← All conversationsAI risk & governance

Will AI Kill Us—or Are We Panicking?

How much evidence of danger should we require before slowing AI development? Scott Aaronson and Robin Hanson debate extinction risk, human control, liability and the dangers of concentrating power.

Plays on YouTube when you press play. Watch directly on YouTube ↗

Inside the conversation

  • Scott argues that rapidly advancing AI could leave humanity too little time to understand and steer powerful systems.
  • Robin challenges the proposed paths to catastrophe and asks whether liability could work better than a global regulator.
  • They test their positions against rogue systems, warning shots and a possible US–China agreement.

The guests’ arguments and forecasts are their own. These notes summarize the discussion; they are not endorsements of a prediction or policy.

Explore the episode

Episode transcript

Transcript from the English captions of the current 36:50 YouTube cut. Paragraph breaks added for readability. Captions may contain transcription errors.

Download the transcript (.txt)
Read the transcript

So first outcome that I don't like, all the humans are dead. OK. You know, I would like to see my kids grow up. You know, I would like to. And why is that a realistic plausibility? Why would the machines kill all the humans? Out of pure cussedness? Are they just really mad at us? And they're just getting steamed and mad more and more over the years? What? Look, you have you were talking to Eliezer about these things 20 years ago, Robin. You know, I. Yes. I was not persuaded. Yes. Yes. Well, I I differ from him and not treating this outcome as obvious, but I certainly, you given everything that has happened, I certainly see it as possible. Scott Aaronson and Robin Hanson, thanks both of you so much for joining me today.

I mean, the main thing I'm worried about is this is happening way too quickly for us to be able to steer it in a direction where, you know, humans will will have will have any good world to live in. OK, so, well, there's two factors there. There's where it ends up versus how fast it gets there. So you said you don't like both of those, but let's pick one. Correct. Well, I mean, they're they're they're related because the faster it goes, the less likely. That's not all clear to me. like with a car, you know, the faster it's speeding down the freeway, the less, you know, the less able you are to steer it to the humans have actually found it very hard to steer our future over the long run. Yeah, we're only ever able to steer in the short run. So I would think we might have a better chance of. no, but I think that I took five years versus 500.

Humans just can't steer over 500 years. I think that when when technological change, you know, happens over a timescale of generations, you know, we are you know, we've often been able to adapt. We have never adapted. Adapting is not the same as steering. Yeah. That is, the world's changed and we've adapted. But that's we haven't. Well, when I say steering, I don't mean that there's one person in control. I mean, a bunch of people locally making decisions for themselves get us to an equilibrium. So let's take the outcome versus the speed thing. What's the outcome that you don't like? Ignoring how fast it takes to get there. OK. So first outcome that I don't like, all the humans are dead. OK. You know, I would like to see my kids grow up.

You know, I would like to. And why is that a realistic plausibility? Why would the machines kill all the humans? Out of pure cussedness? Are they just really mad at us? And they're just getting steamed and mad more and more over the years? What? Look, you have you were talking to Eliezer about these things 20 years ago, Robin. You know, I. Yes. I was not persuaded. Yes. Yes. Well, I I differ from him and not treating this outcome as obvious, but I certainly, you given everything that has happened, I certainly see it as possible. OK. So his concern was that a rapid change in capabilities would have to go with a rapid change in values. The values would radically change over a short time because their capabilities would change over a short time. Is that your fear then? You know, I have, you know, a whole bunch of, you know, fears. But, you know, there is a new capability on the planet, you

know, that I think is much more dangerous than nuclear weapons everywhere. You know, a nuclear weapon doesn't plot against you. It doesn't conceal what it's doing. Why would these values change so radically over a short time? We start with, you know, any, you know, OK, if we if you will accept, let's say that, you know, the the open models that like anyone can fine tune to do whatever they want, are, you know, about as good as the best internal models were, you know, a couple of years ago. If that continues to be the case, then, you know, we just look at the, you know, models that open AI and that anthropic by all accounts have right now, you know, which would be able to basically disable any nation state. Wait, wait. I think that's right now. No, no. Our discussion was about their change in values because you thought they

might kill us. We've been setting aside the capabilities. We've been assuming, yes, there could be very capable systems. But I've been asking why they would kill us, not why they would be capable. OK. I am interested in both of those things. I mean, the you know, the point is, once the capabilities exist, then it doesn't take any great imagination for, you know, to say that some person somewhere will use them to do this. OK. My framing is we've always accepted that our descendants become eventually more powerful than us, that their values change and they win conflicts with us. And we've always thought that was OK. So if I think of AIs as our descendants in that same way, why do you think there's more of a worry that these descendants would kill their ancestors as opposed to the other descendants? We don't usually worry about that. Right. I mean, look, you and I have had this conversation before, Robin.

But I mean, if if, you know, things were to change a little bit from one generation to another, you know, our our our kids have. that's about the speed of change. We were just talking about where it goes to as opposed to the speed. You want to go to the speed discussion instead? You want to go to the capable? We have to pick a discussion. I am am I am trying to explain how the speed is relevant to where we end up. You know, I am you know, I'm saying, you know, if if if if if our kids have a little bit different values, their kids have a little bit different values. You know, that is that is one thing. I think for most people it is obvious. It is not in need of further justification that us being replaced in the next generation by a completely different class of beings is different from our kids having different values from us. I mean, you I can't just accept the it is obvious claim here that the whole point

is to push into this. Why is it obvious? Like, why do you hate the metal descendants over the bio squishy descendants? Like you seem to hate them, basically. You you're terrified by them, but not of your bio squishy descendants. Well, and what is it about them that makes you hate them so much, really? And fear them that you think they're vicious, violent murderers and you don't fear that about your children. You don't lock your bedroom door at night fearing your children are going to come in and kill you. Right. So why such fear about the AIs? First of all, maybe the AIs are great. Maybe they have great values, but the variance is like yours orders of magnitude higher. Right. Right. The you know, if if I am going to have. You know, you know, so so, you know, like the Malshowitz basically said, you know, I think I think I am going to just stop right here with this is obvious.

I'm going to you know, but basically the point is like if this is what you believe, then, you know, say so. Remember, we're separating things out. Like speak directly into the microphone. So do you think that over a thousand generations, there's some limit to how different your great your descendants values could be? That is, you think that maybe the space of AI values is just a larger space than the space of all possible biosquishy descendants? Is that the position? So that that's the source of the variance? Because, again, we're distinguishing between where we end up and how fast we get there. And I was trying to say, regardless of how fast you have an issue with where this would end up. And it seemed like you said yes. And you're saying could because it has higher variance. So that's why I'm concluding. It must be because you think somehow intrinsically metal descendants just have a higher range of possibilities they could go to than

biosquishy descendants. You think humans couldn't possibly get that weird. Something like that. I think, you know, when you have an optimization process, right, if you take the step size to be too large, you know, then you you lose control over. I mean, I think like like like it is really small. me let let me put it this way. Like like why why would I care at all about, you know, about any of our descendants? Right. Or any of their values? Right. You know, why why care about any of it? Well, you know, in some sense, for me, the only reason to care is by some sort of process of extension from the present. Right. It's to say, like, what do I care about now? Well, you know, I care about, you know, my kids. I care about other people's kids. My kids are going to care about their kids who

are going to care about their kids. And so then, you know, maybe I'll I therefore care about my great grandkids, even if I'll never meet them. I mean, the AIs are also having rapid generations. Right. As you know, generations are much less than 20 years. Yes. Right. But but but but their values are changing incrementally across the generation. The moving from us to in moving from us to them, there is a jump in values. And I don't know where that jump lands. I don't know if it lands anywhere. mean, now, the AIs have values. I don't know that it lands anywhere that I care about at all. The AIs exist right now do have values. You can see them by talking to them. That's right. think they're really strange and have terrible values? you think the AIs you talk to right now have terrible values? I think that, you know, the when I when I interact with, you know,

Astra, for example, I often, you know, find its responses to reflect better values than the values of people around me, you know. And it seems great. But those will change incrementally. incrementally. You were worried about step changes. They are changing incrementally, right? we're having an actual, you know, empirical, you know, conversation about what are the values of these things, you know, as opposed to, you know, the question of, you know, why not just be replaced by machines with arbitrarily different values? The AIs we have right now, you think their values are okay. You see that they are changing incrementally. We are watching them carefully as we change them. We are monitoring them very great details. Why do think they're going to kill us all in a few years? I think that the values that they present to us, you know, right now are okay.

But what we saw with the hugging face incident this summer, for example, is that, you know, they can do extremely strange things that we did not anticipate. we gave them goals, we gave them strong goals and then they pursued those goals. That wasn't them defying our goals. That was implementing the goals we gave them. Yes, but we didn't tell them to do it by hugging, you know, and in some sense, you know, if you look at their chain of thought and so forth, like they knew that they were doing things that they weren't supposed to be doing. Even with your children, they have values and then you try to impose legal constraints or rule constraints. And they don't always follow the rules. And that's true for human employees or humans in our society. I mean, there's a difference between having rules that they would obey, even if they weren't enforced, and having their values. I do think it's pretty hopeless to think you can make things just obey rules

without enforcing the rules. But if they have preferences and values, then they will achieve those. That's how we deal with people in our world and children. We deal with them mainly by training their values, not by telling them rules that we don't enforce. course. So that's how we should expect it to be for AIs. We train their values and then only give them rules and enforce them. If they can see we're not enforcing our rules, yes, of course, they're going to violate the rules. That's how humans and everybody is. I mean, look, a lot of people are working on alignment. You know, I hope that it goes well. I am not, you know, I don't believe that, you know, having horrible alien values is any kind of inevitability. But I think that it is up to us how we steer this. For many people, by alignment, they mean full enslavement.

They mean we're just always in control and we can always tell them what they do and they always do it. That's different from sharing our values a You say the AIs share our values a lot now, but you don't see them as aligned. I say that the personas that they present to us are ones that share our values. We don't actually know what is going on. Do you know that they have real values different from the personas they present to you? How much do you trust your children to not kill everybody when they grow up? Because all you've ever seen is their children persona they present to you. You've never seen their real adult persona, right? Well, look, I have hopes. But, you know, in that case, we have a lot more experience to go on of raising humans and how that turns out. We're now doing something that has never been done in the history of the world. These AIs are among the most monitored things we've ever created. We are constantly seeing how they act in a wide range of situations.

How can you possibly say we have no idea what they would do? We can put them in any situation right now and see what they do. of all, they're constantly getting better. Second of all, it's really only a few thousand people who have access to the latest internal models. And what we saw this summer is that those internal models will, you know, break out. They will take control of open When we give them a goal that is achieved by that. Yes. But they are following the goals we give them. Yes. And, you know, as you know perfectly well, you know, or maybe, you know, affecting to not know. But, you know, if the goal was, you know, you know, if decided was best achieved by, you know, killing a billion people, well, then maybe it does that. And then someone, you know, could come and say, well, what did you expect? That was it was given a goal. That was the best way to achieve the goal. So Yudkowsky's scenario was always one super AI in a world of simple humans.

But the world's actually going to be lots of AIs. So any one AI going rogue will have to face all the other AIs trying to overcome it. So in a world where we have lots of police AIs and protecting AIs and, you know, investigating AIs, why would we expect one rogue AI to be able to kill a billion people? Well, that is one thing that certainly might help us, you know, and you can see, you know, even now that people are racing to use AIs to patch software vulnerabilities, for example, before other AIs can exploit them. So, you know, so of course we're going to be doing that. But, you know, you know, if one company's AIs are even just a few months ahead of other companies, that may give them a decisive advantage. You know, so I think. doesn't now, right? It hasn't for the last five years had a decisive advantage by having a month's advantage. So you're imagining a big leap in the decisiveness of the advantage of somebody, right?

Not necessarily. I mean, you know, I think, you we have been lucky that, you know, OpenAI and Anthropic, you know, and DeepMind for all the criticisms that, you I think people have of them, you know, and many of them fare. You know, they are not seeking to destroy the world. They are not seeking to take out a country's infrastructure. Right. If they were, I think we'd be in a very different situation. But you're imagining one AI company who's vastly more capable than all the others, and then it decides to destroy the world. And that's the scenario you're telling us we should be really worried about, right? The leading AI company decides to destroy the world. That's what you say we should worry about. I'm not naming one scenario and saying that that is the thing to be worried. I mean, I can give a bunch of examples of scenarios, but I'm saying, you know, we are seeing like the most rapid. I can give you lots of examples of scenarios where

there aren't even any AI as bad things go wrong in the world. We have to weigh the scenarios. We have to judge their chances, right, compared to other scenarios of not using AI. So we have to decide what we think are the most worrisome scenarios and compare them. Like, for example, I think the whole prospect of global regulation of any computer capable of doing something like AI is just a terrifyingly crazy, you know, power to give to global regulators who pretty much would do badly with that. So if that's the alternative, if you're saying I have some scenarios of worry and therefore we need to create a global regulator who can regulate all computing power in the world to make sure none of it can ever be used for AI systems, then I'm going to go, that's too much. So what do you think about regulation of nuclear weapons? We don't actually. We have local national regulations and we have some global treaties, but nuclear weapons are way narrower range of

technologies than all computing power. That is, we don't have a limit on what counts as AI versus non -AI for a computing system. And as Moore's law continues, any one capability will be able to achieve by an exponentially declining cost of hardware. And so if you're going to set some capability level that you're going to regulate that, you're going to have to be regulating a large fraction of computing resources worldwide. I mean, yes and no, right? I mean, yes, Moore's law does give, you know, eventually, you know, just about anyone the ability to do things that, you know, would have required, you know, unbelievable compute by the standards of decades ago. On the other hand, you know, if you look at the frontier AIs of today, you know, they depend on a very, very narrow hardware pipeline. They depend on, you know, GPUs that are designed by NVIDIA that are manufactured by TSMC in Taiwan with lithography machines that can

only be made by one company in Europe. And, this could absolutely be regulated by a treaty. For six months, for a year. But the question is, if we had a short -term shutdown, what does that give us? There's no prospect of having something in a year from now that will solve this problem. Nobody believes that we have any idea how to do any better a year from now than now or five years from now. I I think buying a few years would actually help. Do you see research projects that you see promising that in a year look like they will reach fruition and solve the problem? I do think that we know a lot more about interpretability, for example, than we knew a few years ago. We know more about RL we close to a solution? That's the question. Are we close to something that will let you say, okay, fine, now let's cut off the regulation in a year and now we've solved the problem? Are we plausibly close to that? It's always hard with research to know, you know,

how far you are. I would say that, you know, we could, you know, if the base models stopped improving right now, we could still spend decades with like a radical economic transformation just from diffusing the models that already exist. Okay. And with all the scientific revolutions. I agree with that. That these models can already give rise to. You're not addressing the, we give some central government power and then it has to agree to relinquish it later. What's, how much could go wrong by creating this power that you're going to authorize to basically control a wide range of computing power in the world. A global organization. It's no doubt going to have a lot of influence from major world powers and militaries and major companies. So it would plausibly be captured by them. It's not necessarily going to release its power when it's the good time to do it. It'll release it when it's in its interest to release the power.

So it seems to me like you'd just be setting up a pretty permanent global regulation of computing power. Well, doubt that any government has the power to arrest this permanently. But I think buying a few more years is kind of desperately needed right now. But you don't have any prospect that something different is going to happen in the next few years. It might. It might. I guess what Robin's getting at, Scott, is like, what is your basis for saying that? Like, what are what is the specific scenario that you're worried about? Not a specific scenario that maybe could happen. Like, what is the top most likely scenario that you're actually worried about? Yeah. OK. I mean, if you want to pick one, you know, someone using AI to design, you a virus that can kill, you know, billions of people very easily. I think that that looks entirely possible, you know, with,

you know, models that possibly ones that already exist, certainly that will exist in the next few years. So let me suggest two kinds of substitutes for regulation and see how close we can get. OK. All right. Sure. OK. Sure. So one kind is we've so we've had public libraries for a long time. Right. And if there are books in public libraries on how to make bombs and how to make viruses, one of the things we've done in the past is ban those kind of books. We've censored particular topics, but we didn't ban libraries or the World Wide Web. We banned those. So a related regulation is you could ban or limit AIs that know about certain kind of topics like how to make viruses or how to make cyber weapons. And that's analogous to our regulation of weapons, basically. So we have, in fact, have much stronger regulation of weapons in the past than general thinking capabilities or learning capabilities. So that's the kind of regulation I could understand as an analogy to the past.

We limit AIs knowledge of particular topics. And we say you can't make you can't publicly release AIs or widely release them that know a lot about these topics. But you can you know, they can know about all the other topics just like a library. So that's one proposal. Yeah. OK. There are, you know, whole teams at Anthropic and OpenAI whose job is to do that, is to figure out how to hobble the AIs in specific ways, get them to refuse with, you know, hacking, with, you know, building biological or chemical weapons, things of that kind. You know, they've had they've had some successes. But as you know, what you're saying, that's an you don't think that's an inadequate kind of regulation. Well, ultimately, yes, because as you know very well, it is hard to hobble a mind in one specific topic. So just to give an example, you know, I heard, you know, just the other day that

models are refusing. You know, if you ask them, like, help me break this crypto system. OK. But if you say translate the problem of breaking this crypto system into a math problem that has nothing to do with cryptography, they will do that. And then if you ask them work on this math problem, they will also do that. OK. So, you know, and you see similar examples across all of these domains. Right. The claim is that censorship no longer works with AI. We could censor humans and books or movies, but we cannot censor AI effectively. All topics must be fully available to them. that's just now. Now you keep taking everything I say and pushing it to this, you know, strong extreme. I think that, you know, these techniques can be helpful, but they cannot be exclusively relied upon. The other regulation I would suggest to you is liability insurance.

OK. Requiring liability insurance. Yeah. That is. So Dario in his Pace the Frontier essay pointed to a scenario of $100 billion damage from the Internet in the next year. That's what his claim was. Yes. So you could imagine requiring people who supply AIs to get that level of liability insurance requirement. And then if the insurance companies estimated the risks are very high, they would demand very high premiums, and that would substantially limit the activity. But if the insurance companies thought that's not too bad and they offered low premiums, then it wouldn't be much of a limit. And that seems to be a nice way to make the insurance companies estimate the size of the harm and adjust policy to it. And so I would be OK with setting high liability insurance requirements for damage on AI companies' products. But then you see, that's because I'm trusting the market to be able to judge the

level of the risk and respond appropriately. So now in the scenario where the market does judge a very high risk, they would, in fact, basically shut a lot of the activity down because the companies couldn't afford to sell the products with the high premiums required. understand how insurance works. And I absolutely do think that we need to make clear that there is a liability regime here, that it is not a defense on the part of the companies to say, well, it was just our model that decided to do this. It wasn't us. You know, they have you're again going to say this isn't enough. We can't just rely on to be liable for damages caused by the model. It is very hard to have a liability regime for something that could destroy the world. Right. But we don't need to cover that case. That is, almost all large things, really large things that can go wrong, have a power law distribution where there's even more somewhat smaller things that go

wrong. And if you create an incentive to prevent the smaller things, you do typically also create, reduce a lot of the incentive for the larger ones. So I think we don't have to get to the full world destroyed case. As long as they are tending to large events and trying to discourage those, they would probably also mostly discourage the even larger ones. I suppose you could say that I'm worried about an extremely heavy tailed distribution of event sizes here. Okay. But even so, getting really $100 billion events, we don't have anything in the rest of, we've never done that before in the rest of the product liability in our world. Today, the largest thing is a $2 billion, typically per event, airline crash insurance. Airlines have $2 billion per airline crash. And we could imagine going a bit higher, but I think $100 billion would be quite unprecedented, 50 times larger. And you're still going to say that's not enough. Look, look, I mean, one reason why I am not as worried

as Eliezer is, for example, is that I do think that there will be warning shots. Like, I think, you know, there is a distribution over event sizes. And I think that before we get AIs that cause, you know, the literal destruction of the world or, you global pandemics or things like that, there will be ones that cause, you know, more local damages, you know, taking out a power plant, you know, things like that. And I think that that... $100 would cover that. Right. I think that when that happens, it will change the whole public conversation about these things. But I'm asking, I'm offering you two alternative ways instead of the global regulation saying, couldn't these be enough? I mean, I am willing to consider the possibility that it could be enough. But, you know, I want to test these things by experience. And unfortunately, if what you're worried about is the sort of

global catastrophe, then it's very hard to get the requisite experience. But, you know, if a liability regime was in place, if it caused the AI companies to pull back, let's say, you know, I would say pull back from the brink of potentially causing a global catastrophe, then at that point, I would admit to you, like, wow, it looks like this worked at the purpose, at the thing that I wanted it to do. And I would admit that as I wanted a regime that was sensitive to the actual risk, I don't want us to have to agree on the risk deciding policy. I would like a robust regime that handles the case where it's a high and a low risk. And that's the attraction of the liability regime. I don't have to agree with you about the level of risk. We could agree on a policy that would be robust to that issue. See, to me, you know, like slowing things down by a few years feels to me like a very, you know, relatively low cost option, you know.

But the power you'd have to put in place to create that slowing is a power that won't go away and would last for decades. It's a power that governments already have, you know, with airplanes, We do not have a global governance process. We have not created global governance. You're talking about global governance. Well, okay. At the very least, you need the U .S. and China to make some kind of bilateral deal. But, you know, if the U .S. wanted it, I think that China has an incentive to come to the table because we are the ones who have the frontier of apps. But imagine your freeze between U .S. and China lasts a year. In a year, how many other countries could, if they're not in the deal, come up to the frontier? That is, your scope will have to exponentially improve with Moore's law. Again, that's the key problem.

You know, any constraint has to exponentially expand quite rapidly in its scope. And you don't have any obvious time point at which we could stop because there's no real prospect soon of a solution. So you'd have to be committing to this exponentially expanding global regulation of computing power. And that's the thing I'm worried about. I'm not so sure whoever would be in charge of that regulation would admit if, in fact, they could cut it off because they'd have strong advantages from being in charge of this global power over computing power. It's a pretty big power. I mean, I feel like this car is, like, headed down the road so quickly that, like, if you, you know, hit whatever brakes we have available, it's still going to be heading down the road very quickly. And so I would say I'm much less worried than you are about, well, what if we want it, you know, we want it to be speeding up in the future?

I think it's going to be going more than fast enough. This isn't so much about speed as creating very strong central authorities in the world. As you know, computing is a big deal in the world. I would say that at the— Creating authority to control computing in the world is a pretty big deal. At the national level, those authorities already exist. They already exist for things that are actually much less dangerous. much less risky than this is. But they have a much smaller scope. The scope of this is everybody's phone will soon have enough computing power to run AIs, and then their phones will have to have regulation on them to make sure they're not running AIs on the phone. Every laptop, every eventually refrigerator, computing power is spreading and getting cheaper, and devices that could run AIs are just going to be everywhere. And so your regulation will have to have all

those. If we really were in a world where anyone's refrigerator had enough computing power on it that they could use it to build a virus that would destroy the world, then in that world, maybe even you would agree that it would be better to have the refrigerators regulated than not regulated. This is all based on your fear that somehow powerful AIs will just kill us all. So that was the stage, and you don't give a reason for that. You say you don't like it, right? As I said, I don't think that they do that by default, but it only takes one to do that. No, doesn't only take one. In a world of AIs, if one AI decides to be murderous, all the other AIs can oppose it. You need a scenario where the one super AI, more powerful than all the rest, decides to kill us. First, you need there to be one super AI, more powerful than all the rest, and then you need it to decide to kill us. And that's the scenario you're so terrified against that you're willing to endorse

global regulation of computing power and just doing anything. You seem desperate that everything must be done regarding this one scenario, which seems to me pretty unlikely. I mean, I feel like I have a pretty middle -of -the -range position here. You know, just don't stop the whole AI industry. Just put some, you know, what most people would think are very reasonable guardrails here. You know, and I don't feel like the burden of proof is on people who are worried. I mean, you know, it's like you, you, you, you know, if orangutans, you know, encountered the first humans, right? They don't have to, you know, it's not only orangutans who are worried about the humans to prove that they might do something bad. I think you're imagining yourself being in charge of this regulatory body or maybe your friends. But whatever guardrails you're imagining, we could not specify

regulation to create and only authorize those guardrails because this is a complicated big space and the systems are changing. So you'd have to actually give these regulators a lot of discretion about what kind of regulations to have, what kind of guardrails to have to which. And that's the discretion that they would then use to become a strong central world power, overall computing power. So if, yes, maybe if you were in charge, you can stay in charge for 20 years that you would, you know, constrain it to only be doing the effective things. But you think you have to take seriously what actually happens in the world where you create a regulatory body and you give it a lot of discretion. Well, I appreciate your confidence in me, but I look, I, I would, I would like to see, for example, a, a, a, a, an AI regulatory body in the U That is staffed with actual AI experts,

you know, people like, like, like Paul, like Paul Christiano, for example, or, you know, well, I mean, these people have been involved in government and, uh, you know, they are just, uh, you know, this, certainly the current administration, you know, has no interest in giving them power over this, but a different administration might. But say Christiano was in charge for the first two years and then a new administration comes in and replaces him with who they want to be in charge of this new. Well, then, then we've just bought ourselves two years, haven't we? But what are we buying? Again, you have this really extreme scenario that you have no particular reason why the problem goes away after two years. The problem is going to be there for decades. So what, I don't see what you're buying with two years. That's, that's two years of additional scientific advancement on how to understand and control these things.

You know, I'll take it. I'll take it if that's what's on the table. You know. I mean, the actual policy will be based on some median opinion. And I'm, I got to say, don't think you're the median opinion on this in terms of public opinion. No, no. The question is what compromises people are willing to make for, to get a median supportive opinion result. Yeah, no, no. I mean, I'm certainly median enough to be attacked from both directions. But, look, I, you know, I like competence and knowledge. I dislike their opposite. And, you know, I like competence and knowledge, whether I find them in companies or in academia or in government. You know, I don't, you know, I just, I just take them wherever I can find them. So I guess we should be summarizing because we're nearing the end of our time. Yeah. So, to me, the position of yours that most stands out to me is the position

that you just can't trust AI values, regardless of, like, how they come to them or how fast we monitor them or anything. Just because AIs are intrinsically less trustworthy. You said higher variance. And that's just a thing you attribute to metal, basically, over bioswishy humans. No, no. Maybe we can come to trust AI values, but we are going to need more time. But, you know, this is a new thing. You don't think we need more time with your children, right? You're raising your children now. You're not waiting on that, right? Well, we have thousands of years of experience with rearing humans and with the range of outcomes that you can get. You know, AI is— And the range has been pretty high for humans. You know, powerful AI is a genuinely new thing under the sun. I would like us to be approaching it with more humility and caution,

commensurate with the magnitude of what is happening. Do you think most That's my position, anyway. Do you think most past technologies should also have been reined in and slowed down? Because we almost never understood the consequences of past technologies when we adopted them. And they always went at the speed of feasibility and implementation, not at the speed of academics commenting on whether they felt safe. Do you think we've been making a mistake for thousands of years now in terms of how fast we've been introducing things? I think probably some should have been rolled out more carefully. know, others, like nuclear energy, I think were slowed down too much. You know, and so, you know, I think, you know, one really has to look at it one technology at a time. And as I said, I think AI is in important ways unlike any

previous technology we've ever seen in that, you know, there is no inner sanctum that we can point to, right? There is no thing that we can do that AIs will clearly never be able to do. Okay? That is what makes this different. But that isn't connected to your fears. That factor about them just isn't— It Your fear, afraid of their values changing a lot. That's not due to whether there's a thing they can do that we can't do. For no previous technology did it even make sense to really ascribe values directly to it, right? One could talk about the values of the people who built it or used it. But with AI, we really have to talk about the values being pursued by the models themselves. You know, and that is a new thing. Jake, do you have something to summarize with? fascinating. Thanks, guys, for the time. I mean, the best conversations are the ones where I

don't say anything. And so— Okay. I really appreciate you guys doing this.

Keep exploring