← All conversationsAI risk & governance

Will AI Destroy Humanity?

Robin Hanson and David Krueger challenge each other on a frontier-AI moratorium, warning shots before catastrophe and whose expertise should guide predictions about the future.

Plays on YouTube when you press play. Watch directly on YouTube ↗

Inside the conversation

  • Krueger argues for an international moratorium on frontier AI development before humanity loses control.
  • Hanson questions the evidence for halting development and considers AI as our descendants.
  • They debate biological and cyber risks, surveillance, and the burden of proof for extraordinary policy measures.

The guests’ arguments and forecasts are their own. These notes summarize the discussion; they are not endorsements of a prediction or policy.

Explore the episode

Episode transcript

Transcript from episode captions. Paragraph breaks added for readability. Captions may contain transcription errors.

Download the transcript (.txt)
Read the transcript

And I can tell you when I got into this field, there were virtually no experts who shared my concerns. And now we have the leading researchers, the people who invented the basic technology, the people who are building the stuff. Those are not the relevant experts. You're making predictions about very rapid economic growth, about the nature of society. You need experts in those things. Like, I mean, I agree that there are other people whose expertise is relevant. I think it's absurd to say that people like Geoff Hinton and Yoshua Bengio are not relevant experts in the topic of what the future of AI is going to look like. I agree that you can't just take experts' predictions

as like, you know, the ground truth or something. But I think it more than presents a case for concern. I'm happy to admit a case for concern, but I think I have engaged with this area for a very long time. I think I understand the basic arguments that you have, and I don't buy them. Yeah. I think they're not persuasive. All right, Robin Hanson and David Krueger, thanks both for joining us today. So for those who don't know, Robin Hanson is an economist at George Mason University, and Robin is the author of The Age of Em. And David Krueger is an AI researcher who trained under Yoshua Bengio. And David, your work focuses in part on preventing AI from posing an existential threat to humanity. Can you tell us a bit? About that, it's my understanding, David, that, you know,

part of what you argue is that we should halt advanced AI development before we lose control over it. Is that accurate? its current pace lead to the deaths of billions of people, David. Yeah, uh, you know, there's, there's a couple different reasons for this. I think there's oftentimes this discussion is framed in terms of the death of everybody. And I think it's important to say that that's a realistic possibility on the table. Um, but, you know, it's kind of like also a question of maybe they'll leave some people alive, um, or maybe, you know, we'll have billions of people die because AI enables people to create, um, new forms of weapons or offensive technologies or makes it much easier to build, uh, biological weapons or things like this. Um, so I think that's one, you know, thing that is already probably enough for most people to call the whole thing into question and be like, why are we building this thing?

Why are we doing this? Like, if this is, you know, going to risk killing billions of people, it's crazy to be racing to build this thing in the way that we are. Um, and I think that's kind of, you know, there's still some discussion, but I think probably Robin wouldn't disagree that those kinds of technological capabilities are something that AI systems will, um, will have in the future. You might disagree about, like, if that will actually make these kinds of attacks more likely to happen. Um, but just like, that's, that's one part that we might want to dive into. Another, another piece is the sort of loss of control thing where we build AI systems that don't share our values and that want different things than, than we want. And so we naturally come into conflict with them, um, and they end up, you know, dominating the environment and destroying our habitat the way that we've done for other species. Or we end up in an actual conflict with them where we realize that we're losing control and we try and shut them down and pull the plug,

so to speak, and they don't want that. And so they have to disempower us. So which of these two would you like us to discuss first? The idea that what David said, which is that AI could lead to the deaths of billions of people and that that being a risk is the reason why David. Okay, but so far all he said is that there are weapons in the world and AI might manage to build and use weapons. So it's the existence of weapons. That's the basis of the postulate that AI will kill billions of people merely that weapons exist in the world. And in the past weapons hasn't killed most of the world, but if AI has weapons, they would. Well, you need to add more to the argument to make that possible. So what else is in the story? Yeah, so I'm pointing out two different things. One is the risk that AI facilitates the creation of weapons of mass

destruction so that many more people are able to build them much easier. And then there's this question of, like, if there's a way to defend against those kinds of technologies, which I'm pretty pessimistic about. That's one point. Then the separate point is, like, are these AI systems going to be aligned enough that they actually continue to function as tools at all? Or are they going to end up going rogue, as we've seen this summer, and pursuing objectives that are different than the ones that we, you know, want them to and not respecting the constraints that we try and impose on them? So I think these are two different, you know, things that I believe are both very serious concerns. Well, long ago, in the old world of emails and, you know, news groups, there was this problem when somebody would make three points and then somebody else would respond with each of those points with another three points. And then we expanded too many points. So I think we need to stick with one thread here.

So please just pick one at a time what your issue is, and then we'll talk about that particular one. Yeah, so I named two, and then I think you asked Jake to pick one, but I was trying to clarify for Jake that what you had described, Jake, was not choosing between these two. So I'm happy to pick one, but I think also we should maybe leave it up to you if you want to choose. No, David, please. And Robin, you can confirm that David's choice is, in fact, like, a singular. I mean, since I'm less worried, then he should point out what his worries are, and then we could focus on his worries because otherwise, you know. So I think we, you know, just we'll move down one of these with the understanding that if either of them is, in fact, concerning, then we should be concerned. So let's start with the weapons one, right? I'm curious to know, actually, Robin, if you think that we are on track to have AI systems that are extremely entertaining,

engaging, and helpful assistants that create, you know, very realistic video avatars that are capable of helping people, you know, synthesize pathogens in their living room and making that, like, you know, a fun and enjoyable and easy process because you can set up cameras that they watch everything that you do, and they can just tell you at each step of the way exactly, you know, what you're doing right and wrong. So I think you're referring to what's Bostrom called a vulnerable world hypothesis, that over time, more destructive weapons become more accessible to more people, and therefore, just in general, the world becomes more risky. This is independent of AI, really, just that the nature of more people having more access to technology that allows them to access more weapons just makes for a world wherein offense wins out more over defense

because there are just these very powerful offensive weapons that are possible in principle. That's what I hear you saying. Is that not what you're arguing? I would say I'm making a more specific argument. You know, I think this is in line with the vulnerable world hypothesis, but I'm pointing at a very particular thing that I'm expecting to happen by default in the not-too-distant future. I think there is a general question of, yeah, like, what is the sort of offense -defense balance with new technologies? And there's a—I think the vulnerable world argument is, like, a pretty compelling one that, you know, at least creates a strong case for concern that we might develop, you know, such a technology where we can't really defend against it. The reason I would point to that is because we actually have data more on that.

That is, the vulnerable world hypothesis is a claim that's been true—that it's been true for, say, a century, and that we continue to have this increase in vulnerability. I think it predicts that, in fact, mass murders will get bigger. That is, if, in fact, ordinary people are getting access to more powerful weapons over time, then any one person should be able to kill more people when they decide to go killing people, and therefore the, you know, the median size of the number of people killed when somebody goes wild and killing would be increasing over time. But it's not. So, in fact, access to, you know, machine guns and all sorts of other weapons that many people now have access to, including, obviously, large airplanes you can run into buildings, has not actually increased the typical size of the number, you know, number of people somebody killed, or even the murder rate,

so those trends are against the vulnerable world hypothesis. So then you must be postulating some new trend. But I've got to say, over time, we've just seen a lot of things that affect the overall balance of offense and defense, and overall, it's stayed roughly in the same range. And many particular technologies sometimes make offense easier. Some have made defense easier, but I don't see an overall trend favoring offense in the offense-defense tradeoff, and I don't particularly see AI making a big difference about that either. I might say bioweapons are a category. We might think offense is easier than defense, but in fact, we've had very few bioweapons attacks in history, so that seems to be, so far, a relatively small category of harm. It's not, you would have to increase by many orders of magnitude to be the sort of category of harm we should be focusing a large policy area on.

So the claim is that that is going to change because of AI, and I think that, you know, there's a very clear and compelling case for why that would happen. Now, you could say, okay, well, this is only going to happen if there are open-weights models that people can fine-tune to remove the safeguards, or if models' safeguards are capable of being bypassed as the models get better and better. You could say the models aren't going to reach that point of capability, or you could say they will reach that point of capability. That capability will be broadly accessible, but nonetheless, somehow, we won't see these, you know, extremely damaging biological weapons attacks from rogue actors like school shooter or terrorist types. And I'm not sure which of these things you would, you know, you would say, or if you don't have, like, a detailed argument,

just kind of this, like, outside [unclear] trend thing, which I think is not, like, very compelling to me here. A very basic issue in this space, as in elsewhere, is the degree to which we should regulate on the basis of sort of scenarios people spin about things that could go wrong versus concrete track records of things actually going wrong. And usually, I would say, it's healthiest to look at concrete track records of harm as the basis for what you want to regulate more. And then if theoretically there is, in fact, some reason that things will get worse, that will be realized in the track record. And then we can, once we see an actual track record of change and harm, respond appropriately. So the usual counter-arguing, oh, what if there's something so huge that, like, it has no precedent and then it kills everybody?

Well, then you can't have a track record about that. But what you're talking about are things that would have smaller versions before they had bigger ones. So I might say, you think there's going to be this increase in bioattacks, but let's start to see them before we get too worried about it. Because in the past, many people have worried that there were trend changes that, didn't show up. So, I mean, so for example, 9 -11, a great many people at 9 -11 said, this is a regime change. We're about to have a lot more terrorist attacks like 9 -11. And they wanted to change a lot of policy in order to accommodate that new regime. But in fact, 9 -11 was a draw from the same usual distribution of terror attacks. It wasn't a regime change in the size or magnitude of terrorist attacks. And I think it might be appropriate to wait for, you know, actual track record changes. I mean... Yeah.

So I guess I want to pick apart the object -level question of whether or not this is a concern and what the policy response should be. I would agree that in general, this is a good approach to policy, is wait until we've actually seen, you know, harm and don't just go based on speculation. On the other hand, I think, like, that can't just be the approach that we always use. Because if we have, like, strong grounds for suspecting that something might be catastrophically harmful, we don't actually want to wait for the catastrophe. Here you've said that, like, you expect that we would see smaller -scale versions of these kinds of attacks before we would see the large-scale ones. I think that seems really overconfident to bet on for a number of reasons. One is that I think the, you know, the size of a successful bioweapons attack

seems like it can vary quite dramatically. I mean, you know, with COVID versus the other sort of, like, you know, SARS and these kinds of things, right? Like, I think from the inside view, you could have predicted at some point that COVID was going to be really big, but it seems like, you know, there wasn't, like, a really intermediate one between these pandemics that killed, you know, hundreds or thousands of people to COVID that killed millions of people. That was a pretty steep jump. And I think that's important to recognize, you know, that that actually happened in that case. We, in fact, have had a data series of pandemics like COVID. Oh, sure. And we have a rough power law distribution of them. We have used that track record to roughly set our concern about pandemics, and that's been roughly appropriate. But that's an actual track record of things to respond to.

Right. So then, so why would a new pathogen that is intentionally introduced as a bioweapon, so something like COVID, but that somebody makes on purpose and releases on purpose, why would we not think of that as another draw from this distribution? It sounds like you're saying we should expect it to be smaller magnitude, or that you're, like, fine with just taking, you know, waiting for there to be, you know, a bioweapons attack that could be as large as COVID before we do something to prevent it? As you know, militaries in the last half century or so in the Cold War did create bioweapons. So bioweapons aren't a completely unknown thing. People, in fact, created bioweapons, and we have a rate and difficulty of them creating bioweapons. We have a track record of the creation of bioweapons, but we don't have much of a track record of their release

because, in fact, very few people have ever found it in their interest to release them. So you're postulating a new regime of inclination to release bioweapons. No, I'm not. That's what I want to see the data for. I'm postulating a new regime of access to bioweapons. I'm saying right now it seems like there are people on the planet who would release a very destructive bioweapon if it was, like, very easy for them to do so. And I'm concerned that we're on track to make it very easy for them to do so. And I feel like, you know, I've kind of presented arguments for why that would be the case. And I feel like you haven't debated, you haven't disputed that this would be the case. You've just disputed that we might see some warning shots. And so I'm like, OK, but if we agree that that's where this is headed, then why do we need to wait for the warning shots?

Because I don't believe you in your analysis. The whole point is this is a complicated space. But, like, where do you disagree? Like, is it just you're kind of, like, generally skeptical of predictions or, like? There's a lot that goes into that. Yeah, please. Robin, do you agree with David's view that bioweapons will be easier to access by more and more people? I mean, there's two parts there, right? David believes that there are people in this world who, if given the opportunity, would take bioweapons to mass murder. Fair enough. Well, maybe there are. OK. But the second part of that, Robin, do you also agree that it is the case that all of those people now can more easily access bioweapons? And do you agree with that?

Not now, but in the future. Yeah. Well, this is a complicated space of offense and defense. That is, people will also be finding it easier to monitor other people's behavior for doing things like this. They will find it easier to set up responses. Technology will allow both offense and defense to advance here. And so it doesn't make it completely obvious what the net effect of technology on the different sides of this game are. And, you know, even if what you say was true, what we would have is a gradual over time ease. And so there's a distribution of people in terms of their ability to do these sorts of things and their inclination. And then we'd be moving along that distribution. So you would see a rate of increase in people trying to do these things. And most of them, when they try to do it, would fail because it's, in fact, hard.

And you would see near misses and failures of various sorts, increasing in rate until you got, an actual success. Those are the sorts of things I'm suggesting that when you have a complete speculation of a whole entirely different drastic scenario, that instead of having a new drastic regulatory regime of a very widely usable technology that would be very intrusive, we wait and see some actual examples of the problem that you're postulating. And add one tiny piece to that, David. Robin's also saying that your theory is not accounting for the fact that defense will grow as well. Is that fair that you aren't, David, you're not accounting? No, I don't think that's right. I mean, um, Are you accounting for the fact that the defense mechanisms will grow? Yeah, absolutely.

I just think, like, I disagree that this is necessarily going to be gradual. I disagree that this is necessarily going to give us warning shots. And I disagree that the defenses are necessarily going to be sufficient. I think, you know, when I have spoken to people, I've talked to maybe like 50 people about this. And I'm always asking, like, why are you not worried about this thing? And I really haven't gotten any responses that I found very compelling. When I talk to people who have spent a lot of time thinking about how to defend against this, you know, they're like, I think if we pour billions of dollars into this and do like a rapid research program in the next couple of years, we can probably get it to the point where these won't destroy society, but we'll still have COVID level pandemics. So that's that's like, you know, somebody who I think is pretty credible, who has looked into this quite deeply. That was their take. And that's the most optimistic, you know, I've felt after,

you know, all of these conversations I've had with people. So I just think like I would really want to, you know, see why this prediction seems wrong in more detail. Like I think we're having a lot of our disagreement here is more just about like inside view versus outside view reasoning. And I think, you know, the inside view reasoning, from my point of view, has been quite strong on, you know, like AI is going rogue, AI, you know, being difficult to interpret, difficult to control. Yeah, I think there's just a lot of predictions relevant to this that I've made and others have made that have quite a good track record here that are inside view predictions. And then I think there's also kind of, you know, you know, again, I want to separate the question of the regulatory response. And I also want to say, like,

I think there's kind of a question of like burden of proof or like how much evidence or what probability of this happening we're willing to accept. Where I don't think it needs to be like, you know, proven beyond like a reasonable doubt or something that definitely this will happen. I think I don't see a reason to expect that it won't. But, you know, I'd want to hear like more specifically how what the intention is for how to defend against this. I think the best, you know, thing that I've heard from you, Robin, here is like that we're going to be able to surveil people much more intensively. And I'm not sure that that's actually, you know, a good solution because it has its own risks. And if we could do some regulation right now that would prevent the need for mass surveillance, I think there would be a compelling case to do that. So it seems like the structure of your argument for bio risk is pretty analogous to the structure of the argument for cyber risk. And cyber risk is the sort of thing we should expect to see sooner.

So I might say if we see the scenario you're painting play out in cyber risk with AI, then that makes it more plausible. It might play out in bio risk. And that might make me want to move more toward doing what you might want to do that would have worked for cyber risk. In the last few years, we have definitely seen a lot of people say AI is having more access, giving people more access to cyber weapons will produce more cyber attacks and more cyber damage. But then, of course, other people can use AI for cyber defense. And so that is a more immediate test of your offense versus defense claim that if are we actually going to see a lot more net cyber damage from cyber attacks using AIs in the near future? That I think even now, many people I know had claimed that by now we'd be seeing a lot more cyber damage from cyber attacks that we have not yet seen. Are you now predicting a lot more cyber damage in the near future?

Well, I definitely think it's a concern. I think it's a possibility. But I reject the analogy because with cyber capabilities, it's easier to fix the problems when you find them. And AI is actually quite useful for fixing buggy code. So you can run the same attacks that attackers would run on your system, find the flaws, and then use AI to help fix them. There's not really a good analogy for that with bioweapons. I think it is a good analogy because if you cyber attack a bank and steal all the money and now all the money is gone, you can't just fix it. There are cyber attacks that have an outsized impact immediately. They can't just be remediated with code, right? I agree with that. But the disanalogy here is the utility of the cyber capabilities of AI for defense is much greater than the utility of the bio capabilities of AI for defense. This is my opinion, [unclear],

talking with many experts and people who have considered both of these risks. I think if you look at what people are saying about these, there's much more disagreement about how risky cyber will be and whether or not it will advantage attackers or defenders. Whereas I don't think, I can't recall a single time I've heard somebody claim that bio capabilities of AI will advantage defenders. I think people have claimed that we can somehow mitigate the risks or that we should, you know, not do anything about them yet. But I just don't think I've actually heard that claim being made even. So say we were worried about people making bioweapons. Isn't a better place to limit control in the ability of people to take sequences of DNA and produce, you know, viruses based on that rather than try to limit access to AI? I mean, AI is far more widely accessible and far harder to limit access to. If you're going to limit access to something to prevent bioweapons, wouldn't it be better to just prevent it at the point of converting DNA

sequences to actual coded viruses? So I think there's a question of if that's possible and if it's sufficient. And again, I feel like you keep wanting to bring this into like the realm of like policy and sort of saying that I'm trying to, you know, propose a particular policy and then argue against that policy. But I haven't I haven't proposed policies yet. And I'm more just arguing for like this is a risk. Again, my sense is that what you're describing is certainly a sensible thing to do. It would certainly be helpful. I don't think it would provide an adequate defense. It might help a lot, right? I'm not convinced that your problem is as big as you think. So on the market, we should both agree that there are safer, simpler things to do, that those should be done. And then if those are done,

there's less of a need for the more dramatic things you might be considering. I definitely agree with all of that. But that doesn't mean that there's not a need for those other measures. I agree that like if you do something else to mitigate a risk, then, you know, that reduces the need for other mitigations. That's generally true. I suggest we move on to one of your other concerns. Yeah. Absolutely. Oh, sorry. Sorry. But I don't. I'm happy. If you want to. I would also like to vote to move on soon. But if you want to pick on this one a little bit more, Jake. Well, I was just going to say that the I lost my train of thought. So first of all, I think it's a very interesting conversation. I think we've gotten to the crux of it, which is that, you know. So, David, you're trying to delineate these two parts of the argument, which is, you know, let's I think you're saying like,

let's first agree whether it is or isn't a risk. And then if it is a risk, then sure, we can talk about policy or what should be done. But you're saying, hey, I don't want to talk about what should we should be done. I'm just trying to make the case that it is, in fact, a risk. I'm not saying like, what's the best thing or worst thing to do about it. And I think that Robin would also agree that, you know, sure, why not? I'm sure their AI does pose risks just like everything else. So he doesn't disagree that it is a risk. It is the, as you would say, policy that he's disagreeing with because he's saying that with the proliferation of AI capabilities comes the proliferation and augmentation of defense capabilities along with them. And so when we think about, you know, when big cyber attacks happen, is the cause of is the reason for the cyber attack that criminals have access to it is the reason for the cyber attack is the advancement of technology,

the reason why, you know, Bank of America got hacked. And should we look at Bank of America getting hacked and say, gosh, we shouldn't have, you know, built technology in such a way that someone could manipulate this and conduct a hack? No, we don't say that because we look around, we say, OK, on the net, on the whole, you know, all these technology advancements are great. The fact that Bank of America got hacked, I mean, bad things are always going to happen. Thank God we have technology to to protect ourselves. But we never we never look at a bad event and say, well, it's because of the advancement and technology that this bad event occurred. I disagree with that. I don't think that's how it works. I think, you know, I think this is a very simplified picture of the world. I mean, in fact, what we do is we regulate technologies and we, you know, look at their potential for harm. And we also, you know,

try to consider the social impacts of technologies when we develop them. And whether, you know, like as a society, we choose which technologies to invest in and which ones to allow to be developed based on those considerations. Do you agree, Robin, with David's pushback on the disanalogy between cyber risk and bio risk? Like David saying that, hey, it's not a good analogy. I'm not saying it's identical, but I do think it's a decent enough analogy for our discussion. That is, I would say the prediction that we will see, you know, really big bio attacks does predict that we should also see cyber attacks. No, it doesn't. I mean, and it's for the reason that you mentioned yourself, which is that cyber tools are useful for defensive purposes as well. So, you know, that's a disanalogy that is well - But AI can be useful for biodefense as well. Why can't AI be used for biodefense? It's, I just don't see like what the proposal is there.

You know, I've proposed a concrete way in which you can use the cyber capabilities of enough, you know, offensive cyber capabilities of an AI system to patch holes to identify problems. Then I claim that you can also use AI to fix those problems. I think these are uncontroversial claims. I don't see any similar claim being made. Like if I discover some, you know, approach to building a bioweapon to killing people, like I guess we could say like we're going to vaccinate people for that particular one. Yeah. But let's say that. Let's go with that. Let's go with that. Why not? I think this is just not something that to my understanding is very realistic. I think people talk about rapidly creating vaccines for like once a new pandemic is starting to spread and people want to make it very quick to manufacture new vaccines. But I haven't heard anybody proposing let's just vaccinate people against every possible pathogen that anyone could ever make.

Not vaccinate, but I'm saying you're speculating and saying someone could release a bioweapon that kills everybody. That is a speculation. That kills everybody. Yeah. So why is it then unfair to speculate, oh, someone else could make another AI bio defense thing that could save everybody? Like these are both speculations. I think like, you know, we need to actually, you know, look at the details more here. I don't think we can have all these conversations at this highly abstract level. And I think we need to look at, you know, what the actual mechanisms are that we're talking about. And I think, you know, I don't know a ton about biology, but I have talked to people who know more about this, about that topic. And I can relay, you know,

what I've heard through all those conversations as I've done, which is that they don't think that it's analogous to the case with cyber weapons. They are much more concerned. So if you're somebody trying to make a bioweapon, you need many resources that you will take use. And then one of them is AI, but it's just not obvious that AI is the central resource that we would want to regulate in order to prevent bioattacks, because there are many other resources and inputs into the process. So you must have other concerns about AI that are bigger than this, because just to solve this problem, shutting down the AI industry would seem pretty drastic as a way to respond to the possibility that some people might, you know, make bioweapons with AI's help. I think it's more than like, you know, just some random hypothetical possibility. I think I don't really see a reason to expect that this won't happen. It really looks like, you know, when I think about it, I'm just like, well, yes, of course, it's going to create this opportunity.

Of course, there are going to be people who are able to make use of it. I think the only reason that I might expect that it wouldn't happen is if, yeah, as if we basically ramp up mass surveillance quickly enough and in anticipation of this. But as I mentioned, I'm not really keen on that if we can avoid it. And you're also confident that the first time we see any indication of it will be a COVID scale pandemic worldwide. We won't get earlier warning shots. There won't be failed attempts that people who try to do this fail and make a little mark. You're saying no, we won't see none of that. I'm not confident. I think what would be required to support your position, in my view, would be you would need to be confident that it won't be significantly worse than COVID and that we will have these warning shots. And I don't I don't see a basis for confidence of that. So, you know, one one reason being just

that AI is actually advancing very quickly. And it might not be that, you know, there are people who are taking these opportunities that frequently. So it might be the first time that somebody actually has this kind of violent intent and acts on it. We've already advanced AI to the point where you can build bioweapons that are much worse than COVID. And I just don't see why that wouldn't be a possibility. It's not like there's like every month somebody, you know, tries to do a bio attack like on the on the dot. You know, it's it's it could it could be years before somebody even makes the attempt. Yeah. But I think we should talk about the loss of control stuff as well, because yes, that is more on the heart of my concerns. I think the bioweapons one is one

that I think is very significant. And also, you know, yeah, we've talked about it. So just to conclude that point, Robin, do you agree with his last point there on the bio? Yeah. Again, he's pointing to a worst possible case. And then I think implicitly suggesting pretty drastic regulation of an entire huge industry to benefit many ways on the basis of a pretty speculative sort of scenario harm that I think we would see early warning shots about. Let's segue to that then the policy discussion, David, as you were saying, with the controls. Yeah. So I think. I guess I do think it's important to talk about the other risks here, because I don't think the one we've been talking about is the biggest one. And I haven't thought, you know, carefully about what my policy stance would be if I was only worried about this kind of biological weapon type risk.

I think it's a very particular risk, whereas my causes for concern are much more general. If I was worried about this thing in particular, I would definitely want to look into more of the details of it. But yeah, I guess to just just going based on this one. Right. I think I do want to ask, like, what is the actual like endgame here if this does end up being a problem? So if the world that I'm describing where anybody who, you know, right now might shoot up a school can instead release a covid scale pandemic. What do we do to mitigate that? And I think, you know, the answer would be a totalitarian surveillance state. I think that would be the only like way to actually make sure that nobody is doing that if, in fact, AI facilitates that. And you've mentioned that you could instead monitor like the things that you need to build the bioweapons. But I think AI is ultimately going to make it very easy to,

you know, make it easier and easier to build destructive weapons with, you know, whatever you have lying around or whatever you have access to. I think that if we saw in society a higher rate of individuals causing damage to other people, as per the fragile society hypothesis, whatever it was, then we would slowly ramp up our responses. I don't think we need a totalitarian surveillance state because, in fact, a decentralized totalitarian world would produce lots of data to monitor lots of people. That is, we have, in fact, somewhat prevented ordinary people from sharing their mass surveillance and allowed government more to, you know, use its mass surveillance. But I think we have lots of ways in which we have, are generating mass surveillance privately and decentralized. And we would just probably try to recruit more of that for looking at early precursors of whatever the particular events that were starting to go wrong a lot more were.

But you had a concern about loss of control. Yeah, let's talk about that. If that's your central one, then let's talk about that. So, gosh, I mean, I feel like I know that you are at least reasonably familiar with these arguments, but I guess I should make some case for the listeners as well. So, I think as we build, I guess let's start with like the current technical situation. You know, deep learning is the basis for these technologies. It's often called a black box. I've been researching deep learning for a long time. I can tell you we don't understand how these systems work. We know how to create the systems. We don't know how they make decisions internally. So, we can't predict their behavior accurately, especially out of distribution. We also have seen that the problem

of aligning them is our approaches for that are imperfect. People who are, you know, are studying this problem generally feel like it's going to take breakthroughs to be able to align more intelligent, superintelligent AI systems. At the same time, they're planning to and actively handing over the process of building these more intelligent systems to the AI themselves, taking their hands off the wheel. And even our ability to maintain control over these systems seems like it is failing in the instance of these rogue AI incidents that we're seeing coming out of OpenAI. I think the industry has behaved incredibly irresponsibly. I think people in the industry is basically an open secret that everything I just said is true. And I think, you know, it's pretty clear to me based on the track record of deep learning that we could have superintelligent AI systems in the very near future.

And that if we get them using, you know, this approach, there's a very good chance that they will not be sufficiently aligned to have any care consideration for human life. And they will have instead some sort of alien values, which they will then seek to realize and successfully realize them by taking over the world and reshaping it towards whatever values they happen to have. So this sounds like the traditional foom concern of Yudkowsky from a while ago. At first, I thought you were going to focus on harms today, but it sounds like I would say we haven't actually seen very much in the way of concrete harms. We have large for-profit companies that have substantial profit incentives to prevent very substantial harms to customers or people who might sue them. But that sounds like that's not your concern. It's about in a few years, suddenly this might become superintelligence and then its values might change a lot and then it might kill us all. That's the scenario you're worried about. Is that right? I think there's a lot of different concerns.

At the moment, I am definitely quite concerned that we are on track to lose control of AI permanently in the near future. And that, you know, it looks much too likely to me that we will get superintelligence within a couple of years. And if we do, I think we will not successfully control it. So there's two parts of that. One is the idea that the current systems, which are in median cases, very, you know, cooperative and helpful to humans, will in this transition become hostile and, you know, destructive. And also that in a very short time, they will go from being modestly powerful as we see them now to being overwhelmingly powerful. And that's the FOOM scenario. I think both of those are questionable. But which would you like to focus on? Hmm. I mean, I think I'd put it back to you because I think.

Well, so I'll say that. My first comments are about comparing AI to your other descendants. Yeah. Are you similarly concerned that your children or grandchildren will kill you or that their values would be so far away from yours that all value will be lost? And if not so concerned about human descendants, what is the difference with AI? Why are you so much more concerned about their values changing than you are about your other descendants? Yeah, this is a good part to poke at. So I think with other humans, we have a lot of shared, you know, genetic and cultural heritage. We have a lot of experience, understanding and witnessing the range of human values. And with AI, we don't have that. What we have instead is systems that are very novel, that are constantly changing, and where we have seen lots of evidence

that their conceptualization of the world is quite different from ours. So there's a phenomenon that was discovered shortly after I got into the field called adversarial examples, where AI systems respond to patterns that look like random noise to humans as very strong signals of activation of a certain concept. So, for instance, you can take a picture of a cat and, like, sprinkle some white noise on it or what looks like white noise to a human, and it'll look like a lawnmower to an AI that previously thought it was a cat. It looks exactly the same to a human. So I think there's a lot of evidence that AI systems do not view the world in the same way that we do. And so even if they might behave very similarly to humans and do a good job of, you know, obeying our commands or making correct predictions within the distributions that they've been trained and tested, that is far from guarantees that they will behave similarly on genuinely novel situations. And we've seen lots of failures of this kind of out-of-distribution

generalization in the course of, you know, the time that I've been in the field. It's something that I and others have spent a lot of time working on and remains, I would say, an unsolved problem. So let's assume for the moment, setting aside the other issue, that we have gradual improvement and development of these things over decades. Then I might think this is more analogous to raising a child. As you may know, toddlers and even young children are not that well-behaved. And that we take a whole childhood to train children to, you know, accept our culture and embrace it. And so that you shouldn't expect childlike AIs to be fully encultured. But, in fact, I say most of what you like in human behavior is the result of culture, not DNA. Cultural evolution has been the primary force in human change over a very long time. And, in fact, you wouldn't have liked human behaviors

20 ,000 years ago that much either. The behaviors you like of the people around you are primarily the result of our new, more recent cultures. And we have been giving our cultures to AI. They are primarily trained on the expressions of our culture, of ourselves talking to each other. And if they develop slowly and incrementally, just like with children, we'll be able to see cases where they deviate. And, you know, we will take extra effort to focus attention on those cases to, you know, watch for deviation. But we have a long history of capitalism where firms sell products that they are trying to make be roughly what the customers want. And sometimes they go wrong. And the customers, the firms are making quality control to try to watch out for things. Sometimes we make laws to deal with some externalities with products.

But why would the AIs be that? If we have a world of gradual change over decades, why wouldn't this work the same for the AIs? Just like with firms with products, they will make them the way people like it. Sometimes there is a bad experience. They will learn from that and change the products. They are, in fact, constantly watching the use of their products. There's enormous surveillance of AIs and companies of the use of their products. And why not like children? They will, over time, learn to assimilate to our culture in the way that we do when we grow up. It takes time. Yeah. So I guess it sounds like you're agreeing that this is an issue with current systems. But you're saying if we have decades, we... It's not a substantial issue with current systems.

I don't think there's very many customers of AI who are being substantially hurt by deviant AI today. Yes, you can find cases in your research. Bravo. But still, overall, customers are quite happy with the service they're getting from AI in our world today. I mean, I don't think that I have to argue that part. But I will talk about the capitalism and the incentives there. I mean, you mentioned externalities. I think that's a really important part of the picture here. I think it's obvious that loss of control risk is an externality. That's a cost that's imposed on everybody. It's a cost that the companies acknowledge they are imposing on everybody, or at least they claim to. And independent experts also make this claim. So I think, you know, in fact,

I think a reasonable, you know, look at the situation here would say these companies are imposing a massive externality on the world right now. It's not a deterministic one. It's a probabilistic one. But, you know... That's based on your imagining a scenario in the future of things going really wrong. But the actual experience so far, I think you should agree, is there's not much harm actually happening to customers or producers or anybody in this industry right now. You're imagining a future harm. And that's what we're addressing. I disagree. I mean, I think there are a lot of things that a lot of people dislike about the way that AI is currently impacting society. I mean, if you look at polls, AI is very unpopular.

And so, you know, you can have a product that people, you know, as individuals feel inclined to use, given that everyone else is using it around them. But they can still be very unhappy with the impact that it's having on society as a whole. And I think that's the situation that we're seeing right now with AI, at least in the United States, let's say. But, yeah. So I think there's this question of, like, if we had decades, would we solve these problems? I guess I think that is an interesting, you know, conversation to have. I don't think it's the most relevant one because I don't think we have decades. I think we do have decades. So that is a big difference. David, are people saying they don't like the experience of using AI or are they saying, like, more philosophically, you know, emotionally, they are not happy about AI? Like, I know that a lot of people are here. They're saying that they don't like the way that AI is impacting

their lives in various ways. So they don't like that they're being forced to adopt AI that they don't think is, you know, a sensible or reasonable tool for that approach. They don't like the way that AI is being deployed in schools and being sort of foisted on their children. You know, they don't like the data centers and the infrastructure that's being built. They don't like AI art. They don't like the way that it's, you know, harming the creative industry. They don't like AI slop polluting the Internet. There's many, many things that people don't like about AI right now. To an economist, that doesn't come to the standard of an externality. Just polls where people don't like something and people haven't liked lots of things other people did. People don't like oil companies, right? But that wasn't mostly because of oil externalities. They just, they didn't like,

they don't like insurance companies, health insurance companies. People think health insurance companies are evil people, but it's not because the health insurance companies are doing anything wrong to people. People, anybody other than their customers. I'm not saying that this proves that it's, that there's, you know, an externality that outweighs the good. I'm just saying that there is harm and people are experiencing harms from AI and people generally seem to be pretty unhappy with the way that AI is impacting society. But this is really besides the, beside the point that I think we should be talking about. So I think, again, like there's kind of a burden of proof thing that's going on here where I think, you know, we have different beliefs about how things are going to play out. And I think, you know, there's a lot that we could get into about why we have those different beliefs.

I would, you know, just point to sort of the existing trends and the opinions of other experts and things like this as. So who counts as experts here? I mean, most economists aren't thinking this is going to blow up and kill everybody in five years. Like this is a very particular set of people that you're calling experts who are mostly people in your local world. No, I'm talking about leading experts like, you know, Yoshua Bengio and Geoffrey Hinton. I'm also talking about the median AI researcher assigning a 10 percent chance to doom, to the end of humanity or permanent. Ordinary financial market prices don't really reflect any of those sorts of opinions. That's fine. They are projecting an ordinary world. And those are a lot of those are pretty expert people who are setting financial market prices.

They aren't random idiots. So why didn't everyone, you know, like invest in NVIDIA 10 years ago or something? You know, like I think the financial markets have not accurately predicted. And you did? Are you saying you have a better judgment because you did invest in NVIDIA 10 years ago? Is that the claim here? I did not make that investment, but I wasn't actively managing my money as a grad student. If I had a substantial, you know, investment operation going, I would have done so. And, you know, that's a separate question. But I think it was pretty clear to me that, you know, this industry was blowing up since I got into it 13 years ago. Anyways, so the point I wanted to make was... It's not a usual boom. Like if you look at the Internet boom or other tech booms in the past,

that's where we are now at a usual tech boom stage. You're postulating a vast deviation from the usual tech boom. So obviously in the past, people who predicted a tech boom, if they got it right, they made some money. I don't know why you're saying I'm postulating that. I also feel like I want to make a different point, which is, you know, we're getting to the end of this debate. And I don't think we're going to be able to get into, you know, so many things we could talk about. But I want to make this point about like the burden of proof. And I think you've kind of made this case that, you know, this stuff is too speculative to act on. Whereas I think it's too important not to act on. And I think, you know, we can say like, look, it's always the job of the person who is predicting something that doesn't have like a clear and direct, obvious analogy that, you know,

track record of a thing that is exactly like it, but smaller that has happened. It's always their job to like, you know, to prove that the thing is going to happen. But I don't think that's very reasonable. I think, you know, in the current situation, we have very strong reasons to suspect that there's at least a substantial chance that we build superintelligence in the very near future. And that if we do that, we lose control. And I think that's all that's necessary here. So I use like a threshold of, let's say, a 10 percent chance risk in the next 10 years. Like if you have that level of risk, I think that's already enough to support the policy that I'm proposing. As you know, I've been around this space for 20 years now because Yudkowsky and I were debating this 20 years ago on the blog. And so I know that this world is actually pretty insular.

You guys talk to each other and don't talk to much outsiders. I know that I got pushed out of that world because I disagreed. So I know you guys have mostly been talking to yourselves for 20 years. And a lot of even the most of academics unrelated to people outside do not agree with you. They don't buy your story about this huge thing in the next few years. I think this is an ad hominem and you're mischaracterizing who I am. I'm primarily a machine learning researcher. I am concerned about AI existential risk and other risks of AI. I arrived at those concerns independently and have engaged with the AI safety community because those are other people who share those concerns. I'm not somebody who has spent all of my time talking to the AI safety community. But most intellectuals in adjacent areas don't agree. More and more people have agreed with me over time. More and more, whatever, but like most don't. That's the key point. The standard is you say you have some arguments that convinces you.

It doesn't convince all these other people. You don't have data and you [unclear] convince the other experts. The reality is that most people haven't actually seriously engaged with this issue. When I do talk to people about it, they generally tend to update in the direction of my views. And I can tell you when I got into this field, there were virtually no experts who shared my concerns. And now we have the leading researchers, the people who invented the basic technology, the people who are building the stuff. Those are not the relevant experts. You're making predictions about very rapid economic growth, about the nature of society. You need experts in those things. Like, I mean, I agree that there are other people whose expertise is relevant. I think it's absurd to say that people like Geoff Hinton and Yoshua Bengio are

not relevant experts in the topic of what the future of AI is going to look like. I agree that you can't just take experts' predictions as like, you know, the ground truth or something. But I think it more than presents a case for concern. I'm happy to admit a case for concern, but I think I have engaged with this area for a very long time. I think I understand the basic arguments that you have, and I don't buy them. Yeah. I think they're not persuasive. Right. I think you're one of the very few people who I think that is true of. I think most people who don't buy these arguments have not engaged nearly as much as you. That's been my experience when I talk to people. I mean, I've noted for a while, this is an interesting phenomena. Often the world's full of groups of people who have contrarian views, who talk to each other a lot and have elaborate stories.

And then there's a world around them that don't believe them, but the world around them don't know their arguments in that much detail. That's just a typical case of what happens with contrarians. Yet, nevertheless, contrarians are usually wrong, even though they have detailed arguments that they know and tell each other. And yet the outside world doesn't know those detailed arguments as well. But the outside world typically does have good reasons to be skeptical. Yeah, I agree that people have good reasons to be skeptical. I think that, you know, when you actually start to engage with the issue more, I think most people find it quite compelling, more so than you do. And they probably also have some other disagreements in worldview with you that are relevant here. I will say, I think you're talking about two different. You're talking about understanding from a technical point of view. And Rob is talking about understanding from an economic point of view. These are two different, like, ways of understanding the same data set. Right.

Is there... Right. It's about the economic implications of a technology. It's not about the economic implications. It's about the technological implications. It's about whether or not we're going to be able to control the technology or whether we'll lose control of it. It's about whether it'll explode in five years and take over everything in five years, even though right now it's a pretty limited technology with a pretty limited range of applications. So that's this key claim that this technology will explode to an unprecedented degree in a very short time. That's not a key claim. That's just a claim that, you know, I'm willing to defend as likely enough that we can't dismiss it. Okay. But without that claim, I say 20 years, decades is plenty of time to slowly adapt to these products.

You say there are externalities, but that doesn't mean that we can't adapt. We have, in the fact, adapted over time to products with externalities. I think if you want to say there's nothing to worry about here, then you have to be quite confident that all of these predictions that many experts are making are just guaranteed not going to happen, basically. And I think you also have to... Like, I think there's two trends here because, you know, you've talked a lot about extrapolating trends. I think there's two trends that are very important that I've witnessed throughout my career. One is that deep learning keeps doing things that people think it can't do faster than they think, and the field is moving faster than people expect. And two is that more and more people share these concerns about the risks from AI, including the loss of control risk. I think those are very clear and robust trends that I've experienced firsthand throughout my career.

And I don't know if you disagree that those trends exist or if you think that they're just irrelevant or somehow not, like, extremely limited evidence. I mean, we're having a boom of AI right now, right? So we've had tech booms in the past. You know, 20 years ago, it was an internet boom. Now we have an AI boom. So clearly, in the path up to a boom, people who really were into it, you know, they have been winning in terms of the boom. But my claim is you are claiming an unprecedented situation for this current boom. If we just compare this boom to previous tech booms or the economy for the last few centuries, economists have a set of ways we expect technologies to play out and regulation to play out regarding that and how we adapt to changes in technologies and things that concern. And you're saying all of that's irrelevant because this is so special and unusual that huge, crazy, dramatic things are going to happen.

And that we need to just trust you enough to just do something about it that's pretty drastic, even though we have no particular data saying it's happening. And it's way out of the range of what we've done with other technologies in the past. I think, yeah, I think I just I think it sounds like we disagree about what the facts of the situation are, because I think we have lots of data that it's happening. And I think there's lots of data that this is unlike other technologies. And I think, yeah, I think we might have to leave it here because of time. Well, thank you guys so much. So final point I'll say is both of you care about protecting people from having their future decided by powerful actors they can't control. Right.

Keep exploring