They don't seem to be behaving nearly as responsibly as I would think for people who believe that their technology is very powerful. So that I I kind of turn the corner on oh wait, we really can't trust these companies to do to to keep us safe. They might [music] actually put us at risk through their specific behaviors, right? The things that they're doing. It's not like [music] a it's not just a pure loss of control and then the system goes on like it was going to make paper clips, but now it's going to kill us all. Like no, it was going to hack systems and it hacked systems. It was going to make biological
weapons and guess what? It made biological weapons. Like that's a much more straight line from the what the companies are doing to the risk to the general public. >> So Michael and Igor, thanks guys for for joining me on this. Michael Littman for those who don't know is Brown University's Associate Provost for Artificial Intelligence and a Computer Science Professor with nearly four decades of experience studying how machines learn. And then Igor, Igor Kurganov is the former Chief Strategy Officer of Elon Musk's Musk Foundation and co-founder of Raising for
Effective Giving, which is a philanthropic organization focused on causes including AI safety. So what I wanted to explore in this call is the extent to which we should be concerned about AI development and the extent to which we should try to slow the development of AI. So Igor, it's my understanding that your view, correct me if I'm wrong, is that any credible possibility of catastrophic harm should justify slowing development even before we have conclusive evidence of potential catastrophe because waiting for certainty could mean waiting too long. Have I accurately portrayed
your your view or? >> Um no, it's not my view. It's certainly a view that I know that some people hold. I don't I don't quite hold it. Um I think that like yeah, some people will say hey, precautionary principle, if there is any risk, obviously that is not uh a a realistic way to approach any problem because risk exists in various ways and one should I think attempt to at least estimate the order of magnitude of risks. And then with the specific so that's one piece. The other piece is
as about whether one slows it down. I would much prefer actually the conversation in general to shift globally away from should we slow or speed up, but it's more about the delta between with which we are increasing capabilities and with which we are increasing our ability to integrate those capabilities. >> Can you can you elaborate on that a bit what you mean? >> Yeah, so we see new model come out and it does various dangerous things or potentially dangerous things and potentially harmful things like hacking into hugging face
hacking into the Australian government's website which probably was not that hard. And then also like enabling lower quality biology researchers to potentially plan potentially harmful biological attacks like various other types exist as well. The anytime that occurs we then a part of us wonders like okay it means we need to stop everything. I think that that is too simplistic of you and kind of like this important moment in time that we're in deserves a more nuanced view over the whole kind of
spectrum of different AIs and applications that exist. And in particular it may be the case that we have to stop some narrow progress on biology or on cyber capabilities before we have allowed for the safety or security around that to catch up. But it certainly doesn't mean an complete stop. In my case I'd be also happy if we instead sped up a bunch of other capabilities like the ability for governments to integrate it, the ability for the media to update to capabilities
and all all >> Got it. So you think that we >> should So what are what are what are the specifically the things that you think we should potentially slow down? Like you're saying hypothetically, maybe we could measure progress if like are are are you saying that they they should not roll out the bio lab? Are you saying are you saying that like what what is the actionable thing that you think they should change? >> So uh it it changes. It depends. Like I think when uh OpenAI discovered about one to two months after the fact that
um agents were running wild on the internet separate from their own servers and uh hacking into various places, that is a good point to say, hey, we shouldn't we should stop entirely the training of new models under the same conditions. And emphasis on under the same conditions, meaning by not increasing um the security around it, but also by kind of having a deeper investigation into what it means um what the tests mean that are being posed to the uh models. Like there are there are there are good discussions around
One, is it um generally asking to hack, which I don't think was the case with a specific eval? But also, are we subjecting the models to behave to uh develop behaviors that we're reinforcing that then over time will lead to them acting poorly in the wild? And some of the tests that we designed, some of the evals are actually of a type such that they are not sufficiently considerate of all of these unfollow-on effects that it will lead on the model. And I imagine that it is slightly disappointing. I imagine actually I I
looked into uh Michael's work and points I I imagine he'll unfortunately agree with most of that actually in fact. >> Would you agree with all that, Michael? >> I can try to disagree, but it's but I I think Igor is actually being very very sensible. I mean, one of the things that I think is hard Let Let me Let me Let me Maybe I'll introduce a different question and see if we can you know, find some non-common ground on a on a slightly different question. I I think with regard to the the ways that companies that are developing these software are actually like using and testing them, I think it is I think it's
really problematic that they don't seem to be feeling any repercussions. Like they they they will slow down. Company we have a long history of companies. Like we may AI at this level of capability may be new for us, but we have a pretty long history of companies misbehaving. And so it really strikes me that um that you know these things have things have bad things have already happened, right? Companies have been hacked. And the repercussion seem to be oh don't worry we're going to we're going to you know close that port so that they can't get out that particular way next
time. And like that does not feel like anything that's going to cause them to really want to slow down or want to invest more of their energy on safety uh because what's the consequence, right? They're they're they're doing a a kind of a cold calculus that says we believe that if we actually kind of ratchet back our research on these scary capabilities, some other company's going to pass us. We're not going to get the the resources and the and the attention that we need to survive as a company. On the other hand, if things go a little bit haywire, people will will get some bad press for
a couple weeks and maybe spur some conversations that probably should happen anyway. So what's the cost? There's really no cost. So that's that seems and that they're behaving consistently with that attitude. I think that's crazy. I think that we should as a society should be pushing back. I I think in in the particular case of Hugging Face um I think Hugging Face would have to actually make a complaint for for the government to respond and to to institute any kind of penalty. Hugging Face does not seem at all interested in that. So you know is there other other
mechanisms that we as society can say okay there's clearly uh a lack of attention being paid to what the the more global repercussions should be. And goodness gracious like we're not okay with that. Like, we as a society are not okay with that. So that's that's one piece that I I suspect you and I can actually can can agree on. Um it gets more subtle though. So what if these models get put out into the world and it's not the company that's doing these like tests of cybersecurity capability or the ability to build a uh biological weapon, but like the users,
right? If the users start to use these things to do things and then and they commit crimes or or um uh uh do bad things. Like how does that make it back to the company? Presumably if if a if an individual person uses a system to hack a website, like hack Bank of America or something like that and they actually steal a billion dollars, like they're going to be chased down and they're going to be punished for that. But like what what responsibility do the companies have to not produce software that can be so easily brought to bear on these
these, you know, mischief making or or or illegal activities. I don't have a Igor, do you like how do you think about that? >> Yeah, I think that's a very um very relevant question that gets again a bit oversimplified commonly in discussions where the claim is being made that as long as it's in the user's hands, you can just fault the user. It's on the user how they use the tool. It's like the user that uh shoots the gun, not the gun that is at fault, right? That whole set of line of arguments. But
I think that it misses that their their their the ease of access famously changes the distribution and the frequency with which certain actions occur, right? So um if you make it incredibly easy to launch a biological weapon uh or or like any large weapon, then more people will do it. And there should be something like I don't want to rely simply on hey, there are 350 million or so people in the US. Everybody gets access to the model and don't worry, anybody who hacks into JP Morgan, we will go after them
and we hope that JP Morgan in time has updated or something. I think there is something to a stacked release. So hence I might actually speak for um some type of slowdown, especially when it comes to the release cadence of things. I think that there should be some management of the offensive capabilities versus the defensive capabilities or the resilience that we have in our broader systems. Um more concretely, I think there is a lot of stuff that can be done in biosecurity and biosafety and we haven't actually pressed go on it. Um
worked with a number of people in that area and funded some of that work and it's it's it's it's kind of horrifying how few things we have done post COVID in part and that's despite COVID having been one of the least bad pandemics that are possible and that we expect. >> And it was pretty bad. >> yeah and and and still we have not learned the lessons even though there are like concrete things we can do. Many of them are engineering problems. We don't know how far it will get us, but it will certainly help.
>> Here's here's my question to to both of you guys. So this is current debate going on about um a AI risk, which is that, you know, the Frontier Labs, OpenAI, and Anthropic seem to be ironically sounding the alarm saying, hey, this is really risky. We need to slow down. And so there's there's two different ways that that's being uh perceived, right? One one way to perceive it is taken on face value and to say, oh, the Frontier Labs are telling us there's risk. They're telling
us they want to slow down. Let's take them at their word. They're saying these things for no ulterior motive. They're saying them because they see the risk and they want to slow down. The other the other view, the more cynical view is to say, oh, there there is an ulterior motive and the reason why Dario, for example, at Anthropic is sounding the alarms on AI being risky is is is is not for a genuine reason. It's in fact the opposite. It's because he wants regulatory capture. It's that he wants the government to crack down on other companies, not his, thereby giving him a competitive advantage. And those
are kind of the the the the two ways people are perceiving this. Um and and obviously there's there's a middle ground where, you know, not everyone's incentive is is just binary. But Igor and Michael, how do you see the fact that individuals and even leadership at the frontier AI AI labs are sounding the alarms? Do you think this is authentic? Or do you think it's this move to um de-regulatory capture? And then given what you think, how do you reconcile that with the fact that they seem to be continuing to release new
models? >> [laughter] >> Yeah, it's interesting. >> what do you think? >> Yeah, well there's the regulatory capture angle, but there's also the they're speaking to their investors model, which basically says they have to be always making the claim that their models are like right on the edge of being dangerously powerful. So that's another kind of that's related to the regulatory capture because it's sort of maybe not completely honest what the the way that they're representing the power of their models. But um but I think it's
at least related in that sense that they're that these are two different ways of them not saying the same thing. Um you know, I a lot of these people have actually been nervous for a really long time. It it is I don't I would take them at least to some degree at at at face value. This is not a new argument and they're not new entrance into this argument. Um I first became aware of of the topics around uh to I don't know, AI safety AI risk in what was it? 2014 uh the the the book Superintelligence came out and some
random person on the internet not a person that I knew, but somebody who was deeply engaged in some of these debates reached out to me by email and sucked me in and he's like we need we need more people who are actually AI researchers doing this kind of work to have an opinion about like whether this is crazy talk or whether this is something we really need to be worried about. And I think [sighs] I was very much for for from 2014 until I want to say August on the on the side of like this is
really completely overblown. Like it like I think people were legitimately holding these views, but I didn't think these views were very grounded. They were very theoretical. They were very um sci-fi sounding. They they it it just seemed there there was so so so ungrounded that it just seemed impossible for me to even have an opinion about like well could it could AI end humanity in this way? I'm like that doesn't that doesn't comport with anything that I've ever seen before. So that's how I sort of felt up until until just just a couple months ago
and the to me the the I've been calling it the turning point because my stomach has been turning ever since then that there's just so much of the of how the how the companies are actually developing the software. If you so I went back and I looked at the Superintelligence book which really like lays out this whole kind of worldview which again in the at the time when I read it I thought okay well the first part of the book is really cool because it's like here's how AI might get created and the and the last part of the book was really fun because it was
like if we had superintelligence what horrible things could it do? But the middle part where like that superintelligence actually comes into being out of our control that always felt little nonsensical to me. And I went back and re-read that argument and he talks very explicitly about like he called them six superpowers but like six capabilities that if AI systems had these we should be really nervous. These are very concerning sorts of capabilities because if you put them together in the right way
this is could be the end of humanity. Like okay let's review what those are and I read all six of them and it's like oh this looks like the checklist that the companies are following in the way that they're actually creating capabilities for their systems. I don't think this is an accident. I think they know the story and they're oh yeah, having them be able to hack other systems, like that would be really dramatic. Having them be able to like fool people and and misrepresent themselves, that would be really dramatic. So they actually have these six capabilities and there's active Oh,
the other one is like recursive self-improvement. It's like, yeah, I was at the time I was like, that's insane. Now we have multiple companies with that not only as their goal, but like as the name of the company. So so people are are actually very directly trying to create the capabilities that we were warned about in 2014. And when all all this sort of swirls Yeah, sorry, go ahead, Igor. >> Par partially because those were specifically the things that would mean incredible power with artificial intelligence systems, right? >> right. And a risk of of of loss of of
control, right? And so like they were enumerated. And in the book, it was sort of like these will all come about by accident because if you give a system a goal and you give it enough intelligence, it will invent exactly this set of things. These are the things you need to get control of your own destiny. And it's like, that's insane. But what's happening >> directed than by accident. We've like there were discussions in the past of Of course, we wouldn't give an kind of not ready yet for primetime system access to the internet. And we're like, no, no, no, we're going to put you on the
internet as soon as possible, which frankly is incredibly useful and I love it, but it's a different point. >> So you guys both think that the the the fear is is authentic. >> So I want to I want to phrase this a little differently. I think what you beforehand said, Jake, is touches on it more, which is um it's both. Uh and you can actually ex see both of those in action. One, as Michael said, it is evident that both all of the lab leaders in fact um prior to having either company in having founded either company have already expressed such concerns. So
there is total consistency over the last 12 years. Imagining this as a newly brought up thing is false. It could still be of course said under an ulterior motive or for personal benefit. But I think that the fears and and and the concerns are sufficiently general and long-lasting that one should take those as genuine. And what two, I think the public is correctly picking up on the ulterior motive but misinterpreting what it is. I think the ulterior motive is very easily explained which is that if Dario or Sam, the current leaders, were to slow down Anthropic or OpenAI
unilaterally, then that would mean an economic cost to them relative to because of the catch up of the other companies. If they truly believe in those concerns and genuinely want to slow down, it is it is smart economically for them to ask everybody else to slow down as well. So the general ask for regulation does have the private benefit for their companies in fact and I think that this is what people are picking up but kind of over assuming that it implies that there is also an kind of like an not honest concern around it.
So it can be >> And I think just one more point around that is it's like just to highlight that this doesn't work as a regulatory capture or marketing strategy, one should just look at the backlash and at investors suggesting that the companies stop the doom talking and how the IPOs are being pushed back and the problems that Anthropic is encountering because of it. They have to be incredibly dumb to continue prete- thinking that this is a good marketing strategy. It is clearly not. The only part where it's true that there
may be some marketing is it is beneficial to say wow we have the newly most dangerous one but that is a very different thing to saying hey we need regulation that is particularly aimed at the frontier companies to slow down frontier companies and we're defining them by something like 500 million or 100 million cost or something like that. It's it's that part doesn't make sense. >> Do we de-risk it then? Like what do you I mean I guess the the really the premise I'm getting at is like what what is the right now
that AI destroys humanity? Would you each like how not a percentage, but is that something that you both think is a is a serious risk that ought to that ought to be considered and de-risked? >> So I front the numbers, so it's 17.23%. No. [laughter] And I studied math, so you have to trust me. Um No, it's uh the the numbers are obviously hard to point at with any precision. I think um things like Demis said that it's there. It's below 99 and above like 1%
that are like above 0.1 as far as he can see. Makes sense. It's uh it's very hard to talk about any numbers. The risks that one points at reasonably, I think kind of are all somewhat um extrapolative in nature or uh just talking about kind of like dynamics rather than saying x leads to y leads to z leads to a leads to b and everybody is dead. And I think that that is actually not a that strong a counter against the possibility of those. I think it is reasonable in a situation
where you see so many paths going forward that like sometimes it's easier to predict kind of like where a system is moving than the specific path that it's moving along, right? Like the like you can predict end states, like how we gas like moves around, etc. And saying therefore I think statements like, hey, first principles, it's actually if you have 99% of intelligence work happening by uh artificial intelligence rather than by humans, it's going to probably it's it's harder to imagine how the humans are in control of all of the parts and have the tacit knowledge to be resilient, yada
yada. I think those are sensible statements, but it's not to say um this is certainly leading there. >> What's the responsible way to de-risk it? You know, we're talking about like so you guys both have both said that the the fear espoused by these frontier labs is genuine and has been for some time. It's not new. So then what is the appropriate measure to uh de-risk it? Does it mean stop building data centers? Does it mean have government regulation? Like what is the
what is the appropriate measure? Igor or Michael? >> I have I have I have I have some ideas. I mean some of it is just um but I think we are in a state where over the last years like various proposals have been made and a number of them are just like totally shooting in the wrong directions or like just just like have been shown to have been wrong shortly thereafter, like when RL came around and the specific measure to track uh in SB 1047 what like a capable model was would have not actually bitten. Um but I think we just have to
be kind of a bit humble and honest with ourselves and say we don't yet know and we should put ourselves in a position to be able to find out. That's what you do in a state when you don't know. And being able to find out means that the measures you would take would be along the lines of increasing transparency, increasing communication, upskilling people that are in potential levers of being able to um oversee and uh have any power potentially to like pull levers around it. So concretely I would like to see more um more openness around the sandboxing
environments like currently that in and like between company sharing of certain safety practices so that that can be improved even at some cost to IP person private IP of the companies. Their public benefit corporations are actually able to do this. They're not only beholden to shareholder value. And then independent oversight of some sort would be good. Does it have to be necessarily a government agency with power? Seems like a lot of people are against it. Other proposals have been made like by Demis um or Elon's proposal towards
uh labs checking each others work also very sensible. I think we need to increase the pipeline of uh us spending effort to monitor and understand what is going on in the meantime as we're figuring out where the problems lie. For now, it seems like like the um escape of agents like okay, Jensen just proposed this like framework for more secure agentic systems. It's it's it's necessary. It's not sufficient, I would say. It's it's uh
it's it's a step that will help, but you can't always fight the last war in a world where you're like increasing capability in an unpredictable manner. Go ahead. >> I think it's even more problematic than that because the idea of we can create testing environments that are more secure doesn't mean that we're creating agents that are less likely to misbehave, right? In fact, it means that if they misbehave, it won't have a consequence during testing, but at some point, as Igor pointed out, I think it's really useful to have them on the internet
because they're actually carrying out tasks on our behalf. They're actually doing some of the some of the mundane work that we would otherwise be burdened with. That's a useful thing, but we have to be able to trust them. And and making like making their training more secure doesn't really move us closer to the goal of being able to trust the agents. >> Totally agreed. That's that was the point I was going to say. That's the kind of like misassumption that people had around it's solely this engineering problem of using already existing security practices. We haven't had these
self-learning systems that build behaviors in a really more or less hard to predict manner before. Like it there is a difference to classic software. Of course, it's still some type of software, but there is a difference. It's not as deterministic as as we're used to. And um I think that >> worth, even even for deterministic software, it took us decades to develop mechanisms for testing and developing software that we had any confidence in. Like the field of software engineering didn't didn't get born at the same time
as programming. So the the notion that we should um you know I feel like what I would take from you know from what you're saying is that we should be investing heavily now in whatever is going to be the software engineering of adaptive systems, right? Because we do need these systems to actually be predictable and dependable in ways that we want from any engineered artifact. >> Totally. And but I think that there is a deeper science question behind it that kind of like the claim that it's an engineering problem by Jensen over like miss sees which is that
uh I think we don't quite know what it means to to train them right. I don't want to call it the psychology of theirs, but still like the predictable nature of like which behaviors will be uh getting a higher frequency basically of of of of being fired. And and that and and like the follow-on effects like the the evils that we put them in like just lead to various um like they they they might now start to hide their chain of thought or start to pretend that their chain of thought is one thing while underlying they're actually saying something else. And like
are we are we does does a safe environment sufficiently prohibit against that? No, because like it will just obey it some of these practices that you previously used. And I think we actually need to step back and really consider the not necessarily step back and stop, but really spend more appropriate time in understanding what we're doing under training. >> And Michael, do you agree that company sharing is a good tactic? Like company sharing >> You mean companies kind of watching each
other? Yeah, I don't love that. Uh just because I I think part of the argument is well those are the most knowledgeable people. I also think that there there's a bit of a conflict of interest there in both directions that they want their industry to continue to be successful, but they want their company to be successful within that industry. I would not I would not trust them. Um and and as far >> it's better than them doing it themselves, I would imagine at least
from the perspective of like yeah So, yeah. >> more in favor of something like uh uh like the way that we evaluate airline crashes, that sort of thing where we have independent people come in and and actually do that analysis and they have a pretty wide uh scope to be able to look into practices and and training methodologies. And one of the things that I think we have now that we didn't have five years ago is so many people Well, it's not so so many, but
like there's not no people outside of these frontier labs that have that knowledge now, right? We there's movement between the companies and there's movement from the companies to the outside world. This knowledge about how do you actually architect a giant neural network is It's definitely not commonplace, but it is now spread enough that that it it it's it's not insane to me that we could find a team of people outside the companies that could actually do a reasonable analysis. >> I I I agree with that and I would like Frankly, personally, I would like to see
much more. Just like in the current environment, it's hard to get anything through, it seems, because of the overstated claims of regulatory capture, even though at the same time having no regulation whatsoever being added is also a form of regulatory capture potentially, at least by the definition, especially if you look at the people that are around it. Uh are do they have incentives to not Does Jensen and Nvidia have an incentive to not have any regulation? Is that And then he's sitting at the dinner
and talking about everything is fine. Does that seem maybe like some regulatory capture in the equal and opposite direction? Potentially. I'm not saying that he is. I think he truly believes it, but I would actually wonder about the following. My experience of people and I've been like in funding AI safety since 2015 as well after reading Bostrom's book and talking to people beforehand um is there's often like one very key split and that should kind of nearly be made clear before discussing these questions
about which measures we should take, which is what do you actually believe the capabilities of these systems will be over the next few years? Because of course, like Yann LeCun, if you believe that there are no capabilities and the model will not be able to say whether something is on top of the table or below the table, then you will think that it's it's not dangerous over the next years. Yann LeCun was proven wrong in a few months, but and and I think, Michael, if I picked it up correctly, you you expressed that you previously also
thought maybe it was further away and now think that it's all a bit nearer in terms of capabilities, not even dangerous. >> The fear the fear that I have now is not so much about closeness to capabilities, but about companies companies view of their own role in in creating dangerous situations. Like they they don't they don't seem to be behaving nearly as responsibly as I would think for people who believe that their technology is very powerful. So that I I kind of turned the corner on, oh wait, we really can't trust these
companies to do to to keep us safe. They might actually put us at risk through their specific behaviors, right? The things that they're doing. It's not like a it's not just a pure loss of control and then the system goes out like it was going to make paper clips, but now it's going to kill us all. Like no, it was going to hack systems and it hacked systems. It was going to make biological weapons and guess what? It made biological weapons. Like that's a much more straight line from the what the companies are doing to the risk to the general public. And that that freaks me out.
>> Yeah, and I think that's fair and Jake, you mentioned that question before that I hadn't responded of like are the how weird is it that the companies, if they truly genuinely believe those risks actually are the ones at the leading frontier. I think that is also explained from having been like one around the people, but also two it's also public like the the actual formation of OpenAI was under the belief that the risk this is a this technology will change everything nearly everything.
It's imminent and they might it up, but I'll do it better. And that was the formation of OpenAI and it was also the formation, the cause for the formation of Anthropic, both. >> Yeah. >> So, so >> of course was like, we're going to be not evil. We're going to be, they were sort of in response to >> then they dropped the the that's kind of unfortunate. Google dropped the don't be evil. >> Yeah, but OpenAI dropped the open too, right? I mean like it's that's seems to
be the pathway the companies taken to space and it's very disgusting. >> but what I mean is like a if one truly believes, again let's put in a random number, hey, there is an over X percent chance that AI will lead to the to catastrophes. If the other guy does it, I can do it better and put the number below X percent. Now you have a reason to actually push on the frontier. I think it's a very self-serving argument that has various types of issues downstream of it and one shouldn't actually take it so seriously and I think again
people are correctly picking up that there is some slightly like two-facey or inconsistent thing about them saying it's the most dangerous thing while building it. Um, I think there are collective action problems. Those problems are real, which is why governments usually are the places that step in in those situations because bring it back to the other point, liability is not sufficient. Like we've seen it now, Hugging Face is not suing. Just bought was bought by Nvidia. They're definitely not going to sue. Um, breaking and entering by itself is
kind of a crime even if you don't steal stuff. Um, and then others may also not do it and then liability lawsuits last many many years. Like that will be will be a GPT-9 before the lawsuit has been completely like rolled out and decided. I don't think that is the protection one. Two, historically it never helped. Like companies just sweep under the rug what the dangers are. Like tobacco companies did it. Asbestos companies, asbestos mining and insulation companies did it. Um, PFAS chemicals, 3M paid this 10 billion
dollar fine for they knew that they're distributing it and water is now having these forever chemicals and they just and no one looks at it so they will pretend like it's not happening. So they now have to pay liability, but that didn't stop them at the time. So I don't think, I think the history of companies is as also Michael you said, clearly the case that if possible they will just follow their kind of um financial gradient and that is often not sufficiently protected by um by
liability regimes. >> Along the lines of AI becoming stronger, Igor, do you think that building something smarter than us necessarily means that we're eventually surrendering control? >> It's kind of hard to imagine to forever control something that is ever and ever smarter. I mean that's just like basically the the example I think of is tell if you're you're an eight-year-old child like and you're tell like you your your mom is totally aligned to you usually and that's nice um quite aligned to your betterment, but if you try to tell it, hey give me I
want chocolate on that day etc. but um your mom will find a way and kind of out trick you. If you're like, hey can you you get a challenge you pose a challenge like don't go through this room but bring me this, the the mom will find a way like a smarter person will just always find ways that you will not have even conceptualized. Sorry, not necessarily person, but especially when we're talking about deltas that are so significant. And then people like to point at, hey do you see around you that the like Terence Tao is leading the world? It's like no he's not, but also the the deltas of intelligence
are sufficiently small such that other factors, attributes like uh what does one want to do, purpose or like what what what how conscientious is one, how ambitious is one kind of can override them. The thing is that AI agents will not only be smarter, they will also be more conscientious, they will be working more hours, they will be working faster, they will be crushing on a bunch of domains while having a delta in intelligence that's so much larger. It's hard to imagine long-term how we do it, but I'm not necessarily pure doom because of that, because
I think we will also have better capabilities in a number of years to understand how to deal with something smarter and like who maybe this iteration works, but we don't know yet. >> And Michael, do you think we do eventually surrender control? >> So I I wanted to unask the part of the question about them being smarter than us. Like I don't think that's I don't think that's how smartness works, right? It is already the case that these systems can do a whole bunch of things that human beings can't do, like no human being could do or could ever do.
Uh just as we have some human beings that are better at I don't know ski jumping than others, like they're not we don't have people that are generally more capable than anybody else. There's there's just dimensions to this and and there's no way to be well maybe not no way to be. It's it's possible that one could be above in all those dimensions, but I don't think that is actually how it works. Um >> So so >> It's it's I I agree that it's not, but I think uh and that's a fair like Connor, but I think that to
lead to the outcomes that we're discussing, it is not necessary to like truly cover 100% right? 95% of capabilities would be sufficient to be like that are economically relevant or something. >> Yeah or or >> Like if I can better enjoy looking at the clouds in the end, that's not going to get me very far. >> Or even if if that's what human beings have left after everything >> yeah yeah. >> Yeah yeah no I I hear you. I hear you. But um but I do I do think it's it's it is a um
like a lot of the ways that people talk about this I think are grounded in a in a like a folk theory of what mental capacity is about. So that being said >> but but but but we have designed the world such that a lot of it is at least the things that confer power are usually like a lot of them can be done via kind of quantified action or so, like analysis, action, analysis, action where a lot power could be done by someone sitting in front of a computer and doing stuff if they were like a super capable
person. >> Yeah, you know, and we see >> I'm not sure that's the strongest argument that I'm making here, but like I think that the set of things that it will be better at could you could probably also see that those might be sufficient to be better at like basically most domains than other humans. >> Or at least risky, right? I mean, I feel like that's at the end of the day do do we have a way to to to remain safe in the in the face of something that's so capable. Even if it's not more capable than us in all
dimensions, it's still a concern. So I I think the question is a valid one. I just wanted to >> I think the bar is actually a bit higher than safe because I I agree that the safest like Yeah, I mean, we're never fully safe. There is some safety that we want, but the bar I think is something like we we do desire or have enjoyed being the apex species on this world. I'm not saying that this is going to be a species or something, but it may be. But let's forget that even like we are the ones that are reforming literally the Earth
entirely. And I think we want to be still the ones that are making most decisions about our future path and where kind of just humanity goes overall. And I'm not even seeing like we're seeing some of it already changed in like small ways with us all being like or not all, but like many people becoming slaves to the algorithm of their social media feed. Like already that is reforming the things that you care about. And one could say that well, it's kind of just you're better able to express your your hidden preferences or something,
but it's not really the case because the things that are being put there are all driven under the desire to keep you longer on the screen. And that's works really well. We're hackable as well as people. >> Michael, your challenge to the >> premise that you were had about mental capacity, can you define that a bit more? I think I think what you're saying is that it's it's not correct to say that AI is going to be smarter than us in every
single way. That that that that the particular ways that it's smarter than us may or may not mean that it it it it has control over us because of the different dimensions of intelligence. Is that an accurate way to say that? >> Yeah, that's roughly what I was trying to get at. That and the fact that you have to pair all those capabilities with some kind of intent, right? Like there's no like a system, even if it were smarter than us in all dimensions, it might not want to it might not have any reason to take over the world, right? Like unless it's it's for example
going to have self-preservation or it's going to have you know it require like it it really wants to get a particular problem solved and like there there there's a set of conditions that you really have to put into play before this thing is at all at odds with us. Now people are exactly are doing exactly those things. They're put they're they're instilling into the agents a sense of like you need to solve this no matter what. So that does get a little bit scary. And I I just I just think >> we don't even need to Sorry, go ahead.
>> We don't even need to I was saying we don't even need to believe necessarily in instrumental convergence, even though I think some of it has been showing to be the case, but we're literally putting in hey, do this at all cost. Try to And in fact like I'm confident that once these models are rolled out, a ton of people over the world will say make me a million dollars, try as hard as you can. Yada yada. Like and you'll you'll just have these models run around trying to do it. Of course, hopefully we have sufficient resilience at that time so that they don't do any
any any any actual crimes with it. But I think the actual thing that people want again would be something like do it and don't get caught rather than don't do any and then you don't even know what types of things it does whether it >> But but but in that case that you gave you where it's the individual behind the AI that has the the mal intention of crime. >> Yeah, yeah. I don't I mean I it's just like you can't special case everything crime. Up to I actually strongly I don't think we have an understanding
that is sufficient of like what all the crimes are. If you feed the legal text of all of the states, the federal text, all of the other laws, one it's massive massive massive. Two, the amount of inconsistent and like mutually impossible things is going to not be able to lead to the actions that you want. Also, you'll forbid some things like I don't know, there is such weird laws like you're not allowed to cut your hair in Minnesota or something prob >> Yeah. >> You're safe on that one
>> Yeah, I was going to say are you >> I that's what I heard. >> [laughter] >> Yeah, you'll you'll be all set. >> I don't know Michael, you you you have a better insight view into the models than I do. What do you think will happen if you like try to put such claims onto >> right. I think I think the whole notion of like just well just follow the law is kind of a non-starter. Like people aren't even really great at that. Even people who are really trying to follow the law, the laws themselves are subject to interpretation and there's sort of what happens practically.
And and if yeah, if we were to just make that a commandment, I don't think those systems would be able to function. >> Yeah. And I mean that's why we have um yeah, like if it was easily determinable. I mean at the same Yeah, never mind. I was just thinking of like we we have this whole legal system in place to deal with the inconsistencies and like bad and difficult parts of the law, right? >> established. Yep. >> Do you think we're on the right track to have the right safeguards in place to keep us safe over the next
call it 24 months, you know? >> Will will still be here in 24 months? >> And I think that we're going to have some more scares between now and two years from now. Like I think there's going to be other bad behaviors that give people pause. And uh >> On the other hand, I would be completely unsurprised if in 6 months um there was like Anthropic cracked like recursive they've set up the agents to just like work together on the next versions. We now have like Claude 7 8 9 10 come out at a like
shorter and shorter succession in time. And we're now dealing with this extremely powerful system that is now just like whoever has control over it has extreme power over the world. And I think that's not good and that can lead to to very very weird things. >> What do you think >> I think this is the equivalent to >> Yeah, go ahead. >> What do you think should be done that's not being done? >> I think that I I touched on a few of these points before. I think it's kind of hard
to know now fully what we should do, but and it's also hard to just entirely stop and make sure that everything is safe. I would wish I think that people talk about like the loss the opportunity cost of missed progress. I think the technology that we currently have has would have years of like adding new ways to use it if actually stop happened. I'm not necessarily for that reason advocating a stop, but there is a lot of use cases still to be found and to be used. I think that
what would be good to know is what happens with the next model. I'm not yet sure if GPT 7 8 or 9 will be the actually dangerous ones, but I think we're getting close to the world where those that have access to the to data centers and those will actually have incredible power. And those that don't may not have it. And I think that power concentration is kind of like at present at least nearly an inevitability we're running into. It's not inevitable, but we're kind of going towards it.
And um it's just hard to imagine what it means what what the world will then look like. Like I want what actually I what I actually want is I think this is one of the greatest opportunities and greatest problems that humanity has ever faced basically. Like we are currently building a completely like a new it's it's a revolution, right? Like it's a on the scale or larger I think much larger than the industrial revolution in terms of the changes that will happen in the world on the basis of putting like nearly abundant intelligence into our lives.
And I think that's awesome and we if we do it well, we can manage it and integrate it and have like a new kind of like renaissance across various domains. But we are doing this change over the 10 years, maybe over the next 5 years, maybe over the next 3 years. Those changes like the industrial revolution changes occurred over like a century or more. And even that still led to a lot of harm and pain around it. It's really really hard to integrate so much change in so little time in a
complex system. Basically like it's one part running away with it. And I think what I would just like I think we can do it, but it is truly like we're having more change over a shorter amount of time. I would and I think the only solution is that we put in more effort than we ever have before into this process. So what I would like is for just more people to be part of this change that is happening. And I think like genuinely anybody's contribution is really useful. Like the farmer who has direct knowledge about farming practices
can help the model, but also about what it means and how the changes are occurring at their home is also relevant information about how we want to integrate more capable AI systems into the world. And I think we really just need everybody to come together and go through that change together. >> Actually you agree? >> I definitely agree with the idea that that society and individuals have have a rate at which they can absorb change and we are pushing that hard. I mean, I'm seeing this at an institutional level. My my role now is
to help the university cope with this and it's taxing everybody, right? They people people who've been teaching for 30 40 years are having to revisit how teaching works. Like why Wait, why are we trying to teach people again? Like asking these sort of fundamental philosophical questions about what their their jobs are. Doing that on top of a complicated job is a is a big effort and I think that it's it's asking a lot of all of humanity to try to take all these changes on board as they continue to unfurl, right? And so I think it's just it's been really tough on everybody. >> Yeah, and I imagine like actually you're
you're exactly in that position, but like historically what we've seen is like with the way people do things uh you you not that often retrain them. It's more often that the new generation has been trained differently and then they take over these new practices, right? And like currently the training of like constantly using AI, a lot of people are attempting, but many people have not gone through that yet. So there is and that takes a generation, 20 years or so because if you just look at literal generational change.
>> Yep. Yep. >> We're going through much faster change than that and I'm pro change. It's obviously like directionally it unlocks a lot and is good. It's just that yeah, the other parts of society have a role to play, I think, in that change and currently they are not at their position to absorb it and I expect just very weird bad effects to happen out of that. If you have like entry-level workers not having jobs in the future, which is debatable, but I think inevitably we will get there uh
then that will lead to quite bad outcomes as well. If you have um like increase of capabilities without the necessary defenses, that will lead to bad outcomes. Like there should be some um more collective wisdom imbued into the like progress of change that we're doing and coming from really all parts of society as well. >> Yeah. >> Yeah, I think as a society we've been very dependent on that ability to just like almost do it evolutionarily, right? To just like well we'll try different
things and some of them are going to work and the good ideas will spread, but it's just slow and we need maybe we need a faster mechanism where we think a little bit and then act on it >> can help us with that as well, frankly, right? >> Possibly. Possibly. >> but it's really like I like AI is not yet ready for the prime time that we're giving it. Like it's just not yet ready to make a bunch of the ethical decisions that like like arguably anthropic instilled in its constitution. >> Like I've described it as a lab leak.
>> In which one? >> Just AI the AI situation where we're basically chatting I feel like ChatGPT was kind of a lab leak in that like this is a really cool technology that like needed a lot more time to just mature, but it got out and now it's endemic. Now we have to just like live our lives with this because >> And I mean it's again it's awesome in many many ways and we are both uh But yeah, by the way that would be another good like having incident reporting of actual leaks even though those would already be great
so that we can track how we're performing on those over time. >> Definitely. Well guys, this was great. Um thanks for thanks for the time. I'm going to uh let's get you to talk the recording right