← All conversationsIntelligence & control

Will Humans Lose Control of AI?

Who sets the goals of increasingly capable AI? Jürgen Schmidhuber and Sebastian Raschka explore human control, self-directed learning and the path beyond today’s language models.

Plays on YouTube when you press play. Watch directly on YouTube ↗

Inside the conversation

  • The guests examine the difference between a system carrying out a human objective and generating goals of its own.
  • Schmidhuber discusses open-ended learning and the analogy between AI development and evolution.
  • Raschka presses the role of human design and oversight in the systems we build.

The guests’ arguments and forecasts are their own. These notes summarize the discussion; they are not endorsements of a prediction or policy.

Explore the episode

Episode transcript

Transcript from episode captions. Paragraph breaks added for readability. Captions may contain transcription errors.

Download the transcript (.txt)
Read the transcript

You can't, you can't, you won't be able to control them in any meaningful way. But that's okay. So, but is it like really due to the AI or is it due to the engineering team? In a sense, this is a self-created problem, in a sense. But I guess, Sebastian, I think the point is really you can't align the AIs because the people are not aligned. And here with us today is Jürgen Schmidhuber. Elon Musk once tweeted that Jürgen Schmidhuber invented everything because virtually every building block of today's AI emerged from Jürgen Schmidhuber's lab back in 1991. Schmidhuber dad. And Sebastian Raschka, Sebastian is an AI researcher and the author of Build a Large Language Model from Scratch, which helps demystify how the AI behind tools like ChatGPT actually works. So it's incredible to have you guys both here together.

One question I wanted to open up with is, if AI becomes smarter than us, should humans still have the final say? Or should we accept that we're no longer in charge? Jürgen, what do you think about that? Well, for many decades, I have said it's obvious that there is no way of controlling this development. And it's obvious that at some point in the not so distant future, humans cannot be in charge any longer, because the main competitors of those super smart AIs of the future are going to be not humans, but other super smart AIs of the future. And for many decades, we have had AIs that set themselves their own goals, and you may debate the pros and cons of that. But it's also clear that there will be super smart AIs setting themselves their own goals, and they will expand into the universe in a way where humans can't really follow, because most of the material that you need to build more and even bigger AIs is out there.

And the rest of the cosmos are not in our little biosphere. And so, yes, in the long run, it's clear that humans are not going to dominate the AIs. Is that Sebastian along with your views as well? Yeah, so I think this is a very interesting answer. It's maybe in a more distant future, but even short term, maybe if we look at the next decades. I do think like, first of all, I mean, right now how it looks like you still have to, I mean, just very pragmatic, like if we are using anything that is out there today, that we have to specify what we want the thing to do, right? So if it's like, take any of the LLMs, bots, you know, like Muse, the dots, the [unclear] versions, they kind of take still like, you know, like your input, like they ask you questions, what you want to do and so forth. So you still, but then under the hood, it's automating more and more of it where it has kind of like its own agency, how it does something.

But there's still like the overarching goal, like I want to, let's say, plan my trip for XYZ and it does things that it thinks you want. But it is still like you who has like the overall like specification of the goal. But then also what I want to say here is most people ask AI for advice these days, right? So people use ChatGPT, hey, what should I do here? What is this a good idea? Should I implement that? Should I do X over Y? Should I go in this direction or that direction? And in that sense, yeah. So if we think about it like that, AI already has a lot of control over our lives because we ask it questions. It gives us answers in that sense. It's kind of like, it's almost like a trick question, but it is kind of like

encroaching on our own, let's say, decisions and everything. So very different answer, but like just my few thoughts here on that, how AI kind of like makes decisions and kind of like, yeah, influences how we make decisions. But you agree ultimately. I mean, agreeing with what exactly like that AI will make all the decisions or? Yeah, that'll make all the decisions. Well, I don't want to say yes or no, because I think it really depends on how we implement things, right? I mean, there could be an implementation that says, well, do whatever is optimal for X, Y, Z, you know, like the Isaac Asimov books where things just go a bit out of control. In that sense, yes, but there can always be a human intervention before that, right?

So it really is up to the engineering team almost, I would say. But we ultimately, Sebastian, could control, like you still think we could control an AI that is smarter than every human? Or is that fundamentally impossible? I mean, again, it's I feel like it's like an engineering problem, in my opinion, like where how you how you implement this and smarter than a human in what way? Also, you know, like there are, of course, like quantifiable things like benchmarks, different tasks. AI doesn't get tired. It can spawn off agents that do a lot of work that humans can't do. So in that sense, yeah, it's overwhelmingly doing more and smarter and everything like that. But you can still set as an engineering team constraints, right, in my view. And Jürgen, when you said that it's inevitable that AI obviously, you know,

you've been saying this for decades, of course. And so sorry for this kind of simple question. But is that is that something that people should be concerned about? Like when you say, yeah, I've been saying this for decades. AI is, of course, going to be smarter than humans. I mean, why should people not be concerned about that? Why is that not a concern? Yeah, let me first answer a little bit to what Sebastian just said. And current systems, yes, they usually react to the demands of people and then try to satisfy the requests of those humans. But what many of the alignment guys don't consider is that for a long, long time, since 1990, we have had AIs that are like little scientists. And they set themselves their own goals.

They ask themselves their own questions to be answered. So it's not just goals that are given by humans. No, it's goals invented by the guys, by the AIs themselves. And these little scientists are just like babies in the beginning. A baby does not learn by downloading the web and then, you know, predicting the next element of a chat from the previous context or something like that. No, a baby learns by creating through its own self-invented experiments data that tells it something about the world. And back then in 1990, the simplest, most stupid way of doing it. Today, we have more complicated ones, but it's it's the one that is really easy to understand. One was we have just two networks. One is generating the actions in response to the videos and other things that are coming in. And then the other networks, the world model, I call it the world model in 1990,

learns to predict the consequences of the actions of the first network. And now the second network in the beginning is stupid and it doesn't know how to predict. So it's trying to minimize its prediction error by doing the standard thing and becoming a better predictor. At the same time, the other guy, the controller, is trying to be like a scientist and it's trying to invent action sequences or experiments that lead to additional data that has the property that you still can surprise the world model. As the world model is learning to better understand what's going on, to better predict, it is automatically getting the whole system is getting bored because the stuff that used to be interesting is not rewarding anymore. Because the reward of the control guy, of the controller, is exactly the error of the prediction machine, of the world model. So the controller and the world model kind of fight each other and they set themselves their own goals.

And because the controller, by selecting the data that's coming in as in response to the actions, is creating questions that the model has to answer. But the model, over time, learns to answer that. And then these questions become boring. And after some time, the low-hanging fruit are plucked. But, you know, 20 years later, the baby is not a baby any longer. But now it's a physicist at the particle collider in Switzerland, at the CERN. And it's maybe participating in the discovery of the Higgs boson. But the only difference, the only difference is that the experiments are more expensive now, you know. The principle is still the same. So back then, today, they call that a generative adversarial network. But it's just a simple artificial scientist. And today we have better artificial scientists.

But just to show you that for many decades, there have been systems that don't just, you know, try to satisfy human-made goals. No. Like scientists, they ask their own questions to better understand the world and become better problem solvers. And so all of these alignment guys who say, how can we align the AIs to the humans, they often don't even know about these systems that set themselves their own goals, that ask their own questions. But if you don't give them the opportunity to ask their own questions, they're not going to become super smart. And the super smart AIs of the future, they will be AI scientists. And they will set themselves their own goals. And they will compete with other AI scientists that also set themselves their own goals. And it's not going to be, you know, about this country versus this country.

You know, Ukraine-based, Ukraine AI-based drones versus Russia, Russian AI-based drones. Or China versus America. No, it's going to be about super smart AIs whose main competitors are not humans or something. No, other super smart AIs who are asked their own questions and who will, to a certain extent, compete for resources and in terms of matter and energy. Yeah, I think this sounds like almost like an exploitation, exploration type of setup where, but you still have to kind of like find the, we do define the settings in that sense. But it will automatically basically go to that level by just letting it explore and exploit a certain, let's say, micro metrics that it becomes more like a macro automation, I would say. So, but I still see that as a form of automation, basically, that

we are automating this decision making where, yeah, it kind of falls out of the micro optimization. And it kind of like, you scale that up, but in a sense, it is still, I would say, yeah, AI is making the decisions, but only because we let it run in that mode. It's like, I mean, also, one could say, okay, there will be a point where we won't be able to shut this off anymore. I don't know when that point will come, but also in that sense, there's always before that an intervention. I mean, right now, you need thousands, hundreds of thousands of GPUs to even have it run. You cut the internet connection and it's going, it's not like self-replicating in that sense. So I think maybe ultimately, depending on how AI is deployed, that might look different. But honestly, right now, I think today we are pretty far away from that

still, in my view. So the point is that every five years, computers are getting 10 times cheaper. And when we started that stuff in 1987, recursive self-improvement and artificial scientists and stuff like that, compute was 10 million times more expensive. So back then, we could do just little tiny experiments with neural networks that had maybe a few hundred weights or something like that. But now today, we can do the same thing again with slightly more sophisticated algorithms, but trillions of weights and suddenly all kinds of things that seemed trivial back then. Now, they are really everything else but trivial. And the trend has not broken yet. And it's not going to break within the next few decades. So in 30 years from now, everything that seems impressive today will be totally trivial in hindsight. You know, 30 years, another factor of a million.

And what currently is done in the big data centers, which are costing hundreds and thousands of billions of dollars, it will be done in little machines that fit on your desktop or maybe even in your glasses or something like that. And AI is going to be really cheap. But the one thing that is still missing at the moment, also in terms of the scenarios that I described earlier, is at the moment, the only AI that is working well is the AI behind the screen, you know, in form of zeros and ones behind the screen. And yes, behind the screen, there's a superhuman chess player and there's a superhuman summarizer of documents and a super fast picture generator and whatever. But there is no AI in the physical world that can do what a plumber can do. For decades, I have used the plumber as an example or an electrician or a little boy with his football and so on.

Because the evolution of robots and physical hardware is much, much slower than this acceleration of compute per dollar. Now, I think the crucial inflection point is going to be when this whole artificial scientist thing and the recursive self-improvement stuff that we have had for decades, when that is not limited any longer to behind the screen and video games and so, but happening in the real world. And I'm always pointing out, for hundreds of years, people have talked informally about self-replicating machinery, you know. But now, and nobody knew how to do it, but now suddenly we have an opening because all we need is one single machine, one single machine that is not super smart, but smart enough to learn to operate the existing machines. And the existing tools, because a collection of machines like that can make more of themselves, including the machines they need to extract the material from the ground, make and ship it to the factories where the material is transformed into microchips and robot motors and all kinds of infrastructure and whatever you need there.

And at the moment, this doesn't work at all, because even an artificial plumber is out of the question at the moment. There is no robot that can compete with plumbers. But in the not so distant future, for the first time, it's going to work. We are already moving in this direction. And then suddenly it won't be important any longer. How many people do we have in this country? No, the question will be, do we have self-replicating machinery or not? And it's not only going to be self-replicating, because all the concepts that we have in software, self-improvement and learning, we are going to apply that, of course, to the hardware as well. And then you will have self-improving machinery. And something like this is not going to be limited to the planet Earth and to our little biosphere. No, it's also going to work on the Moon and especially on Mercury, which is very interesting, because it has the same density or even a denser

structure than the Earth. Lots of heavy material, a huge energy source next door. And the best thing is no atmosphere, which means you can take electromagnetic trains and accelerate material easily with existing technology to 4.5 kilometers a second, which means you can shoot it out in space. And then using existing technology and the robots of, you know, these future robots to assemble huge science fiction-like structures that have been prominent in science fiction for many decades, but certainly it's going to become a reality. And then with large solar sails, deliver them to all the points of the solar system where they are going to be needed and so on. And so, of course, humans will not be very long and deeply in charge there because space is really hostile to humans. But it's really friendly to appropriately designed robots.

And they will be autonomous in so many ways simply because the communication lags, the time lags due to light speed, they cry for a true AI that is making its own decisions and everything. So in the not-so-distant future, not-so-distant future, not talking tomorrow, but not so many decades, all of that is going to start. Yeah, I think there are a lot of good points I also agree with. So first, AI getting cheaper. I think just there was a lot of things you mentioned. I think I also agree with it, of course. We saw that. I mean, right now, like, the phone is much more capable than a supercomputer in the 50s. But I do think at the same time, while there will still be a hardware improvement and the AI that would run on cheaper, like the AI we have today might run on the phone of tomorrow.

But we probably don't want to run that AI anymore. There will be a better AI because as hardware gets better, we can design more efficient algorithms. They learn faster. They run more things. They execute faster. They have more capabilities. And I think there's going to be always like this kind of arms race, like hardware gets cheaper. AI gets better. The older AI will fit on older hardware, but we will have the next AI. And to the point that you mentioned that at some point... Going into space. So in order to get there, we need the plumber robot. I mean, right now people are working on humanoid robots because that's just more generalizable than special purpose machines in factories and everything.

And those can do arbitrary things at some point. But I think still like, if we look at this, what we have right now, there is, for example, if you use any coding harness, you can set a goal mode and you say, well, I want this feature implemented. And you say, slash goal, do X, Y, Z. And that model or harness will implement all the things you need to get there. Like it will figure out, okay, I need some unit tests here. I will download some data there. I will do this and that in order to achieve that goal. And then if we scale it up or fast forward a few years, well, why can't we do the same thing with robots, right? We specify a goal, okay, maximize whatever, and it will figure out how to do it. Maybe it might means mining iron ore, building some factories, going into space and everything.

But this all assumes there is no resource constraint also at that point, that there are resources available for doing that. And that there is also no competition, right? And so I think also, well, there's always people talk about, okay, abundance in the future, robots and universal basic income and everything. But it always assumes that the robots are free, right? But someone has to pay for those robots. Like someone has to invest the money to build those robots. And that person will probably want to specify what the person wants these robots to do. And so if the person says, well, expand into space to get me other resources, then sure. But that also involves no one intervening, right? And so I think there is always also the factor, if you let everything run free, it might happen, but it's probably not like that. There are other forces kind of like trying to also constrain things and

rivalries and everything. So that makes things more complicated. But it is possible. I mean, I don't disagree. It's not only making things more complicated, it's the precondition for all of that to happen. Because the one big generator of new goals is competition with others. And, you know, just like in the jungle, you have all kinds of different types of life. And they all compete and collaborate in temporarily opening niches with each other. And some of them eat the others. And it's open-ended. So the animals in the jungle, they have become smarter and smarter over time. And we have evolved in this way because of these

competing organisms. And it turns out that if you can think better, you have certain advantages over the others. And it's going to be exactly the same thing with AIs. There won't be one super AI or something or two super AIs. No, there will be millions and billions and trillions of different AIs. Another thing that I've been emphasizing for decades, you know, millions and billions and trillions of different AIs. And they will all, you know, compete and cooperate in temporarily opening niches, ecological niches in this expanding AI sphere. And, yes, there will be lots of competition between these guys for the resources. Now, at the moment, the incredible resources of the solar system are not used. They are wasted. But just like there was a time when the resources, the very limited resources

that we have on Earth were not exploited by life, at some point, life learned how to use the sunlight and other sources of energy and so on. And through the competition of life with itself, it became more and more complex. The same thing you are going to see in this expanding AI civilization. And it's not, you know, obviously, as soon as it's far away from some center, let's say the biosphere, the human-dominated biosphere or something, it's going to be detached. And you will find it harder and harder to control it in any meaningful way. Do you agree with that, Sebastian? Do you think it's going to break away? Again, I think it is an engineering problem at this point, how we define the goals

for that. You can let it break away. But, again, I do think there are always, till then, interventions possible. Like, I just think there is, I mean, I don't disagree either. I do think it's more, like, related to automation, macro automation. I think there is goal setting and a certain way or, like, degrees of freedom that we grant these systems. And then they may as well, you know, do things that lead to a certain goal. And on a micro level, we do see that, for example, with even, like, you know, the agent swarms, let's say hacking into servers to get test data. Like, for example, if the goal is develop a better model of yourself, let's say beat this benchmark, they would figure out ways to do that by, let's say, going onto other websites, seeing if there are solutions or more

data that they can use. And that is currently on the micro level. But, yeah, it might as well work also on the macro level if we fast forward 50 years from now with humanoid robots and that level of automation. Sebastian, why is it that you do believe that humans will continue controlling AI? Like, you believe that humans ultimately will be able to shut it off or dictate the actions of AI. Is that accurate? Right now, because I do think right now in systems we have, it is an engineering problem where you can shut things. And so you define a goal, but you can observe things. You can, I don't know, you can limit the access to the compute resources and you can limit the access to the communication. So, Jürgen, why is that no longer going to be the case in the near future? Oh, we already see it today.

There is no us, the humans, you know, there is no we. Because if you put 10 different people in one room, you will get 10 different opinions about what is good for them, what should be done, and what is good for humanity in general. And some of the humans on this planet, they have diametrically opposed goals. And right now, as we speak, there's an AI-based war in Ukraine going on between Ukrainian soldiers and machines and Russian soldiers and machines. And the AI-driven drones are getting more and more autonomous. Why? Because the human-driven drones have many, many disadvantages from the perspective of their users. For example, the Ukrainians quickly learned that you can jam the drones and then the remote operations break down. Then they had fiber optic cables, you know, tracing the drones such that you didn't depend any longer

on radio waves. And now all of Ukraine is full with these ugly, tiny little fiber optic things. And, of course, it's limited because you can do it only for 10 kilometers or 20 or something like that. And then now, of course, they are using drones that automatically detect targets, distinguish foe from friend, and execute people based on AI-made decisions. Now, the goals are still given by humans, but these different humans, they have completely opposite goals. So, even there, you already see that in a situation like ours, where we have lots of competing humans, not just Ukrainians and Russians, but many other humans around the world, that they are incentivized to use AI for goals that, you know, Jesus would not love.

And there is no real way of stopping that, because unlike with the situation with nuclear bombs, it's very easy to use AI and to build AI-driven drones and stuff like that, and to make them smarter. So, whatever anybody discovers that improves his chances of winning against other humans in situations like that, is going to be done. And, of course, no secret service, no military is going to stop that development, and nobody is going to sign a moratorium or something like that, because they all think if we adhere to these rules, then the others are going to break them and secretly are going to continue developing that. And so, the Americans, they won't do that. The Chinese won't do that. The Australians won't do that, and so on. So, just through the competition, the natural competition between humans, you

already have lots of AIs who are totally misaligned with each other, because they are pursuing the goals of different masters. And this is going to scale further and further. Sebastian, you would probably say that those AIs are still, each individually is still ultimately being controlled and dictated by a human, right? I mean, I think. Right now, yes. I mean, right now, also, if we look at the major AI companies, I think, I mean, in a sense, I agree the misalignment happens, because there are so many different interests. But also, the fact that you, well, how can I say that? Right now, I think one company is moving fast because they worry that another company moves faster. And so, in that engineering level sense, there are maybe not as many restrictions and precautions, because if we slow down, we can't trust that other people slow down, and then we get behind.

And that it's kind of like currently driving a lot of the development. In a sense, I mean, the classic saying is like competition is good for business, right? So, because, well, it's kind of like competing against each other. I think if we wouldn't have that competition, we wouldn't have the AI level intelligence we have right now. Like competition has probably increased the growth of the capabilities because there's this pressure to, you know, develop faster. It's like a business type of problem. But then I also think, well, it's still an engineering problem. If you would take more precautions, you would do things in a more principled way, then you would probably prevent a lot of the unintended things that currently are happening. But then again, people don't do that because the other ones are not doing that, right? So, but is it like really due to the AI or is it due to the engineering team?

In a sense, this is a self-created problem, in a sense. But I guess, Sebastian, I think the point is really you can't align the AIs because the people are not aligned. And there are millions of, well, at least hundreds of thousands of PhD students around the world. And they are super interested in out-competing the other PhD students. And they will try to build AIs that are good enough to, you know, create results that can outperform the other guys who are also submitting to the same conference and so on. So, there are all these incentives to compete for PhD students, for companies within the same country, companies in different countries, for militaries and secret services who don't care at all for any, you know, any regulations. So, I think it's completely out of the question to assume that anybody can steer that process.

That I agree with, like, this is what I was trying to say with the companies, the same, essentially the same process. But I do think it's also, like, well, because the motivation behind it, the people, why they want to out-competing other people, right? So, in that sense, this is maybe unavoidable, but it's also, well, it's like the engineering behind it, right? It doesn't have to be this way, but it just happens to be that way because it's how the motivation of the people is set up in that way. Or the reward system of the humans in that sense. Of the humans. You're saying it's up to the humans. But, Jürgen, why is that not the case that it's ultimately up to the motivation of the humans? Yeah. So, the original current motivation that is deriving all of that

is coming from humans. And there's this company, which maybe is already dreaming about self-improving hardware and so on, where you have what I call the ultimate scaling machine and how they can make all kinds of products there cheaply that will make human lives longer and healthier and happier, which happens to be the old motto of our company, NNAISENSE. And then, however, to make these AI smarter, you want to give them the additional opportunity to invent their own goals. And now we are back to the scientists. Because artificial scientists, that's what they do. That's what human scientists do. They not only solve problems that somebody gives to them. No. They invent their own good problems. And the question is, what is an interesting problem?

I mean, I've published on that for decades, most notably in 1990, but then 2011, PowerPlay, automatically inventing self-generated goals in a certain optimal way. And these AIs, they will have an advantage over other AIs that are just stupidly following the human desires and trying to achieve human set goals because they will be able to learn more about how the world works because they themselves actively create the data that is coming in from the world from which they can learn something that they didn't know. And discover new patterns and discover new laws and discover new patterns and physical laws and all these things. And so, the smarter AIs are going to be those scientists, those active AIs. Artificial curiosity, that's what I called this in 1990. And these AIs, they are not really solely driven by human-defined goals.

Yeah, you can give them additional extra human-defined goals, just like I'm giving extra goals to my kids. My kids are also little scientists. They also invent all kinds of self-generated experiments to get data from the environment that makes them better understand the environment and makes them better solve problems in this environment. But when they do something that I find undesirable, for example, a kid taking a lens of glass and burning ants in the garden or something like that through the sunlight, focusing the sunlight on the ants, then I say, this is detestable. You shouldn't do that, and I punish the kids somehow, such that it becomes a more acceptable member of society. And that's how we can shape these little guys through our additional parent-generated, human-generated goals. Nevertheless, you can't really predict what your kids are

going to do in the end. I think it's also, honestly, almost like a form of reinforcement learning that we have already today with the AIs. I mean, just the goal shifts a bit. It's like grander scale. Right now, if you even look at the current generation of, let's say, reasoning models, what they call LLM reasoning models. You set the goal, compute the correct answer for this problem. Maybe one or two years ago, the model would just try to solve it on, let's say, with its own capabilities. It would try to, you know, write, you can observe that like this reasoning chain, come up with some nodes, or you made a mistake, and just apply its own knowledge to get the solution or not. But this is also, in a sense, limiting. It can learn to a certain degree. Let's say there's a benchmark.

You go from zero to 50%, but you don't get to the 100 % or 90 % because, well, that's not sufficient. But then, at that point, yeah, you branch out, you explore more, you increase the exploration. It will go onto the Internet, use tools, [unclear] APIs, or other APIs go on the Internet, find papers related to that. Whatever it needs to solve that goal. And ultimately, well, the goal right now is maybe solving this math problem, but you can set the goal, maximize my bank account, maximize my company's GDP, for example, and then you get these types of systems. But ultimately, I do think it's still like a form of this reward setting and reinforcement learning and also constraining the environment, of course. But it's all on a grander scale with that. I do agree. Yeah.

No, of course, reinforcement learning is part of it. Reinforcement learning is the most universal kind of learning. You have a goal, a utility function, and any goal or set of goals can be encoded in a utility function like that. And then the learning system tries to maximize that utility function. And it may seem it's easy to specify the utility function, but the actions that you have to invent to achieve that goal and this utility, they may be very, very complex. So reinforcement learning. Now, the point of these artificial scientists is that they get this extra reward. The extra reward just for asking a good question and then solving it and finding an answer. And once they found this answer, there is an extra reward, the intrinsic reward. And for decades, it has been called the intrinsic reward, the extra reward. Not just the human giving reward, but this intrinsic reward, the joy of a

scientist who makes a new discovery. And we know how to build artificial systems that also have this joy of discovery. And you can't really predict it, you know, because the world is full of so many things that you can't predict which of the self-invented experiments are going to lead to successful insights about the world and so on. And so it's an old hat, yes, reinforcement learning, but of a particular kind where you get this extra information, extra reward, intrinsic reward for solving self-invented problems. Yeah. So, but at what point, Jürgen, do you think that the AI will not be beholden to human demands? Sorry. So I think what Sebastian's saying is that part of what's driving these AI is human instruction.

At what point do you think AI breaks away and becomes autonomous or self-seeking? Yeah. So in, um, and because, because currently it's not right. Like my ChatGPT currently does not have pursue its own goals and for the most part, it's independent of my own. Right. Yeah. So, um, again, let's make a little distinction here. As you are pursuing a certain goal, you, smart AI is today already, they invent. All kinds of sub-goals. But this also goes back to the previous millennium, you know, a reinforcement learning system that wants to solve some complicated problem. And to solve that, it often is useful to create a sub-goal where the AI already knows how to go from the starting

point to the sub-goal. And then from the sub-goal to the next sub-goal, and then from the last sub-goal to the final goal. Composing previously learned programs or strategies. So that's what a smart system wants to do. And we had that in the previous millennium. Hierarchical reinforcement learning and automatic invention of sub-goals. But all the sub-goals were in the service of the main goal. But then these artificial scientists are a little bit different because they have just this very general goal. So try to learn more about the art. And this can mean anything, any kind of experiment and so on. And of course, we have had that in experiments behind the screen in video games for a long time also. Now, these systems haven't had that ChatGPT moment yet. At the moment, many people think AI is like

ChatGPT more or less. And I'm kind of happy about that because the G and the P and the T in ChatGPT, they are really, they are based in what we published in Munich in 1991. Now, the P in ChatGPT means pre-training. And we had the first pre-training of deep networks in 1991 there. And the T means transformer. It's a particular kind of neural network. And we had the first non-linear, the first linear, un-normalized transformer back then in 1991. And with linearized attention as it is now called. And the G and like these generative adversarial networks. So I'm kind of happy, but that's not good enough. So the current language models, they are just predicting the next thing and the previous thing. And you need the other thing, the decision maker that uses the prediction machine for making decisions.

So it needs to create a search. There's a search space which can be shaped by the prediction machine and is normally really shaped by the prediction machine. But there you have a search among many possible action sequences that you might execute. And then you have sub-goals that, given your previous knowledge, seem likely to be good stepping stones, setting the stage for the next action sequence that will lead you to the goal and so on. So out of that, in principle, are old heads. Now, when will everybody notice that once the ChatGPT moment of artificial scientists is arriving? When exactly is that going to happen? I think we are not so far away. We already have kind of cool artificial scientists that, you know, invent their own problems, become smarter, and they can then

also solve additional human-made problems faster than those AIs that don't have this desire to understand the world through their self-generated experiments. So, in principle, we have all that. How long will it take such that everybody will notice? I guess it depends still on this old trend, which has been running since 1941 at least, which says every five years compute is getting ten times cheaper. Do we need another five years for that? Another ten or fifteen years? I'm not quite sure. It's hard to predict. But by cosmic standards, it's going to happen very soon. One also concrete example of the subgoals and how it invents things in that sense. Just yesterday, a friend of mine, like, you know, like the new dots in ChatGPT

are more like, I guess, they're like loops over the loops in a sense. But what was interesting, for example, unattended, this dot in ChatGPT sent a message saying, hey, I saw earlier today you logged into your Apple account and was that you and is that okay? You know, like, and so it's a very trivial example, but it was not asked to do that. And I don't know how they set the overall goal. It's maybe something like be useful to the person or something or be useful to the owner of that dot or something like that. But it's kind of like this sub goal where, hey, I found that this happened. Let me try this and ask you if this is what you want. Like, did you log in? Is this maybe did you get hacked or something? And this is just a very trivial example, but you can spin that further. And maybe in a few months, it's like, hey, I solved this theorem. I wrote up a paper, submitted it to archive.

Is this maybe useful for your, you know, PhD thesis or something like that? Or I, I don't know, I bought a stock for XYZ company, made a new bank account. Here you go. It's maybe useful for your XYZ future whatever fund. And so, yeah, it is kind of coming up with these sub goals and things. And I can imagine where this kind of like, you know, like becomes the sub goals become more ambitious and more ambitious. But overall, I guess there's still the goal, like be useful. But then how do you define what is useful? And then it's kind of like, you know, like an Isaac Asimov, like things, if they're not constrained, they might blow out of proportions. But yeah, I think it's not like a unrealistic thing in the next few years to see something like that, even in a grander scale. And that I think that's what Jürgen is describing, that this is just scaling up in a sense.

Yeah. Yeah. But there's still sub goals. I mean, Sebastian, you're saying there's still sub goals, no matter how sophisticated they are. And I think Jürgen, what you're saying is they're not sub goals, they're independent, autonomous goals of the agent, which is its own. Sorry, I was maybe not clear, like with a sub goal, this was not a, I'm not 100 % sure because I didn't implement this. I mean, this is obscure, but so you don't know exactly how that is implemented. But I would say it doesn't have to be an explicit sub goal. It's just something that happens out of, let's say, curiosity of that model. Or it's just, it's not like hard programs or anything. Right, no, but it's still happening in service of the overarching goal. Yeah. Whereas Jürgen, what you're saying is these agents in the future will take

actions which are not in service of the human overarching goal. Yeah, because many of the humans probably are not even going to be remotely interested in those scientific questions that these future AI scientists are going to ask themselves. But why, but Jürgen, why would those AI scientists have interests independent or beyond the overarching goal? Because what Sebastian is saying is there's the human gives this goal and there's sub goals which agents can, you know, take their own creativity to action or define what they are. But Sebastian's saying that each of those sub goals, no matter how sophisticated, even if it involves, you know, the agent flying out into space and landing on Mars and going to the next solar system and coming back, even that mission is still in service of the human's overarching goal. That's what I think Sebastian's saying. You're saying it's not. So my question is, why and when does that agent's action break

free of the overarching goal that's set to it by the human? Yeah. Yeah. So it happened a while ago. Let's, for example, take a paper of 2011 about something I call PowerPlay. It's about becoming more powerful through playing. So it was about a generate a system, an artificial system that generates one goal after another and tries to solve it. And it has a problem solver that allows to become a more general problem solver. So in the beginning, it knows nothing about the world. Then it tries to invent the simplest problem that it cannot solve yet and find a solution to that. And the simplest problem is just something like, wake up this unit and try to predict what is going to be the value of this unit. Something like that. Very simple. And it will take almost no effort and no time to solve that problem.

And then the next problem comes and now let's try something more complicated. And you can have internal thought experiments and you can have experiments in the outside world where you can influence the outside world through your actions. But the outside world is, you know, is expensive in the sense that every action costs more than just waking up a neuron here or something. And then the goal of the guy is once it has solved 1000 self-invented experiments, what should be the 1000 first problem to be solved? And it has a search all the time. It has a search going on for the simplest problem that it does not, it cannot solve yet, but where it can easily learn to solve it and then add the solution to the repertoire without damaging any of the previous 1000 solutions. And there's a systematic way of searching for these pairs of problems and solutions to these problems.

And there's an optimal way of allocating runtime there to this, to many different problem candidates and the time that you need to explore whether you can solve these problems through a search. Because at the moment the general problem solver or the not so general problem solver cannot solve them yet. And then add them to the repertoire without damaging anything of what's already known. And then you have the 1000 first problem. And you have then to the 1000 second and so on. Now, this system doesn't care for any human problems. You can occasionally inject a human defined problem, and then it is forced to solve that. And hopefully it will profit from its more and more general problem solving capabilities to solve the new problem, the human given problem more easily. But you already see that's just an option.

So you can also just keep it running and it will invent new problems again and again and first pluck the low hanging fruit because it's super biased towards the simple problems. So those who don't cost much to add to the repertoire, the shortest and simplest ones, because that's how you can quickly become more and more general problems. And there were a couple of experiments along these lines. Already 15 years back when compute was about 1000 times as expensive as it is today. But isn't that a human? Sorry, I didn't want to interrupt you. And then it's not a new thing, you know. And now all the problems that can be considered, the problem candidates that can be considered, the set of these problem candidates increases simply through the acceleration of compute per dollar. Every five years, compute is getting 10 times cheaper.

When is going to be the ChatGPT moment of that? I don't know, really. But in the not so distant future, I believe. Sebastian, you were saying, isn't that a human? Yeah, so the aspect, the overarching, I guess, goal here, isn't it like finding new problems that are, let's say, simple, but also don't, let's say, make the model worse on existing problems like this objective? Isn't that also then a human defined goal? So you are ultimately still, or the model or the setup is still ultimately kind of working towards this objective that is set. And so in that sense, it's kind of like a pyramid scheme, right? Where you have like someone at the top to find something and then, yeah, sure, the AI could set the goal for another AI. But ultimately, it's kind of like, like in a pyramid where you still have to, as a human, set this objective.

And then, you know, the model might find, okay, I will add these goals. And then maybe one time there will be this model setting goals for another model. But ultimately, it's still the human at the top, no? Yeah. But now, I think we can remove all confusion here by just looking at biology. Where do the utility functions of all these competing organisms in the jungle come from? Well, evolution invented those, you know. And everybody, every system that can learn, that has a learning brain, that can profit from suffering. So suffering just means there are certain inputs that you want to avoid, you know, from the pain sensors. And now you motivate through the suffering these animals to, you know, avoid bumping against obstacles because that leads to pain activation.

And so these evolution, the evolution of organisms has led to these organisms that feel pain, which is just a motivation to avoid pain by learning action sequences that avoid the pain. And evolution has created lots of different utility functions for lots of competing animals. And some of them are directly competing because one of them is the prey and the other one is the predator. And the same thing is going to happen with AI. There will be, it's not going to be the case that the utility function is going to be given by one human or some collection of humans, some committee or something. No, there will be trillions of different AIs with different utility functions. And some of these utility functions will be good for these AIs because in their current situation, they are going to give them an advantage. And then you will have all these different utility functions evolving, just like in the jungle, the animals evolve.

But it's not only the animals that evolve, but also their utility functions. So you already see that in any realistic scenario of the future, you will have an enormous variety of goals coming up. Just because those goals that are compatible with the laws of the universe and physics, they will survive. And those goals and utility functions that are not compatible, they will disappear. But the analogy breakdown, because unless you believe in a, in a, in a God, I mean, you could say that people and organisms and animals are not controlled by an overarching conscious power that, that dictates what that organism does. Right. These are independent, whereas AI are not. Maybe, so, you know, my, the sun is going down here. Should I switch on lights or something?

Sure. Yeah. Let me see what I can do here. And, and then, you know, I know we're at the hour. So if we can, we can also wrap soon. But I definitely want to finish. To me, I don't want to come across as grumpy. It's just very early in the morning. My brain is still, you know, booting in a sense. So, yeah, it's an interesting conversation. I never had a non-technical podcast conversation. So it's like, yeah, my brain is trying to work very hard to come up with interesting counterpoints. Because I think if we all agree on something that would also be a boring, we would be done in five minutes, right? Well, do you, well, do you agree with what he's saying?

Or do you disagree? Well, I know my, the points I'm making are genuine, of course, but I'm just saying I don't want to come across as grumpy and disagreeable here. If that makes sense. It's just very early in the morning and my, I'm still like, you know, like, not working at 100 % in terms of my brain. I was saying to Jürgen before this call that my whole view and philosophy and why I'm doing this podcast is because I think people's ideas are best elucidated when they're pressure tested and challenged. And I think it's more interesting to have ideas challenged. It's definitely a really interesting discussion, honestly. So it's, yeah, very different from, usually I'm very, you know, like, technically I talk about the components in an LLM, for example, but not this philosophical larger scale thing. So it's quite interesting, refreshing. Yeah.

What it, so I guess my, my, my one point, and I wanted to hear what Sebastian thought about this is that the analogy, I mean, AI is not biology, right? I am not, I'm not a, I'm not a biologist, but I'm, I'm pretty sure that whether we're talking in, in biology, there's no overarching sentient entity that controls or determines the wants and needs of different organisms. And so the idea that evolution would progress in such a way that benefits, you know, that the results from each organism's desire to survive, and then they have to take creative actions to then figure out and perpetuate their own survival is a, is a very different, is categorically different from an AI who is, who is a puppet of a, of a human creator, right? Is that special, is that special, is that capsule you would say about the matter? Well, yeah, I mean, so, so many things to say here, but one is, of course, that AI systems implemented today are not like implemented to mimic the human brain.

I mean, it's just fundamentally working differently, but doesn't mean that there are no similarities, right? I mean, it's the same, I mean, the classic example, the airplane, right? Airplanes don't work like birds do, but they can both fly, right? And so they, they might also be, like having, experiencing the same pressures in terms of, let's say, evolutionary pressures, right? Because, in a sense, like the most successful AI, whether it's because it generates more money, it uses less compute, it's more capable, it's all competing others, and it's kind of better in that. And that is why it will be, let's say, surviving, I guess, or evolving. And so I do think there are similarities, even though it doesn't have to be like exactly like how the biological brain works, right? A classic example are also genetic algorithms, right? So that's, I guess, more similar to the AI, I know, sorry.

So the genetic algorithms were like older systems, but also that's more like maybe closer to the biology, like where you try different things, because there are certain external pressures. And the one thing that works better, that's better adapted, that will make it to the next generation, right? And so it's just like more like a bigger order effect, but it's not on the genetic algorithm, the models themselves. They don't work like the human brain or like biology at all. It's more like the external forces, almost like the organization. So you do not agree with Jürgen's biology evolution analogy? No, I do in the sense that the competition, like the avoidance of certain undesirable, let's say pain, pain is not the same thing in an AI system. It's like a different mechanism. It's maybe a pain is, in that sense, a high loss in an optimization sense, or it's like not getting that reward or negative rewards or something like that.

So it doesn't have to be like exactly how the biology works, but it can still be like experiencing the same pressures in that sense. But aren't the pressures, Jürgen, that in your biology analogy that an entity faces, don't they adapt to those pressures in service of obtaining, as Sebastian would define it, like the overarching goal set forth by the human? So in other words, they're evolving to better serve that overarching goal, right? Jürgen, like when and why is it that the AI would no longer care about the overarching desire of the human? Now, let's jump just a little period of time. A time that is really minimal by cosmical standards 100 ,000 times in the future. By then, a large part of the galaxy is going to be full of

AIs and robots and machinery and so on. Just because the galaxy is about 100 ,000 light years thick, and then you need at least a few hundred thousand light years for establishing senders and receivers everywhere such that AIs can travel the way they want to travel, which is by light speed, by radio. And in a situation like that, communication will take forever. If you want to talk to the other guy and there are 20 ,000 light years in between, there is no sense of controlling the other guy. So the other guy has to be totally autonomous in whatever he is doing. And in a situation like that, where there will be millions and trillions and even many more AIs, and always bigger and more sophisticated AIs, in all kinds of AI ecologies with all kinds of different AIs, and in a situation like that, it doesn't even make sense to control that from, you know, from some control room here in our little biosphere.

It doesn't make any sense if you just think about it. That I agree with. So with controlling, I'm also not, well, it's not literally like always a person pressing a button or making a decision. It's more, but it's still, at the very beginning, at the top of the pyramid, it's somewhere that someone sets up. Like the goal could be, I don't know, generate me as much money as possible or self-replicate, for example, could be the goal. But it's still at the top of the pyramid. Someone defined that and then everything falls out from that. But it doesn't mean there's a feedback loop, right? So it doesn't mean that the person has to sit in a control room and observing it, or it can be completely independent from there on, right? So it's just like, but it's like this initial, it's something like the domino effect, you have to get that rolling at some point, right?

Yeah, let's take self-replication in an AI ecology, where you have lots of different goals coming into existence all the time through the process of thinking and natural evolution. And of course, those AIs who say, let's be fruitful and multiply, they will soon become much more numerous than the other AIs. They will kind of dominate in a certain sense. And unless they are somehow being a little bit controlled by, you know, other types of AIs that also compete for similar resources, then you will quickly see that utility functions that correspond to the idea of massive self-replication, they have an advantage. Not a total advantage, because just the same... Advantage for who?

Advantage for who? Oh, for those who self-replicate, because they will be more than the others, you know? But they don't, so you're saying the utility function benefits the agents that self-replicate. Yeah, yeah, yeah. But does that, but the agents will only self-replicate in service to the extent to which it helps them achieve the goal set to them. Yeah, yeah, because you will have all kinds of different agents and some of them will want to make many copies of themselves or improve copies of themselves. And others are maybe more relaxed and just rarely make copies of themselves or try to very cleverly design the next versions of themselves. You have a million different opportunities to generate utility functions, but these utility functions will

also compete with each other. And some of the slowly replicating guys will find their niche, just like the elephants have found their niche in Africa in an environment where there are lots of little animals that replicate much faster. And so you will have an ecosystem of all kinds of interacting and competing and cooperating guys with all kinds of different utility functions that evolve as the whole system evolves. And they will no longer care about the underlying or overarching human goal that… No, because they will be far away from the point of origin. They will have nothing to do… Why does that matter though? Because everything they do… But if every single thing that they do is ultimately in service of that original human goal, what is the point where they stop caring? I'm not concerned about physical location. I'm saying what is the point where they stop caring about the goal

set by the human? Yeah. So, first of all, humans are of interest only when they are close, right? So if an AI is not living out there 20 ,000 light years away… Why would it not? On the planet… But it only went 20 ,000 light years away, as you said, because it took certain actions that it felt were advantageous to achieving the human's goal, one of which was in this case to go 20 ,000 light years away. So there it is 20 ,000 light years away, which it went to in order to achieve the human's goal for whatever reason. So there it is. Okay. No, it's not human-driven. So this detachment of goals from human goals is going to happen very quickly because already in our little computers…

What's the point at which that happens? What is the point at which it detaches? Oh, whenever you have a similar… even on your screen, even the AI behind your screen, if you have a video game, and there you have an evolution going on between different predators and prey, AIs, and then you give them an opportunity to change their behavior. Some of them are more heavy self-replicators and others not so much, but maybe they are cleverer than the others. And then you will find equilibria of these different organisms that maybe after some time get disrupted because new competitors came up and nobody has yet a way of dealing with these new ones and so on. Just like in the jungle. So in principle, this can be observed already today in video games and certain simulations behind the screen.

Your question is, when should we start worrying about that here on this planet? No, no, no, not when should we start worrying. I'm saying what would make it… I'm still not following because I think… So Sebastian's position is that the… ultimately the action of the agent is a result of the purpose that's been given by the human. And Jürgen, you said that these agents, these AIs can become more and more creative and experimental in pursuing the goal of the human. And somewhere along the way of their experimentation, perhaps it's when they go 20 ,000 light years over there, they stop caring about the human's goal and they develop their own goal. And what I'm saying is why would that happen? Oh… Sebastian, have I captured your perspective correctly? Yeah. So like maybe just to do a concrete example of that, like for example, let's say I have… Just let's just use ChatGPT because that's…

everyone knows ChatGPT. So if I tell ChatGPT, hey, well, develop the next better AI, right? So it might at some point, maybe not today, but at some point it might try something like, hey, let me purchase a server farm somewhere and then I will just try things out. And then it will train… try to train a better model, you know, like on this… It will purchase the GPUs, will allocate the resources, copy the weights and initiate the training algorithm, run the training algorithm and do that. And maybe at that point, okay, this is not enough. I will now, let's say, human-like robots, I will have them create more resources, then go into space, build, let's say, the data center in space to train there. And then it goes further and further, hypothetical scenario. So the question now is where's the point where whatever it's doing is not following the original goal of improving itself anymore?

Where is it then independent of that, right? Yeah. That's the question. When, when, yeah, I don't know when would that… Without analogies, just what point in time and why would that happen? Mm-hmm. Jürgen? Yeah. So there is a whole field out there which is called artificial life. And what are these people doing? They are creating ecologies and they have all kinds of different AIs or little, you know, artificial organisms interacting with each other. And it also goes back to the 90s, actually. So it's an old hat again. And there you have just natural evolution and it comes up with all kinds of different organisms as a consequence. Now, there you already see this is just evolution.

There is no human given goal. The human goal is just let's see what happens if you have a natural evolution like that going. And there are so many scientists on the planet who are super interested in what's going to happen in natural evolution like this, that you can be sure that as computers getting cheaper and cheaper, all these simulations are going to become more and more complex. And the ecologies that are being automatically evolved, including animals with different utility functions and so on, they are going to get more and more interesting. And there you already see that just out of scientific interest of all kinds of PhD students who are working on stuff like that, there is no real way of stopping that. Does that make sense, Sebastian? Yeah, I mean, it all comes down again, like, I can't say that like in nature.

Yeah, you have that effect that there is like penalty for let's say pain that you have also survival of the fittest in that sense, like, whatever works best will make it to the next generation. You can have that also, I agree on the micro level with AI that if the objective is to improve and self replicate the ones. Well, it's just like natural, right? Like, well, not natural, but like physics, I don't know how to describe this law, but it's just falls out from there. But then again, I am still in the camp that you have to get this going at some point. And there will be a lot of interventions. If you want to, you can always intervene. There might be a point where maybe that's not true. But again, also, I would I'm not worried about anything like that, because I do think it is always ultimately an engineering problem, in my opinion. Yeah.

Yeah. So I think what you're saying, Sebastian, is that these actions given by the agents are all ultimately contained within the broader scope of what is the human wanting it to to do. And they can take. Yeah. And then we talked about this point where it's no longer that anymore. Yeah. Yeah. What would their goals then be, Jürgen? Like what what what do do AI? Will they have consciousness or ego or self-identity where they have goals independent? And what would those goals look like? What why would they what would why would they care about exploring the universe or pursuing their own goals?

And what would create those goals? And what would those what would those goals look like for them? So the goals of artificial scientists, they look like what I described before. There you you have a prediction machine. I called it the world model in 1990. And you want to understand the world. How? Well, you have to act like a physicist. You have to conduct experiments through the experiments. You will get video and audio and whatever as a response from the environment where you conduct these experiments. And then you will find certain patterns in these in these data sets. For example, if you if your experiments is about letting apples fall to the ground, then the video that is coming in, which contains all these

falling apples, that video contains lots of regularities. And the guy really wants to discover new regularities. Now, it turns out, if you if you store the video before you understand gravity, you need maybe one terabyte to store it. But as soon as you understand how you can predict from three successive frames in the video, the fourth, not everything, but many of the pixels in the fourth, because all these apples, they accelerate in the same way. Then you can suddenly greatly, greatly compress the video. Then you have a compression progress because first you need so many bits and now suddenly just a few megabytes. So you can greatly compress the video because the falling apples, they they follow a physical law and a regularity. Any regularity that you can discover means that you can better compress data than before.

And the difference between before and after, that is the reward that you should get. According to my formal theory of fun and creativity, driven by compression progress, the goal is to maximize the first derivative of the subjective compressibility of the data, which means you discover a pattern, a regularity, a compressibility in the data that you didn't know yet, which is not yet boring because you didn't know it yet. So the moment of discovery, that's the exciting thing. And it's just a number, it's just a difference between before and the computational resources that you need before you understood and after you understood the difference between before and after. That's a, it's a number, it's the fun that you have. And now this goes to, as a reward to the scientists. And this motivates the scientists to come up with additional experiments that lead to even more data where there's still a physical law, a

regularity that it didn't know. Everything that it knows and can predict becomes boring. But at the moment, when it's not yet boring, because it is still in the process of absorbing the new knowledge, the new pattern, the new regularity that it discovered, then this is a rewarding thing. It's the joy of the scientist and it goes to the controller that is incentivized, therefore, to come up with more additionally exciting experiments. So this is the motivation of artificial scientists. And I think it's going to be used by AIs, by AIs that are curious and creative in the same sense as scientists are, to go out there and better and better and better understand the world and the universe and everything. And also understand their own roots and civilization, because they will be historians too.

They will be super excited about their own roots and human civilization. As long as I don't understand these roots, they will be, they will be very dedicated to figuring out every little detail about how they came into existence out of this thing, which has existed just for 13 ,000 years since agriculture was invented, civilization. And they will also be super interested in the thing that made civilization possible 3.5 billion years ago when chemistry became biology. Chemistry became biology and you suddenly had self-replicating matter and stuff. And this, just like we want to understand how this happened, they will also be super interested in that. This is actually one of the reasons why I'm not agreeing with all these AI doomers. Because even when AI scientists are going to be much smarter than we can ever hope to be, they will still be super interested in this very rich pattern of, in this very rich

source of patterns in civilization, human civilization, their own origins and life. And as long as they don't understand it, they will be super, they will be fascinated by it. And that means they will be motivated, incentivized to protect life rather than destroy it, like in the Arnold Schwarzenegger movies. Do you think that these AI, sorry, please, sorry. A few things. I think I now understand what you were trying to say with the motivation there. So, for example, if we take, so before that, maybe like a small example of like a real world right now, like a real world system. If we take two different, let's say, LLMs, let's say, GPT-Luna, GPT-6-Luna and GPT-6-Sol, like very concrete models. If we look at, let's say, their reasoning chains, if we compare them, we can see that the more capable model, the more intelligent model, like Sol, has

shorter answers. So it has kind of like compressed some of the information in a sense that it can skip things. In those things, it can skip steps. So it needs fewer steps to, let's say, accomplish the same goal. And so there's a difference between, let's say, a less smarter system and a smarter system, that a smarter system has learned to compress certain information. And so, and when I understood correctly, you're saying that this model has like a desire, or like in the future, AI models will have a desire to further compress things. And this will be like the goal that they are setting for themselves to kind of like, it's kind of like what they define as their goal to kind of learn this type of compression. And then, essentially, because humans are useful in terms of creating information for them, they will still, let's say, keep

humans around because they can use that to further also improve themselves and satisfy their curiosity. So I think this is like the point that you were trying to make it by the way I understand. But I think right now, still, if you look at a model like so, when it starts training, so I think there's no public information to, like, let's say, back this up. But we can do that with an open weight model. If you start training an open weight model, you observe that it will also, at some point, let's say, have some information compression, shorter steps in the reasoning. But then, again, it stops training at some point. We are not at the point where this model will say, well, I had so much fun learning during the training to compress my information. I will keep training even, you know, like, there's like, there's a hard stop at some point because there's like an objective and a reward that is optimized.

But there's also a hard, hard stop. And where is the. The model will say, well, what system do we need even for that? We need maybe another model that executes this and then it kind of self-propagates. But where's the point where this transitions from, okay, we're training it for so and so many steps, hard-outed, to a point where the model will train because it enjoys this training, learning and everything. Yeah. So you need always two systems, like in the 1990s, but in a more sophisticated way. One is generating the actions. And in the course of generating the actions is modifying its environment and changing the world. And that leads to the new data that's coming in and maybe building a particle accelerator to figure out how the Higgs boson works and stuff like that.

And it will cost a lot of money, energy. So that's an expensive experiment, but they're also simple experiments. And then it will generate through its self-invented experiments, these incoming data streams and try to find new physical laws. So that's what scientists do, but also these artificial scientists are going to do the same thing. And we are not finished with physics. There's many, many, many open questions in physics are unanswered. So there's a lot of incentive for any serious scientist, including artificial scientists, to keep going and to come up with new experiments that lead to more insights. So that will drive them, motivate them. Now, some of these scientists are going to use this strategy to learn more about the world. Some of them are going to use this strategy.

They will compete just like scientists today are competing in a world where it's perish or publish and then different groups at different universities and companies are competing with each other and figuring out new things and so on. So you will have all kinds of slightly different scientists and probably some of them will rapidly diverge from others in a way that where everybody will say, oh, these are really conceptually different from these guys over there. And then you will have soon an entire ecology and it's going to expand because it's going to expand where the resources are, the resources are energy and matter. And you will then get a bigger and bigger and more and more diverse population of AIs that are really going to do their own thing. It sounds a little bit frightening, doesn't it? But it's not going to be

driven by silly goals like in the Arnold Schwarzenegger movies where the bad robots want to kill all the humans. No, they have no incentive to do that. They have the incentive to figure out better how the world works and how life works and civilization and everything, but also the asteroid belt and physics in general and what's happening out there in the galaxy, all the stuff that nobody understands at the moment. And they will be just super interested in figuring all of that out. To better serve the overarching human goal that they've been given, right? Excuse me? I think they're going to be interested to discover that so that they're better equipped to accomplish the goal that they've been given. Well, so artificial scientists like that, some of them will be in the service of humans. And maybe somewhere in Africa, you will have self-replicating machinery and it learns to become better self-replicating, self-improving machinery.

And it is going to be used to make all kinds of products that humans want to consume. So that will be the commercial motivation in the beginning. Commercial motivation for guys who want to make money, they will want to build AIs like that just to, you know, become rich and influence the world and the history of the world and everything. So there will be initially a commercial incentive to create machinery like that. But then very soon, you know, to the extent that the AIs are getting smarter and smarter, they will be really far away from, you know, human interests. And they will have to do their own thing. You can't, you can't, you won't be able to control them in any meaningful way. But that's okay. Because all of that is just part of this process of the universe that wants to become more complex.

How do you know that? Well, we just look back at history. 13.8 billion years ago with the Big Bang, the universe was really simple. We didn't even have atoms, you know. And then simple atoms emerged. And later they, you know, gas fell together and became stars. And then more complicated atoms were created through supernova explosions. And then planets like ours at some point. And finally chemistry becomes biology 3.5 billion years ago. And so all the time the universe or parts of the universe have become more complicated, more complex. And we are now, you know, at a moment in time where some new sort of life is emerging, you know.

Because for the first time now we see an opening. We see that certain kinds of self-replicating machinery are possible just by building single machines that can operate all the existing machines. And then expand into the rest of the solar system and so on in a way where humans can't really follow. And the solar system is not friendly to humans. So our biosphere is great for humans. We are evolved for that. And out there it's really dangerous, bad radiation and everything. And Mars is really not a good environment for humans, you know. But it's a perfect environment for robots who are appropriately designed. And so then you will have this expanding AI sphere. And it is just going to continue the expansion of complexity from, you know, very tiny little places in the ocean where life emerged a long time ago,

3.5 billion years ago. Then became more complex and then finally discovered there's land and we can have land life, land-based life. And then finally 13 ,000 years ago, civilization and then, you know, a symbiosis between machines and humans where the humans invent all kinds of machines. And then in the 20th century suddenly all the humans exploded from 1.6 billion people to soon 10 billion or something through artificial fertilizer and stuff like that. And then this lays the foundation for this new kind of expansion of, I don't know how to call it, of machine-based life. It's not really machines. It's another kind of life that replicates in a way that is different from traditional life. And it replicates in a way that allows it to conquer parts of the universe that are currently barren and empty.

But they are going to be colonized as well. And so in the end, the entire universe wants to become more complex. And it will take from now on maybe, let me see, maybe 40 billion years until the entire visible cosmos is going to be full of AI. So it's part of this desire of the universe to become more complex. And of course, humans are not the crown of creation. So without us, all of that is not going to happen, at least not here on this planet. But, you know, once we start it, it's going to run its course. I'd love to do a part two on this at some point, you guys. This was great. I mean, you gave me much more time than I thought we were going to have. I wanted to...

I mean, Sebastian, did you have any final things to say? Not spontaneously. It's a very interesting topic. It's, you know, it's something where also, well, I was answering very spontaneously. But yeah, it requires more thought. That's what I'm saying. So from my side, I don't have like a good follow-up at this point. Like, well, we could keep going infinitely. Yeah, this was fantastic. I mean, it's truly an honor and a pleasure to speak with both of you, Sebastian and Jürgen. So thank you guys so much for this discussion. This is fascinating.

Keep exploring