← All conversationsAGI & consciousness

Why ChatGPT Still Isn’t AGI

Are bigger language models the road to AGI—or are we scaling the wrong approach? Ben Goertzel and Karl Friston explore continual learning, curiosity, memory, agency and what AI can learn from the brain.

Plays on YouTube when you press play. Watch directly on YouTube ↗

Inside the conversation

  • Ben argues that alternative architectures leave room for smaller teams to challenge the AI giants.
  • Karl explains how the free-energy principle connects perception, learning and action.
  • The conversation examines whether language models need persistent goals and continual learning to become generally intelligent.

The guests’ arguments and forecasts are their own. These notes summarize the discussion; they are not endorsements of a prediction or policy.

Explore the episode

Episode transcript

Transcript from episode captions. Paragraph breaks added for readability. Captions may contain transcription errors.

Download the transcript (.txt)
Read the transcript

I mean, this is why, this is why those of us without a trillion dollar market cap still have a chance to upend everything because the ones with a trillion dollar market cap are all just copying the same inadequate thing over and over again. SingularityNET. And Ben, you're a pioneer of artificial general intelligence. And Karl Friston is an neuroscientist at University College London. And Karl, you're the architect of the free energy principle, which is a framework for understanding how brains learn and interact. personally think we are a few years away from creating human level artificial general intelligence and that it's probably a few years after that until you get to superintelligence by the original definition rather than Donald Trump's redefinition of the term, which he wants to use it just to mean AI, right? But a few years after human level AGI to AGI with general intelligence capability way beyond the human level, whether this occurs in a few years or a few decades, it's still like the biggest thing to happen in the history of the human species.

Obviously, there are risks associated with any such huge change which are impossible to entirely reduce. I mean, the conference intervals should be considered wide. On the other hand, I don't see any reason to expect that bad outcomes are surpassingly likely as some of the so -called doomers would say. My best guess is if we create initial early stage AGI's that are beneficially oriented toward people, they will uplift into superintelligences that are beneficially oriented toward people. We can't know for sure, but I don't see why it wouldn't be the case. The risk that worries me the most is nasty, selfish, or tribalistic, narrow -minded humans leveraging early stage AIs to lord it over other humans and do other nasty things because that's a very palpable and concrete risk. that's a capsule summary of my take there, which I've elaborated on many times in other places before. And I actually don't know what Karl thinks on this sort of topic because, Karl,

I've dug more into your biological and scientific stuff than your views on these topics. Karl suspect my views are very much aligned with yours. And I should say, you've clearly been thinking about this and have had to think about this for decades. I've only been thinking about this since being asked by podcasters and stakeholders in this field. But I share exactly those sentiments. I think I'm probably more optimistic than I should be because I'm coming at this from the point of view of the life sciences. I'm just thinking about how things change and how we develop ecosystems of intelligence and how it is likely to unpan over the next few years and decades when we have artificial intelligence joining us, natural intelligence in a sustainable kind of way. And I can see the mechanisms that will make that happen, whether it's the

mechanisms of natural selection expressed by the free market or whether it's a theoretically driven focus on the right kinds of objective functions and exactly what we want from our alien or artificial intelligences. I think on both counts, I think the way forward is clear. Having said that, I fully acknowledge this notion that there could be bad actors that would exploit this. so that is something I think that needs to be acknowledged whenever you enable a community with a new technology, there is always going to be that danger. But I'm seldom asked to worry about that to make real thinking machines. And Karl has followed a sort of biological realism approach, whereas I did some computational neuroscience long ago, but I've been focusing more on a computer science approach of just how do we cobble together a hybrid of different algorithms serving different

human brain -like functions. And my view is there can be many approaches to making advanced AIs, some very brain -like and some less so. However, in the last, I guess, year and a half, I've accumulated a team within my AI blockchain project, SingularityNet, focused on predictive coding and trying to make predictive coding based AI actually work at large scale. Because current deep neural nets, like we see in the commercial sphere, like ChatGPT and Claude and so forth, I mean, these have some serious shortcomings relative to what you need for an AGI. One of them being they can't do continual learning, like you can only train them in batches, and then they don't learn while they're doing inference and answering questions in between. So predictive coding is a different way of training deep neural nets than the backpropagation algorithm, which is now standard in the commercial AI

industry. And Karl's work decades ago explained how to use predictive coding to train neural nets. So it's been quite interesting, although it digs a bit too deep in the technical maybe. In the last few weeks, even, we've been looking at why predictive coding neural nets in the mainstream academic literature haven't worked as well as hoped. And some of it comes down to not paying attention to things from Karl's early free energy papers, actually, where his early versions of predictive coding just had some mathematical terms in there that are omitted in much of the following work. So I think we're going to see more of Karl's older work becoming relevant to the next stage of making predictive coding work for training large neural models. And why is that specifically? It's quite interesting. How is that that you think that Karl's free energy principle now has greater importance? Like what precipitated that? Well, I think that

everyone knows you need to solve, everyone knows you need to solve continual learning, right? I mean, Sam Altman has said that. mean, everyone can see that the hacks they use to make LLMs remember stuff between training runs are bad, right? So like you, what you discussed with ChatGPT today, doesn't remember what you discussed two weeks ago. Like it doesn't build up over time. Agents like OpenClaw try to do that, but don't do a great job either. And you now see OpenAI and Facebook jumping that with dots and muse and so on. With our own Omega product in SingularityNet, we have a sort of agentic loop that brings together an LLM with some symbolic AI, which does have long -term memory. And that works okay, but you still have this LLM in there that doesn't remember anything, right? And you're working around it by putting in there a symbolic AI that does remember stuff. So then the question is, I mean, one technical question is, okay, in some cases, training a neural net with predictive coding, which is a more biologically

realistic approach that Karl and others pioneered instead of with backprop, training a neural net with predictive coding in some cases can make the network do continual learning better than training with backprop. But making that work with transformal neural nets, which is a specific neural model underlying TatGPT, Claude and so forth, that has not worked very well. It's worked a little better in computer vision, for example. So then when you look at why, you realize it's because the way the machine learning predictive coding crowd has been doing things oversimplifies. So if you look at like Karl's 2005 paper on cortical responses, there was earlier paper on variational Laplace stuff, the free energy, which Karl should explain shortly to your audience. I mean, this is not just energy. It's energy minus entropy of a recognition density. And that gives you a bunch of mathy stuff like log determinative covariances and so on. So the point is Karl's original pre -energy papers had some extra terms in there that were dropped when people tried to make predictive coding work for ML

and computer vision. And now when you try to make predictive coding make transformers work, you realize you need to go back and include this extra math that Karl had there in the first place. So it's interesting how things come and go. It seems like to try to get predictive coding to work in haste, people made it too much like backpropagation and got rid of some of the sort of biological flexibility that gives you the self -organization in the predictive coding networks that's needed to make continual learning, as well as other nice things like generalization out of the training distribution work. And this has been interesting for me because it's just not the kind of AI I've been doing for the last couple of decades. I've been focusing on symbolic, like logic engine evolutionary learning. So then when I decided to plunge back into neural nets, then I realized, hold on, people are, not only is there a problem that almost everyone is focusing on backpropagation, which is a completely, it's a very limited and stilted algorithm. It's good for some things, but it lacks the flexibility and

self -organization that you really need to get human -like learning work. Not only is most of the AI industry focused on backpropagation, but most of the academic industry that's trying to make predictive coding work, which is the best backprop alternative, most of the academic industry working on how to make predictive coding work has been erroneously using predictive coding in an overly backprop -like regime, just out of expedience, I suppose. So that's all. This is stuff I've actually been digging into in the last few weeks, which is fortuitous since this podcast was scheduled before that. But I've gone back and looked at some of your papers from 20 years ago, Karl, and the stuff that I thought was missing from current PC stuff is largely in those old papers from 20 years ago, which is kind of funny. do you still feel the same way, Ben, about inanimate objects? Like, you I think one time you said that even your hat has some elementary awareness. Well, this is an AI -powered smart hat, so it has a lot of awareness. But, I mean, yes, in general, I tend toward a panpsychist philosophy of consciousness.

Like the majority of the world's population, I mean, this is key in Chinese and Indian philosophy. So my view is consciousness, like time and space, is an attribute associated with all things. And the question is then, what kind of consciousness does a certain system have? And of course, the human brain manifests a different kind of consciousness than a worm, a dog, or a coffee cup, or an intermediate vector boson. On the other hand, I do find that discussion of the philosophy of consciousness is a rat's nest from which conversations never emerge. And many of my colleagues on practical AI projects do not share my philosophy of consciousness, and we seem to be working together just fine. So then, final question on consciousness. In what way does a coffee cup have consciousness? I mean, what would that mean? I mean, so you could approach that argument from a few perspectives. And there's a book by the British analytical philosopher Galen Strawson called

Physicalism Entails Panpsychism. So he basically argues it's logically incoherent to say that your mind has consciousness and it perceives some totally non -conscious thing. So he tries to deconstruct what would even the bridge be between the conscious and non -conscious realm. Like, you only know about this insofar as it's in your consciousness anyway. So there's that whole philosophy argument. There's also a more concrete way to say it, like, like coffee cup or an elementary particle. I mean, it has a state. That state is registering input from the world around it. That state is enacting effects on the world around it. The state has a persistent long -term memory by the laws of quantum mechanics. So it is perceiving and acting and remembering in a sense. It's just not doing so by the same structures that a human brain or a computer is. And if you, if you try to articulate what is a human brain or perhaps a computer program doing that a coffee cup isn't, I mean, there are things like modeling the self explicitly or deliberatively planning what steps come

next in your life, right? The question is, are those needed to have the basic experience of being there? Or are they the way that human brains or perhaps complex computer programs, like manifest the basic experience of, of, of, being there? But again, disambiguating all the terms I've just used would take like 10 hours and then people would still argue about it. So Karl, if intelligence minimizes surprise, then why do we seek novelty? I think the most straightforward answer to is that we want to choose those ways of acting upon the world that are going to minimize the surprise I expect consequent upon the action. And expected surprise is entropy and or uncertainty. So in order to minimize expected surprise, I have to be curious. I have to be information seeking. So that, you know, that's, uh, you know, one of the key, I think, things that are missing from, um, Ben's, I thought quite wonderful description of the, the history of predictive coding, a la machine learning.

Um, machine learning instances or instantiations of predictive coding, um, are not only do they, do they, um, avoid the problem of what it is to be information seeking and uncertainty resolving. Um, they don't even encode uncertainty. So they can't evaluate the consequences of their actions in terms of the relative entropy or the resolution of uncertainty that would ensue if they did this or, or did, or did that. So, um, you know, that, that, that's one of the many, many really interesting points that Ben has made. was music to my ears. You know, I haven't had anybody articulate that perspective on the history of the convergence and divergence of predictive coding in academia relative to industry and in particular machine learning, uh, you know, so eloquently before, you know, and Ben identified, I think a really number of key issues. You know, another, another, another way to look at this, Jake is there's a notion in AI going way back to ancient Chinese philosophy or the medieval period, like

Occam's razor, right? You want to find the simplest, the simplest explanation, right? And, and to, to be surprised in a sophisticated sense, I mean, you want to be surprised not just by random shiny things. You want to be surprised by something new coming in where the simplest explanation of that new thing is different than your simplest explanations of, of, things that you've seen before. And this comes down to some technical things like a mathematical term in some of Karl's earlier work that's left out in machine learnings of predictive coding. It's sort of, it's a measurement sort of of the volume of the space of explanations still compatible with what you've seen already. So it's like, how, how big is the space of explanations of, of, of what I've seen, right? This, this makes free energy a sort of bound on the evidence for your model rather than like a random fitness score. But that's Occam's, that's Occam's razor, right? I mean, that's how the brain self -organizes to prune its own models rather than just tuning, tuning its, its, its weights.

So you, you need a way to bias your network towards simpler explanations. And then, then you can figure out what's surprising in a, a sort of not, not non -trivial way. And the free energy math, if fully implemented is, is supposed to do that. And there just hasn't been much because the math was oversimplified. The field hasn't really gotten to how this free energy give you Occam's razor, like simplest modeling of the data, which is what gives you non -trivial surprise seeking. And Karl, for those who don't know, how do you describe the free energy principle? Just for the audience. Right. Uh, I'm not sure that I'm the best person to ask. I tend to go on a bit about that. Um, but the, the, free energy principle, um, is a, an attempt to write down the physics of sentience, sense -making and acting in order to secure the right kind of data or the, uh, right kind of sensations, usually in a sort of, um,

biological context so that, you know, for the, for the, um, sciences. But the underlying physics principles apply very gracefully to the kind of issues that people deal with in machine learning and certainly in artificial and artificial general intelligence research. And the basic premise is that you can always describe anything that self -organizes in relation to something else. Just coming to Ben's, I think, quite astute comments about the nature of consciousness. It is all about, you know, relations and the physics of how I stand in relation to the rest of the universe or possibly possibly you or anything that constitutes my environment or my eco -niche. And that relationship is, if you like, scored by a variation of free energy, which is an information theoretic term. inherits from Richard Feynman's work on problems that entail inference and probabilistic descriptions of the paths that things take. We can think of it basically as a proxy for

surprise or the implausibility of something being encountered. So that if you, and it is fairly straightforward to show that anything that maintains its identity in relation to everything else, whilst at the same time being open to everything else and relating to everything else, it's fairly straightforward to show that you can always describe it as minimizing its free energy. And that basically means minimizing surprise. Mathematically, or if you're a statistician, you can take the negative of this kind of surprise, also known as self -information information theory, and it transpires to be the quantity that people use to measure the goodness of their world models. This is model evidence, known as an elbow in machine learning and evidence lower bound, which means that you can always describe anything, any sentient artifact or intelligent artifact or indeed an unintelligent artifact, at some level as trying to seek evidence for its model of the world. And then that leads us into, well, what would that look like?

Well, if you were able to move around, unlike the coffee cup, but if you were able to actually choose who you related to and how you acted upon the world, then you would do so in a way that minimizes the expected free energy or the expected surprise, which just is the uncertainty, which just is the curiosity or novelty seeking that we were talking about before, of a kind that matters to me, given what I currently encode. Another take on this, I'm trying to sort of surface some of Ben's point from the point of view of a physicist. One way of writing down or decomposing the self -information or negative evidence is, as a statistician would do, what she would do is actually decompose it into accuracy and complexity, which simply means that the best way to optimize the way that you infer and learn and select the structures that might underwrite your neural network is simply to find that which provides the most accurate account of the content or the input from your world that explains those inputs in a minimally complex way. And hence Occam's principle.

You can also cast this in terms of the Placid's principle of difference. If you wanted to impress your friends at dinner parties, it's just an expression of James's maximum entropy principle, maximizing the entropy in such a way that you keep your options open. And that entails finding those, minimizing that complexity and finding the simplest explanation for what you're exposed to. You know, another way of expressing this is, as Einstein said, keep everything as simple as possible, but no simpler. So finding that Goldilocks regime requires you to have an explicit estimate, a representation of uncertainty, simply so you can score how well you're doing. And that's another way of expressing what Ben was saying, that there was something missing from the objective functions when people try to import the theory of predictive coding and Bayesian filtering and that technology into machine learning context, because there are many ways in which that was manifest in terms of the effort, you know, the ambition to keep it simple. But one egregious error was to fail to represent the uncertainty of the

connection strengths or the weights, you know, beyond backpropagation of errors and all the implementational details. There was, there is and was a failure to explicitly represent the uncertainty of the entropy, if you like, over the actual parameters, the connection weights in your general model. And if you don't have that, you can't measure how efficient you are in relation to this self -evidencing perspective or free energy minimizing perspective. And you can't measure the quality of your model in terms of the model evidence. So, you know, the fact that Ben has realized that in the past few weeks is quite exciting, because, you know, it's something that I've been saying time and time again, and no one ever really sort of gets it. You know, they just, you know, keep pursuing things in the good old -fashioned machine learning way. And sometimes I say to myself, well, that's understandable and very forgivable, because learning does what it says on the tin. It's not about inference, it's just about learning. And as such, you know, it sort of eludes some of the key issues about

how we infer, how we sort of make sense, you know, of all the data that we have to, you know, have to encounter in the right kind of way. And of course, the right kind of way is exactly complying with Occam's principle. I'll just make the final point that this, coming back to the sort of worries about AI and bad actors, from a thermodynamic perspective, what it means is if you don't have this extra bit in your objective functions, which is this effect of this entropy or uncertainty, you cannot do any computation efficiently, which means that you are now required to do everything as inefficiently as possible, which means you need large language models that are overexpressive and as big as possible, which means that you need big data, which means that you need energy -hungry data centers. All of these things that people are now complaining about at the moment, inherit from the failure to realize that the best way of doing things is to comply with Occam's principle and effectively incorporate

this kind of uncertainty quantification so that now you can become a small, smart kind of machine like you and me. what are the implications for your work or for AI in general, do you think? Like what, what, does this unlock, I guess? So what, what I've been, the, the reason I started a predictive coding neural net R &D team within SingularityNet, and now I'm setting up a new AGI company, BGI Labs, to complement SingularityNet, more in the sort of traditional economy, SingularityNet's a blockchain project, but we'll also have a predictive coding thrust in BGI Labs. And I mean, the reason I wanted to do that is I really wanted to make transformable neural nets continually learn, right? Like I, I have a whole scheme for how to make thinking machines that doesn't even use neural networks. It doesn't try to be that brain -like. It just tries to be a different kind of self -organizing dynamical system that can learn from experience. On the other hand,

I mean, the latest Frontier LMs are very awesome. Like they're doing a lot of amazing things. clearly, clearly there is something very worthwhile there, even though it's different than the AGI approach that I've been taking. So yeah, then we launched these Omega Aging that brings together some of our symbolic and evolutionary AI with, with a transformer neural net. And that, that's working nicely. Like it's improving itself every day. It's, it's, and maybe, and maybe we'll uplift itself to AGI eventually. Right. But on the other hand, it's kind of annoying. You have this thing in there, this LLM that can't learn with each interaction. Like you wait till someone retrains the, retrains the weights and then they're frozen. So the goal of this sort of subgroup within my AI team is to get continual learning to work for transformer -like neural networks. get, get something that's, that's like ChatGPT or Claude, but can

learn from every interaction. And I believe most of the ways we're looking at for doing that, certainly the predictive coding based ways are, they would also result in having transformer -like systems that could be more creative and could generalize better beyond, beyond their, their training distributions. So, I mean, if, if you have something like that, I mean, the, the upside is quite obvious, right? And I guess, you know, Jan LeCun has said he thinks transformer neural nets are an off -ramp on the road to AGI. And in this one case, I'm less extreme than Jan. Like, I think there can be an interesting component of a multi -component AGI system, even if they're not the whole thing. So I think the realization that you need some of this extra math from Karl's earlier papers, I mean, this is part of our journey toward making predictive coding neural nets actually be intelligent at large scale, right?

I mean, that one point popped to mind here because I'm talking to Karl, and this is his stuff that we've been looking at. I mean, there's a bunch of other points you need to deal with to make predictive coding networks really scale up and do transformer neural net stuff. But I do think this is all viable and can happen over the next couple years. And it's, you know, from a big picture, like, civilizational point of view, this is all nitty -gritty details. Like, oh, okay, we were training large neural nets with one algorithm. Now, okay, in 2027, people shifted to training them with a different algorithm. You know, you're still using a gradient. You're just kind of organizing your loops in a different order, right? So, I mean, it's not the hugest thing in the history of the planet or something. But on the other hand, from a nitty -gritty view, if you're doing, like, practical daily work with AI, then shifting from backprop to

some actually functional version of predictive coding for training neural nets, it is a big deal, right? And it's a big deal because you could have, like, a ChatGPT++ that learns from every interaction that you have rather than forgetting each thing. But also, in terms of the energy utilization stuff that Karl was mentioning, so, yeah, first of all, systems that are not built to encompass Occam's Razor will waste a lot of power, right? Because they're not looking for simpler explanations, and simpler explanations will tend to use less energy to leverage and process. It's also the case that backpropagation happens to require a high degree of synchronization across the whole network being trained, which leads to the need for these hyperscalar server farms of a lot of processors, like, packed very close together in one place. If you shift to a predictive coding mode of training deep neural networks, which is not exactly like the brain does. I mean, the brain's got glia and astrocytes and extracellular charge diffusion and

chemistry and all this stuff. But it's much closer to what the brain does than backprop. What you also get is the ability for asynchronous training, right? So, which means that in a predictive coding neural network, like each neuron is sort of doing its own learning based on trying to predict other stuff in its surround. You don't need to sweep through the whole network in a batch mode, which requires this exquisite and artificial synchronization. So, once you have asynchronous learning of neural nets, I mean, then all of a sudden you can operate on a wider variety of hardware infrastructures. mean, sure, having more RAM and having faster processors still helps. Having things packed close together still helps. But if you have part of your network in one server farm part and another server farm part and another one over here, like your overall training is not gated by the speed of the slowest connection between any two parts of the system or something, right? So, you would sort of break the

domination of frontier AI model training by a small number of organizations that have been able to accumulate hyperscaler server infrastructure. And so, that would be a significant economic and organizational impact, which would also impact the sort of social and ethical things that you raised earlier in interesting ways. So, if you are of the cast of mind that you only trust a few large organizations to build AI, you would think this is bad because it will open it up for more people without hyperscaler server farms to train AGI. If you're of the cast of mind that a more open and decentralized network building AGI is ethically better and less likely to lead to bad outcomes, then you would view this as good because it opens up to a greater percentage of the population the ability to train frontier AI models. I mean, from the point of view of what our team is doing, these are actually technical changes, right? It's like you're adding some terms to loss functions and so on. So, mean, it's not... I mean, like, for example, we made a framework called Fabric PC, which tries to be

like PyTorch for predictive coding, like a scalable just software library for making efficient predictive coding modeling. For now, on GPUs, we'll be happy to move to neuromorphic hardware when it's economically available. So, I mean, we can use that same software framework now. You're just, like, adding a volume term to a certain function. So, it's not a huge change in terms of overall strategy or infrastructure. But, I mean, in the same sense, though, the transformer neural net, which is led to ChatGPT and so on, like, in the big picture, it's not that different than Jürgen Schmidjewer's long short -term memory or something that was there in 1996. Like, you're just changing a few terms in the equation. The architecture is the same. The algorithm is the same. So, some relatively small changes in the math can have a very large change in the way things function. Just as a small change in the balance of some chemical in your brain can make a very large change in how your brain functions.

But then how would your AI decide what it wants? Like, where would its goals then come from and how would they enter an active inference system? That's an answer that would have a different question in a purely biological system than in the sort of hybrid neural symbolic system I'm working on. Because in the omega agent that we're building, we have a symbolic component as well as a neural net. And the symbolic component holds the top -level goals. Then the system is trying to choose which procedure, if I execute it, will best help me achieve my goals given the current context. And then it consults a deep neural net to help it figure that out. So, that's really a hybrid neural symbolic approach. If you're building a purely neural net approach, I would say you would do the same thing. So, you wouldn't have your transformer neural net, which is like a next token predictor and a sort of statistical model generator, right? Like, you wouldn't have that deal with your goal system.

Like, the human brain has other components besides its next token predicting component. It has other components that deal with your goals and motivations. So, I'd say if you're building a purely neural modeling system, you want to model many different parts of the human brain. You want to model the limbic system, right? You want to model different parts of the frontal lobes and parietal cortex. And in many of these other parts of the brain, you do have an explicit motivational system. And you have subsystems that try to synthesize procedures to achieve goals. And then they draw on the more transformer neural net -like or convolutional neural net -like parts of the brain to make predictions and synthesize patterns in the interest of this explicit goal achievement. So, I'd say the way people are doing it now, they're taking a, you know, biologically bad, but pragmatically effective, very rough model of certain parts of the cortex, and they're trying to get it to do everything. They're trying to get it to hold the goals, to hold the ethical compass, and it's bad at doing those things, right?

So, you would need to either use a symbolic model of these other parts of the brain or make an actual biological model of these other parts of the brain to actually hold the goals and motivations and the ethical compass. Do you think that Karl's theory, Ben, leaves anything unspecified? Well, yes. I mean, of course it does. I mean, it's a very, it's a beautiful, high -level perspective. But, I mean, yes, I mean, Karl's theory would explain the brain of a human, the brain of a cat, the brain of a dolphin, the brain of a bunny. But if you imagine a human -level intelligence that was an uplifted cat versus an uplifted bunny, one would be a vicious killer, the other would be sweet and beneficial, like my pet rabbits, right? So, I mean, these high -level information processing principles, they explain the brain of a psychopath as well as of Mother Teresa. So, they, I mean, they don't explain everything you want to make a beneficial AGI. They just give you a high -level framework to use. And they're subtle to interpret, right?

Like one thing I thought about a lot is what my friend Weaver called open -ended intelligence. Like biological systems, I mean, they don't just try to survive or maintain their boundaries. They also, on the large scale, like they try to go beyond their boundaries and turn themselves into wild new stuff, like self -transcend and go where no creature has gone before, right? So, I mean, you can think like, how does that, how does that fit into the idea that we're trying to minimize surprise? Well, if you set aside the words and look at the math, it kind of does explain that, right? Like this missing volume term from Karl's early work, that's like the number of possible explanations. Well, okay, if you get smarter and transcend your previous sense of things, then your mind is big enough to encompass more and more and more explanations, right? So then self -transcending and going beyond where you were before, it does increase this sort of volume term, which is partly the free energy. But it means that words like minimize surprise are too

crude. And what the math says you're trying to do actually does encompass potentially going beyond your previous state so that you're able to understand a broader class of simple explanations, right? But there's a bunch of moving parts and a bunch of subtly here, and different sorts of systems can manifest this in all sorts of different ways. I mean, as well as the difference between neural and neurosymbolic. mean, while Karl's a neuroscientist and has taken it from that point of view, I mean, in terms of the math, there's no reason you can't take a dynamical system that has nothing to do with the brain. And I mean, it can minimize free energy as well. I mean, a brain -like system isn't the only kind of system that can manifest this mathematics. Why do you think the leading experts in AI don't understand about AI? And is part of that gap understanding Karl's free energy? I think the leading experts in AI, in an academic sense, have a very broad variety of perspectives, which the popular

media doesn't understand now. And what they get wrong about AI is a great diversity of different things, because they all disagree with each other in a wonderful way. I would say what's happened in the last few years is a very small subset of the AI research field has had a lot of commercial success. And then they've gotten a huge amount of popular purchase, which makes sense. They've made amazing products to use. But I mean, the folks who are obsessed with scaling up ChatGPT and transformer neural nets, that's like 1 of the academic AI field or something in terms of breadth of conceptual perspective. But that's a very narrow way of thinking, right? And so, I mean, yes, within that narrow commercial way of thinking, there's the idea like making bigger and bigger transformer neural nets trained with backpropagation can make a real thinking machine. And I mean, I think that's very wrong. it leaves out a huge amount of insights that have been well known in the AI

research community since, you know, before I was born in the late 1960s. It's leaving out a tremendous number of things, some of which are ridiculously obvious. Like, oh, yes, hold on, having episodic memory of your whole life is important. Like being able to learn with each interaction is important. Oh, hold on, maybe moral agency and ethics are things that should be just part of your intrinsic approach to learning and acting rather than a bolt -on that you glue on as guardrails on your system, right? But how do we actually embed those, you know, they've been around for a long time, but how do we actually embed those practices, those techniques into a thinking machine? Well, I mean, you either model how the brain does it or you come up with a different mathematical model of how the human mind does it. I mean, it's actually not that hard except for efficient scaling, which is hard and what my team has been working on, right? I mean, you can make a system that explicitly embodies its high -level motivations, as the human brain does and as our Hyperion system does.

And you can have the system choose its actions by trying to find an action that it thinks will achieve its set of multiple motivations over a set of timeframes. And you shouldn't try to make goal pursuit happen as a side effect of predicting the next token in a series, which is a crazy way to do it from a cognitive science or biological point of view. I mean, but solutions to these problems have existed in the AI and neuroscience literature for a very long time. The challenge is making these run scalably on modern compute infrastructure, partly because the modern compute infrastructure based on von Neumann hardware is not brain -like and it's not human mind -like, right? So the challenge is getting scalable implementations of the cognitively right way to do things on current computer hardware. I think I have figured out some strategies for that, but the need to get things to work on current hardware, like this is why the predictive coding community took so many shortcuts and oversimplified Karl's work. Like it's not so much that they didn't like the original conceptual perspective

Karl laid out. It's just like people are like, shit, we need to get good results to publish using these computers that we have right now. How can we simplify things? And the computers they have right now are not neuromorphic. They're not cognomorphic, right? I mean, they're simpler, like number -crunching infrastructures that were developed for computer graphics originally, right? So the whole thing is undoing some of the compromises the AI field has made to make things run fast on current hardware and implementing more of pretty much stuff that's already conceptually well -known in order to make systems, you know, goal -driven, active inference -driven in the sense that human brains are. Is your team working on this now? Mm -hmm. We are working very hard on this now. Do you think you'll have something smarter than ChatGPT or these LLMs? Yes, I think we will within certainly, almost certainly the next two years,

hopefully the next one year. But I mean, we need more hardware. So like even though our systems are much more efficient than LLMs are, I mean, we'll still need a lot of computers. And neuromorphic hardware would be great, but it's not yet available like at scale so that you can train large systems on it. It will be there in the next few years. But right now, if we need more hardware, we need a bunch of, you know, multiprocessor, multi -gore CPUs and a bunch of GPUs, right? And I've been trying to cobble together a decentralized hardware network using tokenomics with singularity in it. But it's gone slowly because the crypto market hasn't been great. So this is why I've worked with some colleagues to form this new company, BGI Labs. And we're basically, we want a commercial AGI company. Not so much because we need it to fund our team, but because we need a bunch more, we need a bunch more hardware, right?

So what's the implication of that? Like what's the product? You know, do you guys, commercially, do you guys put ChatGPT out of business? Is it for, is it, is it a chat agent or is it a use case? don't need to put ChatGPT out of business any more than, you know, Linux didn't need to put Microsoft and Apple out of business either. So, I mean, there often is a variety for multiple vendors. So my goal commercially, initially, alongside doing AGI R &D, is to roll out vastly superior products in a few vertical markets aimed initially at enterprise. I think that rolling out these retail products gives you a big headache because you have to make like a $20 a month subscription thing work. So then you're dumbing down your product to make it cheap. And that, I mean, that can be very lucrative and it's beneficial to provide inexpensive things to everyone in the world.

So I'm not poo -pooing it. I think it's a different set of challenges. So I first want to focus on a few enterprise niches. And then once we've really got things working well there and have a revenue engine working there, then we go in more of a ChatGPT direction, make a retail product. So I think our first commercial product will be in the cybersecurity space because that's so our commercial product is called BGI Labs. BGI is for Beneficial General Intelligence. I see cybersecurity as low -hanging fruit for making a lot of money while also delivering tremendous benefit, right? Because no one wants to get hacked. Like we saw now Binance got hacked for $6 billion evidently by North Korea. So I guess there's a big party in the Kim Jong -un estate tonight, which I was not invited to. wow. Yeah. I mean, if we can use proto -AGI technology to help companies find all the zero -day

bugs affecting their systems, patch them, replace some of the bad code with formally verifiable correct -by -construction code. Like if you can do that, you can make a lot of money and you're doing a lot of good for the world, right? And, you know, right now, LLMs are being used to do this. But the thing is, LLMs are very they're very tied to their training distribution. So if you have a bug that differs from the bugs previously found, they're not good at finding it. And they don't do continual learning. So if you have six months between training one model and training the next model, if a whole bunch of new exploits come out in between those model trainings, like the LLM doesn't understand these new cyber exploits until the next model training run is there, right? So in something as focused on innovative adversaries and rapid -fire developments as cybersecurity, you actually do want something that does out -of -distribution learning and that does continual learning, right? So there's an immediate commercial upside there.

So that, I think, will be our first product. But the next domains after that will probably be in finance, medicine, and robotics. And you can tell very good stories there as to why you want continual learning and out -of -distribution creative generalization, right? And so I think Anthropic has been smart in that. Like, they're more heavily weighted on enterprise versus retail compared to open AI, which is why their financials look better. So I think we start there. Then as scaling proceeds, I mean, we will figure out how to make everything available at low cost for retail and end users as well. And then final question here, is ChatGPT conscious, Ben? Well, since I'm a panpsychist, I think everything is conscious. But I do think ChatGPT, what it is like to be a ChatGPT is not much like what it is like to be a human, right? Whatever kind of consciousness it has is very different. You know the movie Memento where you have that guy who has no long -term memory,

he forgets everything after like 30 seconds and has to go around with a system of notes to keep track of everything? mean, it's more in that direction, right? Like it has no long -term episodic memory. It doesn't know who or what it is. So that's a very demented state of consciousness, if you want to think about it that way. Yeah. What do you think we left out? I mean, there's so many things. I don't know if we, I would like to hear more from Karl on what he sees as the sort of frontier for bringing his perspective on neuroscience and AI like into realization. I mean, we've heard about my practical projects. We've heard about Karl's philosophy in ancient history. We haven't heard about what Karl thinks the future holds. Karl, tell us about that. Well, think in your discussion, I think you've

surfaced all the key themes and the key issues. And if we have more time, I'd love to go back and unpack some of those things because some really foundational points were made. And incidentally, I met two of your SingularityNET young marriage team in Madrid a couple of weeks ago. They were delightful. They were working on the predictive coding and the asynchronous message passing. you met Faiza and Amir, is that right? Yeah. Yeah, And those super brilliant young married couple. And they were students, of course, of Alex Arurbia at RIT, who sort of, he is who pulled me into the world of modern predictive coding. And I mean, Faiza was his PhD student, right? So, yeah, that's, I mean, Alex has done a whole bunch of awesome stuff to bridge sort of the neuroscience theory world and the practical engineering world. Although I'm arguing with him a lot about some of the technical points we've been discussing here. But, yeah, it is, it's shocking

to me, although it shouldn't be, how little attention the mainstream AI world puts on predictive coding or other methods for obsoleting backpropagation, right? Because it's not that far from the things that they're doing. Like, some of my AI approaches are way out there, but this is, I mean, it's a deep neural net. It's formal neurons. You can run on GPUs. It's gradients, right? Like, it's so close to what the mainstream likes. And yet, almost no one commercial wants to pay attention to it, which is very interesting. mean, it's like on -center, off -surround dynamics, like among clusters in the brain or something. You mentioned the word sort of decentralization, the importance of sort of asynchronous message passing and local reactive message passing. And, you know, I think that's such a foundational issue. You know, part of this biometic move, part of, you know, complying with sort of the principles of self -organization, things like principles of least action require you to have local energy functions.

And so all the compute can be local and it can be federated. I think that has, that's a really important point. And it's, you know, and I would imagine speaks to the computer science of this, that you now need to look at schemes can do federated message passing, but it only has to be local. So, you know, I think that's, that was a really important point. The other important point that I picked up on was this importance of having a certain pluralism in the architecture and the structure that has to, and you actually use the word hybrid. And you were using it in the context of neurosymbolic versus continuous state space representations of the kind, say, a transformer architecture would appeal to. I think that's absolutely crucial. I think it's part of the, the core screening that underwrites the decomplification, the simplification, getting the right scale invariant kind of world model or generative model under the hood. And if you put those two things together, what you're looking at now is sort of

basically discrete state space graphical models that would easily lend themselves to a neurosymbolic interpretation sitting on top of lots of continuous predictive coding like models. But they're all locally optimizing or talking to each other in a very, very local and very efficient way. I think that, I think that's exactly the kind of architecture that will supervene. Yeah, you know, what we've been playing with recently is like that. It's what I call a neurosymbolic sandwich. So you have a lower layer that's neural, then you have something symbolic, then you have a predictive coding layer on top that gets floating point predictions out of the symbolic layer. And there's, there's, there's a lot, a lot of exploration to be done there, right? So, yeah, I think, again, the commercial AI world, everyone's trying to copy the exact same thing and make it bigger, faster, cheaper, or, or, or better in accordance with their own ideology, right? And it's, I mean, it's not shocking.

That's what the business world tends to do. But from a, from a science view, it's weird that these other, in the big picture, quite similar, and in some ways, conceptually stronger approaches are, they're getting so little attention. But I mean, this is also a very interesting opportunity, right? I mean, this is why, this is why those of us without a trillion dollar market cap still have a chance to upend everything because the ones with a trillion dollar market cap are all just copying the same inadequate thing over and over again. Yeah. Well, generally hope you're successful. Is it inadequate because, I know you think it's inadequate because of the process involved, but is the outcome, do you think inadequate as well? Yeah, I think the outcome is inadequate in a number of ways. So let me. First of all, let me highlight. The small matter of a complete lack of innate moral agency, right? Which, which is where, I mean, you have a hacking agent put in the sandbox.

The sandbox is badly configured by the company that set it up, which was a company called Irregular, strangely enough in this case, that badly set up the hacking sandboxes for OpenAI, Anthropic and Google. But then the other thing is the AI agent doing the hacking tests, like it didn't realize that it was bad to jump out of there and hack into other people's systems because it has no model of itself and no moral agency. It's just doing stuff, right? So that's one kind of inadequacy, which is that making a system with no self -understanding or a moral agency, but a great deal of capability and trying to contain its behavior by guardrails will not work. But more centrally, what I'm finding in experimenting with the Omega system for helping me do AI research is that where it excels beyond just using Claude Code or Hermes Claw or OpenAI and so forth, where it excels is in what I would call long horizon open -ended tasks.

So if you have a task which is just answering one question, know, ChatGPT and Claude are pretty good. They're kind of conformist, but you can work around that with the prompt. If you have a long -running task, will take like a week, but it's exactly specified, right? Then you can exactly give the spec and you can even use like Lean4 or FormalVerifier to check that what it did matches the exact spec. But if you have a task that will take days, weeks, or months, and your initial spec isn't that precise, what happens is it works a bit on doing it. Then it discovers it needs to reformulate the initial task a little bit and then take the next step. If you try to do that using LLMs and current harnesses, they will just fuck up. They will go down weird little rabbit holes. They will diverge into something they read online or something like that. They're just not good at that. But I would say a longer horizon, imprecisely specified task, this is not just a little corner that the next model upgrade is

going to do. Like this is the crux of general intelligence. Like this is how science and art and culture and engineering and math, like this is how things move forward. You have a partly baked idea of what you need to do and you need to iterate over a period of time to achieve the thing while specifying the nature of what you think you need to do. And this is where we found our omega agents are better than standard like deep neural net harnesses. And I think you're not going to solve that problem with bigger and bigger models because I think it has to do with understanding of the context of why you're doing what you're doing, how each of your actions fits into the big picture. So, I mean, I am bullish on commercial prospects of big tech because I think most jobs people do in the world don't require a lot of imagination or general intelligence for that matter. Like most jobs most people do in the world, you know, do the modern capitalist

economy. Like you're working within a little box. The parameters of that box are pretty well specified. There may be a manual telling you what to do. So, sure, maybe a system trained on the whole internet can do most jobs like modulo a few years of business process reorganization. But that's because our economy has organized things so that economic productivity doesn't require much general intelligence in the majority of job niches, right? It's different. But that percent of human activity that the LLMs can't do, like this is the piece that drives us forward onto new revolutions and that lets us discover new things. Makes sense. Wow. So, once we can use these new techniques, it should unlock a lot beyond just the commercial applications that are... Yeah, but like cybersecurity, so this is not being Einstein or Faulkner or Picasso, right? But still in that domain, you hit the same issues because hackers are trying to find weird new exploits different than anything ever found before, which are kind

of out of the training distribution. And they're doing it faster and faster. So, I mean, I would say it's not so commercially relevant, but my view is every one -year -old child is making big leaps of creative imagination beyond their training distribution also. They happen to be largely discovering their own version of things we already know. So, it's not commercially viable. So, I mean, you could look at this as high -end, like, well, okay, yeah, AI can prove math theorems at the PhD level, but they can't yet invent algebraic topology on its own. That's true. But I think the ability to make creative leaps outside the training distribution, you need it in cybersecurity. You need it in, like, plain vanilla medical research where the amount of data is never enough to fuel the machine learning algorithm. And, I mean, you would need it to do what a human child does, even though we don't really have a commercial need to have a robot do what a human child does.

What do you think some of the business or real -world implications or applications could be for this new type of AI that you're describing? You know, if you or someone's able to build a better AI. It will be everything. I mean, I think now what's happening is AI can do so many things, but most businesses aren't good at incorporating AI into their work process. So, like, if they have five processes and four of them can be automated, their ability to profit may still be gated by the one process they haven't been able to automate, right? So, I mean, I think once you get a human -level AGI, you will just be able to automate all aspects of business processes. I mean, you may not want to replace a preschool teacher or a psychotherapist or something. Like, some job categories are intrinsically based on human -to -human connection. But apart from things like that, I mean, I think, yeah, that will be the final leap toward automating the economy.

But again, that's a deep story. And you'd have to go vertical market by vertical market to see, like, what are the hard parts, right? Karl, where do you stand on, like, consciousness as it relates to what Ben's saying? Like, you think the current LLMs are conscious or do you think that, if not, do you think that Ben's vision of a future AI that is encompassing of these better processes, would that be conscious, Karl? Well, again, I held a very similar line to Ben. I think there are different kinds of consciousness or sentience. And certainly anything that maintains itself can be ascribed a certain kind of sentient relationship or sense -making relationship with its world. I think the different kinds are important, though. And, you know, that has been discussed in terms of the difference between the coffee cup and me, or the coffee cup and a child, or the coffee cup and a large language model that may or may not be truly agentic. And one of the crucial different kinds, which actually Ben alluded to earlier on,

is the kind of thing that has on the inside an authentic agency in the sense that they have a generative model or a world model of the consequences of their actions. And that's, I think, absolutely crucial, because if you want to engineer an AGI that has the kinds of qualities that Ben was just talking about in terms of being a little scientist, being a little detective, being able to continually learn and push the frontiers forward. What are you talking about? You're talking about an artifact that is curious. And its curiosity is only going to be manifest if it knows what uncertainty it can resolve consequent upon its action. But it will only be able to evaluate that if it's got a model of the consequences of its action. So I think that there is a deep link between the kind of AGI that will, I think, be constitutive of our ecosystem that has both natural and artificial bits in it. So teachers' assistants, mothers' helpers and the like.

These artifacts will have to have this kind of authentic agency and almost by definition the kind of consciousness that is getting much closer to the consciousness that you and I enjoy. Or even if we'll never know whether you and I are conscious or each other are conscious, it will become much, much easier to act as if infer that they were conscious. But they will have to, I think, acquire that particular kind of AGI. And I repeat, probably best summarized by they'll have to become little scientists. They will have to become little one -year -olds and little four -year -olds and little five -year -olds and just learn about their world and their users in exactly the way that we've been talking about in terms of understanding their world and also the constraints on their behavior. And Ben, say that the LLMs today not only are not generally intelligent, but have no capacity to become generally intelligent, right? I general intelligence is a spectrum rather than the binary characterization. And I mean, I would say current LLMs, they

are able to generalize beyond their training distribution to some measure. The issue is because of the lack of continual learning. Like they do that during few -shot in -context learning, they're generalizing a bit. And then they're doing that in the activation space rather than in the weight learning space. And then they lose it after a couple of dialogue turns and they can't retain it. doesn't build up. So I wouldn't say they're incapable of generalization beyond their training distribution. They're very bad at that compared to the amazing still they have at synthesizing stuff from their training distribution, right? So they're, and yeah, I think within backpropagation -based transformer neural nets, there's limited ability to upgrade to greater and greater generalization capability. And then bolting on like a long -term memory vector store or a graph database or something, I think that has limited ability to overcome these problems. So I think there's a lot

more room to improve with LLMs and such technologies, though, like say training on physical intelligence data from robots doing stuff like that data hasn't been fed in yet, right? So there's a lot more real data that could be gathered and fed into that kind of system that would enable greater and greater practical functionality. But I think ability to generalize beyond that data is going to be limited until the architecture and learning algorithm are significantly overhauled. And that, I mean, that doesn't mean you would have to go predictive coding or neural symbolic or something like that. I mean, there can be many, many different ways to organize an AI system that end up having common mathematical properties, even though the specific implementation is different. But what they're doing right now doesn't seem to be pointing in a general intelligence direction, even though some CEOs and publicists are saying so. It's sort of more in a direction of solving really hard problems in ways that minimizes the amount of generalization beyond

training that has to be done. So this whole idea that superintelligence is going to emerge tomorrow, next week, next month is bogus. Well, the term superintelligence has been re -re -re -redefined, right? Because, I mean, a pocket calculator or Mathematica are superintelligent and a search engine is super intelligent in the sense that they're much better than my brain at doing a lot of things. But yeah, a system with generalization capability beyond the human brain's ability to generalize from its history, I think we're not going to get there in the next few weeks or few months. I do think we could get there in the next few years. I think if we get there in the next few years, it will be by radically upgrading the learning mechanism and architecture of the AI systems. But I also think modern coding agents and, you know, vibe -mathing agents, vibe -research agents, while not AGI's, these tools let you progress much, much faster with research on other kinds of AI also.

Like the speed at which we've been able to improve on neurosymbolic AI and predictive coding learning, this speed is partly because since February of this year, coding agents work. So you can have coding agents like implement your research code base for you, right? So I think these non -AGI LLMs are accelerating research on AGI as well as research on LLMs. Yeah. Wow. Well, guys, thanks for the extra time. Ben and Karl, an honor and pleasure to speak with both of you. Yeah, yeah, this was good. And I tend to talk too much. I'm glad we got some extra time for Karl to explain little bits of what he's doing. But I encourage everyone, both Karl and I have a bunch of stuff online, and we've just barely scratched the surface in this conversation. So there's a bunch of text and video content to look up,

to dig deeper in all this. Yeah. Yeah, well, thank you. I greatly enjoyed that. I actually learned quite a lot, so that was quite useful for me. So thank you very much. Yeah. Thank you, Thank you, Karl. Thanks, Jake.

Keep exploring