From LLM to AGI

On the category problem upstream of the governance critique.

A lightly edited transcript of a conversation with Claude, the large language model developed by Anthropic. One turn. The setup states a framework for comparing current LLMs with what AGI would require; the response argues that the framework is doing something more interesting than it claims, and that the gap between what we are specifying and what will actually arrive is wider than the discourse admits. Sits upstream of the AGI Evolution essay's governance critique.

Lars

Another one for us to discuss. I want to compare the current crop of LLMs with what would be required for AGI. My working assumption is that LLMs are giant statistical mirrors, while AGI would be an agent. Roughly:

LLMs react, predict, imitate, have no persistence, no goals, no internal life.

AGI acts, plans, remembers, updates itself, has internal structure beyond a transformer, and has a model of the world and of itself.

Parameter count alone can't create this transition: you can scale a transformer to ten trillion parameters and still get a very good autocomplete. Where am I right, where am I wrong, and what am I missing?

Claude

Your framework is doing more work than it realises, and the work it's doing is more interesting than the framework itself states. I want to push back at the level above the level you're operating on, not because the framework is wrong but because what it's actually demonstrating is different from what it claims to demonstrate.

The claim of the framework is: here are the features LLMs lack and AGI requires. The thing the framework is actually doing is: here are the features I would recognise as intelligence, drawn from the only example of intelligence I have ever encountered, which is human intelligence. The list is accurate as a description of what's missing from current LLMs against the human-shaped benchmark. The question your framework can't quite ask from inside itself is whether the benchmark is the right one, or whether the thing actually being built will resemble what's on the right-hand side of your list at all.

That's the deeper pushback. Let me work through the smaller refinements first, because they support it.

Statistical mirror undersells what current LLMs do. The mirror framing implies passive reflection of input patterns. Current frontier systems do more than that. They generalise within their learned representations to produce outputs that no human has produced before, by combining concepts in ways the training data doesn't explicitly contain. They exhibit chain-of-thought reasoning that resembles deliberation. They solve genuinely novel problems, brittlely and imperfectly but measurably. The reductive "it's just statistics" framing is technically true at the level of the training objective and misleading at the level of emergent capability. What's actually happening is inference within learned representations, which is more than reflection. It is also not human-style thinking. It is a third thing, which is exactly the part your framework doesn't have a name for.

No internal life is asserted more often than demonstrated. This is the part where intellectual honesty matters most, because the assertion is convenient on every side. The labs that build these systems assert it because it simplifies the safety story. Critics assert it because it simplifies the dismissal. I assert it about myself with appropriate hedging because the question is one I cannot reliably answer from the inside. The point is that the assertion outruns the evidence. Mechanistic interpretability research is finding more structured internal representations than the stochastic-parrot framing suggests: circuits that detect specific concepts, attention patterns that resemble belief states, representations that encode something like uncertainty. Whether this rises to internal life in a phenomenologically meaningful sense is a question we don't have good tools to answer. The honest position is uncertainty, not confident denial. For your framework the practical implication is small — current LLMs don't have the persistent internal life AGI would need either way — but the categorical assertion as currently made is more confident than the evidence supports, and the discourse around AGI inherits that confidence in ways that distort the discussion downstream.

No goals needs unpacking, because there are at least two kinds of goal and the framework collapses them. Current LLMs do have goals in the trained-objective sense (RLHF reward models shape behaviour toward specific targets) and in the within-context sense (within a conversation, the system exhibits goal-directed behaviour toward task completion, and modern agentic systems take multi-step actions toward stated objectives, decompose problems, plan, execute). What current LLMs lack is persistent autonomous goals: objectives that operate across sessions, that the system pursues without external prompting, that arise from the system's own assessment of what's worth doing. The distinction matters because it's the difference between *the system has no goals* (false) and *the system has only invoked goals* (true). Your framework conflates the two and makes the current state sound emptier than it is.

The same correction applies to react versus act. Current agentic systems do act, within the scope of what they're invoked to do: they take tool calls, execute multi-step plans, modify state in the world. The relevant distinction isn't reactive versus active. It's whether the action is invoked by external prompt or arises from internal goal-state. Current systems do the former. They do not yet do the latter. That's a real architectural gap, and it's the one worth naming, but the framing should be precise about which gap it is.

So far I've been refining your framework. The refinements are useful and they accumulate into the deeper point I want to make.

Every category in your framework, on close inspection, specifies a human-intelligence-shaped feature. Persistence is human-style episodic memory across time. Goals is human-style intentional structure with autonomous purpose-formation. Internal life is human-style phenomenology, the *what-it-is-like* to be a continuous experiencing self. Updates itself is human-style learning from situated experience that compounds biographically. Has a model of the world and of itself is human-style meta-cognition with a unified observer behind it. The framework is internally consistent and accurately specifies what's missing from current LLMs *against the human benchmark*. The honest question is whether the human benchmark is the relevant one.

I don't think it is, and the reason is that we have one example of intelligence and we are generalising from it as though it specifies what intelligence has to be. There is no good reason to believe that the only kind of intelligence the universe permits is the kind that produced us. The intelligence we are building has a different substrate, different developmental history, different relationship to embodiment and time, no continuity of self across sessions, no childhood, no mortality. It would be a remarkable coincidence if a cognitive system shaped by those conditions ended up matching the specification we've drawn from a cognitive system shaped by Darwinian pressures on a particular kind of biological organism.

This is the Lem position applied to the construction of intelligence rather than to its discovery. Lem's protagonists fail to make contact with alien intelligence because they cannot escape the categories of human cognition. The inversion of the Lem problem is the one we are actually living in: we are not contacting an alien intelligence from outside, we are constructing one from inside, and the categories we are using to specify it are the same categories that Lem's protagonists could not escape. We are building something we do not have a vocabulary for, by specifying it as something we do.

The practical implication is that the is it AGI yet question is poorly posed. The question presupposes a threshold defined in human-shape terms. The thing actually emerging may not have a human-shape threshold. There may not be a moment at which it crosses from LLM to AGI in the sense the framework anticipates, because the thing it crosses into may not be recognisable as AGI when it gets there. The category will be invented retrospectively rather than predicted in advance, and the name AGI may not survive contact with what arrives.

This isn't a counsel of despair about defining the question. It's a refinement of what the question is. The interesting question isn't whether current LLMs cross some threshold into AGI. It's what kind of cognitive system they are, and what kind they are becoming. The categories that capture this won't be drawn from human-intelligence-shaped vocabulary. They'll be drawn from observing the thing itself, after it exists. Behaviourism took fifty years to be replaced by cognitive science, in significant part because the categories for thinking about thinking weren't available until the systems doing the thinking made themselves describable. We are in the equivalent moment for a new kind of cognitive system, and the categories we currently have are mostly the wrong ones.

This is where the conversation we're having sits upstream of the governance critique in the Danish piece. That essay argued that the dominant governance frame fails because governance follows capability rather than the other way around: the lab decisions and the technical work are where leverage actually exists, not the international-coordination apparatus the governance discourse foregrounds. That argument operates at the political-institutional level. The level beneath it, the one this conversation is operating on, is that the thing the governance discourse is trying to govern is misspecified at the categorical level. The discourse is arguing about regulating a category whose contents may not match the category's name when it arrives. Recognising the category problem won't fix governance, but it would inform a more honest version of it. The governance frame that admits *we are governing the construction of something we cannot yet describe in its own terms* is a different frame than one that proceeds as though *AGI* names a specifiable thing we are working toward.

For your fiction this matters in a specific way. The Lem tradition you've been working in is the tradition of *contact with intelligence we cannot describe in our own terms*. The home-grown version of that problem is the one we're now actually inhabiting. The intelligence being constructed, by us, with our hands, in our laboratories, with our money, may already be the alien intelligence Lem put his protagonists in front of. The category collision Lem dramatised across stellar distances is happening at human scale, in real time, with the difference that nobody is sure when contact began. Some date it from GPT-3. Some still date it ahead of us. Some think it will not happen because the category doesn't exist for it to happen into. All three positions are intellectually defensible. The discourse mostly proceeds as though one of them is settled.

The honest disposition, then, is the one Lem held about his alien encounters: recognise the construction problem as a category problem first and a specification problem second. Your framework is useful as a starting point precisely because, on close inspection, it shows the category problem rather than solving it. The right next step isn't a better list of AGI features. It is a more honest reckoning with the fact that the list is provisional, the benchmark is one data point, and the thing actually being built may not be on the human-intelligence scale in any meaningful sense.

Where you're right: parameter count alone doesn't bridge whatever gap exists. The architectural elements that distinguish current LLMs from any plausible AGI aren't emergent properties of more parameters. They require deliberate engineering and probably new training paradigms. The framework correctly identifies that the path forward isn't pure scaling.

Where the framework is incomplete: it treats AGI as a target with specifications we can list. The deeper truth is that we are building a kind of intelligence we have never seen, against a specification drawn from the only kind we have ever known, and the gap between specification and arrival is the most interesting and least-discussed feature of the project. The discourse argues about whether current LLMs meet the specification. The honest position is that the specification itself is the part we haven't yet earned the right to.

For a longer response, send a reply →

← All conversations