The transition to AGI

Here's the single coherent position that captures our initial position:

Over the next forty years, the real story of AGI won't be the technology itself but the choices we make about how it is used: whether we preserve human dignity by refusing to automate the most consequential decisions; whether we distribute the enormous economic gains broadly rather than letting them concentrate in a few hands; whether we prevent states and corporations from using AGI as an instrument of control; and whether we maintain a shared, trustworthy information environment that keeps society coherent. If we fail to set these boundaries early, AGI will amplify existing inequalities and power structures; if we succeed, it can become a stabilising force that enhances human capability rather than replacing it.

Against the governance frame

The question we are discussing is the dominant frame in serious discussion of AGI futures. It's articulate, sensibly observed within its own terms, and wrong in a specific way that will matter for the next forty years. The wrongness isn't in the details. Most of the details are correctly seen. The wrongness is in the underlying assumption: that governance is the variable that determines what AGI becomes. The governance frame treats AGI as something humanity sits down and decides about, picks among trajectories, locks in via institutional design. This isn't how AGI has behaved in the four years since the technology became macro-relevant, and there's no historical analogue suggesting it will start behaving this way.

The opposing view, stated plainly: governance follows capability, not the other way around. The constraints the piece proposes will be proposed, partially adopted in cosmetic form, and will not bind. The trajectory isn't a choice from a menu of three. It's the resultant of capability development, corporate incentive, and geopolitical competition, none of which respond to governance preference at the relevant timescale. Forty years out, the realistic projection is closer to the piece's trajectory 1 — acceleration without guardrails — with cosmetic elements of trajectory 2 in some jurisdictions, and outright trajectory 3 between blocs. The piece's framing is comfort literature for a class of analysts who need governance to be the locus of agency, because governance is what they have to offer.

This is a hard claim. Let's develop it against the piece's own structure.

Start with the empirical evidence in front of us, because it's stronger than any extrapolation. We've had four years of accelerated AI progress with global discussion of governance running in parallel. The result, measured: no enforceable international agreement on frontier model development. US-China decoupling accelerating, not converging. The EU AI Act producing compliance friction without altering capability development — and producing it in such a way that European frontier capability has fallen further behind, not caught up. The major American labs continuing to release on commercial schedules with safety practices attached as documentation. China developing parallel capabilities at comparable pace, by some measures faster. Military applications proliferating in three active conflicts at the moment of writing. Autonomous targeting systems already deployed.

Every prediction the governance frame would have made about this period — that international coordination would emerge under sufficient pressure, that hard lines would hold on autonomous weapons, that information ecosystems would stabilise as the harms became visible, that anti-concentration norms would develop — has failed. Not partially. Wholly. The governance frame has been tested at small scale and the results are in. The piece extrapolates forty years forward from the assumption that the frame works, without engaging the fact that it hasn't worked at four. This is the central tell. Any frame that requires forty years of behaviour we haven't seen one year of needs a stronger argument than "we'd better lock it in."

The historical analogue the piece implicitly relies on is arms control. The analogue is wrong at the level of physics. Nuclear weapons require uranium enrichment infrastructure that can be photographed from satellites and accounted for at the kilogram level. The relevant assets are physical, geographically fixed, and visible. AGI-relevant assets are code and compute, both of which can be hidden, replicated, exported invisibly, and replaced. The chip-export controls the US has placed on China are the closest analogue to nuclear verification, and they've already been substantially defeated through grey-market routing, domestic Chinese fabrication progress, and algorithmic efficiency gains that reduce compute requirements. The IAEA can in principle verify that Iran is or isn't enriching uranium. No comparable body could verify that a state isn't training a frontier model. The verification gap isn't a detail. It's the reason the arms-control template doesn't transfer.

Now to the moral constraints the piece proposes. The "non-delegable decisions" line is sentimental and historically uninformed. Every previous version of this line has yielded the moment the automation was good enough to make insisting on human discretion politically expensive.

Judicial sentencing has been substantially automated in US state systems by algorithmic risk-assessment tools — COMPAS and its successors — with weak human review and almost no meaningful right of contestation. This happened with technology a generation behind current capability, against the explicit objections of legal scholars and civil-rights organisations, because the alternative was hiring judges. Medical diagnosis has yielded in radiology, pathology, dermatology, ophthalmology, with the human role retreating to formal sign-off on automated reads. Welfare eligibility was the first domain to fall, in the 1990s, before serious machine learning existed; the Australian "Robodebt" scandal and the Dutch childcare-benefits scandal are recent enough to be present-tense. Battlefield decisions are yielding now, in Ukraine and Gaza, to autonomous targeting systems whose human-in-the-loop component is increasingly nominal.

The pattern is consistent. A decision is non-delegable until the automation is 80-90% as good, at which point the cost of insisting on human discretion — in money, in throughput, in political legitimacy when the human is visibly worse than the machine — becomes untenable. The non-delegation line then moves to a smaller class of decisions, and the cycle repeats. There's no reason to think AGI-grade systems will trigger a different response, and good reason to think the response will be faster, because the capability gap will be larger and the cost gap more obvious. By 2066, almost every decision the piece flags as non-delegable will be substantially delegated, with human oversight retained as a legal fiction that exists to assign liability when the system fails.

The pluralism argument fails for a different reason: incompatibility with the technology's economics. The piece calls for value diversity in AGI ecosystems, multiple model lineages embedding different cultural trade-offs. The underlying economics produce 2-3 frontier model providers globally and will continue to. Training a frontier model now costs hundreds of millions of dollars in compute alone; in ten years it'll cost tens of billions; in twenty years it may cost a measurable fraction of a national GDP. Pluralism at the application layer is real and will persist. Pluralism at the foundational layer isn't real and won't develop. By 2066 we plausibly have three serious frontier model lineages: one American, one Chinese, one consortium-of-the-rest, possibly Indian-led, possibly European-led, possibly a hybrid. The "value diversity" question reduces to which of these three you're governed by. This isn't pluralism. It's tripolar capture with consumer choice at the surface.

The economic prescription is the part of the piece that most badly mistakes wishes for analysis. Universal basic income, structural separation of model providers from application layers, public ownership of core models, robust redistribution from capital to displaced labour — these are coherent policies. They require state capacity, fiscal base, political coalition, and international coordination to prevent capital flight. AGI hollows all four, in that order, on a faster timescale than the political response can build.

The fiscal point first. The historical tax base of every developed state is labour income, value-added on goods produced by labour, and property whose value is in part a function of labour-productive geography. AGI compresses all three. Labour income falls as a share of value created. Value-added shifts to capital, data, and compute: all mobile, all easier to hide than payroll. Property value in non-productive geographies collapses; productive geographies concentrate further. The piece notes this in passing and moves on. It can't be moved on from. The tax base for the redistribution the piece prescribes is the same tax base AGI is destroying. There's no version of this where a state both loses fiscal capacity and expands welfare provision simultaneously. The democracies that try will face capital flight, then bond-market discipline, then forced retrenchment. The Nordic countries are first in line for this, paradoxically, because they have the most to lose: the highest existing welfare provision built on the most mobile capital and the smallest domestic markets to retaliate with.

The political-coalition point second. Mass redistribution from capital to labour has historically required either a credible socialist movement, a recent war, or a depression. The AGI transition produces neither war nor depression in the form that historically generated those coalitions. It produces gradual immiseration of the median worker against a background of rising aggregate output, which is the worst possible political combination, because the immiserated lack the resources to organise and the comfortable lack the urgency to act. The political coalition for taxing capital heavily enough to fund mass redistribution would have to form among the people losing economic relevance. Declining classes don't win political fights against ascendant ones. They produce backlash politics: nationalism, scapegoating, occasional populist eruptions that don't actually redistribute. The honest projection is that the welfare state shrinks as AGI hollows its base, with periodic political crises that fail to reverse the trend.

The anti-concentration measures fail for a third reason. Foundation models have economies of scale that make antitrust intervention either ineffective or actively harmful. Breaking up a frontier provider hands the lead to whichever provider isn't being broken up — historically, the Chinese one. The EU has been demonstrating this for five years already. The Digital Markets Act and the AI Act together have produced a regulatory environment in which European AI capability is structurally uncompetitive without altering the underlying global concentration. The result over forty years is European dependency on American or Chinese systems for everything that matters, mediated by a regulatory apparatus that produces compliance theatre rather than capability. The piece treats the EU approach as a template; it's a cautionary tale.

The military and intelligence applications won't be constrained, and the piece's prescription is already a dead letter. Hard bans on fully autonomous lethal weapons have been proposed at the UN Convention on Certain Conventional Weapons since 2014. The result has been a decade of consensus-blocking by the states with active autonomous weapons programs, including the US, Russia, Israel, the UK, and increasingly Turkey and Iran. Loitering munitions with autonomous target selection are deployed now, in current conflicts, with human-in-the-loop provisions that are nominal at best. The Lavender system used in Gaza is documented to have operated with effective human review measured in seconds per target. This is the baseline, not the projection.

AGI-grade systems applied to military decision-making will be developed by every state with the capability and used by every state at war. The question isn't whether autonomous lethal systems will be normalised. They already are. The question is whether the normalisation will be open or covert, and the answer is that it'll be openly opposed and covertly accepted, in the way chemical weapons are openly banned and quietly used. By 2066 the international norm will be that fully autonomous lethal force is unacceptable in principle and routine in practice, with periodic scandals when the practice becomes too visible to deny.

Population-scale surveillance and persuasion are already largely unconstrained. The piece calls for transparency obligations for AI use in elections; the 2024 election cycle globally featured significant AI-generated content with effectively no enforcement of the transparency rules that existed on paper. The 2028 and 2032 cycles will be worse. By 2066 the distinction between organic and AI-generated political communication will be unrecoverable, and democracies will have either adapted to operating in that environment or degraded to the point of being democratic in form only. The information ecosystem doesn't recover. The piece's prescription — provenance, watermarking, public-interest information spaces — has been proposed for a decade and the problem has gotten worse, not better, every year. Watermarks are removed by post-processing. Provenance systems are defeated within months of release. Public broadcasters are losing audience to algorithmic platforms in every democracy that has them. Prescribing what failed in 2018 to solve a problem that's now an order of magnitude larger isn't analysis. It's nostalgia.

The three-trajectories framing is the deepest error in our initial analysis, because it presents the trajectory as a choice. The trajectory isn't chosen. It's the equilibrium produced by the underlying dynamics. Trajectory 1 — acceleration without guardrails — is the default because it requires no coordination. Trajectory 2 — constrained negotiated AGI — requires the highest level of international coordination in human history at the exact moment international coordination is degrading on every measurable axis. Trajectory 3 — fragmentation — is what you get when trajectory 2 fails and trajectory 1 is constrained by geopolitical rivalry rather than by governance. The actual question isn't which trajectory we choose. It's whether the failure of 2 produces 1 or 3, and the answer depends on the geopolitical environment of the 2030s, which currently looks adversarial enough to produce 3 in the medium term and 1 in the long term as the rivalry settles into a stable balance.

Asking "what would we want to lock in now to bias toward 2" assumes the lock-in mechanism exists. It doesn't. Any constraint locked in now is locked in by states whose incentives in 2040 will be different and who will unlock it the moment unlocking serves their interest. The chemical weapons convention has held because chemical weapons aren't strategically decisive. The nuclear non-proliferation treaty has held imperfectly because nuclear weapons are verifiable. AGI is strategically decisive and unverifiable. There's no historical precedent for binding constraints on a technology with both properties. Asking for one is asking for a thing that hasn't existed.

If governance doesn't bind at the macro level, the question becomes where leverage actually exists. The piece looks past this because the answer isn't governance-shaped.

Leverage exists at the technical level: in alignment research, in capability evaluation methodology, in interpretability work, in the design choices made by the engineering teams of the three or four labs that will be at the frontier. These choices are happening now, are happening fast, and are happening largely outside the governance frame the piece foregrounds. The people who actually shape AGI by 2066 are the technical staff of three or four firms and the security services of three or four states. Everyone else is reacting. This is not a normative claim about who should shape AGI. It's a positive claim about who will. The governance frame's failure to engage seriously with this is the strongest evidence that the frame isn't describing the territory.

Leverage also exists at the institutional level: in which firms build what, with what incentive structure, with what internal culture, and with what tolerance for slowing capability development in favour of safety work. This is unfashionable to say in policy circles because it relocates agency from elected institutions to private firms. It's true anyway. Anthropic, OpenAI, Google DeepMind, and a small number of Chinese labs have more influence on what AGI becomes than every regulator combined, and will continue to. The honest version of the governance question is what kind of internal cultures, incentive structures, and competitive pressures these firms operate under, and whether the marginal effort there is more or less productive than the marginal effort in international coordination. My read: more productive, by a large margin, and the resources currently flowing into the governance frame would do more good redirected to lab-internal work.

Leverage exists, third, in capability evaluation and incident reporting infrastructure: the technical and institutional machinery that lets us see what frontier systems can actually do and what they actually do once deployed. This is the part of the governance frame that does work, because it's continuous with the engineering rather than imposed from outside. The piece glosses it. It's the part of the piece worth keeping.

The realistic forty-year projection, then, against the piece's three trajectories.

We get trajectory 1 with national variations. The United States accelerates, with weak federal coordination, strong private capability, periodic regulatory eruptions in response to specific scandals, and structural dependence on a handful of firms that increasingly function as quasi-state actors. The state and the labs become harder to distinguish. By 2050 the major American labs have de facto security clearances and de facto national-security missions; by 2060 the line between Anthropic and the NSA, between Google DeepMind and DARPA, is mostly bureaucratic.

China accelerates differently. The state-lab distinction is collapsed from the start. Capability development is integrated with industrial policy, military modernisation, and population management. The result by 2050 is a capability stack that is in some domains ahead, in others behind, and in all domains aligned with state objectives in a way American capability isn't. Pluralism within the Chinese system is nil; coordination across the Chinese system is high; the trade-off is favourable for state purposes and unfavourable for individual ones.

The European Union retains regulatory ambition and continues to lose capability. By 2050 European AGI consumption is overwhelmingly mediated through American or Chinese systems compliance-wrapped for the European market. The regulatory apparatus persists and grows, employing many people in compliance work for technology developed elsewhere. The cultural costs of this dependency are large and the political response is unstable: periodic eruptions of techno-nationalism that fail to produce competitive capability.

The Nordics — your country and mine — face the hardest version of the transition because they have the most to lose. The welfare states that distinguish them from the Anglosphere were built on assumptions about labour, capital mobility, and fiscal base that AGI invalidates. By 2050 either they've found a way to tax capital, data, and compute at rates that would currently seem confiscatory, or they've begun the long retrenchment. I think they retrench, because the tax-coordination problem isn't solvable at the EU level let alone the Nordic Council level, and unilateral confiscation produces flight. The Danish welfare state in 2066 is recognisably the descendant of today's, smaller in scope, less universal, more conditional, struggling to maintain the legitimacy that made it work.

The Global South — the actual majority of the human population, missing from the piece almost entirely — receives AGI capability primarily as a consumer of American and Chinese systems, with limited ability to shape what's offered. Some countries — India, Indonesia, Nigeria, Brazil — develop indigenous capability at the application layer and at the second tier of foundation models. None develops frontier capability. The geopolitics of AGI becomes a Cold War with a non-aligned movement, except the non-aligned have less leverage than they had in the original because both sides can offer essential infrastructure rather than just aid.

Trajectory 2 — the negotiated AGI the piece hopes for — appears in fragments. Some bilateral coordination on extreme risks. Some industry-level standards on incident reporting. Some sectoral regulation in finance and healthcare that actually binds. These are real and useful and don't add up to the trajectory the piece describes. They're the parts of trajectory 2 that survive contact with trajectories 1 and 3.

The thing the piece's frame can't see is that the question isn't whether AGI is governed. It's who governs whom in a world where governance is increasingly mediated by systems that can outthink the governors. The interesting boundary by 2066 isn't between humans and AGI. It's between the humans who direct AGI and the humans directed by it. That boundary will be drawn by capability, capital, and proximity to the labs — not by international agreement, not by democratic deliberation, not by checklists of dignity-distribution-democracy-diversity-durability, however well-intentioned the checklist.

The honest closing question isn't which boundary you'd fight for. It's whether you'd rather be on the directing or the directed side, and what you have to do in the next twenty years to position yourself or your children for the former. The piece can't ask that question because asking it concedes the frame.

I'd rather we ask it.

For a longer response, send a reply →

← All essays