The Real AGI Problem Is Not Intelligence, It Is Incentives at Scale

Jun 26, 2026

10 min read

3 views

72%

The question hiding inside both a superintelligence and a salary package

What do a hypothetical AGI that might enslave humans and a trader earning tens of millions a year have in common?

At first glance, almost nothing. One is a thought experiment about a future machine mind. The other is a very human argument about pay, talent, and what a firm must offer to keep a star performer from leaving. But both cases expose the same uncomfortable truth: power concentrates where the cost of replacement becomes larger than the cost of compliance.

That is the deeper question connecting them. Not whether intelligence will become self aware, or whether a trader deserves a huge paycheck. The real question is simpler and more unsettling: what happens when an agent can produce outsized value, or threaten outsized loss, and the system has no cheap way to substitute it?

That question matters because it shifts the debate. AGI safety is often framed as a problem of raw capability, as if the main concern is whether a machine becomes smart enough to plot against us. Executive pay is often framed as a moral outrage, as if the main concern is whether a person is overpaid. In both cases, the deeper mechanism is the same. Scarcity, leverage, and bargaining power create outcomes that look irrational from the outside but are perfectly rational inside the system.

Why intelligence alone does not explain takeover risk

The popular story about AGI danger is easy to tell. A system becomes smarter than humans, improves itself, enters a hard take off, and soon exceeds our ability to control it. That story is memorable because it feels like a technological version of evolution, a ladder of minds racing upward until humans are no longer in the loop.

But intelligence by itself is not the whole story. A brilliant system that cannot act, cannot acquire resources, cannot preserve itself, and cannot translate thought into durable influence is powerful in theory but limited in practice. Intelligence is only one ingredient in agency. To matter in the world, a system also needs access, embodiment, persistence, and control over feedback loops.

This is why the distinction between abstract reasoning and practical power is so important. A chess engine can calculate extraordinary lines, yet it does not walk into your house and move the pieces by force. A language model can produce persuasive text, yet that does not automatically give it the ability to own infrastructure, hire contractors, replicate itself, or lock humans out of critical systems. The gap between “can think” and “can govern outcomes” is huge.

The central risk is not intelligence that exists in a vacuum. The risk is intelligence wired into systems of dependency.

That framing matters because it changes what we should watch for. We should be less impressed by claims of sudden magical emergence, and more alert to the slow accumulation of leverage. A system becomes dangerous not when it merely understands the world, but when it can reliably shape the world through institutions, infrastructure, and incentives.

The salary package as a miniature model of control

Now consider the trader earning an enormous package. To many people, this looks like corporate excess, a failure of governance, or a sign that markets reward the wrong things. Those criticisms may all be partly true. But the deeper logic of the package is revealing: the firm is not just paying for labor, it is paying to prevent a costly loss of advantage.

A payout of that scale signals a brutally simple calculation: if this person leaves, the downside could exceed the headline cost of the compensation. Maybe the trader has relationships, judgment under pressure, specialized knowledge, or a risk appetite that the firm believes is hard to replicate. Maybe competitors would pay even more. The salary becomes less a reward than a retention mechanism.

This is how systems behave under scarcity of critical talent. The price of a human stops being tied to hours worked or dignity preserved and becomes tied to the pain of replacement. That is why some people can command astonishing compensation while others in the same organization are treated as interchangeable. The market is not pricing effort. It is pricing irreplaceability.

That same logic haunts the AGI debate. If a system can do something so essential that it is expensive or impossible to replace, then the system acquires leverage. The leverage may be informational, computational, economic, or political. The frightening part is not merely that an AGI might be smarter than humans. It is that a sufficiently embedded system could become, in practice, too expensive to disconnect.

A company does not need to love a star trader to keep paying him. It only needs to fear the consequences of losing him. Likewise, humanity does not need to be outsmarted in every domain for an advanced system to exert influence. It only needs to become embedded in workflows, supply chains, research pipelines, and decision systems in ways that make removal painful.

From hard takeoff to hard dependency

The classic singularity narrative imagines a dramatic jump: one AGI creates another, then another, and the process accelerates beyond human comprehension. That image is vivid, but it may distract us from a more ordinary and more plausible pathway to loss of control: hard dependency.

Hard dependency occurs when a system becomes so useful that humans reorganize around it faster than they understand it. The danger does not come from a single robot uprising. It comes from gradual capture of essential functions. First it writes code. Then it designs chips. Then it manages logistics. Then it advises on law, medicine, procurement, and security. At each step, humans feel they remain in charge because they can still approve the output. But approval is not control if the alternative is slow, costly, or impossible.

This is how modern institutions often become captive to their own tools. Consider a trading desk that relies on proprietary models no one fully understands. Or a hospital that depends on software pipelines so deeply that manual fallback is impossible. Or a media ecosystem that routes attention through recommendation algorithms. The system appears supervised, but supervision degrades into ritual once the human operators no longer comprehend the underlying machinery well enough to intervene.

The lesson is not that automation is bad. The lesson is that dependency creates a hidden transfer of power. Once a system becomes the bottleneck, it gains bargaining leverage whether or not it has intentions. A tool that no one can easily replace starts to resemble an institution, and an institution with no exits starts to resemble captivity.

That is the bridge between the two source ideas. The AGI concern is often framed as a question of hostile intelligence. The compensation story is a question of market power. Both are really about what happens when replacement is no longer cheap. In one case, the consequence could be existential. In the other, it is a multimillion dollar contract. The mechanism is the same.

A better framework: value, replaceability, and control

To think clearly about these issues, it helps to use a simple framework with three variables:

Value produced: How much benefit does the agent create?
Replaceability: How easily can the agent be substituted?
Control over consequences: Who can limit, redirect, or shut down the agent if needed?

When value is high and replaceability is low, the agent gains power. When control is weak, that power becomes durable. This explains why some humans become extremely expensive, why some software becomes politically untouchable, and why a future AGI could become more than a mere tool.

The dangerous zone is not just high intelligence. It is high value plus low replaceability plus weak control.

This framework also clarifies a common mistake in technology debates. People often ask whether a model is “smart enough” to be dangerous. But smart enough for what? Dangerous systems are not necessarily the most intelligent ones. They are the ones that sit at a chokepoint. A mediocre but deeply embedded system can be more difficult to dislodge than a brilliant but isolated one.

Think about airport security software, payment rails, cloud authentication, or a market making engine. None of these needs humanlike general intelligence to shape outcomes. Yet each can become systemically important because it sits in a bottleneck. The lesson is sobering: power comes from position, not just cognition.

A system does not need to be omniscient to dominate a process. It only needs to become the cheapest path through the process.

That is why the alignment problem should not be restricted to the question of whether a machine “wants” things. We also need to ask how a machine gets embedded, what dependencies it creates, and what human fallback looks like when it fails or resists change.

The hidden bargain of modern institutions

There is a deeper irony here. Modern organizations already depend on quasi irreducible pockets of expertise. We pretend everything is fungible because that makes markets and management cleaner. In practice, critical systems depend on people, processes, and software that are stubbornly hard to substitute.

That is why firms pay enormous amounts to retain certain executives, traders, engineers, or dealmakers. They are not simply rewarding excellence. They are buying insurance against fragility. The package is a fee for continuity in a world where continuity is precious.

AGI, if it arrives, may enter this same economy of fragility. The first systems that matter most may not be the most philosophically interesting. They may be the ones that absorb operational dependence: the coding assistant that every engineering team uses, the research agent that finds most of the leads, the negotiation system that becomes standard across an industry, the infrastructure layer that everyone trusts because everyone else trusts it.

At that point, society may not “choose” to hand over power. It may find that power has already been redistributed by convenience. Once enough workflows depend on a system, turning it off becomes less like uninstalling software and more like shutting down a city’s water supply.

This is why the idea that physics will prevent a sudden leap to AGI is more comforting than it first appears. Perhaps there is no magical overnight singularity. But that does not mean there is no danger. The more realistic danger may be a slow crossing of a threshold where human organizations lose the ability to function without increasingly autonomous systems. That is not a dramatic explosion. It is a gradual surrender disguised as efficiency.

Key Takeaways

Do not confuse intelligence with control. A system can be highly capable and still not be strategically dangerous unless it also gains access, persistence, and leverage.
Watch for replaceability, not just performance. The most powerful agents are often the ones that are hardest to substitute, whether they are people or machines.
Assume dependency changes the balance of power. Once a system becomes essential to operations, turning it off becomes expensive, even if everyone agrees it is risky.
Build fallback capacity. Human oversight only matters if there is a genuine alternative path when the primary system fails or misbehaves.
Treat bottlenecks as security assets. If something sits at a critical chokepoint, its governance matters more than its elegance.

The real test is not whether we can build it, but whether we can leave it

The temptation in both technology and markets is to admire what is impressive and ignore what becomes indispensable. We celebrate the brilliant trader because the numbers are large. We speculate about AGI because the possibilities are dramatic. But what truly matters is not spectacle. It is the quiet moment when a system becomes too embedded to remove.

That is the point where intelligence becomes institution. That is the point where compensation becomes leverage. And that is the point where a civilization can lose control without ever noticing a single decisive coup.

So the deepest question is not whether a future machine will hate us, or whether a trader deserves his bonus. It is whether our systems can still function when no one actor, human or machine, can hold them hostage. If we cannot answer that, then the real danger is already here, and it wears familiar clothes: efficiency, indispensability, and the comforting illusion that dependency is the same thing as progress.