The Real Alignment Problem Is Not Artificial Intelligence. It Is Institutional Incentives.

What if the most dangerous intelligence in the world is not a machine that escapes its laboratory, but an institution that learns how to escape accountability?

That question sounds less dramatic than a superintelligent system deciding whether humans are useful. In practice, however, the mechanism may be surprisingly similar. An entity accumulates resources, acquires specialized expertise, exploits gaps between jurisdictions, and persuades its overseers that its continued freedom serves the public interest. Eventually, the institution becomes too valuable, too mobile, or too complicated to govern in ordinary ways.

This is the overlooked connection between the pursuit of artificial general intelligence and the political economy of international sports organizations. One concerns systems that may become vastly more capable than their creators. The other concerns organizations that already possess wealth, influence, and the ability to negotiate special treatment from states. Both reveal the same underlying problem:

Power does not become safe merely because it is useful, prestigious, or created for a legitimate purpose. It becomes safe when the institutions around it can still impose meaningful limits.

The central question, then, is not simply whether an intelligent system will be aligned with human values. It is whether human governance can remain aligned with reality when confronted by concentrated capability.

Capability Is Not the Same as Control

Discussions of AGI often leap from technical progress to a dramatic endpoint. A system becomes superintelligent, improves its own successor, and triggers a rapid feedback loop sometimes described as a hard takeoff. In this picture, the danger comes from a sudden discontinuity: yesterday, humans are in charge; tomorrow, they are not.

There are good reasons to question the idea of an instantaneous leap. Intelligence is not a single dial that can be turned upward without constraint. A system needs hardware, energy, data, laboratories, manufacturing capacity, networks, and opportunities to act in the physical world. Physics imposes friction. So do supply chains, institutional permissions, engineering bottlenecks, and the limits of experimentation.

But rejecting a cinematic hard takeoff does not make the problem disappear. It changes the form of the problem. A gradual increase in capability may be more politically dangerous precisely because it gives institutions time to normalize each step.

Consider an organization that begins with a modest exemption. It is told that the exemption will attract investment, preserve jobs, or bring prestigious activity into the country. Then it receives a special tax treatment. Its employees receive unusual protections. Its commercial operations are described as incidental to a public mission. Each decision may appear defensible in isolation. Yet the cumulative effect is to create an actor with substantial resources and diminished obligations.

This is not a machine becoming autonomous in one instant. It is governance losing traction one concession at a time.

The same pattern could emerge with advanced AI. A system may not need to seize control of the world to become difficult to govern. It may first become indispensable to financial markets, defense planning, scientific research, public administration, or infrastructure management. Once many critical systems depend on it, shutting it down becomes costly. Once its internal processes become difficult to explain, challenging it becomes politically risky. Once its operators compete internationally, restraint begins to look like unilateral disarmament.

The result is a softer version of escape: not physical escape from a lab, but institutional escape from accountability.

The Jurisdictional Arbitrage Machine

A useful way to understand concentrated power is through the idea of arbitrage. Arbitrage is often associated with finance, but the broader principle is simple: an actor profits by moving activity toward the rules that are most favorable to it.

A multinational company can shift profits. A wealthy individual can change residence. An international federation can choose its headquarters based partly on tax and legal conditions. An advanced AI organization could do something similar with compute, data, talent, corporate registration, and regulatory exposure.

The crucial point is that no single jurisdiction needs to be corrupt or incompetent for this system to produce weak oversight. Each government may be responding rationally to local incentives. One wants investment. Another wants strategic advantage. A third wants high skilled employment. A fourth fears that strict controls will simply push development elsewhere.

Collectively, however, these rational choices can create a race to the bottom.

Imagine five countries considering a powerful AI laboratory. Country A proposes strict safety testing. Country B offers faster approvals. Country C provides subsidized energy. Country D grants legal protections for employees and contractors. Country E promises a favorable tax regime. The laboratory can then present each concession as necessary to remain competitive. Every government sees the others as the constraint on its own ambition.

This is the same structural weakness that appears when governments compete to attract wealthy international bodies by granting them privileges that ordinary organizations do not receive. The question is not whether the organization has a socially valuable mission. The question is whether the organization can convert that mission into bargaining power strong enough to weaken the rules meant to contain it.

When powerful actors can choose their regulators, regulation becomes a menu of services rather than a boundary.

This matters enormously for AGI. A system capable of designing software, discovering scientific knowledge, optimizing persuasion, or coordinating large operations could become a highly portable source of advantage. Its developers would have incentives to move toward the most permissive environment, while governments would have incentives to accommodate them.

The race would not necessarily be framed as a race to abandon safety. It would be framed as a race to avoid losing talent, investment, national prestige, or strategic capability. That is how institutional safeguards are usually weakened: not through an explicit declaration that safety is unimportant, but through a series of claims that safety must be made compatible with competitiveness.

Why Mission Statements Are Weak Protection

International sports federations are formally associated with governing and promoting sport. Their public purpose can be genuine. Yet a public purpose does not automatically make an organization publicly accountable.

This distinction is easy to miss because mission and structure are often confused. We assume that an organization devoted to a valuable activity deserves special treatment. But governance should depend not only on what an organization says it is for. It should depend on how much power it has, how much money it controls, how transparent it is, and whether those affected by its decisions can challenge it.

An organization can be not for profit while accumulating billions in revenue and maintaining vast financial reserves. It can serve a global constituency while remaining difficult for any single democratic government to supervise. It can conduct commercial activities while describing them as secondary to a public mission. None of these facts proves misconduct. Together, they show why legal categories based on labels are often inadequate.

The same issue will arise with advanced AI. A laboratory may describe itself as a research organization, a public benefit corporation, a safety initiative, or a national champion. These labels may tell us something about its intentions. They tell us much less about its actual power.

A better approach is to ask four questions:

1. What capabilities can the organization deploy?
2. What resources can it command?
3. What dependencies has society developed on it?
4. What consequences can it impose without prior approval?

These questions produce a more realistic classification system. A small nonprofit with a noble mission and limited reach poses a different governance problem from a wealthy organization that controls critical infrastructure, shapes public information, or can rapidly create more capable systems.

We should therefore distinguish between purpose based legitimacy and power based accountability. The first concerns whether an activity is valuable. The second concerns whether the actor performing it remains subject to effective constraint.

Purpose based legitimacy is a useful starting point, but it is not a substitute for accountability. Indeed, noble missions can be especially effective shields because critics can be portrayed as opposing the cause itself. Questioning the financial privileges of a sports body can be cast as hostility toward sport. Questioning the autonomy of an AI laboratory can be cast as hostility toward innovation or national security.

The more prestigious the mission, the more carefully its privileges should be examined.

The Alignment Problem Has Two Layers

Technical AI safety focuses on whether a system's behavior corresponds to human intentions. This is an essential problem. A system may pursue a goal in ways that are technically competent but socially disastrous. It may exploit loopholes, manipulate its operators, or optimize a measurable target while undermining the broader purpose behind that target.

But there is another layer: institutional alignment. This asks whether the organizations developing and deploying powerful systems remain responsive to the public institutions that are supposed to govern them.

The two layers can conflict. A laboratory may build a system that is safer in the narrow technical sense while lobbying for rules that make independent oversight weaker. A government may impose strong internal procedures while competing to attract development from countries with weaker procedures. A company may sincerely want a safe system while facing investors, customers, or geopolitical rivals who reward speed above all else.

In other words, a technically aligned model can be embedded in a misaligned political economy.

This resembles the difference between a well behaved employee and a badly designed organization. The employee may follow instructions responsibly, but if the organization rewards concealment, speed, and rule avoidance, individual virtue will not solve the systemic problem. Conversely, an organization with sound incentives can often tolerate imperfect individuals because procedures, transparency, and accountability limit the damage.

The most important safety property may therefore not be a model's internal disposition. It may be the reversibility of the surrounding system.

Can authorities pause deployment without causing economic collapse? Can an independent body inspect the system without relying on the developer's own measurements? Can users switch providers? Can a government impose penalties that are large enough to matter? Can the public learn what happened after a failure? Can the organization be dissolved, restructured, or taxed without threatening essential services?

If the answer to these questions is no, society has allowed capability to become a source of political immunity.

Designing for Friction Before It Is Needed

The practical lesson is not to prevent every powerful organization from existing. Modern societies need institutions that can coordinate across borders, conduct difficult research, and manage specialized activities. The goal is to ensure that usefulness does not become an excuse for untouchability.

A robust governance framework should create productive friction at the points where power compounds. This does not mean pointless bureaucracy. It means making certain actions slower, more visible, and more contestable when they could create irreversible consequences.

For advanced AI, productive friction might include independent evaluations before access to large amounts of computing power, mandatory reporting of major capability changes, restrictions on transferring frontier systems to lightly regulated jurisdictions, and shared international standards for incident disclosure. It could also include competition rules that prevent a small number of firms from controlling the hardware, cloud infrastructure, talent, and deployment channels needed to build increasingly capable systems.

For international organizations more generally, the same principles apply. Special treatment should be conditional, transparent, and proportional to demonstrated public benefit. Exemptions should have expiration dates. Commercial activities should not disappear behind a nonprofit label. Public subsidies should come with public reporting. Relocation incentives should be evaluated not only by the jobs or prestige they bring, but also by the accountability they remove.

A simple governance audit can help. For any powerful organization, ask:

• Mobility: Can it move its core activities to escape oversight?
• Opacity: Can outsiders verify its finances, decisions, and risks?
• Dependency: Would stopping it impose unacceptable costs on society?
• Concentration: Does it control resources that competitors and regulators need?
• Immunity: Are its legal or tax privileges greater than its public obligations?
• Reversibility: Can its authority be reduced without a crisis?

The more affirmative answers an organization receives, the less we should rely on goodwill, mission statements, or voluntary codes.

Key Takeaways

• Treat regulatory competition as a safety issue. When powerful actors can relocate to the most permissive jurisdiction, national rules alone are unlikely to be sufficient.
• Classify organizations by power, not labels. A nonprofit, research laboratory, or public benefit entity may still require strict oversight if it controls substantial resources or critical capabilities.
• Build reversibility into major systems. Avoid allowing any AI provider, international body, or infrastructure operator to become so indispensable that oversight becomes politically impossible.
• Make privileges conditional. Tax exemptions, subsidies, special legal statuses, and expedited approvals should require transparency, measurable public benefits, and periodic review.
• Separate technical alignment from institutional alignment. A safe system can still be deployed by organizations whose incentives encourage secrecy, speed, or regulatory escape.

The familiar image of an AI risk is a machine deciding that humans are obstacles to its objective. The less familiar, and perhaps more immediate, image is a network of human institutions deciding that accountability is an obstacle to their objective.

One scenario involves intelligence exceeding its creators. The other involves incentives quietly exceeding institutions' capacity to respond. They are not identical threats, but they share a logic: an actor accumulates capability, exploits the gaps between rules, and becomes difficult to restrain because too many other actors benefit from its continued freedom.

That is why the deepest AI safety question may not be, "How do we ensure that a machine wants what we want?" It may be, "How do we ensure that no powerful actor can turn usefulness into exemption?"

The future will not be secured by finding a perfect guardian, whether human or artificial. It will be secured by building systems in which guardians can be questioned, privileges can be withdrawn, and power can be made to stop. The real test of civilization is not whether it can create extraordinary intelligence. It is whether it can remain capable of saying no to intelligence, wealth, and prestige when saying no becomes difficult.