Why the Biggest AI Question Is Not Intelligence, but Power

The real question is not whether AI can think

What if the most important question about AI is not whether it becomes intelligent, but who it will let extract value from everyone else?

That may sound like a distraction. The public conversation about AI is dominated by model benchmarks, AGI timelines, and philosophical debates about consciousness. But those debates can hide a more practical and more consequential truth: every major general purpose technology becomes a struggle over who gets locked in, who gets priced, and who gets to keep the surplus.

That is the deep link between the hype around artificial intelligence and the uglier history of digital platforms. The danger is not simply that AI will become smarter. The danger is that it will become the perfect instrument for surveillance pricing, wage compression, and dependency creation. In other words, AI may matter less as a mind than as a machine for rearranging bargaining power.

The central issue is not whether machines can reason like humans. It is whether institutions can use machine intelligence to make everyone else easier to exploit.

From intelligence to extraction

The AGI conversation often starts with a category error. We ask whether a system can match human intelligence, as if that were the threshold that determines its social impact. But the more revealing question is: what incentives will surround the system once it is good enough to be useful?

A calculator did not need self-awareness to transform accounting. A spreadsheet did not need emotions to reorganize finance. Likewise, AI does not need consciousness to become economically disruptive. It only needs to be good at a few high-leverage tasks: predicting behavior, automating persuasion, segmenting people into price tiers, and reducing the cost of replacing labor.

That is why so much AI investment is not really about making better tools for human flourishing. It is about finding a cleaner way to do what companies have always wanted to do: pay less for labor, charge more for demand, and keep the difference. Once you see that, the obsession with AGI starts to look like a smoke screen. The more urgent question is not “Can it think?” but “What leverage does it create for the people who own it?”

A system can be mediocre at solving human problems and still be extraordinarily valuable if it is excellent at helping a boss decide whom to fire, what to charge, and how tightly to squeeze each worker or customer. That is the overlooked business model of much of modern AI.

The platform playbook has already shown us the ending

This is not a new story. Digital platforms have already taught us the basic choreography of extraction.

First, they are charming to users. They promise convenience, openness, and a better deal than the incumbents. Then they create lock-in: your photos are there, your friends are there, your customers are there, your work history is there, your reputation is there. Once leaving becomes painful, the platform can begin to change the terms.

The pattern is almost theatrical:

1. Stage one: win the end user. Offer something attractive and frictionless.
2. Stage two: win the business customer. Use the locked-in user base to sell access, advertising, or distribution.
3. Stage three: squeeze both sides. Raise prices, lower quality, increase fees, and shift value to shareholders.

This pattern matters because it shows that deterioration is not accidental. It is often the business model itself.

Facebook is a perfect illustration. It began by promising not to spy and not to clutter the feed with unwanted content. Once users were trapped by social dependence, it could violate those promises. Then advertisers became dependent on access to those users. After that, the platform could degrade the experience for both sides, because neither side could easily escape the networked dependency it had created.

That three-step move is now everywhere. The interface improves just long enough to create habit, then the terms worsen because the lock-in is doing the work.

AI is becoming the next lock-in engine

AI is poised to intensify this same pattern, but with a more intimate form of control.

A search engine does not just answer questions. It can become the gateway through which a person discovers reality. A scheduling assistant does not just save time. It can become the layer that intermediates all decisions. A workplace copilot does not just help write emails. It can become the system through which management observes, evaluates, and disciplines labor.

Once a system sits between people and the world, it can start shaping the terms of the relationship. That is where AI gets dangerous. It can become the best imaginable tool for personalized extraction, because it can infer exactly how much pressure each person can bear.

Think about it in concrete terms:

• A worker applying for a job can receive an offer adjusted to their desperation.
• A customer shopping online can be shown the highest price they are likely to tolerate.
• A contractor can be assigned shifts based on financial vulnerability.
• A creator can be offered lower compensation once the platform knows they have no better option.

This is not science fiction. It is the logical extension of surveillance systems already used in labor markets and advertising markets. The only difference is scale and precision.

The most alarming possibility is that AI will not simply automate tasks. It will automate asymmetric bargaining. It will help the more powerful side learn more, decide faster, and extract more while appearing neutral.

Why definitions of AGI miss the point

The debate over AGI often gets stuck because the term is slippery. Intelligence is not one thing. It includes reasoning, pattern recognition, language, memory, judgment, emotional attunement, social intuition, and more. Consciousness is even harder to define. A machine can be astonishingly capable in one domain and still be profoundly limited in others.

But the social risk of AI does not wait for philosophers to settle the definition. A system can be non-conscious, non-human, and still deeply disruptive if it is deployed inside institutions that already favor concentration.

That is the key insight: capability matters less than deployment context.

A model trained to predict, classify, and persuade is not just a model. In the hands of a monopolist, it becomes a lever. In the hands of a competitive market, it may be a tool. In the hands of a platform with lock-in, it becomes a tollbooth. The same technical object can produce radically different outcomes depending on the surrounding rules.

This is why the most important question is not whether AI reaches some abstract threshold of general intelligence. It is whether AI gets embedded in systems that already have the power to trap users, suppress wages, and distort markets.

A useful mental model: AI as a force multiplier for choke points

Here is a better way to think about modern AI: it is not primarily a creator of value, but a multiplier of control at choke points.

A choke point is any place where one actor can control access to many others. Search, app stores, cloud infrastructure, ad marketplaces, labor platforms, and proprietary device ecosystems are all choke points. Whoever controls them can tax, steer, surveil, or exclude.

AI makes choke points more dangerous for three reasons:

1. It reduces the cost of individualized control

Instead of showing everyone the same ad, offer or policy, AI can tailor the message to each person’s weaknesses.

2. It deepens dependency

If the system writes your emails, sorts your leads, recommends your prices, and manages your workflow, it becomes harder to leave, even if it worsens.

3. It obscures responsibility

When decisions are made by models, institutions can hide behind complexity. A human manager can blame the system. A platform can claim neutrality. A monopoly can present extraction as optimization.

This is why AI should be understood less as a magic brain and more as a control layer. Control layers do not need to be perfect. They only need to be useful enough to concentrate advantage.

The most powerful technologies are not always the ones that understand the world best. They are the ones that let someone else understand you better than you understand them.

The policy problem hiding inside the technical debate

If this sounds like a policy issue, that is because it is.

Businesses cannot solve this alone by building better products. Individuals cannot solve it by opting out one at a time. Once lock-in and scale have created a chokepoint, the problem becomes structural. You cannot outcompete a system that is allowed to use regulation, data, and network effects as weapons.

That is why antitrust, interoperability, privacy rules, labor protections, and procurement standards matter. They are not abstract legal topics. They are the mechanisms that determine whether a technology becomes competitive infrastructure or extractive infrastructure.

Interoperability is especially important because it lowers switching costs. If users can leave, discipline returns. If workers can move between platforms, wage suppression becomes harder. If businesses can serve customers without handing all their leverage to a gatekeeper, the gatekeeper loses its tollbooth power.

The same logic applies to data. If platforms can buy and weaponize intimate personal data, they can price people according to vulnerability. If data brokers are left unchecked, the most desperate people will quietly pay the most, whether as workers, borrowers, or consumers.

In this sense, privacy is not merely about dignity. It is about market fairness and human bargaining power.

What to do with this insight

The practical lesson is not to panic about sentient machines. It is to get more suspicious of systems that claim to be neutral while quietly redistributing power.

Whenever a new AI feature is launched, ask four questions:

1. Where is the choke point? Who controls the interface between people and opportunity?

2. What lock-in does it create? What gets harder to leave once users adopt it?

3. What does it know that others do not? Is it learning private information that can be used to price, rank, or pressure people?

4. Who captures the surplus? Does the value go to users, workers, businesses, or only to owners and shareholders?

If you cannot answer these questions, the conversation about “innovation” is premature.

The most valuable organizations in the AI era may not be the ones with the largest models. They may be the ones that refuse to use the model as a surveillance engine. They may be the ones that build portability, transparency, and competition into the product from the start. And for policymakers, the priority should be clear: do not let AI become the new justification for old monopolistic behavior.

Key Takeaways

• Stop asking only whether AI is intelligent. Ask what kinds of power it concentrates.
• Treat lock-in as the central risk. Once users, workers, or customers cannot leave, the quality of service will usually decline.
• Watch for surveillance pricing. If a system learns your vulnerability, it can charge or pay you accordingly.
• Support interoperability and portability. These are not technical niceties. They are defenses against extraction.
• Evaluate AI by its bargaining effects. The key question is whether it strengthens the weaker side of a market or the stronger one.

The future of AI is a future of institutions

The biggest mistake is to imagine that AI’s destiny is contained inside the model itself. It is not. AI will not simply tell us what intelligence is. It will reveal what our institutions allow intelligence to be used for.

That is the more unsettling and more useful frame. The real threat is not that machines become human. The real threat is that human organizations become more machine-like in their ability to observe, segment, and extract.

So the next time someone asks when we will reach AGI, a better question may be: what happens first, artificial general intelligence, or artificial general extraction?

The answer to that question will shape far more of daily life than any abstract definition of consciousness ever could.