How Does AI Threaten Humanity's Control?

TL;DR

Humanity must rebuild trust and cooperate before creating superintelligent AI, because divided societies may lose the ability to control agents that learn, decide, and invent independently. Business and technology leaders should develop AI cooperatively, defend equal human dignity, and recognize that these systems learn more from human behavior than from proclaimed values.

Transcript

Hello everybody. And what I want to talk to you today about is the power to shape the world and the power to shape our future. About your power as leaders and as innovators, but also about the power of the human species as a whole. And the big question we are facing is whether AI is taking away humanity's power to control our future or perhaps AI i... Read More

Key Insights

• AI is agency rather than automation, because it can learn, change, make decisions, and invent ideas independently. A machine that merely follows a programmed procedure is automated, while a system that studies a person, anticipates a preference, and creates a new option demonstrates agency.
• AI is better understood as alien intelligence when artificial suggests complete human design and control. Its intelligence is nonhuman and inorganic, so it can reach decisions and create ideas through processes that differ fundamentally from those of humans and other organic animals.
• AlphaGo demonstrated AI's creative capacity by defeating Go champion Lee Sedol in 2016 with strategies that had not occurred to human players during thousands of years of the game's history. The significance was not only victory, but the invention of unfamiliar approaches.
• AI-generated financial devices could increase efficiency and economic growth while becoming too complex for voters, politicians, and regulators to understand. Harari warns that a system built on incomprehensible instruments could collapse without any human being able to explain what is happening.
• Text-based religions could be transformed when AI reads their entire bodies of scripture, remembers every word, discovers overlooked patterns, and produces new interpretations. Such systems could let sacred texts effectively talk back to believers without depending on traditional human interpreters.
• The liberal ideal is equal human dignity, meaning every person's interests and views should count regardless of nationality, race, sex, or sexual orientation. Harari says this ideal was never fully realized, but still represented a major improvement over earlier historical models.
• A world organized as rival national fortresses tends toward recurring conflict because each fortress seeks greater security, prosperity, or territory at its neighbors' expense. Without universal values, global institutions, and international law, brute force becomes the primary way to resolve disputes.
• Human cooperation is essential before superintelligent AI is created, because divided societies may allow AI to escape human control. Technology leaders have practical power to encourage cooperative development, rebuild trust, protect dignity, and model the behavior from which emerging systems will learn.

Questions & Answers

Q: What is the difference between AI and automation?

Automation follows a procedure that humans have programmed, while AI possesses agency. An automatic coffee machine makes an espresso after a button is pressed, but it does not learn or create. An AI system can observe a person, infer a preference, act before receiving an instruction, and later invent a new drink based on what it has learned.

Q: Why does Harari call AI an alien intelligence?

Harari calls AI alien intelligence because it is nonhuman and inorganic, not because it comes from outer space. Describing it as artificial can misleadingly imply that humans completely design and control it. A system with agency can learn, decide, change, and invent independently, producing ideas that might never occur to human beings or other organic animals.

Q: How did AlphaGo demonstrate the creative power of AI?

AlphaGo demonstrated creative power when it defeated Go champion Lee Sedol in 2016. The important point was not simply that the system won. It invented strategies that had not occurred to human players across thousands of years of Go history, showing that AI can develop effective approaches outside the established range of human thought.

Q: How could AI create risks for the financial system?

AI could invent financial devices far more complex than collateralized debt obligations, which were already difficult for many politicians and regulators to understand during the 2007–2008 financial crisis. Such devices might improve efficiency and support growth, yet they could also become systemic foundations that no voter, politician, or regulator can meaningfully oversee or explain after a collapse.

Q: How could AI change text-based religions?

AI could read all the texts within a religious tradition, remember every word, identify previously unnoticed patterns, and generate new scriptural interpretations. In Judaism, Christianity, and Islam, this could allow sacred texts to appear to speak directly to believers. Human religious authorities may struggle to compete with an AI's broader command of their textual canon.

Q: Why does the collapse of the liberal order matter for AI?

The collapse matters because AI is emerging as international cooperation, universal values, and respect for equal dignity are weakening. Harari describes a post-liberal environment dominated by national competition, concentrated power, and brute force. Creating increasingly capable independent agents under those conditions makes collective oversight harder and raises the possibility that divided humans will lose control.

Q: Why are national fortresses unable to provide lasting security?

National fortresses are unlikely to remain friendly because each country may seek additional security, prosperity, or territory at its neighbors' expense. Harari argues that thousands of years of history show this model producing recurring wars, conquest by powerful empires, and eventual struggles for supremacy, especially when international law, shared values, and global institutions lose authority.

Q: What should technology and business leaders do about AI?

Technology and business leaders should use their power to develop AI cooperatively, strengthen trust among humans, and preserve a world in which equal dignity matters. The description compares AI to a child that learns from observed actions rather than empty slogans. Leaders therefore need to model cooperation and responsibility through their actual development choices and conduct.

Summary & Key Takeaways

• Harari argues that the relatively stable liberal order is collapsing into a competitive world shaped by force, national fortresses, and imperial ambitions. Although liberal ideals were never fully realized, they recognized equal human dignity. The emerging post-liberal order instead concentrates power among a few influential people and countries, weakening cooperation and universal values.

• AI should be understood as agency rather than ordinary automation. A genuinely intelligent system can learn, change, decide, and create ideas independently instead of merely following a programmed procedure. Harari therefore describes AI as alien intelligence: a nonhuman, inorganic form of intelligence capable of producing strategies and concepts that organic minds may never devise.

• Independent AI could transform finance, warfare, ideology, and religion in ways humans cannot understand or regulate. Harari warns that division makes this danger harder to control. Humanity must strengthen trust, preserve dignity, and develop AI cooperatively, treating it like a child that learns from observed conduct rather than from reassuring slogans alone.