How Could Superintelligent AI Shape Humanity?

TL;DR

Superintelligent AI could move civilization beyond the present human condition, but the outcome could involve either lasting technological advancement or extinction. Because future existential risks are likely to arise from human activity, progress in machine intelligence should be considered alongside its potentially profound consequences for society and humanity’s long-term future.

Transcript

Ladies and gentlemen, please welcome Professor, University of Oxford, Director, Future of Humanity Institute and of Strategic Artificial Intelligence Research Center, Nick Bostrom. Uh, thank you all for coming. Um, I run this, um, research group. We are a bunch of, uh, mathematicians, philosophers, computer scientists, economists, and we have the u... Read More

Key Insights

• The modern human condition is probably a transitional phase rather than a stable endpoint. Its unusual position on historical, evolutionary, geological, and spatial scales makes dramatic future change plausible, even though departures from present conditions are often treated as extraordinary possibilities.
• Humanity is increasingly likely to leave its current band of technological capability as the timeframe expands. The downward path could end in extinction, while the upward path could produce technological maturity, space colonization, and continued development lasting until cosmic expansion prevents access to additional resources.
• Extinction is an attractor state because a species that becomes extinct normally remains extinct. Technological maturity may represent another attractor state because a civilization capable of expanding into space could continue developing and obtaining resources for billions of years.
• Serious existential risks over the next decades or hundred years are likely to be anthropogenic. Humanity has survived natural dangers for more than a hundred thousand years, so a future extinction event would probably involve something new, with human activity supplying most new threats.
• Biotechnology, nanotechnology, and artificial intelligence are potential sources of existential danger. These technological advances could move humanity toward the downward outcome, even though advanced technology and superintelligence could also provide a route toward a more capable civilization.
• Machine intelligence could eventually outperform biological cognition because machine substrates can process information closer to physical limits than brains can. Biological enhancement and hybrid systems are other possible routes to superintelligence, but machines offer the greatest long-term information-processing potential described in the talk.
• Modern artificial intelligence emphasizes learning rather than handcrafted expert knowledge. Earlier expert systems depended on humans entering knowledge tokens and permitted only simple logical derivations, whereas machine-learning systems can acquire capabilities from experience and therefore may progress much further.
• Artificial intelligence progress depends substantially on both hardware and algorithms. In chess computing, roughly half the improvement after Deep Blue came from faster computers and roughly half from better algorithms, a rule of thumb that appears to apply across multiple domains.

Questions & Answers

Q: Why might the modern human condition be temporary?

The modern human condition appears temporary because it is an anomaly across several scales. Humanity lived as hunter-gatherers for most of the past hundred thousand years, while recent economic and technological capabilities have grown rapidly. A plot of world GDP over ten thousand years does not suggest a settled state. Over longer periods, the probability of remaining within the current narrow capability range appears to diminish.

Q: What long-term outcomes could humanity face?

Humanity could eventually leave its present capability range in either a downward or upward direction. The downward outcome is extinction, which is stable because extinct species tend to remain extinct. The upward outcome involves technological maturity, possible space colonization, and continued development. Once that stage is reached, expansion might continue for billions of years until cosmic expansion prevents access to more distant resources.

Q: Why are future existential risks likely to be human-made?

Future existential risks are likely to be anthropogenic because humanity has already survived natural threats for more than a hundred thousand years. Those threats include asteroids, supervolcanoes, and firestorms. If something causes human extinction over the next decades or hundred years, it will probably be a new danger. Human activity is the source of most new things, including powerful emerging technologies.

Q: Which technologies could create existential risks?

Biotechnology, nanotechnology, and artificial intelligence are identified as technological advances that could place humanity in the downward outcome of extinction. The argument does not claim that these technologies must cause catastrophe. Instead, it treats them as plausible sources of new, human-created risks that differ from natural dangers humanity has already survived throughout its history.

Q: How could superintelligence change civilization?

Superintelligence could serve as a general route from the current human condition to a more technologically mature state. Such a civilization might begin colonizing space and continue developing for billions of years. However, advanced artificial intelligence also belongs among the technologies capable of creating existential danger, so the transition could have profoundly different consequences depending on how it unfolds.

Q: Why could machine intelligence surpass biological brains?

Machine intelligence could surpass biological brains because machine substrates can support information processing that moves closer to physical limits. Biological cognitive enhancement and hybrid human-machine systems are also possible paths toward superintelligence. In the long run, however, the information-processing capacity available in machines is expected to far exceed what is possible within a biological substrate such as the human brain.

Q: How is modern AI different from early expert systems?

Early artificial intelligence relied on expert systems built from knowledge that humans carefully encoded by hand. Those systems could make simple logical derivations, but their outputs largely reflected what people had explicitly entered. Modern AI focuses on machine learning, allowing systems to acquire capabilities from data or experience. The Atari system illustrated this shift by learning multiple games directly from raw pixel input.

Q: What factors are driving progress in artificial intelligence?

Artificial intelligence progress is driven by algorithmic advances and improvements in hardware capability. Chess computing provides an example: roughly half of its improvement after Deep Blue came from faster computers, while roughly half came from better algorithms. The precise proportions can vary by field, but both drivers have contributed substantially across a range of domains and enabled increasingly capable learning systems.

Summary & Key Takeaways

• Modern civilization is unlikely to represent a permanent human condition. Humanity spent most of the past hundred thousand years living as hunter-gatherers, while recent technological and economic growth suggests a continuing transition. Over longer timeframes, civilization may leave its current capability range through extinction or further technological development.

• The most serious existential risks over the coming decades or hundred years are likely to be anthropogenic because humanity has already survived natural threats for more than a hundred thousand years. New technologies, particularly biotechnology, nanotechnology, and artificial intelligence, could create risks capable of pushing humanity toward extinction.

• Superintelligence could provide a route toward technological maturity, space colonization, and a durable post-human condition. Machine intelligence is especially important because machine-based information processing could eventually exceed biological cognition. Progress is being driven by both improved algorithms and faster hardware, while modern AI increasingly learns instead of relying on handcrafted knowledge.