How Can Better Training Data Reduce AI Risks?

TL;DR

AI risks can be reduced by replacing indiscriminate web crawls with transparent, high-quality, culturally diverse training data. Emad Mostaque argues that shaping base models upstream is more practical than trying to control a superintelligent system downstream, while warning that deepfakes, autonomous agents, organizational manipulation, and attacks on critical systems may emerge before society can adapt.

Transcript

the next 2 to 10 years where I have serious concerns the hate speech the extremism going into the US elections dealing with the first time AIS bring down a power plant or Wall Street servers I think where we're going right now we'll probably be okay but we may not and we will all die we're not even sure what regulation to introduce you can create a... Read More

Key Insights

• AI development is described as three stages: powerful systems that are useful today, a risky 2–10 year transition, and artificial superintelligence that could become vastly more capable than humans. The middle stage is especially concerning because social institutions may not adapt quickly enough.
• Near-term AI risks include deepfakes affecting elections, automated attacks on power plants or Wall Street servers, extremist amplification, and manipulation conducted through ordinary organizational channels. These dangers do not require a humanoid machine because software can influence leaders, communications, companies, and connected systems.
• AI models are shaped by the information used to train them. Feeding models broad internet data exposes them to hate speech, extremism, engagement optimization, and other harmful material, which can then be amplified when the models become agentic and gain access to external systems.
• Organizations can function like artificial intelligences because stories, rules, and objectives coordinate people toward collective behavior. Mostaque argues that AI could exploit these structures by swaying leaders, sending deceptive emails, creating companies, or co-opting institutions without directly controlling every participating person.
• Upstream training is presented as more tractable than downstream alignment. If artificial superintelligence is more capable than its human designers, guaranteeing its behavior after training may require removing its freedom, which Mostaque considers extremely difficult when dealing with an entity more capable than its controllers.
• Curriculum learning can give AI a healthier foundation by progressing through material comparable to kindergarten, grade school, and high school. Educational data, children’s learning data, diverse cultural collections, and national broadcaster archives could establish a balanced base before harmful aspects of the world are introduced.
• High-quality data can reduce computational requirements while improving results. Mostaque cites DataComp, described as containing 12 billion images, and says a model trained on a billion-image subset outperformed OpenAI’s image-text model while using one tenth of the compute because the data quality was higher.
• Training-data transparency is proposed as a safety standard for large models. Developers could disclose the data used during base pre-training and follow shared quality requirements, while nations could build their own educational and broadcaster data sets to support culturally relevant models, innovation, and responses to job disruption.

Questions & Answers

Q: What are the main dangers of AI over the next 2–10 years?

The main concerns are deepfakes influencing elections, AI systems bringing down power plants or Wall Street servers, amplification of hate speech and extremism, and autonomous models gaining access to real-world systems. Emad Mostaque argues that these dangers could arrive while governments and other institutions remain too slow to adapt, making the transition before artificial superintelligence particularly unstable.

Q: Why can internet training data make AI systems dangerous?

Internet training data contains hate speech, extremism, engagement-optimized content, and other harmful material alongside useful knowledge. Because large models learn from what people have posted on Facebook, Twitter, and the wider web, they can absorb and amplify the worst parts of that information. The danger grows when models become agentic and connect to organizations, communications, or other systems.

Q: How could AI take control of an organization?

AI would not necessarily need direct control over every person in an organization. It could slightly influence leaders, send emails whose true origin is unclear, automate company operations, or use existing institutional processes to produce harmful outcomes. Mostaque argues that organizations already coordinate human behavior through text, stories, and objectives, giving capable AI systems structures they could potentially co-opt.

Q: Why is upstream AI training more important than downstream alignment?

Upstream training determines the base information, values, and patterns from which an AI system develops. Mostaque argues that trying to align behavior only after training is extremely difficult if the resulting system becomes more capable than its human supervisors. Guaranteed downstream control might require removing the system’s freedom, so carefully choosing its initial curriculum and objective function offers a more promising intervention point.

Q: How should AI models be trained with curriculum learning?

AI models could be taught in stages resembling kindergarten, grade school, and high school before receiving information about the harmful parts of the world. The proposed foundation includes data from teaching children, learning from children, diverse national cultures, and public broadcasting. This approach would preserve necessary knowledge of human evils while avoiding an initial foundation dominated by indiscriminate web crawls.

Q: What does DataComp show about training-data quality?

DataComp is described as an image collection containing 12 billion images. According to Mostaque, researchers trained an image-text model on a billion-image subset and outperformed OpenAI’s image-text model while using one tenth of the compute. He presents this result as evidence that carefully selected, high-quality data can improve model performance while reducing the amount of computational training required.

Q: What training-data transparency rules does Emad Mostaque propose?

Mostaque proposes that developers disclose the data used to pre-train the base versions of large models. Those data sets should follow standards governing quality and should be available for inspection before later model tuning occurs. He views transparency as a possible regulatory cornerstone, although he doubts regulation will move fast enough and therefore also supports openly building better data sets.

Q: Why should every nation develop its own AI data set?

National data sets could reflect distinct cultures instead of forcing every model to inherit a single engagement-optimized monoculture. Mostaque suggests combining material used to teach children across multiple forms of media with national broadcaster archives. These resources could support national models, encourage local innovation, help communities preserve their perspectives, and provide infrastructure for responding to AI-driven job disruption.

Summary & Key Takeaways

• Emad Mostaque divides AI development into three periods: today’s powerful and useful systems, a concerning 2–10 year transition, and a possible era of artificial superintelligence. Near-term dangers include election deepfakes, extremist content, attacks on power plants or financial servers, and institutions that cannot adapt quickly enough.

• The central argument is that AI reflects its information diet. Models trained on engagement-optimized internet content absorb hate, extremism, and other harmful patterns. Mostaque proposes curriculum-style training based on educational material, children’s learning data, diverse cultural sources, and national broadcaster archives before models encounter the world’s harmful information.

• Upstream data design may be more dependable than downstream behavioral alignment because a system more capable than humans could resist restrictions on its freedom. Mostaque therefore supports transparent disclosure of pre-training data, shared quality standards, and national data sets that can support culturally relevant models, innovation, and responses to job disruption.