Alexandr Wang on Why Data Is the Bottleneck for AI

TL;DR

AI rests on three pillars: algorithms, compute, and data, and in 2016 nobody was working on the data pillar, which is why Scale AI was founded. The easy data on the internet is now exhausted, so progress depends on frontier data production: expert reasoning chains, agent workflow logs, multilingual and multimodal data. Data abundance or data scarcity is a choice the industry makes.

Transcript

hi listeners and welcome to no priors today I'm excited to welcome Alex Wang who started scale AI as a 19-year-old College Dropout scale has since become a juggernaut in the AI industry modern AI is powered by three pillars compute data and algorithms while research labs are working on algorithms and AI chip companies are working on the compute pil... Read More

Key Insights

• Modern AI is powered by three pillars: compute, data, and algorithms. Research labs work on algorithms and chip companies work on compute, while Scale positioned itself as the data foundry, the pillar that in 2016 had nobody focused on it.
• Neural networks are very much a product of their data. Wang reached this conclusion while trying to train image recognition models at MIT, and playing that observation forward is what convinced him data would only become more important over the long arc of the technology.
• The first data engine Scale built supported sensor-fused data, combining 2D camera imagery with 3D lidar from vehicles. It quickly became an industry standard across autonomous driving players including General Motors, Toyota, and Stellantis.
• Scale's government business began with data engines for geospatial, satellite, and overhead imagery around 2019 to 2020, a period when the future of AI applications was genuinely unclear. That technology fueled the first AI program of record for the US Department of Defense.
• RLHF work with OpenAI began on GPT-2 in 2019, in what Wang calls the primordial days. The models were rudimentary enough that they did not seem like anything at the time, but Scale partnered anyway on the reasoning that OpenAI had smart people.
• Data abundance versus data scarcity is a choice the industry makes, not a fixed constraint. Wang frames Scale's role as building data abundance, because the ability to scale data is what determines whether models can progress from one generation to the next.
• The easy data, meaning everything on the internet, is already exhausted. Models can no longer learn much more from sources like Reddit comments, so quality requirements have increased dramatically and the field has shifted to forward data production with high supervisory signal.
• Frontier data includes reasoning chains of thought from mathematicians, physicists, biologists, chemists, lawyers, and doctors, plus agent workflow data from enterprise, consumer, and coding agents, multilingual coverage, video and audio, and esoteric data types inside enterprises.

Questions & Answers

Q: What are the three pillars of modern AI?

Modern AI is powered by compute, data, and algorithms. Wang identified this framework while studying AI and machine learning at MIT, around the time DeepMind released AlphaGo and Google released TensorFlow. He observed that research labs such as OpenAI and Google's labs were working on the algorithms pillar, and Nvidia was already a clear leader in building compute for AI systems. Nobody, however, was focused on data, even though it was clear that over the long arc of the technology data would only become more and more important. That gap is the reason Scale AI exists as a data foundry.

Q: Why did Alexandr Wang drop out of MIT to start Scale AI?

While at MIT he was trying to train image recognition neural networks and realized very quickly that these models were very much a product of their data. Playing that forward, he concluded that AI rests on three fundamental pillars: algorithms, compute, and data. Companies already existed for the first two, but nobody was solving data. In 2016 he dropped out of MIT, went through Y Combinator, and started Scale to be the organization that would solve the hard problems of producing and creating enough data to fuel the AI ecosystem.

Q: How did Scale AI start in autonomous vehicles?

Scale followed the standard early startup advice to focus, and autonomous driving was the prime-time AI use case at the time. The company got started in 2016 at the very beginning of the autonomous vehicle cycle, and Cruise was acquired around the time Scale was going through Y Combinator. Scale built the first data engine supporting sensor-fused data, meaning a combination of 2D camera data plus 3D lidar data captured from the vehicles. That engine quickly became an industry standard, and Scale worked with General Motors, Toyota, Stellantis, and many others.

Q: How did Scale AI get into government work?

Starting around 2019 to 2020 it was actually pretty unclear where the future of AI use cases and applications would come from, since this was before language models and generative AI. During that period of high uncertainty, Scale turned to government applications, an area with clear applicability that was becoming more important globally. The company built the first data engines to support government data, mostly geospatial, satellite, and other overhead imagery. That work fueled the first AI program of record for the US Department of Defense and proved critical years later in the Ukraine conflict.

Q: When did Scale AI start working with OpenAI on RLHF?

Scale partnered with OpenAI around 2019 to run the very first experiments on RLHF on top of GPT-2, during what Wang describes as the primordial days of RLHF. The models at that time were really rudimentary and truly did not seem like anything, but Scale partnered anyway on the reasoning that OpenAI was a bunch of smart people worth working with. Scale worked with the team that originally invented RLHF and kept innovating with them from 2019 onward, including on the InstructGPT paper that fed into GPT-3 and served as a precursor to ChatGPT.

Q: What is frontier data and why does it matter?

Frontier data is forward-produced data with high supervisory signal, created because the community has exhausted the easy data available on the internet. Quality requirements have increased dramatically, and models can no longer learn much more from sources such as Reddit comments. Frontier data includes reasoning chains of thought from the world's experts, mathematicians, physicists, biologists, chemists, lawyers, and doctors. It also includes agent workflow data from enterprise, consumer, and coding agents, multilingual data spanning the many languages spoken in the world, and multimodal data such as video and audio, plus esoteric data types inside enterprises and industrial use cases.

Q: Will AI run out of training data?

Wang treats running out of tokens as a choice rather than an inevitability. The industry can choose either data abundance or data scarcity, and Scale views its role in the ecosystem as building data abundance. The ability to scale data is key to scaling large language models, and one of the fundamental bottlenecks standing between GPT-4 and the next generation is whether enough data exists to get there. Since easy internet data is exhausted, the answer is forward data production of high-quality frontier data rather than accepting scarcity.

Q: How has the scope of AI use cases changed for Scale AI?

In the autonomous vehicle era it felt like building one very specific use case that would be very valuable. Wang says that for the first time in maybe the entire history of AI, the technology now truly feels general purpose and applicable across a very large number of business use cases. At the mathematical level AI is functions that can approximate nearly any function, including intelligence. As an infrastructure provider, Scale's burden is to anticipate where breakthrough use cases will appear and lay down the tracks before the freight train of AI comes rolling through.

Summary & Key Takeaways

• Alexandr Wang studied AI and machine learning at MIT during the year DeepMind released AlphaGo and Google released TensorFlow. Training image recognition neural networks showed him that models are largely a product of their data. He identified three pillars behind AI, algorithms, compute, and data, saw labs working on algorithms and Nvidia leading on compute, and found nobody focused on data.

• Scale dropped out of MIT in 2016, went through Y Combinator, and focused first on autonomous driving, then the prime-time AI use case. The company built the first data engine supporting sensor-fused data, combining 2D camera and 3D lidar inputs, which became an industry standard across players including General Motors, Toyota, and Stellantis.

• Around 2019 to 2020, with the future of AI applications unclear, Scale moved into government work, building the first data engines for geospatial, satellite, and other overhead imagery. That work fueled the first AI program of record for the US Department of Defense and proved critical years later in the Ukraine conflict.

• In 2019 Scale partnered with OpenAI on the first RLHF experiments on top of GPT-2, working with the team that originally invented RLHF. That collaboration continued through GPT-3 and the InstructGPT paper, a precursor to ChatGPT. After DALL-E 2 and ChatGPT arrived in 2022, Scale concentrated on being the data foundry for generative AI.

• Today Scale's data foundry fuels essentially every major large language model in the industry, including work with OpenAI, Meta, and Microsoft. Ambitions have expanded from GPT-3, which Wang describes as a landmark model with a modesty to it, toward agents, complex reasoning, multimodality, and multilinguality, each of which places new demands on infrastructure.

• The community has exhausted the easy data available on the internet, so the next phase is forward data production with high supervisory signal. Frontier data means reasoning chains of thought from mathematicians, physicists, biologists, chemists, lawyers, and doctors, agent workflow data from enterprise, consumer, and coding agents, plus multilingual, video, audio, and esoteric enterprise data types.