Jim Fan on Nvidia's Embodied AI and Robot Foundation Models

TL;DR

Nvidia's robotics strategy combines three data types: internet-scale video for common-sense priors, GPU-accelerated simulation for infinite action data, and teleoperated real-robot data with no sim-to-real gap. Jim Fan, who leads Nvidia's GEAR group behind Project GR00T, expects a GPT-3 moment for robotics within two to three years, echoing Jensen Huang's belief that everything that moves will eventually be autonomous.

Transcript

so from uh the the chip level which is the J and Thor family to the foundation model uh project Gro and also to the to the simulation and the utilities that we build along the way it will become a platform a Computing platform for humanoid robots and then also for intelligent robots in general so I want to quote Jensen here um one of my favorite qu... Read More

Key Insights

• Nvidia's GEAR group (Generalist Embodied Agent Research), co-led by Jim Fan, builds AI agents that generate actions in both virtual worlds (gaming AI, simulation) and the physical world (robotics), summarized in three words: 'we generate actions.'
• Project GR00T is Nvidia's moonshot effort to build foundation models for humanoid robotics, unveiled at Jensen Huang's GTC keynote in March, aiming to create the AI brain for humanoid robots and intelligent robots in general.
• A successful robotics data strategy combines three buckets: internet-scale text and video, simulation-generated synthetic data, and real robot data collected by teleoperating robots, each with complementary strengths that offset the others' weaknesses.
• Internet-scale data is the most diverse and encodes common-sense priors because most online videos are human-centered, but it lacks action signals since motor control cannot be downloaded from the internet.
• Simulation provides effectively infinite data that scales with GPU compute, and GPU-accelerated simulators can accelerate real time by up to 10,000x, enabling far higher data throughput than the 24-hours-per-day limit of physical collection.
• Simulation's weakness is the sim-to-real gap: physics and visuals differ from the real world, and simulated content is less diverse than real-world scenarios encountered by robots.
• Real robot data has no sim-to-real gap because it is collected on actual hardware, but it is expensive, requiring hired human operators and limited by the 24-hour physical clock.
• Nvidia's competitive advantages in robotics are compute resources for scaling foundation models and decades of simulation expertise in physics simulation, rendering, and real-time GPU acceleration from its origins as a graphics company.

Questions & Answers

Q: What is Nvidia's GEAR group and what does it do?

GEAR stands for Generalist Embodied Agent Research, a team co-led by Jim Fan at Nvidia. Summarized in three words, the team 'generates actions' because it builds embodied AI agents that take actions in different worlds. When actions happen in virtual worlds, that is gaming AI and simulation; when they happen in the physical world, that is robotics. The group currently focuses on Project GR00T, building the AI brain for humanoid robots.

Q: What is Project GR00T from Nvidia?

Project GR00T is Nvidia's moonshot effort at building foundation models for humanoid robotics. It was unveiled by Jensen Huang at his GTC keynote in March. The GEAR team is focused on GR00T with the goal of building the AI brain for humanoid robots and even beyond, extending to intelligent robots in general. The cute GR00T humanoid robots appeared on stage with Jensen at GTC.

Q: What are the three kinds of data used to train robotics foundation models?

Nvidia divides its robotics data strategy into three buckets. First is internet-scale data, meaning all the text and videos online. Second is simulation data, where Nvidia's simulation tools generate large amounts of synthetic data. Third is real robot data, collected by teleoperating robots and recording the results on robot platforms. Jim Fan believes a successful strategy combines all three, mixing them into a unified solution that leverages each one's strengths.

Q: Why is simulation important for robotics data generation?

Simulation provides effectively infinite data that scales with compute: the more GPUs put into the simulation pipeline, the more data is produced. It is also fast, since GPU-accelerated simulators can accelerate real time by up to 10,000x, collecting data at much higher throughput than real robots limited to 24 hours per day. In simulation you can capture all actions and observe their consequences within the environment, unlike internet data which lacks action signals.

Q: What are the weaknesses of simulation data for robotics?

Simulation's main weakness is the sim-to-real gap. No matter how good the graphics pipeline is, the simulated physics will differ from the real world, and the visuals will not look exactly as realistic as reality. There is also a diversity issue: the contents in simulation will not be as diverse as all the scenarios a robot encounters in the real world. These gaps limit how well simulation-trained behavior transfers to physical robots.

Q: Why is internet-scale data useful but insufficient for robotics?

Internet data is the most diverse and encodes a lot of common-sense priors. Most online videos are human-centered because people love taking selfies and recording activities, and many are instructional videos, so robots can learn how humans interact with objects and how objects behave in different situations. However, internet data does not come with actions: you cannot download the motor control signals of robots from the internet, which is why simulation and real robot data are also needed.

Q: When does Jim Fan expect a GPT-3 moment for robotics?

Jim Fan hopes to see a research breakthrough in robot foundation models within the next two to three years, which he calls a GPT-3 moment for robotics, though he notes this is pure speculation. After that breakthrough, robots entering people's daily lives is harder to predict because it involves more than the technical side: robots need to be affordable, mass produced, safe in hardware, and to satisfy privacy and regulatory requirements, all of which will take longer.

Q: What is Nvidia's competitive advantage in building robotics foundation models?

Nvidia has two main advantages. First is compute resources, since all foundation models require significant compute to scale up, and Nvidia believes in scaling laws even though the scaling laws for embodied AI and robotics are yet to be studied. Second is simulation expertise: before being an AI company, Nvidia was a graphics company with many years of experience in physics simulation, rendering, and real-time acceleration on GPUs, which it uses heavily in its robotics approach.

Summary & Key Takeaways

• Jim Fan, senior research scientist at Nvidia, co-leads the GEAR group, which builds embodied AI agents that generate actions across virtual worlds like gaming and simulation and the physical world of robotics. His path ran from being OpenAI's first intern in 2016, working on the World of Bits agent project, to a Stanford PhD with Fei-Fei Li, then to Nvidia.

• Nvidia's robotics approach rests on a three-pronged data strategy. Internet-scale video supplies diverse common-sense priors but lacks action signals. GPU-accelerated simulation yields effectively infinite data, accelerating real time up to 10,000x, but suffers a sim-to-real gap. Real teleoperated robot data avoids that gap but is costly and clock-limited.

• Fan hopes for a research breakthrough, a GPT-3 moment for robotics, within the next two to three years. Wider adoption into daily life will take longer, requiring affordable mass production, hardware safety, privacy, and regulation. He endorses Jensen Huang's prediction that everything that moves will eventually be autonomous.