How to train a robotic hand policy using simulation

TL;DR

The core idea is to train a robot hand policy and a vision model entirely in simulation and then transfer to the real robot. This works thanks to domain randomization, which exposes the system to varied noise and friction so the learned policy is robust. Training is done separately for vision and policy and then combined at run time on the real device.

Transcript

last hi folks I'm Alex Rey I'm on the robotics team here and this is gonna be a different sort of talk this is an overview of a recent large team result and instead of being these like really great in turn projects unfortunately they're they're limited on people and limited on time this is about 12 months worth of work for about 12 people so there'... Read More

Key Insights

• The task is to achieve arbitrary rotations of an object held in a five finger dexterous hand.
• Domain randomization is used to cover unmodeled effects like friction and tendon creep, enabling robust policy transfer to the real robot.
• Vision and policy models are trained separately in simulation and then rolled out together on the real robot.
• A simple recurrent policy controls the robot hand, while the vision system localizes the object inside the hand.
• The system uses a lightweight PPO-like policy structure with an actor-critic setup and a Floyd connect layer for action outputs.
• Training is distributed across thousands of CPU cores and multiple GPUs to handle large-scale simulated data.
• The researchers attempted several tracking and vision approaches but settled on a simple, effective combination that could run in real time.
• Gloves did not improve performance, highlighting that domain randomization, not added hardware, drove the robustness.
• More videos from this channel:
  • How to amplify internal expertise with AI
  • How to Build AI Apps with OpenAI APIs for Developers
  • How close are we to automating scientific discovery?
• More from this channel: OpenAI

Questions & Answers

Q: What is the North Star task described for the robot hand?

The North Star task is to manipulate an object in the robot hand to achieve a sequence of 50 independent randomly drawn rotations. This requires the object to be oriented arbitrarily without being dropped, representing a challenging, robust manipulation goal that guided the research throughout.

Q: How is the policy trained and deployed in this project?

The policy is trained as a simple recurrent model using an actor-critic framework. It runs on the real robot, receiving observations from the vision system and camera data, and outputs actions to control the hand. Training used large-scale simulation resources, while real-world deployment relied on the pre-trained model combined with live observations.

Q: What role does vision play in the system and how is it trained?

Vision is handled by a separate model trained in simulation to localize the object inside the hand. The vision model uses a simple three-branch architecture that shares parameters and predicts position and rotation. It is trained offline in simulation and then used at run time to provide observations to the policy.

Q: What is domain randomization and why is it important here?

Domain randomization is the technique of exposing the simulator to a wide variety of noise and material variations rather than perfectly modeling the real world. This makes the learned policy robust to unmodeled effects like friction, material differences, and mechanical idiosyncrasies, enabling successful transfer to the real robot hand.

Q: Why did the researchers separate vision and policy training instead of joint training?

Separating vision and policy training reduces computational complexity and allows each model to specialize. In practice, training them separately and then combining them at run time performed better and was more computationally efficient than joint training, which did not yield the same benefits in their experiments.

Q: What challenges did the team encounter with data and hardware during development?

The team faced limited personnel and time, frequent hardware breakage, and difficulty achieving realistic friction and material interactions in the simulator. They addressed this by iterating on simpler subproblems, contracting a roboticist to repair the hand, and relying on domain randomization to bridge the reality gap.

Q: How is the training infrastructure described and what is its scale?

The training infrastructure is called rapid, sharing similarities with the system used for training Dota 2 bots. It leverages thousands of CPU cores and multiple GPUs to run many simulations in parallel, enabling the creation of diverse training data and faster iteration for both policy and vision models.

Q: What was the impact of attempting gloves on the hand for this task?

Trials with gloves did not improve performance, which was surprising to the researchers. This outcome underscored that simply adding physical constraints did not substitute for the robustness gained through domain randomization and large-scale simulated training.

Summary & Key Takeaways

• The project trains a five finger dexterous hand policy and a vision module separately before combining them for real-world execution, using a simple recurrent policy and a basic vision stack. The approach emphasizes keeping models simple while leveraging large-scale simulated data to achieve real-world transfer. The process iterates rapidly, improving from months to days per model to accelerate progress.

• Domain randomization is central, varying friction, colors, and other noise to avoid relying on an accurate world model. This makes the simulator a practical stand-in for real hardware, enabling robust control policies that survive real-world imperfections.

• The system uses a lightweight perceptual model and an actor-critic policy, with training distributed across many CPUs and GPUs. Despite simplicity, the approach achieves manipulation tasks in real hardware that prior robotics work struggled to solve, validating the method and encouraging broader use of simulation-first robotics research.