How Do Gemini-Powered Robots Think and Act?

TL;DR

Gemini-powered robots combine vision, language, and physical actions to interpret instructions, plan movements, and adapt to unfamiliar objects and scenes. Google DeepMind demonstrates this through precise lunch packing and spoken commands involving previously unseen items, while longer-horizon agents can connect small actions into useful tasks instead of requiring separate instructions for every step.

Transcript

Welcome to Google DeepMind, the podcast with me, your host, Hannah Fry. Now, you might remember that earlier this year, I got to sit down with Carolina Parada, who is the Head of Robotics at Google DeepMind, and she was talking all about taking Gemini's multi-modal reasoning and putting it, embedding it into a physical body. And since we were comin... Read More

Key Insights

• Modern robotic generalization is built on large vision-language models that already understand many general concepts about the world. This foundation helps robots respond more flexibly to unfamiliar scenes, visual conditions, and natural-language instructions than the systems shown in the laboratory four years earlier.
• Vision-language-action models place physical actions on the same footing as vision and language tokens. By modeling these elements as sequences, a robot can determine which actions are appropriate when it receives a new instruction or encounters a situation that differs from its training examples.
• Action generalization is the ability to infer a suitable sequence of physical movements in a new situation. Google DeepMind reports major improvements in this area, moving beyond short tasks such as picking, placing, or unzipping toward systems that can coordinate multiple smaller behaviors.
• Long-horizon robotic tasks are created by orchestrating shorter learned actions. An agent can interpret a high-level goal, determine the necessary intermediate steps, and connect them, reducing the need for a person to issue a separate instruction for every individual movement.
• The thinking component improves robotic performance by having the model produce thoughts about an intended action before executing it. The team describes this as applying a reasoning principle used with language models to physical manipulation, where even basic actions can be difficult for robots.
• Dexterous lunch packing requires millimeter-level precision and careful force control. The demonstrated robot visually handled a zip-lock bag, bread, chocolate, and grapes, showing that successful manipulation depends on accurate positioning as well as avoiding excessive pressure on delicate objects.
• Teleoperation supplies demonstrations for learning physical tasks. A human effectively embodies the robot and performs the target activity through it, giving the model examples of what successful execution looks like from the robot's perspective instead of relying only on autonomous trial and error.
• Open-ended generalization allows a robot to manipulate objects it has not previously seen. After receiving spoken instructions, the demonstrated system opened a small green container, placed an unfamiliar pink stress ball inside, and attempted to return the lid using a general robotic policy.

Questions & Answers

Q: What is a vision-language-action model in robotics?

A vision-language-action model combines visual information, language instructions, and physical robot actions within the same modeling framework. Google DeepMind places action sequences on the same footing as vision and language tokens, enabling the system to infer what movements should follow from a new scene or instruction. This supports action generalization rather than limiting the robot to one rigidly preprogrammed routine.

Q: How do Gemini-powered robots understand spoken instructions?

The demonstrated robot runs a general policy with a Gemini layer placed on top, allowing a person to speak ordinary instructions to it. The system connects language with its visual understanding of the scene and its available physical actions. It can then identify relevant objects, describe what it is doing, and carry out requests such as moving colored blocks into specified trays.

Q: How do robots generalize to objects they have never seen?

The robots build on large vision-language models that understand general concepts and can transfer that understanding to robotics. In the laboratory, a robot encountered a small container and a travel stress ball that it had not seen before. It responded to spoken requests by opening the container, placing the unfamiliar soft object inside, and attempting to replace the lid.

Q: Why does a robot think before taking an action?

Thinking before acting helps the robot consider the movement it is about to perform instead of immediately executing it. In Gemini Robotics 1.5, the system outputs thoughts and then takes the physical action. The team says this process makes the robot more general and improves performance, which is valuable because manipulation tasks that humans perform intuitively remain difficult for robots.

Q: How are long-horizon robotic tasks performed?

Long-horizon tasks are performed by using an agent to coordinate a sequence of smaller robotic actions. Instead of requiring a person to provide one instruction after another, the system can receive a high-level request and determine the intermediate steps. The example discussed is packing luggage for London by checking the weather, deciding what is needed, and then packing the bag.

Q: How does Google DeepMind train robots for dexterous tasks?

The lunch-packing demonstration was trained through teleoperation. A human embodies or controls the robot and performs the task using the robot, providing examples of correct behavior from its perspective. The robot then learns the relationship between visual input and physical actions end to end, including delicate operations such as handling a zip-lock bag and positioning bread inside it.

Q: Why is packing a lunch difficult for a robot?

Packing a lunch requires a long sequence of precise manipulations involving objects with different shapes and physical properties. The robot must grasp a zip-lock bag with millimeter-level precision, place bread into a small space, avoid crushing it too hard, add other food, and close the bag. The demonstration emphasizes both dexterity and careful control of position and force.

Q: What changed in Google DeepMind robotics over four years?

The newer systems rely on more robust visual backbones and large multimodal models, so they are less dependent on controlled lighting, fixed backgrounds, or privacy screens around each setup. Their visual and instructional generalization has improved, and robotics models now build on broad vision-language understanding. The laboratory therefore focuses increasingly on general, open-ended behavior rather than narrowly repeated routines.

Summary & Key Takeaways

• Google DeepMind builds its newer robotics systems on large vision-language models that understand general human concepts. Vision-language-action models extend this foundation by treating physical actions alongside vision and language tokens, allowing robots to infer appropriate action sequences when they encounter new instructions, scenes, or objects.

• The laboratory demonstrations separate two major capabilities. A lunch-packing task highlights dexterity through delicate manipulation of a zip-lock bag, bread, chocolate, and grapes. Another robot highlights generalization by following spoken instructions and manipulating a small container and stress ball that it had never encountered before.

• Gemini Robotics 1.5 includes agent and thinking components. The agent can connect short actions into longer tasks, while the thinking component makes a robot consider an action before executing it. Together, these capabilities aim to make robots more adaptable and useful for complete, open-ended human requests.