How Do AI Agents Learn Through Video Games?

TL;DR

AI agents are entities that act in an environment, changing it and learning from the consequences of their actions. Video games provide varied, scalable virtual environments where agents can practice autonomy, while language models can contribute conceptual knowledge and planning that is connected to actions in virtual spaces or through physical robots.

Transcript

HANNAH FRY: Welcome back to "Google DeepMind-- The Podcast," with me, your host, Professor Hannah Fry. [MUSIC PLAYING] I think it's fair to say that generative AI, as we have it now, is a bit of a mismatch for our science fiction fantasies about what technology might become. But it is worth remembering that this isn't the only possibility. There ar... Read More

Key Insights

• An agent is an entity that can act in an environment. The environment provides observations, and the agent’s actions change that environment, allowing the system to encounter and potentially understand the consequences of what it does.
• Agency is the ability to take actions, while autonomy is the ability to act independently toward a task. An agent may possess agency because it follows programmed instructions without having the higher autonomy needed to explore, adapt, or pursue goals with limited input.
• Autonomy is a spectrum rather than a single capability. A self-driving car can make navigation decisions with some autonomy, while less autonomous agents need substantial user direction, and a more autonomous agent could explore an environment and learn new things with minimal instruction.
• High-level tasks require many short-horizon subtasks. Picking up an object or placing it on a table is a basic mechanical action, while preparing lunch requires planning, coordinating intermediate steps, solving multiple problems, and potentially deciding what the user would want.
• General agents are considered important for reaching artificial general intelligence. The stated goal is to create systems that adapt to new situations, reason broadly, and perform many tasks, potentially supporting scientific progress, driving, household assistance, and online shopping.
• Agents can operate in physical or virtual environments. Robots act through physical bodies, game agents act inside simulated spaces, and software agents can take nonphysical actions such as writing Python code, compiling functions, building software, or looking for information online.
• Chatbots are comparatively passive agents because their typical output is natural language. Their responses can influence what a user asks next, but they are generally not trained through repeated environmental action, failure analysis, and new attempts in the same way as interactive agents.
• Video games are useful proving grounds because they offer rich, varied experiences across many different scenarios and environments. Their virtual nature also makes them easier to scale, allowing researchers to create multiple instances for training and evaluating agent performance.

Questions & Answers

Q: What is an AI agent and how does it interact with an environment?

An AI agent is an entity that can take actions within an environment. The environment supplies observations that inform the agent, and the agent’s actions can change what happens next. This interaction is important because it exposes the agent to the consequences of its decisions. The concept is broad enough to include robots, autopilot systems, game-playing systems, chatbots, and software-producing agents.

Q: What is the difference between agency and autonomy in AI?

Agency is the capacity to take actions, including actions that a system was explicitly programmed to perform. Autonomy is the ability to act independently in pursuit of a task. An autopilot can therefore count as an agent even when its logic is handcrafted, while a more autonomous system makes more decisions by itself, explores its environment, or learns with less ongoing direction from a person.

Q: Why does autonomy exist on a spectrum for AI agents?

Autonomy exists on a spectrum because agents differ in how much human input they require and how independently they can pursue objectives. A self-driving car may choose how to navigate roads, giving it some autonomy. Other systems require frequent instructions. At the more autonomous end, an agent could explore the world, discover new information, and learn without needing detailed guidance for every action.

Q: What is the difference between short-horizon and high-level tasks?

Short-horizon tasks are basic actions with limited scope, such as grabbing an object and placing it on a table. High-level tasks require many connected steps and successful completion of multiple subtasks. Preparing lunch, for example, involves more than manipulating objects. It can also require planning the sequence of work and determining what the person wants before carrying out the necessary actions.

Q: How could general AI agents contribute to artificial general intelligence?

General agents are described as a key path toward artificial general intelligence because they must act within environments, adapt to unfamiliar situations, and reason across different tasks. If systems reach human-level generality, they could support scientific progress and assist with activities such as driving, household work, and online shopping. The discussion also suggests that additional applications may only become clear as the technology develops.

Q: How are physical agents different from virtual agents?

Physical agents are embodied systems, such as robots, that observe and act in the real world. Virtual agents operate in abstract or simulated environments, including video games. An agent does not need a physical or spatial body, however. A software agent can still qualify by taking consequential actions such as writing Python code, compiling a function, building software, or looking for information online.

Q: How do AI agents differ from language-model chatbots?

Language-model chatbots primarily produce natural-language responses, sometimes supplemented by specific functions such as internet search. Their output affects the user and may shape the next prompt, so they can be viewed as limited agents. However, they are generally trained from human-generated internet data rather than by acting repeatedly in an environment, failing, identifying why an attempt failed, and trying again based on those consequences.

Q: Why are video games useful for training AI agents?

Video games provide rich and varied experiences in neatly constrained virtual environments. Different games present different scenarios, rules, objectives, and possible actions, making them useful for testing whether agents can develop broad capabilities and approach human performance. Because games are virtual, researchers can also scale training more easily by creating many environment instances in which agents can act and learn.

Summary & Key Takeaways

• An agent is defined broadly as an entity capable of acting within an environment. The environment supplies observations and responds to the agent’s actions, while those actions alter subsequent conditions. Humans, robots, autopilot systems, chatbots, and software-producing systems can therefore qualify as agents, although their capabilities and degrees of autonomy differ significantly.

• Agency means having the capacity to take actions, while autonomy describes acting independently to accomplish a task. Autonomy exists on a spectrum, from systems that require frequent human input to agents that explore and learn with little instruction. Long-horizon goals, such as preparing lunch, also require coordinating many simpler subtasks successfully.

• General agents are presented as a route toward artificial general intelligence because they must adapt, reason, and act across unfamiliar situations. Games offer rich, varied, and scalable virtual environments for developing these abilities. Language models can add broad conceptual knowledge and planning, while additional training connects those capabilities to robotic or virtual actions.