How Can AI Learn to Cooperate With Humans?

TL;DR

AI can learn cooperation through reinforcement learning that rewards actions benefiting both the agent and others. Communication helps agents understand different needs, but effective cooperation also requires inferring hidden preferences, balancing self-interest with collective welfare, and accounting for possible deception. These capabilities could help machines coordinate with people in areas such as transportation and environmental protection.

Transcript

welcome back to deepmind the podcast i'm hannah fry a mathematician who's been following the remarkable progress in artificial intelligence in recent years in this series i'm talking to scientists researchers and engineers about how their latest work is changing our world deepmind's big goal is solving intelligence and over the next few episodes we... Read More

Key Insights

• Cooperation is a central component of human intelligence because civilization, scientific achievements, engineering projects, art, literature, and vaccine creation depend on many people coordinating their efforts rather than acting as isolated individuals.
• Communication is a booster for cooperation because participants must understand both their own needs and the needs of others before they can identify actions that improve their joint welfare.
• Cooperative AI is the effort to help humans and machines find ways to improve their joint welfare, and its researchers view successful cooperation as a crucial milestone toward artificial general intelligence.
• Reinforcement learning works by giving an agent numerical rewards for desired outcomes, much like positive reinforcement shapes animal behavior, and the agent learns actions that maximize its accumulated reward.
• Mixed-motive scenarios are interactions in which participants’ incentives are neither completely aligned nor entirely opposed, creating opportunities for agents to find solutions that provide shared benefits.
• Social value orientation is the tendency to incorporate another participant’s reward into one’s own decisions, allowing an agent to change future behavior based on how its actions affect someone else.
• An agent’s balance between selfishness and altruism is partly controlled by its reward design, which can prioritize its own welfare, combine multiple participants’ rewards, or focus almost entirely on another entity’s welfare.
• Inferring another participant’s preferences is a major challenge because rewards are not normally visible, expressions can be misleading, and negotiators may hide or misrepresent their actual preferences to influence an outcome.

Questions & Answers

Q: What is cooperative AI?

Cooperative AI is an area of research focused on helping humans and machines find ways to improve their joint welfare. Its goal is not merely to make an isolated agent perform well, but to enable agents to understand other participants, coordinate their behavior, and discover beneficial solutions when individual incentives may be partly aligned and partly conflicting.

Q: Why is cooperation important for artificial intelligence?

Cooperation is important because the highest level of intelligence discussed in the episode is human intelligence, and humans are unusually capable cooperators. Major accomplishments in science, art, engineering, literature, transportation, and vaccine development arise from groups working toward common goals. Researchers therefore argue that machines seeking comparable intelligence will also need the ability to cooperate.

Q: How does communication help AI agents cooperate?

Communication helps cooperation by allowing participants to convey their needs, desires, and intentions. An agent cannot coordinate effectively by considering only its own objective. It must also develop an understanding of what other people or agents want. Communication supports that understanding, although communicated preferences may still be incomplete, inaccurate, or deliberately deceptive.

Q: How does reinforcement learning teach an agent to cooperate?

Reinforcement learning assigns numerical rewards to actions and outcomes, and an agent learns to maximize the points it receives. Cooperative behavior can be encouraged by rewarding an agent not only for its own success but also when its actions benefit others. This expands the objective from individual performance toward outcomes that improve welfare across multiple participants.

Q: What is social value orientation in cooperative AI?

Social value orientation describes how much a participant incorporates another person’s welfare into its own decisions. For an artificial agent, designers can expose it to rewards received by other agents and include those rewards in its objective. The agent can then modify future actions according to both its personal outcome and the effects experienced by others.

Q: How can designers adjust an AI agent’s selfishness?

Designers can adjust selfishness through the reward function. At one end, an agent can focus only on its own reward. At the other, it can care almost entirely about another person’s or agent’s well-being, producing highly altruistic or self-sacrificial behavior. Intermediate settings combine personal and shared rewards, creating different balances between individual and collective interests.

Q: Why is inferring another agent’s preferences difficult?

Preferences are difficult to infer because people and agents do not necessarily reveal their rewards directly. Facial expressions and statements may offer clues, but they can also be misleading. Someone might claim to enjoy an outcome to avoid hurting another person’s feelings, causing both participants to repeat a choice that neither of them actually wants.

Q: How can deception interfere with cooperative AI?

Deception interferes with cooperation when a participant conceals or misrepresents its true preferences to influence another participant’s actions. This is especially relevant in negotiations, where someone may pretend to dislike an acceptable option to steer the group toward a preferred alternative. Cooperative agents must therefore infer rewards while accounting for strategically unreliable signals.

Summary & Key Takeaways

• Cooperative AI seeks to help humans and machines improve their joint welfare. DeepMind researchers argue that cooperation is essential to advanced intelligence because humanity’s greatest achievements usually depend on coordinated groups. A machine approaching human-level intelligence may therefore need to understand other participants, communicate effectively, and work toward shared goals.

• Reinforcement learning trains agents by assigning numerical rewards to actions and outcomes. Earlier systems focused on single-agent tasks or zero-sum games, but cooperative research examines mixed-motive situations where interests are only partly aligned. Agents can develop cooperative behavior when their rewards include benefits received by nearby agents or people.

• Social value orientation represents the degree to which another participant’s welfare affects an agent’s decisions. Designers can adjust this from complete selfishness to strong altruism. However, real preferences are not always directly observable, so agents must infer them from behavior and communication while recognizing that people or agents may conceal or misrepresent their desires.