What Is Reinforcement Learning? DeepMind x UCL RL Lecture Series [1/13]

TL;DR

Reinforcement learning is learning to make decisions that achieve goals through interaction with an environment. It differs from passive learning because an agent’s actions shape later experiences and may make future choices possible or impossible. The DeepMind x UCL lecture also connects reinforcement learning to artificial intelligence, where machines find solutions after people specify a problem and goal. Read on to understand its active, sequential, and autonomous nature.

Transcript

hello and welcome to this course on reinforcement learning my name is harvan husselt and i'm a research scientist at deepmind in london and every year we teach this course on reinforcement training at ucl this year it's a little bit different because due to the pandemic situation with covet 19 we are pre-recording the lectures so instead of talking... Read More

Key Insights

• Reinforcement learning is concerned with learning how to make decisions that achieve goals. Learning, autonomous decision-making, and explicit goals are therefore the central concepts used to connect reinforcement learning with the lecture’s working definition of artificial intelligence.
• The industrial revolution automated repeated physical or manual solutions. Machines such as steam trains and steamboats replaced some work previously performed through human or animal labor, producing large productivity increases and enabling solutions that had not previously been practical.
• The digital revolution automated repeated mental solutions. Calculators illustrate this transition because arithmetic procedures can be described precisely, implemented as programs, and executed more quickly and precisely than slower mental calculation.
• Artificial intelligence can be viewed as allowing machines to find solutions themselves. Under this view, people specify a problem and a goal, while a learning system uses data and interaction to determine how the goal can be achieved.
• Learning may be easier to program than a complete collection of adult capabilities. Alan Turing’s child-machine idea suggests starting with limited initial mechanisms and developing more advanced capabilities through education and other experience.
• Reinforcement learning is active rather than passive. The learner does not merely receive experience that is entirely outside its control, because its chosen actions can influence the data and subsequent experiences available to it.
• Reinforcement learning interactions can be sequential. An early action may alter the world, making later actions possible or impossible, so a decision must be understood partly through its effects on future interactions.
• Goal-directed learning does not require examples of optimal behavior for every action. People and animals can acquire useful behavior through interaction, even when nobody specifies the exact sequence of actions needed to perform a task.

Questions & Answers

Q: What is reinforcement learning and how does it work?

Reinforcement learning is the problem of learning to make decisions that achieve goals through interaction with an environment. The learner chooses actions that influence later experiences, and earlier decisions may change which future actions are possible. Useful behavior can therefore emerge through goal-directed interaction without exact examples of the optimal action in every situation.

Q: How is reinforcement learning related to artificial intelligence?

The lecture describes artificial intelligence as allowing machines to find solutions themselves. People specify a problem and a goal, while the machine uses data and interaction to learn how to achieve that goal. Learning, autonomous decision-making, and goals are therefore central to both artificial intelligence and reinforcement learning.

Q: Why is reinforcement learning considered active learning?

Reinforcement learning is active because the learner’s actions influence the experience it receives. Its choices can change the surrounding world and determine which observations or actions become available later. The learner therefore participates in producing the data from which it learns.

Q: Why are sequential decisions important in reinforcement learning?

A decision can have consequences beyond its immediate result. An early action may alter the world, making later actions possible or impossible. Reinforcement learning must therefore consider how present behavior affects future interactions and progress toward the goal.

Q: Can reinforcement learning work without examples of optimal behavior?

Yes, useful behavior can be learned without receiving exact examples of the optimal action in every situation. People and animals demonstrate that goal-directed skills can develop through interaction and experience. The learner can discover a workable sequence of decisions rather than copying a fully specified solution.

Q: How does artificial intelligence differ from earlier forms of automation?

The industrial revolution automated repeated physical or manual solutions, while the digital revolution automated repeated mental procedures such as arithmetic. Both depended on people discovering the solution before encoding or mechanizing it. The lecture presents artificial intelligence as the next step: machines learn to find solutions after people specify the problem and goal.

Q: What does Alan Turing’s child-machine idea suggest about learning?

Alan Turing’s child-machine idea suggests beginning with limited initial mechanisms and developing more advanced capabilities through education and experience. The lecture uses this idea to explain why programming a system that can learn may be easier than directly specifying a complete collection of adult capabilities.

Q: What does the DeepMind x UCL reinforcement learning course cover?

The course examines reinforcement learning concepts and the different algorithms that can be built. This first lecture introduces reinforcement learning and explains its relationship to artificial intelligence. Other lectures in the series are taught by Harvan Husselt, Diana Bursa, and Matteo Hessel.

Summary & Key Takeaways

• Reinforcement learning addresses how a system can learn to make decisions that achieve goals. The lecture connects this problem to artificial intelligence, defined here as building machines that find solutions themselves instead of requiring people to discover each solution first and then encode it as a program.

• Earlier waves of automation replaced repeated physical and mental solutions, but both depended on people determining the solutions beforehand. Artificial intelligence could extend automation by letting machines learn solutions after humans specify a problem and goal, using data and interaction as part of the learning process.

• Learning through interaction is active, sequential, and goal directed. An agent’s actions can affect the experience it receives and change which future actions become possible. People and animals also demonstrate that behavior can be learned without being given exact examples of the optimal action in every situation.