How Does Backpropagation Power Deep Learning?

TL;DR

Backpropagation trains neural networks by computing, for every weight at once, how a small change would affect the output, making it about a million times more efficient than tinkering with weights one at a time. Once given large labeled datasets and compute power, it drove speech recognition past Siri by 2012 and cut image-recognition error from 25% to under 3%.

Transcript

[Applause] I'm going to talk about some basic share and I imagine the some people in the audience who don't really have a good grip of what the backpropagation algorithm is so I'm actually going to explain that very briefly so you know what we're talking about and now I'm sure a few examples of what it can do and these are all things that are now a... Read More

Key Insights

• Neural networks learn a different way than traditional programming: instead of writing step-by-step instructions, you build a network with a learning algorithm and show it examples, letting it solve problems like turning image pixels into descriptive words that people failed to program for 50 years.
• An artificial neuron takes values on its input lines, multiplies each by a weight that can be positive or negative, sums them into a total input, then outputs a nonlinear function that stays silent until the total input passes a threshold and grows as the input grows.
• The naive learning algorithm resembles evolution: change one weight, run a batch of training examples, keep the change if the output improves and discard it if not. It works but is inefficient because it requires running examples twice for every one of possibly a billion weights.
• Backpropagation uses calculus instead of physical tinkering: because the network knows its own weights, it can compute how a small change to each weight would affect the output and adjust all weights in parallel, making it roughly a million times more efficient with a million weights.
• Machine learning largely abandoned backpropagation by the 1990s because datasets were small, other algorithms performed better and were provable, and backpropagation gave different answers on different runs with no guarantee it would work.
• Speech recognition networks broke statistics 101 rules by using around four million parameters between fully connected layers with only a few million training examples, yet worked; the brain similarly has about 10,000 synapses per fixation over roughly 10 billion lifetime fixations.
• By 2012 Google deployed neural networks in Android speech recognition, producing a large performance jump that surpassed Siri, and modern systems now train end-to-end on data alone with backpropagation rather than hand-built front ends.
• In the 2012 ImageNet competition, conventional computer vision systems had plateaued around 25% error while the neural network reached almost half that; by 2015 systems hit human-level performance and now sit below 3% error, a tenth of the older approach.

Questions & Answers

Q: What is backpropagation and how does it work?

Backpropagation is an algorithm for training neural networks efficiently using calculus. Because the network already knows all its own weights, it does not need to physically tinker with each weight and observe the effect. Instead it looks at the discrepancy between the desired output and the actual output, then sends information backwards through the network to compute, for every weight, how a small increase would improve or worsen the output. It adjusts all weights in parallel in the same time the naive method would take to test just one.

Q: Why is backpropagation more efficient than the naive learning method?

The naive method changes one weight, runs a batch of training examples through the network, checks whether the output improved, and keeps or discards the change. This requires running examples twice for each weight, and a network might have a billion weights, so it is enormously slow. Backpropagation instead computes what to do with all weights at once in the same amount of time the naive method figures out a single weight. With a million weights, that makes it about a million times more efficient, which is enough to make a real difference.

Q: How does an artificial neuron work?

An artificial neuron has input lines coming from sensors or other neurons, and each input line carries a weight that can be positive or negative. The neuron takes the value on each input line, multiplies it by that line's weight, and adds everything up to get a total input. It then produces an output that is a nonlinear function of that total input: if the total input is not big enough, the neuron stays silent, but once the total input passes a threshold it starts responding, and the response grows as the total input grows. Learning happens by changing the strengths of the weights.

Q: Why did machine learning researchers abandon backpropagation in the 1990s?

By the 1990s, people in machine learning had given up on backpropagation for several reasons. Their datasets were relatively small, and other algorithms worked better on those small datasets. Those competing algorithms also let researchers prove things about them, whereas with backpropagation you could not prove it would work. Additionally, when different people ran backpropagation they got different answers, so anyone obsessed with getting one provably correct answer found it unsuitable. The naive version also did not work well in deep networks or recurrent networks.

Q: How did neural networks change speech recognition?

Graduate students at the University of Toronto applied a neural network to speech recognition, feeding coefficients describing the sound wave through multiple layers of hidden units to predict which phoneme was being spoken from a spectrogram. These probabilistic answers were strung together to find plausible utterances. It worked just a little bit better than existing systems, but experts realized this small edge from a few months of work would soon wipe out the state of the art. By 2012 Google used it in Android, producing a big jump that beat Siri, and modern systems train end-to-end.

Q: What happened in the 2012 ImageNet competition?

Two of Hinton's graduate students, Ilya Sutskever and Alex Krizhevsky, entered the ImageNet competition, which had a million images with a thousand examof each class and a secret test set so competitors could not cheat. Their neural network performed so much better than everyone else that the person running the test set did not believe the results and had to run it three times before accepting them. Conventional computer vision systems had plateaued at about 25% error, while the neural network reached almost half that rate.

Q: How do neural networks defy the rules of statistics 101?

The speech recognition network used about four million parameters between each pair of fully connected layers but had only a few million training examples. Statistics 101 says this cannot possibly work because there are far more parameters than training examples. Hinton counters that the brain violates the same rule: over a lifetime you make about 10 billion fixations, yet you have roughly 10,000 times more synapses than that, giving about 10,000 synapses for each fixation. So the brain also fails the statistics 101 test, yet both clearly work in practice.

Q: How much has image recognition error improved with deep learning?

Before neural networks, conventional computer vision systems had plateaued at about a 25% error rate on the ImageNet dataset. In 2012 the neural network system cut that to almost half. As thousands of graduate students and experienced developers adopted the approach, performance kept improving. By 2015 systems reached human levels on that dataset, where a person named Andrej Karpathy did the task himself and got 5% error. Systems are now below 3% error, roughly a tenth of the error rate of the previous computer vision systems.

Summary & Key Takeaways

• Traditional programming requires spelling out exactly how to solve a task, but neural networks flip this: you build a network of artificial neurons with adjustable weights and a learning algorithm, then show examples. This handles problems like describing image contents that resisted 50 years of hand-coded programming attempts.

• Learning can be done naively by changing one weight at a time and keeping changes that improve output, an evolution-like process that works but is extremely slow. Backpropagation instead uses calculus to compute the effect of adjusting every weight simultaneously, making it about a million times more efficient.

• Despite being abandoned in the 1990s for small datasets and unprovable behavior, backpropagation now works amazingly well given large labeled data and compute. It powered speech recognition past Siri by 2012 and cut ImageNet image-recognition error from 25% to under 3%, reaching human levels by 2015.