Big neural networks: Does size matter? | Oriol Vinyals and Lex Fridman

TL;DR

Size matters because some neural-network capabilities appear only after a scale threshold, but expanding an existing model while reusing its trained weights is extraordinarily hard. Flamingo demonstrates a modular alternative: it froze Chinchilla’s 70 billion parameters and added 10 billion parameters for vision-language tasks. Read on to see how this 80-billion-parameter system gained image-dialogue and few-shot abilities without retraining its largest component.

Transcript

you mentioned early on like Psy it's hard to grow what did you mean by that because we're talking about scale might change uh there might be and we'll talk about this too like there's a emergent there's certain things about these neural networks that are emerging so certain like performance we can see only with scale and there's some kind of thresh... Read More

Key Insights

• 💗 Growing neural networks with a large number of parameters is challenging, but modularity offers a solution.
• 📰 Modularity allows for the reuse of pre-trained components and the addition of new capabilities, facilitating scalability.
• 😑 The Flamingo model showcases modular growth by combining a pre-trained language model with a vision capability.
• 👻 Modularity in neural networks is similar to modular software engineering, allowing for the building of increasingly complex models.

Questions & Answers

Q: Does neural network size matter?

Yes. Oriol Vinyals says certain performance and emerging capabilities can appear only with scale, potentially after a threshold. The challenge is not merely training a larger network, but expanding an existing one while preserving and reusing the valuable weights already built.

Q: Why is it hard to grow an existing neural network?

Retraining a network at a larger size is possible, but reusing its existing weights while expanding it into a larger model is extraordinarily hard. Those weights represent substantial work and already provide a strong initial brain for the tasks researchers care about.

Q: How can modularity help neural networks grow?

Modularity lets researchers freeze a trained component and attach smaller networks that add a new capability. This approach can reuse existing weights instead of rebuilding the whole system from scratch, although research is still needed to add capabilities without destroying others.

Q: How was Flamingo built from Chinchilla?

Researchers took Chinchilla, a language-only model trained to predict the next word, and froze its weights. They then inserted small neural-network components at selected places and trained the combined system with datasets connecting vision and language.

Q: How many parameters do Chinchilla and Flamingo have?

Chinchilla has 70 billion parameters. The added components contribute another 10 billion, making the largest released Flamingo model a total of 80 billion parameters.

Q: What new capability was added to Chinchilla in Flamingo?

Flamingo added the ability to see to Chinchilla’s language capabilities. Its input can include images and text, and its output is text, allowing a user to upload an image and conduct a dialogue about it.

Q: Were all parts of Flamingo trained from scratch?

No. The main and largest portion, Chinchilla, was frozen, while some added parameters were trained from scratch. Other components were initialized from self-supervised learning in a model that already understood vision.

Q: What abilities emerged after Flamingo was trained?

Flamingo showed few-shot learning and could be taught a new vision task through prompting. Vinyals says researchers also observed abilities they had not measured during development, including discussing the subtle joke in an image, though he describes that example as anecdotal rather than proof that the broader problem was solved.

Summary & Key Takeaways

• Growing a neural network like the meow network is challenging due to the large number of parameters involved.

• Modularity in neural networks allows for the addition of new capabilities without starting from scratch, as seen in the Flamingo model.

• Flamingo, a chatbot model, combines language and vision capabilities by building on top of a pre-trained language model and adding a small sub-network for vision processing.