"1. Scale continues to be an important factor"

"One theme that has remained constant in deep learning over the past few years is the drive to create bigger neural networks."

"2. Unsupervised learning continues to deliver"

"There has been tremendous progress in this field, in recent years, especially in LLMs, which are mostly trained on large sets of raw data gathered from around the internet."

"For example, there were phenomenal advances in text-to-image models this year. Models like OpenAI’s DALL-E 2, Google’s Imagen, and Stability AI’s Stable Diffusion have displayed the power of unsupervised learning"

"Unlike older text-to-image models, which required well-annotated pairs of images and descriptions, these models use large datasets of loosely captioned images that already exist on the internet."

"The sheer size of their training datasets (which is only possible because there’s no need for manual labeling) and variability of the captioning schemes enables these models to find all kinds of intricate patterns between textual and visual information"

"3. Multimodality takes big strides"

"Text-to-image generators have another interesting characteristic: they combine multiple data types in a single model. Being able to process multiple modalities enables deep learning models to take on much more complicated tasks."

"Evidently, multimodality has played an important role in making deep learning systems more flexible."

"This was perhaps best displayed by DeepMind’s Gato, a deep learning model trained on a variety of data types, including images, text and proprioception data. Gato showed decent performance in multiple tasks, including image captioning, interactive dialogues, controlling a robotic arm and playing games."

"Despite the impressive achievements of deep learning, some of the field’s problems remain unsolved. Among them are causality, compositionality, common sense, reasoning, planning, intuitive physics, and abstraction and analogy-making."

"Likewise, text-to-image generators create stunning graphics but make basic mistakes when asked to draw images that require compositionality or have complex descriptions."

"For example, larger LLMs can maintain coherence and consistency over longer stretches of text. But they fail on tasks that require meticulous step-by-step reasoning and planning."