What Is Google Titans: AI Memory at Test Time?

TL;DR

Google's Titans architecture gives AI a human-like long-term memory and lets it keep learning after training, during use ('test time'). It scales beyond a 2-million-token context window with higher accuracy and, in benchmarks shown, beats Transformer and Mamba architectures. It remembers 'surprising' events that violate expectations, mimicking how the human brain prioritizes memorable moments.

Transcript

AI is evolving right now it's like a Pokemon evolving to the next stage or Goku turning super saiyan it's really exciting so you don't want to miss out on what's going on right now in this video I'm going to go over a few recent breakthroughs in AI if true this is going to be huge this is going to make AI way smarter than even the smartest models w... Read More

Key Insights

• Titans is a Google Research architecture that gives AI a human-like memory and lets models keep learning new things even after training, which the video claims makes them smarter than today's leading models.
• The Transformer, introduced in Google's 2017 'Attention Is All You Need' paper, underpins nearly all top models including GPT-4o, o1, Gemini, Claude, DeepSeek, and Llama, and excels at understanding context and relationships in data.
• Transformers have quadratic time and memory complexity with context length, making them accurate but slow and expensive when fed large amounts of data, and limited by a fixed context window.
• Context windows vary by model: GPT-4o has about 128,000 tokens, Claude has 200,000 tokens, and Google Gemini 2 has 2 million tokens, which can fit a 2-hour video, over 60,000 lines of code, or 1.4 million words.
• Titans can scale to context windows larger than 2 million tokens with higher accuracy, aiming to process inputs too large for current Transformers, such as entire novels, hundreds of documents, or an organization's full codebase.
• Test time (also called inference time) is when you use a model; normally model weights stay frozen after training, so most current AI does not learn or improve while being used.
• Titans changes memory during test time, meaning the model can continue learning after training and, in theory, get smarter as it is used, unlike standard frozen-weight models.
• Titans uses a 'surprise' metric inspired by human memory: it prioritizes storing events that violate expectations, and also retains important information following a surprising moment, not just the peak surprise itself.

Questions & Answers

Q: What is Google's Titans architecture?

Titans is an architecture from Google Research described in the paper 'Titans: Learning to Memorize at Test Time.' It gives AI models a human-like long-term memory and the ability to keep learning new things even after training. According to benchmark tables and charts shown in the video, Titans beats existing architectures like the Transformer and Mamba, as well as leading models such as GPT-4o, 4o mini, and Llama 3, which the video calls an insane improvement if true.

Q: Why are Transformer models limited by a context window?

The Transformer's output is conditioned on the direct dependencies of tokens in the current context window, which comes with quadratic time and memory complexity in terms of context length. This means there is a limit on how long a prompt can be. If a prompt is too long, it becomes too computationally expensive and inefficient to handle, which is why ChatGPT can return an error saying the submitted message was too long. That limit is known as the context window.

Q: How large are the context windows of GPT-4o, Claude, and Gemini 2?

According to the video, GPT-4o has a context window of about 128,000 tokens, Claude has a context window of 200,000 tokens, and the most recent Google Gemini 2 has a context window of 2 million tokens. The 2-million-token window is already huge and can fit roughly a 2-hour video, over 60,000 lines of code, and over 1.4 million words. Even so, the video notes it is still not enough for some real-world tasks.

Q: What does 'test time' mean in AI?

Test time, also called inference time, is the stage when you actually use the AI model, such as when you prompt ChatGPT. Your prompt is broken into tokens that flow through the neural network, and the model produces an answer. This contrasts with train time, the training stage where the model's dials and knobs, or weights, are adjusted. Normally, during test time these weights do not change, so the AI does not learn new things while being used.

Q: How does Titans let AI keep learning after training?

Most AI models freeze their weights after training, so during test time the dials and knobs do not change and the model does not really learn or improve. Titans is designed so that it can change these weights, or at least the model's memory, during test time. This is captured in the paper title 'Learning to Memorize at Test Time.' It means Titan models can continue learning new things even after training and, in theory, could get smarter and smarter as they are used.

Q: How does Titans decide what to remember?

Titans uses a long-term memory module with a learning process and a surprise metric inspired by human memory. Humans do not remember everything, only important things, and an event that violates expectations, meaning a surprising event, is more memorable. The video's example is a normal daily walk to work that is forgettable, versus being robbed or seeing a car crash, which the brain clearly remembers. Titans is designed to remember such surprising, abnormal events rather than routine ones.

Q: Why does Titans remember events after a surprising moment, not just the peak surprise?

Relying only on a surprise metric can miss important information that comes after a big surprising moment. Using the robbery example, the start of the event is most shocking, but you also want to remember who robbed you, where and how it happened, and how to avoid it next time. From a human memory perspective, an initial moment can be surprising enough to hold attention across a long time frame, leading to memorizing the entire event. Titans follows this same logic.

Q: What was the significance of the 2017 'Attention Is All You Need' paper?

Released by Google Research in 2017, 'Attention Is All You Need' is described in the video as the most iconic and influential AI research paper today, driving the AI explosion of recent years. It introduced the Transformer architecture used in nearly all top models, including GPT-4o, o1, Gemini, Claude, DeepSeek, and Llama. The Transformer is revolutionary because it understands context by focusing on relationships between different parts of the data, making it strong at language, writing, coding, and reasoning.

Summary & Key Takeaways

• The video breaks down Google Research's Titans paper, which gives AI models a human-like long-term memory and the ability to keep learning after training. Benchmark tables and charts shown in the video indicate Titans outperforms existing architectures like the Transformer and Mamba, and leading models such as GPT-4o and Llama 3.

• Today's top models derive from the 2017 Transformer architecture, which is excellent at context but has quadratic time and memory cost with context length. This creates a fixed context window limit, so very long prompts fail, and even Gemini 2's 2-million-token window is not enough for some real-world tasks.

• Titans addresses these limits by scaling past 2 million tokens with higher accuracy and by updating memory at test time. Its long-term memory module uses a surprise metric inspired by the human brain, remembering events that violate expectations along with the important details that follow them.