What Are Google's Titans and MIRAS Memory Models?

TL;DR

Google's Titans blends sharp short-term window attention with a long-term memory module that updates while the model runs, storing information based on surprise. It handles context windows over 2 million tokens, hit over 95% accuracy on needle-in-a-haystack at 16,000 tokens, and beat GPT-4, Llama 3.1 70B, and retrieval-augmented models on the BABILong benchmark.

Transcript

All right, so Google has been on a wild streak lately and the past couple of days dropped update after update across completely different parts of the AI ecosystem. They introduced a new long-term memory system for huge context windows, prepared a cheaper nano banana 2 flash model that performs close to the pro version, quietly tested AI generated ... Read More

Key Insights

• Titans is Google's attempt to blend two worlds: it keeps sharp, precise short-term memory through window attention and adds a separate long-term memory module that actually updates while the model is running, unlike most models that don't genuinely learn during use.
• Titans decides what to store based on surprise. The more unexpected something is, the more likely it gets saved, and it forgets in a smart way instead of just wiping old information indiscriminately.
• Google built three types of long-term memory for Titans: memory as context, memory as gate, and memory as layer. The MAC version in particular dominates long-sequence tests with context windows over 2 million tokens.
• Transformers fall apart on super long inputs because compute cost shoots up as context grows, while modern RNNs and state space models gain speed but forget too much by squeezing history into one tiny state.
• A 760 million parameter Titans model handled the needle-in-a-haystack test with more than 95% accuracy at 16,000 tokens and beat GPT-4, recurrent Gemma 9B, Llama 3.1 70B, and even Llama 3 paired with retrieval tools on the BABILong benchmark.
• MIRAS is a unifying theory for sequence models, showing that Transformers, Mamba, RetNet, RWKV, and DeltaNet are all versions of the same idea: an internal memory system that stores and retrieves information with different rules.
• MIRAS breaks memory design into four core questions: what shape the memory is, how the model decides what to store, how fast new info pushes out old entries, and how memory gets updated over time. It produced attention-free models Moneta, Yaad, and Memora.
• The shift points toward models that learn continuously during use rather than frozen, one-time pre-trained models, echoing Ilya Sutskever's goal of models that learn the way a talented teenager learns: continuously and actively, not through mountains of static training data.

Questions & Answers

Q: What is Google's Titans memory system?

Titans is a Google research architecture that blends short-term and long-term memory to overcome transformer limitations on long inputs. It keeps short-term memory through window attention, which is sharp and precise, and adds a separate long-term memory module that updates while the model is running. This means it genuinely learns during use, unlike most models today. It stores information based on surprise, saving the more unexpected content, and forgets in a smart way instead of just wiping old info.

Q: Why do transformers struggle with long inputs?

Transformers fall apart on super long inputs because, even with all the fancy optimization tricks, the compute cost shoots up like crazy as the context gets longer. Once you push past a few hundred thousand tokens, things stop working the way they should. The alternative extreme, using a modern RNN or state space model, gains speed but loses detail because the whole history gets squeezed into one tiny state, making it efficient but forgetful. Titans tries to blend both worlds.

Q: How well does Titans perform on benchmarks?

Titans performs exceptionally on long-sequence tests. The MAC version handles context windows over 2 million tokens, and a 760 million parameter model punches well above its weight. It handled the needle-in-a-haystack test with more than 95% accuracy at 16,000 tokens and dominated the BABILong benchmark, where models must connect facts scattered across huge documents. Titans beat GPT-4, recurrent Gemma 9B, Llama 3.1 70B, and even Llama 3 paired with retrieval tools.

Q: What is MIRAS and how does it relate to sequence models?

MIRAS is Google's unifying theory for sequence models. Zooming out, Transformers, Mamba, RetNet, RWKV, and DeltaNet are all versions of the same idea: an internal memory system that stores and retrieves information with different rules. MIRAS breaks that idea into four core questions: what shape the memory is, how the model decides what is worth storing, how quickly new info pushes out old entries, and how memory gets updated over time. It is a roadmap, not a single model.

Q: What is Lux, the computer-use model?

Lux is a computer-use model from the OpenAGI Foundation that feels less like a research toy and more like infrastructure. It is not a chat model with a browser plugin; it actually looks at the screen, reads the UI, and outputs clicks, scrolls, and key presses. It is meant to operate full desktops, browsers, spreadsheets, editors, and email clients, handling real user interfaces instead of depending on APIs. On the online Mind2Web benchmark of over 300 real tasks, Lux scores 83.6.

Q: How does Lux compare to other agent models?

On the online Mind2Web benchmark, covering over 300 real tasks pulled from actual websites, Lux scores 83.6, while Gemini CUA sits at 69.0, OpenAI Operator at 61.3, and Claude Sonnet 4 at 61.0. Mind2Web is brutally unforgiving because every task depends on visual context, shifting layouts, random UI behaviors, and inconsistent design choices. Lux also ends up around 10 times cheaper per token than OpenAI Operator, changing the economics of long multi-step automation.

Q: What is Nano Banana 2 Flash?

Nano Banana 2 Flash is a new image model Google appears to be preparing, keeping the food naming theme. The pro version was internally known as ketchup, and references to mayo started appearing in code. Early access signals suggest the Flash version performs almost identically to the Pro model but at much lower operational cost. Google uses Pro models for premium performance and deploys Flash versions for high-volume scenarios. The public announcement is probably coming sometime in December.

Q: Why did Google's headline rewriting experiment upset newsrooms?

Google quietly rolled out an experiment in Google Discover that rewrote publishers' headlines, sometimes completely changing their meaning. For example, a PC Gamer story about a Baldur's Gate 3 glitch became 'BG3 players exploit children,' and an Ars Technica piece stating Valve had not announced Steam Machine pricing was rewritten to 'Steam Machine price revealed.' The AI label only appears after tapping into the interface, so on the feed it looks like the publisher wrote it, which editors worry will erode reader trust.

Summary & Key Takeaways

• Google research opened up about Titans, which solves the transformer weakness on long inputs by combining sharp short-term window attention with a long-term memory module that updates while running. Titans stores information based on surprise, forgets intelligently, and comes in three types: memory as context, gate, and layer.

• The MAC version of Titans handles context windows over 2 million tokens. A 760 million parameter model scored over 95% accuracy on needle-in-a-haystack at 16,000 tokens and beat GPT-4, recurrent Gemma 9B, Llama 3.1 70B, and retrieval-augmented Llama 3 on the BABILong benchmark connecting scattered facts.

• MIRAS unifies sequence models like Transformers, Mamba, RetNet, RWKV, and DeltaNet under one memory framework built around four core questions. It spawned attention-free models Moneta, Yaad, and Memora, some of which outperform Mamba 2 and classic Transformers in ultra-long context tests, serving as a roadmap for next-generation architectures.