What Are the Latest AI Breakthroughs from Google, OpenAI, Deepseek, and Sakana AI? CTM, GPT-5, and More

TL;DR

The latest breakthroughs include Sakana AI Labs’ Continuous Thought Machines, Deepseek Sparse Attention, and GPT-5-assisted scientific research. CTM gives individual neurons memory and adjusts its internal thinking time, while Deepseek’s lightning indexer can reduce long-context work by roughly 50x by attending to about 2,000 rather than 100,000 tokens. GPT-5 has helped expert researchers investigate active problems across biology, mathematics, and physics. Read on to see how each advance works.

Transcript

So, there have been many different recent AI breakthroughs. So, let's talk about them. So, in this video, this is going to be a video mainly focused on AI research papers and all of the cutting edge research that is probably going to shape the next 6 to 12 months, including a few research papers that were rather interesting in what they discovered ... Read More

Key Insights

• Continuous Thought Machines (CTM), from Sakana AI Labs, give each neuron its own small memory and mini brain that remembers past steps and keeps updating, unlike normal AI neurons that simply map an input to an output.
• CTM represents ideas through neuron synchronization rather than plain numbers: when two neurons' activity rises and falls together, that shared dance pattern becomes the model's main way of thinking.
• CTM decides its own thinking time using internal ticks, spending few ticks on easy problems and many on hard ones, and was tested on maze solving, image recognition, math puzzles, sorting, and reinforcement learning.
• Sakana CTO Llion Jones, one of the eight original Transformer inventors at Google, now argues it is time to move beyond Transformers, and Sakana is investigating neuroevolution approaches while publishing research openly.
• Deepseek Sparse Attention (DSA) replaces full attention with a lightning indexer that quickly scores previous tokens, selects only the top-K most relevant, and runs attention on those, focusing on the most useful context.
• DSA delivers roughly a 50x reduction in work for long context by attending to about 2,000 tokens instead of 100,000, making 128K, 256K, and even 1 million token context realistic because compute scales linearly.
• GPT-5 cannot run research autonomously, but in expert hands it meaningfully accelerates discovery across biology, math, physics, algorithms, cosmology, and material science on active unsolved real-world problems.
• GPT-5 helped solve a step of a decades-old Erdos problem by suggesting a pattern-breaking argument involving one odd number, and proposed an immune-cell mechanism from an unpublished chart within minutes.

Questions & Answers

Q: What are the latest AI research breakthroughs from Sakana AI, Deepseek, and OpenAI?

Sakana AI Labs introduced Continuous Thought Machines, which use neuron memory, synchronization patterns, and adjustable internal thinking time. Deepseek developed Sparse Attention to select only the most relevant past tokens, while OpenAI showed GPT-5 helping expert scientists accelerate research across fields including biology, mathematics, and physics.

Q: What is a Continuous Thought Machine (CTM)?

A Continuous Thought Machine is an architecture from Sakana AI Labs that continues thinking through a series of internal ticks instead of producing an answer after one thinking step. Each neuron has a small memory and its own mini brain, allowing it to remember recent steps and keep updating.

Q: How does CTM use neuron synchronization to think?

CTM examines how neurons’ activity changes together over time. When two neurons rise and fall together, their synchronization pattern becomes part of the model’s main way of representing and processing ideas.

Q: How does a CTM decide how long to think?

A CTM reasons through internal ticks, similar to taking several attempts to solve a maze or mathematics problem. Easy problems may require only a few ticks, while difficult ones may require many, and the CTM decides its thinking time on its own.

Q: What tasks was the Continuous Thought Machine tested on?

CTM was tested on maze solving, image recognition, mathematics puzzles, number sorting, and reinforcement learning. It solved mazes larger than those used in training, scanned different parts of images, learned simple puzzle rules, sorted step by step, and controlled moving robots after thinking multiple times.

Q: What is Deepseek Sparse Attention and how does it work?

Deepseek Sparse Attention uses a module called a lightning indexer to scan previous tokens and score their relevance. It selects the top-K most relevant tokens and runs attention only on them instead of comparing every token with the entire preceding context.

Q: Why can Deepseek Sparse Attention reduce long-context computation?

It can attend to about 2,000 selected tokens rather than all 100,000 tokens, producing roughly a 50x reduction in work for long contexts. The indexer is trained to mimic full-transformer attention, helping the system retain nearly the same accuracy while using less computation.

Q: How is GPT-5 accelerating scientific research?

GPT-5 has helped expert researchers work on active problems across biology, mathematics, physics, algorithms, cosmology, and material science, although it cannot conduct research autonomously. Reported examples include proposing an immune-cell mechanism from an unpublished chart, suggesting an argument involving one odd number for a step in a decades-old Erdos problem, and exposing flaws in a robotics decision-making method.

Summary & Key Takeaways

• Continuous Thought Machines from Sakana AI Labs treat each neuron as a tiny brain with its own memory that keeps updating step by step. The model thinks through internal ticks, uses neuron synchronization patterns as its reasoning, and decides on its own how long to think about each problem.

• Deepseek introduced Deep Seek Sparse Attention, using a lightning indexer to score all previous tokens and attend only to the top-K most relevant ones. Because the indexer mimics full-transformer attention, it keeps nearly the same accuracy while cutting compute up to 50x and enabling million-token context.

• OpenAI, with Oxford, Cambridge, Harvard and others, showed GPT-5 accelerating real research across biology, math, and physics. It explained an immune-cell change from an unpublished chart, solved a step of a decades-old Erdos problem, and exposed flaws in a robotics decision-making method.