How Can One AI Model Learn Across Physics?

Transcript

What's up everybody? I'm R.J. I'm here with my co-host Brandon for a latent space lightning AI for science podcast. A week ago, we released a fantastic episode with Ana Anan Kumar. It got a great reception, probably helped by the fact that she brought her company Accelerated Understanding Out of Stealth days before we released. But when we recorded... Read More

Key Insights

• Accelerated Understanding is building one foundation model across multiple physical domains, following the pattern by which general language models replaced separate systems for tasks such as spellchecking, translation, and autocomplete.
• Physical laws are essential because real-world scientific data are limited, and genuinely new discoveries cannot already exist in training data. Existing simulators can initialize training, while physics can guide later self-improvement.
• Shared physical structure exists across apparently different applications. Fluid behavior in catheters and rockets follows common fluid-dynamics principles, while conservation, causality, object permanence, and temporal variation provide broader features across systems governed by different equations.
• Full physical simulation requires three spatial dimensions plus time. Modeling all four dimensions together preserves continuity and the relationship between inputs and outputs, while one-step autoregressive rollouts can accumulate errors over time.
• The reported system supports context lengths far beyond typical language settings. Accelerated Understanding says it has trained with trillion-context inputs and outputs and performed inference with a five-trillion context.
• Custom distributed infrastructure is necessary because individual samples can exceed an accelerator or full node, and model layers can be too large to reassemble inside one GPU using conventional sharding approaches.
• Multiphysics training reportedly improves performance through shared learning. A model trained across several physics areas performed better than equally sized separate models that devoted the same parameter capacity to each individual area.
• Neural operators make the architecture resolution invariant, so applications can use lower resolution during early design exploration and reserve very large contexts for physical problems that require detailed simulation.

Questions & Answers

Q: What is a foundation model for the physical world?

A foundation model for the physical world is one model trained across multiple kinds of physical systems rather than a separate model for every application. Accelerated Understanding aims to combine domains such as fluid dynamics, semiconductors, energy, and aerospace by learning shared features including conservation, causality, object permanence, and change over time, while using physical laws to guide its predictions and improvement.

Q: Why can one model learn across different areas of physics?

Different physical applications can share underlying structure even when their geometries, operating regimes, or governing equations differ. Fluid in a catheter and fluid in a rocket still reflect fluid-dynamics principles, with regimes parameterized by quantities such as Reynolds numbers. Across broader systems, recurring properties such as energy conservation, causality, object permanence, and temporal variation give a neural model transferable patterns to learn.

Q: How does physics-based self-improvement address limited training data?

Real physical data are not available in a single internet-scale collection, and scientific discovery seeks results that are new by definition. The proposed approach therefore begins with existing simulators and other available sources, then uses the laws of physics as a guiding signal. Those laws provide objective constraints that can evaluate and improve model behavior even when the desired discovery was absent from the original training data.

Q: Why does the model use full four-dimensional rollouts?

The model represents three-dimensional space and also follows the system through time, producing a four-dimensional rollout. The founders prefer this approach because predicting only one time step at a time can cause errors to accumulate. Full rollouts are also intended to preserve continuity and clarify how conditions in the input led to the resulting output, which matters for engineering design tasks.

Q: How can a physics model process trillion-token contexts?

Accelerated Understanding says its models can train with trillion-context inputs and outputs and perform inference at a five-trillion context. Achieving that scale required rethinking distributed training because layers may be too large to reassemble inside one GPU, while individual samples may not fit on an accelerator or even a full node. The company therefore developed different sharding infrastructure and strategies for its unusually large data and model shapes.

Q: What role do neural operators play in the architecture?

Neural operators provide resolution invariance, allowing the model to work at different spatial resolutions and context lengths rather than requiring one fixed resolution for every task. A difficult simulation can receive very high resolution, while early design exploration or a less demanding application can use less compute. This flexibility supports both detailed physical modeling and more economical iteration within the same general architecture.

Q: Does training across multiple physics domains improve performance?

The founders report that multidomain training produces benefits beyond simply increasing model size. A model of a given size trained across multiple physics areas performed better than separate models that each received parameter capacity equal to the multidomain model. They interpret this result as evidence of shared learning, because adding diverse physics areas improved performance within the individual areas rather than merely combining unrelated capabilities.

Q: Why are standard transformer architectures unsuitable for this approach?

The founders argue that standard transformer architectures cannot support a five-trillion context because their quadratic complexity makes that scale infeasible. They also consider unrestricted all-to-all correlation unnecessary for physical systems, which possess more structure than arbitrary token sequences. Their architecture instead builds on neural operators and physical structure, enabling variable resolution and extremely large contexts without relying on the same assumptions used by language transformers.

Summary & Key Takeaways

• Accelerated Understanding is developing a foundation model for the physical world, inspired by the consolidation of specialized language systems into general models. The goal is to learn across areas such as fluid dynamics, semiconductors, energy, and aerospace by exploiting shared physical properties, including conservation, causality, object permanence, and variation over time.

• The model represents three-dimensional space and full temporal rollouts without reducing the problem to compressed video or one-step autoregressive prediction. This creates four independently growing dimensions and extremely large contexts. The company reports trillion-context training, five-trillion-context inference, and custom sharding infrastructure for samples and layers that exceed individual accelerator or node capacity.

• Neural operators provide resolution invariance, allowing computation and context length to vary with the precision required by each application or design stage. Simulator-generated data supplies initial training material, while physical laws provide dense, objective signals for self-improvement. Reported early opportunities include semiconductor design, manufacturing, energy, geothermal exploration, and scientific discovery.