How Does Llama 4 Handle 10 Million Tokens?

TL;DR

Llama 4 Scout supports a 10 million token context window, roughly five million words, while using 17 billion active parameters from 109 billion total parameters. The Llama 4 family also includes Maverick, a larger multimodal model available now, and Behemoth, a two trillion parameter model that remains in preview and training.

Transcript

Meta recently released Llama 4, which is their new open-source large language model. It's a family of three, and it's available right now, and I partnered with Meta for this video to give you the breakdown of all the different versions of Llama 4 and what they have to offer. And later in the video, I'll show you a couple of different websites where... Read More

Key Insights

• • Llama 4 is a family of three multimodal models: Scout, Maverick, and Behemoth. Scout is presented as the smallest general-purpose model, Maverick as the medium-sized option, and Behemoth as the largest model, which remains in preview and training.
• • Scout’s context window is 10 million tokens, which the transcript estimates as roughly five million words. This capacity is designed to accommodate very large collections of text without requiring users to trim documents or divide them into smaller files.
• • Scout uses 17 billion active parameters from 109 billion total parameters. Its 16-expert mixture-of-experts architecture selects only the model components that are relevant to a request, allowing it to preserve strong performance while using fewer computational resources at a given moment.
• • Scout can run on a single NVIDIA H100 GPU according to the transcript. Although that still requires substantial hardware, activating only 17 billion parameters at a time makes the model more resource-friendly than its total parameter count might initially suggest.
• • Maverick uses 17 billion active parameters from 400 billion total parameters and includes 128 experts. The cited benchmarks show it competing with or beating GPT-4o, Gemini 2.0 Flash, and DeepSeek V3.1 across tested areas, while retaining multimodal capabilities.
• • Maverick’s listed pricing starts at 19 cents per one million input and output tokens. The transcript describes this as comparable with Gemini 2.0 Flash and less expensive than DeepSeek in the displayed comparison, with GPT-4o priced considerably higher there.
• • Behemoth has two trillion total parameters, 288 billion active parameters, and 16 experts. The presented STEM benchmark shows it outperforming Gemini 2.0 Pro and Claude Sonnet 3.7 even though Behemoth is still in preview and continues to undergo training.
• • Open-source models provide developers with control beyond API-only access, including the ability to self-host, customize, and fine-tune models. Scout and Maverick can be requested or obtained through Hugging Face, while Behemoth is not yet available for download.

Questions & Answers

Q: What models are included in the Llama 4 family?

The Llama 4 family includes three multimodal models: Scout, Maverick, and Behemoth. Scout is described as the smallest general-purpose option, Maverick as the medium-sized model, and Behemoth as the largest. Scout and Maverick are currently available for access or download, while Behemoth remains in preview and is still being trained.

Q: How large is Llama 4 Scout’s context window?

Llama 4 Scout has a 10 million token context window, which the transcript estimates as approximately five million words. This allows it to process much larger inputs than models cited with 128,000-token or two million-token limits. The capacity could reduce the need to trim, organize, and split large documents into smaller chunks before analysis.

Q: How does the Llama 4 mixture-of-experts architecture work?

The mixture-of-experts architecture activates only the parts of the model that are most relevant to a particular request. Depending on the task, the model may select two, three, or four experts instead of engaging the entire model. This approach is intended to maintain high performance while making processing faster, more efficient, and less demanding on hardware.

Q: What are the parameter specifications of Llama 4 Scout?

Llama 4 Scout has 109 billion total parameters, 17 billion active parameters, and 16 experts. The total parameter count represents the broader collection of learned model settings, while the active count describes how many parameters are used at a given time. This selective activation allows Scout to run entirely on one NVIDIA H100 GPU.

Q: How is Llama 4 Maverick different from Scout?

Llama 4 Maverick is larger than Scout in total capacity, with 400 billion total parameters and 128 experts, compared with Scout’s 109 billion total parameters and 16 experts. Both use 17 billion active parameters. Scout’s highlighted advantage is its 10 million token context window, while Maverick emphasizes competitive reasoning, coding, vision, and benchmark performance.

Q: What are the specifications and status of Llama 4 Behemoth?

Llama 4 Behemoth is the largest member of the family, with two trillion total parameters, 288 billion active parameters, and 16 experts. It remains in preview and is still undergoing training, so it is not available for download. The presented STEM benchmark shows it outperforming Gemini 2.0 Pro and Claude Sonnet 3.7.

Q: Why might developers choose an open-source Llama 4 model?

An open-source Llama 4 model gives developers and companies more control over how the technology is deployed and adapted. Unlike models accessed only through a provider’s API, an open-source model can be self-hosted, customized, and fine-tuned. The transcript presents this flexibility as useful for teams building applications that require greater control over their model infrastructure.

Q: Where can users try or download Llama 4?

Users can try Llama 4 through Meta AI at meta.ai, and Meta has also introduced it across WhatsApp, Messenger, and Instagram. Groq’s chat website offers selectable access to Scout and Maverick. Developers can request access to the downloadable models through Meta or obtain them through Hugging Face, although Behemoth remains unavailable while in preview.

Summary & Key Takeaways

• Meta’s Llama 4 family consists of Scout, Maverick, and Behemoth. All three models are multimodal, meaning they can work with text and images. Scout is the smallest general-purpose option, Maverick is the medium-sized model, and Behemoth is the largest model, although Behemoth remains in preview and is still undergoing training.

• Scout has 109 billion total parameters, 17 billion active parameters, and 16 experts. Its mixture-of-experts architecture activates only the model components relevant to each request, improving efficiency. Its distinguishing feature is a 10 million token context window, which the transcript equates to approximately five million words of usable context.

• Maverick has 400 billion total parameters, 17 billion active parameters, and 128 experts, while Behemoth reaches two trillion total parameters and 288 billion active parameters. Developers can request Scout or Maverick, access them through Hugging Face, or test Llama 4 through Meta AI and Groq’s model-selection interface.