How to Improve Open Model Pretraining at Scale

TL;DR

Strong model pretraining depends first on high-quality, diverse data, then on architecture, information extraction, gradient quality, and stable scaling. Hugging Face applies this framework through open resources such as FineWeb-Edu2, FinePDF, and SmolLM 3, while exploring optimizer alternatives and sparse Mixture of Experts designs that can improve learning and inference under practical constraints.

Transcript

Hey everyone, welcome to the latest in space podcast. This is Allesio, founder of Colonel Labs, and I'm joined by Swix from Cognition. First, let me introduce you the new role. We we haven't uh figured out the official title yet, but yes, uh uh we're recording this the day after we announced the series C and uh it's uh it's very good. Uh you know, ... Read More

Key Insights

• Data quality and diversity are the foundation of successful pretraining because models can only learn from the information supplied to them. Hugging Face therefore treats the relevance of raw training data as the most important of its five model-training pillars.
• Model architecture is an optimization problem shaped by practical constraints, including inference efficiency, the number of GPUs on which a model must fit, and key-value cache consumption. The best architecture is therefore not simply the largest design, but one suited to its intended operating conditions.
• Information extraction efficiency measures how much useful learning a model obtains from its data at every training step. Distillation and multi-token prediction are two approaches discussed, with multi-token prediction identified as particularly useful for coding data.
• Gradient quality determines how effectively each update improves the model. Optimizer research can approximate additional matrix information or project updates into a data-related space, making the learning process itself an important target for innovation beyond dataset and architecture improvements.
• Training stability is essential when scaling because gradients or activations can diverge and cause training to fail. Appropriate hyperparameters, an understanding of small-to-large model scaling, and effective feature learning help ensure that each gradient contributes useful information without destabilizing the run.
• FinePDF explores PDFs as an underused source of high-quality training data. Its pipeline benchmarks multiple PDF extractors and can retrieve documents again from the internet when Common Crawl copies are poorly extracted, supporting cleaner data that can be mixed with other web datasets.
• Open optimizer research is moving beyond widespread reliance on AdamW. The discussion identifies Muon, Shampoo, and work associated with Kimi K2 as recent directions, while noting that some open model reports provide limited detail about optimizer configuration, stability, and scaling methods.
• Mixture of Experts research focuses on sparsity, expert specialization, and routing. The discussion highlights granular routing mechanisms associated with DeepSeek and high sparsity in Alibaba's Qwen models, showing how architecture research can allocate computation selectively while addressing practical production constraints.

Questions & Answers

Q: What are the five pillars of model pretraining?

The five pillars are data quality and diversity, model architecture, information extraction efficiency, gradient quality, and training stability at scale. The framework begins by maximizing the relevance of raw information, then designs the model for practical constraints, improves how much it learns per step, strengthens parameter updates, and selects scaling behavior and hyperparameters that prevent divergence.

Q: Why are data quality and diversity important for model training?

Data quality and diversity are important because the model's capabilities depend directly on the raw information used during pretraining. The discussion describes this as the most important pillar and invokes the principle of garbage in, garbage out. A strong dataset team must therefore improve relevance, coverage, and variety before architecture or optimizer innovations can realize their full value.

Q: What is FinePDF and why was it created?

FinePDF is a Hugging Face data effort that explores PDF documents as a source for model pretraining. It was created because PDFs were considered underexplored compared with established web datasets. The pipeline benchmarks multiple PDF extraction methods and supports mixing the resulting content with other web data, where the presented results showed strong comparative performance.

Q: Why does the FinePDF pipeline retrieve some documents again?

The FinePDF pipeline can retrieve a document again from the internet when the PDF copy found through Common Crawl has not been extracted well. A newer or better available version may produce cleaner content. This step is part of a broader pipeline that evaluates PDF extractors and seeks to improve the quality of information ultimately used for model training.

Q: How should model architecture reflect inference constraints?

Model architecture should be selected according to the conditions under which the trained model must operate. The relevant constraints include inference efficiency, how many GPUs can hold the model, and how much key-value cache it consumes. Architecture research therefore aims to construct the strongest model possible within explicit deployment limits, rather than optimizing capability without regard to resource use.

Q: How can pretraining extract more information from each data sample?

Pretraining can extract more information by changing how learning signals are obtained at each step. The discussion identifies distillation as one approach and multi-token prediction as another. Multi-token prediction is described as especially useful for coding. These techniques complement better data and architecture by increasing how much useful information training derives from the available examples.

Q: Why are researchers exploring optimizers beyond AdamW?

Researchers are exploring alternatives because the optimizer deeply influences how a model learns and how effectively gradients update its parameters. The discussion notes that AdamW was widely used, while Muon, Shampoo, Kimi K2, and earlier use of Adafactor represent other directions. It also questions whether optimizer parameters reused across very different model scales can truly remain optimal.

Q: What problems must be controlled when scaling model training?

Scaling requires control of gradients, activations, hyperparameters, and the relationship between smaller experiments and larger training runs. Gradients or activations can diverge as scale increases, causing instability. The training setup must therefore preserve stable behavior and effective feature learning, so that each update extracts useful information rather than allowing the run to explode or drift away from productive learning.

Summary & Key Takeaways

• Elie Bakouch presents model training as five connected optimization problems: maximizing the relevance of raw data, designing architectures around efficiency constraints, extracting more information at each training step, improving gradient updates, and maintaining stability as models scale. This framework helps separate research areas that can be explored in parallel.

• Hugging Face treats data quality and diversity as the most important foundation of pretraining. FinePDF expands the available sources by processing PDF documents, benchmarking extraction methods, and sometimes retrieving documents again from the internet when Common Crawl versions are poorly extracted. The resulting data can then complement web datasets.

• The discussion also covers optimizer research beyond AdamW, sparse Mixture of Experts architectures, expert routing, data rephrasing, and lessons from training SmolLM. These areas reflect a broader effort to make open model development more efficient, reproducible, and useful to researchers who want to build upon publicly released datasets, tools, and findings.