The New Language Model Stack: Customization, Retrieval, and Open Source

The New Language Model Stack: Customization, Retrieval, and Open Source

Language models have become an integral part of various industries, with companies incorporating them into their products and workflows. From code autocompletion to chatbots, language models have proven to be a valuable tool in enhancing user experiences and improving productivity. In this article, we will explore the new stack for these applications, focusing on customization, retrieval mechanisms, and the growing trend of open source usage.

Customization of language models is a crucial aspect for companies that want to tailor the models to their specific needs. There are three main ways to achieve customization: training a custom model from scratch, fine-tuning a base model, or using a pre-trained model with relevant context retrieval. Each approach has its own level of difficulty and requirements.

Training a custom model from scratch is the most challenging method. It requires skilled ML scientists, substantial amounts of relevant data, and significant computational resources. Historically, this approach was limited to mega-cap tech companies due to the resources and expertise required. However, open source tooling, such as Hugging Face, is making custom model training more accessible, enabling more companies to innovate with language models.

Fine-tuning a base model is a less complex method compared to training from scratch. It involves updating the weights of a pre-trained model with proprietary or domain-specific data. While this approach is becoming more accessible through open source innovations, it still requires a skilled team and may have unintended consequences, such as model drift or loss of other skills. Fine-tuning has the potential to become more common in the future as advancements continue to simplify the process.

Using a pre-trained model with relevant context retrieval is the easiest way to customize language models. Instead of fine-tuning or training from scratch, this approach focuses on providing the model with the right information at the right time. By making data searchable using natural language, developers can retrieve relevant context for the model to reason about. This can be achieved through structured queries to databases, product catalog searches, calling external APIs, or using embeddings retrieval.

Embeddings retrieval is particularly useful as it allows unstructured data to be easily searchable using natural language. This process involves converting data into embeddings, storing them in a vector database, and searching for the most relevant context when a query occurs. Embeddings retrieval helps overcome the model's limited context window, reduces costs, solves data freshness issues, and can be implemented by developers without formal machine learning training. Startups typically use purpose-built vector databases, while larger companies often rely on tools provided by their cloud providers.

The new stack for language models also emphasizes the importance of retrieval mechanisms. In a survey, 88% of respondents believed that a retrieval mechanism, such as a vector database, would remain a key part of their stack. Retrieving relevant context helps improve result quality, reduce inaccuracies, and address data freshness concerns. Purpose-built vector databases, as well as offerings from cloud providers, are commonly used to facilitate efficient storage, searching, and updating of embeddings at scale.

Open source usage is also gaining traction in the language model space. In a recent survey, 65% of companies reported having language model applications in production, with 94% using a foundation model API. OpenAI's GPT model was the most popular choice among respondents, but Anthropic's model also saw increased interest. Additionally, 38% of companies expressed interest in an LLM orchestration and application development framework like LangChain. This indicates a growing demand for open source tools that simplify the development and deployment of language models.

While the language model stack is still evolving, there are actionable steps that companies can take to leverage these advancements effectively:

1. Assess your customization needs: Evaluate whether training a custom model from scratch, fine-tuning a base model, or using a pre-trained model with retrieval mechanisms aligns with your requirements. Consider the level of difficulty, available resources, and the desired level of customization.

2. Explore open source tools: Keep an eye on the latest open source innovations in the language model space. Projects like Hugging Face and LangChain are making customization and development more accessible. Experiment with these tools to harness their potential for your specific use cases.

3. Consider a scalable retrieval mechanism: If retrieval of relevant context is crucial for your language models, explore purpose-built vector databases or cloud provider offerings. These solutions can help streamline the storage, searching, and updating of embeddings, ensuring efficient and scalable retrieval processes.

In conclusion, the new language model stack revolves around customization, retrieval mechanisms, and the growing adoption of open source tools. Companies are increasingly seeking ways to tailor language models to their unique contexts, and the three main approaches – training from scratch, fine-tuning, and retrieval-based customization – offer varying levels of difficulty and requirements. Additionally, the use of purpose-built vector databases and open source tools is on the rise, enabling more companies to leverage the power of language models effectively. By understanding these trends and taking actionable steps, businesses can unlock the full potential of language models and drive innovation in their respective industries.