Perspectives in AI: From LLMs to Reasoning with Edward Hu, Inventor of LoRA and μTransfer - Pear VC

Perspectives in AI: From LLMs to Reasoning with Edward Hu, Inventor of LoRA and μTransfer - Pear VC

In the world of artificial intelligence (AI), there are constantly new advancements and techniques being developed to improve the capabilities of models. One such method is Low Rank Adaptation (LoRA), which allows for the adaptation of large, pre-trained models to specific tasks or domains without the need for extensive retraining. This concept was invented by Edward Hu, the creator of LoRA and μTransfer, and has gained significant attention in the AI community.

So, what exactly is LoRA? It is a method that involves having a smaller module containing enough domain-specific information, which can be appended to a larger model. This smaller module acts as an auxiliary component that adjusts the characteristics of the larger model without the need for rebuilding or retraining it. In other words, LoRA allows for the injection of domain-specific knowledge into a larger model, enabling it to understand and process information within a specific field.

The implementation of LoRA is based on the mathematical concept of low rank approximation. By creating a smaller, adaptable module with specific characteristics or information, it can be integrated into larger models to customize them for a particular task. This approach has proven to be highly efficient, especially when compared to other methods such as fine-tuning or using adapters.

Fine-tuning is a common technique used to adapt models to specific tasks. However, it can be a costly and time-consuming process. For example, the storage cost of saving every checkpoint during the fine-tuning process can be significant, especially when dealing with large models. Additionally, the process of switching models for customization purposes can be network-intensive, I/O-intensive, and slow, leading to a less practical user experience.

On the other hand, LoRA offers impressive efficiencies in a production environment. Through the fine-tuning and adaptation of a 175 billion parameter model, the resource usage was significantly reduced to just 24 V100s. Moreover, the reduction in checkpoint sizes from 1 TB to just 200 megabytes opened the door to innovative engineering approaches such as caching in VRAM or RAM and swapping them on demand. This improvement in switching models swiftly greatly enhanced the user experience.

One of the primary benefits of LoRA in a production environment is the acceleration of training and the reduction in training costs by decreasing the number of GPUs required. The base model remains the same, but the adaptive part is faster and smaller, making it quicker to switch between different tasks or domains. Additionally, LoRA contributes to a significant reduction in storage costs, which can be up to a factor of 1000 to 5000, resulting in substantial savings for AI teams.