Why your brain is so good at rewriting history: Navigating the High Cost of AI Compute

Why your brain is so good at rewriting history: Navigating the High Cost of AI Compute

Memory plays a crucial role in how we perceive and remember events. It not only helps us recall information but also shapes our understanding of past experiences. However, our memory is not always accurate, and there are several factors that contribute to this phenomenon.

One factor that influences our memory is schema-driven memory. Schemas are frameworks in our memory that help us predict the sequence of events and understand the reasons behind our actions. When we think back on the past, we often rely on these schemas to structure our memories. This can lead to misremembering the order of events or attributing certain ideas or contributions to ourselves when they may not have been entirely our own. Our memory tends to emphasize our own contributions because they are most vividly represented in our minds.

Additionally, the information we receive after an event can also influence our memory. Things we hear about after the fact can become mixed in with our visual memories, further distorting our recollection of events. This phenomenon highlights the malleability of memory and how easily it can be altered or rewritten.

In the realm of artificial intelligence (AI), there is a different kind of cost that poses challenges for developers and companies alike. The high cost of AI compute is a predominant factor driving the industry today. The demand for compute resources far exceeds the supply, with some companies spending more than 80% of their total capital raised on compute resources. The cost of training and inference depends on the size and type of the model, measured in terms of parameters and tokens.

For example, GPT-3, a widely known AI model, has approximately 175 billion parameters. Training a model like GPT-3 can take an enormous amount of computational power, with estimates ranging from $500,000 to $4.6 million for a single run. The complexity of AI algorithms makes AI infrastructure expensive, and it becomes essential to pick the smallest model that solves the desired use case.

To optimize AI infrastructure costs, there are several actionable pieces of advice to consider:

1. Start with hosted model services: For many startups, especially app companies, building their own AI infrastructure may not be necessary from the outset. Hosted model services like OpenAI or Hugging Face provide a platform to search for product-market fit without the need to manage underlying infrastructure or models. Developers can achieve meaningful control over model performance through prompt engineering and higher-order fine-tuning abstractions.

2. Consider cloud-based solutions: In most cases, the cloud is the right place for AI infrastructure. Cloud providers offer compute capacity, availability, and network interconnects that can meet the demands of AI workloads. However, at a very large scale, it may become more cost-effective to run your own data center if specific hardware or geopolitical considerations are essential.

3. Optimize model size and memory requirements: Choosing the right model size is crucial to balancing performance and cost. Large language models (LLMs) with high parameter counts may require specialized hardware and memory optimizations. Working with third parties that specialize in model optimizations can help reduce infrastructure costs and improve performance.

The high cost of AI infrastructure may seem like a barrier for new entrants, but the landscape is still evolving. Open-source models and advancements in software optimizations have the potential to disrupt the market and change the cost dynamics. As technology progresses and GPU performance improves, there is hope for more cost-effective AI infrastructure solutions in the future.

In conclusion, our memory's ability to rewrite history and AI infrastructure's high cost share a common thread – they both highlight the complexities of perception and the challenges of managing resources. Understanding the factors that influence memory and taking actionable steps to optimize AI infrastructure can help navigate these challenges effectively. By leveraging the power of memory and AI, we can continue to push the boundaries of human knowledge and innovation.