Navigating the High Cost of AI Compute: What it takes to build an $11B startup

Navigating the High Cost of AI Compute: What it takes to build an $11B startup

In today's rapidly evolving tech industry, artificial intelligence (AI) has become a dominant force. However, one of the biggest challenges faced by companies in this space is the high cost of AI compute. The cost of training and inference can be a significant drain on resources, with some companies spending up to 80% of their total capital on compute resources alone. The demand for compute is so high that it outstrips supply by a factor of 10. This has led to a prevailing factor driving the industry today – the cost of AI compute.

The cost of AI compute is determined by factors such as the size and type of the model being used. For transformers, a common type of generative AI model, a forward pass or inference for a model with a certain number of parameters and input/output tokens takes approximately 2np floating point operations (FLOPs), where n is the length of the input/output sequence and p is the number of parameters. Training a model takes approximately 6*p FLOPs per token. These estimates allow us to approximate the compute and memory requirements for a model of a certain size.

Memory requirements for transformers depend on the model size. For inference, the model parameters need to fit into memory, while for training, additional intermediate values per parameter between the forward and backward pass need to be stored. Optimizations such as using shorter floating point representations can help reduce memory requirements. However, as model sizes continue to grow, specialized chips such as GPUs (graphics processing units) are needed to accelerate compute tasks. GPUs offer significant performance improvements over traditional processors and can reduce inference time from hours to seconds.

While cloud-based AI infrastructure is often the right choice for most startups, there are cases where running your own AI hardware can offer a competitive advantage. Companies that require fine-grained control over training and inference or whose models are the product itself may benefit from managing their own infrastructure. However, this comes with its own challenges, including hardware selection, power, space, and cooling considerations.

To optimize the cost and performance of AI infrastructure, there are several factors to consider. The availability of GPUs, network interconnects, customer support, and latency requirements all play a role in selecting the right hardware and cloud provider. Additionally, software optimizations can greatly impact model performance and running time. Techniques such as mixed or shorter precision, weight partitioning, and scheduling AI tasks can lead to significant improvements in efficiency.

Building a successful startup in the AI space requires a combination of speed and quality. It is possible to move fast and deliver a high-quality product, but it requires urgency in fixing underlying issues and challenging assumptions about what is possible. This sense of urgency may make the CEO appear unreasonable at times, but it forces teams to think critically and pressure-test their assumptions.

In the interview with Parker Conrad, CEO of Rippling, he emphasizes the importance of not accepting the set of implicit assumptions about what is possible. By challenging these assumptions and finding the right people to lead product development within the company, a compound startup model can be successful. This model involves founders of specific products within the company who have previous experience in founding companies.

In conclusion, navigating the high cost of AI compute is a challenge for companies in the industry. However, by understanding the compute and memory requirements of AI models, optimizing infrastructure and hardware selection, and leveraging software optimizations, companies can mitigate these costs. Balancing speed and quality is crucial, and adopting a compound startup model can help drive innovation and success in the AI space.

Actionable Advice:

1. Optimize compute and memory requirements by selecting the right AI model size and leveraging software optimizations such as mixed precision and weight partitioning.
2. Evaluate the cost and performance of different cloud providers and hardware options to find the right balance for your AI infrastructure.
3. Challenge assumptions and create a sense of urgency in fixing underlying issues to deliver a high-quality product at a fast pace.