The Intersection of Nvidia H100 GPUs and the Qualitative Aspect of Investing

The Intersection of Nvidia H100 GPUs and the Qualitative Aspect of Investing

Introduction:
In the rapidly evolving world of technology and investing, two seemingly unrelated topics - Nvidia H100 GPUs and qualitative investing - converge to highlight important aspects of their respective domains. This article aims to explore the common points between these subjects and shed light on their significance in today's landscape. From the demand for high-end GPUs to the qualitative evaluation of investment opportunities, we delve into the intricacies that drive decision-making in these realms.

The Growing Demand for Nvidia H100 GPUs:
Startups engaged in fine-tuning large open-source models form a significant user base for high-end GPUs like the Nvidia H100. These companies, often building models from scratch, rely on the computational power and speed offered by these GPUs to fulfill multimillion-dollar contracts spanning several years. Additionally, private cloud-based companies, such as CoreWeave and Lambda, utilize hundreds or thousands of H100s for tasks like fine-tuning existing models and exploring diffusion models. The H100 emerges as the preferred choice due to its superior performance and scalability for both training and inference in large language models (LLMs).

The Need for H100 GPUs in LLM Training and Inference:
In the realm of LLM training, memory bandwidth, FLOPS (tensor cores or equivalent matrix multiplication units), caches, cache latencies, additional features like FP8 compute, compute performance, and interconnect speed play crucial roles. The H100 stands out as the go-to option due to lower cache latencies, FP8 compute capabilities, and its potential to scale effectively with higher GPU numbers. Conversely, when it comes to LLM inference, performance per dollar becomes the primary consideration. While the H100 and A100 GPUs offer a competitive performance per dollar ratio, the former remains favored due to its faster training times and the importance of compressed launch or training periods for startups.

The Role of CUDA as Nvidia's Moat:
Despite the theoretical possibility of using AMD GPUs, the practical challenges and time required for implementation hinder their widespread adoption. Devoting significant time to getting everything to work could result in delays, potentially giving competitors an advantage. Therefore, Nvidia's CUDA framework serves as a moat, solidifying its position as the preferred choice for companies in the LLM space. The risks associated with deploying a large number of AMD GPUs or unproven startup silicon chips on a significant scale further reinforce the preference for Nvidia GPUs.

The Comparison of H100s and A100s:
The H100 GPUs outperform A100 GPUs in terms of speed, boasting approximately 3.5 times faster inference and 2.3 times faster training for 16-bit operations. Consequently, most companies opt for the 8-GPU HGX H100s over the DGX H100s or 4-GPU HGX H100 servers. However, the cost of these GPUs must also be considered, with a single DGX H100 with 8 H100 GPUs amounting to $460,000, including required support. Startups can avail the Inception discount, offering approximately $50,000 off on up to 8 DGX H100 boxes.

The Growing Need for H100 GPUs:
Several prominent companies, including GPT-4, Meta, Tesla, Stability AI, and Inflection, rely on large numbers of A100 GPUs for their operations. Estimates suggest that OpenAI may require around 50,000 H100s, while Inflection and Meta could potentially need 22,000 and 25,000 H100s, respectively. Cloud service providers like Azure, Google Cloud, AWS, and Oracle might each require up to 30,000 H100s. Furthermore, private clouds operated by companies like Lambda Labs and CoreWeave may collectively demand 100,000 H100s. These estimates hint at a significant market demand, excluding potential requirements from Chinese companies such as ByteDance, Baidu, and Tencent.

Production and Bottlenecks in H100 Manufacturing:
TSMC serves as the manufacturer for H100 GPUs, and the production process, including packaging and testing, takes approximately six months. While wafer starts are not a bottleneck, the 3D stacking packaging technique known as CoWoS poses challenges. The limited availability of MI250, an alternative to H100, further complicates the situation, as it may not be viable due to its availability concerns.

Nvidia Allocations and the Importance of End Customers:
Nvidia allocates GPUs per customer, considering factors such as the identity of the end customer and their potential impact on the market. Cloud service providers like Azure, Oracle, Lambda Labs, and AWS have all launched their H100 previews at different times, receiving allocations based on their specific requirements. Nvidia prefers customers with strong brand names or startups with impressive credentials, while also avoiding providing large allocations to companies competing directly with them.

Qualitative Investing: The Secret Sauce:
Qualitative investing, as highlighted by Todd Combs, Warren Buffett's protégé, emphasizes the importance of understanding the unique aspects of a business beyond quantitative analysis. This approach involves evaluating qualitative factors such as moats, barriers to entry, and other intangible aspects that go beyond what is presented in filings or annual reports. Warren Buffett and Todd Combs prioritize qualitative evaluation, focusing on details that are often overlooked. They emphasize starting with facts and building narratives from there, rather than starting with preconceived narratives.

The Role of Gratitude and Mentorship in Investing:
Todd Combs emphasizes the role of mentorship and gratitude in achieving success as an investor. He acknowledges that luck plays a significant part, but he also recognizes the impact of others who have supported and mentored him throughout his journey. Combs highlights the feeling of gratitude when someone sees potential in you, taking a chance on your abilities. He firmly believes in standing on the shoulders of those who have come before and acknowledges that investing is a collaborative effort.

Conclusion:
The convergence of Nvidia H100 GPUs and qualitative investing sheds light on the intricacies and unique aspects of these domains. From the demand for high-performance GPUs in the LLM space to the importance of qualitative evaluation in investment decisions, these topics showcase the dynamic nature of technology and finance. As the demand for GPUs continues to rise and investors seek to uncover hidden gems, understanding the nuances of both domains becomes increasingly crucial.

Actionable Advice:

1. For startups in the LLM space, prioritize the adoption of Nvidia H100 GPUs due to their superior performance and scalability for training and inference tasks. Consider the impact of compressed launch or training periods on the overall success of your models.
2. Investors should incorporate qualitative analysis into their decision-making process, going beyond quantitative metrics. Evaluate factors such as moats, barriers to entry, and other intangible aspects to gain a deeper understanding of investment opportunities.
3. Cultivate a sense of gratitude and recognize the importance of mentorship in your investing journey. Surround yourself with individuals who believe in your potential and learn from their guidance and experiences.

Disclaimer: The information in this article is for informational purposes only and should not be considered as financial or investment advice.