A Comprehensive Guide to Open AI Server Design and Overcoming Limitations

A Comprehensive Guide to Open AI Server Design and Overcoming Limitations

In late 2019, the Open Compute Project (OCP) officially released the OAI-UBB1.0 design specifications and subsequently launched an open acceleration hardware platform based on the OAI-UBB1.0 specifications. This platform supports various vendors' OAM products without the need for hardware modifications. The OCP also established the OAI working group in 2019 to define a form factor for AI acceleration cards that are better suited for large-scale deep learning training. The goal was to support higher power consumption and greater interconnect bandwidth for AI acceleration cards, while also addressing the issue of diverse card form factors and inconsistent interfaces. The OAI working group unified the AI acceleration card baseboard design specifications, known as OAI-UBB, building upon the OAM foundation. The OAI-UBB specifications further defined the host interface, power supply method, cooling method, management interface, inter-card interconnect topology, and scale-out approach for the baseboard. The UBB link can be divided into 8x links, and if all seven ports are configured as 16x, external expansion cannot be achieved. To form an interconnected cluster for expansion, the UBB baseboard limits the interconnect links to within 8x and by default designates the second half of port 1 (8x, commonly referred to as the 1H port) as the external expansion port.

In a related topic, let's explore the limitations in the swordplay of NVIDIA's Huang Jia Dao technique. GPUs with cxl extended memory can be used as CUDA malloc memory, which refers to VRAM. On the other hand, memory extended by Grace is simply host temporary open ATS memory for the GPU's pin memory. In terms of programming affinity, these two approaches are vastly different. Taking it a step further, cxl extended VRAM can be used as an HBM cache, allowing HBM to serve as a pool of DRAM cache. If implemented successfully, this can provide a software perspective of TB-level HBM VRAM. The bottleneck in this process is likely due to the missing microarchitecture during the native definition of GPGPU. Interrupts are not a major issue, and memory page table structures are challenging but not insurmountable. The most probable Achilles' heel lies in the complexity of CUDA's software stack and scheduler, making virtualization difficult to embed. CUDA is the towering fortress of NVIDIA, a barrier that almost all AI successors struggle to overcome. Have you ever wondered why you had to train a large model quickly with thousands of NVIDIA GPUs at your disposal, only for it to crash within a day? Well, your superiors imposed constraints on you, requiring you to tolerate crashes and add checkpoints for recovery within 20 minutes of each crash. As a result, you gradually became accustomed to the constant restarts during large-scale model training and the frequent swapping of faulty GPUs with new ones. Full virtualization of heterogeneous programming is possible (although I cannot directly provide existing industry examples), but NVIDIA has not achieved it. Do not fall into the trap of assuming that if NVIDIA doesn't do it, it cannot be done in the architecture. On the other hand, cloud computing requires virtualization.

Based on these insights, here are three actionable pieces of advice for designing open AI servers and overcoming limitations:

1. Embrace Open Compute Project (OCP) specifications: By adhering to the OAI-UBB design specifications, you can ensure compatibility and interoperability between different vendors' AI acceleration cards. This promotes flexibility and scalability in building AI server infrastructure.

2. Explore alternative memory extension technologies: While cxl extended VRAM is suitable for CUDA allocation, Grace's memory extension approach offers unique advantages in terms of host-gpu affinity. Consider the benefits of using Grace's pin memory for specific use cases, such as temporary memory allocation for GPU tasks.

3. Look beyond CUDA for virtualization solutions: Although CUDA is a dominant force in GPU programming, explore other virtualization options that can overcome its limitations. Investigate emerging technologies and frameworks that offer comprehensive virtualization capabilities for heterogeneous programming scenarios.

In conclusion, the Open Compute Project's OAI-UBB design specifications have revolutionized the open AI server landscape, enabling seamless integration of AI acceleration cards from different vendors. However, it is crucial to overcome the limitations and challenges posed by proprietary software stacks and complex schedulers. By embracing open standards, exploring alternative memory extension technologies, and seeking virtualization solutions beyond CUDA, organizations can design more efficient and scalable AI server infrastructure.