"Choosing and Comparing Network Architectures for Smart Computing Centers: Optimizing End-to-End Communication Latency"

"Choosing and Comparing Network Architectures for Smart Computing Centers: Optimizing End-to-End Communication Latency"

Introduction:
In the realm of smart computing centers, reducing end-to-end communication latency between multiple machines and multiple cards is crucial. One key technology that accomplishes this is RDMA (Remote Direct Memory Access), which bypasses the operating system kernel to enable direct memory access between host machines. There are four main methods of implementing RDMA: InfiniBand, RoCEv1, RoCEv2, and iWARP. Among these, RoCEv1 is currently outdated and iWARP is less commonly used. The two primary solutions for RDMA technology currently in use are InfiniBand and RoCEv2. By bypassing the kernel protocol stack, these solutions offer significant improvements in latency performance compared to traditional TCP/IP networks.

Comparing InfiniBand and RoCEv2:
In the case of scenarios where communication within the same cluster is only one hop away, InfiniBand and RoCEv2 demonstrate significantly reduced end-to-end latency compared to traditional IP networks. Laboratory test data shows that bypassing the kernel protocol stack can reduce application-layer end-to-end latency from 50us (TCP/IP) to 5us (RoCE) or even 2us (InfiniBand). InfiniBand networks consist of key components such as the Subnet Manager (SM), InfiniBand network cards, InfiniBand switches, and InfiniBand cables. NVIDIA introduced the 400Gbps Quantum-2 series switch in 2021, featuring 32 800G OSFP ports that require cable conversion to 64 400G QSFP ports. InfiniBand switches do not run any routing protocols, as the entire network's forwarding table is calculated and distributed by a centralized Subnet Manager (SM). The SM also manages InfiniBand subnet configurations, including partitions and QoS, in addition to the forwarding table. Specialized cables and optical modules are required for interconnections between InfiniBand switches and network cards. InfiniBand's Adaptive Routing, based on per-packet dynamic routing, ensures optimal network utilization in massively scaled networks. InfiniBand networks have been successfully deployed in large GPU clusters, including those of Baidu Intelligent Cloud and Microsoft Azure. Notable InfiniBand network solution and equipment providers in the market include NVIDIA, Intel Corporation, Cisco Systems, and Hewlett Packard Enterprise (HPE). NVIDIA holds the majority market share, exceeding 70%.

RoCEv2, on the other hand, offers greater versatility and relatively lower prices compared to InfiniBand. Apart from constructing high-performance RDMA networks, RoCEv2 can also be used in traditional Ethernet networks. However, configuring parameters such as Headroom, PFC (Priority-based Flow Control), and ECN (Explicit Congestion Notification) on switches can be complex. In massively scaled scenarios like those with thousands of cards, the throughput performance of RoCEv2 networks may be slightly weaker compared to InfiniBand networks. Notable switch vendors supporting RoCE include H3C (New H3C) and Huawei, with NVIDIA's ConnectX series of network cards holding a significant market share.

Optimizing Large-scale Model Inference:
When performing each step of decode inference, attention plays a crucial role. Attention calculates the relevance between the query (Q) and the key (K) and value (V) matrices. While the query is based on the output token from the previous step, the key and value matrices incorporate information not only from the current token but also from all previous tokens, including user input prompts. Consequently, the multiplication operation between Q and K becomes a vector-matrix multiplication, resulting in an attention matrix that is also a vector. This vector is then multiplied with the V matrix. This transformation of the core matrix multiplication operation into vector-matrix multiplication is significantly different for GPUs. The former can be efficiently computed on tensor cores with greater computational power, while the latter is typically computed on vector cores. Decode inference requires the K and V tensors to contain information from both the prompt and the history of model-generated tokens, which increases as the inference steps progress. In this attention computation, the input consists of complete prompts with a certain length, generating equally long Q, K, and V tensors for matrix multiplication and softmax operations. This process is similar to previous transformer-based model inferences like BERT.

Conclusion:
Choosing the appropriate network architecture for smart computing centers is essential for optimizing end-to-end communication latency. The InfiniBand and RoCEv2 solutions offer significant improvements over traditional TCP/IP networks by bypassing the kernel protocol stack. InfiniBand networks, with their Subnet Manager, network cards, switches, and specialized cables, have been successfully deployed in various large-scale GPU clusters. On the other hand, RoCEv2 provides versatility and lower costs, making it suitable for both RDMA networks and traditional Ethernet networks. When it comes to large-scale model inference, attention computation plays a critical role in optimizing the performance on GPUs. By understanding the nuances of vector-matrix multiplication and efficiently managing the K and V tensors, the inference process can be streamlined.

Actionable Advice:

1. Evaluate the specific needs of your smart computing center and consider the scale and latency requirements before choosing between InfiniBand and RoCEv2 solutions.
2. When implementing InfiniBand networks, ensure proper configuration of the Subnet Manager and take advantage of adaptive routing for optimal network utilization.
3. When optimizing large-scale model inference, pay close attention to attention computation and efficiently manage the K and V tensors to maximize GPU performance.

By considering these factors and implementing suitable network architectures and optimizing large-scale model inference, smart computing centers can achieve significant improvements in end-to-end communication latency and overall performance.