What Is Homa and Why It Improves AI Clusters

TL;DR

Homa is designed to reduce tail latency in data center AI workloads by rethinking congestion control and prioritizing small, latency sensitive messages over large transfers. The talk explains how conventional TCP and RDMA struggle with incast and head of line blocking, and how Homa’s clean slate design targets these issues to improve overall compute utilization in AI clusters.

Transcript

[music] Please welcome to the stage the professor emmeritus at Stanford University, John Sterhout. Good morning. It's really great to be here to talk about the network side of AI applications and in particular to make the case that latency matters and it's probably going to be mattering more in the future. But I just want to say this is a talk is u... Read More

Key Insights

• AI workloads are shifting from throughput dominated transfers to latency sensitive small messages. The key bottleneck is tail latency which can stall GPUs during synchronization.
• Incast congestion from multiple nodes sending to one destination causes queue buildup and drops, increasing delays for all messages including small ones.
• TCP and RDMA rely on sender driven congestion control and a byte stream model, which poorly differentiate message boundaries and hinder prioritization of small messages.
• ECN marks in switches provide congestion signals, but the reaction time and control stability make it hard to optimize without new protocols.
• Head of line blocking in byte streams causes small messages to wait behind large messages, worsening tail latency in mixed traffic scenarios.
• A clean slate redesign like Homa can coordinate congestion control more effectively for data center workloads with mixed message sizes.
• Homa aims to cut tail latency by rethinking how congestion signals are used and how messages are prioritized.
• The presentation argues that a protocol designed specifically for modern AI cluster traffic can achieve substantial reductions in tail latency compared to legacy approaches.

Questions & Answers

Q: What is Homa and why is it needed for AI clusters

Homa is a new protocol designed for data center workloads where small, latency sensitive messages are common alongside large transfers. It addresses tail latency, which becomes a bottleneck when synchronization messages must wait for other traffic. The need arises because legacy protocols like TCP and RDMA rely on sender driven congestion control and a byte stream model, which do not differentiate message boundaries or prioritize short messages. Homa redesigns congestion signaling and message handling to reduce tail latency, leading to better utilization of AI compute resources.

Q: Why do small messages cause problems in traditional networks

Small messages become problematic when they interact with large transfers in a shared network due to incast and queue buildup at the destination. Traditional protocols treat data as a continuous stream of bytes and rely on sender driven control, which makes it hard to quickly detect and react to congestion. This can cause head of line blocking and long tail latencies for the small messages even if the large transfers are proceeding well, leading to GPUs waiting and reduced throughput.

Q: What is incast and how does it affect AI workloads

Incast occurs when several nodes simultaneously transfer data to a single destination, causing multiple high arrival rates to contend for a single ingress queue. This can fill the queue and cause packet drops or delays, which in turn increases tail latency for all traffic directed at that destination. In AI workloads, where synchronization and metadata exchanges are frequent, incast can severely stall progress and waste compute cycles by delaying the next compute phase.

Q: How does ECN help and what are its limits in current systems

ECN marks packets that pass through a congested switch so receivers can inform senders to slow down. While helpful, ECN signaling is limited by the challenge of stable control and the time it takes for senders to adjust their rate in response to congestion. In large and dynamic AI clusters, relying solely on ECN with traditional congestion control can still lead to oscillations and suboptimal utilization because feedback is indirect and slow to propagate, especially with mixed traffic patterns.

Q: What distinguishes a clean slate design like Homa from TCP RDMA

A clean slate design starts from the ground up with an architecture tailored to the realities of modern AI traffic, including many small, latency sensitive messages and mixed workloads. It can differentiate message boundaries, prioritize short messages, and implement congestion control that responds quickly to local conditions without being overly conservative or unstable. This contrasts with TCP and RDMA where the byte stream model and sender driven control make it hard to achieve low tail latency in the presence of incast and head of line blocking.

Q: What workloads benefit most from Homa according to the talk

Workloads that involve frequent small exchanges such as metadata lookups, barrier synchronizations, and timely coordination across GPUs benefit most. In these scenarios, latency and tail latency dominate performance because even a few slow message exchanges can stall compute progress. Inference and agentic workloads, where iterations require quick synchronizations, stand to gain significant improvements from a protocol designed to minimize tail latency and efficiently handle mixed message sizes.

Q: How does tail latency impact GPU utilization in AI pipelines

Tail latency determines how long a given synchronization step can hold up the next compute phase. If one small message is slow, all subsequent steps must wait, leaving GPUs idle while waiting for data exchanges to complete. Reducing tail latency means synchronization completes quickly, enabling more consistent compute progress and higher effective throughput across the pipeline.

Q: What is the expected performance benefit of Homa

Homa is claimed to reduce tail latency by an order of magnitude or more in data center workloads that mix small and large messages. This implies a substantial improvement in overall AI cluster utilization, as faster and more predictable synchronization allows GPUs to stay busy with compute rather than waiting on network delays. The performance gains come from targeted congestion signaling and message handling that aligns with latency sensitive operations.

Summary & Key Takeaways

• Homa targets a shift in AI workloads from large bulk transfers to frequent small messages, where latency and tail latency become the bottleneck. It explains the problems with legacy protocols that rely on sender driven congestion control and a byte stream model.

• The talk outlines how congestion and incast cause queueing, drops, and retransmissions, and argues that early congestion notification and a different data plane design are needed for modern data center traffic.

• Homa is presented as a redesign that reduces tail latency by an order of magnitude or more, aligning protocol behavior with the needs of small message exchanges in AI inference and coordination tasks.