Building Software Systems At Google and Lessons Learned

Transcript

Stanford University. Okay. Can you hear me? Okay. Okay. Um, welcome to I guess this is E380, but it's also been sort of overridden with our distinguished lecture series. Um, today's speaker is Jeff Dean of Google. And um, you know, Jeff, you know, I don't want to use up all his time describing all his accomplishments, but I'll say he did get his de... Read More

Summary

Jeff Dean of Google discusses the evolution of various systems at Google, including hardware, web search and retrieval systems, infrastructure software, and techniques for building high-performance and reliable systems. He also touches on the challenges of indexing, caching, and availability in large-scale systems.

Questions & Answers

Q: Can you provide an overview of the evolution of Google's computing hardware?

Google started out using a mix of different machines, but eventually built their own hardware using commodity components. They designed their own computers with trays of four machines sharing a power supply. However, this approach led to additional failure modes. They then moved to a rack design without cases, which improved airflow. Google continuously upgraded their hardware to improve computational power and efficiency.

Q: How did Google's search and retrieval systems evolve over time?

The original Google search system was inspired by research on search engines for web pages and used the link structure of the web for ranking. However, as traffic and index sizes grew, they had to scale their systems. They added an ad system, caching servers, and doc servers. Over time, they improved performance and latency, with significant revisions to the search system without changing the user interface.

Q: How did Google handle the growth in index size and traffic?

Google believed that a large index size was crucial for search quality. They partitioned the index and added more machines and replicas to handle the increasing traffic and index size. They also introduced caching servers to improve performance and reduce query latency. However, increasing the index size led to seek operations that slowed down the system. To tackle this, they transitioned to an in-memory index system, which improved throughput and query latency.

Q: How did Google address the issues of variance and availability in the in-memory index system?

In an in-memory index system, variance caused by randomized cron jobs can impact CPU usage and performance. Google learned that spacing out cron jobs at fixed intervals was more beneficial than randomization. Additionally, availability became a concern as machines failed, leading to crashes and data center outages. To mitigate this, Google implemented canary requests, where a request is sent to one machine first, and if it fails, it is not sent to all machines. This prevented widespread crashes and allowed for investigation of problematic queries.

Q: How did Google approach the challenge of integrating different corpora into universal search?

Google introduced universal search to search multiple corpora simultaneously when users go to google.com. However, different corpora had varying traffic levels and ranking functions not optimized for higher traffic. Google had to address performance issues and determine which corpora were relevant for each query. Rather than predicting relevance solely based on the query, Google issued the query to all corpora and used the scores to determine relevance. User interface decisions were also made to organize and present the results from different corpora.

Q: How did Google manage data storage and availability in their systems?

Google developed the Google File System (GFS), optimized for large files. The system had a master that managed file system metadata and distributed the data across multiple chunk servers. Each file was divided into chunks and replicated across multiple machines to tolerate failures. Google clusters consisted of thousands of machines running chunk server processes, with homogeneous hardware configurations. Availability relied on software rather than hardware, as commodity hardware still experienced failures.

Q: Can you explain the design and advantages of GFS master and chunk servers?

The GFS master managed file system metadata, while chunk servers stored and served the actual data. Clients communicated with the master to determine the location of data chunks and read directly from chunk servers. Chunks were replicated across multiple machines for fault tolerance. This design allowed for high read and write bandwidth and efficient processing of large files. It also enabled easy experimentation and scalability in managing the storage and retrieval of data.

Q: What are some of the challenges faced in large-scale system infrastructure?

Large-scale system infrastructure poses various challenges, including individual machine failures, disk drive failures, and network issues. Additionally, long-distance links between data centers can experience unexpected problems like fiber cuts caused by horse graves or drunk hunters. Reliability and availability in such environments need to be ensured through software rather than relying solely on hardware. The ability to store data persistently with high availability and run large-scale computations reliably are crucial aspects of managing large-scale system infrastructure.

Q: Did Google make any improvements to their encoding formats for efficient data storage?

Yes, Google developed a compact and fast decoding format for variable-length integers. By using a one-byte prefix for groups of numbers, they reduced shifting and masking operations needed for decoding. This enabled faster decoding and instruction-level parallelism. Google aimed for performance improvements when reading data stored in their systems.

Q: How did Google deal with machine failures and reliability in their system infrastructure?

Google acknowledged that hardware failures are inevitable in large-scale systems. They focused on using more machines rather than relying on more reliable hardware. Reliability and availability were designed to come from the software layer, making it essential to handle machine failures gracefully. Google preferred having more machines that are duplicable, as it provided more computing power per dollar, as long as software was capable of managing failures efficiently.

Takeaways

Jeff Dean's talk highlighted the evolution of Google's computing hardware, search systems, and infrastructure software. The key takeaways include the need for scalability in handling larger indices and traffic, the adoption of in-memory index systems for improved performance, the challenges in integrating different corpora into universal search, the design advantages of GFS for distributed data storage, and the importance of software for reliability and availability in large-scale systems. Additionally, optimizations in encoding formats and the use of more machines for increased computing power were emphasized.