How We've Scaled Dropbox

Transcript

Stanford University Welcome to dou 380 winter 201122 I'm Andy Freeman the other course organizer is Dennis Allison we're approaching the end of the quarter so if you're taking the class for credit please caught up remember no incompletes um we've talked a lot about large systems and scalable systems but we haven't talked much if at all about rapidl... Read More

Summary

This video is a talk by Kevin Modzelewski, the server team lead at Dropbox. He discusses the technical challenges and solutions Dropbox faced to handle rapid growth and scaling with limited resources. He focuses on the back-end architecture of Dropbox, including the file synchronization and metadata storage systems.

Questions & Answers

Q: What is the main focus of this talk?

The main focus of this talk is to discuss the technical challenges of rapidly growing systems and how Dropbox dealt with them.

Q: What are some technical challenges faced by Dropbox?

Dropbox faced challenges in terms of high read and write volumes, maintaining consistency and correctness, and scaling the back-end infrastructure to handle the growing demand.

Q: How does Dropbox handle the high read and write volumes?

Unlike most web applications, Dropbox has a read-to-write ratio of approximately 1-to-1 because each client has a complete copy of their entire Dropbox. This requires a different approach to caching and scaling compared to other companies. Dropbox measures its cache in petabytes, which means they have to consider a different set of cacheability rules.

Q: What are the implications of Dropbox's high consistency and correctness requirements?

Dropbox cannot afford to be wrong in scenarios where it involves user expectations, such as sharing files and folders. This means Dropbox has to prioritize the correctness of its operations and ensure high levels of consistency, even if it means sacrificing other factors like availability or performance.

Q: How did Dropbox's back-end architecture evolve over time?

The initial architecture consisted of a single server running all the application servers, web servers, and database instances. As Dropbox grew, they added more servers for file uploading and downloading, split the workload into separate meta and block servers, and added caching and load balancing layers. The architecture today remains similar to the initial version, but with multiple copies of each component for scalability and availability.

Q: What were some challenges in scaling the database tier?

Scaling the database tier was challenging because the assumptions about running in a single transaction had to be changed. Dropbox had to identify and resolve dependencies on the assumption of a single database, and instead, use sharding, partitioning, and caching techniques to improve performance and scalability.

Q: How does Dropbox handle security and encryption?

Although specific details cannot be provided, Dropbox takes security and privacy seriously and responds aggressively to any security incidents. They ensure encryption and protection of user data while maintaining user-friendly access and sharing features.

Q: How does Dropbox handle deduplication of files?

Dropbox uses block-level deduplication, where files are divided into blocks and each block is assigned a hash. If two files have blocks with the same hash, only one copy is stored in the storage tier.

Q: How does Dropbox handle file sharing and collaboration?

Dropbox faces challenges in shared folders and collaborative environments that cross user boundaries. These scenarios require careful consideration of data storage, queries, and consistency to ensure accurate and efficient sharing of files and folders.

Q: What metrics and monitoring does Dropbox use?

Dropbox monitors server load, request rates, request breakdown, response times, bandwidth usage, and other relevant metrics to track system performance and troubleshoot issues.

Q: How has Dropbox evolved its instrumentation and monitoring over time?

Initially, simple tools like "top" were used for monitoring. However, as the system grew, Dropbox built out better graphing, trending, and monitoring tools to gain insight into system behavior and diagnose performance problems.

Takeaways

Dropbox successfully managed rapid growth and scaling by evolving its back-end architecture and addressing technical challenges as they arose. They prioritized consistency and correctness while handling high read and write volumes. Dropbox also leveraged caching, load balancing, and storage optimizations to improve performance and scalability. The company takes security and privacy seriously, continuously monitors system metrics, and responds proactively to maintain user satisfaction.