Mai-Lan Tomsen Bukovec on How Amazon S3 Scales

Transcript

AWS S3 is the world's largest cloud storage service, but just how big is it and how is it engineered to be as reliable as it is at such a massive scale? Milon is the VP of data and analytics at AWS and has been running S3 for 13 years. Today we discuss the sheer scale of S3 in the data stored and the number of servers it runs on. How seemingly over... Read More

Key Insights

• S3 holds over 500 trillion objects and hundreds of exabytes of data, and serves hundreds of millions of transactions per second. It processes over a quadrillion requests every year, which puts it among the largest storage systems in the world.
• The physical base of S3 is tens of millions of hard drives across millions of servers. These sit in racks inside buildings, spread across 120 availability zones in 38 regions. Stacked one on top of another, the drives would reach the International Space Station and nearly back.
• An exabyte is a thousand petabytes. Some individual S3 customers store exabytes of data in what they call a data lake. Sony's group CEO called such a store a data ocean, because at exabyte scale the term lake no longer fits.
• S3 originally used eventual consistency: a put was acknowledged only once the data was stored, but a list might not show the new object right away. This design favored durability and availability and worked well for e-commerce sites, where a human could just refresh.
• Data lakes on S3 came about when frontier data customers like Netflix and Pinterest paired Apache Hadoop with S3's unlimited storage, solid performance and low price. Those cloud-native companies adopted data lakes around 2013 to 2015, and enterprises adopted the same pattern from about 2015 to 2020.
• Apache Iceberg is an open-source table format that adds table attributes to Parquet data stored in S3. Customers value it because any Iceberg-compliant analytics engine can read the data, which allows a decentralized analytics architecture and lets teams swap engines or adopt new AI tools later.
• The core of S3 is the put and the get: storing data in and reading it back out at scale. AWS deliberately keeps the developer experience simple. When engineers decide what to build next, they start from how to keep S3 easy to use.
• Conditional writes let developers make S3 operations depend on the current state of an object. AWS added put if absent and put if match first, then copy if absent and delete if match, so developers no longer have to build that coordination themselves.

Questions & Answers

Q: How big is Amazon S3?

Amazon S3 stores over 500 trillion objects and hundreds of exabytes of data. It serves hundreds of millions of transactions per second and processes over a quadrillion requests a year. Its hardware includes tens of millions of hard drives across millions of servers, spread over 120 availability zones in 38 regions. Stacked up, those drives would reach the International Space Station and nearly back.

Q: When did Amazon S3 launch and why was it built?

S3 development started in 2005, and S3 launched in 2006 as the first AWS service. Amazon engineers had large amounts of unstructured data, such as PDFs, images and backups, and needed somewhere to store it at a low enough price that they could stop worrying about storage growth. Many customers at the time were building e-commerce websites like Amazon.com, and S3 was designed for that kind of storage.

Q: What is eventual consistency in Amazon S3?

S3's original 2006 design used eventual consistency. S3 acknowledged a put only once it actually had the data, but a list request might not show the new object straight away. That consistency model let S3 optimize for durability and availability. It suited e-commerce sites, where an image that was not yet visible was acceptable because the user could refresh. S3 later moved to strong consistency.

Q: How did S3 become the foundation for data lakes?

Apache Hadoop began as a community in 2006, the same year S3 launched. Frontier data customers like Netflix and Pinterest combined Hadoop with S3's unlimited storage, good performance and low price to build what came to be called data lakes, extending S3 from unstructured to tabular data. These cloud-native companies adopted data lakes around 2013 to 2015, and enterprises followed from about 2015 to 2020.

Q: What is Apache Iceberg and why do S3 customers use it?

Apache Iceberg is an open-source table format for large-scale analytic workloads. It gives table attributes to Parquet data stored in S3. Customers use it to run a decentralized analytics architecture: teams can choose any analytics engine as long as it is Iceberg compliant. That also gives chief data officers and CTOs a way to replace analytics tools or adopt new AI tools without moving their data.

Q: What are Amazon S3 Tables and S3 Vectors?

S3 Tables launched in December 2024, after AWS saw many large customers running Iceberg tables on top of Parquet data in S3. AWS added more than 15 features to S3 Tables in the following year. S3 Vectors entered preview in July and became generally available about a week before the interview. Both extend S3 beyond unstructured storage to newer data and AI uses.

Q: What are the basic operations an engineer should know when using S3?

S3 is built around the put, which stores data, and the get, which reads it back. Other primitives, such as delete, list and copy, sit alongside these, and AWS has added many capabilities over time. Simplicity remains the guiding principle: when S3 engineers decide what to build next, they ask how to keep S3 easy for developers to use.

Q: What are conditional writes in Amazon S3?

Conditional writes make an S3 operation depend on the state of the object it targets. AWS first added put if absent and put if match, then copy if absent and delete if match. With these, developers can perform puts and other operations based on whether an object exists or matches an expected version, instead of building that coordination logic themselves. The additions were driven by what developers were trying to do.

Summary & Key Takeaways

• Amazon S3 stores over 500 trillion objects and hundreds of exabytes of data. It serves hundreds of millions of transactions per second and over a quadrillion requests a year. Underneath are tens of millions of hard drives in millions of servers, spread across 120 availability zones in 38 regions worldwide.

• S3 development began in 2005, and it launched in 2006 as the first AWS service. It was built for unstructured data such as PDFs, images and backups, using eventual consistency. That trade-off favored durability and availability, and it suited e-commerce sites, where a user could simply refresh if an image had not appeared yet.

• S3 grew from unstructured storage into the base of data lakes. Frontier customers like Netflix and Pinterest combined it with Hadoop around 2013 to 2015, enterprises followed, and Parquet and then Iceberg brought tabular data. AWS launched S3 Tables in December 2024 and made S3 Vectors generally available after a July preview.