S3 NYC Iceberg meetup 2025-07-10

S3 object storage
NYC Iceberg meetup
2025-07-10

Transcript

1. S3 object storage Sean Sullivan NYC Iceberg meetup July 10 2025
2. 🔵 Amazon S3 🔵 Iceberg and S3 🔵 Kafka and S3
3. Launched on March 14, 2006 S3 “Simple Storage Service”
4. S3 is an object storage service with a HTTP REST API
5. S3
6. Objects are stored in buckets
7. 🔵 General purpose buckets 🔵 Directory buckets 🔵 Table buckets
8. With S3 Express One Zone, data is stored in a different bucket type — an S3 directory bucket — which can support up to 2 million requests per second Directory buckets
9. select a specific Availability Zone within an AWS Region to store your data S3 Express One Zone
10. co-locate your storage and compute resources in the same Availability Zone to further optimize performance S3 Express One Zone
11. “Table buckets” were announced at AWS re:Invent 2024 Table buckets
12. S3 Tables fully managed Iceberg tables Iceberg tables as a first class AWS resource
14. S3 Analytics Accelerator
15. S3 Analytics Accelerator 📍 integrates with AWS CRT 📍 sequential prefetching 📍 small object prefetching
16. S3 Analytics Accelerator 📍 Parquet footer caching 📍 predictive column prefetching
17. S3 Analytics Accelerator
18. Iceberg S3 properties s3.analytics-accelerator.enabled=true S3FileIOProperties
20. July 2023: WarpStream WarpStream is an Apache Kafka protocol compatible data streaming platform built directly on top of S3
21. June 2024: WarpStream
22. May 2025: KIP-1150
24. “Apache Kafka is often operated in cloud hyperscaler environments where high-reliability object storage is available and more cost-effective than block storage for equivalent workloads”
26. https://github.com/aiven/inkless
28. The End
29. Bonus content
30. RisingWave: S3 as primary storage
33. S3 Analytics Accelerator