Amazon S3 Boston 2025-05-07

Boston Lakehouse Meetup
2025-05-07
Cambridge Massachusetts

Transcript

1. Amazon S3 Sean Sullivan Boston Lakehouse meetup May 7, 2025
3. Developers AWS
4. Iceberg table format
5. S3 buckets
6. org.apache.iceberg.aws.s3.S3FileIO Developers AWS
8. Agenda AWS re:Invent 2024 Amazon S3 AWS SDK Bonus topics
9. AWS re:Invent 2024 S3 Tables S3 Metadata
10. AWS re:Invent 2024 fully managed Iceberg tables
11. AWS re:Invent 2024 “Table buckets”
12. AWS re:Invent 2024 Iceberg tables as fi rst-class AWS resources
13. AWS re:Invent 2024
14. AWS re:Invent 2024 pom.xml
15. AWS re:Invent 2024 automatic generation of metadata that is captured when S3 objects are added or modi fi ed stored in fully managed Apache Iceberg tables
16. Amazon S3 “Simple Storage Service” Launched on March 14, 2006
17. Amazon S3 S3 is an object storage service with an HTTP REST API https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
18. Amazon S3 “There is a frontend fl eet with a REST API, a namespace service, a storage fl eet that’s full of hard disks, and a fl eet that does background operations.” https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
19. S3 core concepts Buckets Objects
20. S3 core concepts An Amazon S3 object represents a fi le or collection of data Every object must reside within a bucket
21. S3 bucket types General purpose buckets Directory buckets Table buckets
22. S3 bucket names an Amazon S3 bucket name is globally unique the namespace is shared by all AWS accounts
23. S3 pricing https://aws.amazon.com/s3/pricing/ “You pay for storing objects in your S3 buckets. The rate you’re charged depends on your objects' size, how long you stored the objects during the month, and the storage class”
24. S3 storage classes https://aws.amazon.com/s3/storage-classes/ “Amazon S3 o ff ers a range of storage classes that you can choose from based on the performance, data access, resiliency, and cost requirements of your workloads.”
25. S3 storage classes https://aws.amazon.com/s3/storage-classes/ Standard Intelligent Tiering Express One Zone … and many others
26. Storage class choice matters https://www.youtube.com/watch?v=RxgYNrXPOLw
27. S3 REST API 3314 pages
28. S3 operations Upload object List objects Download object Copy Move Delete
29. using S3 in a Java application
30. AWS SDK for Java v1 AWS SDK for Java v2 AWS SDK for Kotlin
31. AWS SDK for Java v1 https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/
32. Open source SDK’s
33. AWS SDK for Java v2 pom.xml
34. AWS SDK for Java v2 pom.xml
35. AWS SDK for Java v2 : HTTP clients Apache Client Netty Client CRT Client
36. AWS SDK for Java v2: CRT client pom.xml
37. CRT @ AWS re:Invent 2024 https://youtu.be/2DSVjJTRsz8?t=833
38. Di ff erent fl avors of S3 clients Async Sync
39. how to create an S3 bucket?
40. Creating an S3 bucket AWS Console UI AWS CLI AWS SDK CloudFormation AWS CDK Terraform Pulumi Infrastructure as Code Other
41. CloudFormation
42. Pulumi
43. testing S3 locally Localstack MinIO Adobe S3Mock Testcontainers Testcontainers Testcontainers
44. S3 with MinIO S3MinioTest.java
45. how to upload an object? PutObjectRequest
46. how to retrieve an object? GetObjectRequest
47. how to upload large objects? CreateMultipartUploadRequest
48. parallel uploads? S3TransferManager
49. Iceberg @ Net fl ix 2018 June 2018 https://www.youtube.com/watch?v=nWwQMlrjhy0 S3
50. AWS re:Invent 2023 Ryan Blue
51. AWS re:Invent 2023
52. AWS re:Invent 2023 “Too many small fi les are a problem”
53. Apache Iceberg project https://github.com/apache/iceberg
54. Iceberg pull request https://github.com/apache/iceberg/pull/11349
55. S3 Conditional Writes https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ Conditional writes can ensure there is no existing object with the same key name in you bucket during PUT operations
56. S3 bucket permissions https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleCon fi guration.html By default, all Amazon S3 resources are private, including buckets, objects, and related subresources
57. S3 performance considerations?
58. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 pre fi x”
59. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "There are no limits to the number of pre fi xes in a bucket. You can increase your read or write performance by using parallelization”
60. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html “While Amazon S3 is scaling to your new higher request rate, you may see some 503 (Slow Down) errors. These errors will dissipate when the scaling is complete.”
61. KubeCon November 2024
62. re:Invent December 2024
63. S3
64. Thank you!
65. S3 resources
66. AWS re:Invent 2024
67. AWS re:Invent 2024
68. AWS re:Invent 2024
69. AWS re:Invent 2024
70. AWS re:Invent 2024
71. AWS re:Invent 2024
72. Bonus content
73. I was just hoping you might give me some insight into the evolution of the Apache Iceberg table specification