Amazon S3 Chicago 2025-06-04

Chicago Java User Group (CJUG)
June 4 2025

Transcript

1. Amazon S3 Sean Sullivan Chicago Java User Group June 4, 2025
2. S3
4. Amazon S3 AWS re:Invent 2024 AWS SDK Hawley-Smoot Tariff Act
5. “Simple Storage Service” Launched on March 14, 2006
6. S3 is an object storage service with an HTTP REST API
7. Amazon S3 “There is a frontend fl eet with a REST API, a namespace service, a storage fl eet that’s full of hard disks, and a fl eet that does background operations.” https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
8. Amazon S3 Buckets Objects
9. S3 core concepts An Amazon S3 object represents a fi le or collection of data Every object must reside within a bucket
10. S3 bucket types General purpose buckets Directory buckets Table buckets
11. S3 directory buckets With S3 Express One Zone, data is stored in a different bucket type — an S3 directory bucket — which can support up to 2 million requests per second aws.amazon.com
12. Structured Data Unstructured Data S3
13. Open table formats
15. Iceberg table format
16. November 2024
17. org.apache.iceberg.aws.s3.S3FileIO
19. AWS re:Invent 2024 S3 Tables S3 Metadata
20. AWS re:Invent 2024 fully managed Iceberg tables
21. AWS re:Invent 2024 “Table buckets”
22. AWS re:Invent 2024 Iceberg tables as fi rst-class AWS resources
23. AWS re:Invent 2024
24. AWS re:Invent 2024 pom.xml
25. AWS re:Invent 2024 automatic generation of metadata that is captured when S3 objects are added or modi fi ed stored in fully managed Apache Iceberg tables
26. S3 bucket names an Amazon S3 bucket name is globally unique the namespace is shared by all AWS accounts
27. S3 pricing https://aws.amazon.com/s3/pricing/ “You pay for storing objects in your S3 buckets. The rate you’re charged depends on your objects' size, how long you stored the objects during the month, and the storage class”
28. S3 storage classes https://aws.amazon.com/s3/storage-classes/ “Amazon S3 o ff ers a range of storage classes that you can choose from based on the performance, data access, resiliency, and cost requirements of your workloads.”
29. S3 storage classes https://aws.amazon.com/s3/storage-classes/ Standard Intelligent Tiering Express One Zone … and many others
30. Storage class choice matters https://www.youtube.com/watch?v=RxgYNrXPOLw
31. S3 REST API 3314 pages
32. S3 operations Upload object List objects Download object Copy Move Delete
33. using S3 in a Java application
34. AWS SDK for Java v1 AWS SDK for Java v2 AWS SDK for Kotlin
35. AWS SDK for Java v1 https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/
36. Open source SDK’s
37. AWS SDK for Java v2 pom.xml
38. AWS SDK for Java v2 pom.xml
39. AWS SDK for Java v2 : HTTP clients Apache Client Netty Client CRT Client
40. AWS SDK for Java v2: CRT client pom.xml
41. CRT @ AWS re:Invent 2024 https://youtu.be/2DSVjJTRsz8?t=833
42. Iceberg S3_CRT_ENABLED
43. Di ff erent fl avors of S3 clients Async Sync
44. how to create an S3 bucket?
45. Creating an S3 bucket AWS Console UI AWS CLI AWS SDK CloudFormation AWS CDK Terraform Pulumi Infrastructure as Code Other
46. CloudFormation
47. Pulumi
48. testing S3 locally Localstack MinIO Adobe S3Mock Testcontainers Testcontainers Testcontainers
49. S3 with MinIO S3MinioTest.java
50. how to upload an object? PutObjectRequest
51. how to retrieve an object? GetObjectRequest
52. how to upload large objects? CreateMultipartUploadRequest
53. parallel uploads? S3TransferManager
54. Iceberg @ Net fl ix 2018 June 2018 https://www.youtube.com/watch?v=nWwQMlrjhy0 S3
55. AWS re:Invent 2023 Ryan Blue
56. AWS re:Invent 2023
57. AWS re:Invent 2023 “Too many small fi les are a problem”
58. Apache Iceberg project https://github.com/apache/iceberg
59. Iceberg pull request https://github.com/apache/iceberg/pull/11349
60. S3 Conditional Writes https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ Conditional writes can ensure there is no existing object with the same key name in you bucket during PUT operations
61. S3 bucket permissions https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleCon fi guration.html By default, all Amazon S3 resources are private, including buckets, objects, and related subresources
62. S3 performance considerations?
63. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 pre fi x”
64. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "There are no limits to the number of pre fi xes in a bucket. You can increase your read or write performance by using parallelization”
65. S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html “While Amazon S3 is scaling to your new higher request rate, you may see some 503 (Slow Down) errors.” “These errors will dissipate when the scaling is complete.”
66. Kafka and S3
67. WarpStream November 2023 September 2024
68. WarpStream
69. Diskless Kafka https://aiven.io/blog/diskless-kafka-is-the-tide-and-its-rising
70. KIP-1150
71. KIP-1150 “Apache Kafka is often operated in cloud hyperscaler environments where high-reliability object storage is available and more cost- effective than block storage for equivalent workloads.”
72. Exiting S3
73. 37 Signals : S3 Exit
74. 37 Signals : S3 Exit
75. 37 Signals : S3 Exit
76. 37 Signals : S3 Exit May 2025
77. Data transfer
78. Final thoughts
79. KubeCon November 2024
80. re:Invent December 2024
81. Questions?
82. Thank you!
83. S3 resources
84. AWS re:Invent 2024
85. AWS re:Invent 2024
86. AWS re:Invent 2024
87. AWS re:Invent 2024
88. AWS re:Invent 2024
89. AWS re:Invent 2024
90. Bonus content