Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Amazon S3 NYJavaSIG 2024-12-12
Search
sullis
December 12, 2024
Programming
0
190
Amazon S3 NYJavaSIG 2024-12-12
Amazon S3
NYJavaSIG
New York, NY
sullis
December 12, 2024
Tweet
Share
More Decks by sullis
See All by sullis
Dependency management for Java applications 2025-09-11
sullis
0
8
S3 NYC Iceberg meetup 2025-07-10
sullis
0
41
Amazon S3 Chicago 2025-06-04
sullis
0
100
Amazon S3 Boston 2025-05-07
sullis
0
55
Netty ConFoo Montreal 2025-02-27
sullis
0
100
GitHub Actions ConFoo Montreal 2025-02-26
sullis
0
63
Netty Portland Java User Group 2025-02-18
sullis
0
12
Amazon S3 - Portland Java User Group 2024-09-17
sullis
0
110
Netty - Montreal Java User Group 2024-05-21
sullis
0
180
Other Decks in Programming
See All in Programming
The Past, Present, and Future of Enterprise Java
ivargrimstad
0
400
Processing Gem ベースの、2D レトロゲームエンジンの開発
tokujiros
2
130
複雑なドメインに挑む.pdf
yukisakai1225
5
1.2k
MCPとデザインシステムに立脚したデザインと実装の融合
yukukotani
4
1.4k
請來的 AI Agent 同事們在寫程式時,怎麼用 pytest 去除各種幻想與盲點
keitheis
0
120
プロパティベーステストによるUIテスト: LLMによるプロパティ定義生成でエッジケースを捉える
tetta_pdnt
0
3.3k
知っているようで知らない"rails new"の世界 / The World of "rails new" You Think You Know but Don't
luccafort
PRO
1
180
もうちょっといいRubyプロファイラを作りたい (2025)
osyoyu
1
450
HTMLの品質ってなんだっけ? “HTMLクライテリア”の設計と実践
unachang113
4
2.9k
2025 年のコーディングエージェントの現在地とエンジニアの仕事の変化について
azukiazusa1
24
12k
ファインディ株式会社におけるMCP活用とサービス開発
starfish719
0
1.9k
Compose Multiplatform × AI で作る、次世代アプリ開発支援ツールの設計と実装
thagikura
0
170
Featured
See All Featured
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
48
9.7k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
How STYLIGHT went responsive
nonsquared
100
5.8k
We Have a Design System, Now What?
morganepeng
53
7.8k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.1k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Code Review Best Practice
trishagee
71
19k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
36
2.5k
Why You Should Never Use an ORM
jnunemaker
PRO
59
9.5k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
The Cost Of JavaScript in 2023
addyosmani
53
8.9k
Transcript
Amazon S3 NYJavaSIG 2024-12-12 Sean Sullivan
AWS Developers
Agenda AWS re:Invent 2024 Amazon S3 AWS SDK Bonus topics
AWS re:Invent 2024 S3 Tables S3 Metadata
AWS re:Invent 2024 fully managed Iceberg tables
AWS re:Invent 2024 “Table buckets”
AWS re:Invent 2024 Iceberg tables as fi rst-class AWS resources
AWS re:Invent 2024
AWS re:Invent 2024 pom.xml
AWS re:Invent 2024 automatic generation of metadata that is captured
when S3 objects are added or modi fi ed stored in fully managed Apache Iceberg tables
Amazon S3 “Simple Storage Service” Launched on March 14, 2006
Amazon S3 S3 is an object storage service with an
HTTP REST API https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
Amazon S3 “There is a frontend fl eet with a
REST API, a namespace service, a storage fl eet that’s full of hard disks, and a fl eet that does background operations.” https://www.allthingsdistributed.com/2023/07/building-and-operating-a-pretty-big-storage-system.html
S3 core concepts Buckets Objects
S3 core concepts An Amazon S3 object represents a fi
le or collection of data Every object must reside within a bucket
S3 bucket types General purpose buckets Directory buckets Table buckets
S3 bucket names an Amazon S3 bucket name is globally
unique the namespace is shared by all AWS accounts
S3 pricing https://aws.amazon.com/s3/pricing/ “You pay for storing objects in your
S3 buckets. The rate you’re charged depends on your objects' size, how long you stored the objects during the month, and the storage class”
S3 storage classes https://aws.amazon.com/s3/storage-classes/ “Amazon S3 o ff ers a
range of storage classes that you can choose from based on the performance, data access, resiliency, and cost requirements of your workloads.”
S3 storage classes https://aws.amazon.com/s3/storage-classes/ Standard Intelligent Tiering Express One Zone
… and many others
Storage class choice matters https://www.youtube.com/watch?v=RxgYNrXPOLw
S3 REST API 3314 pages
S3 operations Upload object List objects Download object Copy Move
Delete
using S3 in a Java application
AWS SDK for Java v1 AWS SDK for Java v2
AWS SDK for Kotlin
AWS SDK for Java v1 https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/
Open source SDK’s
AWS SDK for Java v2 pom.xml
AWS SDK for Java v2 pom.xml
AWS SDK for Java v2 : HTTP clients Apache Client
Netty Client CRT Client
AWS SDK for Java v2: CRT client pom.xml
CRT @ AWS re:Invent 2024 https://youtu.be/2DSVjJTRsz8?t=833
Di ff erent fl avors of S3 clients Async Sync
how to create an S3 bucket?
Creating an S3 bucket AWS Console UI AWS CLI AWS
SDK CloudFormation AWS CDK Terraform Pulumi Infrastructure as Code Other
CloudFormation
Pulumi
s3-playground https://github.com/sullis/s3-playground
s3-playground https://github.com/sullis/s3-playground
testing S3 locally Localstack MinIO Adobe S3Mock Testcontainers Testcontainers Testcontainers
S3 with MinIO S3MinioTest.java
how to upload an object? PutObjectRequest
how to retrieve an object? GetObjectRequest
how to upload large objects? CreateMultipartUploadRequest
parallel uploads? S3TransferManager
Big Data analytics?
Iceberg @ Net fl ix 2018 June 2018 https://www.youtube.com/watch?v=nWwQMlrjhy0 S3
Apache Iceberg 2024 A table format is a method of
structuring a dataset’s files to present them as a unified “table.”
Apache Iceberg 2024 In a data lake, all your data
is stored as files in some storage solution (e.g. Amazon S3)
AWS re:Invent 2023 Ryan Blue
AWS re:Invent 2023 S3
AWS re:Invent 2023
AWS re:Invent 2023 “Too many small fi les are a
problem”
Apache Iceberg project https://github.com/apache/iceberg
Iceberg pull request https://github.com/apache/iceberg/pull/11349
S3 Conditional Writes https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/ Conditional writes can ensure there is
no existing object with the same key name in you bucket during PUT operations
S3 bucket permissions https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleCon fi guration.html By default, all Amazon
S3 resources are private, including buckets, objects, and related subresources
S3 performance considerations?
S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "your application can achieve at least 3,500
PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned Amazon S3 pre fi x”
S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html "There are no limits to the number
of pre fi xes in a bucket. You can increase your read or write performance by using parallelization”
S3 performance https://docs.aws.amazon.com/AmazonS3/latest/userguide/optimizing-performance.html “While Amazon S3 is scaling to your
new higher request rate, you may see some 503 (Slow Down) errors. These errors will dissipate when the scaling is complete.”
KubeCon November 2024
re:Invent December 2024
The End
Bonus content
AWS re:Invent 2024
AWS re:Invent 2024
AWS re:Invent 2024
AWS re:Invent 2024
AWS re:Invent 2024
AWS re:Invent 2024