Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Batch Processing
Search
Rodrigo Graciano
April 11, 2022
Programming
0
290
Batch Processing
Deck from the Batch Processing in Action presentation
Rodrigo Graciano
April 11, 2022
Tweet
Share
More Decks by Rodrigo Graciano
See All by Rodrigo Graciano
Your Java isn't the same
graciano
0
150
Cloud Batch
graciano
0
130
Java 9 ao 17 - Oracle no TDC
graciano
0
96
KnoxJava - Java 9-17
graciano
0
140
Of concurrency and other demons
graciano
0
94
Java 9 to Java 16: A review of recent changes to the language
graciano
0
110
Java 9 ao 15 - Evolução da Linguagem Java
graciano
0
240
Other Decks in Programming
See All in Programming
ECS Service Connectのこれまでのアップデートと今後のRoadmapを見てみる
tkikuc
2
250
Generative AI Use Cases JP (略称:GenU)奮闘記
hideg
1
300
初めてDefinitelyTypedにPRを出した話
syumai
0
420
Macとオーディオ再生 2024/11/02
yusukeito
0
370
型付き API リクエストを実現するいくつかの手法とその選択 / Typed API Request
euxn23
8
2.2k
Snowflake x dbtで作るセキュアでアジャイルなデータ基盤
tsoshiro
2
520
Click-free releases & the making of a CLI app
oheyadam
2
120
CSC509 Lecture 11
javiergs
PRO
0
180
Flutterを言い訳にしない!アプリの使い心地改善テクニック5選🔥
kno3a87
1
200
距離関数を極める! / SESSIONS 2024
gam0022
0
290
3 Effective Rules for Using Signals in Angular
manfredsteyer
PRO
1
100
[Do iOS '24] Ship your app on a Friday...and enjoy your weekend!
polpielladev
0
110
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
The Cost Of JavaScript in 2023
addyosmani
45
6.8k
How to Think Like a Performance Engineer
csswizardry
20
1.1k
Thoughts on Productivity
jonyablonski
67
4.3k
Into the Great Unknown - MozCon
thekraken
32
1.5k
Music & Morning Musume
bryan
46
6.2k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
28
2k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.1k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
47
2.1k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
0
100
Measuring & Analyzing Core Web Vitals
bluesmoon
4
130
Why You Should Never Use an ORM
jnunemaker
PRO
54
9.1k
Transcript
BATCH PROCESSING IN ACTION HILLMER CHONA RODRIGO GRACIANO
WHAT SPRING BATCH IS? Spring Batch is a lightweight, comprehensive
batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
WHAT SPRING BATCH IS NOT? Spring Batch is not a
scheduler https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
BATCH VOCABULARY
THE BASICS
SHOW ME SOME CODE: BASIC DEMO This Photo by Unknown
Author is licensed under CC BY-NC-ND
JOB STEP REPOSITORY RUNNING
BATCH ARCHITECTURE (SIMPLIFIED ) https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader
Writer 1 * 1 1 1 1
JOB REPOSITORY •It’s how Job, Step, JobParameters, JobExecution, and StepExecution
are persisted
• JobRepository • JobLauncher • JobRegistry • JobExplorer • PlatformTransactionManager
• JobBuilderFactory • StepBuilderFactory @EnableBatchProcessing
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container
SHOW ME SOME CODE: RUNNING JOBS This Photo by Unknown
Author is licensed under CC BY-NC-ND
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container • Synchronous x Asynchronous
PROCESSORS CHUNKS WE CAN DO BETTER
STEP SEQUENCE Process Step Read Write
BATCH ARCHITECTURE https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1
* 1 1 1 1 Processor 1 1
PROCESSORS •Transforming •Filtering •Validations
STEP SEQUENCE Process Step Read Write
CHUNK SEQUENCE Process Step Read Write Read Process
MORE CODE MORE CODE: PROCESSORS & CHUNKS This Photo by
Unknown Author is licensed under CC BY-NC-ND
+READERS +WRITERS TASKLETS
READERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file
MULTI-FILE READER •Read multiple file in a single step •Files
must have the same format •Works with XML, JSON and flat files
SHOW ME SOME CODE: MULTI-READERS This Photo by Unknown Author
is licensed under CC BY-NC-ND
WRITERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file • Email
TASKLETS • 1 time operation instead of read/write • Run
a DB query • Copy Files • Zip/unzip files
BEYOND THE BASICS LISTENERS STEP FLOW SCALING AND PARALLEL PROCESSING
LISTENERS JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener ChunkListener ItemProcessListener SkipListener
JOB EXECUTION LISTENERS
STEP LISTENERS • before/after step StepExecutionListener • before/after read •
onReadError ItemReadListener • before/after write • onWriteError ItemWriteListener
STEP LISTENERS • before/after/after error chunk ChunkListener • before/after process
• onProcessError ItemProcessListener • onSkipInRead • onSkipInProcess • onSkipInWrite SkipListener
STEP FLOW • Sequential
STEP FLOW: SEQUENTIAL Step 1 Step 2 Step 3
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider
STEP FLOW: CONDITIONAL & DECIDER Condition Step 1 Step 3
Step 2
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider •
Split
STEP FLOW: SPLIT Step 1 Step 2 Step 3 Start
Step 4
SCALING AND PARALLEL PROCESSING • Multi-threaded (Async Executor) • Parallel
steps (Split) • Remote Chunking (Data processed by slaves – Real data sent not a reference) • Remote Partitioning (Data in chunks and local to slaves)
MORE FUNCTIONALITIES RESTARTING SKIPING FAILURES RETRY
USEFUL FUNCTIONALITIES - START •startLimit(n) •allowStartIfComplete(boolean)
USEFUL FUNCTIONALITIES – SKIPPING FAILURES •faultTolerant() •skip(SomeException.class) •skipLimit(n) •noSkip(SomeException.class) •noRollback(SomeException.class)
USEFUL FUNCTIONALITIES - RETRY •faultTolerant() •retry(SomeException.class) •retryLimit(n)
CODE CODE This Photo by Unknown Author is licensed under
CC BY-NC-ND
QUESTIONS? https://github.com/rodrigolgraciano/batch-demo.git https://speakerdeck.com/graciano/batch-processing @rodrigograciano @ HillmerCh
CHUNK-ORIENTED STEP https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png
JOB Job JobInstance JobExecution JobParameters Present at Devnexus title =
Batch Processing in Action Present Batch Processing in Action at Devnexus First try to present – hopefully, it will work
…STEP Job JobInstance JobExecution Step StepExecution Read count Write count
Filter count…
CHUNK-ORIENTED STEP + PROCESSORS https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png