Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Batch Processing
Search
Rodrigo Graciano
April 11, 2022
Programming
0
280
Batch Processing
Deck from the Batch Processing in Action presentation
Rodrigo Graciano
April 11, 2022
Tweet
Share
More Decks by Rodrigo Graciano
See All by Rodrigo Graciano
Your Java isn't the same
graciano
0
140
Cloud Batch
graciano
0
130
Java 9 ao 17 - Oracle no TDC
graciano
0
96
KnoxJava - Java 9-17
graciano
0
140
Of concurrency and other demons
graciano
0
94
Java 9 to Java 16: A review of recent changes to the language
graciano
0
110
Java 9 ao 15 - Evolução da Linguagem Java
graciano
0
240
Other Decks in Programming
See All in Programming
LangChainでWebサイトの内容取得やGitHubソースコード取得
shukob
0
160
ECMAScript、Web標準の型はどう管理されているか / How ECMAScript and Web standards types are maintained
petamoriken
3
390
今インフラ技術をイチから学び直すなら
yuhta28
1
140
あなたのアプリ、ログはでてますか?あるいはログをだしてますか? (Funabashi.dev用 軽量版)
uzulla
2
120
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
0
120
Modern Angular with the NGRX Signal Store New Rules for Your Architecture @BASTA! 2024 in Mainz
manfredsteyer
PRO
0
150
私のEbitengineの第一歩
qt_luigi
0
450
Boost Performance and Developer Productivity with Jakarta EE 11
ivargrimstad
0
490
Go1.23で入った errorsパッケージの小さなアプデ
kuro_kurorrr
2
390
2024 컴포즈 정원사
jisungbin
0
150
はじめてみよう量子プログラミング
itokoichi01
0
210
【TID2024】模擬講義:プログラマと一緒にゲームをデザインしてみよう!
akatsukigames_tech
0
660
Featured
See All Featured
Raft: Consensus for Rubyists
vanstee
135
6.5k
Fashionably flexible responsive web design (full day workshop)
malarkey
401
65k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
158
15k
Building an army of robots
kneath
302
42k
10 Git Anti Patterns You Should be Aware of
lemiorhan
653
58k
How To Stay Up To Date on Web Technology
chriscoyier
786
250k
Optimizing for Happiness
mojombo
375
69k
How to Ace a Technical Interview
jacobian
274
23k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
230
17k
Code Reviewing Like a Champion
maltzj
517
39k
What's in a price? How to price your products and services
michaelherold
242
11k
The World Runs on Bad Software
bkeepers
PRO
64
11k
Transcript
BATCH PROCESSING IN ACTION HILLMER CHONA RODRIGO GRACIANO
WHAT SPRING BATCH IS? Spring Batch is a lightweight, comprehensive
batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
WHAT SPRING BATCH IS NOT? Spring Batch is not a
scheduler https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
BATCH VOCABULARY
THE BASICS
SHOW ME SOME CODE: BASIC DEMO This Photo by Unknown
Author is licensed under CC BY-NC-ND
JOB STEP REPOSITORY RUNNING
BATCH ARCHITECTURE (SIMPLIFIED ) https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader
Writer 1 * 1 1 1 1
JOB REPOSITORY •It’s how Job, Step, JobParameters, JobExecution, and StepExecution
are persisted
• JobRepository • JobLauncher • JobRegistry • JobExplorer • PlatformTransactionManager
• JobBuilderFactory • StepBuilderFactory @EnableBatchProcessing
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container
SHOW ME SOME CODE: RUNNING JOBS This Photo by Unknown
Author is licensed under CC BY-NC-ND
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container • Synchronous x Asynchronous
PROCESSORS CHUNKS WE CAN DO BETTER
STEP SEQUENCE Process Step Read Write
BATCH ARCHITECTURE https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1
* 1 1 1 1 Processor 1 1
PROCESSORS •Transforming •Filtering •Validations
STEP SEQUENCE Process Step Read Write
CHUNK SEQUENCE Process Step Read Write Read Process
MORE CODE MORE CODE: PROCESSORS & CHUNKS This Photo by
Unknown Author is licensed under CC BY-NC-ND
+READERS +WRITERS TASKLETS
READERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file
MULTI-FILE READER •Read multiple file in a single step •Files
must have the same format •Works with XML, JSON and flat files
SHOW ME SOME CODE: MULTI-READERS This Photo by Unknown Author
is licensed under CC BY-NC-ND
WRITERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file • Email
TASKLETS • 1 time operation instead of read/write • Run
a DB query • Copy Files • Zip/unzip files
BEYOND THE BASICS LISTENERS STEP FLOW SCALING AND PARALLEL PROCESSING
LISTENERS JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener ChunkListener ItemProcessListener SkipListener
JOB EXECUTION LISTENERS
STEP LISTENERS • before/after step StepExecutionListener • before/after read •
onReadError ItemReadListener • before/after write • onWriteError ItemWriteListener
STEP LISTENERS • before/after/after error chunk ChunkListener • before/after process
• onProcessError ItemProcessListener • onSkipInRead • onSkipInProcess • onSkipInWrite SkipListener
STEP FLOW • Sequential
STEP FLOW: SEQUENTIAL Step 1 Step 2 Step 3
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider
STEP FLOW: CONDITIONAL & DECIDER Condition Step 1 Step 3
Step 2
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider •
Split
STEP FLOW: SPLIT Step 1 Step 2 Step 3 Start
Step 4
SCALING AND PARALLEL PROCESSING • Multi-threaded (Async Executor) • Parallel
steps (Split) • Remote Chunking (Data processed by slaves – Real data sent not a reference) • Remote Partitioning (Data in chunks and local to slaves)
MORE FUNCTIONALITIES RESTARTING SKIPING FAILURES RETRY
USEFUL FUNCTIONALITIES - START •startLimit(n) •allowStartIfComplete(boolean)
USEFUL FUNCTIONALITIES – SKIPPING FAILURES •faultTolerant() •skip(SomeException.class) •skipLimit(n) •noSkip(SomeException.class) •noRollback(SomeException.class)
USEFUL FUNCTIONALITIES - RETRY •faultTolerant() •retry(SomeException.class) •retryLimit(n)
CODE CODE This Photo by Unknown Author is licensed under
CC BY-NC-ND
QUESTIONS? https://github.com/rodrigolgraciano/batch-demo.git https://speakerdeck.com/graciano/batch-processing @rodrigograciano @ HillmerCh
CHUNK-ORIENTED STEP https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png
JOB Job JobInstance JobExecution JobParameters Present at Devnexus title =
Batch Processing in Action Present Batch Processing in Action at Devnexus First try to present – hopefully, it will work
…STEP Job JobInstance JobExecution Step StepExecution Read count Write count
Filter count…
CHUNK-ORIENTED STEP + PROCESSORS https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png