Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Batch Processing
Search
Rodrigo Graciano
April 11, 2022
Programming
0
360
Batch Processing
Deck from the Batch Processing in Action presentation
Rodrigo Graciano
April 11, 2022
Tweet
Share
More Decks by Rodrigo Graciano
See All by Rodrigo Graciano
Your Java isn't the same
graciano
0
190
Cloud Batch
graciano
0
160
Java 9 ao 17 - Oracle no TDC
graciano
0
110
KnoxJava - Java 9-17
graciano
0
150
Of concurrency and other demons
graciano
0
120
Java 9 to Java 16: A review of recent changes to the language
graciano
0
140
Java 9 ao 15 - Evolução da Linguagem Java
graciano
0
260
Other Decks in Programming
See All in Programming
Streamlitで実現できるようになったこと、実現してくれたこと
ayumu_yamaguchi
2
270
Reactの歴史を振り返る
tutinoko
1
170
一人でAIプロダクトを作るならAIにはもっと働いてもらいたい / I want AI to work harder
rkaga
3
190
0から始めるモジュラーモノリス-クリーンなモノリスを目指して
sushi0120
0
250
中級グラフィックス入門~効率的なメッシュレット描画~
projectasura
4
2.4k
変化を楽しむエンジニアリング ~ いままでとこれから ~
murajun1978
0
660
Go製CLIツールをnpmで配布するには
syumai
2
1.1k
大規模FlutterプロジェクトのCI実行時間を約8割削減した話
teamlab
PRO
0
440
Understanding Kotlin Multiplatform
l2hyunwoo
0
250
画像コンペでのベースラインモデルの育て方
tattaka
3
1k
あまり知られていない MCP 仕様たち / MCP specifications that aren’t widely known
ktr_0731
0
220
DataformでPythonする / dataform-de-python
snhryt
0
150
Featured
See All Featured
Embracing the Ebb and Flow
colly
86
4.8k
Art, The Web, and Tiny UX
lynnandtonic
301
21k
Side Projects
sachag
455
43k
Navigating Team Friction
lara
188
15k
Documentation Writing (for coders)
carmenintech
73
5k
The Invisible Side of Design
smashingmag
301
51k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
A designer walks into a library…
pauljervisheath
207
24k
What's in a price? How to price your products and services
michaelherold
246
12k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
RailsConf 2023
tenderlove
30
1.2k
Transcript
BATCH PROCESSING IN ACTION HILLMER CHONA RODRIGO GRACIANO
WHAT SPRING BATCH IS? Spring Batch is a lightweight, comprehensive
batch framework designed to enable the development of robust batch applications vital for the daily operations of enterprise systems. https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
WHAT SPRING BATCH IS NOT? Spring Batch is not a
scheduler https://docs.spring.io/spring-batch/docs/current/reference/html/index-single.html#spring-batch-intro
BATCH VOCABULARY
THE BASICS
SHOW ME SOME CODE: BASIC DEMO This Photo by Unknown
Author is licensed under CC BY-NC-ND
JOB STEP REPOSITORY RUNNING
BATCH ARCHITECTURE (SIMPLIFIED ) https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader
Writer 1 * 1 1 1 1
JOB REPOSITORY •It’s how Job, Step, JobParameters, JobExecution, and StepExecution
are persisted
• JobRepository • JobLauncher • JobRegistry • JobExplorer • PlatformTransactionManager
• JobBuilderFactory • StepBuilderFactory @EnableBatchProcessing
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container
SHOW ME SOME CODE: RUNNING JOBS This Photo by Unknown
Author is licensed under CC BY-NC-ND
RUNNING JOBS • From the command Line (CommandLineJobRunner) • Within
a web container • Synchronous x Asynchronous
PROCESSORS CHUNKS WE CAN DO BETTER
STEP SEQUENCE Process Step Read Write
BATCH ARCHITECTURE https://docs.spring.io/spring-batch/docs/current/reference/html/images/spring-batch-reference-model.png JobRepository JobLauncher Job Step Reader Writer 1
* 1 1 1 1 Processor 1 1
PROCESSORS •Transforming •Filtering •Validations
STEP SEQUENCE Process Step Read Write
CHUNK SEQUENCE Process Step Read Write Read Process
MORE CODE MORE CODE: PROCESSORS & CHUNKS This Photo by
Unknown Author is licensed under CC BY-NC-ND
+READERS +WRITERS TASKLETS
READERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file
MULTI-FILE READER •Read multiple file in a single step •Files
must have the same format •Works with XML, JSON and flat files
SHOW ME SOME CODE: MULTI-READERS This Photo by Unknown Author
is licensed under CC BY-NC-ND
WRITERS • Database (JDBC, Mongo, Neo4j, Hibernate, etc.) • Flat
Files (delimited, fixed-length) • XML, JSON • JMS, Kafka, Amqp • Multi-file • Email
TASKLETS • 1 time operation instead of read/write • Run
a DB query • Copy Files • Zip/unzip files
BEYOND THE BASICS LISTENERS STEP FLOW SCALING AND PARALLEL PROCESSING
LISTENERS JobExecutionListener StepExecutionListener ItemReadListener ItemWriteListener ChunkListener ItemProcessListener SkipListener
JOB EXECUTION LISTENERS
STEP LISTENERS • before/after step StepExecutionListener • before/after read •
onReadError ItemReadListener • before/after write • onWriteError ItemWriteListener
STEP LISTENERS • before/after/after error chunk ChunkListener • before/after process
• onProcessError ItemProcessListener • onSkipInRead • onSkipInProcess • onSkipInWrite SkipListener
STEP FLOW • Sequential
STEP FLOW: SEQUENTIAL Step 1 Step 2 Step 3
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider
STEP FLOW: CONDITIONAL & DECIDER Condition Step 1 Step 3
Step 2
STEP FLOW • Sequential • Conditional (ExitStatus) • Decider •
Split
STEP FLOW: SPLIT Step 1 Step 2 Step 3 Start
Step 4
SCALING AND PARALLEL PROCESSING • Multi-threaded (Async Executor) • Parallel
steps (Split) • Remote Chunking (Data processed by slaves – Real data sent not a reference) • Remote Partitioning (Data in chunks and local to slaves)
MORE FUNCTIONALITIES RESTARTING SKIPING FAILURES RETRY
USEFUL FUNCTIONALITIES - START •startLimit(n) •allowStartIfComplete(boolean)
USEFUL FUNCTIONALITIES – SKIPPING FAILURES •faultTolerant() •skip(SomeException.class) •skipLimit(n) •noSkip(SomeException.class) •noRollback(SomeException.class)
USEFUL FUNCTIONALITIES - RETRY •faultTolerant() •retry(SomeException.class) •retryLimit(n)
CODE CODE This Photo by Unknown Author is licensed under
CC BY-NC-ND
QUESTIONS? https://github.com/rodrigolgraciano/batch-demo.git https://speakerdeck.com/graciano/batch-processing @rodrigograciano @ HillmerCh
CHUNK-ORIENTED STEP https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png
JOB Job JobInstance JobExecution JobParameters Present at Devnexus title =
Batch Processing in Action Present Batch Processing in Action at Devnexus First try to present – hopefully, it will work
…STEP Job JobInstance JobExecution Step StepExecution Read count Write count
Filter count…
CHUNK-ORIENTED STEP + PROCESSORS https://docs.spring.io/spring-batch/docs/current/reference/html/images/chunk-oriented-processing-with-item-processor.png