Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
The Design Philosophy of Kuroko2
Search
Naoto Takai
January 18, 2017
Programming
3
2.2k
The Design Philosophy of Kuroko2
Internal lightning talks in Cookpad
Naoto Takai
January 18, 2017
Tweet
Share
More Decks by Naoto Takai
See All by Naoto Takai
Building the Perfect Custom Keyboard
takai
1
630
キースイッチ潤滑入門 / A Brief Introduction to Lube MX Switches
takai
2
800
みなさん、分かっていますか / What is a difference?
takai
1
920
Agile and DevOps with Ruby
takai
2
1.1k
Software Quality and Testing #1
takai
6
680
Microservices in Action
takai
37
5.4k
Ruby App on Netflix Microservices Stack
takai
6
1.1k
How We Use Jenkins?
takai
21
8.5k
COOKPAD and Test Automation
takai
19
2.8k
Other Decks in Programming
See All in Programming
Django Ninja による API 開発効率化とリプレースの実践
kashewnuts
0
920
デミカツ切り抜きで面倒くさいことはPythonにやらせよう
aokswork3
0
190
そのpreloadは必要?見過ごされたpreloadが技術的負債として爆発した日
mugitti9
2
3k
Web技術を最大限活用してRAW画像を現像する / Developing RAW Images on the Web
ssssota
2
1.2k
Let's Write a Train Tracking Algorithm
twocentstudios
0
220
ポスターセッション: 「まっすぐ行って、右!」って言ってラズパイカーを動かしたい 〜生成AI × Raspberry Pi Pico × Gradioの試作メモ〜
komofr
0
940
WebエンジニアがSwiftをブラウザで動かすプレイグラウンドを作ってみた
ohmori_yusuke
0
170
明日から始めるリファクタリング
ryounasso
0
110
ててべんす独演会〜Flowの全てを語ります〜
tbsten
1
220
Pythonスレッドとは結局何なのか? CPython実装から見るNoGIL時代の変化
curekoshimizu
4
1.3k
育てるアーキテクチャ:戦い抜くPythonマイクロサービスの設計と進化戦略
fujidomoe
1
150
ソフトウェア設計の実践的な考え方
masuda220
PRO
3
470
Featured
See All Featured
Docker and Python
trallard
46
3.6k
Done Done
chrislema
185
16k
YesSQL, Process and Tooling at Scale
rocio
173
14k
The Language of Interfaces
destraynor
162
25k
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
890
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
9
960
Scaling GitHub
holman
463
140k
Optimizing for Happiness
mojombo
379
70k
Building an army of robots
kneath
306
46k
Music & Morning Musume
bryan
46
6.8k
Typedesign – Prime Four
hannesfritz
42
2.8k
Statistics for Hackers
jakevdp
799
220k
Transcript
The Design Philosophy of
• Web-based (not “enterprisy”) • Embedded workflow engine (ruote). •
Distributed architecture • with on demand EC2 provisioning. • communicates with message queue. Kuroko1
kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch
exec command web interface web workflow Kuroko1 Overview
kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch
exec command web interface web workflow Kuroko1 Pitfalls × × × × × × × × × ×
• It’s really hard to isolate the problem in a
distributed environment. • “Programmable” means programming is always needed. • Nothing is reliable except MySQL. Lessons from Kuroko1
There is data inconsistency between the actual state and the
stored state. Lessons from Kuroko1 (cont.)
State management is a key factor.
• Monolithic is the best. • Data oriented, not process
oriented. • Monitor as possible. Kuroko2
Monolithic Architecture kuroko-console kuoko-worker workflow engine worker process exec command
web interface DB
Kuroko2 Job State job_definitions job_instances tokens noop noop noop noop
/0-noop A job definition A launched job A job state
it do engine.process(token) expect(token.path).to eq '/0-noop' engine.process(token) expect(token.path).to eq '/1-noop'
engine.process(token) expect(token).to be_finished expect(Kuroko2::Token.all.count).to eq 0 end
Kuroko2 Execution State kuroko-worker kuroko-console executions workflow-processor command-executor exec command
loop do execution = Execution.unstarted.take() execution.touch(:started_at) execute_shell(execution.shell) execution.update(exit_status: exit_status, finished_at:
Time.current) execution.finish() end
kuroko-worker kuroko-console executions workflow-processor command-executor exec command Kuroko2 Pitfalls ×
× ×
Kuroko2 Worker Processes command-executor kill process monitor process shell process
exec command
def check_assignment_delay(exec_no_pid) if exec_no_pid.started_at < 1.minutes.ago Kuroko2::Notifications .not_assigned(exec_no_pid, @hostname).deliver_now exec_no_pid.touch(:mailed_at)
end end
def check_process_absence(execution) begin Process.kill(0, execution.pid) rescue Errno::EPERM true rescue Errno::ESRCH
if Execution.exists?(execution.id) notify_process_absence(execution) end end end
• /v1/stats/waiting_execution • /v1/stats/instance Kuroko2 Monitoring API
• State management is a key factor: • monolithic is
the best. • data oriented, not process oriented. • monitor as possible. Kuroko2
is open source product. We are waiting your contributions!