The Design Philosophy of Kuroko2

Internal lightning talks in Cookpad

Transcript

1. The Design Philosophy of
2. • Web-based (not “enterprisy”) • Embedded workflow engine (ruote). • Distributed architecture • with on demand EC2 provisioning. • communicates with message queue. Kuroko1
3. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch exec command web interface web workflow Kuroko1 Overview
4. kuroko-console RabbitMQ kuoko-worker workflow engine worker process EC2 API launch exec command web interface web workflow Kuroko1 Pitfalls × × × × × × × × × ×
5. • It’s really hard to isolate the problem in a distributed environment. • “Programmable” means programming is always needed. • Nothing is reliable except MySQL. Lessons from Kuroko1
6. There is data inconsistency between the actual state and the stored state. Lessons from Kuroko1 (cont.)
7. State management is a key factor.
8. • Monolithic is the best. • Data oriented, not process oriented. • Monitor as possible. Kuroko2
9. Monolithic Architecture kuroko-console kuoko-worker workflow engine worker process exec command web interface DB
10. Kuroko2 Job State job_definitions job_instances tokens noop noop noop noop /0-noop A job definition A launched job A job state
11. it do engine.process(token) expect(token.path).to eq '/0-noop' engine.process(token) expect(token.path).to eq '/1-noop' engine.process(token) expect(token).to be_finished expect(Kuroko2::Token.all.count).to eq 0 end
12. Kuroko2 Execution State kuroko-worker kuroko-console executions workflow-processor command-executor exec command
13. loop do execution = Execution.unstarted.take() execution.touch(:started_at) execute_shell(execution.shell) execution.update(exit_status: exit_status, finished_at: Time.current) execution.finish() end
14. kuroko-worker kuroko-console executions workflow-processor command-executor exec command Kuroko2 Pitfalls × × ×
15. Kuroko2 Worker Processes command-executor kill process monitor process shell process exec command
16. def check_assignment_delay(exec_no_pid) if exec_no_pid.started_at < 1.minutes.ago Kuroko2::Notifications .not_assigned(exec_no_pid, @hostname).deliver_now exec_no_pid.touch(:mailed_at) end end
17. def check_process_absence(execution) begin Process.kill(0, execution.pid) rescue Errno::EPERM true rescue Errno::ESRCH if Execution.exists?(execution.id) notify_process_absence(execution) end end end
18. • /v1/stats/waiting_execution • /v1/stats/instance Kuroko2 Monitoring API
19. • State management is a key factor: • monolithic is the best. • data oriented, not process oriented. • monitor as possible. Kuroko2
20. is open source product. We are waiting your contributions!