Share
Embed
Transcript
25 slides
-
ࣗݾհ • େੴӳհ • ٕज़෦ ։ൃج൫άϧʔϓ • ڞ௨ܾࡁج൫kuroko2ͷӡ༻ ͳͲ •
ෳαʔϏεؒͷ߹ੑͷऔ ΓΈʹ͍ͭͯ -
LVSPLPͱ • web-based job scheduler / workflow engine. • ࢀߟࢿྉ
• repository: https://github.com/cookpad/kuroko2 • ΫοΫύουͷδϣϒཧγεςϜ kuroko2 ͷհ • The Design Philosophy of Kuroko2 • kuroko2 documentation -
LVSPLPͷԊֵ • 2015/04 ٕ࣌ज़෦ͩͬͨ Naoto Takai (ݱΤϯδχΞ౷ׅϚωʔδϟʔ )ͷεΧϯΫ ϫʔΫʹΑͬͯ։ൃ͞Εɺࣾʹಋೖ͞ΕΔ •
2015/05 େੴ͕ kuroko2 ͷϝϯςΛҾ͖ܧ͙ • 2015/12 ΫοΫύου։ൃऀϒϩάʮΫοΫύουͷδϣϒཧγεςϜ kuroko2 ͷ հʯ • OSSʹ͠ͳ͍ͭΓͩͬͨ • 2016/10 ಥવͷOSSԽ • ॾൠͷࣄʹΑΓαΠϨϯτͳϦϦʔε • ʮOSS ʹͳͬͨ Kuroko2 ΛͲ͜ΑΓૣ͘ಋೖͨ͠ͷͰհ͍ͨ͠ϒϩάʯͰ OSSԽ͕ੈؒʹΕΔ • 2017/01 Naoto Takai ͷ͚ࣾLTʮThe Design Philosophy of Kuroko2ʯެ։ -
LVSPLPͷํ • ݱ࣮తͳӡ༻Λݟਾ͑ͨ҆ఆͨ͠ઃܭΛอͭ (The Design Philosophy of Kuroko2 p.20) •
ࡶʹͳΓ͕ͪͳόονδϣϒཧͷΛUI/UXͳͲͰղ ܾ͢Δ • γϯϓϦγςΟ • εέδϡʔϥʔͱ൚༻ੑͷߴ͍ϫʔΫϑϩʔཧʹప͢Δ • ͏·͘֎ͷػߏͱ࿈ܞ͢Δ (Bricolage, Hako) • ඞཁͳ෦͚ͩʹ։͔Ε֦ͨுੑ -
ΧελϜλεΫ 1. λεΫͷ࡞ $ cd your_kurko2_rails_apps/ $ mkdir -p lib/kuroko2/workflow/task/
$ cat > my_project_runner.rb module Kuroko2 module Workflow module Task class MyProjectRunner < Execute def chdir '/home/alice/my_project' end def shell "bundle exec ./bin/rails runner -e production #{Shellwords.escape(option)}" end end end end end 2. kuroko2.yml એݴ custom_tasks: my_project_runner: MyProjectRunner 3. kuroko2 script env: VAL1=A env: VAL2=B my_project_runner: MyProject::Batch1.run -
,VSPLP"QQMJDBUJPO$POUSPMMFS֦ு 1. extentionͷ࡞ $ cd your_kurko2_rails_apps/ $ cat > lib/controller_extention.rb
module ControllerExtention extend ActiveSupport::Concern included do before_action :additional_before_action end private def additional_before_action if signed_in? # do something end end end 2. kuroko2.yml એݴ extentions: controller: - ControllerExtention -
,VSPLP"QQMJDBUJPO$POUSPMMFS֦ுͷ͍ॴ • ΫοΫύουͰ Raven::Transports::Fluentd ͷί ϯςΩετՃͳͲͰར༻ (Τϥʔ௨) • ϩάͳͲͷτϨʔγϯά •
ϢʔβʔຖͷΞΫηε੍ݶ • Controller͚͍ͩ·ͷͱ͜ΖରԠ͍ͯ͠Δ͕ɺ͠ඞ ཁέʔε͕͋Ε֦ுϙΠϯτΛ૿͢͜ͱݕ౼͠ ͍ͨ -
͜Ε͔Β • ·ͩ·ͩະඋ • υΩϡϝϯτ • ͬͱރΕ͍ͤͨ͞ • ϫʔΫϑϩʔͷػೳڧԽ •
Hako (Docker) ͱͷޮతͳ࿈ܞ -
Α͋͘Δ࣭ • OSSͷ kuroko2 ΫοΫύουࣾͰ͍ͬͯΔͷʁ • YES • ࣾόʔδϣϯଘࡏ͍ͯ͠ͳ͍ •
masterϒϥϯνͷϦϦʔεલͷͷΛৗʹ͏Α͏ ʹͯ͠ɺࣾͰͳ͍͔֬ೝͯ͠ϦϦʔε͍ͯ͠Δ -
ن • 700ݸͷόονδϣϒ (2017/06/01࣌) • Ұ͋ͨΓͷ࣮ߦճ 6000 • kuroko2 ͷ্Ͱͯ͢ཧ
• αʔϏεຖʹ͚ͣʹίϯιʔϧ͚̍ͭͩ • ͦ͜·ͰେنͰͳ͍ (ͱࢥ͍ͬͯΔ) -
ߏ • worker (commnad-executor) ͷ 230 • 34ݸͷworker༻Πϯελϯε • ൚༻
worker༻Πϯελϯε x 10 • CPUΛ͏όον༻Πϯελϯε x 2 • Hako (docker) ΞϓϦέʔγϣϯ༻ x 3 • େྔϝʔϧ৴༻ x 8 • ϓογϡ௨༻ x 4 • DWH༻ x 1 • ͦͷଞ x 7 -
ࢹ • CPU, Disk, Network, ࢮ׆ࢹ ͳͲͷఆܗͷͷʹՃ͑ͯɺδϣϒͷঢ়گΛࢹ • OSS kuroko2
ʹ͋ΔҎԼͷAPIΛఆظతʹୟ͍͍ͯΔ • /v1/stats/waiting_execution • /v1/stats/instance -
LVSPLPͷσϓϩΠ • kuroko2ຊମ capistrano ͰσϓϩΠ • console, scheduler, processor, command-
executer ͦΕͧΕʹroleΛ͚ͯσϓϩΠͯ͠ worker Λ restart • HUPγάφϧʹΑΔ graceful shutdown -
όονδϣϒͷσϓϩΠ • όονͷσϓϩΠͦΕͧΕͷΞϓϦέʔγϣϯຖʹ ߦ͍ͬͯΔ • ࠷ۙͷͷ hako oneshot Ͱ ECS
λεΫΛ࣮ߦ͠ ͍ͯΔͷͰ worker ʹର͍ͯ͠Βͳ͍ • ݹ͍ΞϓϦέʔγϣϯ worker͍ͯ͠Δ -
όονδϣϒͷཧ • όονδϣϒ֤։ൃऀɺ෦ॺ͕Λͭ • δϣϒ͕ࣦഊͨ͠߹ɺ୲ऀʹϝʔϧSlackͰ௨ɺ֤୲෦ॺͰϦ τϥΠͳͲͷରԠ͢Δ • δϣϒͷ description ʹো࣌ͷӨڹɺରԠํ๏ɺϦτϥΠͷඞཁੑͳͲΛ
ॻ͍͓ͯ͘͜ͱͰେ͖ͳো࣌ʹଞͷ෦ॺ(SRE ͳͲ)ͰରԠͰ͖ΔΑ͏ʹ ͍ͯ͠Δ • ϩά kuroko2 ͔Β AWS CloudWatch Logs ʹอଘͭͭ͠ɺΞϓϦέʔγϣ ϯ͕ͦΕͧΕཧ͍ͯ͠Δϩάଘࡏ͍ͯ͠Δ • ෦ॺδϣϒʹ·͕ͨͬͨδϣϒͷ࿈ܞʹ wait λεΫ͕Α͘ΘΕ͍ͯΔ -
·ͱΊ • OSS ͷ kuroko2 Λ͍ͬͯ·͢ • αʔϏεຖʹ͚ΔͷͰͳͯ͘ɺ΄΅ͯ͢ͷδϣϒΛ1ͭͷkuroko2 ΞϓϦͰཧ •
தن͘Β͍? • 700ݸͷδϣϒ • 6000࣮ߦ/ • 230 worker • kuroko2 ͷػೳΛར༻ͨ͠όονઐ༻ͷࢹ߲͕͋Δ • kuroko2 ͷ job definition ͷςϯϓϨ௨Γʹӡ༻