Upgrade to Pro — share decks privately, control downloads, hide ads and more …

インフラエンジニアがConsulとStretcherをつかったデプロイ改善で開発効率の向上に貢...

インフラエンジニアがConsulとStretcherをつかったデプロイ改善で開発効率の向上に貢献した話 / jtf2017-consul-stretcher-deploy

Shuichi Ohsawa

August 27, 2017
Tweet

More Decks by Shuichi Ohsawa

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ • େᖒलҰ • Blog: http://blog.jicoman.info/ • Sansanגࣜձࣾ • σʔλԽγεςϜͷΠϯϑϥΛ୲౰

    • Πϯϑϥߏஙɾӡ༻ɾվળɺ։ൃج൫ͷվળͳͲ • લ৬͸WebΤϯδχΞ(PHPer)ͱͯ͠ܞଳίϯςϯπαΠτͷ։ൃ΍AWS΁ͷ Ҡߦͱ͔΍͍ͬͯ·ͨ͠ 2/64
  2. Data Strategy & Operation Center • ໊ࢗͷσʔλԽ&σʔλ׆༻ • Development Group

    • ໊ࢗͷσʔλԽͷγεςϜʮGEESʯશମͷ։ൃɾӡ༻ • ओʹRuby on RailsͰ։ൃ • R&D Group • ໊ࢗͷσʔλԽͷࣗಈԽͷͨΊʹඞཁͳը૾ೝࣝɺػցֶशɺςΩετղੳͳͲ • σʔλαΠΤϯςΟετ΋ࡏ੶͠ɺσʔλ׆༻ʹ஫ྗ • ओͳ։ൃݴޠ: C#, Python • Πϯϑϥ୲౰͸Devlopment Groupʹॴଐ͍ͯ͠Δ͕ɺR&D Group΋ݟ͍ͯΔ • 3໊(಺Ұਓ͸։ൃͱ݉຿) 7/64
  3. ։ൃऀͱΠϯϑϥ୲౰ͱͷؔΘΓ • جຊతʹ։ൃऀͱҰॹʹϓϩδΣΫτʹೖΔ͜ͱ͸ͳ͍ • ΠϯϑϥνʔϜͱͯ͠ผϓϩδΣΫτΛ૸Β͍ͤͯΔ͜ͱ͕ଟ͍ • ґཔɾ૬ஊϕʔεͰͷؔΘΓ͕࠷΋ଟ͍ • αʔόߏஙɺ˓˓Λ࢖͍͍ͨ etc

    • ΞϥʔτରԠ͸։ൃऀɾΠϯϑϥ୲౰ • ։ൃऀ͸Πϯϑϥͷίʔυ(Chef, TerraformͳͲ)Λ΄ͱΜͲ͍͡Βͳ͍ • Πϯϑϥ୲౰͸αʔϏεͷιʔείʔυΛ͍͡Βͳ͍ 8/64
  4. ݩʑ͸ҰϓϩδΣΫτͷҰ؀ • εςʔδϯά؀ڥͷࣗಈߏஙϓϩδΣΫτ • ༻్ʹ߹Θͤͨෳ਺ͷεςʔδϯά؀ڥ͕ඞཁͩͬͨ • TerraformΛ׆༻ͯࣗ͠ಈతʹߏஙͱഁյ͕Ͱ͖ΔΑ͏ʹ͢Δ • ࣗಈߏங͢Δʹ͸σϓϩΠͷࣗಈԽ΋ඞཁ •

    ैདྷͷσϓϩΠͰ͸ࣗಈԽ͕೉͔ͬͨ͠ • ຊ൪؀ڥʹ͓͚ΔσϓϩΠʹ՝୊Λײ͍ͯͨͨ͡ΊվળʹऔΓ૊Ή͜ͱʹͨ͠ • ݩʑ͸ΠϯϑϥνʔϜ಺ͷϓϩδΣΫτ͕ͩͬͨɺσϓϩΠʹؔͯ͠෼͔Βͳ͍͜ͱ͕ ଟ͍ͷͰ։ൃऀʹڠྗͯ͠΋Βͬͨ 10/64
  5. αʔόߏ੒ • ϩʔϧ • WebɺBatchɺDeploy etc • Webαʔό • ELB

    + EC2(c3.xlarge × 20਺୆) • Batchαʔό • EC2(r3.xlarge × े਺୆) • Deployαʔό • Web/BatchαʔόʹσϓϩΠ 12/64
  6. ैདྷͷσϓϩΠ • ఆظϦϦʔε(िҰ) • ͢΂ͯͷRailsΞϓϦΛσϓϩΠ • ඞཁʹԠͯ͡hotfixϦϦʔε(िʹ਺ճ) • ඞཁͳΞϓϦ͚ͩσϓϩΠ •

    Web/Batchαʔόશ୆Ұ੪ʹσϓϩΠ • σϓϩΠαʔό͔Β࣮ߦ • PushܕσϓϩΠ/In place/All at once 15/64
  7. ैདྷͷσϓϩΠ • σϓϩΠεΫϦϓτ(Rakefile)ͰσϓϩΠ࣮ߦ • Capistrano 2ϕʔε • ΞϓϦຖʹCapλεΫϑΝΠϧͷੜ੒ • ฒྻͰෳ਺ΞϓϦΛσϓϩΠ

    • ϩʔϧ͸Chef ServerͷλάͰ؅ཧ • Web, BatchͰॲཧ͕ҟͳΔͨΊϩʔϧͰ۠ผ • knife tagίϚϯυͰ෇͚֎͢͠Δ 16/64
  8. σϓϩΠπʔϧબఆͷํ਑ • PullܕσϓϩΠ • αʔόࣗ਎͕σϓϩΠ͢Δ • σϓϩΠ͕࣌ؒҰఆ(୆਺ʹൺྫ͠ͳ͍) • Amazon S3ʹϏϧυࡁͷϑΝΠϧΛ഑ஔ

    • μ΢ϯϩʔυ͠഑ஔɺ࠶ىಈ͢Δ͚ͩͰσϓϩΠ׬ྃ • S3ͷՄ༻ੑ͸99.99ˋ • In place/All at once • Blue/GreenσϓϩΠͩͱ࢓૊Έ͕ෳࡶԽ͢ΔͨΊҰ୴ݟૹΓ • One by one(1୆ͣͭ)ͩͱ͕͔͔࣌ؒΔ • ϗετͷొ࿥ɾ࡟আ͸ࣗಈతʹߦ͍͍ͨ 23/64
  9. AWS CodeDeploy • ࠾༻Λݟૹͬͨཧ༝! • ෳ਺ΞϓϦΛҰ੪ʹσϓϩΠ͢Δͱ஗͍(γϦΞϧʹ࣮ߦ͞ΕΔ) • All at onceͷ৔߹ɺ1୆Ͱ΋ࣦഊͨ͠ΒσϓϩΠࣦഊͱͳΒͳ͍

    • σϓϩΠࣦഊʹͳΒͳ͍͔ΒϩʔϧόοΫͰ͖ͳ͍ • CodeDeployͷ૝ఆ͍ͯ͠Δ؀ڥͱࣗ෼ୡͷͱϚον͠ͳ͔ͬͨ 25/64
  10. Stretcher • https://github.com/fujiwara/stretcher • Consulͱ૊Έ߹ΘͤΔ͜ͱͰɺΠϕϯτΛτϦΨʔͱͨ͠PullܕσϓϩΠ͕Մೳ • ࠾༻ͨ͠ཧ༝! • ෳ਺ΞϓϦΛҰ੪ʹσϓϩΠͯ͠΋ૣ͍ •

    20ऑͷRailsΞϓϦΛฒྻσϓϩΠ͢Δͱ1ʙ2෼ఔ౓ͰऴΘΔ • ෳ਺ͷઃఆϑΝΠϧ(ϚχϑΣετϑΝΠϧ)Λ༻ҙ͢Δ͜ͱͰɺৼΔ෣͍Λม͑Δ͜ͱ͕Ͱ͖Δ • σϓϩΠϑΝΠϧ഑ஔ༻ • γϯϘϦοΫϦϯΫ੾ସ༻ • Capistranoͷ࢓૊ΈΛྲྀ༻ • Consulͱͷ਌࿨ੑ͕ߴ͍ 26/64
  11. Consul • https://www.consul.io/ • HashiCorpࣾఏڙͷΦʔέετϨʔγϣϯπʔϧ • αʔόͱΫϥΠΞϯτͱ͍͏̎ͭͷ໾ׂͰߏ੒͞ΕɺΫϥελΛܗ੒͢Δ • ো֐ݕ஌ɺ಺෦DNSɺKey/ValueετΞ(KVS) •

    Πϕϯτൃߦ • ֤ConsulΫϥΠΞϯτ͸ΠϕϯτΛ଴ͪड͚͍ͯΔঢ়ଶͰɺಛఆͷΠϕϯτ͕ൃߦ͞ΕͨΒ೚ҙͷΞ ΫγϣϯΛ࣮ߦ • αʔϏελάʹΑΔϩʔϧ؅ཧ • ىಈ࣌ʹΫϥελϦϯά΁ࣗಈδϣΠϯ 27/64
  12. طଘͷ࢓૊ΈΛ௥͏ • σϓϩΠεΫϦϓτΛಡΉ • ͘Θ͍͠ਓʹ֓ཁΛฉ͍ͯɺৄࡉ͸ಡΜͰཧղ • Capistrano 2ͷιʔείʔυ΋ซͤͯಡΉ • cap

    deployͰͲͷΑ͏ʹ࣮ߦ͞Ε͍ͯΔͷ͔ • ϝϯςφϯε੾ସɺϩʔϧόοΫͷ࢓૊Έ΋௥ͬͯΈΔ • ։ൃऀͰ΋෼͔Βͳ͍͜ͱ͕ҙ֎ͱଟ͍ 28/64
  13. σϓϩΠεΫϦϓτͷվम • capistrano-stretcher • https://github.com/pepabo/capistrano-stretcher • Capistrano͔ΒStretcher + ConsulΛݺͼग़ͯ͠σϓϩΠͰ͖Δ •

    ͦͷ··࢖͍͔͚ͨͬͨͲͰ͖ͳ͔ͬͨŋŋŋ • Capistrano3 ରԠ • طଘͷσϓϩΠεΫϦϓτͰ͸ಠࣗίϚϯυ͕ଟ͍ • ࢀߟʹͭͭ͠ɺҰ͔ΒػೳΛ࣮૷ 29/64
  14. υΩϡϝϯτ࡞੒ • ֓ཁ • ৽͍͠σϓϩΠͷશମ૾Λ௫ΜͰ΋Β͏ • ίϚϯυͱ࣮ߦํ๏ • σϓϩΠ͢Δͱ͖ͷϚχϡΞϧ •

    Կ΋ߟ͑ͣʹίϐϖ͢Δ͚ͩͰσϓϩΠͰ͖ΔΑ͏ʹ • τϥϒϧγϡʔςΟϯά • ૝ఆ͞ΕΔΤϥʔ಺༰ͱͦͷରॲํ๏ • ࣗ෼͕͍ͳͯ͘΋ղܾͰ͖ΔΑ͏ʹ • ໰୊͕ىͬͨ͜ͱ͖ʹ୭Ͱ΋ରॲͰ͖ΔΑ͏ʹͳΔͷ͕ཧ૝త 32/64
  15. ֓ཁ • ιʔείʔυΛϏϧυɺtar.gzͰݻΊͯɺS3ʹΞοϓϩʔυ • ϏϧυαʔόʹσϓϩΠ • ϩʔϧຖʹϚχϑΣετϑΝΠϧΛੜ੒ɺS3ʹΞοϓϩʔυ • consul eventͰΠϕϯτൃՐɺWeb/BatchαʔόͰStretcher࣮ߦ

    • σϓϩΠϑΝΠϧͷऔಘɺ഑ஔ • ϦϦʔεલ·ͰʹऴΘΒ͓ͤͯ͘ • γϯϘϦοΫϦϯΫ੾ସɺRails࠶ىಈ • ϦϦʔε࣌ʹ࣮ߦ(ϦϦʔε࣌ؒͷ୹ॖ) • σΟϨΫτϦߏ଄͸Capistranoͱಉ͡ 35/64
  16. update_code_#{application}_#{version}.yml src: s3://<%= s3_bucket %>/<%= application %> checksum: <%= checksum

    %> dest: app/<%= application %>/release/YYYYMMDDHHMMSS commands: pre: - mkdir -p app/<%= application %> success: - echo 'success' failure: - echo 'failed!!' excludes: - "*.pid" - "*.socket" 40/64
  17. deploy_#{application}_#{role}_#{version}.yml sync_strategy: mv commands: post: - ln -sfn "app/<%= application

    %>/release/YYYYMMDDHHMMSS" "app/<%= application %>/current" <%=# Rails࠶ىಈ %> - cd app/<%= application %>/release/YYYYMMDDHHMMSS && touch tmp/restart.txt <%=# ಈ࡞֬ೝ & ஆػӡస %> - <%= File.join(script_path, "pre-warming.sh") %> <%= application %> success: <%=# աڈͷϦϦʔεΛ࡟আ %> - <%= File.join(script_path, "cleanup.sh") %> <%= application %> - echo 'release success' failure: - echo 'release failed!!' 41/64
  18. Consul KVSͰόʔδϣϯ؅ཧ • σϓϩΠ੒ޭޙɺConsul KVSʹ੒ޭͨ͠࠷৽ͱ1ͭલͷόʔδϣϯΛ ߋ৽͢Δ • http://127.0.0.1:8500/v1/kv/#{application}/current • http://127.0.0.1:8500/v1/kv/#{application}/previous

    • όʔδϣϯ͸ϦϦʔεσΟϨΫτϦͷ೔෇( YYYYMMDDHHMMSS ) • αʔό௥Ճ͢Δͱ͖͸࠷৽ɺϩʔϧόοΫ͢Δͱ͖͸1ͭલͷόʔδ ϣϯΛࢦఆ͢Δ 45/64
  19. ·ͱΊͯσϓϩΠ͢Δ ͢΂ͯͷΞϓϦΛϚϧνεϨουͰฒྻσϓϩΠ task deploy_all: :setup do deploy_projects = "application_A,apllication_B,..." threads_count

    = ENV['THREADS'] Parallel.map(deploy_projects, in_threads: threads_count.to_i) do |project| # Ϗϧυαʔό΁ͷσϓϩΠ # ϚχϑΣετϑΝΠϧͷੜ੒ # S3΁ͷΞοϓϩʔυ # ΞϓϦέʔγϣϯαʔόʹ഑෍ end end task release_all: :setup do deploy_projects = "application_A,apllication_B,..." threads_count = ENV['THREADS'] Parallel.map(deploy_projects, in_threads: threads_count.to_i) do |project| # σϓϩΠ(γϯϘϦοΫϦϯΫ੾ସ) end end 47/64
  20. ݁Ռ 1. σϓϩΠʹ͕͔͔࣌ؒΔ • => ϦϦʔε࣌ؒͷ୹ॖ(30෼->5෼) 2. ಛఆͷαʔό͚ͩσϓϩΠͰ͖ͳ͍ • =>

    ࠷৽ͷσϓϩΠϑΝΠϧΛ࣋ͬͯ͘Δ͚ͩ 3. GitHub/RubyGems͕མͪΔͱσϓϩΠͰ͖ͳ͍ • => S3ͳͷͰ΄΅໰୊ͳ͠ 4. ࣗಈԽͮ͠Β͍ • => ConsulʹΑͬͯΫϥελϦϯά΁ͷࣗಈδϣΠϯ/୤ୀ͕Մೳ 49/64
  21. ·ͱΊ • Πϯϑϥ୲౰͕σϓϩΠվળΛͨ͜͠ͱͰ։ൃج൫ʹߩݙͰ͖ͨ • ։ൃͱΠϯϑϥͷ྆ํΛ஌Βͳ͍ͱվળͰ͖ͳ͍ྖҬ͕͋Δ • Πϯϑϥ͚ͩͰͰ͖Δ෼໺͸গͳ͍ • ։ൃ΋Ͱ͖ΔΠϯϑϥ(ࣗ෼΋ΩϟϦΞతଆ໘΋ؚΊͯ) •

    օʹͱͬͯ޾ͤͳಓΛબͿ • ։ൃଆͱΠϯϑϥଆͷཁٻ͕Ϛον͢Ε͹ϕετ • গͣͭ͠ม͍͑ͯ͘ • Ұؾʹ΍ΓํΛม͑ͣʹɺͰ͖Δ͚ͩࠓ·Ͱͷํ๏Λอͪͳ͕Β • ௕ظؒվળ͍͚ͯ͠͹݁Ռͱͯ͠େ͖͘มΘ͍͚ͬͯΔ 51/64
  22. Consul - ΫϥΠΞϯτ # /etc/consul.d/settings.json { "data_dir": "/opt/consul", "log_level": "err",

    "start_join": [ "consul-server001", "consul-server002", "consul-server003" ] } 55/64
  23. Consul - ϩʔϧλά # /etc/consul.d/role.json { "service": { "name": "role",

    "tags": [ # ෳ਺ઃఆՄ "web", "newrelic_monitoring" ] } } 56/64
  24. Consul - Πϕϯτ(σϓϩΠ) # /etc/consul.d/deploy.json { "watches": [ { "type":

    "event", "name": "deploy_application_A", "handler": "export TMPDIR=/ephemeral0; stretcher -retry 3 >> $TMPDIR/deploy_application_A.log 2>&1" }, { "type": "event", "name": "deploy_application_B", "handler": "export TMPDIR=/ephemeral0; stretcher -retry 3 >> $TMPDIR/deploy_application_B.log 2>&1" }, ... } • consul eventίϚϯυͰΠϕϯτൃՐ • consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file} • σϓϩΠϑΝΠϧͷμ΢ϯϩʔυઌ(TMPDIR)ΛEC2ͷΠϯελϯεετΞʹมߋ 57/64
  25. ϩʔϧ # ϩʔϧͷࢦఆ role :web, *consul_role('web') role :batch, *consul_role('batch') #

    ϩʔϧࢦఆ͞Εͨϗετͷऔಘ def consul_role(roles) response = `curl -fsSL http://127.0.0.1:8500/v1/catalog/service/role` nodes = JSON.parse(response) lists = [] nodes.each do |node| if (node['ServiceTags'] & roles.split(',')) == roles.split(',') lists << node['Address'] end end lists end 58/64
  26. Consul - Πϕϯτ(ϩʔϧλάͷ෇͚֎͠) # /etc/consul.d/change_role.json { "watches": [ { "type":

    "event", "name": "change_role", "handler": "change_consul_role.rb >> /tmp/change_log.log 2>&1" } ] } #!/bin/env ruby require "json" consul_url = "http://127.0.0.1:8500" response = %x(curl -fsSL #{consul_url}/v1/catalog/node/$(curl -fsSL #{consul_url}/v1/agent/self | jq -r .Member.Name)) node = JSON.parse(response) new_role = node["Services"]["role"]["Tags"] consul_role_file_path = "/etc/consul.d/role.json" if ! File.exist?(consul_role_file_path) then exit 0 end consul_role = open(consul_role_file_path) do |io| JSON.load(io) end if consul_role["service"]["tags"] != new_role then consul_role["service"]["tags"] = new_role open(consul_role_file_path, 'w') do |io| io.puts(JSON.pretty_generate(consul_role)) end %x(consul reload) end 59/64
  27. S3΁ͷΞοϓϩʔυ def upload2s3(local_src_path, bucket, path) run "aws s3api put-object --region

    #{aws_region} --bucket #{bucket} --key #{path} --body #{local_src_path}" end def create_tarball run "cd #{latest_release} && rm -rf #{tarball_name} && shopt -s dotglob && tar zcf #{tarball_name} *" end def upload_tarball upload2s3("#{latest_release}/#{tarball_name}", app_s3_bucket, "#{app_s3_path}/#{tarball_name}") end
  28. σϓϩΠϑΝΠϧͷ഑෍ namespace :stretcher do task :update_code, roles: :build do roles.split(',').each

    do |role| manifest_file = "update_code_#{application}_#{version}.yml" capture "consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}" end puts 'Finished(Not released yet!)' end end
  29. ϦϦʔε(γϯϘϦοΫϦϯΫ੾ସ) namespace :stretcher do task :release, roles: :build do transaction

    do on_rollback do rollback end roles.split(',').each do |role| manifest_file = "deploy_#{application}_#{role}_#{version}.yml" capture "consul event -service=role -tag=#{role} -name=deploy_#{application} #{manifest_file}" end unless check_event_success('release_failure') puts 'Release Failed!! & Starting rolling back.' exit 1 end puts 'Release finished.' end end end
  30. όʔδϣϯߋ৽ ConsulͷKVSʹ֨ೲ͍ͯ͠Δόʔδϣϯ஋Λߋ৽ after 'stretcher:release', 'stretcher:update_release_version' namespace :stretcher do task :update_release_version,

    roles: :build do # લճ੒ޭͨ͠σϓϩΠͷόʔδϣϯΛpreviousʹ͢Δ capture "curl -sS -X PUT -d '#{current_version}'http://127.0.0.1:8500/v1/kv/#{application}/previous" # ੒ޭͨ͠σϓϩΠͷόʔδϣϯΛߋ৽͢Δ capture "curl -sS -X PUT -d '#{latest_version)}' http://127.0.0.1:8500/v1/kv/#{application}/current" end end
  31. ϩʔϧόοΫ namespace :stretcher do task :rollback, roles: :build do fetch(:deploy_roles).split(',').each

    do |target_role| manifest_file = "update_code_#{application}_#{previous_version}.yml" capture "consul event -service=role -tag=#{target_role} -name=deploy_#{application} #{manifest_file}" end reset_release_status fetch(:deploy_roles).split(',').each do |target_role| manifest_file = "deploy_#{application}_#{role}_#{previous_version}.yml" capture "/usr/local/bin/consul event -service=role -tag=#{target_role} -name=deploy_#{application} #{manifest_file}" end end