Upgrade to Pro — share decks privately, control downloads, hide ads and more …

インフラチームを持たない会社でのインフラ運用

Yusuke
July 19, 2013

 インフラチームを持たない会社でのインフラ運用

始める DevOps ( http://atnd.org/events/41286 ) での発表資料です
#init_devops

Yusuke

July 19, 2013
Tweet

More Decks by Yusuke

Other Decks in Technology

Transcript

  1. ΞΧπΩ ιʔγϟϧήʔϜ࡞͍ͬͯΔձࣾ ૑ۀ͔࣌Β࢖͍ͬͯΔ΋ͷ͸ Ruby on Rails ͱ Nginx ΤϯδχΞ 9

    ਓɻ։ൃͭͭ͠Πϯϑϥӡ༻ ӡ༻ͷޮ཰Խ͕ॏཁ 13೥7݄20೔౔༵೔
  2. ૑ۀظ(҉ࠇ࣌୅) 2010~ 6~7 ສDAU Session ͷอଘઌ͕ MySQL DB ʹ →

    Master DBʹॲཧ͕ूத 5෼ஔ͖ʹམͪΔDBΛ࠶ىಈ͢Δ࡞ۀɻ ప໷Ͱ memcached Λಋೖ͠ɺͳΜͱ͔ མͪͳ͍Α͏ʹͳͬͨ͜ͱ΋ɻ 13೥7݄20೔౔༵೔
  3. ӡ༻҆ఆԽ 2012~ EC2ͷλάʹΑͬͯ໾ׂΛ؅ཧ EC2ʹ "Role" λάΛ෇͚ͯɺ໾ׂΛࢦఆ /etc/rc.local Ͱ git pull

    & λάʹԠͨ͡ε ΫϦϓτΛ࣮ߦ ShellͰႈ౳ੑΛอͭͨΊʹؤு͍ͬͯͨ 13೥7݄20೔౔༵೔
  4. ϛυϧ΢ΣΞઃఆ Chef Solo αʔυύʔςΟ cookbook Λ Berkshelf ͷ ֎Ͱ؅ཧͨ͠Γɺfork ͢Δͷ͸Ξϯνύ

    λʔϯ ΞΧπΩͰ͸ɺsite-cookbooksͰ্ॻ͖͠ ͍ͯΔ 13೥7݄20೔౔༵೔
  5. ϛυϧ΢ΣΞઃఆ Chef Solo ൓ө͸खಈɻ·ͩ production ؀ڥ΁ͷࣗ ಈ൓ө͸ͪΐͬͱා͍... serverspec ͕ॆ࣮ͨ͠Β Git

    ϦϙδτϦ ͷ master ϒϥϯνͷߋ৽ʹԠͯࣗ͡ಈ Ͱద༻͢Δ 13೥7݄20೔౔༵೔
  6. #  config/deploy.rb def  tagged_servers(tag_key,  tag_value,  default=[])    @ec2  ||=  AWS::EC2.new(ec2_endpoint:

     'ec2.ap-­‐northeast-­‐1.amazonaws.com')    ret  =  @ec2.instances.map  do  |instance|        next  if  instance.tags[tag_key]  !=  tag_value        next  if  instance.status  !=  :running        instance.dns_name.empty?  ?  instance.ip_address  :  instance.dns_name    end.compact    return  default  if  ret.empty?    ret end   def  tag(tag_value,  *args)    AWS.memoize  {        tagged_servers(tag_key,  tag_value).each  do  |host|            server(host,  *args)        end    } end   #  config/deploy/environment.rb tag  'app',  :web,  :app σϓϩΠ 13೥7݄20೔౔༵೔
  7. ؂ࢹ ؂ࢹϨΠϠ πʔϧ ର৅ OS؂ࢹ Amazon CloudWatch CPU / ϝϞϦ

    / σΟεΫ / Πϯελϯε਺ ϓϩηε؂ࢹ God Unicorn / Resque ϛυϧ΢ΣΞ؂ࢹ nagios MySQL Slow query౳ ΞϓϦέʔγϣϯ؂ࢹ NewRelic ύϑΥʔϚϯε 13೥7݄20೔౔༵೔
  8. God Ϛελʔϓϩηε͸GodͰ؂ࢹ͠ɺWorker͸ kzk/unicorn-worker-killer Ͱɺఆظతʹ࠶ىಈ require  ::File.expand_path('../config/environment',    __FILE__) require  'unicorn/oob_gc'

    require  'unicorn/worker_killer'   #  ϦΫΤετΛ࣮ߦ͍ͯ͠Δͱ͖͸GC͠ͳ͍ use  Unicorn::OobGC #  3072~4096ϦΫΤετ࣮ߦͨ͠Β࠶ىಈ͢Δ use  Unicorn::WorkerKiller::MaxRequests,  3072,  4096   run  XXXXXXX::Application 13೥7݄20೔౔༵೔
  9. Chariot $  ruby  bin/watcher  app-­‐name 10  minuts  PV  dataset:  

     2013-­‐07-­‐19  17:06:00  +0900:  1416.0    2013-­‐07-­‐19  17:07:00  +0900:  1269.0    2013-­‐07-­‐19  17:08:00  +0900:  1220.0    2013-­‐07-­‐19  17:09:00  +0900:  1286.0    2013-­‐07-­‐19  17:10:00  +0900:  1293.0    2013-­‐07-­‐19  17:11:00  +0900:  1352.0    2013-­‐07-­‐19  17:12:00  +0900:  1252.0    2013-­‐07-­‐19  17:13:00  +0900:  1248.0    2013-­‐07-­‐19  17:14:00  +0900:  1232.0    2013-­‐07-­‐19  17:15:00  +0900:  1266.0 10  minuts  PV  average:  1283.4 Current  [3] Expect  [2]    but  config  min  value  is  [3] -­‐-­‐-­‐  Do  nothing  -­‐-­‐-­‐ #  AWSͷઃఆ access_key_id:  AWS_ACCESS_KEY_ID secret_access_key:  AWS_SECRET_KEY ec2_endpoint:  ec2.ap-­‐northeast-­‐1.amazonaws.com cloud_watch_endpoint:  monitoring.ap-­‐northeast-­‐1.amazonaws.com elb_endpoint:  elasticloadbalancing.ap-­‐northeast-­‐1.amazonaws.com   #  ΞϓϦͷઃఆ min:  3  #  ࠷௿ىಈΠϯελϯε਺ event-­‐min:    20130722_2220-­‐20130722_2310:  15    20130723_2115-­‐20130723_2205:  30    20130723_2205-­‐20130723_2255:  15   sampling:  10  #  CloudWatch͔Βऔಘ͢ΔαϯϓϦϯά਺(1෼ʹ͖ͭ1ͭ) base:  1000  #  1Πϯελϯε͕ॲཧͰ͖Δ෼ؒPV਺ 13೥7݄20೔౔༵೔
  10. Cookbook ΍ Chariot ͷ໰୊͸ Github ͷ Issue ʹ ొ࿥ͯ͠ɺमਖ਼ग़དྷͦ͏ͳਓ͕ Pull

    Request ͢Δ ͜Ε͔Βಋೖ͍ͨ͠ࣄ΍ӡ༻ͷ໰୊͸ɺ Redmine ʹνέοτొ࿥͢Δ ίϥϘϨʔγϣϯ 13೥7݄20೔౔༵೔
  11. จԽ ΤϯδχΞνʔϜͷཧ೦ 2. Keep changing มԽͷத৺ʹ਎Λஔ͖ɺৗʹվળ͠ଓ͚Δ "ٕज़ͷྲྀΕʹ෇͍͍͚ͯͩ͘Ͱ͸ͳ͘ɺྲྀΕͷதʹੵۃతʹ਎Λஔ͖ɺৗ ʹࣗ෼ࣗ਎͕มԽͷத৺఺ʹ͍ΔΑ͏ʹมΘΓଓ͚Δɻ" 5. Do

    it yourself ࣗΒͷ؀ڥ͸ࣗΒ૑Γͩ͠ɺվળ͢Δ "؀ڥ͸୭͔͕༩͑ͯ͘ΕΔ΋ͷͰ΋ɺ࠷ॳ͔Βͦ͜ʹ͋Δ΋ͷͰ΋ͳ͍ɻ ࣗΒ͕׆ಈ͢ΔͨΊʹ࠷దͳ؀ڥ͸ࣗΒ͕ߟ͑ɺࣗΒखΛಈ͔͠૑Γͩ࢟͢ ੎ͰऔΓ૊Ήɻ" 13೥7݄20೔౔༵೔
  12. ͲͷΑ͏ʹͯ͠޿Ί͍͔ͯ͘ খ࢝͘͞ΊΔ ·ͣ͸ Chef Solo ͔Β ৽نPJͷαʔόߏஙεΫϦϓτΛ Chef ʹஔ ͖׵͑ͯΈΔͱ͜Ζ͔Β

    ੒ޭମݧΛڞ༗͢Δ πʔϧΛཧղͯ͠΋͏ͷ͸೉͍͚͠Ͳɺετ ʔϦʔΛཧղͯ͠΋Β͏ͷ͸؆୯ 13೥7݄20೔౔༵೔