Upgrade to Pro — share decks privately, control downloads, hide ads and more …

トレタのインフラ運用

 トレタのインフラ運用

【トレタ×プレイド】TechBlog Deep Dive Meetup #1 〜インフラ編〜
http://plaidtech.connpass.com/event/33511/

での発表内容。
Terraform, Packer, Roadworker, Ansible, Serverspec, Circle CIなどをトレタでどう使っているか?何が罠か?など。

Hiroaki Sano

July 06, 2016
Tweet

More Decks by Hiroaki Sano

Other Decks in Technology

Transcript

  1. NF • Name: • ࠤ໺༟ষ(Hiroaki Sano) • Personal website: •

    https://hiroakis.com/blog/ • Company: • NEC Soft, Ltd. (2006/4-) • CyberAgent, Inc. (2011/3-) • Toreta, Inc. (2014/11-)
  2. τϨλ • ҿ৯ళ޲͚༧໿؅ཧΞϓϦέʔγϣϯΛiPadΞϓϦͱͯ͠ఏڙ • ҰൠϢʔβ޲͚ʹ΢Σϒ༧໿΋ఏڙ • ސ٬ɿϨετϥϯɺډञ԰ͳͲͷҿ৯ళ • ྫ͑͹… •

    Զͷגࣜձ༷ࣾ(ԶͷΠλϦΞϯɺԶͷϑϨϯν…etc) • גࣜձࣾେঙ༷(ঙ΍ɺ΍Δ͖஡԰…etc) • ΤʔɾϐʔΧϯύχʔ༷(௩ా೶৔…etc) • ϩʔετϗʔε༷ • etc • ϩʔϯν͸2013೥12݄ • ಋೖళฮ਺ɿ6000ళฮʙ • ళฮ͋ͨΓͷֹ݄՝ۚ • ίʔϙϨʔτΧϥʔ͸྘ • ΦϑΟε͸ौ୩ -> ໨ࠇ -> ܙൺण -> ޒ൓ా
  3. ࢲͷॻ͍ͨهࣄ • Raspberry PiͱԹ౓ɾ࣪౓ɾޫηϯαʔͰΦϑΟεͷঢ়گΛՄࢹԽ͢Δ • http://tech.toreta.in/entry/2016/02/15/153203 • τϨλͷΠϯϑϥӡ༻ɺࢧ͍͑ͯΔಓ۩(Packer, Terraform, Serverspec,

    Ansible, Roadworker, Circle CI)ɺߟ͑ํ • http://tech.toreta.in/entry/2016/04/14/143248 • ϝʔϧͷ഑৴ঢ়گΛՄࢹԽɺ௥੻͢Δ • http://tech.toreta.in/entry/2016/04/20/153852 • Engineyard͔ΒAWSʹҠઃͯ͠Auroraͷӡ༻Λ։࢝ͨ͠ • http://tech.toreta.in/entry/2016/06/16/114919
  4. ࠓ೔࿩͢͜ͱ • ओʹAWSͰͷΠϯϑϥӡ༻ʹ͍ͭͯ • τϨλͷAPIαʔόͷΤίγεςϜ • ओʹϒϩάωλʹొ৔ͨ͠ࣄฑΛத৺ʹ… • Terraform •

    Roadworker • Packer • Ansible • Serverspec • Aurora • Circle CI • ࠓޙʹ͍ͭͯ • ٕज़తʹਅ৽͍͠΋ͷ͸ͳ͍Ͱ͕͢ࢀߟʹ͍͚ͯͨͩ͠Ε͹…
  5. Amazon Route 53 Amazon RDS(Aurora) api
 worker Redis memcached Amazon

    SNS Amazon SQS τϨλ"1*ͱͦͷपลͷΤίγεςϜ FAX/SMS… Amazon S3 ը૾/Ի੠… ଞαϒγεςϜ΁ͷ௨஌ ϩά ϝʔϧ fax, sms API worker fluentd
  6. τϨλ"1*ͱͦͷपลͷΤίγεςϜ • Amazon Web Services • 5݄ʹEngineyard͔ΒҠઃ(ϒϩάهࣄͷ௨Γ) • Appαʔό: APIɺWorkerಉډ.

    nginx, Rails(unicorn), sidekiqͰ࣮૷. • Redis: WorkerͷΩϡʔΠϯά༻ • memcached: Ωϟογϡ • RDS Aurora: ϝΠϯͷσʔλϕʔε • Google Cloud Platform • BigQuery: fluentd͔Βϩάͷసૹ • ιʔϦʔαʔό΍Ұ෦ͷόοναʔό: GCEͷݕূΛ݉Ͷͯͪ͜Βʹߏஙͯ͋͠Δ • Monitoring • Mackerel: Ϧιʔε؂ࢹɺϝτϦΫε • Pingdom: ΤϯυϙΠϯτ؂ࢹ • Pagerduty: ΞϥʔτରԠͷεέδϡʔϦϯάɺి࿩௨஌ • CI • Circle CI: CI͓ΑͼσϓϩΠɺ͓ΑͼΦϖϨʔγϣϯͷத৺
  7. app
 worker Redis Aurora Writer memcached app
 worker Redis memcached

    Availability Zone A 71$ઃܭ public subnet Availability Zone C Aurora Reader public subnet 10.0.0.0/16 10.0.0.0/24 10.0.1.0/24
  8. 71$ઃܭ • ֤ίϯϙʔωϯτ͸ϚϧνAZͰ഑ஔ • 10.0.X.0/24 -> AZ-A • 10.0.Y.0/24 ->

    AZ-C • ύϒϦοΫαϒωοτͷΈ • RDSͳͲ΋ύϒϦοΫαϒωοτʹஔ͘ • ݎ࿚ੑ͸ηΩϡϦςΟάϧʔϓͰ୲อ • ࡶ͡Όͳ͍͔ʁͱͨ·ʹݴΘΕΔ • ݱঢ়ͷ࡞ΓͰผʹࠔ͍ͬͯͳ͍ • ࠔͬͨΒ࡞Γͳ͓͢
  9. 5FSSBGPSN • VPC, SecurityGroup, EC2Λ؅ཧର৅ͱ͍ͯ͠Δ • EC2ߏங࣌͸Environment, Role, ServiceλάΛ෇༩͢Δ •

    ͜ΕΒ͸σϓϩΠ΍ϓϩϏδϣχϯάͰ࢖͏ • AWSͷ࢖͍ํͱͯ͠యܕతͱ͍͏͔ݹయతͳखஈ
  10. ͳͥ5FSSBGPSN͔ʁ • CloudFormationΑΓ͸׳Ε͍͔ͯͨΒ • Google Cloud Platform΋࢖͍͔ͬͯͨΒ • ͪͳΈʹࣅͨΑ͏ͳπʔϧʹ͜Μͳͷ΋͋Γ·͢ •

    Apache Libcloud • https://libcloud.apache.org/index.html • छʑͷΫϥ΢υʹରԠͨ͠boto(aws sdk for Python)ͷ Α͏ͳ΋ͷ
  11. ؊UGTUBUFϑΝΠϧͷѻ͍ • terraform࣮ߦޙͷঢ়ଶ͕هड़͞Ε͍ͯΔϑΝΠϧ • terraform͸͜ͷϑΝΠϧͷঢ়ଶΛਖ਼ͱͯ͠มߋΛద༻͢Δ • terraform࣮ߦ࣌ʹ͜ͷϑΝΠϧ͕յΕͨΓઌ૆ฦΓͨ͠ঢ়ଶͰ࣮ߦ͢Δͱ ࣄނΔ • ͜ͷϑΝΠϧ͕յΕͨΒʁר͖໭ͬͯ͠·ͬͨΒʁ

    => ؤு࣏ͬͯͦ ͏:(ʀƄƅ’Тƅ'): • खݩ͔Β࣮ߦ͢Δ৔߹͸githubͳͲͰ؅ཧͭͭ͠࡞ۀऀ֤Ґ͕ඞͣ࠷৽ͷ ΋ͷΛpull͔ͯ͠Β࢖͏ • τϨλͰ͸… • S3ʹஔ͖ͭͭɺCircle CI͔ΒterraformΛ࣮ߦ • ΦϖϨʔγϣϯΛ࣮ߦ͢Δਓ෺͸Ұਓ(Circle CI)ʹ͢ΔͨΊ • ඞͣ࠷৽ͷtfstateϑΝΠϧΛ࢖͏Α͏ͳ࢓૊Έʹ͢ΔͨΊ
  12. 3PBEXPSLFS • Engineyard࣌୅͸EngineyardͱAWSͷϋΠϒϦουӡ༻ • Engineyard: EC2Πϯελϯε(App, Redis, MySQL…) • AWS:

    Route53, S3… • ೖࣾ࣌ʹAWS্ͷ΋ͷͷίʔυԽʹணखͨ͠ • ࠷ॳ͔ΒTerraformʹ͠ͳ͔ͬͨͷ͸౰࣌ͷTerraform͸ طଘͷDNSϨίʔυͷexport͕Ͱ͖ͳ͔ͬͨ • Roadworker͸Ͱ͖ͨ
  13. શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ • શ෦ೖΓʹͨ͠৔߹ͷ… • ϝϦοτ • ىಈ͢Δ͚ͩͰαʔϏε౤ೖՄೳͱͳΔ • σϝϦοτ •

    ίϯϑΟάมߋͷͨͼʹAMIΛࣽࠐΉඞཁ͕͋Δ • ·ͨͦͷͨͼʹΠϯελϯεશೖΕସ͑ • ͜ͷ࢓૊ΈΛࣗಈԽ͢Ε͹σϝϦοτ͸ͳ͘ͳΔ͔΋͠Εͳ͍ • ͨͩࣽ͜͠Ή࣌ؒΛ଴ͭͷ͕μϧ͍ͱ͖΋͋Δ…
  14. શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ • શ෦ೖΓͷํ͕Ϋϥ΢υతͰ͸͋Δ • ίϯςφͷӡ༻ͱۙ͘ͳΔ͸ͣ • ͭ·Γঢ়ଶͷมߋ = ΠϯελϯεΛ৽͘͠࡞ΔˍೖΕସ͑Δ •

    ίϯϑΟάྨͷมߋΛAnsible/chefͳͲͰద༻Λ܁Γฦ͢ͷ͸ݩ ͸ΦϯϓϨͷߟ͑ํ • ͭ·Γಉ͡αʔόΛ࢖͍ճ͢ͱ͍͏લఏͷιϦϡʔγϣϯ • ͍ΘΏΔႈ౳ੑ • Ϋϥ΢υ͸”ࣺͯΔ”ͱ͍͏બ୒ࢶ͕͋Δ • ࣺͯΔ->࠶ߏங͕୹࣌ؒͰՄೳ • ΋ͪΖΜΠϯελϯε͕յΕΔ·Ͱ࢖͍ճ͢͜ͱ΋Ͱ͖Δ
  15. ͳͥ"OTJCMF͔ʁ • ผʹͳΜͰ΋ྑ͔ͬͨ • Α͋͘Δ(?)ࣄҊ • αʔόߏஙεΫϦϓτ͕ൿ఻ͷλϨͩᵆ( :^o^)ᵒ • αʔόߏஙखॱॻ͕ոจॻͩᵎ(^o^;

    )ᵊ • chef/ansibleΛಋೖͩ(^q^) • cookbook/playbook͕ݹจॻʹͳͬͨʗ(^o^)ʘ • ͜ͷखͷπʔϧ͸ͲΕ࢖ͬͯ΋هड़ϧʔϧ΍ӡ༻ϧʔϧΛܾΊ͓͔ͯͳ͍ ͱυπϘʹϋϚΔ(ܦݧ্) • Πϯϑϥίʔυ΋ͪΌΜͱϨϏϡʔ͢Δʢ͋ͨΓ·͑Ͱ͕͢…ʣ • ςετɺServerspecΛॻ͍͓ͯ͜͏ • ʮ༨ܭͳ͜ͱ͕Ͱ͖ͳ͍puppet͕Ұ൪ྑ͍ʯbyಉۀऀͷ༑ਓ ͱ͍͏ҙݟ΋ ͋Δ
  16. "VSPSB • ಋೖʹ͋ͨͬͯݕূͨ͜͠ͱ • Ҡߦલ(MySQL on Engineyard)ͱ࣮ߦܭը͕มΘΒͳ͍͜ͱͷ֬ೝ • ͪΌΜͱࠓ·Ͱ௨ΓʹΠϯσοΫεΛ࢖ͬͯ͘ΕΔ͔ •

    ίϯϑΟάϨʔγϣϯͷਫ਼ࠪ • MySQL͓͡͞Μͱͯ͠͸ඇৗʹؾʹͳͬͨ… • Ͳ͜ΛͲ͏มߋ͔ͨ͠͸ϒϩάهࣄΛࢀর͍ͯͩ͘͠͞ • εέʔϧΞοϓɺσʔλϦΧόϦ…ͳͲͷΦϖखॱɺཁ͢Δ࣌ؒͳͲ • ༗ࣄʹඋ͑ͯखॱΛཱ͓֬ͯͨ͘͠Ί • ཁ͸ૉৼΓɻ࿅शɻ
  17. "VSPSBͰ͸࣮ݱͰ͖ͳ͍͜ͱ • θϩμ΢ϯλΠϜ • ϝϯςφϯε΢Οϯυ΢͕͋ΔͷͰ͍͔ͭ͸࠶ىಈ͠ͳ͖Ό ͳΒͳ͍ͱ͖͕དྷΔɻ • Ͱ΋ϑΣΠϧΦʔόͰ1෼ఔ౓Ͱ׬ • 1෼΋ࢭΊͨ͘ͳ͍৔߹͸…

    • MySQL on EC2 with MHA • ࢲͷهԱͰ͸MHAͷํ͕ϑΣΠϧΦʔό͸ૣ͍ • ׬શθϩμ΢ϯλΠϜΛ໨ࢦ͔ͨͬͨ͠ΒϚϧνϚελͳ෼ ࢄDBͰ΋…Cassandraͱ͔Ͱ…ؤுͬͯ…͍ͩ͘͞…
  18. ࠓޙ • ίϯςφʁ • ΍ΔͳΒ৽نαʔϏε։ൃ࣌ • ։ൃख๏ɺσϓϩΠํ๏ɺӡ༻ख๏͕มԽ͢ΔɺଞΤϯδχΞʹ΋ͦΕΛ΍ͬ ͯ΋Β͏ඞཁ͕͋Δ • طଘͷαʔϏεΛίϯςφԽ͢Δ͜ͱ͸ࠓ͸ߟ͍͑ͯͳ͍

    • ৽نαʔϏεͰίϯςφಋೖͷޮՌ͕ग़ͨΒ΍Δ͔΋ • ASGʁ • ୆਺͕૿͑ͯඇϐʔΫ࣌ͷίετ࡟ݮޮՌ͕ݟ͑ͦ͏ʹͳͬͨΒ • ͨͩ͠Terraformͱͷ૬ੑ͕ѱͦ͏ͩ