Upgrade to Pro — share decks privately, control downloads, hide ads and more …

fluct-migrate-aws-from-on-premises

 fluct-migrate-aws-from-on-premises

Avatar for Takuya Nishigori

Takuya Nishigori

June 03, 2021
Tweet

More Decks by Takuya Nishigori

Other Decks in Programming

Transcript

  1. ࣗݾ঺հ @_nishigori גࣜձࣾ fluct (CARTA HOLDINGS ͷάϧʔϓձࣾ ) ։ൃຊ෦ ΤϯδχΞ

    ◦ SRE (Site Reliability Engineering ) ◦ Release Engineerin g ◦ Ϗϧυ৬ਓ ◦ ΞϓϦέʔγϣϯίʔυ΋ಡΈॻ͖ ◦ AWS͸ fluct ೖࣾޙʹ৮Γ࢝Ίͨ
  2. About fluct גࣜձࣾ fluc t “גࣜձࣾ fl uct͸ʮϝσΟΞͷ੒௕Λڞ૑͢ΔʯΛϛογϣϯʹWebϝσΟ Ξ΍εϚʔτϑΥϯΞϓϦʹର͠ɺSSP(Supply-Side Platform)ʮ

    fl uctʯͱࠃ ಺Ͱ3ࣾͷΈೝఆΛड͚ͨʮαΠτӡӦऀ޲͚Googleೝఆύʔτφʔʯͱ͠ ͯɺʮDoubleClick Ad ExchangeʯͷಋೖٴͼίϯαϧςΟϯάΛߦ͏ϝσΟ ΞϚωλΠζࢧԉࣄۀΛ࣠ͱͨ͠ΞυςΫϊϩδʔΧϯύχʔͰ͢ɻ ” https://corp. fl uct.jp/ fluct SSP (Supply Side Platform ) WebαΠτ΍ωΠςΟϒΞϓϦͳͲͷഔମʹ࠷దͳ޿ࠂΛࣗಈ Ͱ഑৴͠ɺ޿ࠂऩӹΛ࠷େԽ͢ΔͨΊͷαʔϏεΛఏڙ ◦ RTB (Real Time Bidding ) ◦ etc …
  3. About fluct SSP γεςϜʹٻΊΒΕΔੈք ◦ ߴτϥϑΟοΫʢ݄ؒ2000ԯճҎ্ͷ޿ࠂϦΫΤετʣ ◦ ௿ϨΠςϯγ ◦ ʢεέʔϥϏϦςΟʣ

    ޿ࠂ഑৴αʔόʔʢ޿ࠂΛબͿ໾ׂʣ ͸ಠཱՔಇ ◦ αʔόʔҟৗ͕αʔϏεʹۃྗӨڹΛ༩͑ͳ͍ঢ়ଶʹ ◦ ނো࣌ʹ͸ࣗಈͰαʔϏεΞ΢τ ◦ ΦϯϓϨϛεɾAWSͷϋΠϒϦουͳ഑ஔ ◦ ಠཱ͍ͯ͠ΔͷͰ৔ॴΛબ͹ͳ͍ ◦ ΦϯϓϨϛε͸ৗ࣌਺ඦ୆ ◦ AWS͸਺े~ ਺ඦ୆ʹΦʔτεέʔϧ ௿ϨΠςϯγͱ҆ఆੑΛੜΉΞʔΩςΫνϟ - SSPͷݱ৔ʹֶͿɺߴՄ༻ੑͷͭ͘Γํ: https://eh-career.com/engineerhub/entry/2019/07/12/103000
  4. ϋΠϒϦουͳߏ੒ʹࢸΔ·Ͱ ◦ αʔϏεϩʔϯν౰ॳ͸ΦϯϓϨϛεͷΈʢ਺୆ʣ ◦ ΦϯϓϨϛεαʔόʔௐୡ·ͰͷຒΊ߹ΘͤͰAWSར༻Λ͸͡Ίͨ ◦ ౰࣌ͷ༧૝Λ௒͑ΔτϥϑΟοΫ૿ ◦ ޿ࠂ഑৴γεςϜ͸ΦϯϓϨϛεো֐࣌ͷୀආઌͱͯ֬͠อ ◦

    ωοτϫʔΫػثো֐ͰɺશϦΫΤετΛAWSʹৼͬͨաڈ΋ Engineers in VOYAGE - ࣄۀΛΤϯδχΞϦϯά͢Δٕज़ऀͨͪ: https://www.lambdanote.com/products/engineers-in-voyage About fluct SSP
  5. ͜ͷൃද͸ɺ ॻ੶ ʮEngineers in VOYAGEʯ ୈҰষ “fluct — ޿ࠂ഑৴ͷ෣୆ཪͷٕज़ऀͨͪ ”

    ͷଓ͖ͷ෺ޠͰ͢ Engineers in VOYAGE - ࣄۀΛΤϯδχΞϦϯά͢Δٕज़ऀͨͪ: https://www.lambdanote.com/products/engineers-in-voyage About fluct SSP
  6. γεςϜͷॅΈ෼͚ ◦ 30લޙͷαϒγεςϜ͕ଘࡏ ◦ ։ൃ؀ڥ΍ࢼ࡞ΞϓϦέʔγϣϯ͸ओʹAWSͰ TCO (Total Cost of Ownership)

    ͷ؍఺ ◦ ༧ଌՄೳͳγεςϜෛՙ͸ΦϯϓϨϛεͰ ◦ ʰn೥ઌ΋ಉن໛ͷ޿ࠂ഑৴Λ͍ͯ͠ΔͩΖ͏ʱ ◦ ༧ଌෆՄೳͳྨ͸AW S ◦ ৽͍͠Ϗδωενϟϯε ◦ ಥൃతͳ˓˓ About fluct SSP 2019೥౰࣌ͷfluct γεςϜશମ
  7. ݱঢ়ҡ࣋ͱ՝୊ղܾͷδϨϯϚ ࣮ࡍͣͬͱ೰ΜͰ·ͨ͠ ◦ 5೥΄Ͳ೰ΜͰ͸ͷ܁Γฦ͠ ◦ ೥ʹҰ౓͸ग़Δʮશ෦Ϋϥ΢υʹҠΔͷ͸Ͳ͏ͩΖ͏ʁʯ ◦ ͱ΋͋ΕɺΦϯϓϨϛεͷ΄͏͕·͍ͩ҆…ʁ ◦ ʢ͋ΕʁΦϯϓϨϛεͷ΄͏͕ো֐ස౓ଟ͘ͳ͍…ʁʣ

    ◦ λΠϛϯά͸͍͕ͭద੾ͳͷ͔ ◦ ͍ͣΕશͯΫϥ΢υαʔϏεͷ৐Δ͔΋͠Εͳ͍͕ ◦ ݮՁঈ٫ͷλΠϛϯάɺϥοΫ͝ͱʹόϥόϥ ◦ ʮAWSͰ΋ผͷۤ࿑͕ੜ·ΕΔͰ͠ΐʁʯ
  8. զʑ͕΍Γ͍ͨ͜ͱ͸Կͳͷ͔ ࣄۀͷ੒௕଎౓Λམͱ͞ͳ͍બ୒Λ ◦ ෳࡶԽ͢Δ޿ࠂࣾձ΁ɺٕज़ΛؚΊͨదԠ ◦ ւ֎ͷࢢ৔֫ಘ ࣄۀ๊͕͑Δ໰୊ղܾ΁ͷۙಓΛ ◦ ΑΓιϑτ΢ΣΞͷ։ൃʹ஫ྗ͍ͨ͠ ◦

    શͯͷΤϯδχΞ͕AWSʹ৮ΕΔঢ়گ͸͖ͭͬͯͨ͘ ◦ ʮΠϯϑϥʯͱ͍͏۠ผΛ࣋ͨͳ͍ੈք΁ ◦ ʢ΋ͪΖΜ N/W ஌ࣝ౳͕ඞཁͳ͍ͱ͍͏બ୒Ͱ͸ͳ͍ʣ
  9. αϙʔτͷखް͞ͱ૬ੑ ◦ ΤϯλʔϓϥΠζαϙʔτܖ໿ ◦ ஸೡɻ޿ࠂۀքपΓͷ͜ͱ΋ৄ͍͠୲౰ऀ ◦ Slack Ͱͷίϛϡχέʔγϣϯ ◦ ϥϑͳฐࣾελΠϧʹ߹Θͤͯ΋Β͍͑ͯΔ

    ◦ Ϩε͸΍͍ νʔϜ͕AWSʹ׳Ε͍ͯΔ ◦ νʔϜͰ৽نɾະ࢖༻ʢAWSʣαʔϏε΋Ωϟον Ξοϓ͢Δ౔৕͕Ͱ͖ͨ͜ͱ΋ىҼ ◦ Ϋϥ΢υαʔϏεͰ͸AWS͕Ұ൪େ͖ͳن໛Ͱѻͬͯ ͍ΔʢҰ෦γεςϜͰ͸ผͷΫϥ΢υαʔϏεΛར༻ ͳͥAWSͳͷ͔
  10. AWS EC2 Graviton Processor (Graviton2 ) "Amazon EC2 C5 ΠϯελϯεΑΓ΋ɺ࠷େ

    40ˋ ߴ͍ίετ ύϑΥʔϚϯεΛൃش͠·͢ " https://aws.amazon.com/jp/ec2/graviton/ https://aws.amazon.com/jp/about-aws/whats-new/2020/06/ ◦ Architecture = arm6 4 ◦ ϋϠΫφϧʂʂʂϠεΫφϧʂʂʂ AWS Migration Acceleration Program: https://aws.amazon.com/jp/migration-acceleration-program/ ͳͥAWSͳͷ͔
  11. AWS Migration Acceleration Program (MAP ) ◦ AWSͷΫϥ΢υҠߦϓϩάϥϜ ◦ 2019೥ͷ

    re:Invent Ͱ΋ηογϣϯτϥοΫ͋Γ ◦ https://youtu.be/lR27wRbmzoo ◦ Ҡߦ४උͷͨΊͷϓϩάϥϜ΋༻ҙ͞Ε͍ͯΔʢMRPʣ ◦ ϑΟʔυόοΫͰز͔ͭࢪࡦఏҊΛ͍͍ͨͩͨ ◦ ن໛ʹΑͬͯ͸Ҡߦର৅ͷAWSར༻ྉΫϨδοτ͕෇༩ ◦ ͜Ε͕ʢૣʑʹʣҠߦ͢ΔܾΊखʹʂ AWS Migration Acceleration Program: https://aws.amazon.com/jp/migration-acceleration-program/ ͳͥAWSͳͷ͔
  12. ঈ٫ࢼࢉ ͱ AWSར༻ྉΫϨδοτ ◦ ࢒฽ՁͱAWS MAP࣮ࢪͰ෇༩͞ΕͨΫϨδοτͷֹۚਪҠͷܭࢉ ◦ ࢒฽Ձ͕େ͖͍͏ͪ͸ΦϯϓϨϛε΋Ҿ͖ଓ͖Քಇ ◦ ঈ٫ظ͕ؒ༨ͬͯͯ΋Ҡߦ׬਱ͨ͠΄͏͕ϝϦοτ͕େ͖͍

    ΦϯϓϨϛεҡ࣋ͷ··Ͱ͍ͬͨ৔߹ͱൺֱ ◦ গ҆͘͠ͳΓͦ͏ ◦ ࢼࢉͷஈ֊Ͱ͸҆͘ͳΔ͜ͱ͚ͩ෼͔Ε͹͍͍Ԙകʹ ◦ ίετ࡟ݮ͸Ҡߦ໨ॲͷ͍ͭͨޙʹूதతʹߦ͍ͬͯ͘ AWSҠߦݕূ
  13. FYI: ͦͷ΄͔ͷׂҾࢪࡦ ʢಉΠϕϯτͰઆ໌͕͋ΔͨΊׂѪʣ ◦ Reserved Instanc e ◦ fluct Ͱ͸ߪೖͯ͠΋1೥୯Ґ

    ◦ Saving Plan s ◦ Spot Instanc e ◦ https://aws.amazon.com/jp/ec2/spot/instance-advisor/ ◦ ҠߦظؒதʹEC2 Instanceྉ͕ۚ҆͘ͳΔ͔΋…ʁʢظ଴ AWSҠߦݕূ AWS΁ͷϚΠάϨʔγϣϯͷͦͷઌʹ ʙϦϑτˍγϑτ͔ΒͷΫϥ΢υ࠷దԽʙ: https://pages.awscloud.com/JAPAN-event-OE-Migration-20210603-reg-event.html
  14. ΤϯδχΞͷ൑அɻձࣾͷܾஅ ͜͜·ͰશͯΤϯδχΞνʔϜͷ൑அ ◦ ܦӦਞ΁ͷઆ໌͸ผ్ߦͬͨʢ੔ཧ͍ͭͰʹʣ ◦ ໾һձٞͰ΋ʢCTO͕?ʣ͍͍ײ͡ʹઆ໌ͯ͘͠Εͨ΋Α͏ ◦ ద੾ͳݖݶҠৡ ◦ ೚ͤͯ΋Β͍ͬͯΔ

    ◦ Ή͠ΖΦϯϓϨϛεߪೖ౳Ͱͷ᜛ٞӠʑ͸ϒϩοΫཁૉͩͬͨͷͰ͸ ◦ νʔϜͰ ΦϯϓϨϛε V.S. AWS ͱ͍ͬͨਪਐରཱ͸ੜ·Εͳ͔ͬͨ ◦ ߹ཧతͳཧ༝͕͋Ε͹ͦΕΛબ୒͢΂͖ ◦ Α͘΋ѱ͘΋ݻ͍ࣥͯ͠ͳ͍ʢΑ͍ʣ Ҡߦͷܾஅ Engineers in VOYAGE - ࣄۀΛΤϯδχΞϦϯά͢Δٕज़ऀͨͪ: https://www.lambdanote.com/products/engineers-in-voyage
  15. ࣮ࡍ͸̏ਓͰҠߦ ◦ ෦ॺͷΤϯδχΞ͸30໊લޙ ◦ ࿹ྗ͸େࣄ ◦ ͜ͷ3໊͡Όͳͯ͘΋͍ͣΕ୭͔͕΍͔ͬͨ΋ ◦ ۩ମతʹ͸1೥Ҏ಺ͰશͯҠߦ͢Δ໨ॲΛݐͯΔ໨ඪ ◦

    Ҡߦޙ͸AWS࠷దԽ΁޲͚ͯγϑτʢޙड़ʣ ◦ ΦϯϓϨϛεͷϥοΫఫୀʢ෺ཧͰӡͿʣ͸৽ଔͷ໘ʑͱҰॹʹ ◦ ΤϯδχΞਓੜͰ෺ཧ৮Δ࠷ޙͷػձ͔΋ Ϧϑτˍγϑτ
  16. ޿ࠂ഑৴αʔόʔͷҠߦ ◦ ݩʑͷϋΠϒϦουߏ੒ΛঃʑʹAWSʹൺ཰Ҡಈ ◦ Route53 Record > weigh t ◦

    EC2 Instance++ (AutoScaling ) ◦ EC2 Instance family ͸ C5d serie s ◦ σΟεΫॻ͖ࠐΈ͕ଟ͍ͷͰ Ephemeal-Dis k ◦ ಉλΠϛϯάͰ Root EBS Λ gp2 ͔Β gp3 ΁ Ϧϑτˍγϑτ
  17. ॲཧΛ෦෼తʹEC2͔Β੾Γग़͠ ◦ CRON JOB, ϩάऩू etc … ◦ ϦΫΤετॲཧΛछผ͠ECS, Lambda౳΁Ҡߦ

    ◦ ΤϯυϙΠϯτϨϕϧͰͷ੾Γ෼͚ ◦ ϚωʔδυαʔϏεͷར༻ ◦ DynamoDB, Kinesis Data Stream, … ◦ ೰ΈϙΠϯτ͸౎౓AWSαϙʔτʹ૬ஊ ◦ ઌͷݮՁঈ٫λΠϛϯάΛΈͯΏͬ͘Γ࣮ࢪ Ϧϑτˍγϑτ ϞϊϦε͔ΒϚΠΫϩαʔϏε΁ ― ϞϊϦεΛਐԽͤ͞Δ࣮ફҠߦΨΠυ: https://www.oreilly.co.jp/books/9784873119311/
  18. ޿ࠂ഑৴αʔόʔ͸ϞϊϦεҡ࣋ ◦ ϞδϡϥʔϞϊϦεͰͷ޻෉ͱݱঢ় ◦ monorepo؅ཧ ◦ ಠࣗͷฒྻϏϧυγεςϜ ◦ ̍೔ʹ਺ճͷϦϦʔε ◦

    ਺छྨ͋Δ޿ࠂ഑৴αʔόʔͰ͍ͪΞϓϦέʔ γϣϯͷσϓϩΠ͕஗ΕΔ͜ͱ͸ك ◦ ϨΠςϯγʔΛԿΑΓؾʹ͢Δʢ਺msͷੈքʣ ◦ ෼ղΑΓ΋݁߹ʹΑΔར఺Λڗड ◦ ϞχλϦϯά߲໨͸೔ʑ૿Ճத ͱ͸ݴ͑ɺద੾ͳ෼ղ୯ҐΛ౎౓໛ࡧ͍ͯ͠Δ Ϧϑτˍγϑτ ϞϊϦε͔ΒϚΠΫϩαʔϏε΁ ― ϞϊϦεΛਐԽͤ͞Δ࣮ફҠߦΨΠυ: https://www.oreilly.co.jp/books/9784873119311/
  19. Graviton2 ରԠͷલ౗͠ ◦ Apple Silicon (M1 Mac) ొ৔΋͋Γ ◦ ։ൃ؀ڥͷ

    arm64 ରԠ΋߹ΘͤͨରԠΛ ◦ x86_64 ͱฒྻՔಇ͠ͳ͕ΒҠߦ ◦ ࠷ऴతʹ͸࣮஍ݕূ ◦ ͍ͭͰ΋ϩʔϧόοΫՄೳʹ ◦ ޿ࠂ഑৴αʔόʔͷϕʔεAMI͸ࣗલϏϧυ ◦ RockyLinux 8. 4 ◦ Kernel Update (like amazon-linux-extras kernel-ng ) ◦ Install SSM Agent, AWS CL I ◦ etc… Ϧϑτˍγϑτ x86_64 ͔Β arm64 ΁༷ࢠݟ͠ͳ͕Β੾ΓସΘΔ༷ࢠ
  20. Multi Architecture Support for Graviton 2 https://github.com/aws/aws-graviton-getting-started ◦ RPM ύοέʔδ

    ◦ aarch64ެࣜఏڙ͕ແ͍΋ͷ͸ࣗલϏϧυ ◦ Puppet manifest (Configuration Management) ◦ ΞϓϦέʔγϣϯίʔυͷରԠ ◦ fluctͷίʔυϨϕϧͰ͸΄΅มߋͳ͠ ◦ ϏϧυγεςϜ (Makefile) ͰࠩҟΛٵऩ ◦ ΞʔΩςΫνϟ͝ͱʹArtifact Λ ฒߦϏϧυ ◦ s3://xxxxx/fluct-xxx-aarch64.tar.g z ◦ s3://xxxxx/fluct-xxx-x86_64.tar.g z ◦ Amazon ECR Image (docker buildx) Ϧϑτˍγϑτ
  21. ͍ͪΞϓϦέʔγϣϯͰGraviton2ҠߦલޙͷίετਪҠʢCost Explorerʣ ޿ࠂ഑৴αʔόʔ͸શͯGraviton2΁Ҡߦ ◦ ͋ͱ͸ΦϯϓϨϛεଆͷঈ٫ʹ߹Θͤͯਵ࣌ൺ཰มߋ ύϑΥʔϚϯε޲্ʹΑΔίετ࡟ݮ ◦ ϐʔΫ࣌ ඵؒ 3

    ~ 5 ສϦΫΤετΛࡹ͖ͭͭ ◦ Πϯελϯε୆਺͕ C5d ͱൺ΂ 60% ͰՔಇ (C6gd ) ◦ Price͸ 30 ~ 40% ࡟ݮʢεϙοτՁ֨มಈ͋Γʣ ◦ ٯʹ୆਺ͷগͳ͍ʢΠϯελϯε਺୆ͷʣαϒγεςϜͰ ͸ͦ͜·Ͱֹۚ͸࡟ݮ͞Εͳ͔ͬͨ Ϧϑτˍγϑτ
  22. Spot by NetAp p “Going beyond cloud analytics and recommendations,

    Spot by NetApp automates and optimizes your cloud infrastructure in AWS, Azure or Google Cloud to deliver SLA-backed availability and performance at the lowest possible cost.” - https://docs.spot.io/ Case fluc t ◦ AWSͰҰ޻෉ඞཁͳ෦෼Λ୅ସར༻ ◦ EC2, ECS on EC2 ͷεϙοτΠϯελϯε؅ཧ ◦ Cost Analyzing for Multi Clou d Ϧϑτˍγϑτ
  23. Deployment U I AWS Code γϦʔζͰσϓϩΠج൫Λߏங͢ΔͷͰ͸ͳ͘ Spot ElastiGroup (ίϯϐϡʔτϦιʔε؅ཧʣΛར༻ ◦

    batch size ࢦఆʢn%ͣͭΠϯελϯεೖΕସ͑ʣ ◦ ࣗಈͰELB΁ͷdetach / attac h AutoScal e ◦ ϦΫΤετ࣮੷͔Βͷ༧ଌεέʔϦϯά ◦ %ϕʔεͷΩϟύγςΟࢦఆ ◦ AWS্Ͱ࣮ݱͤ͞Δʹ͸ Lambda ར༻Λ૝ఆ ◦ Scheduled Scalin g ◦ னɾ໷ͷϐʔΫ௚લʹઌʹεέʔϧΞ΢τ͍ͯ͠Δ ◦ ECS on EC2 εέʔϧΠϯ௚લͷࣗಈdraining Ϧϑτˍγϑτ ഑৴αʔόʔͷσϓϩΠ࣌ͷը໘
  24. EC2 Spot Request, Spot Instanc e ◦ SpotՁ֨͸ࣗಈௐ੔ ◦ ઓུΛબ୒Մೳʢίετॏࢹɾόϥϯεܕ౳ʣ

    ◦ ྫ. AZຖʹ୆਺Λἧ͑Δ ◦ EC2 Spot InstanceมಈʹΑΔΦϯσϚϯυࣗಈ੾׵͑ ◦ fluct Ͱ͸͢΂ͯεϙοτΠϯελϯεࢦఆ ◦ C6g εϙοτΠϯελϯεΛ༏ઌ ◦ ރׇͨ͠৔߹ʹ M6g ͕ར༻͞ΕΔ ΦϯσϚϯυൺֱͰׂҾ͞ΕͨྉۚͷҰఆׂ߹Λɺ Spot by NetApp ʹࢧ෷͍ Ϧϑτˍγϑτ spot priceߴಅ࣌ɺΦϯσϚϯυΠϯελϯεʹࣗಈͰ੾ΓସΘΔ༷ࢠ
  25. Ϧϑτˍγϑτ FYI: ΍Βͳ͔ͬͨ͜ͱ ◦ AWS τϨʔχϯά ◦ शख़౓ʹΑͬͯ͸࣮ࢪ͔ͨ͠΋ ◦ ંΛݟͯ

    ◦ OSมߋ ◦ ·ͣ͸ΦϯϓϨϛεͱಉ͡Ͱ (CentOS վΊ RockyLinux) ◦ Ubuntu ৐Γ׵͑͸গ͠໎ͬͨʢ͍͔ͭػձ͕͋Ε͹ ◦ ECS ͸ Bottlerocket Ͱͳ͘ Amazon Linuxʢͨͩͷ༷ࢠݟ
  26. Ϧϑτˍγϑτ FYI: ࠓޙ΍Γͦ͏ͳ͜ͱ ◦ ϚωʔδυαʔϏεʹ৐ΕΔ΋ͷ͸৐Δ ◦ ʢ͓଴ͪͯ͠·͢ʂʣAmazon TimeStream at Tokyo

    regio n ◦ Region୯Ґͷো֐࣌ͷϑΣΠϧΦʔόʔʢOsaka Regionʣ ◦ ʢ͓଴ͪͯ͠·͢ʂʣEC2 Graviton2 at Osaka region