Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MENTAをAWSに移行して振り返る(ECS/Fargate + Laravel編)

MENTAをAWSに移行して振り返る(ECS/Fargate + Laravel編)

https://lancersrecruit.connpass.com/event/219434/

【SPACEMARKET×Lancers】シェアリングエコノミーを支えるインフラ/SREでのスライドとなります。

Avatar for adachin0817

adachin0817

August 25, 2021
Tweet

More Decks by adachin0817

Other Decks in Programming

Transcript

  1. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ΞδΣϯμ  ɾࣗݾ঺հ ɾ3೥લͷγεςϜߏ੒/͘͞ΒͷΫϥ΢υʹ͍ͭͯ(ۤ࿑࿩) ɾ։ൃ؀ڥΛ࡞ΔϙΠϯτ ɾStg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ ɾCircleCIʹΑΔίϯςφσϓϩΠ ɾECS Scheduled

    TasksʹΑΔόον࣮૷ʹ͍ͭͯ ɾϩά/ίϯςφ؂ࢹʹ͍ͭͯ ɾຊ൪DBϚεΩϯάࣗಈԽʹ͍ͭͯ ɾECSΦʔτεέʔϧʹ͍ͭͯ ɾ෼ੳج൫ͷ঺հ(Digdag,Embulk,BigQuery,Redash) ɾҠߦલͷ՝୊ͱҠߦޙͷτϥϒϧ ɾ·ͱΊ
  2. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE 3೥લͷγεςϜߏ੒  ɾ2018೥ϦϦʔεɹ ɾ෭ۀͰࣗ෼͕αʔόʔอक͍ͯͨ͠ 
 ɾ͘͞ΒͷΫϥ΢υ ɾPHP 7.2 /

    Laravel 5.5 ɹɾNginx / MariaDB /Redis / Ansible ɹɾSendGrid/Redash 
 ɹɾ؂ࢹ/Mackerel ɹɾ΄ͱΜͲVPSతͳӡ༻ ɾ։ൃ؀ڥ ɹɾDocker ɾϦϦʔεํ๏ ɹɾSSHʹΑΔγΣϧεΫϦϓτ
  3. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ։ൃ؀ڥΛ࡞ΔϙΠϯτ  ɾຊ൪؀ڥͱಉ౳ͷߏ੒Λ࠶ݱ ɾELB(Nginx)ίϯςφ(ϦόʔεϓϩΩγ) ɾApp(PHP7.2-alpine/Nginx) ɾMySQL 5.7 ɾSendGrid/phpMyAdmin/Redis ɾStripe/Selenium/node/Digdag(Embulk)

    ɾWordPress(PHP7.3-alpine/Nginx) ɾ֤ΞϓϦέʔγϣϯͷىಈ͸SupervisorͰ؅ཧ ɾcomposer install͸AppίϯςφͰϩάΠϯ࣌ʹ࣮ߦ ɹɾsupervisorctlͰΞϓϦΛrestart͢ΔΑ͏ʹ 
 ɾM1 MacͰ΋ಈ࡞ݕূࡁΈ
  4. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Stg/ຊ൪؀ڥͷߏ੒ʹ͍ͭͯ  TerraformͰશϦιʔείʔυԽ ɾStaging؀ڥ͸ۃྗίετΛ཈͑λεΫ਺͸1ͭ ɾࣾ಺༻͸ϥϯαʔζRProxyͰΞΫηε(phpMyAdmin,SendGrid,Digdag,Redash) ɾWordPressίϯςφʹΑΔϝσΟΞӡ༻ ɾphpMyAdminίϯςφʹΑΔσʔλग़͠ ɾΦʔτεέʔϧʹΑΔ࠷େ6୆ͷAppίϯςφͰӡ༻ ɾdevops/SSHίϯςφͷར༻Ͱaws

    cli.MySQLͷૢ࡞ ɹɾݱࡏ͸ഇࢭ͕ͨ͠EC2ຊ൪DBϚεΩϯά༻ͷ؀ڥͱͯ͠ར༻ ɾDigdag/Embulk/BigQueryʹΑΔ෼ੳج൫ͷߏங ɾDB͸RDS AuroraʹҠߦ ɾElastiCacheʹΑΔϩάΠϯΩϟογϡԽ ɾը૾౳S3Խ ɾϩά͸CloudWatch Logs ɾόον͸ECS Scheduled Tasks ɾ಺෦؂ࠪ(ϩάͷू໿) ɹɾAmazon Kinesis ɾAWS KMSʹΑΔ҉߸Խ ɾAWS WAFͰͷDDoSΞλοΫ੍ޚ ɾ؂ࢹ͸DatadogͰίϯςφϦιʔεͱAPMͰͷΤϥʔݕ஌ ɾຊ൪؀ڥ͸ΧφϦΞ؀ڥ΋
  5. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE SSH/࡞ۀ༻ίϯςφʮdevopsʯͷݱࡏ  ɾdevopsίϯςφΛഇࢭ(2021/04~) ɾECS Execͷొ৔ ɾAppίϯςφʹ௚઀ϩάΠϯ ɾγΣϧεΫϦϓτͰ࣮૷ ɹɾݱࡏ͸ecspressoͰίϯςφ؅ཧ ↑ݸਓϒϩάΛνΣοΫ👍

    $ ls /etc/ecspresso/menta/ README.md app.yaml digdag.yaml $ export AWS_PROFILE=menta $ ecspresso --config /etc/ecspresso/menta/app.yaml exec --command bash $ ecspresso --config /etc/ecspresso/menta/digdag.yaml exec --command bash
  6. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE APIܦ༝ͰͷStg؀ڥͷσϓϩΠ  ɾStg؀ڥͷσϓϩΠ ɹɾgithub flow ɹɾγΣϧεΫϦϓτ 
 ɹɹɾ೚ҙͷϒϥϯνΛࢦఆ ɾૉૣ͘σϓϩΠ͕Մೳ

    ɹɾ։ൃޮ཰͕Ξοϓ ɾMasterϚʔδͰຊ൪؀ڥϦϦʔε ɾTerraform CI؀ڥΛ࣮૷ $ sh deploy.sh ԼهͷΑ͏ʹϒϥϯνΛࢦఆ࣮ͯ͠ߦ͍ͯͩ͘͠͞ɻ (ྫ) sh deploy.sh ϒϥϯν໊ $ sh deploy.sh fix-nginx { "number" : 74, "state" : "pending", "id" : "xxxxxx-xxxxx-xxxxx-xxx-xxxxxxxx", "created_at" : “2021-06-27T06:14:43.973Z" } ϒϥϯν໊/ fix-nginx ͷStgσϓϩΠΛ։࢝͠·ͨ͠!
  7. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ECS Scheduled TasksʹΑΔόον࣮૷ʹ͍ͭͯ  ɾࠓ·Ͱ 
 ɹɾLaravelͷλεΫεέδϡʔϧ ɾݱࡏ ɹɾECS

    Scheduled Tasks 
 ɹɹɾCloudWatchͷΠϕϯτϧʔϧ 
 ɹɹɾTerraformԽ { "containerOverrides": [ { "name": "menta-app", "command": [ "php", "artisan", "admin:coupon-mail:send" ] } ] }
  8. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ϩά/ίϯςφ؂ࢹʹ͍ͭͯ  ɾΞΫηε/ΞϓϦϩάɺRDS؂ࠪ/Τϥʔϩά 
 ɹɾCloudWatch Logsʹ ɹɾࠓ·Ͱ ɹɹɾLambdaͰS3ʹγϯΫɹɹ ɹɹɾPythonͰ࣮૷͍ͯͨͨ͠Ίӡ༻ίετΞοϓ

    ɹɾࠓճ ɹɹɾAmazon Kinesis Data FirehoseͰS3ʹू໿ ɹɹɾTerraformԽ 
 ɾ؂ࢹ ɹɾMackerel→DatadogʹҠߦ ɹɾλεΫఆٛʹdatadog-agentΛಈ࡞ͤ͞ΔΑ͏ʹ 
 ɹɾcpu:10ɺmemory:256 TerraformԽ ɹɾAPMͰͷΞϓϦΤϥʔݕ஌
  9. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ຊ൪DBϚεΩϯάࣗಈԽ 
  ɾStg؀ڥͷDB ɹɾຖि1ճຊ൪DBͱಉ͡σʔλΛೖΕͯ΄͍͠ʂ ɹɾݸਓ৘ใ͸΋ͪΖΜϚεΩϯά ɾdevops(EC2)ͷ෮׆ ɾECS/FargateͰ΍Ζ͏ͱࢥͬͨ ɹɾσΟεΫྔ͕গͳ͍ͨΊஅ೦

    ɾauto-masking ɹɾprd-masking.sh ຖि౔༵೔0࣌ʙ ɹɹɾDump ɹɾstg-masking-restore.sh ຖि౔༵೔5࣌ʙ ɹɹɾRestore ɾpost_slack.sh ݁ՌΛSlack௨஌ ɾ։ൃ؀ڥ(MySQLίϯςφ) ɾauto-renew-mysql-container ຖि౔༵೔9࣌ʙ ɾCodeBuild ɹɾMySQLίϯςφΛbuild͠ϚεΩϯάσʔλΛRestore
  10. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦ࣌ͷ՝୊ͱΑ͔ͬͨ͜ͱ  ɾϩάΛίϯςφ಺Ͱ؅ཧͯ͠͠·͏ͱσϓϩΠ͢Δͨͼফ͑ͯ͠·͏ ɹɾCloudWatch Logsʹඪ४ग़ྗ͠ͳ͚Ε͹ͳΒͳ͍ ɹɾLaravelΞϓϦϩάͷstdout͕5.8Ҏ্͡Όͳ͍ͱରԠͰ͖ͳ͍ ɹɾόʔδϣϯΞοϓΛߦͳͬͨ ɾը૾ܥ ɹɾS3

    ը૾Ξοϓϩʔυ͕Ұ෦ͷػೳ͔͠ରԠ͍ͯ͠ͳ͔ͬͨ 
 ɹɹɾ1ϲ݄Ҏ্͔͚ͯҠߦ ɾόον΍աڈͷاըͷιʔεΛ୨Է ɹɾෛ࠴ͱͳ͍ͬͯͨιʔεΛ੔ཧͰ͖ͨ ɾUnit/Feature/E2Eͷࣗಈςετ੔උ ɹɾPHPCSΛ࢖ͬͨن໿౷Ұ ɹɾPHPStanʢLarastanʣ࢖ͬͨ੩తղੳͰͷ඼࣭޲্όά࡟ݮ ɹɾϦϑΝΫλϦϯά 💦
  11. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE Ҡߦޙͷτϥϒϧ  ɾECS Scheduled Tasks ɹɾຖ෼ͷόονͰNAT Gatewayͷ௨৴අ͕ංେԽ 
 ɹɾECR΍CloudWatch

    Logsͷ޲͖ઌΛมߋ 
 ɹɹɾVPCΤϯυϙΠϯτܦ༝ʹ 
 ɾଞͷάϧʔϓձࣾ΋10ສԁίετΞοϓ͍ͯͨ͠ 
 ɹɾ೥ؒ120ສ࡟ݮʹվળͨ͠ ɾݸਓϒϩάॻ͍͍ͯΔͷͰࢀߟʹʂ 💦
  12. 2021/08/25ʲLancers×SPACEMARKETʳγΣΞϦϯάΤίϊϛʔΛࢧ͑ΔΠϯϑϥ/SRE ɾࠓޙ ɹɾ։ൃ؀ڥͷDB΋ϚεΩϯά͞ΕͨDBΛར༻ 
 ɹɹɾcodebuild ɹɾPHP/LaravelͷόʔδϣϯΞοϓ ɹɾίϯςφͷ੬ऑੑݕ஌ (Trivy) ɹɾDatadogͰΞΫηεϩά࿈ܞ ɹɾϥϯαʔζຊՈ΋ίϯςφҠߦʹνϟϨϯδ

    
 ɹɹɾࣾ಺Ͱӡ༻͍ͯ͠Δ΋ͷ͸ҠߦࡁΈ ·ͱΊ  ɾ൒೥ͰҠߦ͢Δ͜ͱ͕Ͱ͖ͨ(2020/10~2021/03) 
 ɹɾٕज़తͳෛ࠴ͷ੔ཧΛߦ͍ҠߦΛ࣮ࢪͰ͖ͨ ɾاըɾΞΠσΟΞϕʔεͷϓϩτλΠϓ։ൃ͔Β 
 ɹɾকདྷΛݟਾ͑ӡ༻ɾอक΋ҙࣝͨ͠։ൃελΠϧʹ 
 ɹɾϥϯαʔζͱಉ͡։ൃελΠϧʹมΘͬͨ ɾImmutableͳαʔόʔͷӡ༻ ɹɾ҆ఆӡ༻ɺӡ༻ޮ཰ͷΞοϓ ɹɾ৑௕Խ΍ΦʔτεέʔϦϯάʹΑΔߴ͍Մ༻ੑΛ࣮ݱͨ͠ߏ੒ ɾDatadogͷAPMͰΞϓϦͷΤϥʔ௨஌͕Ͱ͖ΔΑ͏ʹ ɾϗετΠϯελϯεͷ؅ཧ͕ল͚Δ ɾλεΫ୯ҐͰࣗ༝ʹϦιʔεมߋՄೳ 
 ɾλεΫͷىಈ଎౓͕஗Ί ɹɾόον࣮ߦ΋਺෼஗͘ͳΔ 
 ɾ͘͞ΒͷΫϥ΢υ࣌୅ֹ݄Ͱ਺ेສԁˢ