Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Amazon ECS とマネージドサービスを活用したフルコンテナ構成によるゲームサーバの構築と...

Amazon ECS とマネージドサービスを活用したフルコンテナ構成によるゲームサーバの構築と運用 / Amazon-ECS-game-server

FUJIWARA Shunichiro

August 22, 2018
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. αʔϏε - Service λεΫΛෳ਺ݸଋͶͨ΋ͷ • ϩʔυόϥϯαʔ (ALB, NLB) ʹऩ༰ ͢Δ୯ҐʹͳΔ

    • ࢦఆͨ͠λεΫ਺Λҡ࣋͢Δ • λεΫ਺ΛΦʔτεέʔϧͰ͖Δ k8s ͷ Deployment ʹ૬౰
  2. ࠓ·Ͱͷ EC2 Ͱͷαʔόӡ༻ͱҧ͏఺ EC2 ্ʹΞϓϦέʔγϣϯ/ϛυϧ΢ΣΞΛىಈ → λεΫʹϓϩηεΛ·ͱΊΔ ϩʔυόϥϯαʔʹEC2ΠϯελϯεΛొ࿥ → αʔϏεΛొ࿥

    Φʔτεέʔϧ͸EC2ΠϯελϯεΛ૿ݮ͢Δ → λεΫΛ૿ݮ͢Δ …ͨͩ͠ EC2 ίϯςφΠϯελϯε΋૿ݮ͠ͳ͍ͱ࣮ࡍʹ࢖͑ΔϦιʔ ε͸มΘΒͳ͍ / Fargate Ͱ͸ղܾ
  3. ECS ʹͯ͠Α͔ͬͨ͜ͱ αʔόͷ௥Ճ࡟আָ͕ ΫϥελͷEC2Πϯελϯε͸͢΂ͯಉ͡΋ͷ ͜Ε·Ͱ͸αʔόͷछྨ͝ͱʹΠϯελϯεΛ࡞͍ͬͯͨ App, WebSocket, Batch, ϩάू໿ etc...

    OS ʹ࠶ىಈඞਢͳύονΛ౰͍ͯͨ৔߹ 1.৽͍͠ AMI ͔ΒΠϯελϯεىಈ 2.λεΫΛ৽͍͠ΠϯελϯεʹҠಈ 3.ݹ͍ΠϯελϯεΛࣺͯΔ
  4. ECS Ͱ Spot ΠϯελϯεΛ҆શʹ੾Γ཭͢ ΦϯσϚϯυͷधཁ͕૿͑ͯ Spot ΁ׂΓ౰ͯΔϦιʔε͕ ͳ͘ͳΔͱ120ඵલʹ termination ௨஌͕དྷΔ

    http://169.254.169.254/latest/meta-data/spot/instance-action Λ polling → ঢ়ଶมԽΛݕ஌ͨ͠Βࣗ෼ࣗ਎ΛΫϥελ͔Β֎͢ ͜ΕΛ͠ͳ͍ͱλεΫ͕ಥવࢮ͢Δ
  5. ֤ΠϯελϯεͰୀ໾ॲཧλεΫΛಈ͔͢ #!/bin/bash while sleep 5; do CONTENT=$(curl -sf http://169.254.169.254/latest/meta-data/spot/instance-action) if

    [ -z "$CONTENT" ]; then continue fi CLUSTER=$(curl -s http://localhost:51678/v1/metadata | jq -r .Cluster) CONTAINER_INSTANCE=$(curl -s http://localhost:51678/v1/metadata | jq -r .ContainerInstanceArn) aws ecs update-container-instances-state \ --cluster "$CLUSTER" \ --container-instances "$CONTAINER_INSTANCE" \ --status DRAINING \ && exit 0 done localhost:51678 = ECS agent ! ਖ਼௚͜Ε͙Β͍͸ ECS agent ͕΍ͬͯ΄͍͠
  6. ͜Ε·Ͱͷ EC2 ্ͷσϓϩΠ • ֤ EC2 ʹΞʔΧΠϒϑΝΠϧΛ഑෍ • S3 ʹ

    tar.gz ΛΞοϓϩʔυ • ֤ϗετ͕ S3 ͔Βμ΢ϯϩʔυ ͯ͠ల։ • ϓϩηεΛ graceful ʹ࠶ىಈ Consul + Stretcher (ࣗ࡞OSS) Ͱ࣮ݱ
  7. ECSʹ͓͚ΔσϓϩΠ ECS ͕΍Δ͜ͱ ৽͍͠λεΫఆٛͰλεΫΛىಈ (LB ͕͋Δ৔߹͸ LB ʹ૊ΈࠐΉ) ਖ਼ৗʹىಈɺ૊ΈࠐΊͨΒݹ͍λεΫΛ Λམͱ͢

    ਖ਼ৗʹىಈ͠ͳ͔ͬͨΒݹ͍΋ͷ͸མͪ ͳ͍ αʔϏε಺ͷ͢΂ͯͷλεΫ͕৽͍͠΋ ͷʹͳͬͨΒ׬ྃ
  8. σϓϩΠπʔϧ ͍Ζ͍Ζ͋Δ ecs-cli github.com/aws/amazon-ecs-cli ɹGo / docker-compose Ͱఆٛͨ͠ઃఆΛ ECS ʹσϓϩΠ

    hako github.com/eagletmt/hako ɹRuby / ఆٛ͸ YAML ecs-deploy github.com/silinternational/ecs-deploy ɹbash + aws-cli
  9. { "taskDefinition": { "cpu": "{{ env `VCPU_NUM` `4` }} vCPU",

    "containerDefinitions": [ { "name": "app", "image": "{{ must_env `SERVICE` }}/app:{{ must_env `TAG` }}", VCPU_NUM : prod | staging ͰҟͳΔCPUΛՄมʹ SERVICE : prod | staging σϓϩΠઌͷ؀ڥΛ੾Γସ͑Δ TAG : σϓϩΠ͝ͱʹมΘΔΠϝʔδͷλά ͻͱͭͷ JSON Λෳ਺؀ڥʹ࢖͍ճͤΔͷͰҡ࣋؅ཧָ͕
  10. SSMύϥϝʔλετΞ • ֊૚Խͨ͠ path ໊Ͱ஋Λอଘ • ઃఆ஋ͷόʔδϣχϯάɺมߋऀͷཤ ྺ؅ཧ • KMSͰ҉߸Խͨ͠஋ΛอଘͰ͖Δ

    • KMS΁ͷݖݶ͕ͳ͚Ε͹஋͸ಡΊ ͳ͍ • ίϯςφىಈ࣌ʹ͔͜͜Β஋ΛऔΓ ग़ͯ͠؀ڥม਺ʹઃఆ͢Ε͹Α͍
  11. ύϥϝʔλετΞ͔Β஋ΛऔΓग़͢ aws-ssm-env github.com/jamietsao/aws-ssm-env $ aws-ssm-env --paths=/prod/ API_KEY=xxxxxx DB_PASS=productionpass entry point

    ͷ sh Ͱ࣮ߦɺexport ͔ͯ͠Β exec ͢Δ #!/bin/sh export AWS_REGION=ap-northeast-1 export $(aws-ssm-env --paths=/prod/) exec /path/to/myapp
  12. ύϥϝʔλετΞͷ஋Λ export & exec ssmwrap github.com/handlename/ssmwrap ฐࣾಉ྅࡞ $ ssmwrap -paths=/prod/

    -- /path/to/myapp ssm ͔Βऔಘͨ͠஋Λ env ʹઃఆͯ͠ exec ·Ͱͯ͘͠ΕΔ ɾexec ·Ͱ͢ΔͷͰ Docker ͷ entrypoint ʹ௚઀ࢦఆͰ͖Δ ɾshell scirpt Λܦ༝͠ͳ͍ͷͰվߦΛؚΉ஋ͷ໰୊͕ͳ͍ ɾ-retries ΦϓγϣϯͰϦτϥΠճ਺ΛࢦఆͰ͖Δ
  13. SSMύϥϝʔλετΞͷ໰୊ API Rate Limit ͕ݫ͠Ί େྔͷλεΫΛҰؾʹىಈ͢Δͱ Rate Limit Error (੍ݶ஋͸αϙʔτʹ໰͍߹Θ͕ͤͨඇެ։)

    ɾaws-ssm-env ͸ΤϥʔʹͳΔͱ panic Ͱࢮ͵ ɹσϓϩΠ࣌ʹࢮΜͩλεΫ͸࠶౓ىಈ͞ΕΔ͕… ɾssmwrap ͸ -retries ઃఆͰϦτϥΠͰ͖Δ(͓નΊ) ɾىಈ࣌ʹ random sleep ͰλΠϛϯάΛͣΒͯ͠؇࿨ ! ਖ਼௚ECSଆͰ؀ڥม਺ʹઃఆͯ͠ىಈͯ͠΄͍͠
  14. ֤ίϯςφ͕ STDOUT, STDERR ʹు͖ग़ͨ͠΋ͷ Docker logging driver ʹΑͬͯѻΘΕΔ λεΫఆٛͰίϯςφ͝ͱʹࢦఆͰ͖Δ awslogs

    : CloudWatch Logs ΁ૹ৴ fluend : Fluentd ΁ૹ৴ json-file : ϑΝΠϧ΁อଘ syslog : syslogd ΁ૹ৴ (ଞʹ΋͋Δ)
  15. awslogs or fluentd? CloudWatch Logs ɹpros: ϚωʔδυͳͷͰԿ΋ߟ͑ͳͯ͘Α͍ ɹcons: ͔ͦ͜Βߋʹผʹྲྀ͢ͷ͕ଟগ໘౗ (Lambda

    ͕ඞཁ) Fluentd: ɹpros: ߦಈϩάଞͱಉ༷ͷѻ͍͕Ͱ͖Δ ɹcons: fluentdΛࣗ෼ͰཱͯΔඞཁ͕͋Δ
  16. Fluentd ʹ౷Ұ͢Δ ߦಈϩά͸ͲͪΒʹ͠Ζ Fluentd Ͱѻ͍͍ͨͷͰඞཁ ߦಈϩά ɾλεΫʹΞϓϦέʔγϣϯͱҰॹʹ fluentd Λ഑ஔ (sidecar)

    ɾΞϓϦ͔Β localhost:24224 ʹૹ৴͢Δ(ߏ଄Խϩά) STDOUT, STDERR ɾECSαʔϏεͱͯ͠ NLB Λհͯ͠ fluentd Λ഑ஔ ɾ֤λεΫ͸ Docker logging driver Ͱ NLB:24224 ΁ૹ৴͢Δ
  17. ECS Ͱ fluentd ΛͲ͏࢖͏͔ AWS ͷ৔߹ fluentd + plugin S3

    Ͱ S3 ΁อ؅͢Δͷ͕ఆ൪ S3 ΁ͷॻ͖ग़͠͸ 1ʙ5 ෼ఔ౓ʹ͢Δ ɹ͋·Γ୹͍ͱ S3 ͷΦϒδΣΫτ͕ࡉ੾ΕʹͳΔ) ো֐࣌ͷ͜ͱΛߟ͑Δͱ਺෼ఔ౓Ͱ΋ίϯςφ಺ͷόοϑΝ (ϝϞϦ or ϩʔΧϧϑΝΠϧ)ʹอଘ͢Δͷ͸ফࣦϦεΫ → ৴པͷஔ͚ΔόοϑΝ͕΄͍͠
  18. ʮtail -f ͍ͨ͠ΜͰ͕͢ʯ ϩά͕ϑΝΠϧʹॻ͔Εͳ͘ͳͬͨ → tail -f ͸Ͳ͏͢Δ? kinesis-tailf github.com/fujiwara/kinesis-tailf

    Go + aws-sdk-go Ͱࣗ࡞ $ kinesis-tailf -stream docker-logs Kinesis Streams Λ௥ඌ͠ඪ४ग़ྗʹు ͖ग़͢ CLI -start -end ΦϓγϣϯͰಛఆ࣌ؒଳΛ நग़Մೳ
  19. Τϥʔϩά͚ͩநग़͍ͨ͠ શϩάΛ kinesis-tailf | grep ? ྲྀྔ͕ଟ͍ͱͭΒ͍ Streams ͷσʔλΛ Lambda

    Ͱॲཧ ύλʔϯʹϚονͨ͠ϩά͚ͩผͷ Streams ʹྲྀ͠௚͢ ɾαʔόΤϥʔ status:5 ɾॲཧʹ͕࣌ؒ3ඵҎ্ֻ͔ͬͨ΋ͷ ɹapptime:([3-9]|[1-9][0-9]+)\. ɾ(ERROR|WARN)
  20. ϩάͷू໿ͱετϦʔϛϯάॲཧ ·ͱΊ • ϩά͸ STDOUT/STDERR ʹग़ྗ Docker Logging Driver Ͱૹ৴

    • Fluentd + ৴པͰ͖ΔόοϑΝͱͯ͠ͷ Kinesis Streams • Firehose + S3 + Lambda ͰΠϕϯτυϦϒϯͳσʔλॲཧ΋
  21. AWS / ECS ͸伱͕ؒଟ͍ ࠓճࣗલͰ։ൃͨ͠΋ͷ ecspresso github.com/kayac/ecspresso σϓϩΠπʔϧ ssmwrap github.com/handlename/ssmwrap

    SSM ύϥϝʔλετΞͷ஋Λ؀ڥม਺ʹઃఆͯ͠ exec kinesis-tailf github.com/fujiwara/kinesis-tailf Kinesis StreamsΛ௥ඌ͢Δ
  22. 伱ؒՈ۩ͷྫ Rin github.com/fujiwara/Rin Redshift data Importer by SQS messaging. S3

    Πϕϯτ௨஌Ͱ SQS ʹૹ৴ SQS ͷϝοηʔδ͔Β Redshiftʹ copy ͯ͠औΓࠐΈΛߦ͏πʔϧ 2015.05 ʹ։ൃ 2015.10 Firehose ൃද 2017.07 ౦ژϦʔδϣϯʹ Firehose
  23. ίϯςφ؀ڥ޲͚ͷ։ൃ Tips ίϚϯυϥΠϯΦϓγϣϯͷ஋Λ؀ڥม਺͔Β΋ಡΉ1 func main() { var port int var

    host string flag.IntVar(&port, "port", 8080, "port number") flag.StringVar(&host, "host", "localhost", "hostname") flag.VisitAll(func(f *flag.Flag) { if s := os.Getenv(strings.ToUpper(f.Name)); s != "" { f.Value.Set(s) } }) flag.Parse() fmt.Printf("%s:%d\n", host, port) } 1 https://mattn.kaoriya.net/software/lang/go/20170609110526.htm
  24. ίϚϯυϥΠϯΦϓγϣϯͷ஋Λ؀ڥม਺͔Β΋ಡΉ $ myapp -host example.com -port 9999 example.com:9999 $ HOST=example.com

    PORT=9999 myapp example.com:9999 ίϯςφͰ͸ίϚϯυϥΠϯΑΓ ؀ڥม਺ͷ΄͏͕ઃఆ͠΍͍͢
  25. ઃఆϑΝΠϧʹ؀ڥม਺͔Β஋ΛຒΊࠐΉ github.com/kayac/go-config Go ͷઃఆϑΝΠϧಡΈࠐΈ package # yaml account_id: "{{ must_env

    `MY_ACCOUNT_ID` }}" profile: "{{ env `PROFILE` `default` }}" must_env: ઃఆ͞Εͳ͍ঢ়ଶͰ࣮ߦ͞ΕΔͱ panic env: ઃఆ͞Ε͍ͯͳ͍৔߹͸σϑΥϧτ஋Λࢦఆ YAML / JSON / TOML ʹରԠ
  26. ઃఆϑΝΠϧΛϦϞʔτ͔Βऔಘ͢Δ ઃఆ͕ϑΝΠϧʹͳ͍ͬͯΔͱ มߋͷࡍʹίϯςφͷ࠶Ϗϧυ & Ξοϓϩʔυ͕ඞཁ (ࢼߦࡨޡ͕໘౗) File / S3 /

    HTTP ͰऔಘͰ͖ΔΑ͏ʹ࡞͓ͬͯ͘ͱศར2 $ CONFIG=s3://my-config-bucket/config.yaml mydaemon 2 GoͰHTTPͱS3Λಁաతʹѻ͏ https://shogo82148.github.io/blog/2018/06/09/go-s3-protocol/