Amazon ECS とマネージドサービスを活用したフルコンテナ構成によるゲームサーバの構築と運用 / Amazon-ECS-game-server

CEDEC 2018 の講演資料です
https://2018.cedec.cesa.or.jp/session/detail/s5abc66488c745

Transcript

1. Amazon ECS ͱ ϚωʔδυαʔϏεΛ׆༻ͨ͠ ϑϧίϯςφߏ੒ʹΑΔ ήʔϜαʔόͷߏஙͱӡ༻ 2018.08.22 CEDEC 2018 ໘ന๏ਓΧϠοΫ ౻ݪढ़Ұ࿠
2. ࣗݾ঺հ @fujiwara github.com/fujiwara sfujiwara.hatenablog.com
3. Game & Community
4. Agenda • Amazon ECS Ͱͷαʔόߏஙͱӡ༻ • σϓϩΠख๏ͱൿಗ৘ใͷ؅ཧ • ϩάͷू໿ͱετϦʔϛϯάॲཧ • GoݴޠʹΑΔӡ༻πʔϧ/ϛυϧ΢ΣΞ։ൃ
5. ࠷ॳʹ͓࿳ͼ ·ͨɺຊ൪ϦϦʔεલʹඞਢͰ͋Δෛՙࢼݧ΋ίϯςφΛ׆༻͢Δ͜ ͱͰॊೈʹߦ͑ΔͨΊɺͦͷख๏ʹ͍ͭͯ΋આ໌͠·͢ɻ0 ෛՙࢼݧʹ͍ͭͯ͸ऩ·Γ͖Βͳ͔ͬͨͨΊ ผͷࢿྉΛ͝ࢀর͍ͩ͘͞ ! GoͱίϯςφͰ࡞ΔWebΞϓϦέʔγϣϯϕϯνϚʔΧʔ speakerdeck.com/fujiwara3/benchmarker-built-with-go-and-container 0 https://2018.cedec.cesa.or.jp/session/detail/s5abc66488c745
6. Amazon ECS Ͱͷαʔόߏஙͱӡ༻
7. Amazon ECS AWSͷίϯςφ؅ཧϚωʔδυαʔϏε ͍ΘΏΔʮΦʔέετϨʔγϣϯπʔϧʯ ෳ਺ͷίϯςφͰىಈͨ͠ϓϩηε܈Λ؅ཧ͢Δ΋ͷ ଞʹ༗໊ͳͷ͸ Kubernetes(k8s)
8. ECS ͷ֓೦
9. λεΫ - Task ؔ࿈͢ΔίϯςφϓϩηεΛҰͭʹ·ͱ Ίͨ΋ͷ λεΫ಺ͷίϯςφ͸૬ޓʹ௨৴Ͱ͖Δ (localhost or hostname) k8s ͷ Pod ʹ૬౰
10. λεΫఆٛ - Task Definition λεΫͷߏ੒Λఆٛ͢Δ΋ͷ ؚ·ΕΔ(ෳ਺ͷ)ίϯςφɺλεΫࣗମ ͷωοτϫʔΫઃఆͳͲΛఆٛ • Πϝʔδ • CPU • ϝϞϦ • ϙʔτׂΓ౰ͯ ͳͲ
11. αʔϏε - Service λεΫΛෳ਺ݸଋͶͨ΋ͷ • ϩʔυόϥϯαʔ (ALB, NLB) ʹऩ༰ ͢Δ୯ҐʹͳΔ • ࢦఆͨ͠λεΫ਺Λҡ࣋͢Δ • λεΫ਺ΛΦʔτεέʔϧͰ͖Δ k8s ͷ Deployment ʹ૬౰
12. Ϋϥελʔ - Cluster λεΫ͕഑ஔ͞ΕΔ EC2 Πϯελϯεͷू߹ EC2 Πϯελϯεࣗମ͕Ϛωʔδυʹͳͬͨ Fargate ΋
14. ࠓ·Ͱͷ EC2 Ͱͷαʔόӡ༻ͱҧ͏఺ EC2 ্ʹΞϓϦέʔγϣϯ/ϛυϧ΢ΣΞΛىಈ → λεΫʹϓϩηεΛ·ͱΊΔ ϩʔυόϥϯαʔʹEC2ΠϯελϯεΛొ࿥ → αʔϏεΛొ࿥ Φʔτεέʔϧ͸EC2ΠϯελϯεΛ૿ݮ͢Δ → λεΫΛ૿ݮ͢Δ …ͨͩ͠ EC2 ίϯςφΠϯελϯε΋૿ݮ͠ͳ͍ͱ࣮ࡍʹ࢖͑ΔϦιʔ ε͸มΘΒͳ͍ / Fargate Ͱ͸ղܾ
15. ECS Ͱͷαʔόߏ੒ͷํ਑ ঢ়ଶΛ࣋ͨͳ͍ΞϓϦέʔγϣϯͷΈΛಈ͔͢ ௕࣌ؒͷঢ়ଶΛ࣋ͭϛυϧ΢ΣΞ͸ಈ͔͞ͳ͍ (RDBMSͳͲͷετϨʔδ) ঢ়ଶ͸͢΂ͯϚωʔδυαʔϏε΁อଘ͢Δ (RDS, S3, ElastiCache...)
16. ECS Ͱͷαʔόߏ੒ͷํ਑ ঢ়ଶΛ࣋ͨͳ͍ΞϓϦέʔγϣϯͷΈΛಈ͔͢ σϓϩΠͷͨͼʹ৽͍͠λεΫ͕ىಈ͠ɺݹ͍λεΫ͸མͪΔ λεΫ͕ऴྃ/མͪΔͱϑΝΠϧ΁ͷॻ͖ࠐΈ͸ࣦΘΕΔ EC2ϗετͷϑΝΠϧγεςϜΛϚ΢ϯτ͸Ͱ͖Δ͕… Ϋϥελ಺ͷͲͷEC2ͰλεΫΛىಈ͢Δ͔͸ECS͕ܾΊΔ → Ӭଓ͢Δঢ়ଶ͸͢΂ͯϚωʔδυαʔϏε΁ʂ ϩά΋ϩʔΧϧϑΝΠϧʹ࣋ͨͣϚωʔδυαʔϏε΁సૹ
17. ECS ʹͯ͠Α͔ͬͨ͜ͱ EC2্Ͱͷϛυϧ΢ΣΞͷߏ੒؅ཧ͕ෆཁ ͜Ε·Ͱ͸ Chef Ͱ؅ཧ ੈؒͰ͍͏΄Ͳਏ͘͸ͳ͍͕… ႈ౳ʹ͢Δͷ͕໘౗ɺΞϓϦέʔγϣϯΤϯδχΞʹ͸ෛ୲ → Dockerfile ͰΫϦʔϯͳঢ়ଶ͔ΒΠϯετʔϧ͢Δ͚ͩ
18. ECS ʹͯ͠Α͔ͬͨ͜ͱ αʔόͷ௥Ճ࡟আָ͕ ΫϥελͷEC2Πϯελϯε͸͢΂ͯಉ͡΋ͷ ͜Ε·Ͱ͸αʔόͷछྨ͝ͱʹΠϯελϯεΛ࡞͍ͬͯͨ App, WebSocket, Batch, ϩάू໿ etc... OS ʹ࠶ىಈඞਢͳύονΛ౰͍ͯͨ৔߹ 1.৽͍͠ AMI ͔ΒΠϯελϯεىಈ 2.λεΫΛ৽͍͠ΠϯελϯεʹҠಈ 3.ݹ͍ΠϯελϯεΛࣺͯΔ
19. ECS ʹͯ͠େมͩͬͨ͜ͱ ͜Ε·Ͱͱͷҧ͍Λҙࣝͯ͠৐Γӽ͑Δ ֓೦ΛΞϓϦέʔγϣϯΤϯδχΞʹཧղͯ͠΋Β͏ͷ͕େม ϩʔΧϧϑΝΠϧʹґଘ͠ͳ͍࢓૊ΈΛ࡞Δඞཁ͕͋Δ → ঢ়ଶΛ࣋ͨͳ͍͜ͱΛపఈ͢Δ ʲྫʳϩάΛ tail -f ͍͚ͨ͠ͲϑΝΠϧʹ͸ॻ͚ͳ͍… → ϚωʔδυαʔϏεʴࣗ࡞πʔϧͰղܾ (ޙड़)
20. ECS ʹͯ͠େมͩͬͨ͜ͱ ௚઀ϓϩηεͷঢ়ଶΛΈΔௐ͕ࠪ໘౗ ʮstrace ͍ͨ͠ʂʯ ɾૂͬͨλεΫ͕ಈ࡞͍ͯ͠ΔEC2ͷϗετΛݟ͚ͭΔ ɾEC2 ʹ ssh ͯ͠ docker exec sh & strace ίϯςφʹೖͬͯ΋ඞཁͳπʔϧ͕ͳ͔ͬͨΓ͢Δ
21. ECS / ίϯςφԽͷ෭࡞༻ ঢ়ଶΛ࣋ͨͳ͍ɺϗετ͸͍ͭফ͑ͯ΋໰୊ͳ͍Α͏ʹͳͬͨ → Spot Πϯελϯε͕׆༻Ͱ͖Δ ௨ৗͷΦϯσϚϯυΠϯελϯεͷ༨৒ϦιʔεΛ҆͘ (30%͙Β͍Ͱ) ఏڙ͢Δ΋ͷ ͨͩ͠༨৒Ϧιʔε͕ͳ͘ͳΔͱམͱ͞ΕΔ
22. Spot Fleet ෳ਺ͷΠϯελϯελΠϓ AZ Λ૊Έ߹ΘͤͯϦιʔεΛ֬อ མͪͨ෼͸ࣗಈతʹ୅ସ͕ىಈ
23. ECS Ͱ Spot ΠϯελϯεΛ҆શʹ੾Γ཭͢ ΦϯσϚϯυͷधཁ͕૿͑ͯ Spot ΁ׂΓ౰ͯΔϦιʔε͕ ͳ͘ͳΔͱ120ඵલʹ termination ௨஌͕དྷΔ http://169.254.169.254/latest/meta-data/spot/instance-action Λ polling → ঢ়ଶมԽΛݕ஌ͨ͠Βࣗ෼ࣗ਎ΛΫϥελ͔Β֎͢ ͜ΕΛ͠ͳ͍ͱλεΫ͕ಥવࢮ͢Δ
24. ֤ΠϯελϯεͰୀ໾ॲཧλεΫΛಈ͔͢ #!/bin/bash while sleep 5; do CONTENT=$(curl -sf http://169.254.169.254/latest/meta-data/spot/instance-action) if [ -z "$CONTENT" ]; then continue fi CLUSTER=$(curl -s http://localhost:51678/v1/metadata | jq -r .Cluster) CONTAINER_INSTANCE=$(curl -s http://localhost:51678/v1/metadata | jq -r .ContainerInstanceArn) aws ecs update-container-instances-state \ --cluster "$CLUSTER" \ --container-instances "$CONTAINER_INSTANCE" \ --status DRAINING \ && exit 0 done localhost:51678 = ECS agent ! ਖ਼௚͜Ε͙Β͍͸ ECS agent ͕΍ͬͯ΄͍͠
25. Amazon ECS Ͱͷαʔόߏஙͱӡ༻ ·ͱΊ • ΞϓϦέʔγϣϯʹঢ়ଶΛ࣋ͨͳ͍͜ͱΛపఈ͢Δ • ঢ়ଶΛ࣋ͨͳ͍͜ͱͰಘΒΕΔϝϦοτΛڗड͠Α͏
26. σϓϩΠख๏ͱൿಗ৘ใͷ؅ཧ
27. ͜Ε·Ͱͷ EC2 ্ͷσϓϩΠ • ֤ EC2 ʹΞʔΧΠϒϑΝΠϧΛ഑෍ • S3 ʹ tar.gz ΛΞοϓϩʔυ • ֤ϗετ͕ S3 ͔Βμ΢ϯϩʔυ ͯ͠ల։ • ϓϩηεΛ graceful ʹ࠶ىಈ Consul + Stretcher (ࣗ࡞OSS) Ͱ࣮ݱ
28. ECSʹ͓͚ΔσϓϩΠ Ϣʔβ͕΍Δ͜ͱ ৽͍͠ΞϓϦέʔγϣϯؚ͕·ΕͨΠ ϝʔδΛ ECR ʹొ࿥ ৽͍͠ΠϝʔδΛ࢖͏λεΫఆٛΛొ࿥ αʔϏεઃఆͰ৽͍͠λεΫఆٛΛ࢖͏ Α͏ʹมߋ
29. ECSʹ͓͚ΔσϓϩΠ ECS ͕΍Δ͜ͱ ৽͍͠λεΫఆٛͰλεΫΛىಈ (LB ͕͋Δ৔߹͸ LB ʹ૊ΈࠐΉ) ਖ਼ৗʹىಈɺ૊ΈࠐΊͨΒݹ͍λεΫΛ Λམͱ͢ ਖ਼ৗʹىಈ͠ͳ͔ͬͨΒݹ͍΋ͷ͸མͪ ͳ͍ αʔϏε಺ͷ͢΂ͯͷλεΫ͕৽͍͠΋ ͷʹͳͬͨΒ׬ྃ
30. ϩʔϧόοΫ ݹ͍λεΫఆٛΛ࢖͏Α͏ʹαʔϏεΛมߋ͢Δ ಛʹ ECS ʹػೳ͕͋ΔΘ͚Ͱ͸ͳ͍ Ϣʔβ͕ࣗ෼ͰʮҰͭલʹσϓϩΠ͍ͯͨ͠λεΫఆٛʯ ΛσϓϩΠ͠௚͢ ! ىಈޙʹಈతͳ͜ͱΛͳΔ΂͘͠ͳ͍ͷ͕ॏཁ ! ίϯςφىಈޙʹ࠷৽ίʔυΛϨϙδτϦ͔Βऔಘ " ίʔυ͸Πϝʔδʹম͖ࠐΉ
31. σϓϩΠπʔϧ ͍Ζ͍Ζ͋Δ ecs-cli github.com/aws/amazon-ecs-cli ɹGo / docker-compose Ͱఆٛͨ͠ઃఆΛ ECS ʹσϓϩΠ hako github.com/eagletmt/hako ɹRuby / ఆٛ͸ YAML ecs-deploy github.com/silinternational/ecs-deploy ɹbash + aws-cli
32. σϓϩΠπʔϧࣗ࡞ ecspresso github.com/kayac/ecspresso 1. ݩʑίϯιʔϧૢ࡞ͰσϓϩΠ͍ͯͨ͠΋ͷ͕͋ͬͨ 2. aws-cli ͰऔಘͰ͖ΔλεΫఆٛJSONΛͦͷ··࢖͏ͨΊʹ shell script Λॻ͍ͨ 3. ίʔυ͕ෳࡶʹͳͬͨͷͰ Go + aws-sdk-go Ͱॻ͖௚ͦ͏ ͍ͭͷؒʹ͔ҭͬͯ͠·ͬͨ
33. ecspresso ͷಛ৭ λεΫఆٛ͸ aws-cli Ͱѻ͏JSONΛͦͷ··࢖͑Δ →ݱঢ়ಈ͍͍ͯΔ΋ͷΛͦͷ··؅ཧର৅ʹͰ͖Δ JSON ͷதͷ஋Λ࣮ߦ࣌ͷ؀ڥม਺Ͱஔ׵ͯ͠ొ࿥Ͱ͖Δ { "taskDefinition": { "cpu": "4 vCPU", "containerDefinitions": [ { "name": "app", "image": "prod/app:f6fb4dcc20",
34. { "taskDefinition": { "cpu": "{{ env `VCPU_NUM` `4` }} vCPU", "containerDefinitions": [ { "name": "app", "image": "{{ must_env `SERVICE` }}/app:{{ must_env `TAG` }}", VCPU_NUM : prod | staging ͰҟͳΔCPUΛՄมʹ SERVICE : prod | staging σϓϩΠઌͷ؀ڥΛ੾Γସ͑Δ TAG : σϓϩΠ͝ͱʹมΘΔΠϝʔδͷλά ͻͱͭͷ JSON Λෳ਺؀ڥʹ࢖͍ճͤΔͷͰҡ࣋؅ཧָ͕
35. ίϯςφ΁ͷൿಗ৘ใͷઃఆ Πϝʔδʹൿಗ৘ใΛম͖ࠐΉͷ͸ආ͚͍ͨ ম͖ࠐΉͱ docker pull ͨ͠Βͦ͜ʹ࢒Δ ഁغ͍ͨ͠৔߹ɹΠϝʔδΛ͢΂ͯݸผʹഁغ͢Δ͔͠ͳ͍ → ؀ڥม਺Ͱ౉͢ λεΫఆٛͰ environment ઃఆ͸Ͱ͖Δ͕ → λεΫఆٛࣗମ͸ฏจͰอଘ͞ΕΔ ɹݹ͍৘ใͷഁغ΋Πϝʔδಉ༷ʹ໰୊ʹͳΔ
36. ͲͷΑ͏ʹ؀ڥม਺Λઃఆ͢Δ͔ λεΫىಈ࣌ʹ҆શͳͱ͜Ζ͔Βऔಘ ؀ڥม਺ʹઃఆ͔ͯ͠ΒϓϩηεΛىಈ͢Δ ҆શͳઃఆอଘ৔ॴ AWS Systems Manager (SSM) ύϥϝʔλετΞ
37. SSMύϥϝʔλετΞ
38. SSMύϥϝʔλετΞ • ֊૚Խͨ͠ path ໊Ͱ஋Λอଘ • ઃఆ஋ͷόʔδϣχϯάɺมߋऀͷཤ ྺ؅ཧ • KMSͰ҉߸Խͨ͠஋ΛอଘͰ͖Δ • KMS΁ͷݖݶ͕ͳ͚Ε͹஋͸ಡΊ ͳ͍ • ίϯςφىಈ࣌ʹ͔͜͜Β஋ΛऔΓ ग़ͯ͠؀ڥม਺ʹઃఆ͢Ε͹Α͍
39. ύϥϝʔλετΞ͔Β஋ΛऔΓग़͢ aws-ssm-env github.com/jamietsao/aws-ssm-env $ aws-ssm-env --paths=/prod/ API_KEY=xxxxxx DB_PASS=productionpass entry point ͷ sh Ͱ࣮ߦɺexport ͔ͯ͠Β exec ͢Δ #!/bin/sh export AWS_REGION=ap-northeast-1 export $(aws-ssm-env --paths=/prod/) exec /path/to/myapp
40. ύϥϝʔλετΞͷ஋Λ export & exec ssmwrap github.com/handlename/ssmwrap ฐࣾಉ྅࡞ $ ssmwrap -paths=/prod/ -- /path/to/myapp ssm ͔Βऔಘͨ͠஋Λ env ʹઃఆͯ͠ exec ·Ͱͯ͘͠ΕΔ ɾexec ·Ͱ͢ΔͷͰ Docker ͷ entrypoint ʹ௚઀ࢦఆͰ͖Δ ɾshell scirpt Λܦ༝͠ͳ͍ͷͰվߦΛؚΉ஋ͷ໰୊͕ͳ͍ ɾ-retries ΦϓγϣϯͰϦτϥΠճ਺ΛࢦఆͰ͖Δ
41. SSMύϥϝʔλετΞͷ໰୊ API Rate Limit ͕ݫ͠Ί େྔͷλεΫΛҰؾʹىಈ͢Δͱ Rate Limit Error (੍ݶ஋͸αϙʔτʹ໰͍߹Θ͕ͤͨඇެ։) ɾaws-ssm-env ͸ΤϥʔʹͳΔͱ panic Ͱࢮ͵ ɹσϓϩΠ࣌ʹࢮΜͩλεΫ͸࠶౓ىಈ͞ΕΔ͕… ɾssmwrap ͸ -retries ઃఆͰϦτϥΠͰ͖Δ(͓નΊ) ɾىಈ࣌ʹ random sleep ͰλΠϛϯάΛͣΒͯ͠؇࿨ ! ਖ਼௚ECSଆͰ؀ڥม਺ʹઃఆͯ͠ىಈͯ͠΄͍͠
42. σϓϩΠख๏ͱൿಗ৘ใͷ؅ཧ ·ͱΊ • Blue-Green ͷར఺Λੜ͔ͤΔΑ͏ʹ࢖͏ • SSMύϥϝʔλετΞ͸ศར • Ͱ΋΋ͬͱڧ͘ͳͬͯ΄͍͠ • ֤ࣗ޻෉ͯ͠΍͍͖ͬͯ·͠ΐ͏
43. ϩάͷू໿ͱετϦʔϛϯάॲཧ
44. ϩάू໿ͷੲͱࠓ ੲ (ʙ2011 Before Fluentd) ɾϑΝΠϧΛఆظతʹճऩ ɾλΠϜϥάେ
45. ϩάू໿ͷੲͱࠓ ۙ୅ (2012ʙ After Fluentd) ϑΝΠϧΛ fluentd ͕ஞ࣍ಡΈऔΓ ૹ৴ ΞϓϦέʔγϣϯ͕ fluentd ΁௚઀ૹ৴ λΠϜϥάখ
46. ECS ͔Βͷϩάू໿ ൃੜ͢Δϩά͸େ·͔ʹ3छྨ 1.ΞΫηεϩά 2.ΞϓϦέʔγϣϯ/ϛυϧ΢ΣΞ͕ ɹඪ४ग़ྗɺඪ४Τϥʔग़ྗʹు͖ग़͢ϩά 3.ΞϓϦέʔγϣϯతʹҙຯ͕͋Δߦಈϩά ΞΫηεϩά͸ nginx ͔Β /dev/stdout ʹग़ͤ͹2छྨ ͲͷΑ͏ʹूΊΔ͔
47. ֤ίϯςφ͕ STDOUT, STDERR ʹు͖ग़ͨ͠΋ͷ Docker logging driver ʹΑͬͯѻΘΕΔ λεΫఆٛͰίϯςφ͝ͱʹࢦఆͰ͖Δ awslogs : CloudWatch Logs ΁ૹ৴ fluend : Fluentd ΁ૹ৴ json-file : ϑΝΠϧ΁อଘ syslog : syslogd ΁ૹ৴ (ଞʹ΋͋Δ)
48. awslogs or fluentd? CloudWatch Logs ɹpros: ϚωʔδυͳͷͰԿ΋ߟ͑ͳͯ͘Α͍ ɹcons: ͔ͦ͜Βߋʹผʹྲྀ͢ͷ͕ଟগ໘౗ (Lambda ͕ඞཁ) Fluentd: ɹpros: ߦಈϩάଞͱಉ༷ͷѻ͍͕Ͱ͖Δ ɹcons: fluentdΛࣗ෼ͰཱͯΔඞཁ͕͋Δ
49. Fluentd ʹ౷Ұ͢Δ ߦಈϩά͸ͲͪΒʹ͠Ζ Fluentd Ͱѻ͍͍ͨͷͰඞཁ ߦಈϩά ɾλεΫʹΞϓϦέʔγϣϯͱҰॹʹ fluentd Λ഑ஔ (sidecar) ɾΞϓϦ͔Β localhost:24224 ʹૹ৴͢Δ(ߏ଄Խϩά) STDOUT, STDERR ɾECSαʔϏεͱͯ͠ NLB Λհͯ͠ fluentd Λ഑ஔ ɾ֤λεΫ͸ Docker logging driver Ͱ NLB:24224 ΁ૹ৴͢Δ
51. ECS Ͱ fluentd ΛͲ͏࢖͏͔ AWS ͷ৔߹ fluentd + plugin S3 Ͱ S3 ΁อ؅͢Δͷ͕ఆ൪ S3 ΁ͷॻ͖ग़͠͸ 1ʙ5 ෼ఔ౓ʹ͢Δ ɹ͋·Γ୹͍ͱ S3 ͷΦϒδΣΫτ͕ࡉ੾ΕʹͳΔ) ো֐࣌ͷ͜ͱΛߟ͑Δͱ਺෼ఔ౓Ͱ΋ίϯςφ಺ͷόοϑΝ (ϝϞϦ or ϩʔΧϧϑΝΠϧ)ʹอଘ͢Δͷ͸ফࣦϦεΫ → ৴པͷஔ͚ΔόοϑΝ͕΄͍͠
52. Kinesis Data Streams ౤͛ࠐΜͩϝοηʔδ(ϩάʹݶΒͳ͍)Λ24࣌ؒɺॱংΛอͬͯอ؅ͯ͘͠ΕΔετϦʔϛϯάॲཧج൫ͷϚωʔδυαʔϏε OSS Ͱ͸ Apache Kafka ͕ྨࣅ ϩάΛ S3 ʹॻ͖ग़͢લͷʮ৴པͰ͖ΔʯόοϑΝͱͯ͠࠷ద
53. Kinesis Data Firehose ετϦʔϛϯάσʔλΛఆظతʹ ɾS3 ɾRedshift ɾElasticsearch ʹॻ͖ग़͢ϚωʔδυαʔϏε ιʔεͱͯ͠ Kinesis Streams Λ࢖͑Δ
54. ࠷ऴతʹ͸͜͏
55. ʮtail -f ͍ͨ͠ΜͰ͕͢ʯ ϩά͕ϑΝΠϧʹॻ͔Εͳ͘ͳͬͨ → tail -f ͸Ͳ͏͢Δ? kinesis-tailf github.com/fujiwara/kinesis-tailf Go + aws-sdk-go Ͱࣗ࡞ $ kinesis-tailf -stream docker-logs Kinesis Streams Λ௥ඌ͠ඪ४ग़ྗʹు ͖ग़͢ CLI -start -end ΦϓγϣϯͰಛఆ࣌ؒଳΛ நग़Մೳ
56. Τϥʔϩά͚ͩநग़͍ͨ͠ શϩάΛ kinesis-tailf | grep ? ྲྀྔ͕ଟ͍ͱͭΒ͍ Streams ͷσʔλΛ Lambda Ͱॲཧ ύλʔϯʹϚονͨ͠ϩά͚ͩผͷ Streams ʹྲྀ͠௚͢ ɾαʔόΤϥʔ status:5 ɾॲཧʹ͕࣌ؒ3ඵҎ্ֻ͔ͬͨ΋ͷ ɹapptime:([3-9]|[1-9][0-9]+)\. ɾ(ERROR|WARN)
57. ϩά͔ΒͷΤϥʔ௨஌ S3 Event notification S3ʹΦϒδΣΫτ͕࡞੒/࡟আ͞ΕͨΒ LambdaΛݺͿ Τϥʔநग़ετϦʔϜ͔Β S3 ͷΦϒ δΣΫτ͕ੜ੒͞ΕͨΒ Slack ʹ௨஌ ΠϕϯτυϦϒϯͳόονॲཧʹ Firehose + S3 + Lambda ͕࢖͑Δ
58. ϩά͔ΒͷΤϥʔ௨஌ நग़͞Εͨλάɺߦ਺ͱͱ΋ʹsnippet౤ߘ Τϥʔൃੜ͔Β1෼Ͱ௨஌ େྔʹΤϥʔ͕ൃੜͯ͠΋1෼͝ͱʹ͔͠௨஌͸དྷͳ͍(ॏཁ)
59. ϩάͷू໿ͱετϦʔϛϯάॲཧ ·ͱΊ • ϩά͸ STDOUT/STDERR ʹग़ྗ Docker Logging Driver Ͱૹ৴ • Fluentd + ৴པͰ͖ΔόοϑΝͱͯ͠ͷ Kinesis Streams • Firehose + S3 + Lambda ͰΠϕϯτυϦϒϯͳσʔλॲཧ΋
60. GoݴޠʹΑΔ ӡ༻πʔϧ/ϛυϧ΢ΣΞ։ൃ
61. AWS / ECS ͸伱͕ؒଟ͍ ࠓճࣗલͰ։ൃͨ͠΋ͷ ecspresso github.com/kayac/ecspresso σϓϩΠπʔϧ ssmwrap github.com/handlename/ssmwrap SSM ύϥϝʔλετΞͷ஋Λ؀ڥม਺ʹઃఆͯ͠ exec kinesis-tailf github.com/fujiwara/kinesis-tailf Kinesis StreamsΛ௥ඌ͢Δ
62. 伱ؒՈ۩Λࣗ࡞͢Δ 伱ؒΛຒΊΔπʔϧɺϛυϧ΢ΣΞΛ։ൃͯ͠ӡ༻վળ খ͘͞ɺద౓ʹ൚༻తͳ΋ͷΛ࡞Δ ຊՈ͕伱ؒΛຒΊͨΒࣺͯΒΕΔΑ͏ʹ
63. 伱ؒՈ۩ͷྫ Rin github.com/fujiwara/Rin Redshift data Importer by SQS messaging. S3 Πϕϯτ௨஌Ͱ SQS ʹૹ৴ SQS ͷϝοηʔδ͔Β Redshiftʹ copy ͯ͠औΓࠐΈΛߦ͏πʔϧ 2015.05 ʹ։ൃ 2015.10 Firehose ൃද 2017.07 ౦ژϦʔδϣϯʹ Firehose
64. Before / After Fluentd ͔ΒϩάΛૹ৴ S3 / Redshift ʹॱ࣍औΓ͜·ΕΔ ͱ͍͏ߏ଄͸ͦͷ··Ϛωʔδυʹ
65. πʔϧ։ൃݴޠͱͯ͠ͷ Go γϯάϧόΠφϦʹͳΔ ϥϯλΠϜෆཁ ίϯςφͷϕʔεΠϝʔδΛબ͹ͳ͍ ॻ͖΍͢͞ΑΓಡΈ΍͢͞Λॏࢹ → ϝϯςφϯεੑ͕ߴ͍ aws-sdk-go ػೳɺΞοϓσʔτ͸ਃ͠෼ͳ͍ ࢖͍উख͸ਖ਼௚ඍົͳͱ͜Ζ͕…
66. ίϯςφ؀ڥ޲͚ͷ։ൃ Tips ίϚϯυϥΠϯΦϓγϣϯͷ஋Λ؀ڥม਺͔Β΋ಡΉ1 func main() { var port int var host string flag.IntVar(&port, "port", 8080, "port number") flag.StringVar(&host, "host", "localhost", "hostname") flag.VisitAll(func(f *flag.Flag) { if s := os.Getenv(strings.ToUpper(f.Name)); s != "" { f.Value.Set(s) } }) flag.Parse() fmt.Printf("%s:%d\n", host, port) } 1 https://mattn.kaoriya.net/software/lang/go/20170609110526.htm
67. ίϚϯυϥΠϯΦϓγϣϯͷ஋Λ؀ڥม਺͔Β΋ಡΉ $ myapp -host example.com -port 9999 example.com:9999 $ HOST=example.com PORT=9999 myapp example.com:9999 ίϯςφͰ͸ίϚϯυϥΠϯΑΓ ؀ڥม਺ͷ΄͏͕ઃఆ͠΍͍͢
68. ઃఆϑΝΠϧʹ؀ڥม਺͔Β஋ΛຒΊࠐΉ github.com/kayac/go-config Go ͷઃఆϑΝΠϧಡΈࠐΈ package # yaml account_id: "{{ must_env `MY_ACCOUNT_ID` }}" profile: "{{ env `PROFILE` `default` }}" must_env: ઃఆ͞Εͳ͍ঢ়ଶͰ࣮ߦ͞ΕΔͱ panic env: ઃఆ͞Ε͍ͯͳ͍৔߹͸σϑΥϧτ஋Λࢦఆ YAML / JSON / TOML ʹରԠ
69. ઃఆϑΝΠϧΛϦϞʔτ͔Βऔಘ͢Δ ઃఆ͕ϑΝΠϧʹͳ͍ͬͯΔͱ มߋͷࡍʹίϯςφͷ࠶Ϗϧυ & Ξοϓϩʔυ͕ඞཁ (ࢼߦࡨޡ͕໘౗) File / S3 / HTTP ͰऔಘͰ͖ΔΑ͏ʹ࡞͓ͬͯ͘ͱศར2 $ CONFIG=s3://my-config-bucket/config.yaml mydaemon 2 GoͰHTTPͱS3Λಁաతʹѻ͏ https://shogo82148.github.io/blog/2018/06/09/go-s3-protocol/
70. OSS ͱͯ͠࡞Δ OSS ʹͯ͠͠·͏ (Ծʹࣗ෼Β͔͠࢖Θͳͯ͘΋) README ͙Β͍͸ؤுͬͯॻ͘(ϞνϕʔγϣϯʹͳΔ) ա౓ͳࣾ಺ࣄ৘ͷࠞೖΛ๷͙ ʮͦͷϓϩδΣΫτҎ֎ʹҙຯ͕͋Δػೳͳͷ͔?ʯ ίϐϖͰຐվ଄൛͕૿৩͢ΔͷΛ๷͙ ʮGitHubͰόΠφϦఏڙͯ͠ΔͷͰͦΕΛ࢖͍ͬͯͩ͘͞ʯ
71. GoݴޠʹΑΔ ӡ༻πʔϧ/ϛυϧ΢ΣΞ։ൃ ·ͱΊ • AWS / ECS ͷ伱ؒΛখ͘͞ॻ͍ͨπʔϧͰຒΊΑ͏ • ͋͑ͯOSSʹ͢Δ͜ͱͰ͖Ε͍ʹอͭ • Go ͸ίϯςφ؀ڥͰ࢖͍উख͕Α͍
72. Questions? • Amazon ECS Ͱͷαʔόߏஙͱӡ༻ • σϓϩΠख๏ͱൿಗ৘ใͷ؅ཧ • ϩάͷू໿ͱετϦʔϛϯάॲཧ • GoݴޠʹΑΔӡ༻πʔϧ/ϛυϧ΢ΣΞ։ൃ