Next FRESH! Applications with Amazon ECS

2015/11/20 JAWS-UGコンテナ支部 #3 #jawsug_ct

Transcript

1. Next FRESH! Applications with Amazon ECS 2015/11/20 JAWS-UGίϯςφࢧ෦ #3 #jawsug_ct @stormcat24
2. Who? ‣ Akinori Yamada ( @stormcat24 ) ‣ http://stormcat.hatenablog.com/ ‣ CyberAgent, Inc ‣ Ameba౷ׅຊ෦ FRESHάϧʔϓ ‣ ServerSide / DevOps ‣ ޷͖ͳݴޠ͸Scala
3. Agenda ‣ AmebaFRESH! ‣ Architecture & ECS ‣ Blue Green Deployment ‣ Diet Docker Image ‣ Local Development ‣ ecs-formation ‣ Others
4. AmebaFRESH!
5. ‣ ੜ์ૹಈը഑৴ϓϥοτϑΥʔϜ ‣ جຊແྉ ‣ PCϒϥ΢β / iOS / Android ‣ εϚϗφΠζυ͞ΕͨUI ‣ ߴ඼࣭ͳಈը഑৴ ‣ 2015೥12݄Ұൠެ։༧ఆ ‣ ݱࡏΫϩʔζυެ։த ‣ ※AbemaTVͱ͸ผͷαʔϏεͰ͢YO
10. ‣ 2015/04͔ΒϓϩδΣΫτ։࢝ ‣ ϓϩδΣΫτϝϯόʔ໿30໊ ‣ Serverside ☓ 6 ‣ Frontend ☓ 6 ‣ iOS ☓ 4 ‣ Android ☓ 4 ‣ Designer ☓ 3
11. ‣ Movie & Broadcasting ‣ RTMP Publishing ‣ HTTP Live Streaming (HLS)
12. ‣ Frontend ‣ Node.js v4 ‣ TypeScript1.6 ‣ React / Flux ‣ SPA + SSR ‣ iOS ‣ Swift2.0 ‣ Android ‣ Kotlin
13. ‣ Serverside & Infrastructure ‣ Amazon Web Services (AWS) ‣ Go1.5 ‣ Docker1.9.0 ‣ Microservices ‣ RESTful API (goji)
14. ‣ Tools ‣ Slack + Hubot (ChatOps) ‣ Github Enterprise ‣ CircleCI (Enterprise) ‣ Crashlytics ‣ JIRA Agile ‣ Mackerel ‣ ࡳଋ
15. Architecture & ECS
16. Architecture ‣ ࢥ૝ ‣ جຊํ਑ ‣ Task Definitionͷߏ੒ ‣ ClusterͱServiceͷѻ͍
17. ໨ࢦ͍ͯ͠Δ͜ͱ ‣ ۃྗϝϯςΛೖΕͳ͍ ‣ θϩμ΢ϯλΠϜϦϦʔε → Blue Green Deployment ‣ Πϯϑϥ͸࢖͍ࣺͯͯೖΕସ͑Δ → Immutable Infrastructure ‣ ૉૣ͘සൟͳϦϦʔε ‣ γεςϜ͸ݶΓͳ͘ૄ݁߹ͷํ͕ྑ͍ → Microservices ‣ ίϯςφͩ → Docker
18. Microservices ‣ ಛఆͷ։ൃݴޠʹґଘ͠ͳ͍ʢ๞͖Δ͠ʣ ‣ ௨৴ϓϩτίϧ ‣ RESTFul API (HTTP) ‣ ͦͷ͏ͪgRPC(HTTP2)͍ͨ͠ ‣ αʔϏεͷཻ౓ΛఆΊΔͷ͸΍͸Γ೉͍͠ ‣ γεςϜతͳυϝΠϯྖҬͰ෼͚Δͷ΋Ұͭͷࢦඪ ‣ ޙ͔ΒผαʔϏεͱͯ͠੾Γམͱ͢΋OK
19. Infrastructure ‣ AWS + EC2 Container Service(ECS)Λ࠾༻ ‣ ͪΐ͏Ͳٕज़ݕূதʹECSͷ౦ژϦʔδϣϯ͕དྷͨ ‣ ࠷௿ݶͷίϯςφߏ੒؅ཧͱεέδϡʔϦϯά͕Ͱ͖Ε͹े෼ ‣ ECSҎ֎ʹ΋࢖͍͍ͨ΋ͷ͕͋ͬͨ͠ ‣ Lambda ‣ RDS Aurora
20. جຊํ਑ ‣ MicroservicesຖͰίϯςφΛߏ੒͢ΔTask DefinitionΛͭ͘Δ ‣ 1ECS Clusterʹ͖ͭ1Service ‣ 1Clusterʹ͸1ͭͷAutoScaling Group ‣ ಈը഑৴αʔό͸ಛघͳͷͰྫ֎ ‣ Clusterؒ௨৴͸Internal ELBΛར༻
21. Task Definitionͷํ਑ ‣ ϩάͷసૹʹtd-agentΛ࢖͏ ‣ ֤ίϯςφϩά͸ϗετʹϚ΢ϯτ ‣ ϗετʹϚ΢ϯτ͞ΕͨϩάΛtd-agentͰసૹ ‣ logging driver͸·ͩಋೖͯ͠ͳ͍ ‣ Internal Service(API)Ͱ͋ͬͯ΋جຊNginxΛ௨͢ ‣ ΞΫηεϩάग़͢ͷָͩ͠
22. Task Definition(Service API)
23. Task Definition(WEB+API)
24. Task Definition(Job)
25. Task Definition(movie)
26. جຊతʹ͜ΕΒͷ૊Έ߹Θͤ
27. cluster = serviceͷσϝϦοτ ‣ Ϧιʔεޮ཰తʹ͸ϕετͰ͸ͳ͍ ‣ Cluster : Service = 1:N ʹൺ΂Δͱ ‣ Πϯελϯε਺͸૿͑Δ܏޲ʹ͋Δ ‣ ։ൃ؀ڥͰ͸t2.microΛ༗ޮ׆༻͍ͯ͠Δ ‣ nanoΠϯελϯεщ ƅшƅщ)ŜŹŖƃ
28. cluster = service ʹͯ͠Δཧ༝ ‣ ࢹ֮తʹΘ͔Γ΍͍͢ʢϏΪφʔϑϨϯυϦʔʣ ‣ Service୯ҐͰIAM roleΛઃఆͰ͖ͳ͍ ‣ IAM RoleͰݫີͳݖݶ੍ޚΛ͢Δʹ͸଍Γͳ͍ ‣ Service୯ҐͰͷSecurity GroupʹະରԠ ‣ ͨͩ͠ɺELBલఏʹͯ͠͠·͑͹ղܾͰ͖Δ
29. Blue Green Deployment
30. 2AutoScalingύλʔϯ ‣ BlueɺGreenܥ౷ͷClusterΛ࡞Δ ‣ ͦΕͧΕ͕AutoScalingGroupʹଐ͢Δ ‣ api1-blue, api1-green Έ͍ͨͳ ‣ AutoScalingGroup୯ҐͰELBΛ੾Γସ͑Δ
32. 2AutoScalingͷಛ௃ ‣ ͱͯ΋҆શ ‣ DeployޙͷϩʔϧόοΫ༰қ ‣ DeployલͷStandbyܥ౷ͷ΢ΥʔϜΞοϓඞཁ ‣ 10෼લʹ͸΍͓͖͍ͬͯͨ ‣ ίετͷέΞ͕ॏཁ ‣ DeployޙɺStandbyʹͳͬͨܥ౷Λམͱ͢౳
33. Diet Docker Image
34. Πϝʔδ͸খ͍͞΄Ͳྑ͍ ‣ docker build࣌ؒˣ ‣ CI࣌ؒˣ ‣ Registry͔ΒͷΠϝʔδμ΢ϯϩʔυ࣌ؒˣ ‣ AutoScaleͰ࡞੒͞ΕͨΠϯελϯε͕αʔϏεΠϯ͢Δ ·Ͱͷ࣌ؒ↓
35. docker hub ‣ hub.docker.com ‣ ਺ଟ͘ͷެࣜΠϝʔδ ‣ αΠζΛ࡟͗མͱͨ͠΋ͷ͹͔ΓͰ͸ͳ͍ ‣ 1GB௒͑ΔΠϝʔδ͸σϒ
36. ෆཁͳ΋ͷ͸࡟আ͢Δ ‣ ෆཁͳϑΝΠϧΛݟམͱ͞ͳ͍ ‣ ϏϧυͷͨΊʹੜͨ͡࢈ۀഇغ෺Λ࡟আ ‣ npm cache clear ‣ rm -rf ~/.grade ‣ apt-get clean ‣ Data VolumeΛ࢖͏ʢϙʔλϏϦςΟ͸མͪΔʣ
37. RUNͷճ਺ΛݮΒ͢ ‣ RUNͷ਺͚ͩΠϝʔδͷϨΠϠʔ͕ॏͳΓɺΠ ϝʔδ༰ྔ͸૿͑Δ ‣ && ͰνΣΠϯͯ͠ɺRUNͷճ਺ΛݮΒ͢ ‣ ௕͍docker buildͷ৔߹ɺ్தͰࣦഊ͢ΔͱRUN ͷ಄͔Β΍Γͳ͓͠ͳͷͰফ໣͸͢Δ
38. RUNͷճ਺ΛݮΒ͢ FROM ubuntu:15.10 RUN apt-get update RUN apt-get install -y curl RUN apt-get apt-get clean FROM ubuntu:15.10 RUN apt-get update && \ apt-get install -y curl && \ apt-get apt-get clean
39. ܰྔΠϝʔδΛ࢖͏ ‣ ࡟͗མͱ͞ΕͨܰྔΠϝʔδΛ࢖͏ ‣ ࠷ۙ͸ΦϑΟγϟϧͰslimΠϝʔδ͕͋Δ΋ͷ΋͋Δ ‣ e.g. Node, Go ‣ busybox௒ઈ͍ܰ
40. ‣ ࠷ۙ͸ܰྔΠϝʔδ(slim)΋༻ҙ͞ ΕͯΔ ‣ Docker HubΛීஈ͔Β८ճ͓ͯ͠ ͜͏
41. ‣ ٻΊΒΕΔϙʔλϏϦςΟˢ ‣ GoͰ͋Ε͹࣮ߦϑΝΠϧΛࡌͤΔ ͚ͩ ‣ busyboxͷதͰϏϧυ͸ΩπΠ ʢ೉қ౓ߴʣ
42. ݮྔʹΑΔࢥΘ͵ฐ֐ ‣ x509: failed to load system roots and no roots provided ‣ ίϯςφ͔ΒHTTPS௨৴͕Ͱ͖ͳ͘ͳΔ ‣ apt-get install -y ca-certificates Ͱղܾ ‣ ֎෦πʔϧ΁ͷґଘʢΞϓϦ಺͔ΒͷΩοΫʣ
43. ϕʔεΠϝʔδΛͭ͘Δ ‣ apt updateɺapt-get install ͏Μ͵ΜΛऴΘΒͤͨ ΋ͷ ‣ ຖ౓΍ͬͯͨΒCIͷ͕࣌ؒ૿͑Δʢdocker buildͷ ҆ఆʣ ‣ ΞϓϦଆͷDockerfileͰ͸ຊ࣭ͷॲཧʹઐ೦ͤ͞Δ
44. Local Development
45. ϩʔΧϧ։ൃͰ࢖͏΋ͷ ‣ docker-machine + VirtualBox ‣ docker-compose
46. docker-machine ‣ VirtualBox, AWS, Azure, Digital Ocean্ʹ DockerϗετΛߏங͢Δ ‣ ͔͋ͨ΋ϩʔΧϧ্ʹDocker؀ڥ͕͋Δ͔ͷΑ͏ ʹDockerͷૢ࡞͕Ͱ͖Δ
47. docker-machine ‣ docker-machine + VirtualBoxΛબ୒ ‣ Vagrant͸ࣺͯͨ ‣ Dockerʹൺ΂Δͱ࢖͍ࣺͯίετˢ ‣ ΞϓϦέʔγϣϯɺϛυϧ΢ΣΞؚΊ͍ͯͭͰ΋ϩʔΧϧͰ֬ ೝͰ͖ΔΑ͏ʹ ‣ VirtualBoxͷϙʔτϑΥϫʔυར༻
48. ϚγϯϦιʔεඞཁ ‣ ϩʔΧϧ΋ϑϧDockerͩͱɺٻΊΒΕΔϚγϯεϖο Ϋ͸ߴ͘ͳΔ ‣ 16GBͳ͍ͱ݁ߏਏ͍ ‣ αʔόαΠυΤϯδχΞʹ͸ඞਢ ‣ ίϯςφ͍ͬͺ্͍͛ͯɺշదʹTwitter͢Δͷॏཁ
49. ׳Ε͸ා͍
50. docker-compose ‣ Docker ToolboxͷҰ෦ʢݩʑfigʣ ‣ Dockerίϯςφ܈ͷߏ੒ΛYAMLͰ؅ཧ ‣ docker-compose up -d Ͱىಈ
51. σʔλετΞ΋Docker ‣ ϩʔΧϧͰ͸σʔλετΞ΋Dockerίϯςφར༻ ‣ library/mysql, library/redis ‣ VagrantΑΓ΋ߴ଎ͰؾܰʹεΫϥοϓϏϧυ
52. DBϚΠάϨʔγϣϯॏཁ ‣ ؀ڥ͸ἧͬͯ΋σʔλෆඋ͋Δͱҙຯ͕ແ͍ ‣ FRESH! Ͱ͸ goose ‣ https://bitbucket.org/liamstask/goose/ ‣ SQL͚ͩ͡Όͳ͘ɺGoͰϚΠάϨʔγϣϯ΋ॻ͚Δ
53. ecs-formation
54. ecs-formation ‣ https://github.com/stormcat24/ecs-formation ‣ docker-composeͷΑ͏ʹɺYAMLϑΝΠϧͰίϯςφ ͷߏ੒Λ؅ཧ͢Δ ‣ ౰࣌͸ecs-cliͱ͔ແ͔ͬͨͷͰ࡞ͬͨ ‣ aws-sdk-goར༻
55. ecs-formation features ‣ Task Definitionsͷߋ৽ ‣ Clusterʹ഑ஔ͢ΔServiceͷߋ৽ ‣ Blue-Green Deploymentͷ࣮ߦ
56. Task Definitions (task/jawsug-api.yml) nginx: image: registry.jawsug.local:5000/jawsug/nginx:latest ports: - 80:80 environment: SERVER_NAME: jawsug.example.com volumes: - /var/log/container/nginx:/var/log/nginx links: - api memory: 512 cpu_units: 512 essential: true
57. Services on cluster (service/jawsug-cluster.yml) api-service: task_definition: jawsug-api desired_count: 1
58. Blue Green Deployment (bluegreen/jawsug-cluster.yml) blue: cluster: jawsug-cluster-blue service: api-service autoscaling_group: jawsug-cluster-blue green: cluster: jawsug-cluster-green service: api-service autoscaling_group: jawsug-cluster-green primary_elb: jawsug-api-primary standby_elb: jawsug-api-standby
59. ecs-formationͷӡ༻ ‣ ecs-formationઐ༻ͷϦϙδτϦΛͭ͘Δ ‣ Task, Service, BlueGreenͷఆٛͷYAML ‣ masterʹϚʔδ͞ΕΔͱTask Definitionߋ৽ ‣ service update͸Hubot -> CircleCI͔Β
60. Update Task Definitions push merge PR webhook test update task notification webhook ecs-formation repository Amazon ECS ※࠷৽ͷTask Definitionͷόʔδϣϯʹߋ৽͞ΕΔ
61. Update Service(Deploy) webhook test update service deploy webhook ecs-formation repository Amazon ECS ※࠷৽ͷTask DefinitionͷόʔδϣϯΛར༻͠ɺServiceΛߋ৽͢Δ push deploy branch notification
62. Others
63. Others ‣ AMI ‣ Private Registry ‣ CircleCI + Docker ‣ Terraform ‣ Mackerel
64. EC2-Optimized AMI ‣ Current version 2015.09.b ‣ Amazon Linuxϕʔε ‣ Docker + ECS Agent ‣ ศར͚ͩͲࣾ಺Ͱ໘౗ݟͯ͘Εͳͦ͞͏ͩͬͨ
65. Ubuntu ‣ FRESH!͸UbuntuΛ࠾༻ ‣ DockerͷΠϯετʔϧ → ηϧϑαʔϏε ‣ ECS AgentͷಋೖͱαʔϏεԽʢUpstartʣ → ηϧϑαʔϏε ‣ ੵۃతͳDockerͷΞοϓσʔτ → ਓப ‣ apt-get install docker-engine=1.9.0-0~trusty
66. Private Registry ‣ S3ΛόοΫΤϯυʹɺPrivate RegistryΛ࡞ΕΔ ‣ registry:2.2.0 ‣ konradkleine/docker-registry-frontend:v2
68. CircleCI + Docker ‣ ΞϓϦέʔγϣϯ͸1ϦϙδτϦʹ1Dockerfile ‣ nginx΍td-agentͷΑ͏ͳϛυϧ΢ΣΞܥ͸ผ్ઐ ༻ϦϙδτϦ ‣ CircleCIͰmasterϏϧυ࣌ʹdocker build + push
69. Build Docker Image push merge PR webhook test docker build private registry docker push notification webhook
70. Terraform ‣ ΠϯϑϥߏஙͷͨΊͷΦʔέετϨʔγϣϯπʔ ϧ ‣ ଟ࠼ͳProviderΛఏڙ ‣ AWSͰͷΠϯϑϥߏஙʹར༻
71. Terraform؅ཧͯ͠Δ΋ͷ ‣ EC2 ‣ Security Group ‣ Route53 (ΠϯλʔφϧυϝΠϯͷΈʣ ‣ ECS Cluster ‣ AutoScaling Groupͷىಈߏ੒
72. Terraform؅ཧͷߟ͑ํ ‣ ӡ༻ʹΑͬͯঢ়ଶ͕มΘΔ΋ͷ؅ཧʹ͸޲͔ͳ͍ ‣ ELB ‣ AutoScaling Group ‣ εΫϥοϓʴϏϧυʹ͕͔͔࣌ؒΔ΋ͷ ‣ RDS ‣ ElastiCache ‣ EC2͸CloudinitͰߏ੒͢Δ ‣ Provisioning͸͠ͳ͍ɻ࢖͍ࣺͯ
73. ΫϦςΟΧϧͳ΋ͷ͸ආ͚Δ ‣ Route53 ‣ Ͳ͔ͬͷϓϩδΣΫτͰɺϨϏϡʔ͕ܗ֚ԽˠηϧϑϚ ˠେࣄނ͕͋ͬͨΒ͍͠ ‣ roadworkerઐ༻ϦϙδτϦ༻ҙ͠ɺݫॏʹΫϩεϨ Ϗϡʔ͢Δ ‣ IAM
74. Terraformͷӡ༻ ‣ tfϑΝΠϧ͸GHE্Ͱ؅ཧ ‣ tfstate͸S3্ʹอ࣋ ‣ શͯΛҰͭͷtfstateͰ؅ཧ͠ͳ͍ ‣ dev/shared/staging/production/load ͘Β͍ ‣ CircleCI ‣ PRͰ terraform plan ͷࠩ෼νΣοΫ ‣ planͰ͸ݫີͳνΣοΫ͸Ͱ͖ͳ͍ͷͰͦΕͳΓʹ৺؟ඞཁ ‣ hubot -> CircleCIͰ terraform apply
75. Mackerel ‣ ؂ࢹ͸جຊతʹMackerel ‣ ݟ΍͍͢ɺ͖Ε͍ ‣ ࠷ۙDockerͷϝτϦΫε͕औΕΔΑ͏ʹͳͬͨ
76. Mackerel
77. Mackerel
78. Mackerel ˡίϯςφ୯Ґͷ
    ϝτϦΫε
79. ࠷ޙʹॴײ ‣ ECS͸΋ͪΖΜपลπʔϧ΋ἧ͖ͬͯͯɺDocker ຊ൪ӡ༻ͷෑډ͕େ͖͘Լ͕͍ͬͯΔ ‣ ೰ΜͰΔͳΒͱΓ͋͑ͣࢼͯ͠ΈΑʁ
80. Thank you for listening