ADC2016: Axion meets HashiCorp

社内イベント向けの資料

Transcript

1. Axion meets HashiCorp @atty303 / AdTech Core Development Group © 2016 CyberAgent, Inc. 1
2. ࣗݾ঺հ Ѩ઒ ߞ࢘ 4 2016-02 ʙ ج൫։ൃάϧʔϓ © 2016 CyberAgent, Inc. 2
3. Agenda 4 Consul : Service Discovery 4 Nomad : Container Scheduling 4 Terraform : Infrastructure as a Code © 2016 CyberAgent, Inc. 3
4. Axion Overview © 2016 CyberAgent, Inc. 4
5. What's Axion ? 4 ࣾ಺޲͚(ϚΠΫϩ)αʔϏε 4 σʔλϕʔεͷΑ͏ͳ΋ͷ 4 ൺֱతߴ͍ࢀরස౓ (10k ops/s ~) 4 εϥΠυ಺ͷ IP ΍ݻ༗໊ࢺͳͲ͸μϛʔͰ͢ © 2016 CyberAgent, Inc. 5
6. Axion ։ൃʹ͓͚Δ ٕज़త௅ઓ © 2016 CyberAgent, Inc. 6
7. Docker ͱ ޲͖߹͏! (AWS/GCP ʹཔΒͣ) © 2016 CyberAgent, Inc. 7
8. Docker - Technology Radar 4 Technology Radar 2016-04 1 Ͱ ADOPT(࠾ ༻ͤΑ) 4 ADOPT = ͜ͷτϨϯυ ʹࠓ͙͢৐Δ΂͖ 1 https://www.thoughtworks.com/radar/tools/ docker © 2016 CyberAgent, Inc. 8
9. Docker ͱ޲͖߹͏ͨΊʹ… Nomad ΫϥελʹΞϓϦΛσϓϩΠͯ͠ɺ σϓϩΠͨ͠ΞϓϦͷΤϯυϙΠϯτ؅ཧʹ Consul ͕ඞཁͰɺ͜ΕΒͷ؀ڥߏஙΛ Terraform ʹ΍ΒͤΑ͏ɻ © 2016 CyberAgent, Inc. 9
10. Consul Service Discovery © 2016 CyberAgent, Inc. 10
11. What's Consul ? 4 Service ͷ؅ཧͱݕࡧ 4 Key-Value Storage 4 ߴ͍Մ༻ੑΛඋ͑ͨ෼ࢄγεςϜ © 2016 CyberAgent, Inc. 11
12. Technology Radar 4 Technology Radar 2016-04 2 Ͱ ADOPT(࠾ ༻ͤΑ) 4 ADOPT = ͜ͷτϨϯυ ʹࠓ͙͢৐Δ΂͖ 2 https://www.thoughtworks.com/radar/tools/ consul © 2016 CyberAgent, Inc. 12
13. Service Discovery ωοτϫʔΫΛ௨ͯ͠ػೳΛఏڙ͢Δ Service Λ ൃݟ͢Δػߏ 4 Service ͷྫ: api, mysql, kafka 4 Service ͷϝϯόʔ͸ IP:PORT ͷϦετ 4 ϔϧενΣοΫͰϝϯόʔͷࢮ׆؂ࢹ 4 HTTP ͱ DNS Ͱ໰͍߹Θ͕ͤͰ͖Δ © 2016 CyberAgent, Inc. 13
14. DNS Ͱ A Ϩίʔυ໰͍߹Θͤ $ dig @127.0.0.1 -p 8600 kafka.service.consul ;; QUESTION SECTION: ;kafka.service.consul. IN A ;; ANSWER SECTION: kafka.service.consul. 0 IN A 192.168.0.21 kafka.service.consul. 0 IN A 192.168.0.20 kafka.service.consul. 0 IN A 192.168.0.19 © 2016 CyberAgent, Inc. 14
15. DNS Ͱ A Ϩίʔυ໰͍߹Θͤ 4 Consul agent Λ DNS αʔόʹࢦఆͯ͠໰͍߹ Θͤ 4 *.service.consul Λղܾ͢ΔͱαʔϏεͷϝ ϯόʔ͕ฦͬͯ͘Δ 4 Ϧετͷฒͼ͸ϥϯμϜ (DNS ϥ΢ϯυϩϏϯ) 4 ϔϧενΣοΫΛ௨ͬͨੜ͖͍ͯΔϝϯόʔͩ ؚ͚͕·ΕΔ © 2016 CyberAgent, Inc. 15
16. DNS Ͱ SRV Ϩίʔυ໰͍߹Θͤ $ dig @127.0.0.1 -p 8600 axion-api-admin.service.consul SRV ;; QUESTION SECTION: ;axion-api-admin.service.consul. IN SRV ;; ANSWER SECTION: axion-api-admin.service.consul. 0 IN SRV 1 1 42319 acd-stg-axion-dock-y002.node.openstack-stg.consul. axion-api-admin.service.consul. 0 IN SRV 1 1 38475 acd-stg-axion-dock-y001.node.openstack-stg.consul. ;; ADDITIONAL SECTION: acd-stg-axion-dock-y002.node.openstack-stg.consul. 0 IN A 192.168.0.26 acd-stg-axion-dock-y001.node.openstack-stg.consul. 0 IN A 192.168.0.27 © 2016 CyberAgent, Inc. 16
17. DNS Ͱ SRV Ϩίʔυ໰͍߹Θͤ 4 SRV ϨίʔυͰ͋Ε͹ϙʔτ·Ͱ෼͔Δ 4 ϙʔτΛಈతׂΓ౰ͯ͢Δͱ͖ʹ༗༻ 4 ͨͩ͠ར༻ଆͰ SRV ͷϨκϧό͕ඞཁ 4 طʹ SRV ΛಡΉ࣮૷͕ͳ͚Ε͹ HTTP API ΛಡΉ࣮૷Λॻ͍ͨ΄͏ָ͕͔ͱ © 2016 CyberAgent, Inc. 17
18. HTTP Ͱͷ໰͍߹Θͤ $ curl -s http://localhost:8500/v1/catalog/service/kafka | jq . [ { "Node": "acd-stg-axion-kafka-y001.local", "Address": "192.168.0.19", "ServiceID": "kafka", "ServiceName": "kafka", "ServiceTags": [], "ServiceAddress": "", "ServicePort": 9092, "ServiceEnableTagOverride": false, "CreateIndex": 55508, "ModifyIndex": 151479 }, ... ] © 2016 CyberAgent, Inc. 18
19. HTTP Ͱͷ໰͍߹Θͤ 4 ී௨ʹ JSON Ͱฦͬͯ͘Δ 4 DNS ͱҧͬͯࢮΜͰΔϝϯόʔΛ஌Δ͜ͱ͕ Ͱ͖Δ © 2016 CyberAgent, Inc. 19
20. HTTP Blocking Query 4 HTTP API Ͱఏڙ͞Ε͍ͯΔσʔλͷมߋΛ؂ ࢹͰ͖Δ 4 DNS ΍ HTTP ΛϙʔϦϯάͤͣʹଈ࠲ʹϝϯ όʔมߋΛݕ஌Ͱ͖Δ 4 ͍ΘΏΔ Server Push (HTTP Long Polling) 4 Non-Blocking ͳ HTTP Client Ͱ࢖͏͜ͱ © 2016 CyberAgent, Inc. 20
21. αʔϏεͷొ࿥ 4 ઃఆϑΝΠϧͰ੩తʹొ࿥ 4 ϗετʹΠϯετʔϧ͞Ε͍ͯΔϛυϧ΢ ΣΞͳͲ 4 HTTP API Ͱಈతʹొ࿥ 4 ಈతʹσϓϩΠ͢ΔΞϓϦέʔγϣϯͳͲ 4 Docker registrator ΍ Nomad ͱ౷߹ © 2016 CyberAgent, Inc. 21
22. ྫ͑͹ Kafka Λొ࿥͢Δ { "service": { "name": "kafka-broker", "port": 9092, "checks": [ { "id": "kakfa-broker", "tcp": "localhost:9092", "interval": "5s", "timeout": "1s" } ] } } 4 Kafka ͕ಈ͍͍ͯΔϗετͰ Consul ͷઃఆϑΝΠϧʹهड़ 4 tcp/9200 ΛϔϧενΣοΫ © 2016 CyberAgent, Inc. 22
23. Docker ͱͷ࿈ܞ (registrator 3) docker run --name=registrator -d --net=host --restart=unless-stopped --log-driver=journald \ -v /var/run/docker.sock:/tmp/docker.sock \ gliderlabs/registrator:v6 \ -tags registrator -resync 3600 \ -ip $(curl -s http://169.254.169.254/2009-04-04/meta-data/local-ipv4) \ consul://localhost:8500 4 Docker ίϯςφΛࣗಈతʹ Consul ΁ Service ͱͯ͠ొ࿥ͯ͘͠ΕΔπʔϧ 3 https://github.com/gliderlabs/registrator © 2016 CyberAgent, Inc. 23
24. Docker registrator docker run --name=cadvisor -d \ --restart=unless-stopped --log-driver=journald \ --dns=192.168.168.1 --privileged=true \ -p 9101:8080 \ -e "SERVICE_NAME=cadvisor" \ -e "SERVICE_CHECK_HTTP=/" \ -e "SERVICE_CHECK_INTERVAL=30s" \ google/cadvisor:v0.22.0 4 ؀ڥม਺ SERVICE_* ʹ Service ͷఆٛΛॻ͍ͯίϯςφΛىಈ 4 registrator ͕ίϯςφىಈΛݕग़ͯ͠ Consul ΁ొ࿥ 4 ίϯςφ͕ࢭ·ͬͨͱ͖͸ Consul ͔Β࡟আ © 2016 CyberAgent, Inc. 24
25. Key-Value Storage $ curl -X PUT -d 'test' http://localhost:8500/v1/kv/web/key1 true $ curl http://localhost:8500/v1/kv/web/key1 [ { "CreateIndex":97, "ModifyIndex":97, "Key":"web/key1", "Flags":0, "Value":"dGVzdA==" } ] © 2016 CyberAgent, Inc. 25
26. Key-Value Storage 4 ֊૚Խ͞ΕͨΩʔͷ GET/PUT/DELETE 4 Blocking Query Ͱ watch Ͱ͖Δ͠ɺ CAS ΋ ͋Δ 4 εέʔϥϏϦςΟ͸ແ͍ͷͰ஫ҙ 4 ϗετ΍ΞϓϦͷઃఆͳͲࢀরස౓ͷ௿͍ σʔλͷΈ֨ೲ © 2016 CyberAgent, Inc. 26
27. consul-template 4 Service ΍ KVS ͷมߋΛݕ஌ͯ͠ςϯϓϨʔ τϑΝΠϧΛߋ৽ɺίϚϯυΛ࣮ߦͯ͘͠ΕΔ πʔϧ 4 Consul ʹରԠ͍ͯ͠ͳ͍πʔϧ/ϛυϧ΢ΣΞ ͱ Consul ͷΪϟοϓΛຒΊΔ © 2016 CyberAgent, Inc. 27
28. consul-template ͱ HAProxy ͷྫ global daemon maxconn {{key "service/haproxy/maxconn"}} listen mysql-slave bind *:3306{{range service "mysql-slave"}} server {{.Node}} {{.Address}}:{{.Port}}{{end}} 4 haproxy.conf ͷςϯϓϨʔτΛ४උ $ consul-template -consul localhost:8500 -template \ haproxy.ctmpl:/etc/haproxy/haproxy.conf:systemctl reload haproxy 4 consul-template Λىಈ͓ͯ͘͠ © 2016 CyberAgent, Inc. 28
29. consul-template ͱ HAProxy ͷྫ global daemon maxconn 128 listen mysql-slave bind *:3306 server dbslave01 10.0.0.1:3306 server dbslave02 10.0.0.2:3306 4 Service ͷมߋΛݕ஌ͯ͠ HAProxy ͷઃఆϑΝΠ ϧΛߋ৽ͯ͠Ϧϩʔυͯ͘͠ΕΔ © 2016 CyberAgent, Inc. 29
30. Consul ͷΞʔΩςΫνϟ 4 Server(Master) ઐ༻ϗετΛ 3 or 5 ୆ 4 ଞͷશͯͷϗετͰ Agent(Client) Λ࣮ߦ © 2016 CyberAgent, Inc. 30
31. Axion ͱ Consul © 2016 CyberAgent, Inc. 31
32. Consul ͱ dnsmasq # /etc/resolv.conf nameserver 127.0.0.1 # /etc/dnsmasq.conf server=/consul/127.0.0.1#8600 4 શͯͷϗετͰ Consul ͱ dnsmasq Λ࣮ߦ 4 Consul ͷଘࡏΛҙࣝ͠ͳͯ͘΋ *.consul Λ ໊લղܾͰ͖Δ © 2016 CyberAgent, Inc. 32
33. Service 4 શͯͷωοτϫʔΫαʔϏεΛ Consul ʹొ࿥ © 2016 CyberAgent, Inc. 33
34. ϛυϧ΢ΣΞ΁ͷ઀ଓ 4 ΞϓϦ͔Βͷϛυϧ΢ΣΞ΁ͷ઀ଓͰ Consul ͷ DNS Λࢀর // Aerospike new com.aerospike.client.async.AsyncClient( policy, "aerospike.service.consul", 3000) // Kafka Consumer kafka.consumer.bootstrap.servers = "kafka.service.consul:9092" // ࣮ࡍ͸ Configurable Ͱ͢ © 2016 CyberAgent, Inc. 34
35. ΫϥελϦϯάͱ Consul 4 Kafka, Aerospike, Cassandra, Redis Cluster ͷ Α͏ͳΫϥελΛ૊Ήϛυϧ΢ΣΞͰ઀ଓઌͱͯ͠ ࢦఆ͢Δͷ͸ Ϋϥελʔ΁ͷίϯλΫτϙΠϯτ 4 ΫϥΠΞϯτ͸ੜ͖͍ͯΔϊʔυʹ઀ଓͰ͖Ε͹ɺ ͔ͦ͜ΒΫϥελʔϝϯόʔΛऔಘ͢Δ 4 Consul ͷ DNS ͸ੜ͖͍ͯΔϝϯόʔ͚ͩฦ͢ͷ ͰɺՄ༻ੑͷͨΊʹෳ਺ϗετΛྻڍ͢Δඞཁ͕ແ ͘ͳΔ © 2016 CyberAgent, Inc. 35
36. axion-client 4 Axion ͷ API αʔόʹ઀ଓ͢ΔΫϥΠΞϯτϥΠϒϥϦ 4 τϥϯεϙʔτʹ Scala ͷ RPC ϥΠϒϥϦͰ͋Δ finagle Λ ར༻ 4 finagle ͕ΫϥΠΞϯταΠυͷϩʔυόϥϯαʔΛ͍࣋ͬͯ ΔͨΊɺͦͷϝϯόʔΛ Consul ͷαʔϏεͱಉظ͢Δίʔυ Λ࣮૷ͯ͠ར༻ 4 Finagle Resolver ͷॻ͖ํ͸ finagle-consul 4 Λࢀߟʹ Blocking Query Λར༻࣮ͯ͠૷ 4 https://github.com/kachayev/finagle-consul © 2016 CyberAgent, Inc. 36
37. axcsync 4 Consul ͷ Service ͱ BIG-IP ͷ Pool Member Λಉظ 4 consul-template + axc 5 + python εΫϦϓ τͰ࣮૷ 5 https://adtech.cyberagent.io/techblog/archives/59 © 2016 CyberAgent, Inc. 37
38. Prometheus 4 Pull ܕͷ؂ࢹγεςϜ Prometheus Λ࢖͍ͬͯΔ 4 ؂ࢹର৅Λऔಘ͢Δͷʹ Consul Service Λར༻ (૬ੑൈ܈ !) scrape_configs: - job_name: 'consul' consul_sd_configs: - server: '192.168.168.1:8500' services: - prometheus-pushgateway - prometheus-node-exporter - prometheus-consul-exporter - prometheus-burrow - cadvisor - axion-api-admin © 2016 CyberAgent, Inc. 38
39. Consul Λ࠾༻ͨ͠ཧ༝ Docker ίϯςφͷಈత഑ஔ͢Δ৔߹ɺαʔϏε ͷ഑ஔΛܾΊଧͪͰ͖ͳ͍ͷͰ Service Discovery ػߏ͕ඞਢʹͳΔ 4 ZooKeeper ʹ͸ͳ͍ DNS ͱ HTTP ʹΑΔ؆ ୯ͳΞΫηε 4 etcd ʹ͸ͳ͍ Service ͷϔϧενΣοΫ 4 HashiCorp ͱ͍͏ϒϥϯυ © 2016 CyberAgent, Inc. 39
40. Consul ·ͱΊ 4 Service Discovery ͕͋Δ͜ͱΛલఏʹγες ϜΛ૊Ή͜ͱͰίϯϙʔωϯτͷґଘؔ܎Λ៉ ྷʹදݱͰ͖Δ 4 αʔϏεͷಈతͳมԽ΁ͷରԠ͕ඇৗʹߦ͍΍ ͍͢ 4 Docker ʹؔ܎ͳ͘༗༻ © 2016 CyberAgent, Inc. 40
41. Consul ·ͱΊ 4 Consul ͸ϋϚΓͲ͜Ζ͕ͳ͘ૉ௚ʹ࢖͑ͨ ! 4 ඞཁ࠷খݶͷγϯϓϧͳػೳηοτͰ͋Γͳ͕ ΒԠ༻ੑ͕ߴ͍ 4 ಋೖͷϦεΫ͕ແ͍Α͏ʹࢥ͑Δ 4 Kubernetes ͳͲ SD Λ͍࣋ͬͯΔ΋ͷΛ࢖͏ ৔߹͸ඞཁͳ͍͔΋ © 2016 CyberAgent, Inc. 41
42. Nomad Container Scheduling © 2016 CyberAgent, Inc. 42
43. What's Nomad ? 4 Docker Ϋϥελͷ؅ཧπʔϧ 4 AWS ECS, Kubernetes, Docker Swarm, Mesos ͱಉ͡໾ׂ 4 Ϋϥελʹίϯςφ(ΞϓϦ)Λεέδϡʔϧ (σϓϩΠ)͢Δ 4 Consul ͱͷΠϯςάϨʔγϣϯ © 2016 CyberAgent, Inc. 43
44. Technology Radar 4 Technology Radar 2016-04 6 Ͱ ASSESS(ௐࠪͤΑ) 4 ASSESS = ࠓ͙͢࠾༻͢΂͖ͱ ͸ݴΘͳ͍͕ɺ஫໨͓ͯ͘͠΂ ͖ 4 ͪͳΈʹ Kubernetes ͸ TRIAL(ࢼߦͤΑ) 4 TRIAL = ϦεΫͷ௿͍ϓϩδΣ ΫτͰ࠾༻ͯ͠ΈΔ͜ͱΛਪ঑ ͢Δ 6 https://assets.thoughtworks.com/assets/ technology-radar-apr-2016-en.pdf © 2016 CyberAgent, Inc. 44
45. Nomad ͷΞʔΩςΫνϟ 4 Server ઐ༻ϗετΛ 3 or 5 ୆ 4 Docker Ϋϥελʹ͢ΔϗετͰ Client Λ࣮ߦ © 2016 CyberAgent, Inc. 45
46. Job ఆٛ job "axion-api" { type = "service" # or "batch" or "system" group "api" { count = 2 task "api" { driver = "docker" config { image = "internal-repository/axion-api:$TAG" port_map { thrift = 8000 } } } } } © 2016 CyberAgent, Inc. 46
47. ϦιʔεׂΓ౰ͯ task "api" { resources { cpu = 8000 # MHz memory = 4096 # MB disk = 20000 # MB network { mbits = 100 # MBits port "thrift" { static = 8888 } } } } © 2016 CyberAgent, Inc. 47
48. Consul Service ొ࿥ task "api" { service { name = "axion-api-thrift" port = "thrift" check { type = "tcp" interval = "2s" timeout = "1s" } } } © 2016 CyberAgent, Inc. 48
49. Job ͷ࣮ߦ $ nomad run axion-api.nomad ==> Monitoring evaluation "50f80881" Evaluation triggered by job "axion-api" Allocation "73526b78" created: node "0cc2a12f", group "api" Evaluation status changed: "pending" -> "complete" ==> Evaluation "50f80881" finished with status "complete" © 2016 CyberAgent, Inc. 49
50. Job ͷঢ়ଶ $ nomad status axion-api ID = axion-api Name = axion-api Type = service Priority = 50 Datacenters = openstack-stg Status = running Periodic = false ==> Evaluations ID Priority Triggered By Status 3bc903ef 50 rolling-update complete 3b772796 50 job-register complete ==> Allocations ID Eval ID Node ID Task Group Desired Status 45f2e143 3bc903ef 7a6b1272 api run running 546a9c55 3b772796 8959a37c api run running © 2016 CyberAgent, Inc. 50
51. ഑ஔ(Alloc)ͷঢ়ଶ $ nomad alloc-status 45f2e143 ID = 45f2e143 Eval ID = 3bc903ef Name = axion-api.api[1] Node ID = 7a6b1272 Job ID = axion-api Client Status = running ==> Task Resources Task: "api" CPU Memory MB Disk MB IOPS Addresses 8000 4096 20000 0 admin: 10.4.85.27:40660 thrift: 10.4.85.27:9999 ==> Task "api" is "running" Recent Events: Time Type Description 07/07/16 16:38:49 JST Started Task started by client 07/07/16 16:38:45 JST Received Task received by client © 2016 CyberAgent, Inc. 51
52. ϩάͷ֬ೝ $ nomad fs ls 45f2e143 alloc/logs Mode Size Modfied Time Name -rw-r--r-- 3.5 kB 07/07/16 16:38:52 JST api.stderr.0 -rw-r--r-- 4.6 kB 08/07/16 09:14:31 JST api.stdout.0 # nomad fs cat 45f2e143 alloc/logs/api.stdout.0 © 2016 CyberAgent, Inc. 52
53. શ Job ͷ֬ೝ $ nomad status ID Type Priority Status axion-api service 50 running axion-another-service service 50 running axion-foo-batch batch 50 running axion-foo-batch/periodic-1467924600 batch 50 dead axion-foo-batch/periodic-1467928200 batch 50 dead axion-foo-batch/periodic-1467931800 batch 50 dead axion-foo-batch/periodic-1467935400 batch 50 dead © 2016 CyberAgent, Inc. 53
54. Axion ͱ Nomad © 2016 CyberAgent, Inc. 54
55. λεΫͷ഑ஔ 4 Nomad Ϋϥελ͸ 1 ͚ͭͩ 4 ͲͷϊʔυʹλεΫ͕഑ஔ͞ΕΔ͔͸෼͔Βͳ ͍ 4 ύϑΥʔϚϯεʹහײͳ api ͸ϊʔυΛ઎༗͠ ͍ͨ 4 ֤छ worker ΍όον͸ద౰ʹۭ͍͍ͯΔϊ ʔυͰಈ͍ͯ͘ΕΕ͹͍͍ © 2016 CyberAgent, Inc. 55
56. Job constraint constraint { attribute = "${meta.role}" value = "api" } 4 Nomad ͷϊʔυʹઃఆͰ͖Δ meta ม਺Λ΋ ͱʹλεΫ഑ஔΛ੍ݶ © 2016 CyberAgent, Inc. 56
57. Nomad ͷ meta ઃఆ 4 Consul ͷ KVS ͰϊʔυʹରԠ͢ΔΩʔʹ஋Λઃఆ͢Δͱ consul-template ͕൓Ԡ ͯ͠ Nomad ͷઃఆΛม͑Δ © 2016 CyberAgent, Inc. 57
58. Nomad Λ࢖ͬͨ Axion ͷσϓϩΠϑϩʔ © 2016 CyberAgent, Inc. 58
59. 1). Bot ʹϦϦʔε४උΛґཔ © 2016 CyberAgent, Inc. 59
60. 2). ϦϦʔε PR ͕࡞੒͞ΕΔ 4 master -> deployment/release ΁ͷϚʔδ © 2016 CyberAgent, Inc. 60
61. master ʹϚʔδ͞Εͨ PR Λऩूͯ͠Ϧετ Խ © 2016 CyberAgent, Inc. 61
62. ֤ PR ͷ ## Release Notes ηΫγϣϯΛ ू໿ © 2016 CyberAgent, Inc. 62
63. 3). PR ϚʔδͰ CircleCI ͕Ϗϧυ deployment: release: branch: deployment/release commands: - tar zcf deploy.tar.gz deploy/* - slack-post.sh @beckyy: deploy acd-axion #$CIRCLE_BUILD_NUM using deploy.tar.gz with deploy/deploy.sh stg general: artifacts: - deploy.tar.gz 4 ϦϙδτϦ಺ͷ deploy εΫϦϓτͳͲΛ artifact ͱͯ͠อଘ 4 Bot ΁ͷσϓϩΠίϚϯυΛ Slack ΁ϙετ © 2016 CyberAgent, Inc. 63
64. 4). Bot ͕ stg ΁σϓϩΠΛ࣮ߦ 4 CircleCI ͔Β deploy artifact Λऔಘͯ͠ίϚϯυΛ࣮ߦ © 2016 CyberAgent, Inc. 64
65. 5). ୲౰͕ prd σϓϩΠΛࢦࣔ 4 stg Ͱͷ֬ೝͱϦϦʔεͷ߹ҙΛͱ͔ͬͯΒ࣮ ߦ © 2016 CyberAgent, Inc. 65
66. deploy.sh # Copy job spec to deploy server scp -pq *.nomad centos@deploy:/home/centos/axion/$ENV/nomad # Run new job spec on deploy server cat <<EOF | ssh -qT centos@deploy cd /home/centos/axion/$ENV nomad run nomad/axion-api.nomad EOF © 2016 CyberAgent, Inc. 66
67. Nomad Λ࠾༻ͨ͠ཧ༝ 4 Kubernetes ͱ໎ͬͨ 4 Nomad / Kubernetes ͱ΋ʹ΄΅஌͕ࣝແ͍ ঢ়ଶ 4 طʹ Consul Λ࢖͏͜ͱ͸ܾΊ͍ͯͨ 4 ߏ੒͕γϯϓϧͰػೳ΋গͳ͍͜ͱ͔Βֶशί ετͷ௿ͦ͏ͳ Nomad Λબ୒ © 2016 CyberAgent, Inc. 67
68. Nomad Ͱࠔͬͨͱ͜Ζ 4 Web UI (μογϡϘʔυ)͕ͳ͍ 4 Docker ͷ Volume Λαϙʔτ͍ͯ͠ͳ͍ 4 Rolling Upadte ͱϔϧενΣοΫ͕౷߹͞Ε ͍ͯͳ͍ 2015-09 ͷϦϦʔε͔Β·ͩ 1 ೥ܦ͍ͬͯͳ͍ͷ Ͱ࢓ํͳ͍໘΋… © 2016 CyberAgent, Inc. 68
69. Nomad ·ͱΊ 4 খ͞ͳ࢓ࣄΛ࣮֬ʹ͜ͳͯ͘͠ΕΔϠπͰ͸͋ Δ 4 ΞϓϦͷ഑ஔ͚ͩɺͳͲͱখ͘͞ར༻͢Δʹ͸ ྑ͍ 4 ͓΋ͯͳ͠ײ͸ͳ͍ 4 ے͸͍͍ͷͰࠓޙͷ੒ख़ʹظ଴ © 2016 CyberAgent, Inc. 69
70. Terraform Infrastructure as a Code © 2016 CyberAgent, Inc. 70
71. What's Terraform ? 4 Πϯϑϥͷ࡞੒ɾมߋɾόʔδϣϯ؅ཧͷͨΊ ͷπʔϧ 4 Πϯϑϥͷߏ੒ཁૉΛશͯίʔυͰهड़ 4 ༷ʑͳϦιʔεͷ؅ཧʹରԠ 4 AWS CloudFormation ΍ OpenStack Heat ʹ૬౰͢Δπʔϧ © 2016 CyberAgent, Inc. 71
72. Technology Radar 4 Technology Rader 2015-01 7 Ͱ ASSESS (ௐࠪͤΑ) 7 https://www.thoughtworks.com/radar/tools/terraform © 2016 CyberAgent, Inc. 72
73. Resource Providers Atlas / AWS / Azure (Service Management) / Azure (Resource Manager) Chef / CenturyLinkCloud / CloudFlare / CloudStack / Cobbler / Consul Datadog / DigitalOcean / DNSMadeEasy / DNSimple / Docker / Dyn GitHub / Fastly / Google Cloud / Heroku / InfluxDB / Librato Mailgun / MySQL / OpenStack / Packet / PostgreSQL / PowerDNS Rundeck / StatusCake / SoftLayer / Template / Terraform / TLS Triton / UltraDNS / VMware vCloud Director / VMware vSphere © 2016 CyberAgent, Inc. 73
74. ఆٛϑΝΠϧ resource "openstack_compute_instance_v2" "dock" { region = "${var.region}" name = "${format("acd-${var.env}-axion-dock-y%03d", count.index + 1)}" count = "10" image_name = "centos-7.2.1511" flavor_name = "s2.medium" security_groups = ["default"] network { name = "${var.network_name}" access_network = true } user_data = "${template_cloudinit_config.dock_cloudinit.rendered}" lifecycle { ignore_changes = ["user_data"] } } © 2016 CyberAgent, Inc. 74
75. terraform plan $ terraform plan + module.axion.openstack_compute_instance_v2.dock.1 access_ip_v4: "" => "<computed>" flavor_name: "" => "s2.medium" image_name: "" => "centos-7.2.1511" name: "" => "acd-stg-axion-dock-y001" security_groups.#: "" => "1" security_groups.3814588639: "" => "default" user_data: "" => "4a068b7a5a43f31d9bd280a20a04823bbf4082d8" © 2016 CyberAgent, Inc. 75
76. terraform apply $ terraform apply -parallelism=1 module.axion.openstack_compute_instance_v2.dock.1: Creating... access_ip_v4: "" => "<computed>" flavor_name: "" => "s2.medium" image_name: "" => "centos-7.2.1511" name: "" => "acd-stg-axion-dock-y001" security_groups.#: "" => "1" security_groups.3814588639: "" => "default" user_data: "" => "b0f6c60d289b151ef07f89fe543ee6edb46f0f1a" module.axion.openstack_compute_instance_v2.dock.1: Still creating... (10s elapsed) module.axion.openstack_compute_instance_v2.dock.1: Still creating... (20s elapsed) module.axion.openstack_compute_instance_v2.dock.1: Still creating... (30s elapsed) module.axion.openstack_compute_instance_v2.dock.1: Creation complete Apply complete! Resources: 1 added, 0 changed, 0 destroyed. © 2016 CyberAgent, Inc. 76
77. Axion ͱ Terraform © 2016 CyberAgent, Inc. 77
78. ؅ཧର৅ʹ͍ͯ͠ΔϦιʔε 4 openstack_compute_instance_v2 4 openstack_compute_servergroup_v2 ηΩϡϦςΟάϧʔϓ΍ωοτϫʔΫ͸ Terraform ͷ؅ཧର৅֎ͱ͠ɺCIA ͕༻ҙͨ͠ ΋ͷΛ ID ΍໊લͰࢀর͢ΔΑ͏ʹ͍ͯ͠ΔɻΑͬ ͯɺεΫϥον͔Β Terraform ͚ͩͰߏஙͰ͖ ΔΘ͚Ͱ͸ͳ͍ɻ © 2016 CyberAgent, Inc. 78
79. Terraform ઃఆͷϨΠΞ΢τ |-- modules | `-- axion | |-- main.tf | ڞ௨Ϧιʔεఆٛ | |-- dock.tf | dock ϩʔϧఆٛ | |-- council.tf | council ϩʔϧఆٛ | `-- variables.tf | Ϟδϡʔϧม਺ఆٛ |-- dev | |-- .envrc | OpenStack ؀ڥม਺ (not in repository) | `-- main.tf | dev ؀ڥઃఆ |-- stg | |-- .envrc | `-- main.tf | stg ؀ڥઃఆ `-- prd |-- .envrc `-- main.tf | prd ؀ڥઃఆ ࢀߟ: https://atlas.hashicorp.com/help/intro/use-cases/multiple- environments © 2016 CyberAgent, Inc. 79
80. Terraform ͷ࣮ߦ؀ڥ $ cd ~/axion/terraform $ ls dev modules prd stg 4 ؅ཧαʔόʹ Terraform ϦϙδτϦΛνΣο ΫΞ΢τ 4 ߋ৽࣌ʹ pull ͨ͠Γ͢Δͷ͸౎౓खಈ… ! © 2016 CyberAgent, Inc. 80
81. Terraform ͷ࣮ߦ $ cd stg direnv: loading .envrc direnv: export +OS_PROJECT_NAME +OS_REGION_NAME +OS_TENANT_ID $ terraform plan 4 ૢ࡞ର৅ͷ؀ڥͷσΟϨΫτϦʹೖΔͱ direnv ʹΑͬͯ OpenStack ؀ڥม਺͕ઃఆ ͞ΕΔ 4 terraform ίϚϯυΛ࣮ߦ΋खಈ… ! © 2016 CyberAgent, Inc. 81
82. OpenStack ؀ڥม਺ $ cat .envrc export OS_AUTH_URL="http://openstack.local/v2.0/" export OS_TENANT_ID=**** export OS_TENANT_NAME=**** export OS_PROJECT_NAME=**** export OS_USERNAME=**** export OS_PASSWORD=**** export OS_REGION_NAME=**** 4 OpenStack ͷ API Λୟ͘πʔϧΛ࢖͏ͱ͖ʹඞཁͳ؀ڥม ਺ 4 OpenStack ͷ Dashboard ͔ΒςϯϓϨʔτΛμ΢ϯϩʔυ Մೳ © 2016 CyberAgent, Inc. 82
83. .tfstate ʹ͍ͭͯ $ cd dev; ls -1 main.tf terraform.tfstate <-- ͜Ε terraform.tfstate.backup 4 ࠷ޙʹ apply ͨ͠ͱ͖ͷΠϯϑϥͷঢ়ଶΛه࿥ 4 ద༻͢΂͖มߋͷࠩ෼Λܭࢉ͢ΔͨΊʹ ඞਢ 4 refresh Ͱ࣮Πϯϑϥ͔Β͋Δఔ౓͸࠶ߏஙͰ͖Δ ͕ɺखಈͰϦιʔεঢ়ଶͷ੔߹ੑΛऔΔඞཁ͕͋Δ 4 Terraform Λӡ༻͢Δ্Ͱͷ೰ΈͲ͜Ζ © 2016 CyberAgent, Inc. 83
84. .tfstate ͷ؅ཧ 4 ͻͱ·ͣ terraform ͷ࣮ߦΛ؅ཧαʔόʹݶ ఆͯ͠ɺͦͷ··ϩʔΧϧσΟεΫʹஔ͍͍ͯ Δ 4 Πϯϑϥͷมߋ͸ස౓͕௿͍ͷͰʮӡ༻Ͱ Χόʔʯ 4 ֎෦ετϨʔδ(s3, artifactory ͳͲ)ʹ഑ஔ ͢Δ͜ͱ΋Մೳ © 2016 CyberAgent, Inc. 84
85. Axion ͷॳظߏங 4 ݸਓ؀ڥΛαϯυϘοΫεʹઃఆΛॻ͘ 4 ͻͨ͢Β apply and destroy ! 4 ͋Δఔ౓ݻ·ͬͨΒຊ൪Ϧʔδϣϯ༻ʹม਺ͩ ͚มߋͯ͠ద༻ 4 dev ؀ڥͰ࠷ऴௐ੔ 4 stg / prd ͷߏங͸ apply ͢Δ͚ͩͰ׬ྃ © 2016 CyberAgent, Inc. 85
86. Axion ͷ cloud-init 4 Πϯελϯε໊͔Β hostname ઃఆ 4 ಺෦ DNS ΁ͷϨίʔυొ࿥ 4 LDAP ϩάΠϯͷηοτΞοϓ 4 chef-client ࣮ߦ 4 CIA ؅ཧͷجຊ Cookbook ͷ࣮ߦ 4 ϛυϧ΢ΣΞͷΠϯετʔϧ 4 ϩʔϧݻ༗ͷηοτΞοϓ 4 ϛυϧ΢ΣΞ΍ Docker ίϯςφͷىಈ cloud-init ͚ͩͰηοτΞοϓ͕׬݁͢ΔΑ͏ʹ͍ͯ͠Δ © 2016 CyberAgent, Inc. 86
87. Terraform ӡ༻ͷཧ૝ 4 ΞϓϦͱಉ͘͡ GitHub Ͱมߋ͔ΒσϓϩΠ ·ͰͷαΠΫϧΛճ͍ͨ͠ 4 PR ʹ plan ͷ݁ՌΛࣗಈతʹϙετͯ͠มߋ ఺ΛϨϏϡʔ 4 Ϛʔδ͢Δͱ CI ͕૸ͬͯมߋΛద༻ © 2016 CyberAgent, Inc. 87
88. Terraform ·ͱΊ 4 ࠓ·ͰΠϯελϯεͷ಺෦͸ Chef ͳͲͰߏ੒؅ཧͰ͖͍ͯͨ ͕ɺΠϯελϯεͦͷ΋ͷ͸γΣϧεΫϦϓτΛୟ͍ͯ࡞੒͠ ͍ͯͨ 4 Adhoc ͳૢ࡞ + ࡞ۀϩά 4 ࣗಈԽͱ͍͏จ຺Ͱ͸͞΄Ͳखؒ͸มΘΒͳ͍͔΋͠Εͳ͍ 4 ΠϯϑϥΛίʔυͱͯ͠දݱ͢ΔϝϦοτ 4 ࠶ݱੑͷ୲อ 4 ҉໧஌ͷഉআ 4 ཤྺͷ؅ཧ (git log) © 2016 CyberAgent, Inc. 88
89. Terraform ·ͱΊ 4 Terraform ͦͷ΋ͷ͸ಛʹ໰୊ͳ͘ѻ͑ͨ ! 4 Πϯελϯε಺෦Λ cloud-init Ͱߏங͢Δ ͱ͜Ζ͸େมͩͬͨ… " 4 มߋద༻લʹ plan Λ֬ೝͰ͖Δͷ͸ྑ͍ 4 ੲͷ CloudFormation ͸ dry-run ͕Ͱ͖ ͳͯ͘ා͔ͬͨ © 2016 CyberAgent, Inc. 89
90. Terraform ·ͱΊ 4 ॳظߏஙޙͷӡ༻͕ઙ͍ͷͰࠓޙͭΒ͍͜ͱ͕͋ Δ͔΋͠Εͳ͍ ! 4 ϓϩάϥϛϯάͱಉ͘͡ɺಡΈ΍͍͢ίʔυΛॻ ͘৺ֻ͚͕ඞཁ 4 Axion ͸ cloud-init ·ΘΓ͕͔ͳΓԚͳ͍… 4 ׬શͳ Container ϕʔεͷੈքͰ͋Ε͹Πϯελ ϯε؅ཧͱͯ͠ͷ Terraform ͸ෆཁʹͳΓͦ͏ © 2016 CyberAgent, Inc. 90
91. Axion Πϯϑϥͷ શମ૾ © 2016 CyberAgent, Inc. 91
92. © 2016 CyberAgent, Inc. 92
93. ΦϨ͸Α͏΍͘ ͷ΅Γ͸͡Ίͨ ͹͔Γ͔ͩΒͳ ͜ͷ͸ͯ͠ͳ͘ԕ͍ Docker ࡔΛΑ… © 2016 CyberAgent, Inc. 93