Consulと自作OSSを活用した100台規模のWebサービス運用

YAPC::Asia 2015
http://yapcasia.org/2015/talk/edit/b335dee0-09ad-11e5-8d7a-67dc7d574c3a

Transcript

1. Consulͱࣗ࡞OSS Λ׆༻ͨ͠ 100୆ن໛ͷWeb αʔϏεӡ༻ YAPC::Asia 2015 @fujiwara
2. ౻ݪ ढ़Ұ࿠ @fujiwara github.com/fujiwara sfujiwara.hatenablog.com ٕज़෦
3. Agenda Lobi Consul Consul & My OSS in Lobi
4. Game & Community
5. Lobi - εϚʔτϑΥϯήʔϜʹಛԽͨ͠ίϛϡχςΟΛఏڙ
6. Lobi - εϚʔτϑΥϯήʔϜʹಛԽͨ͠ίϛϡχςΟΛఏڙ
7. Lobiͷαʔό
8. Lobiͷαʔόมભ 1. 2010~2011 AWS (US) : 4୆(?) 2. 2011~2013 ࣗࣾαʔό : 4 ~ 20୆ 3. 2013.11~ AWS (Tokyo) : 20 ~ 100୆
10. Lobiͷαʔό EC2ͰՔಇ͍ͯ͠Δϗετͷछྨ͕ଟ͍ • app, sdk, stream, sdk-stream, db(3shard), transcode, log(aggregate,analyze),batch, deploy... ࣮૷ݴޠ Perl, Node.js, Go ϛυϧ΢ΣΞ͍Ζ͍Ζ • Nginx, MySQL, Starlet, Fluentd, Norikra, memcached, HAProxy, gearman, twemproxy, MHA, dnsmasq...
11. AWS(Tokyo)Ҡߦޙͷ೰Έ(2014) ಺෦DNS͕ͳ͍ • /etc/hostsΛChefͰੜ੒ • DNSαʔόࣗલ͸໘౗…(౰࣌Internal Route53 ͳ͠) ϗετ௥Ճɼ࡟আͷස౓͕ߴ͍ • Φʔτεέʔϧ΋͍ͨ͠ ConsulΛ࢖͓͏ʂ
12. Consul?
13. What's Consul www.consul.io - HashiCorp product • Service Discovery • Health Checking • Key/Value Store • Multi Datacenter
14. Architecture
17. Agent Ϋϥελ಺ͷnodeશ୆Ͱಈ࡞͢Δdaemon Client mode or Server mode ͷͲͪΒ͔Ͱಈ࡞ ϢʔβʹDNS, HTTP interfaceΛఏڙ͢Δ • Ϣʔβ͸جຊతʹlocalhostͷagentͱ௨৴ • Agentಉ͕࢜RPCͰ௨৴(͋·Γҙࣝ͢Δඞཁ͸ ͳ͍) GoͰॻ͔Ε͍ͯͯ1όΠφϦͰಈ࡞ (CLI΋ಉҰ)
18. Server Ϋϥελ಺ͷগ਺nodeͰಈ࡞(3Ҏ্ͷح਺ਪ঑) ৗʹͲΕ͔1୆ͷΈLeaderʹͳΔ • RaftΞϧΰϦζϜʹΑͬͯLeader͕બग़͞ΕΔ σʔλͷॻ͖ࠐΈ͸ৗʹLeaderʹ σʔλ͸Leader͔ΒଞͷServer΁ϨϓϦϨʔγϣ ϯ
19. Raft Raft is a protocol for implementing distributed consensus. ෼ࢄ؀ڥͰͷ߹ҙΞϧΰϦζϜ(ϓϩτίϧ) Leaderબग़ʹserver nodeͷա൒਺ͷಉҙ͕ඞཁ ͳͷͰ࠷௿3 node͕ඞཁ
20. Raft ࢀߟࢿྉ thesecretlivesofdata.com/raft/ • ΞχϝʔγϣϯͰΘ͔Γ΍͍͢ www.slideshare.net/pfi/raft-36155398 • ೔ຊޠͰͷղઆࢿྉ github.com/hashicorp/raft • Consul͕࢖༻͍ͯ͠ΔGo࣮૷
21. Service / Node Discovery
22. Service / Node Discovery αʔϏε಺ͷnode͢΂ͯͰΫϥελΛߏ੒͢Δ • nodeɺnode্Ͱఏڙ͢ΔαʔϏεͷൃݟ • DNS, HTTP API ࢮ׆؂ࢹࣦഊͨ͠node͸Ԡ౴͔ΒࣗಈͰ֎ΕΔ
24. Node Discovery consul members ͰΫϥελ಺ͷnodeΛҰཡ $ consul members Node Address Status Type Build Protocol DC my-app-i-123456 192.168.1.12:8301 alive server 0.5.2 2 dc1 my-app-i-234567 192.168.1.23:8301 alive server 0.5.2 2 dc1 my-app-i-345678 192.168.1.34:8301 alive server 0.5.2 2 dc1 my-db-i-456789 192.168.1.45:8301 alive client 0.5.2 2 dc1 my-db-i-567890 192.168.1.56:8301 alive client 0.5.2 2 dc1 my-db-i-678901 192.168.1.67:8301 alive client 0.5.2 2 dc1 my-app-i-987654 192.168.1.99:8301 failed client 0.5.2 2 dc1 my-app-i-876543 192.168.1.87:8301 left client 0.5.2 2 dc1 (࣮ࡍ͸݁Ռͷॱ൪͸ෆఆ)
25. Node Discovery Status=failed : agentͷࢮ׆؂ࢹʹࣦഊͨ͠node $ consul members -status failed Node Address Status Type Build Protocol DC my-app-i-987654 192.168.1.99:8301 failed server 0.5.2 2 dc1 Status=left : ਖ਼ৗʹΫϥελ͔Β཭୤ͨ͠node $ consul members -status left Node Address Status Type Build Protocol DC my-app-i-876543 192.168.1.87:8301 left server 0.5.2 2 dc1
26. Node Discovery via DNS interface consul agent (127.0.0.1:8600) ʹ໰͍߹ΘͤΔ $ dig @127.0.0.1 -p 8600 my-app-i-123456.node.consul ;; QUESTION SECTION: ;my-app-i-123456.node.consul. IN A ;; ANSWER SECTION: my-app-i-123456.node.consul. 0 IN A 192.168.1.12
27. Node Discovery via DNS interface Status=failed : DNSͰΞυϨε͕Ҿ͚Δ • Ұ࣌తʹࣄނͰ཭୤͍ͯ͠ΔՄೳੑ͕͋Δ Status=left : DNSͰΞυϨε͕Ҿ͚ͳ͘ͳΔ • .consul υϝΠϯͷ໊લղܾΛ consul agent ʹ ౤͛Δ͜ͱͰ಺෦DNSͱͯ͠ར༻Ͱ͖Δ • υϝΠϯ໊͸ઃఆͰมߋՄೳ
28. Service Difinition of App my-app-i-*:/etc/consul.d/app.json { "service": { "name": "app", "port": 3000 } }
29. Service Discovery of App via DNS interface consul agent (127.0.0.1:8600) ʹ໰͍߹ΘͤΔ $ dig @127.0.0.1 -p 8600 app.service.consul ;; QUESTION SECTION: ;app.service.consul. IN A ;; ANSWER SECTION: app.service.consul. 0 IN A 192.168.1.12 app.service.consul. 0 IN A 192.168.1.23 app.service.consul. 0 IN A 192.168.1.34
30. Service Discovery via DNS interface Answerͷॱ൪͸ϥϯμϜ (≒ Round Robin) UDPͰͷ໰͍߹ΘͤͰ͸ର৅͕4ΞυϨεҎ্ ͋Δ৔߹ɺ3ΞυϨεͷΈฦΔ • TCPͰ͸͢΂ͯฦΔ TTL ઃఆՄೳ (default 0)
31. Service Discovery of App via HTTP API http://127.0.0.1:8500 ʹΞΫηε $ curl http://127.0.0.1:8500/v1/catalog/service/app [ { "Node": "my-app-i-123456", "Address": "192.168.1.12", "ServiceID": "app", "ServiceName": "app", "ServicePort": 3000, ... }, { "Node": "my-app-i-234567", "Address": "192.168.1.23", ... } { "Node": "my-app-i-345678", "Address": "192.168.1.34", ... } ]
32. Service Difinision of DB (master) my-db-i-456789:/etc/consul.d/db.json { "service": { "name": "db", "port": 3306, "tags": ["master"] } }
33. Service Difinision of DB (slave) my-db-i-567890,678901:/etc/consul.d/db.json { "service": { "name": "db", "port": 3306, "tags": ["slave"] } }
34. Service Discovery of DB (master/slave) {tag}.{service}.service.consul Ͱ໊લղܾ $ dig @127.0.0.1 -p 8600 master.db.service.consul master.db.service.consul. 0 IN A 192.168.1.45 $ dig @127.0.0.1 -p 8600 slave.db.service.consul slave.db.service.consul. 0 IN A 192.168.1.67 slave.db.service.consul. 0 IN A 192.168.1.56 $ dig @127.0.0.1 -p 8600 db.service.consul db.service.consul. 0 IN A 192.168.1.56 db.service.consul. 0 IN A 192.168.1.45 db.service.consul. 0 IN A 192.168.1.67
35. External Service nodeʹؔ࿈͠ͳ͍ɺ֎෦DNSͰఆٛ͞Ε͍ͯΔ ໊લ΍IPΞυϨε΋αʔϏεͱͯ͠ఆٛͰ͖Δ $ curl -X PUT -d '{ "Node":"rds", "Address":"my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com", "Service":{"Service": "rds"} }' http://127.0.0.1:8500/v1/catalog/register $ dig @127.0.0.1 -p 8600 rds.service.consul. ;; ANSWER SECTION: rds.service.consul. 0 IN CNAME my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com. my-rds.xxxxx.ap-northeast-1.rds.amazonaws.com. 10 IN A 192.168.1.100
36. External Service DNS໊Λొ࿥͢Δ৔߹ɺconsulͷઃఆʹ recursors (֎෦ͷ໊લղܾΛ͢ΔDNSαʔόͷ ΞυϨε) Λ͓ͯ͘͠ • ઃఆ͠ͳ͍ͱ CNAME ͷΈ͕ฦΔ consul agent͸֎෦ͷ໊લղܾ݁ՌΛcache͠ͳ ͍
37. Health Checking
38. Health Checking ֤αʔϏεʹ͍ͭͯɺ3λΠϓͷϔϧενΣοΫ ΛఆٛͰ͖Δ • Script • HTTP • TTL
39. Health Checking by script ϢʔβఆٛͷϔϧενΣοΫίϚϯυΛ࣮ߦ exit codeͰঢ়ଶΛ௨஌ (Nagios/Sensuޓ׵) • 0 : success • 1 : warning • 2 : fail failͷ৔߹͸DNS, HTTPͷԠ౴͔Β֎ΕΔ
40. Health Checking by HTTP consul agent͕HTTPͰΞΫηε • HTTP 2xx : success • HTTP 429 : warning • ͦΕҎ֎ : fail
41. Health Checking by TTL ఆظతʹagentʹHTTP PUTͯ͠ੜଘΛ఻͑Δ TTL͕੾ΕΔ·ͰʹPUT͕ͳ͚Ε͹failѻ͍
42. Key/Value Store
43. Key/Value Store ೚ҙͷ஋Λग़͠ೖΕͰ͖ΔKVS $ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1' true $ curl http://127.0.0.1:8500/v1/kv/web/key1 [{ "CreateIndex": 112, "ModifyIndex": 112, "LockIndex": 0, "Key": "web/key1", "Flags": 0, "Value": "dGVzdA==" }]
44. Key/Value Store URLҾ਺Ͱϝλσʔλ (flags) Λอ࣋Ͱ͖Δ 64bit int, ༻్͸Ϣʔβͷ೚ҙ $ curl -XPUT -d 'test' 'http://127.0.0.1:8500/v1/kv/web/key1?flags=123' true $ curl 'http://127.0.0.1:8500/v1/kv/web/key1' [{ ... "Key": "web/key1", "Flags": 123, // <------- ͜Ε "Value": "dGVzdA==" }]
45. Key/Value Store jsonϨεϙϯεͷ஋͸Base64 encode͞Ε͍ͯΔ jsonͰ͸ͳ͘஋͚ͩੜͰऔΓ͍ͨ৔߹͸Ҿ਺ raw $ curl "http://127.0.0.1:8500/v1/kv/web/key1?raw" test
46. Key/Value Store ͋Δ֊૚ͷԼͷ஋Λ࠶ؼతʹऔΓ͍ͨ৔߹͸ Ҿ਺ recurse $ curl -s "http://127.0.0.1:8500/v1/kv/web/?recurse" [ {"CreateIndex":112,"ModifyIndex":115,"LockIndex":0, "Key":"web/key1","Flags":123,"Value":"dGVzdA=="}, {"CreateIndex":122,"ModifyIndex":122,"LockIndex":0, "Key":"web/key2","Flags":0,"Value":"dGVzdDI="}, {"CreateIndex":124,"ModifyIndex":124,"LockIndex":0, "Key":"web/test/1","Flags":0,"Value":"dGVzdDM="} ] όοΫΞοϓʹ΋ར༻Մೳ
47. Key/Value Store Benchmark GET $ wrk -c 10 -d 10 -t 2 http://127.0.0.1:8500/v1/kv/web/key1 Server(Leader): 41,832 Requests/sec Server/Client(Follower): 17,281 Server(Follower) stale mode: 37,013 Client(Follower) stale mode: 16,938 Consul v0.5.2 on EC2 c4.2xlarge, GOMAXPROCS=4
48. Key/Value Store Benchmark PUT $ wrk -c 10 -d 10 -t 2 -s put.lua http://127.0.0.1:8500/v1/kv/web/key1 -- put.lua wrk.method = "PUT" wrk.body = "test" wrk.headers["Content-Type"] = "application/x-www-form-urlencoded" Server/Client(Follower): 427.56 Requests/sec
49. Key/Value Store ৗʹ 127.0.0.1:8500 Λ࢖͑͹Α͍ͷͰָ ͔ͳΓߴ଎ͳͷͰGET͸ԕྀͳ͘࢖͑Δ σϑΥϧτͰ͸͢΂ͯͷ໰͍߹ΘͤΛ Leader node͕ॲཧ͢Δ • stale modeʹ͢ΔͱLeaderҎ֎ͷServer΋Ԡ ౴Ͱ͖Δ (Ұ؏ੑʹ͍ͭͯ͸ޙड़)
50. ಺෦DNSͱͯ͠ConsulΛ࢖͏
51. ಺෦DNSͱͯ͠ConsulΛ࢖͏ node,serviceͷ໊લղܾΛConsul Agent΁޲͚Δ ! resolv.confͰ͸ϙʔτࢦఆͰ͖ͳ͍ ! Port 53͸ಛݖ͕ඞཁ → ͦͷͨΊ͚ͩʹAgentΛrootͰಈ͔͢ͷ͸ ! Agent͸࠶ؼ໰͍߹Θͤ͸(ҰԠ)Ͱ͖Δ͕cacheػ ೳ͕ͳ͍ ! → dnsmasq, bindͳͲ͔Β.consulυϝΠϯͷΈ forward !
53. ಺෦DNSͱͯ͠ConsulΛ࢖͏ dnsmasqΛશ୆Ͱىಈ .consul υϝΠϯͷ໊લղܾ͸consul agent΁ 127.0.0.1:53 Λdnsmasq͕Listen͢Δ # dnsmasq.conf server=/consul/127.0.0.1#8600 bind-interfaces listen-address=127.0.0.1
54. ಺෦DNSͱͯ͠ConsulΛ࢖͏ resolv.conf Ͱ (node|service).consul Λݕࡧυ ϝΠϯʹࢦఆ → node໊ɺservice໊͚ͩͰ઀ଓͰ͖Δ # /etc/resolv.conf search node.consul service.consul nameserver 127.0.0.1 # dnsmasq nameserver 172.16.0.2 # VPC resolver nameserver 172.16.0.254 # Unbound on EC2
55. ಺෦DNSͱͯ͠ConsulΛ࢖͏ consul.io/docs/guides/forwarding.html BINDΛ࢖͏ྫ
56. ConsulΛຊ൪؀ڥͰӡ༻͢Δ ͨΊʹ
57. Deployment Table Server node: ຊ൪Ͱ͸࠷௿3୆, 3 or 5͕ਪ঑ consul.io/docs/internals/consensus.html
58. ServerʹඞཁͳϦιʔε • CPU: 2CPUͰे෼ • GOMAXPROCS=2 Ҏ্Ͱಈ࡞ͤ͞ΔͷΛڧ͘ਪ ঑ • Memory: 20MBʙ • Disk: 2MBʙ Memory, Disk͸KVͷར༻ঢ়گ࣍ୈ
59. Serverʹ͸ઐ༻ϗετ͕ඞཁʁ consul agentࣗମ͸ͦΕ΄ͲϦιʔεΛ ࢖༻͠ͳ͍ͨΊɺಉډՄೳ͕ͩ… Disk IO͕ߴෛՙͳ৔߹ʹRaftͷHeartbeat͕ ࣦഊ͠΍͍͢ • Timeout 500ms • Heartbeatʹࣦഊ͢ΔͱLeaderબग़͕ߦΘΕΔ • ௨ৗ2,3ඵͰબग़͸׬ྃ͢Δ • ͦͷؒॻ͖ࠐΈॲཧ͕Ͱ͖ͳ͍
60. Daemonize consul agentࣗ਎͸Deamonಈ࡞Ϟʔυ͕ͳ͍ • Daemontools • RPM & init script • github.com/tomhillable/consul-rpm • Systemd ͲΕͰ΋͓޷ΈͰ
61. Bootstrapping consul agent͸ىಈޙɺΫϥελʹjoin͢Δඞཁ ͕͋Δ 1. ίϚϯυͰ consul joinΛ࣮ߦ 2. ઃఆϑΝΠϧͰ start_join Λࢦఆ 3. Atlas࿈ܞ
62. 1. consul join αʔόͷΞυϨεΛࢦఆͯ͠join $ consul join 192.168.1.11 192.168.1.12 192.168.1.13 ࣗಈԽͮ͠Β͍ͷͰςετҎ֎Ͱ͸࢖Θͳ͍
63. 2. start_join consul.confʹαʔόͷΞυϨεΛࢦఆ͓ͯ͘͠ { "start_join": [ "192.168.1.11", "192.168.1.12", "192.168.1.13"] } consul serverʹ͸ݻఆIPΞυϨεΛৼΔ͔ɺผͷ DNSͰղܾͰ͖ΔΑ͏ʹ͓ͯ͘͠
64. 3. Atlas࿈ܞ Atlas - atlas.hashicorp.com Vagrant Packer Terraform ConsulΛ౷߹͢ΔαʔϏε $ consul agent ... \ -atlas=ATLAS_USERNAME/infrastructure \ -atlas-join \ -atlas-token="YOUR_ATLAS_TOKEN" \ ! ServerͷΞυϨεΛ؅ཧ͢Δඞཁ͕ͳ͍ͷͰָ " 11nodeҎ্͸$40/node
65. ߴՄ༻ੑͷͨΊʹ
66. ߴՄ༻ੑͷͨΊʹ Server୆਺ʹΑΓಉ࣌ʹো֐ Λىͯ͜͠΋໰୊ͳ͍node਺ ͕มΘΔ • 3 node → 1 • 5 node → 2 3 nodeߏ੒࣌ɺ2୆མͪͯ࢒ Γ1୆ʹͳͬͯ͠·͏ͱLeader ͕બग़Ͱ͖ͳ͍ ௕࣌ؒ੾Γ཭͢ϝϯςφϯε ࣌ʹ͸Ұ࣌తʹServer nodeΛ ૿΍͢ख΋
67. ߴՄ༻ੑͷͨΊʹ Server nodeͷfailover͸ࣗಈ ! Ϣʔβ͸ৗʹlocalhostͷagent͚ͩΛΈ͍ͯΕ ͹Α͍
68. nodeো֐࣌ͷӨڹ ! LeaderͰ͸ͳ͍ → " ଞnodeʹ͸Өڹͳ͠ ! Leader → " Leader࠶બग़ σϑΥϧτͰ͸͢΂ͯͷಡΈॻ͖ΛLeader͕ॲཧ (ڧҰ؏ੑ) Leader͕ܾ·Δ·ͰΞΫηεෆೳ (DNS, HTTP)
69. Stale mode (DNS) Leader࠶બग़͸௨ৗ2ʙ3ඵͰ׬ྃ ͦͷؒ΋DNSͰNode, Service໊ղܾΛ͍ͨ͠ʁ → Stale mode : Leaderະબग़Ͱ΋Ԡ౴Մೳ "dns_config":{ "allow_stale": true, // default false "max_stale": "10s" // default 5s } ݁Ռ͸ݹ͍Մೳੑ͕͋Δ(݁Ռ੔߹ੑ)
70. DNS TTL default͸TTL 0 → cache͞Εͳ͍ node, serviceผʹTTLΛઃఆՄೳ DNS cacheΛલஈʹ഑ஔͯ͠cacheͰ͖Δ "dns_config":{ "node_ttl": "60s", "service_ttl": { "*": "15s" } }
71. Stale mode (HTTP API) HTTP APIͰstale modeʹ͢Δ৔߹͸Ҿ਺ stale $ curl "http://127.0.0.1:8500/v1/kv/web/key1?stale" staleҾ਺ͳ͠ͰLeaderબग़தʹΞΫηε → 500 Internal Server Error
72. ӡ༻தͷUpgrade consul.io/docs/upgrading.html consul.io/docs/upgrade-specific.html όʔδϣϯผʹ஫ҙ఺͕͋ΔͷͰυΩϡϝϯτΛ ॱ൪ʹAgentΛೖΕସ͑Δ͜ͱͰ Rolling upgradeՄೳ (Leader nodeೖΕସ͑Ͱ࠶બग़͸ى͖Δ)
73. ҆ఆੑ v0.2࣌୅͔Β1೥Ҏ্ӡ༻ Agentϓϩηε͕མͪͨ͜ͱ͸ͳ͍ ! ΦϖϛεͰશServerΛಉ࣌ʹམͱ͢ͱճ෮ෆೳ → KV͸ఆظόοΫΞοϓΛ (Ϣʔβσʔλ͸ೖΕͪΌμϝ ! )
74. Agentࣗମͷ؂ࢹ Agent processࣗମͷ؂ࢹ͸ผ్ • process؂ࢹ(consul agent) • TCP/UDP 8600 (DNS) • TCP 8500 (HTTP) • http://127.0.0.1:8500/v1/status/leader ಺༰มߋݕ஌ • Leader Lost→࠶બग़ͰมΘΔ
75. Φʔτεέʔϧ؀ڥͰ ૿ݮ͢ΔαʔόΛѻ͏
76. 2014.02ʙ Lobi Rec SDKϦϦʔε εϚϑΥΞϓϦʹSDKΛ૊ΈࠐΜͰPlayಈը࿥ը αʔόʹuploadͯ͠ม׵ɺӾཡ ม׵ػೳ͸ElasticTranscoderͰ࣮૷
77. ElasticTranscoder ! Managed ServiceͳͷͰ؅ཧָ͕ ! ม׵ೳྗ͸উखʹεέʔϧ ! ͪΐͬͱ͓ߴ͍… • SD $0.017/min • HD $0.034/min ౤ߘ͋ͨΓ4ύλʔϯ ฏۉ2෼ = $0.204 ≒ 25ԁ
78. 2014.10 ϞϯελʔετϥΠΫ Lobi Rec SDKಋೖʂʂʂ
79. ϞϯελʔετϥΠΫ Lobi Rec SDKಋೖ ௒ώοτήʔϜʹ͖ͭ େྔͷಈը͕… !""""""!
80. EC2 Spot InstanceͰಈըม׵ ! ؅ཧ͕໘౗ ! উखʹεέʔϧ͸ͯ͘͠Εͳ͍ ! ElasticTranscoderΑΓѹ౗తʹ҆Ձ • ElasticTranscoder = $0.204/౤ߘ • EC2 Spot cc2.8xlarge(32core) = $0.45/hour
81. Spot InstanceͰΦʔτεέʔϧ
82. Spot InstanceͰΦʔτεέʔϧ • εέʔϧΞ΢τ: CPUෛՙͰ͸ͳ͘JobͷྔͰ • Job͕ཷ·Βͳ͍ݶΓCPU 100%͸ ! • εέʔϧΠϯ: CPU idle͕શମͷ25%Ҏ্
83. Φʔτεέʔϧ؀ڥͰͷΠϯελϯεىಈ 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ 2. consul join 3. ChefʹΑΔϓϩϏδϣχϯά 4. ΞϓϦέʔγϣϯͷ࠷৽൛Λdeploy 5. Zabbix΁ࣗಈొ࿥ͯ͠؂ࢹର৅ʹ௥Ճ
84. 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ ConsulͷͨΊʹҰҙͳhostname͕ඞཁ Cloud-InitͰىಈ͢ΔϗετͷλΠϓΛઃఆ #cloud-config runcmd: - [sh, -c, 'echo "HOSTNAME_PREFIX=transcode" > /etc/sysconfig/hostname-prefix'] rc.localͰಡΈࠐΉ # /etc/rc.local if [ -f /etc/sysconfig/hostname-prefix ]; then . /etc/sysconfig/hostname-prefix fi
85. 1. ࣗಈͰuniqueͳϗετ໊Λ෇͚Δ hostnameΛ prefix + InstanceID EC2 Name tag ෇༩ # /etc/rc.local instance_id=$(curl -s 169.254.169.254/latest/meta-data/instance-id) new_hostname="${HOSTNAME_PREFIX}-$instance_id" hostname $new_hostname aws ec2 create-tags \ --resources $instance_id \ --tags "Key=Name,Value=$new_hostname"
86. 2. ConsulΫϥελʹjoin consul agent -node $(hostname) ࣗಈతʹ಺෦DNSͰ໊લղܾՄೳʹ
87. 3. ChefʹΑΔϓϩϏδϣχϯά Consul KV͔ΒϗετλΠϓ͝ͱͷJSONΛऔಘ͠ ࣮ͯߦ JSON=$(curl -s "localhost:8500/v1/kv/nodes/bootup/${HOSTNAME_PREFIX}.json?raw") echo "$JSON" > /tmp/chef-bootup.json chef-client -j /tmp/chef-bootup.json ॳճChef࣮ߦ࣌ʹChef-Serverʹొ࿥͞ΕΔ
88. 4. ΞϓϦέʔγϣϯͷ࠷৽൛Λdeploy Stretcher(ޙड़)Ͱdeploy 5. Zabbix΁ࣗಈొ࿥ͯ͠؂ࢹର৅ʹ௥Ճ deploy࣮ߦޙɺzabbix-agentΛىಈˠࣗಈొ࿥ deploy׬ྃલʹىಈ͢ΔͱΞϥʔτ͕ൃ๒ͯ͠͠ ·͏
89. StretcherΛར༻ͨ͠σϓϩΠ Dan Zen https://www.flickr.com/photos/danzen/2288626158/
90. What's Stretcher github.com/fujiwara/stretcher Consul / Serf ͱ࿈ܞͯ͠ಈ͘σϓϩΠπʔϧ
91. Why Stretcher? Archer(rsync) ʹΑΔதԝϗετ͔Βͷdeploy ! pushͰ͸ΦʔτεέʔϧʹରԠͰ͖ͳ͍ ! ֤ϗετ͔ΒrsyncͰpull? → buildதʹrsync͞ΕͨΒ… ! ֤ϗετ͔Βgit pull? → grunt, GoͳͲͷbuildੜ੒෺ΛೖΕͨ͘ͳ͍ ! ୆਺͕ଟ͍ͱssh+rsync΋git pull΋πϥ͍ ! AMI࡞Γ௚͠&ೖΕସ͑͸଴ͯͳ͍
92. Inspired by github.com/sorah/mamiya & AWS CodeDeploy
93. ઃܭํ਑ AWS͡Όͳͯ͘΋ಈ͘ rsync & ίϚϯυ࣮ߦɺͱ͍͏ϑϩʔ͸౿ऻ Consul eventͰΠϕϯτ௨஌ Consul ͱ͸ૄ݁߹ (consul watchͰىಈ) GoͰॻ͘
94. Architecture
95. Consul event ֤nodeʹGossip ProtocolͰΠϕϯτΛ ૹ৴͢Δ࢓૊Έ $ consul event -name EVENT_NAME [-node REGEX] PAYLOAD Event ID: 3b1f3199-6e69-4b82-4812-b35058864fdd ࢦఆͨ͠Πϕϯτ໊Ͱ (ਖ਼نදݱʹϚον͢Δnodeʹ) payloadΛૹ৴
96. Consul watch ࢦఆͨ͠ΠϕϯτΛड৴ͨ͠ΒίϚϯυΛ ࣮ߦ͢Δ࢓૊Έ $ consul watch -type event -name EVENT_NAME COMMAND payload͸ඪ४ೖྗ͔ΒJSONͰ౉͞ΕΔ [{ "ID": "3b1f3199-6e69-4b82-4812-b35058864fdd", "Name": "test", "Payload": "TXkgcGF5bG9hZA==", ... }]
97. Deployment process 1. ΞϓϦέʔγϣϯΛbuildͯ͠tar.gzʹ͢Δ ґଘcpan moduleͳͲ͢΂ͯݻΊΔ 2. खॱॻ(manifest)Λॻ͘ 3. tar.gz, manifestΛS3(or httpd)ʹ্͛Δ 4. consul event Ͱ manifest URLΛ௨஌ consul event -name deploy s3://... ✄---------- ͜͜·ͰstretcherͰ͸ͳ͍ ---------✄
98. ✄------------ stretcher͔͜͜Β ------------✄ consul watch -type event -name deploy stretcher 1. event͔Βmanifest URLΛऔಘ 2. tar.gzΛऔಘͯ͠TMPDIRʹల։ 3. rsync -av --deleteͰߋ৽ 4. command࣮ߦ • ΞϓϦέʔγϣϯ࠶ىಈͳͲ
99. Manifest src: s3://example.com/app.tar.gz checksum: e0840daaa97cd2cf2175f9e5d133ffb3324a2b93 dest: /home/stretcher/app commands: pre: - echo 'staring deploy' post: - echo 'deploy done' success: - cat >> /path/to/success.log failure: - cat >> /path/to/failure.log excludes: - "*.pid" - "*.socket"
100. LobiͰͷdeploy tar.gz ໿200MB ల։͢Δͱ໿400MB • CPAN modules 110MB • node_modules 10MB × 5 • Go app binaries 8MB × 5 • Static files (S3ʹஔ͖͍ͨ)
101. LobiͰͷdeploy 1. ‐ Push to production branch 2. ! Build (1 min~) carton install, grunt, npm install, go build ... 3. " Pack tar.gz & Upload (1 min) 4. # Deploy by Startecher (10~30 sec)
102. LobiͰͷdeploy consul event ૹ৴͔Β10ʙ20ඵͰ׬ྃ ! 2015/08/05 14:58:30 Starting up stretcher agent 2015/08/05 14:58:30 Waiting for consul events from STDIN... 2015/08/05 14:58:30 Executing manifest: s3://... 2015/08/05 14:58:33 Extract archive: /dev/shm/stretcher539962129 to /dev/shm/stretcher_src648982332 2015/08/05 14:58:36 rsync [-av --delete --exclude-from /dev/shm/stretcher_src648982332/conf/rsync_exclude.web /dev/shm/stretcher_src648982332/ /home/xxx/web/] 2015/08/05 14:58:36 sending incremental file list ... sent 787780 bytes received 5230 bytes 1586020.00 bytes/sec total size is 359702435 speedup is 453.59 2015/08/05 14:58:36 invoking command: /home/xxx/web/refresh_services.sh 2015/08/05 14:58:41 success. 2015/08/05 14:58:41 Deploy manifest succeeded. Rollback͍ͨ͠ˠ௚લͷmanifestΛeventૹ৴ →10secͰ໭Δ
103. Deployͷ׬ྃΛ଴ͭ Πϕϯτૹ৴ݩͰ͸࣮ߦ׬͕ྃ෼͔Βͳ͍ • ࣦഊ͢Δϗετ͕͋Δ͔΋͠Εͳ͍ • Քಇதͷશ୆Ͱdeploy׬ྃͨ͠ͷΛ଴͍ͪͨ ……ϩά΋ݟ͍͚ͨͲ100୆෼Ͳ͏΍ͬͯʁ
104. Consul KV Dashboard
105. Chef, Serverspec, Deploy, etc... ֤ϗετͰ࣮ߦͨ݁͠ՌΛ֬ೝ͍ͨ͠΋ͷ • Chef • Serverspec • Stretcher • etc
106. IRC / Slackʹ௨஌ʁ ࣾ಺༻ nopaste-cli command $ nopaste-cli -channel "lobi" -summary "Deploy done!" < deploy.log nopasteʹPOSTͱಉ࣌ʹURL͕௨஌͞ΕΔ 100୆͋Δͱ௨஌͕……!!!!!!!!×100 Ͳ͔͜ͰҰཡͯ͠ݟ͍ͨʂ
107. Consul KV Dashboard github.com/fujiwara/consul-kv-dashboard Consul KVΛσʔλετΞʹͨ͠ μογϡϘʔυWebΞϓϦ
108. σʔλొ࿥ Consul HTTP APIͰ௚઀ૹΔ $ curl -X PUT -d "message" \ '127.0.0.1:8500/v1/kv/dashboard/example/myhostname?flags=1422607461000'
109. keyߏ଄ /v1/kv/dashboard/{category}/{nodename}? flags=({unixtime} * 1000 + {status}) • category: chef, serverspec, deploy... • nodename: Consulͷnode໊ • flags: unixtime * 1000 + status • status: 0=Success 1=Warning 2=Danger 3=Info
110. ը໘ͷଈ࣌ߋ৽ ϒϩοΩϯάΫΤϦΛ࢖͏ consul.io/docs/agent/http.html $ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?recurese HTTP/1.1 200 OK Content-Type: application/json X-Consul-Index: 261975 [{"CreateIndex":261891,"ModifyIndex":261975,"LockIndex":0, "Key":"dashboard/chef/myhost","Flags":1422602855000,"Value":".....
111. Blocking query Ϩεϙϯεϔομͷ X-Consul-Index Λ ࣍ͷϦΫΤετͷҾ਺ʹࢦఆ $ curl -i 127.0.0.1:8500/v1/kv/dashboard/chef/myhost HTTP/1.1 200 OK X-Consul-Index: 261975 ... $ curl 127.0.0.1:8500/v1/kv/dashboard/chef/myhost?index=261975 ৽͍͠σʔλ͕ൃੜ͢Δ·ͰϨεϙϯε͕஗Ԇ ͍ΘΏΔ Long pooling
112. Blocking queryͷ׆༻ Consul Template github.com/hashicorp/consul-template KV, node, service౳ͷঢ়ଶมԽΛଈ൓ө Template→fileߋ৽ˠcommand࣮ߦ $ consul-template \ -consul 127.0.0.1:8500 \ -template "/tmp/template.ctmpl:/var/www/nginx.conf:service nginx restart"
113. consul-kv-dashboard͕΍Δ͜ͱ(1) GoͰॻ͔ΕͨWebΞϓϦέʔγϣϯ HTML, CSS, JavaScriptΛ഑෍ • go-bindataͰ·ͱΊͨϑΝΠϧΛ http.FileServerͰ഑৴ • όΠφϦҰݸΛஔ͍ͯىಈ͢Δ͚ͩͰ ੩తϑΝΠϧ΋഑৴Ͱ͖Δ
114. consul-kv-dashboard͕΍Δ͜ͱ(2) Consul HTTP API ΁ͷ reverse proxy • /api/... → 127.0.0.1:8500/v1/kv/dashboard/... • ϨεϙϯεͷJSONΛ੔ܗ {"Flags":1422608524001} ! {"timestamp":"2015-01-30 18:02:04 +0900","status":"warning"} • /v1/catalog/nodes Λblocking queryͰ؂ࢹ ଘࡏ͢ΔnodeͷσʔλͷΈϑΟϧλ
115. consul-kv-dashboard trigger $ consul-kv-dashboard -trigger COMMAND ΧςΰϦຖʹঢ়ଶ(success→warningͳͲ)͕ มԽͨ͠ΒίϚϯυ࣮ߦՄೳ JSON͕ඪ४ೖྗʹ౉͞ΕΔ { "category":"testing", "node":"web01", "address":"192.168.1.10", "timestamp":"2015-01-21 11:22:33 +0900", "status":"danger", "key":"","data":"failure!!" }
116. ΦʔτεέʔϧͰͷ Deploy Tips
117. ىಈޙʹ࠷৽൛Λdeploy deploy࣌ɺKVʹ࠷৽ͷmanifest URLΛอଘ 1. consul join 2. stretcherىಈ 3. ࠷৽ͷmanifest URLΛKV͔Βऔಘ 4. ࣗ෼ࣗ਎ʹ event (manifest URL) Λૹ৴ 5. deploy !
118. ΦʔτεέʔϧͰͷ஫ҙ఺ ! AMIʹ࢒͍ͬͯΔݹ͍ΞϓϦ͕ىಈ " ࠷৽ͷ deploy IDͰͳ͍৔߹͸ىಈ͠ͳ͍ deploy࣌: unique ͳ ID Λൃߦ • ϑΝΠϧʹॻ͍ͯ tar ʹೖΕΔ • KV ʹ΋ೖΕΔ ىಈ࣌:ϩʔΧϧϑΝΠϧͷ ID ͱ KV Λൺֱ • ҟͳ͍ͬͯͨΒ sleep 10 && exit → restart
119. bash-completionͰ sshͷϗετ໊ิ׬
120. bash-completionͰsshͷϗετ໊ิ׬ σϑΥϧτͰ͸ ~/.ssh/known_hosts Λݩʹิ׬ ! աڈʹsshͨ͜͠ͱ͕͋Δϗετ͕ग़ͯ͘Δ " ݱࡏ͸طʹଘࡏ͠ͳ͍Մೳੑ͕͋Δ " Ұ౓΋sshͨ͜͠ͱ͕ͳ͍৽نϗετ͸ग़ͯ͜ ͳ͍
121. bash-completionͰsshͷϗετ໊ิ׬ ~/.bash_profile _known_hosts_real() { local members=$(consul members -status=alive | awk '!/Node/{printf("%s ", $1)}') COMPREPLY=( $( \ compgen -W "$members" \ ${COMP_WORDS[COMP_CWORD]} \ ) ) return 0 } ! ݱࡏaliveͳϗετͷΈ͕ग़ͯ͘Δʂ
122. ·ͱΊ
123. ·ͱΊ Consul͸ػೳ๛෋…͕ͩ ࢖͍͍ͨػೳ͚ͩ࢖͑͹Α͍ • KVɺEventɺHealth Check... DNSɺHTTPΛ࢖͍͜ͳ͢ͱߴ౓ͳࣗಈԽ͕Մೳ • consul-templateͰಈతLBઃఆ
124. Questions? • Architecture, Service Discovery Health Checking, Key/Value Store • ಺෦DNS • ຊ൪؀ڥӡ༻ / ߴՄ༻ੑ • Φʔτεέʔϧ • Stretcher • Consul KV Dashboard