平和なConsul Cluster運用 / consul-casual-1

Consul Casual Talks #1
http://connpass.com/event/35836/

Transcript

1. ฏ࿨ͳ Consul cluster ӡ༻ Consul Casual Talks #1@fujiwara
2. ౻ݪ ढ़Ұ࿠ @fujiwara github.com/fujiwara sfujiwara.hatenablog.com ٕज़෦
3. Game & Community
4. Agenda Consulͷ׆༻ࣄྫ ฏ࿨ʹӡ༻͢ΔͨΊͷϙΠϯτ
5. Consulͷ׆༻ࣄྫ 1. Internal DNS (node, service) 2. maintͰϝϯςφϯε 3. StretcherʹΑΔσϓϩΠɺChef࣮ߦ 4. consul-templateʹΑΔnginxͷઃఆߋ৽ 5. 1୆͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ
6. Internal DNS
7. Internal DNS (node, service) node໊ (ྫ kayac-web-i-1234567...) service໊(ྫ) • log-aggregator : Fluentdͷू໿αʔό • log-analyzer : Norikra • internal-proxy : ֎ʹग़ͯߦͨ͘ΊͷSquid • internal-mta : ֎ʹग़ͯߦͨ͘ΊͷPostfix
8. Internal DNS (node, service) dnsmasqΛશ୆Ͱىಈ .consul υϝΠϯͷ໊લղܾ͸consul agent΁ 127.0.0.1:53 Λdnsmasq͕Listen͢Δ # dnsmasq.conf server=/consul/127.0.0.1#8600 bind-interfaces listen-address=127.0.0.1
10. Internal DNS (node, service) resolv.conf Ͱ (node|service).consul ΛݕࡧυϝΠϯʹࢦఆ → node໊ɺservice໊͚ͩͰ઀ଓͰ͖Δ # /etc/resolv.conf search node.consul service.consul nameserver 127.0.0.1 # dnsmasq nameserver 172.16.0.2 # VPC resolver nameserver 172.16.0.254 # Unbound on EC2
11. bash-completionͰsshͷϗετ໊ิ׬ ~/.bash_profile _known_hosts_real() { local members=$(consul members -status=alive | awk '!/Node/{printf("%s ", $1)}') COMPREPLY=( $( \ compgen -W "$members" \ ${COMP_WORDS[COMP_CWORD]} \ ) ) return 0 } ੜ͖͍ͯΔϗετͷΈิ׬ީิʹͳΔ http://qiita.com/sfujiwara/items/f4fa907ead53ed104e1a
12. Fluentdͷू໿αʔό΁ૹΔઃఆ ConsulͰఏڙ͢ΔDNS໊΁ϥ΢ϯυϩϏϯͰૹ৴ <match **> type forward expire_dns_cache 15 dns_round_robin true heartbeat_type tcp <server> host log-aggregator.service.consul </server> </match> ૹ৴ઌͷྻڍෆཁɺࣗಈ੾Γ཭͠
13. maintͰϝϯςφϯε
14. consul maint consul maint -enable [-reason "..."] ͋ΔnodeΛϝϯςφϯεϞʔυʹ͢Δ → serviceͷ໊લղܾ͔Β͸֎ΕΔ → nodeͷ໊લղܾ͸Ͱ͖Δ (sshͱ͔)
15. consul maint࢖༻ྫ Fluentdू໿αʔόΛϝϯςφϯε͍ͨ͠৔߹ʹmaint → DNS͔Β֎ΕΔͷͰૹ৴͕ࢭ·Δ (expire_dns_cache ͷઃఆ͕ඞཁ) NorikraͷΠϯελϯεΛऔΓସ͍͑ͨͱ͖ʹ • ৽͍͠ϗετΛ maint -enable ͰηοτΞοϓ • چͰ maint -enable, ৽Ͱ maint -disable • DNSͰೖΕସΘΔͷͰૹ৴ઌ͕੾ΓସΘΔ
16. PackerͰ AMI ࡞੒࣌ʹ maint 1. consul cluster ʹ join 2. maint -enable (ߏஙதʹ૊Έࠐ·Εͳ͍Α͏ʹ) 3. ChefͰߏங 4. maintঢ়ଶͷ·· AMI ࡞੒ 5. AMI͔Βىಈͨ͠Πϯελϯε΋maintͷ·· 6. ىಈޙͷॾʑ͕ऴΘͬͨΒ maint -disable → αʔϏεΠϯ
17. maintͳΒىಈ͠ͳ͍ daemontools ͷ run script #!/bin/bash maint=$(consul maint) if [[ $maint != "" ]]; then echo "$maint" sleep 10 exit 1 fi exec ... ϝϯς࣌ʹىಈͯ͠ཉ͘͠ͳ͍daemonΛ੍ޚ (maint -enableʹͳͬͯ΋stopͨ͠Γ͸͠ͳ͍)
18. StretcherʹΑΔσϓϩΠ
19. StretcherʹΑΔσϓϩΠ github.com/fujiwara/stretcher Consul / Serf ͱ࿈ܞͯ͠ಈ͘σϓϩΠπʔϧ
21. StretcherͰChef࣮ߦ Chef-Server → Stretcher + Chef-Solo • Chef-Serverr͕ SPOF / ϘτϧωοΫʹͳΒͳ͍ • શ୆ʹಉ͡tar, eventΛ഑෍ˠద༻͢ΔjsonΛ֤ϊʔυͰܾఆ # /etc/sysconfig/hostname-prefix HOSTNAME_PREFIX="xxx-app" → nodes/xxx-app.json Λద༻
22. ChefͷroleݕࡧΛserviceఆٛͰ /etc/consul.d/role.json { "service": { "name": "role", "tags": [ "batch-server", "db-client", ... ] } } Serviceͱͯ͠ఆٛͯ͠ݕࡧՄೳʹ http://localhost:8500/v1/catalog/service/role? tag=db-client
23. http://localhost:8500/v1/catalog/service/role? tag=internal-proxy [ { "Node": "xxx-i-10bf0fe2", "Address": "10.0.0.123", "ServiceID": "role", "ServiceName": "role", ... }, { "Node": "xxx-i-3c1b72b3", "Address": "10.0.1.234", "ServiceID": "role", "ServiceName": "role", ... } ]
24. Daemontools؅ཧԼͷdaemon΋serviceఆٛ { "service": { "name": "daemontools", "tags": [ "app", stretcher", "gunfish", ... ] } }
25. ͋Δdaemontools؅ཧϓϩηεΛ࠶ىಈ͍ͨ͠ curl http://localhost:8500/v1/catalog/service/ daemontools?tag=gunfish | jq -r ".[].Node" xxx-admin-i-0391d6162be552655 xxx-app-i-01a7ff42f4796be4f xxx-app-i-05bd652734828b522 xxx-batch-i-0095ac858fe87d8e5 Regexp::TrieͰ࠷దͳਖ਼نදݱʹͯ͠ consul exec consul exec -node '(?:xxx\-(?:a(?:dmin|pp)|batch))' "svc -h /service/gunfish"
26. consul-templateʹΑΔnginxͷઃఆߋ৽
27. consul-template https://github.com/hashicorp/consul-template • Consul KVͷ஋ɺServiceͷղܾ݁ՌͳͲΛτϦΨʹ • ςϯϓϨʔτߋ৽ɺ೚ҙscript kick͕Ͱ͖Δ
28. nginxͷઃఆߋ৽ # config.hcl template { source = "/etc/nginx/spam.ip.conf.ctmpl" destination = "/etc/nginx/spam.ip.conf" command = "service nginx reload" perms = 0644 backup = true } # spam.ip.conf.ctmpl {{key "spam_ips"}} localhost:8500/v1/kv/spam_ips ʹPUT͢Δ͚ͩͰઃఆߋ৽
29. 1୆͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ
30. 1୆͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ WebSocketड৴Ͱಈ͘Slack bot→ 2୆Ҏ্Ͱಈ͘ͱ෼਎͢Δ Ͱ΋Մ༻ੑΛ͍࣋ͨͤͨ…
31. 1୆͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ consul lock Λ࢖͏ ϩοΫΛऔಘͰ͖ͨΒࢦఆͨ͠ίϚϯυ͕࣮ߦ͞ΕΔwrapper consul lock -n 1 nuko "/path/to/run-nuko.sh" Consul leader͕ೖΕସΘΔͱϩοΫ͕ղ์͞ΕΔͷͰ஫ҙ
32. ฏ࿨ʹӡ༻͢ΔͨΊͷϙΠϯτ
33. ฏ࿨ʹӡ༻͢ΔͨΊͷϙΠϯτ RaftΛ(େ·͔ʹͰ΋͍͍ͷͰ)஌͓ͬͯ͘ http://thesecretlivesofdata.com/raft/ ෼ࢄ؀ڥͰͷ߹ҙܗ੒ΞϧΰϦζϜ • Ϧʔμʔબग़ʹʮա൒਺ʯͷ߹ҙ͕ඞཁ • 2୆ = 1୆མͪΔͱա൒਺(=2)͕औΕͳ͍ • 3୆ = 1୆མͪͯ΋ա൒਺(=2)͕औΕΔ • 4୆ = 2୆མͪΔͱա൒਺(=3)͕औΕͳ͍
34. Deployment Table ຊ൪Ͱ͸࠷௿3୆, 3 or 5͕ਪ঑ consul.io/docs/internals/consensus.html
35. ServerʹඞཁͳϦιʔε • CPU: 2CPUͰे෼ • Memory: 20MBʙ • Disk: 2MBʙ Memory, Disk͸KVͷར༻ঢ়گ࣍ୈ KV dump JSON 10MB, data_dir/raft 120MB → consul agent RSS 250MB
36. Serverʹ͸ઐ༻ϗετ͕ඞཁʁ consul agentࣗମ͸ͦΕ΄ͲϦιʔεΛ࢖༻͠ͳ͍ Disk IO͕ߴෛՙͳ৔߹ʹRaftͷHeartbeat͕ࣦഊ͠΍͍͢ • Timeout 500ms • Heartbeatʹࣦഊ͢ΔͱLeaderબग़͕ߦΘΕΔ • ௨ৗ2,3ඵͰબग़͸׬ྃ͢Δ • Consul server ͸ tmpfs Λอଘઌʹͯ͠Disk IOͷӨڹճආ
37. ߴՄ༻ੑͷͨΊʹ Server୆਺ʹΑΓಉ࣌ʹো֐Λىͯ͜͠ ΋໰୊ͳ͍node਺͕มΘΔ • 3 node → 1 • 5 node → 2 3 nodeߏ੒࣌ɺ2୆མͪͯ࢒Γ1୆ʹͳ ͬͯ͠·͏ͱLeader͕બग़Ͱ͖ͳ͍ ௕࣌ؒ੾Γ཭͢ϝϯςφϯε࣌ʹ͸Ұ࣌ తʹServer nodeΛ૿΍͢ख΋
38. ߴՄ༻ੑͷͨΊʹ Server nodeͷfailover͸ࣗಈ ! Ϣʔβ͸ৗʹlocalhostͷagent͚ͩΛΈ͍ͯΕ͹Α͍
39. nodeো֐࣌ͷӨڹ ! LeaderͰ͸ͳ͍ → " ଞnodeʹ͸Өڹͳ͠ ! Leader → " Leader࠶બग़ σϑΥϧτͰ͸͢΂ͯͷಡΈॻ͖ΛLeader͕ॲཧ (ڧҰ؏ੑ) Leader͕ܾ·Δ·ͰΞΫηεෆೳ (DNS, HTTP)
40. Stale mode (DNS) Leader࠶બग़͸௨ৗ2ʙ3ඵͰ׬ྃ ͦͷؒ΋DNSͰNode, Service໊ղܾΛ͍ͨ͠ʁ → Stale mode : Leaderະબग़Ͱ΋Ԡ౴Մೳ "dns_config":{ "allow_stale": true, // default false "max_stale": "10s" // default 5s } ݁Ռ͸ݹ͍Մೳੑ͕͋Δ(݁Ռ੔߹ੑ)
41. DNS TTL default͸TTL 0 → cache͞Εͳ͍ node, serviceผʹTTLΛઃఆՄೳ DNS cache(ͨͱ͑͹dnsmasq)Λલஈʹ഑ஔͯ͠cacheͰ͖Δ "dns_config":{ "node_ttl": "60s", "service_ttl": { "*": "15s" } }
42. Stale mode (HTTP API) HTTP APIͰstale modeʹ͢Δ৔߹͸Ҿ਺ stale $ curl "http://127.0.0.1:8500/v1/kv/web/key1?stale" staleҾ਺ͳ͠ͰLeaderબग़தʹΞΫηε → 500 Internal Server Error
43. ӡ༻தͷUpgrade consul.io/docs/upgrading.html consul.io/docs/upgrade-specific.html όʔδϣϯผʹ஫ҙ఺͕͋ΔͷͰυΩϡϝϯτΛ ॱ൪ʹAgentΛೖΕସ͑Δ͜ͱͰ Rolling upgradeՄೳ (Leader nodeೖΕସ͑Ͱ࠶બग़͸ى͖Δ)
44. ҆ఆੑ v0.2࣌୅͔Β2೥Ҏ্ӡ༻ Agentϓϩηε͕མͪͨ͜ͱ͸1ճ͚ͩ(0.4.1࣌୅) EBS(gp2)ͷΫϨδοτރׇ → IO waitେྔ → panic: Timeout starting MDB transaction ΦϖϛεͰServerΛམͱ͗͢͠Δͱճ෮ෆೳ
45. KVͷόοΫΞοϓ ͋Δ֊૚ͷԼͷ஋Λ࠶ؼతʹऔΓ͍ͨ৔߹͸ recurse $ curl -s "http://127.0.0.1:8500/v1/kv/?recurse" [ {"CreateIndex":112,"ModifyIndex":115,"LockIndex":0, "Key":"key1","Flags":123,"Value":"dGVzdA=="}, {"CreateIndex":122,"ModifyIndex":122,"LockIndex":0, "Key":"key2","Flags":0,"Value":"dGVzdDI="}, {"CreateIndex":124,"ModifyIndex":124,"LockIndex":0, "Key":"test/1","Flags":0,"Value":"dGVzdDM="} ] Key, Flags, ValueΛPUT͠ͳ͓͠ͰϨετΞͰ͖Δ
46. ࣮ࡍʹLeader͕ೖΕସΘͬͨͱ͖ͷϩά 2016/07/30 10:07:28 [WARN] raft: Heartbeat timeout reached, starting election 2016/07/30 10:07:28 [INFO] raft: Node at 10.0.2.132:8300 [Candidate] entering Candidate state 2016/07/30 10:07:30 [WARN] raft: Election timeout reached, restarting election 2016/07/30 10:07:30 [INFO] raft: Node at 10.0.2.132:8300 [Candidate] entering Candidate state 2016/07/30 10:07:30 [INFO] raft: Election won. Tally: 3 2016/07/30 10:07:30 [INFO] raft: Node at 10.0.2.132:8300 [Leader] entering Leader state 2016/07/30 10:07:30 [INFO] consul: cluster leadership acquired 2016/07/30 10:07:30 [INFO] consul: New leader elected: xxx-consul-i-ff26ca5a 2ඵఔ౓Ͱճ෮ DNSͷcache / stale mode ͰαʔϏεӨڹͳ͠ stale໌ࣔ͠ͳ͍HTTP API͸500ʹͳΔˠৗ࣌ୟ͖·͘Δͷ͸…?
47. ࣮ࡍʹ͋ͬͨා͍࿩
48. consul exec Ͱେྔ݁Ռऔಘ consul exec "cat /var/log/foo.log" | grep ... ֤ϗετͷϩάΛconsul execͰऔಘ͠Α͏ͱͨ͠ → consul exec ͸KVʹҰ୴อଘ͢ΔͷͰϝϞϦ/DBංେԽ serverΛ1୆ͣͭ࠶ىಈͯ͠ճ෮
49. ΦϖϛεͰΫϥελ่յ upgrade͔ͨͬͨ͠ 3୆ߏ੒ͷserverͷ1୆Λམͱͯ͠ɺ৽͍͠όΠφϦͰىಈͨ͠ (ͭ΋Γͩͬͨ) ͪΌΜͱىಈ͍ͯ͠ͳ͍ͷʹ2୆໨ͷαʔόΛམͱͨ͠ → ่յ
50. ่յͨ͠ΒͲ͏͢Ε͹ 1. མͪண͘ 2. serverΛશ෦ࢭΊΔ 3. σʔλ(data_dir)΋શ෦ফ͢ 4. serverΛ -bootstrap-expect N Ͱىಈ • start_join ·ͨ͸ खಈͰ join 5. (ඞཁͳΒ) KVΛόοΫΞοϓ͔Β໭͢