Upgrade to Pro — share decks privately, control downloads, hide ads and more …

平和なConsul Cluster運用 / consul-casual-1

平和なConsul Cluster運用 / consul-casual-1

Consul Casual Talks #1
http://connpass.com/event/35836/

Avatar for FUJIWARA Shunichiro

FUJIWARA Shunichiro

August 01, 2016
Tweet

More Decks by FUJIWARA Shunichiro

Other Decks in Technology

Transcript

  1. Consulͷ׆༻ࣄྫ 1. Internal DNS (node, service) 2. maintͰϝϯςφϯε 3. StretcherʹΑΔσϓϩΠɺChef࣮ߦ

    4. consul-templateʹΑΔnginxͷઃఆߋ৽ 5. 1୆͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ
  2. Internal DNS (node, service) node໊ (ྫ kayac-web-i-1234567...) service໊(ྫ) • log-aggregator

    : Fluentdͷू໿αʔό • log-analyzer : Norikra • internal-proxy : ֎ʹग़ͯߦͨ͘ΊͷSquid • internal-mta : ֎ʹग़ͯߦͨ͘ΊͷPostfix
  3. Internal DNS (node, service) dnsmasqΛશ୆Ͱىಈ .consul υϝΠϯͷ໊લղܾ͸consul agent΁ 127.0.0.1:53 Λdnsmasq͕Listen͢Δ

    # dnsmasq.conf server=/consul/127.0.0.1#8600 bind-interfaces listen-address=127.0.0.1
  4. Internal DNS (node, service) resolv.conf Ͱ (node|service).consul ΛݕࡧυϝΠϯʹࢦఆ → node໊ɺservice໊͚ͩͰ઀ଓͰ͖Δ

    # /etc/resolv.conf search node.consul service.consul nameserver 127.0.0.1 # dnsmasq nameserver 172.16.0.2 # VPC resolver nameserver 172.16.0.254 # Unbound on EC2
  5. bash-completionͰsshͷϗετ໊ิ׬ ~/.bash_profile _known_hosts_real() { local members=$(consul members -status=alive | awk

    '!/Node/{printf("%s ", $1)}') COMPREPLY=( $( \ compgen -W "$members" \ ${COMP_WORDS[COMP_CWORD]} \ ) ) return 0 } ੜ͖͍ͯΔϗετͷΈิ׬ީิʹͳΔ http://qiita.com/sfujiwara/items/f4fa907ead53ed104e1a
  6. Fluentdͷू໿αʔό΁ૹΔઃఆ ConsulͰఏڙ͢ΔDNS໊΁ϥ΢ϯυϩϏϯͰૹ৴ <match **> type forward expire_dns_cache 15 dns_round_robin true

    heartbeat_type tcp <server> host log-aggregator.service.consul </server> </match> ૹ৴ઌͷྻڍෆཁɺࣗಈ੾Γ཭͠
  7. PackerͰ AMI ࡞੒࣌ʹ maint 1. consul cluster ʹ join 2.

    maint -enable (ߏஙதʹ૊Έࠐ·Εͳ͍Α͏ʹ) 3. ChefͰߏங 4. maintঢ়ଶͷ·· AMI ࡞੒ 5. AMI͔Βىಈͨ͠Πϯελϯε΋maintͷ·· 6. ىಈޙͷॾʑ͕ऴΘͬͨΒ maint -disable → αʔϏεΠϯ
  8. maintͳΒىಈ͠ͳ͍ daemontools ͷ run script #!/bin/bash maint=$(consul maint) if [[

    $maint != "" ]]; then echo "$maint" sleep 10 exit 1 fi exec ... ϝϯς࣌ʹىಈͯ͠ཉ͘͠ͳ͍daemonΛ੍ޚ (maint -enableʹͳͬͯ΋stopͨ͠Γ͸͠ͳ͍)
  9. StretcherͰChef࣮ߦ Chef-Server → Stretcher + Chef-Solo • Chef-Serverr͕ SPOF /

    ϘτϧωοΫʹͳΒͳ͍ • શ୆ʹಉ͡tar, eventΛ഑෍ˠద༻͢ΔjsonΛ֤ϊʔυͰܾఆ # /etc/sysconfig/hostname-prefix HOSTNAME_PREFIX="xxx-app" → nodes/xxx-app.json Λద༻
  10. ChefͷroleݕࡧΛserviceఆٛͰ /etc/consul.d/role.json { "service": { "name": "role", "tags": [ "batch-server",

    "db-client", ... ] } } Serviceͱͯ͠ఆٛͯ͠ݕࡧՄೳʹ http://localhost:8500/v1/catalog/service/role? tag=db-client
  11. http://localhost:8500/v1/catalog/service/role? tag=internal-proxy [ { "Node": "xxx-i-10bf0fe2", "Address": "10.0.0.123", "ServiceID": "role",

    "ServiceName": "role", ... }, { "Node": "xxx-i-3c1b72b3", "Address": "10.0.1.234", "ServiceID": "role", "ServiceName": "role", ... } ]
  12. ͋Δdaemontools؅ཧϓϩηεΛ࠶ىಈ͍ͨ͠ curl http://localhost:8500/v1/catalog/service/ daemontools?tag=gunfish | jq -r ".[].Node" xxx-admin-i-0391d6162be552655 xxx-app-i-01a7ff42f4796be4f

    xxx-app-i-05bd652734828b522 xxx-batch-i-0095ac858fe87d8e5 Regexp::TrieͰ࠷దͳਖ਼نදݱʹͯ͠ consul exec consul exec -node '(?:xxx\-(?:a(?:dmin|pp)|batch))' "svc -h /service/gunfish"
  13. nginxͷઃఆߋ৽ # config.hcl template { source = "/etc/nginx/spam.ip.conf.ctmpl" destination =

    "/etc/nginx/spam.ip.conf" command = "service nginx reload" perms = 0644 backup = true } # spam.ip.conf.ctmpl {{key "spam_ips"}} localhost:8500/v1/kv/spam_ips ʹPUT͢Δ͚ͩͰઃఆߋ৽
  14. ServerʹඞཁͳϦιʔε • CPU: 2CPUͰे෼ • Memory: 20MBʙ • Disk: 2MBʙ

    Memory, Disk͸KVͷར༻ঢ়گ࣍ୈ KV dump JSON 10MB, data_dir/raft 120MB → consul agent RSS 250MB
  15. ߴՄ༻ੑͷͨΊʹ Server୆਺ʹΑΓಉ࣌ʹো֐Λىͯ͜͠ ΋໰୊ͳ͍node਺͕มΘΔ • 3 node → 1 • 5

    node → 2 3 nodeߏ੒࣌ɺ2୆མͪͯ࢒Γ1୆ʹͳ ͬͯ͠·͏ͱLeader͕બग़Ͱ͖ͳ͍ ௕࣌ؒ੾Γ཭͢ϝϯςφϯε࣌ʹ͸Ұ࣌ తʹServer nodeΛ૿΍͢ख΋
  16. nodeো֐࣌ͷӨڹ ! LeaderͰ͸ͳ͍ → " ଞnodeʹ͸Өڹͳ͠ ! Leader → "

    Leader࠶બग़ σϑΥϧτͰ͸͢΂ͯͷಡΈॻ͖ΛLeader͕ॲཧ (ڧҰ؏ੑ) Leader͕ܾ·Δ·ͰΞΫηεෆೳ (DNS, HTTP)
  17. Stale mode (DNS) Leader࠶બग़͸௨ৗ2ʙ3ඵͰ׬ྃ ͦͷؒ΋DNSͰNode, Service໊ղܾΛ͍ͨ͠ʁ → Stale mode :

    Leaderະબग़Ͱ΋Ԡ౴Մೳ "dns_config":{ "allow_stale": true, // default false "max_stale": "10s" // default 5s } ݁Ռ͸ݹ͍Մೳੑ͕͋Δ(݁Ռ੔߹ੑ)
  18. Stale mode (HTTP API) HTTP APIͰstale modeʹ͢Δ৔߹͸Ҿ਺ stale $ curl

    "http://127.0.0.1:8500/v1/kv/web/key1?stale" staleҾ਺ͳ͠ͰLeaderબग़தʹΞΫηε → 500 Internal Server Error
  19. KVͷόοΫΞοϓ ͋Δ֊૚ͷԼͷ஋Λ࠶ؼతʹऔΓ͍ͨ৔߹͸ recurse $ curl -s "http://127.0.0.1:8500/v1/kv/?recurse" [ {"CreateIndex":112,"ModifyIndex":115,"LockIndex":0, "Key":"key1","Flags":123,"Value":"dGVzdA=="},

    {"CreateIndex":122,"ModifyIndex":122,"LockIndex":0, "Key":"key2","Flags":0,"Value":"dGVzdDI="}, {"CreateIndex":124,"ModifyIndex":124,"LockIndex":0, "Key":"test/1","Flags":0,"Value":"dGVzdDM="} ] Key, Flags, ValueΛPUT͠ͳ͓͠ͰϨετΞͰ͖Δ
  20. ࣮ࡍʹLeader͕ೖΕସΘͬͨͱ͖ͷϩά 2016/07/30 10:07:28 [WARN] raft: Heartbeat timeout reached, starting election

    2016/07/30 10:07:28 [INFO] raft: Node at 10.0.2.132:8300 [Candidate] entering Candidate state 2016/07/30 10:07:30 [WARN] raft: Election timeout reached, restarting election 2016/07/30 10:07:30 [INFO] raft: Node at 10.0.2.132:8300 [Candidate] entering Candidate state 2016/07/30 10:07:30 [INFO] raft: Election won. Tally: 3 2016/07/30 10:07:30 [INFO] raft: Node at 10.0.2.132:8300 [Leader] entering Leader state 2016/07/30 10:07:30 [INFO] consul: cluster leadership acquired 2016/07/30 10:07:30 [INFO] consul: New leader elected: xxx-consul-i-ff26ca5a 2ඵఔ౓Ͱճ෮ DNSͷcache / stale mode ͰαʔϏεӨڹͳ͠ stale໌ࣔ͠ͳ͍HTTP API͸500ʹͳΔˠৗ࣌ୟ͖·͘Δͷ͸…?
  21. consul exec Ͱେྔ݁Ռऔಘ consul exec "cat /var/log/foo.log" | grep ...

    ֤ϗετͷϩάΛconsul execͰऔಘ͠Α͏ͱͨ͠ → consul exec ͸KVʹҰ୴อଘ͢ΔͷͰϝϞϦ/DBංେԽ serverΛ1୆ͣͭ࠶ىಈͯ͠ճ෮
  22. ่յͨ͠ΒͲ͏͢Ε͹ 1. མͪண͘ 2. serverΛશ෦ࢭΊΔ 3. σʔλ(data_dir)΋શ෦ফ͢ 4. serverΛ -bootstrap-expect

    N Ͱىಈ • start_join ·ͨ͸ खಈͰ join 5. (ඞཁͳΒ) KVΛόοΫΞοϓ͔Β໭͢