Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
平和なConsul Cluster運用 / consul-casual-1
Search
FUJIWARA Shunichiro
August 01, 2016
Technology
17
2.8k
平和なConsul Cluster運用 / consul-casual-1
Consul Casual Talks #1
http://connpass.com/event/35836/
FUJIWARA Shunichiro
August 01, 2016
Tweet
Share
More Decks by FUJIWARA Shunichiro
See All by FUJIWARA Shunichiro
alecthomas/kong はいいぞ
fujiwara3
6
1.8k
ecspressoの設計思想に至る道 / sekkeinight2025
fujiwara3
12
2.9k
さくらのIaaS基盤のモニタリングとOpenTelemetry/OSC Hokkaido 2025
fujiwara3
3
1.2k
監視のこれまでとこれから/sakura monitoring seminar 2025
fujiwara3
11
5.3k
k6による負荷試験 入門から日常的な実践まで/Re:TechTalk #01
fujiwara3
2
160
困難を「一般解」で解く
fujiwara3
10
3.8k
「隙間家具OSS」に至る道/Fujiwara Tech Conference 2025
fujiwara3
7
13k
alecthomas/kong はいいぞ / kamakura.go#7
fujiwara3
1
1.2k
ISUCONに強くなるかもしれない日々の過ごしかた/Findy ISUCON 2024-11-14
fujiwara3
11
1.4k
Other Decks in Technology
See All in Technology
GC25 Recap+: Advancing Go Garbage Collection with Green Tea
logica0419
1
360
インサイト情報からどこまで自動化できるか試してみた
takas0522
0
140
Tomorrow graphlib, Let us use everybody
hayaosuzuki
0
160
Why React!?? Next.jsそしてReactを改めてイチから選ぶ
ypresto
10
4.3k
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
11
77k
PLaMo2シリーズのvLLM実装 / PFN LLM セミナー
pfn
PRO
2
940
[2025-09-30] Databricks Genie を利用した分析基盤とデータモデリングの IVRy の現在地
wxyzzz
0
450
Function calling機能をPLaMo2に実装するには / PFN LLMセミナー
pfn
PRO
0
860
From Prompt to Product @ How to Web 2025, Bucharest, Romania
janwerner
0
110
Flaky Testへの現実解をGoのプロポーザルから考える | Go Conference 2025
upamune
1
400
関係性が駆動するアジャイル──GPTに人格を与えたら、対話を通してふりかえりを習慣化できた話
mhlyc
0
130
Windows で省エネ
murachiakira
0
160
Featured
See All Featured
Faster Mobile Websites
deanohume
310
31k
Bash Introduction
62gerente
615
210k
Building an army of robots
kneath
306
46k
Side Projects
sachag
455
43k
Producing Creativity
orderedlist
PRO
347
40k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Become a Pro
speakerdeck
PRO
29
5.5k
Practical Orchestrator
shlominoach
190
11k
Six Lessons from altMBA
skipperchong
28
4k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
Thoughts on Productivity
jonyablonski
70
4.9k
Transcript
ฏͳ Consul cluster ӡ༻ Consul Casual Talks #1@fujiwara
౻ݪ ढ़Ұ @fujiwara github.com/fujiwara sfujiwara.hatenablog.com ٕज़෦
Game & Community
Agenda Consulͷ׆༻ࣄྫ ฏʹӡ༻͢ΔͨΊͷϙΠϯτ
Consulͷ׆༻ࣄྫ 1. Internal DNS (node, service) 2. maintͰϝϯςφϯε 3. StretcherʹΑΔσϓϩΠɺChef࣮ߦ
4. consul-templateʹΑΔnginxͷઃఆߋ৽ 5. 1͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ
Internal DNS
Internal DNS (node, service) node໊ (ྫ kayac-web-i-1234567...) service໊(ྫ) • log-aggregator
: Fluentdͷूαʔό • log-analyzer : Norikra • internal-proxy : ֎ʹग़ͯߦͨ͘ΊͷSquid • internal-mta : ֎ʹग़ͯߦͨ͘ΊͷPostfix
Internal DNS (node, service) dnsmasqΛશͰىಈ .consul υϝΠϯͷ໊લղܾconsul agent 127.0.0.1:53 Λdnsmasq͕Listen͢Δ
# dnsmasq.conf server=/consul/127.0.0.1#8600 bind-interfaces listen-address=127.0.0.1
None
Internal DNS (node, service) resolv.conf Ͱ (node|service).consul ΛݕࡧυϝΠϯʹࢦఆ → node໊ɺservice໊͚ͩͰଓͰ͖Δ
# /etc/resolv.conf search node.consul service.consul nameserver 127.0.0.1 # dnsmasq nameserver 172.16.0.2 # VPC resolver nameserver 172.16.0.254 # Unbound on EC2
bash-completionͰsshͷϗετ໊ิ ~/.bash_profile _known_hosts_real() { local members=$(consul members -status=alive | awk
'!/Node/{printf("%s ", $1)}') COMPREPLY=( $( \ compgen -W "$members" \ ${COMP_WORDS[COMP_CWORD]} \ ) ) return 0 } ੜ͖͍ͯΔϗετͷΈิީิʹͳΔ http://qiita.com/sfujiwara/items/f4fa907ead53ed104e1a
FluentdͷूαʔόૹΔઃఆ ConsulͰఏڙ͢ΔDNS໊ϥϯυϩϏϯͰૹ৴ <match **> type forward expire_dns_cache 15 dns_round_robin true
heartbeat_type tcp <server> host log-aggregator.service.consul </server> </match> ૹ৴ઌͷྻڍෆཁɺࣗಈΓ͠
maintͰϝϯςφϯε
consul maint consul maint -enable [-reason "..."] ͋ΔnodeΛϝϯςφϯεϞʔυʹ͢Δ → serviceͷ໊લղܾ͔Β֎ΕΔ
→ nodeͷ໊લղܾͰ͖Δ (sshͱ͔)
consul maint༻ྫ FluentdूαʔόΛϝϯςφϯε͍ͨ͠߹ʹmaint → DNS͔Β֎ΕΔͷͰૹ৴͕ࢭ·Δ (expire_dns_cache ͷઃఆ͕ඞཁ) NorikraͷΠϯελϯεΛऔΓସ͍͑ͨͱ͖ʹ • ৽͍͠ϗετΛ
maint -enable ͰηοτΞοϓ • چͰ maint -enable, ৽Ͱ maint -disable • DNSͰೖΕସΘΔͷͰૹ৴ઌ͕ΓସΘΔ
PackerͰ AMI ࡞࣌ʹ maint 1. consul cluster ʹ join 2.
maint -enable (ߏஙதʹΈࠐ·Εͳ͍Α͏ʹ) 3. ChefͰߏங 4. maintঢ়ଶͷ·· AMI ࡞ 5. AMI͔Βىಈͨ͠Πϯελϯεmaintͷ·· 6. ىಈޙͷॾʑ͕ऴΘͬͨΒ maint -disable → αʔϏεΠϯ
maintͳΒىಈ͠ͳ͍ daemontools ͷ run script #!/bin/bash maint=$(consul maint) if [[
$maint != "" ]]; then echo "$maint" sleep 10 exit 1 fi exec ... ϝϯς࣌ʹىಈͯ͠ཉ͘͠ͳ͍daemonΛ੍ޚ (maint -enableʹͳͬͯstopͨ͠Γ͠ͳ͍)
StretcherʹΑΔσϓϩΠ
StretcherʹΑΔσϓϩΠ github.com/fujiwara/stretcher Consul / Serf ͱ࿈ܞͯ͠ಈ͘σϓϩΠπʔϧ
None
StretcherͰChef࣮ߦ Chef-Server → Stretcher + Chef-Solo • Chef-Serverr͕ SPOF /
ϘτϧωοΫʹͳΒͳ͍ • શʹಉ͡tar, eventΛˠద༻͢ΔjsonΛ֤ϊʔυͰܾఆ # /etc/sysconfig/hostname-prefix HOSTNAME_PREFIX="xxx-app" → nodes/xxx-app.json Λద༻
ChefͷroleݕࡧΛserviceఆٛͰ /etc/consul.d/role.json { "service": { "name": "role", "tags": [ "batch-server",
"db-client", ... ] } } Serviceͱͯ͠ఆٛͯ͠ݕࡧՄೳʹ http://localhost:8500/v1/catalog/service/role? tag=db-client
http://localhost:8500/v1/catalog/service/role? tag=internal-proxy [ { "Node": "xxx-i-10bf0fe2", "Address": "10.0.0.123", "ServiceID": "role",
"ServiceName": "role", ... }, { "Node": "xxx-i-3c1b72b3", "Address": "10.0.1.234", "ServiceID": "role", "ServiceName": "role", ... } ]
DaemontoolsཧԼͷdaemonserviceఆٛ { "service": { "name": "daemontools", "tags": [ "app", stretcher",
"gunfish", ... ] } }
͋ΔdaemontoolsཧϓϩηεΛ࠶ىಈ͍ͨ͠ curl http://localhost:8500/v1/catalog/service/ daemontools?tag=gunfish | jq -r ".[].Node" xxx-admin-i-0391d6162be552655 xxx-app-i-01a7ff42f4796be4f
xxx-app-i-05bd652734828b522 xxx-batch-i-0095ac858fe87d8e5 Regexp::TrieͰ࠷దͳਖ਼نදݱʹͯ͠ consul exec consul exec -node '(?:xxx\-(?:a(?:dmin|pp)|batch))' "svc -h /service/gunfish"
consul-templateʹΑΔnginxͷઃఆߋ৽
consul-template https://github.com/hashicorp/consul-template • Consul KVͷɺServiceͷղܾ݁ՌͳͲΛτϦΨʹ • ςϯϓϨʔτߋ৽ɺҙscript kick͕Ͱ͖Δ
nginxͷઃఆߋ৽ # config.hcl template { source = "/etc/nginx/spam.ip.conf.ctmpl" destination =
"/etc/nginx/spam.ip.conf" command = "service nginx reload" perms = 0644 backup = true } # spam.ip.conf.ctmpl {{key "spam_ips"}} localhost:8500/v1/kv/spam_ips ʹPUT͢Δ͚ͩͰઃఆߋ৽
1͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ
1͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ WebSocketड৴Ͱಈ͘Slack bot→ 2Ҏ্Ͱಈ͘ͱ͢Δ ͰՄ༻ੑΛ͍࣋ͨͤͨ…
1͔͠ಈ͔ͨ͘͠ͳ͍daemonͷഉଞىಈ consul lock Λ͏ ϩοΫΛऔಘͰ͖ͨΒࢦఆͨ͠ίϚϯυ͕࣮ߦ͞ΕΔwrapper consul lock -n 1 nuko
"/path/to/run-nuko.sh" Consul leader͕ೖΕସΘΔͱϩοΫ͕ղ์͞ΕΔͷͰҙ
ฏʹӡ༻͢ΔͨΊͷϙΠϯτ
ฏʹӡ༻͢ΔͨΊͷϙΠϯτ RaftΛ(େ·͔ʹͰ͍͍ͷͰ)͓ͬͯ͘ http://thesecretlivesofdata.com/raft/ ࢄڥͰͷ߹ҙܗΞϧΰϦζϜ • Ϧʔμʔબग़ʹʮաʯͷ߹ҙ͕ඞཁ • 2 = 1མͪΔͱա(=2)͕औΕͳ͍
• 3 = 1མͪͯա(=2)͕औΕΔ • 4 = 2མͪΔͱա(=3)͕औΕͳ͍
Deployment Table ຊ൪Ͱ࠷3, 3 or 5͕ਪ consul.io/docs/internals/consensus.html
ServerʹඞཁͳϦιʔε • CPU: 2CPUͰे • Memory: 20MBʙ • Disk: 2MBʙ
Memory, DiskKVͷར༻ঢ়گ࣍ୈ KV dump JSON 10MB, data_dir/raft 120MB → consul agent RSS 250MB
Serverʹઐ༻ϗετ͕ඞཁʁ consul agentࣗମͦΕ΄ͲϦιʔεΛ༻͠ͳ͍ Disk IO͕ߴෛՙͳ߹ʹRaftͷHeartbeat͕ࣦഊ͍͢͠ • Timeout 500ms • Heartbeatʹࣦഊ͢ΔͱLeaderબग़͕ߦΘΕΔ
• ௨ৗ2,3ඵͰબग़ྃ͢Δ • Consul server tmpfs Λอଘઌʹͯ͠Disk IOͷӨڹճආ
ߴՄ༻ੑͷͨΊʹ ServerʹΑΓಉ࣌ʹোΛىͯ͜͠ ͳ͍node͕มΘΔ • 3 node → 1 • 5
node → 2 3 nodeߏ࣌ɺ2མͪͯΓ1ʹͳ ͬͯ͠·͏ͱLeader͕બग़Ͱ͖ͳ͍ ࣌ؒΓ͢ϝϯςφϯε࣌ʹҰ࣌ తʹServer nodeΛ૿͢ख
ߴՄ༻ੑͷͨΊʹ Server nodeͷfailoverࣗಈ ! Ϣʔβৗʹlocalhostͷagent͚ͩΛΈ͍ͯΕΑ͍
nodeো࣌ͷӨڹ ! LeaderͰͳ͍ → " ଞnodeʹӨڹͳ͠ ! Leader → "
Leader࠶બग़ σϑΥϧτͰͯ͢ͷಡΈॻ͖ΛLeader͕ॲཧ (ڧҰ؏ੑ) Leader͕ܾ·Δ·ͰΞΫηεෆೳ (DNS, HTTP)
Stale mode (DNS) Leader࠶બग़௨ৗ2ʙ3ඵͰྃ ͦͷؒDNSͰNode, Service໊ղܾΛ͍ͨ͠ʁ → Stale mode :
Leaderະબग़ͰԠՄೳ "dns_config":{ "allow_stale": true, // default false "max_stale": "10s" // default 5s } ݁Ռݹ͍Մೳੑ͕͋Δ(݁Ռ߹ੑ)
DNS TTL defaultTTL 0 → cache͞Εͳ͍ node, serviceผʹTTLΛઃఆՄೳ DNS cache(ͨͱ͑dnsmasq)Λલஈʹஔͯ͠cacheͰ͖Δ
"dns_config":{ "node_ttl": "60s", "service_ttl": { "*": "15s" } }
Stale mode (HTTP API) HTTP APIͰstale modeʹ͢Δ߹Ҿ stale $ curl
"http://127.0.0.1:8500/v1/kv/web/key1?stale" staleҾͳ͠ͰLeaderબग़தʹΞΫηε → 500 Internal Server Error
ӡ༻தͷUpgrade consul.io/docs/upgrading.html consul.io/docs/upgrade-specific.html όʔδϣϯผʹҙ͕͋ΔͷͰυΩϡϝϯτΛ ॱ൪ʹAgentΛೖΕସ͑Δ͜ͱͰ Rolling upgradeՄೳ (Leader nodeೖΕସ͑Ͱ࠶બग़ى͖Δ)
҆ఆੑ v0.2͔࣌Β2Ҏ্ӡ༻ Agentϓϩηε͕མͪͨ͜ͱ1ճ͚ͩ(0.4.1࣌) EBS(gp2)ͷΫϨδοτރׇ → IO waitେྔ → panic: Timeout
starting MDB transaction ΦϖϛεͰServerΛམͱ͗͢͠Δͱճ෮ෆೳ
KVͷόοΫΞοϓ ͋Δ֊ͷԼͷΛ࠶ؼతʹऔΓ͍ͨ߹ recurse $ curl -s "http://127.0.0.1:8500/v1/kv/?recurse" [ {"CreateIndex":112,"ModifyIndex":115,"LockIndex":0, "Key":"key1","Flags":123,"Value":"dGVzdA=="},
{"CreateIndex":122,"ModifyIndex":122,"LockIndex":0, "Key":"key2","Flags":0,"Value":"dGVzdDI="}, {"CreateIndex":124,"ModifyIndex":124,"LockIndex":0, "Key":"test/1","Flags":0,"Value":"dGVzdDM="} ] Key, Flags, ValueΛPUT͠ͳ͓͠ͰϨετΞͰ͖Δ
࣮ࡍʹLeader͕ೖΕସΘͬͨͱ͖ͷϩά 2016/07/30 10:07:28 [WARN] raft: Heartbeat timeout reached, starting election
2016/07/30 10:07:28 [INFO] raft: Node at 10.0.2.132:8300 [Candidate] entering Candidate state 2016/07/30 10:07:30 [WARN] raft: Election timeout reached, restarting election 2016/07/30 10:07:30 [INFO] raft: Node at 10.0.2.132:8300 [Candidate] entering Candidate state 2016/07/30 10:07:30 [INFO] raft: Election won. Tally: 3 2016/07/30 10:07:30 [INFO] raft: Node at 10.0.2.132:8300 [Leader] entering Leader state 2016/07/30 10:07:30 [INFO] consul: cluster leadership acquired 2016/07/30 10:07:30 [INFO] consul: New leader elected: xxx-consul-i-ff26ca5a 2ඵఔͰճ෮ DNSͷcache / stale mode ͰαʔϏεӨڹͳ͠ stale໌ࣔ͠ͳ͍HTTP API500ʹͳΔˠৗ࣌ୟ͖·͘Δͷ…?
࣮ࡍʹ͋ͬͨා͍
consul exec Ͱେྔ݁Ռऔಘ consul exec "cat /var/log/foo.log" | grep ...
֤ϗετͷϩάΛconsul execͰऔಘ͠Α͏ͱͨ͠ → consul exec KVʹҰ୴อଘ͢ΔͷͰϝϞϦ/DBංେԽ serverΛ1ͣͭ࠶ىಈͯ͠ճ෮
ΦϖϛεͰΫϥελ่յ upgrade͔ͨͬͨ͠ 3ߏͷserverͷ1Λམͱͯ͠ɺ৽͍͠όΠφϦͰىಈͨ͠ (ͭΓͩͬͨ) ͪΌΜͱىಈ͍ͯ͠ͳ͍ͷʹ2ͷαʔόΛམͱͨ͠ → ่յ
่յͨ͠ΒͲ͏͢Ε 1. མͪண͘ 2. serverΛશ෦ࢭΊΔ 3. σʔλ(data_dir)શ෦ফ͢ 4. serverΛ -bootstrap-expect
N Ͱىಈ • start_join ·ͨ खಈͰ join 5. (ඞཁͳΒ) KVΛόοΫΞοϓ͔Β͢