Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
サービスのパフォーマンス数値と 依存関係を用いたサービス同士の 協調スケール構想 / Web ...
Search
itkq
December 23, 2017
0
770
サービスのパフォーマンス数値と 依存関係を用いたサービス同士の 協調スケール構想 / Web System Architecture #1
Web System Architecture 研究会 #1
itkq
December 23, 2017
Tweet
Share
More Decks by itkq
See All by itkq
アクセス制御にまつわる改善 / Improving access control
itkq
0
770
SRE Lounge #13 Service Level at Ubie
itkq
3
2.3k
Cookpad summer internship 2019 - infra intro
itkq
1
9.8k
Where Chaos Engineering comes from, and what's next
itkq
6
7.6k
Re:silience から始めるカオスエンジニアリング生活 / A Life of Chaos Engineering Starting with Resilience
itkq
12
3.8k
Featured
See All Featured
For a Future-Friendly Web
brad_frost
175
9.4k
The Cult of Friendly URLs
andyhume
78
6k
Raft: Consensus for Rubyists
vanstee
136
6.6k
Thoughts on Productivity
jonyablonski
67
4.3k
We Have a Design System, Now What?
morganepeng
50
7.2k
Speed Design
sergeychernyshev
25
620
Automating Front-end Workflow
addyosmani
1366
200k
Producing Creativity
orderedlist
PRO
341
39k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
329
21k
Into the Great Unknown - MozCon
thekraken
32
1.5k
Docker and Python
trallard
40
3.1k
Practical Orchestrator
shlominoach
186
10k
Transcript
αʔϏεͷύϑΥʔϚϯεͱ ґଘؔΛ༻͍ͨαʔϏεಉ࢜ͷ ڠௐεέʔϧߏ Web System Architecture ݚڀձ #1 @itkq
Me 2 @itkq Takuya Kosugiyama ౦େ ใ௨৴ܥ M2 όΠτ (SRE)
͍ͨ͜ (͖Ύʔ)
ࢀՃత • Web System ͷՄೳੑɺݶքɺֶज़తՁʹ ڵຯ͕͋Δ • ༷ʑͳਓʑͱ͍ٞͨ͠ • ए͍͏ͪ
[ཁग़య] ʹੈքΛ͍͛ͨ 3
ࢀՃత • Web System ͷՄೳੑɺݶքɺֶज़తՁʹ ڵຯ͕͋Δ • ༷ʑͳਓʑͱ͍ٞͨ͠ • ए͍͏ͪ
[ཁग़య] ʹੈքΛ͍͛ͨ 3 • ϦΞϧ id:y_uuki ͞ΜΛݟʹ͖ͨ
ൃද༰ • ࠷ۙͷ Web ΞʔΩςΫνϟͷಈͱ SRE ຊΛ ಡΜͰߟ͑ͨ͜ͱ • ίϯςφϕʔεͷαʔϏεϝογϡΛ׆༻ͨ͠
ӡ༻্ͷεέʔϧղܾͷͨΊͷ͕͔Γ 4
എܠɿΞϓϦέʔγϣϯίϯςφ • ίϯςφܕԾԽ • খΦʔόʔϔου • Container as a Service
• Production ready • Docker, Kubernetes • Managed services (GKE, ECS, EKS) 5
എܠɿαʔϏεࢦΞʔΩςΫνϟ • ϞϊϦεͷେنԽͱݶք • อकੑɺ։ൃޮɺ… • αʔϏεࢦΞʔΩςΫνϟ • ػೳΛαʔϏεͱͯ͠Γग़͢ •
αʔϏεಉ͕࢜࿈ܞ 6 Q. αʔϏεςΟεΧόϦ? ϦτϥΠ? λΠϜΞτ?
എܠɿαʔϏεϝογϡ • service-to-service ௨৴ͷࡍͰϓϩΩγΛհ͢Δ • αʔϏεϝογϡɿ L7 ϓϩΩγʹΑΔωοτϫʔΫͷநϨΠϠʔ 7 Service
A Proxy Service B Proxy Controller Data Plane Control Plane
എܠɿαʔϏεϝογϡͷ༻్ • Envoy, Linkerd • Advanced load balancing • Circuit
breaking • Rate limiting • Service discovery • Observation • Statistics • Logging • Tracing 8 ։ൃج൫ͷྖҬʹۙ͘ɺӡ༻ͷԠ༻ߟ͑Δ༨͋Γ
ίϯςφɾϚΠΫϩαʔϏεͷ࣮ӡ༻՝ 1. ϗετΩϟύγςΟϓϥϯχϯά • ίϯςφͷਫฏεέʔϧ༰қ • ͔͠͠ϗετΩϟύγςΟඞཁ 2. ίϯςφϦιʔεͷܾఆ๏ •
“desired count” ୭͕Ͳ͏ܾΊΔͷ͔ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ • ґଘઌαʔϏεͷΩϟύγςΟߟྀ͢Δඞཁ 9 ͍ͣΕΩϟύγςΟͷͱͯ͠ूͰ͖Δ
ΩϟύγςΟϓϥϯχϯάͷ͔͏͖࢟ • Site Reliability Engineering 18.2 10 ΠϯςϯτϕʔεͷΩϟύγςΟඪ Ϧιʔε੍ɾྉۚ࠷খԽͷ࠷దԽ ґଘؔɾύϑΥʔϚϯεϝτϦΫεͷѲ
ΩϟύγςΟϓϥϯχϯάͷ͔͏͖࢟ • Site Reliability Engineering 18.2 10 ΠϯςϯτϕʔεͷΩϟύγςΟඪ Ϧιʔε੍ɾྉۚ࠷খԽͷ࠷దԽ ґଘؔɾύϑΥʔϚϯεϝτϦΫεͷѲ
࠶ܝɿ࣮ӡ༻՝ 11 1. ϗετΩϟύγςΟϓϥϯχϯά 2. ίϯςφϦιʔεͷܾఆ๏ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ
࠶ܝɿ࣮ӡ༻՝ 11 1. ϗετΩϟύγςΟϓϥϯχϯά 2. ίϯςφϦιʔεͷܾఆ๏ 3. αʔϏεಉ͕࢜ڠௐͨ͠εέʔϧ جૅݕ౼ɿ •
ύϑΥʔϚϯεఆྔԽ
՝Πϝʔδɿఆঢ়گ 12 A B C user facing internal × 2
× 1 × 1 D System × 1
՝Πϝʔδ 1 13 A B C user facing internal ×
2 × 1 × 1 D System × 1 desired count: 6 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ desired count: 3
՝Πϝʔδ 1 13 A B C user facing internal ×
2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ × 3 desired count: 3 C ͷߟྀ࿙Ε
՝Πϝʔδ 1 13 A B C user facing internal ×
2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ C × 3 desired count: 3 C ͷߟྀ࿙Ε
՝Πϝʔδ 1 13 A B C user facing internal ×
2 × 1 × 1 D System × 1 desired count: 6 × 6 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ C A × 3 desired count: 3 C ͷߟྀ࿙Ε
՝Πϝʔδ 2 14 A B C user facing internal ×
2 × 1 × 1 D System × 1 desired count: 6 desired count: 3 desired count: 3 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ
՝Πϝʔδ 2 14 A B C user facing internal ×
2 × 1 × 1 D System × 1 desired count: 6 × 6 × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε
՝Πϝʔδ 2 14 A B C user facing internal ×
2 × 1 × 1 D System × 1 desired count: 6 × 6 B × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε
՝Πϝʔδ 2 14 A B C user facing internal ×
2 × 1 × 1 D System × 1 desired count: 6 × 6 B A × 3 × 3 desired count: 3 desired count: 3 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ D ͷߟྀ࿙Ε
ղܾΠϝʔδɿఆঢ়گ 15 A 100 rps/container A B C user facing
internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7
ղܾΠϝʔδ 16 A 100 rps/container A B C user facing
internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ
ղܾΠϝʔδ 16 A 100 rps/container A B C user facing
internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ × 6
ղܾΠϝʔδ 16 A 100 rps/container A B C user facing
internal × 2 × 1 × 1 D System × 1 B 200 rps/container C 100 rps/container D 100 rps/container 0.4 0.2 0.7 ex. ϐʔΫ࣌ A ͷෛՙ 3 ഒ (100 * 6 * 0.4 + 100 * 0.7) / 200 ~ 2 (100 * 6 * 0.2) / 100 ~ 2 × 6 × 2 × 2 ύϑΥʔϚϯε + ґଘ → ඞཁ࠷ݶͷίϯςφ
ɿύϑΥʔϚϯεఆྔԽͱґଘ • ఆྔԽʹؔͯ͠ • ͲͷΑ͏ʹଌఆ͢Δ͔ • ϝτϦΫε rps ͰΑ͍͔ •
ͲͷΤϯυϙΠϯτʹର͢Δ rps ͔ • ϦΫΤετύλʔϯ prod ͱಉҰ͔ • αʔϏεͷґଘͲ͏ܾ·Δ͔ 17 ύϑΥʔϚϯεࣗಈଌఆɾఆྔԽͷ࣮ྫ͕গͳ͍
ɿύϑΥʔϚϯεఆྔԽͱґଘ • ఆྔԽʹؔͯ͠ • ͲͷΑ͏ʹଌఆ͢Δ͔ • ϝτϦΫε rps ͰΑ͍͔ •
ͲͷΤϯυϙΠϯτʹର͢Δ rps ͔ • ϦΫΤετύλʔϯ prod ͱಉҰ͔ • αʔϏεͷґଘͲ͏ܾ·Δ͔ 17 ύϑΥʔϚϯεࣗಈଌఆɾఆྔԽͷ࣮ྫ͕গͳ͍ αʔϏεϝογϡ + shadowing
αʔϏεϝογϡߏྫ 18 front-proxy Request B proxy A proxy stats discovery
• connection / request count • 1xx, 2xx, … 5xx count • etc. per proxy metrics:
shadowing 19 front-proxy Request B proxy A proxy Prod Shadow
# routes { "virtual_hosts": [ { "name": "service_a", "domains": [ "*" ], "routes": [ { "prefix": "/", "cluster": "service_a", "shadow": { "cluster": "service_a_prime" } ... A’ B’ strage front->A: GET / front->A: GET /users A->B: GET /awesome_process record log shadow shadow
ύϑΥʔϚϯεଌఆҊ • ೖྗ • ϦΫΤετύεॏΈ d • ಉ࣌ଓ c •
ϨεϙϯελΠϜඪ r • ग़ྗɿϝτϦΫε • d ʹैͬͨ WRR, c ฒྻͰϦΫΤετ 20 RTT ≦ r Ҏ͔ͭ 5xx Ҏ֎ͷϦΫΤετ ୯Ґ࣌ؒ rps = GET / 0.3 GET /users 0.2 POST /users 0.2 GET /depends_b 0.1 …
·ͱΊ • ϚΠΫϩαʔϏεʹ͏αʔϏεϝογϡͷଘࡏ • ࣮ӡ༻ͷԠ༻͕ݕ౼༨͋Γ • ࠷ߴͷΩϟύγςΟϓϥϯχϯάͷ͕͔Γ ͱͯ͠ɺύϑΥʔϚϯεఆྔԽʹண • αʔϏεϝογϡͰϦΫΤετύεॏΈ
Λࢉग़͠ϝτϦΫεΛܭࢉ • ύϑΥʔϚϯε࣮ଌɺґଘࠓޙͷ՝ 21