Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Search
taiki45
November 28, 2018
Technology
1
2.2k
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Cookpad Tech Kitchen #20 クックパッドのマイクロサービスプラットフォーム現状
https://cookpad.connpass.com/event/106913/
taiki45
November 28, 2018
Tweet
Share
More Decks by taiki45
See All by taiki45
RustでAWS Lambda functionをいい感じに書く
taiki45
1
250
SPIFFE Meetup Tokyo #2 LT: Envoy SDS
taiki45
0
570
builderscon Tokyo 2019: Intro Service Mesh
taiki45
6
3k
NoOps Meetup Tokyo #7: 入門サービスメッシュ
taiki45
4
1.7k
CloudNative Days Tokyo 2019: Understanding Envoy
taiki45
3
3.2k
Cloud Native Meetup Tokyo #8 ServiceMesh Day Recap
taiki45
2
300
EnvoyCon 2018: Building and operating service mesh at mid-size company
taiki45
4
4.3k
Building and operating service mesh at mid-size company
taiki45
2
2.2k
Observability, Service Mesh and Microservices
taiki45
25
6.2k
Other Decks in Technology
See All in Technology
What's is Bluesky
shinoharata
0
230
RAGだけじゃない! 古くて新しいベクトル検索の世界 | DevelopersIO 2024 福岡
quiver
2
680
効果的なLLM評価法 LangSmithの技術と実践
knishioka
0
120
Productivity-Conference-GitHub-20240629
yuhattor
2
2.3k
Small_Start_Conscious_Development_Productivity_Improvement_Practices_at_freee
hikarumiyazawa
2
2.1k
Godot Engineを使った vision OSアプリ開発について
limes2018
0
140
GA technologiesの経営戦略から駆動するデータ基盤構築
pacocat
0
530
AWSアーキテクチャ図をスマートに描く方法をいろいろ試してみた
kiku3
0
180
初めてアジャイルテストに挑戦してみたら、スクラムチームが良いサイクルを回し始めた話
hacomono
PRO
1
100
IaCツールのいろいろ
takesection
0
220
爆速開発文化を支えるProduct Engineerの 開発生産性向上の取り組み
shnjtk
9
3.5k
20240626_PLATEAU_AWARD説明会
nokonoko1203
0
210
Featured
See All Featured
From Idea to $5000 a Month in 5 Months
shpigford
377
46k
Design by the Numbers
sachag
277
18k
Bash Introduction
62gerente
606
210k
jQuery: Nuts, Bolts and Bling
dougneiner
61
7.3k
Building Your Own Lightsaber
phodgson
101
5.8k
RailsConf 2023
tenderlove
12
670
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
27
2.4k
The Language of Interfaces
destraynor
151
23k
Imperfection Machines: The Place of Print at Facebook
scottboms
261
12k
Making Projects Easy
brettharned
110
5.6k
YesSQL, Process and Tooling at Scale
rocio
165
14k
Creatively Recalculating Your Daily Design Routine
revolveconf
213
11k
Transcript
クックパッドでの サービスメッシュについて Taiki Ono, Cookpad Inc.
Agenda • Background • Problems • Introducing and operations •
Key results • Next challenges
Background
Cookpad • "Make everyday cooking fun!" • Originally started in
Japan in 1997 • Operate in over 23 languages, 68 countries
Scale • 200+ product developers • 100+ production services •
90M Monthly Average User
Organization structure Service Team SRE team etc
Technology stack • Ruby on Rails for both web frontend
and backend apps • Python for ML apps • Go for backend app • Rust, Swift, Java etc.. for internal apps
None
Problems
Operational problems • Decrease in system reliability • Hard to
trouble shoot and debug ‣ Increase of time detect root causes of incidents ‣ Capacity planing
Solutions • Expeditor ‣ Ruby library inspired by Netflix's Hystrix
• aws-xray ‣ Ruby library for distributed tracing using AWS's X-Ray service https://github.com/cookpad
http://techlife.cookpad.com/entry/2017/09/06/115710
None
GoPythonRustJavaSwift apps? • Limitation of library model approach ‣ More
for product development ‣ Controlling library versions is hard • Planning to develop our proxy and mixed with consul-template
Service mesh to the rescue
at SRECON America 2017 "Lyft's Envoy: Experiences Operating a Large
Service Mesh"
Replacing libraries to proxy
control-plane
Introducing and operating service mesh
Timeline • Early 2017: making plan • Late 2017: building
MVP • Early 2018: generally available
Envoy • Publicity released at mid 2016 • Lightweight •
Graceful reloading • gRPC support https://github.com/envoyproxy/envoy
Plan: in-house • Early 2017: no Istio • We use
Amazon ECS • Not using full features of Envoy • Resiliency and observability parts
Goals • Control resiliency settings by Ops ‣ Centrally managed
‣ Review flow • All metrics should go into Prometheus • Low operation cost ‣ Less components, use of managed services
None
Configuration contents • Jsonnet • Route config ‣ Retry, timeouts
for paths, domains • Cluster config ‣ DNS name of internal ELB ‣ Circuit breaker settings https://github.com/cookpad/kumonos
Drop statsd-relay • Adding tags to metrics with DogStatsd format
• Less component is preferable ‣ Send PRs to Envoy ‣ dog_statsd sink and fixed tag configuration are available
gRPC infrastructure • Need L7 proxy for HTTP/2 traffic •
Let's extend control-plane
ServiceDiscoveryService API • lyft/discovery ‣ Reference implementation of SDS API
• Moved to cookpad/sds
The hard point of ECS • Copy current ECS service
• Wait then switch • Delete old one
Generally available
Operations
Dashboards • Prometheus • Grafana ‣ Per service ‣ Per
servie-to-service ‣ Envoy instances • Vizceral ‣ promviz, promviz-front
None
None
None
None
None
Envoy on EC2 • Build and distribute as a in-house
deb package • Manage as a systemd service • Use hot-restarter.py ‣ Generate starter script for each host role
wait-side-car • Sidecar Envoy containers need a few seconds to
be up ‣ For background jobs • Wrapper command-line tool ‣ cookpad/wait-side-car https://github.com/cookpad/wait-side-car
https://techlife.cookpad.com/entry/2018/04/02/140846
Key results
Resiliency • Eliminates temporal burst of errors from backend services
• Speed of reviewing settings and deployment • Fault isolation: not yet remarkable result
Observability • Decrease of time to detect root causes around
service communication issues • Visualization of how resilience mechanism is working • One of sources of Service Level Indicator
Continuous Growth of platform • Improve application platform without application
deployment • Increase velocity of platform development team
Next challenges
Next challenges • v2 xDS migration • More effective traffic
control • Chaos engineering platform • Distributed tracing • Auth[z, n]
Wrap up
Wrap up • Issues around service communications • Introducing service
mesh instead of doing library approach • Key results: resiliency, observability, platform improvement
Q&A • Twitter: @taiki45 • http://techlife.cookpad.com/ • EnvoyCon 2018 https://
events.linuxfoundation.org/events/ kubecon-cloudnativecon-north- america-2018/co-located-events/ envoycon/