Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Search
taiki45
November 28, 2018
Technology
1
2.4k
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Cookpad Tech Kitchen #20 クックパッドのマイクロサービスプラットフォーム現状
https://cookpad.connpass.com/event/106913/
taiki45
November 28, 2018
Tweet
Share
More Decks by taiki45
See All by taiki45
Mocking in Rust Applications
taiki45
2
590
Error Handling in Rust Applications
taiki45
3
620
Efficient Platform for Security and Compliance
taiki45
4
1.4k
RustでAWS Lambda functionをいい感じに書く
taiki45
2
650
SPIFFE Meetup Tokyo #2 LT: Envoy SDS
taiki45
0
750
builderscon Tokyo 2019: Intro Service Mesh
taiki45
6
3.4k
NoOps Meetup Tokyo #7: 入門サービスメッシュ
taiki45
4
1.9k
CloudNative Days Tokyo 2019: Understanding Envoy
taiki45
3
3.5k
Cloud Native Meetup Tokyo #8 ServiceMesh Day Recap
taiki45
2
380
Other Decks in Technology
See All in Technology
新卒3年目の後悔〜機械学習モデルジョブの運用を頑張った話〜
kameitomohiro
0
300
脅威をモデリングしてMCPのセキュリティ対策を考えよう
flatt_security
4
1.7k
マルチテナント+マルチプロダクト SaaS への AI Agent の組み込み方
kworkdev
PRO
2
350
キャディでのApache Iceberg, Trino採用事例 -Apache Iceberg and Trino Usecase in CADDi--
caddi_eng
0
150
Grafana MCP serverでなんかし隊 / Try Grafana MCP server
kohbis
0
360
AI技術トレンド勉強会 #1MCPの基礎と実務での応用
nisei_k
1
210
Kotlinで学ぶ 代数的データ型
ysknsid25
5
1.1k
IAMのマニアックな話 2025を執筆して、 見えてきたAWSアカウント管理の現在
nrinetcom
PRO
4
580
「伝える」を加速させるCursor術
naomix
0
640
Agentic DevOps時代の生存戦略
kkamegawa
0
410
本部長の代わりに提案書レビュー! KDDI営業が毎日使うAIエージェント「A-BOSS」開発秘話
minorun365
PRO
14
1.8k
OCI Oracle Database Services新機能アップデート(2025/03-2025/05)
oracle4engineer
PRO
1
150
Featured
See All Featured
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
26
2.8k
Why Our Code Smells
bkeepers
PRO
337
57k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
228
22k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.3k
Why You Should Never Use an ORM
jnunemaker
PRO
56
9.4k
Code Review Best Practice
trishagee
68
18k
Designing for humans not robots
tammielis
253
25k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
650
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
The Cost Of JavaScript in 2023
addyosmani
50
8.4k
Transcript
クックパッドでの サービスメッシュについて Taiki Ono, Cookpad Inc.
Agenda • Background • Problems • Introducing and operations •
Key results • Next challenges
Background
Cookpad • "Make everyday cooking fun!" • Originally started in
Japan in 1997 • Operate in over 23 languages, 68 countries
Scale • 200+ product developers • 100+ production services •
90M Monthly Average User
Organization structure Service Team SRE team etc
Technology stack • Ruby on Rails for both web frontend
and backend apps • Python for ML apps • Go for backend app • Rust, Swift, Java etc.. for internal apps
None
Problems
Operational problems • Decrease in system reliability • Hard to
trouble shoot and debug ‣ Increase of time detect root causes of incidents ‣ Capacity planing
Solutions • Expeditor ‣ Ruby library inspired by Netflix's Hystrix
• aws-xray ‣ Ruby library for distributed tracing using AWS's X-Ray service https://github.com/cookpad
http://techlife.cookpad.com/entry/2017/09/06/115710
None
GoPythonRustJavaSwift apps? • Limitation of library model approach ‣ More
for product development ‣ Controlling library versions is hard • Planning to develop our proxy and mixed with consul-template
Service mesh to the rescue
at SRECON America 2017 "Lyft's Envoy: Experiences Operating a Large
Service Mesh"
Replacing libraries to proxy
control-plane
Introducing and operating service mesh
Timeline • Early 2017: making plan • Late 2017: building
MVP • Early 2018: generally available
Envoy • Publicity released at mid 2016 • Lightweight •
Graceful reloading • gRPC support https://github.com/envoyproxy/envoy
Plan: in-house • Early 2017: no Istio • We use
Amazon ECS • Not using full features of Envoy • Resiliency and observability parts
Goals • Control resiliency settings by Ops ‣ Centrally managed
‣ Review flow • All metrics should go into Prometheus • Low operation cost ‣ Less components, use of managed services
None
Configuration contents • Jsonnet • Route config ‣ Retry, timeouts
for paths, domains • Cluster config ‣ DNS name of internal ELB ‣ Circuit breaker settings https://github.com/cookpad/kumonos
Drop statsd-relay • Adding tags to metrics with DogStatsd format
• Less component is preferable ‣ Send PRs to Envoy ‣ dog_statsd sink and fixed tag configuration are available
gRPC infrastructure • Need L7 proxy for HTTP/2 traffic •
Let's extend control-plane
ServiceDiscoveryService API • lyft/discovery ‣ Reference implementation of SDS API
• Moved to cookpad/sds
The hard point of ECS • Copy current ECS service
• Wait then switch • Delete old one
Generally available
Operations
Dashboards • Prometheus • Grafana ‣ Per service ‣ Per
servie-to-service ‣ Envoy instances • Vizceral ‣ promviz, promviz-front
None
None
None
None
None
Envoy on EC2 • Build and distribute as a in-house
deb package • Manage as a systemd service • Use hot-restarter.py ‣ Generate starter script for each host role
wait-side-car • Sidecar Envoy containers need a few seconds to
be up ‣ For background jobs • Wrapper command-line tool ‣ cookpad/wait-side-car https://github.com/cookpad/wait-side-car
https://techlife.cookpad.com/entry/2018/04/02/140846
Key results
Resiliency • Eliminates temporal burst of errors from backend services
• Speed of reviewing settings and deployment • Fault isolation: not yet remarkable result
Observability • Decrease of time to detect root causes around
service communication issues • Visualization of how resilience mechanism is working • One of sources of Service Level Indicator
Continuous Growth of platform • Improve application platform without application
deployment • Increase velocity of platform development team
Next challenges
Next challenges • v2 xDS migration • More effective traffic
control • Chaos engineering platform • Distributed tracing • Auth[z, n]
Wrap up
Wrap up • Issues around service communications • Introducing service
mesh instead of doing library approach • Key results: resiliency, observability, platform improvement
Q&A • Twitter: @taiki45 • http://techlife.cookpad.com/ • EnvoyCon 2018 https://
events.linuxfoundation.org/events/ kubecon-cloudnativecon-north- america-2018/co-located-events/ envoycon/