Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Search
taiki45
November 28, 2018
Technology
1
2.4k
Cookpad Tech Kitchen #20 クックパッドでのサービスメッシュについて
Cookpad Tech Kitchen #20 クックパッドのマイクロサービスプラットフォーム現状
https://cookpad.connpass.com/event/106913/
taiki45
November 28, 2018
Tweet
Share
More Decks by taiki45
See All by taiki45
Mocking in Rust Applications
taiki45
2
560
Error Handling in Rust Applications
taiki45
3
560
Efficient Platform for Security and Compliance
taiki45
4
1.3k
RustでAWS Lambda functionをいい感じに書く
taiki45
2
610
SPIFFE Meetup Tokyo #2 LT: Envoy SDS
taiki45
0
720
builderscon Tokyo 2019: Intro Service Mesh
taiki45
6
3.3k
NoOps Meetup Tokyo #7: 入門サービスメッシュ
taiki45
4
1.8k
CloudNative Days Tokyo 2019: Understanding Envoy
taiki45
3
3.4k
Cloud Native Meetup Tokyo #8 ServiceMesh Day Recap
taiki45
2
370
Other Decks in Technology
See All in Technology
より良い開発者体験を実現するために~開発初心者が感じた生成AIの可能性~
masakiokuda
0
150
Рекомендации с нуля: как мы в Lamoda превратили главную страницу в ключевую точку входа для персонализированного шоппинга. Данил Комаров, Data Scientist, Lamoda Tech
lamodatech
0
710
低レイヤを知りたいPHPerのためのCコンパイラ作成入門 / Building a C Compiler for PHPers Who Want to Dive into Low-Level Programming
tomzoh
1
230
Goの組織でバックエンドTypeScriptを採用してどうだったか / How was adopting backend TypeScript in a Golang company
kaminashi
6
5.3k
The Tale of Leo: Brave Lion and Curious Little Bug
canalun
1
120
JPOUG Tech Talk #12 UNDO Tablespace Reintroduction
nori_shinoda
1
140
AIで進化するソフトウェアテスト:mablの最新生成AI機能でQAを加速!
mfunaki
0
140
LLM as プロダクト開発のパワードスーツ
layerx
PRO
1
240
AWSのマルチアカウント管理 ベストプラクティス最新版 2025 / Multi-Account management on AWS best practice 2025
ohmura
4
280
Стильный код: натуральный поиск редких атрибутов по картинке. Юлия Антохина, Data Scientist, Lamoda Tech
lamodatech
0
700
AIコーディングの最前線 〜活用のコツと課題〜
pharma_x_tech
1
920
フロントエンドも盛り上げたい!フロントエンドCBとAmplifyの軌跡
mkdev10
2
270
Featured
See All Featured
The World Runs on Bad Software
bkeepers
PRO
67
11k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5.3k
RailsConf 2023
tenderlove
30
1.1k
Reflections from 52 weeks, 52 projects
jeffersonlam
349
20k
[RailsConf 2023] Rails as a piece of cake
palkan
54
5.4k
GraphQLの誤解/rethinking-graphql
sonatard
71
10k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.1k
Building an army of robots
kneath
304
45k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
5
520
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
47
2.5k
Adopting Sorbet at Scale
ufuk
76
9.3k
Code Review Best Practice
trishagee
67
18k
Transcript
クックパッドでの サービスメッシュについて Taiki Ono, Cookpad Inc.
Agenda • Background • Problems • Introducing and operations •
Key results • Next challenges
Background
Cookpad • "Make everyday cooking fun!" • Originally started in
Japan in 1997 • Operate in over 23 languages, 68 countries
Scale • 200+ product developers • 100+ production services •
90M Monthly Average User
Organization structure Service Team SRE team etc
Technology stack • Ruby on Rails for both web frontend
and backend apps • Python for ML apps • Go for backend app • Rust, Swift, Java etc.. for internal apps
None
Problems
Operational problems • Decrease in system reliability • Hard to
trouble shoot and debug ‣ Increase of time detect root causes of incidents ‣ Capacity planing
Solutions • Expeditor ‣ Ruby library inspired by Netflix's Hystrix
• aws-xray ‣ Ruby library for distributed tracing using AWS's X-Ray service https://github.com/cookpad
http://techlife.cookpad.com/entry/2017/09/06/115710
None
GoPythonRustJavaSwift apps? • Limitation of library model approach ‣ More
for product development ‣ Controlling library versions is hard • Planning to develop our proxy and mixed with consul-template
Service mesh to the rescue
at SRECON America 2017 "Lyft's Envoy: Experiences Operating a Large
Service Mesh"
Replacing libraries to proxy
control-plane
Introducing and operating service mesh
Timeline • Early 2017: making plan • Late 2017: building
MVP • Early 2018: generally available
Envoy • Publicity released at mid 2016 • Lightweight •
Graceful reloading • gRPC support https://github.com/envoyproxy/envoy
Plan: in-house • Early 2017: no Istio • We use
Amazon ECS • Not using full features of Envoy • Resiliency and observability parts
Goals • Control resiliency settings by Ops ‣ Centrally managed
‣ Review flow • All metrics should go into Prometheus • Low operation cost ‣ Less components, use of managed services
None
Configuration contents • Jsonnet • Route config ‣ Retry, timeouts
for paths, domains • Cluster config ‣ DNS name of internal ELB ‣ Circuit breaker settings https://github.com/cookpad/kumonos
Drop statsd-relay • Adding tags to metrics with DogStatsd format
• Less component is preferable ‣ Send PRs to Envoy ‣ dog_statsd sink and fixed tag configuration are available
gRPC infrastructure • Need L7 proxy for HTTP/2 traffic •
Let's extend control-plane
ServiceDiscoveryService API • lyft/discovery ‣ Reference implementation of SDS API
• Moved to cookpad/sds
The hard point of ECS • Copy current ECS service
• Wait then switch • Delete old one
Generally available
Operations
Dashboards • Prometheus • Grafana ‣ Per service ‣ Per
servie-to-service ‣ Envoy instances • Vizceral ‣ promviz, promviz-front
None
None
None
None
None
Envoy on EC2 • Build and distribute as a in-house
deb package • Manage as a systemd service • Use hot-restarter.py ‣ Generate starter script for each host role
wait-side-car • Sidecar Envoy containers need a few seconds to
be up ‣ For background jobs • Wrapper command-line tool ‣ cookpad/wait-side-car https://github.com/cookpad/wait-side-car
https://techlife.cookpad.com/entry/2018/04/02/140846
Key results
Resiliency • Eliminates temporal burst of errors from backend services
• Speed of reviewing settings and deployment • Fault isolation: not yet remarkable result
Observability • Decrease of time to detect root causes around
service communication issues • Visualization of how resilience mechanism is working • One of sources of Service Level Indicator
Continuous Growth of platform • Improve application platform without application
deployment • Increase velocity of platform development team
Next challenges
Next challenges • v2 xDS migration • More effective traffic
control • Chaos engineering platform • Distributed tracing • Auth[z, n]
Wrap up
Wrap up • Issues around service communications • Introducing service
mesh instead of doing library approach • Key results: resiliency, observability, platform improvement
Q&A • Twitter: @taiki45 • http://techlife.cookpad.com/ • EnvoyCon 2018 https://
events.linuxfoundation.org/events/ kubecon-cloudnativecon-north- america-2018/co-located-events/ envoycon/