Upgrade to Pro — share decks privately, control downloads, hide ads and more …

EnvoyCon 2018: Building and operating service mesh at mid-size company

December 11, 2018

EnvoyCon 2018: Building and operating service mesh at mid-size company

At EnvoyCon 2018 as a co-located event of KubeCon + CloudNativeCon North America 2018.



December 11, 2018

More Decks by taiki45

Other Decks in Technology


  1. Our business • "Make everyday cooking fun!" • Originally started

    in Japan in 1997 • Operate in over 23 languages, 68 countries • World largest recipe sharing site: cookpad.com
  2. Our scale and organization structure • 90M monthly average user

    • ~200 product developers • ~100 production services • 3 platform team members ◦ 1 for service mesh dev Each product team owns their products but all of operations are still owned by central SRE team Product team SRE team
  3. Technology stack • Ruby on Rails for both web frontend

    and backend apps • Python for ML apps • Go, Rust for backend apps • Java for new apps • Other languages for internal apps
  4. Operational problems • Decrease in system reliability • Hard to

    troubleshoot and debug distributed services ◦ Increase of time detect root causes of incidents ◦ Capacity planing
  5. Library approach solutions • github.com/cookpad/expeditor ◦ Ruby library inspired by

    Netflix's Hystrix ◦ Parallel executions, timeouts, retries, cirbuit breakers • github.com/cookpad/aws-xray ◦ Ruby library for distributed tracing using AWS's X-Ray service
  6. GoPythonJava apps? • Limitation of library model approach ◦ Save

    resources for product development • Controlling library versions is hard in a large organization • Planning to develop our proxy and mixed with consul-template
  7. Timeline • Early 2017: making plan • Late 2017: building

    MVP • Early 2018: generally available
  8. In-house control-plane • Early 2017: no Istio • We are

    using Amazon ECS • Not to use full features of Envoy ◦ Resiliency and observability parts only • Small start with in-house control-plane, but planned to migrate to future managed services.
  9. Considerations • Everyone can view and manage resiliency settings ◦

    Centrally managed ◦ GitOps with code reviews • All metrics should go into Prometheus • Low operation cost ◦ Less components, use of managed services
  10. Our service mesh components • kumonos (github.com/cookpad/kumonos) ◦ v1 xDS

    response generator • sds (github.com/cookpad/sds) ◦ Fork of github.com/lyft/discovery to allow multiple service instances on the same IP address ◦ Implements v2 EDS API • itacho (github.com/cookpad/itacho) ◦ v2 xDS response generator (CLI tool) ◦ v2 xDS REST HTTP POST-GET translation server ▪ GitHub#4526 “REST xDS API with HTTP GET requests”
  11. v1 ELB based with v1 xDS app envoy internal- ELB

    HTTP/1.1 app envoy nginx app nginx app S3 v1 CDS/RDS kumonos HTTP/1.1 ECS Task ECS Task Product devloper frontend service A backend service B
  12. Configuration file • Single Jsonnet file represents single service configuration

    ◦ 1 service - N upstream dependencies • Route config ◦ Retry, timeouts for paths, domains ◦ Auto retry with GET,HEAD routes • Cluster config ◦ DNS name of internal ELB ◦ Port, TLS, connect timeout ◦ Circuit breaker settings local circuit_breaker = import 'circuit_breaker.libsonnet'; local routes = import 'routes.libsonnet'; { version: 1, dependencies: [ { name: "user", cluster_name: "user-development", lb: "user-service.example.com:80", tls: false, connect_timeout_ms: 250, circuit_breaker: circuit_breaker.default, routes: [routes.default], }, ], }
  13. v1.1 with v1 SDS for backend gRPC apps app envoy

    HTTP/2 app envoy envoy app envoy app S3 v1 CDS/RDS sds registrator registrator v1 SDS Health checking DDB registration frontend service A backend service B
  14. v2 with v2 xDS app envoy HTTP/2 app envoy envoy

    app envoy app v2 CDS/RDS (POST) sds registrator registrator v2 EDS (POST) Health checking DDB registration S3 itacho server v2 CDS/RDS (GET) itacho CLI
  15. Sending metrics app envoy app envoy statsd_exporter prometheus dog_statsd sink

    dog_statsd sink DogStatsD prometheus format EC2 instance discover with EC2 SD ECS task stats_config.stats_tags - tag_name: service-cluster fixed_value: serviceA - tag_name: service-node fixed_value: serviceA
  16. Dashboards • Grafana ◦ Per service (1 downstream - N

    upstreams) ◦ Per service-to-service (1 downstream - 1 upstream) ◦ Envoy instances • Netflix’s Vizceral ◦ github.com/nghialv/promviz ◦ github.com/mjhd-devlion/promviz-front
  17. Integrate with in-house platform console • Show service dependencies •

    Link to service mesh config file • Show SDS/EDS registration • Link to Grafana dashboards ◦ Per service dashboard ◦ Service-to-service dashboard
  18. Envoy on EC2 instances • Build and distribute as a

    in-house deb package ◦ Setup instances with configuration management tool like Chef: github.com/itamae-kitchen/itamae • Manage Envoy process as a systemd service • Using hot-restarter.py ◦ Generate starter script for each host role
  19. Wait initial xDS fetching • Sidecar Envoy containers need a

    few seconds to be up ◦ Background jobs are service-in quickly ◦ ECS does not have an API to wait the initializing phase • Wrapper command-line tool ◦ github.com/cookpad/wait-side-car ◦ Wait until an upstream health check succeed • Probably move to GitHub#4405
  20. The hard points • Limitation of ECS and its API

    ◦ Without ELB integration, we need to manage lots of things on deployments. ◦ We needed AWS Cloud Map (actually we made almost the same thing in our environment).
  21. Observability • Both SRE and product team have confidence in

    what’s happened in service-to-service communication area ◦ Visualization of how resiliency mechanism is working ◦ Decrease of time to detect root causes around service communication issues • Necessary to encourage collaboration between multiple product teams
  22. Failure recovery • Be able to configure proper resiliency setting

    values with fine-grained metrics • Eliminates temporal burst of errors from backend services • Fault isolation: not yet remarkable result
  23. Continuous development of app platform • Improve application platform without

    product application deployment • Increase velocity of platform development team
  24. Next challenges • Fault injection platform • Distributed tracing •

    Auth{z, n} • More flexibility on traffic control ◦ Envoy on edge proxies? • Migration to managed services