Velocity 2017: Lyft's Envoy: Experiences Operating a Large Service Mesh

1. Lyft's Envoy: Experiences Operating a Large Service Mesh Velocity Matt Klein / @mattklein123, Software Engineer @Lyft
2. Lyft ~4 years ago PHP / Apache monolith MongoDB Internet Clients AWS ELB Simple! No SoA! (but still not that simple)
3. Lyft ~2 years ago PHP / Apache monolith (+haproxy/nsq) MongoDB Internet Clients AWS external ELB DynamoDB AWS internal ELBs Python services Not simple! SoA! With monolith! (and some haproxy/nsq)
4. State of SoA networking in industry • Languages and frameworks. • Protocols (HTTP/1, HTTP/2, gRPC, databases, caching, etc.). • Infrastructures (IaaS, CaaS, on premise, etc.). • Intermediate load balancers (AWS ELB, F5, etc.). • Observability output (stats, tracing, and logging). • Implementations (often partial) of retry, circuit breaking, rate limiting, timeouts, and other distributed systems best practices. • Authentication and Authorization. • Per language libraries for service calls.
5. State of SoA networking in industry A really big and confusing mess...
6. What is Envoy The network should be transparent to applications. When network and application problems do occur it should be easy to determine the source of the problem. This sounds great! But it turns out it’s really, really hard.
7. What is Envoy • Out of process architecture: Let’s do a lot of really hard stuff in one place and allow application developers to focus on business logic. • Modern C++11 code base: Fast and productive. • L3/L4 filter architecture: A byte proxy at its core. Can be used for things other than HTTP (e.g., MongoDB, redis, stunnel replacement, TCP rate limiter, etc.). • HTTP L7 filter architecture: Make it easy to plug in different functionality. • HTTP/2 first! (Including gRPC and a nifty gRPC HTTP/1.1 bridge). • Service discovery and active/passive health checking. • Advanced load balancing: Retry, timeouts, circuit breaking, rate limiting, shadowing, outlier detection, etc. • Best in class observability: stats, logging, and tracing. • Edge proxy: routing and TLS.
8. Envoy service to service topology Service Cluster Envoy Service Discovery Service Cluster Envoy Service External Services HTTP/2 REST / gRPC
9. Lyft today Legacy monolith (+Envoy) MongoDB Internet Clients “Front” Envoy (via TCP ELB) DynamoDB Python services (+Envoy) Service mesh! Awesome! No fear SoA! Go services (+Envoy) Stats / tracing (direct from Envoy) Discovery
10. Eventually consistent service discovery • Fully consistent service discovery systems are very popular (ZK, etcd, consul, etc.). • In practice they are hard to run at scale. • Service discovery is actually an eventually consistent problem. Let’s recognize that and design for it. • Envoy is designed from the get go to treat service discovery as lossy. • Active health checking used in combination with service discovery to produce a routable overlay. Discovery Status HC OK HC Failed Discovered Route Don’t Route Absent Route Don’t Route / Delete
11. Advanced load balancing • Different service discovery types. • Zone aware least request load balancing. • Dynamic stats: Per zone, canary specific stats, etc. • Circuit breaking: Max connections, requests, and retries. • Rate limiting: Integration with global rate limit service. • Shadowing: Fork traffic to a test cluster. • Retries: HTTP router has built in retry capability with different policies. • Timeouts: Both “outer” (including all retries) and “inner” (per try) timeouts. • Outlier detection: Consecutive 5xx • Deploy control: Blue/green, canary, etc. • Fault injection
12. Observability • Observability is by far the most important thing that Envoy provides. • Having all SoA traffic transit through Envoy gives us a single place where we can: ◦ Produce consistent statistics for every hop ◦ Create and propagate a stable request ID / tracing context ◦ Consistent logging ◦ Distributed tracing
13. Observability: Per service auto-generated panel Links to logging and tracing
14. Observability: Service to service template dashboard Template with drop down for every service
15. Observability: Envoy global health dashboard
16. Observability: Distributed tracing
17. Observability: Logging
18. Performance matters for a service proxy • For most companies developer time is worth more than infra costs (cost vs. throughput). • However, Latency and predictability is what matters. And in particular tail latency (P99+). • Virtual IaaS, multiple languages and runtimes, languages that use GC: Niceties that improve productivity and reduce upfront dev costs, but make debugging really difficult. • Ability to reason about overall performance and reliability is critical.
19. Envoy thin clients @Lyft from lyft.api_client import EnvoyClient switchboard_client = EnvoyClient( service='switchboard' ) msg = {'template': 'breaksignout'} headers = {'x-lyft-user-id': 12345647363394} switchboard_client.post("/v2/messages", data=msg, headers=headers) • Abstract away egress port • Request ID/tracing propagation • Guide devs into good timeout, retry, etc. policies • Similar thin clients for Go and PHP
20. Istio and config APIs svcA Envoy Pod Service A svcB Envoy Service B Pilot Control Plane API Mixer Discovery & Config data to Envoys Policy checks, telemetry Control flow during request processing Istio-Auth TLS certs to Envoy Traffic is transparently intercepted and proxied. App is unaware of Envoy’s presence • “Raw Envoy” is still too hard to configure for most folks • Hard to decouple network from deploy orchestration • Build higher level control systems (universal dataplane) • Istio! • K8s first. More later
21. Community • Amazing growth in 9 months across multiple organizations • 59 contributors • 2 orgs with commit access (Lyft/Google) • Don’t believe the FUD / technosphere. C++ is alive and well. ;) • Join us! And many more ...
22. Q&A • Thanks for coming! Questions welcome on Twitter: @mattklein123 • We are super excited about building a community around Envoy. Talk to us if you need help getting started. • Stickers! • https://lyft.github.io/envoy/ (@envoyproxy) • https://istio.io/ (@istiomesh)