latency, high performance, dev productivity • L3/L4 filter architecture • HTTP L7 filter architecture • HTTP/2 first • Service/config discovery • Active/passive health checking • Advanced load balancing • Best in class observability • Service/middle/edge proxy • Hot restart
Worker thread(s) Worker thread(s) Worker thread(s) File flush thread(s) Listeners xDS Runtime Stat flush Admin Connections Process management • Main thread handles non-data plane misc tasks • Worker thread(s) embarrassingly parallel and handle listeners, connections, and proxying • File flush threads avoid blocking • Designed to be 100% non-blocking • Designed to scale to massive parallelism (# of HW threads)
primitive heavily used in the Linux kernel • Scales extremely well for R/W locking that is read heavy New Ref-counted Data Reader Updater Event loop Ref-counted Data “Quiescent period” Copy
event loop (4) IO event / load balancer (7) Post handler / TLS update (5) Health checker (2) xDS/DNS (3) TLS (6) Main Worker • Complete example of TLS and RCU for cluster updates
Service A Service A Service A Service A’ 33% 67% Load balancer Service A -> A’ Rolling deploy Hot restart deploy Service A -> A’ Service A -> A’ • Full binary reload without dropping any connections • Very useful in legacy/non-container scheduler worlds
Hot restarter Primary process Stats Logs Hot restarter Secondary process UDS • Stats/locks kept in shared memory • Simple RPC protocol over unix domain sockets (UDS) • Sockets, some stats, etc. passed between processes • Built for containers
first looked up in TLS cache 3. Not found, allocated in central cache, added to TLS cache 4. Counters/gauges in shared memory 5. Histograms in process memory 6. Scope deletion causes a TLS cache flush on all threads Store (main thread) (1) Shared memory (4) Stat entries Scope central cache (3) Gauges Counters TLS scope cache flush (6) Histograms Scope central cache Scope central cache Scope TLS cache (2) Gauges Counters Histograms Process memory (5) Histograms
TLS histogram TLS histogram Histogram A Histogram B &Current histogram recordValue(...) (1) Merge histogram (2) TLS post() to swap current (3) TLS post() back to continue merge (4) Merge all background histograms (5) (1) TLS histogram values recorded into “current” without locks (2) Period merge/flush (3) Post to each worker to swap current histogram (record now happens on alternate) (4) Post back to main thread to continue merge (5) Merge all TLS histograms without locks
and low latency • Architecture embarrassingly parallel and designed for mostly lock free scaling across high HW thread count • Heavy use of RCU locking paradigm and TLS • Design for containerized world • Extensibility is key Cloud native summary
• We are super excited about building a community around Envoy. Talk to us if you need help getting started. • https://www.envoyproxy.io/ • Lyft is hiring!