The Evolution of Twitter's Edge

Each day, Twitter responds to hundreds of billions of requests from users around the world. Today, Envoy is the point of entry for 100% of these requests. This hasn't always been the case. Not long ago, requests to Twitter passed through an ancient and highly bespoke edge proxy that was created internally and predated the open-source Envoy. In this talk, Ryland will describe how Twitter runs Envoy at the edge at scale, some of the unique benefits that Envoy provides in Twitter's edge architecture, and highlight features that Twitter has contributed to Envoy to support the edge use-case. He will outline how Twitter's edge architecture has evolved over time as the number of users and services has grown, what the next steps are for Twitter's edge, and the role Envoy will play in the future.

Transcript

1. The Evolution of Twitter’s Edge EnvoyCon North America October 11, 2021
2. Placeholder for image. Size should be 6” x 6” Placeholder for image. Size should be 6” x 6” Ryland Degnan Manager, TFE @Twitter Formerly, Edge @Netflix @rjdegnan 2 Who am I? TFE (Twitter Front End) uses @EnvoyProxy to handle hundreds of billions of requests per day #ExpandTheEdge Twitter @Twitter 7:35PM · Oct 10, 2021
3. 3 Twitter Challenges 1. 200+ million daily active users 2. Millions of requests per second 3. Users distributed globally 4. Data centers concentrated in US 5. Unpredictable load spikes 6. Largely unauthenticated API 7. Variety of clients
4. Example of slide 4 Twitter’s Edge Pre-2014 • In many locations, latency too high for users to consistently access Twitter content • Too much time spent establishing connections, transmitting and receiving data
5. 8899A6 Example of slide 5 Twitter Streaming Aggregator (TSA) • Created by Matt Klein in 2013 • L7 reverse proxy, speaks HTTP 1.1/HTTP 2.0 • Terminates SSL, multiplexes requests coming from many connections onto a few high bandwidth links • First listeners installed in strategic locations for World Cup 2014
6. Example of slide 6 Expanding the Edge • Bringing connection termination closer to users is faster, more reliable • Put TSA in POPs and then backhaul requests over reliable high bandwidth links
7. 7 TSA Challenges • Highly bespoke, not always RFC compliant • Not scalable to modern hardware • Does not support modern protocols • Ancient forked version of OpenSSL • Static configuration, no hot restarts • 2017 - Started looking at replacing TSA with OSS based alternative • 2019 - Started development of Envoy-based edge proxy + service mesh with CSL team • 2020-21 - T3 starts taking production traffic
8. 8 Twitter TLS Terminator (T3) • L7 proxy based on Envoy, a mature OSS project backed by a very active and supportive community • Drop-in replacement for TSA • Provides over 2x throughput per thread • Enables us to make vastly more efficient use of modern hardware • Modern protocol support • Security patches deployable in hours • Hot restarts • Dynamic configuration via control plane, yay!
9. 9 IPv6, HTTP/3 gRPC (trailers) SNI, TLS 1.3 OSS for the Win!
10. Image Source 10 Image Source TLS 1.3 • Shorter TLS handshakes, faster connection establishment • We observed 38.2% drop in TLS handshake time and 28.8% drop in the Home Timeline Pull-To-Refresh request duration on average
11. 11 Twitter Edge Architecture • Scala control plane (Finagle) • Uses existing libraries for service discovery, feature switches • Implements a number of xDS APIs (EDS, SDS, RTDS etc.) • Envoy data plane with custom filters • Finer grained upstream cluster specific stats aggregated by vhost, route etc. • Globally unique connection hash for application layer metrics aggregated per tcp connection • Mux protocol support (Twitter RPC multiplexing protocol) • And more!
12. 12 Bot protection at Twitter #T3 Twitter @Twitter 7:35PM · Jan 15, 2021 T3 Bot Mitigation • Constant abuse of unauthenticated endpoints • Existing rate limiting not useful in an edge context • Per-connection rate limiting • Rate limit quotas on a per connection basis so as to not affect legitimate users • Achieved a 20% improvement in TFE tail latency by blocking at the edge • TLS fingerprinting • Based on research by Lee Brotherston, JA3 project by John Althouse (Salesforce) • Generate unique fingerprints based on the SSL Client Hello packet of the incoming connection • Take actions (such as deny or redirect to honeypot) based on abusive usage patterns
13. Example of slide 13 Twitter’s Edge 2021... and Beyond • Run Envoy inside cloud POPs that can be spun on on demand • Finer-grained routing based on user, requested resource
14. 14 The battle for tomorrow #EnvoyProxy Twitter @Twitter 7:35PM · Jan 15, 2021 Pushing More Intelligence to the Edge • Caching on the edge with Envoy • Envoy HTTP cache filter + Memcached to cached materialized views at edge • Precompute edge cache • Envoy Mobile • Unified iOS/Android networking stack greatly simplifies large migrations • Unified observability stack • xDS support • Multidimensional connection mesh as an extension of Twitter Control Tower • Standardized push (opportunistic push to clients)
15. 15 More Fun With Envoy • Cloud Gateway • Transparent forward proxy using Envoy support for tunneling TCP over HTTP/2 CONNECT, SNI dynamic forward proxy • Service Proxy • Developer tool for quick access to services using Envoy dynamic forward proxy along with DNS-SRV service discovery • Service Mesh • Increasing need for language interoperability at Twitter • Existing client libraries in Scala • Replacing existing Scala-based sidecar with Envoy + Scala control plane
16. Thank You! 16 We are hiring! rdegnan@twitter.com