Envoy Gateway CNCF Canada Toronto

Up until now, Ingress routes into K8s clusters have been defined by the Ingress kind, or by vendor-specific CRDs. Neither of these were satisfactory, so a new set of built-in k8s APIs was developed - the Gateway API. In this talk, we will cover the motivation for a new API, its design, and show some examples of its use. we’ll then also cover implementations of it today and in the future, and talk about the exciting merging of several of the existing ingress controllers into one new de facto standard - Envoy Gateway.

Transcript

1. Gateway APIs and API Gateways Modern Ingress Demystified Sergey Marunich, Tetrate.io
2. Outline • Recap: Ingress • Gateway API • Envoy Gateway • [Envoy] [API] Gateway • Where Next?
3. Ingress Networking
4. Service A Backend *.example.com Service A Service A
5. Service A Backend Load Balancer *.example.com Service A Service A
6. Service A Backend Load Balancer Node port *.example.com Service A Service A
7. Backend Load Balancer Node port Cluster IP *.example.com Service A Service A Service A
8. Backend Proxy Proxy Proxy Proxy Ingress Load Balancer Node port Cluster IP *.example.com Service A Service A Service A
9. Service A Backend Proxy Proxy Proxy Proxy Ingress Load Balancer Node port Cluster IP *.example.com Cluster IP Service A Service A
10. Service A Backend Proxy Proxy Proxy Proxy Ingress Load Balancer Node port Cluster IP *.example.com Cluster IP Service A Service A
11. Service A Backend Proxy Proxy Proxy Proxy Ingress Load Balancer Node port *.example.com Service A Service A Ingress Controller Service A
12. Service A Backend Proxy Proxy Proxy Proxy Ingress Load Balancer Node port *.example.com Service A Service A Ingress Controller Service A K8s API Server etcd
13. Service A Backend Proxy Proxy Proxy Proxy Ingress Load Balancer Node port *.example.com Service A Service A Ingress Controller Service A K8s API Server etcd Ingress
14. 😅
15. The Ingress API
16. Ingress API apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minimal-ingress spec: rules: - host: example.com http: paths: - path: /test backend: service: name: test port: number: 80
17. Ingress API apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minimal-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: example.com http: paths: - path: /test backend: service: name: test port: number: 80
18. Ingress API apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minimal-ingress annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: example.com http: paths: - path: /test backend: service: name: test port: number: 80
19. Ingress API apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minimal-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: example.com http: paths: - path: /test pathType: Prefix backend: service: name: test port: number: 80
20. Ingress API apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nginx spec: controller: example.com/nginx-ingress-controller
21. Ingress API apiVersion: networking.k8s.io/v1 kind: IngressClass metadata: name: nginx spec: controller: example.com/nginx-ingress-controller --- kind: Deployment spec: template: spec: containers: - name: nginx args: - /nginx-ingress-controller - '--ingress-class=k8s.io/nginx' - '--controller-class=example.com/nginx-ingress-controller'
22. Ingress API apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: minimal-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / nginx.ingress.kubernetes.io/configuration-snippet: | more_set_headers "Request-Id: $req_id"; spec: ingressClassName: nginx rules: - host: example.com http: …
23. Ingress API: Implementations • Nginx • Haproxy • Apache • Traefik • Contour • Ambassador • Kong • Tyk • Avi • Istio • etc
24. 😅😅
25. Gateway API
26. xkcd, Creative Commons Attribution 2.5
27. Recap: The Storage API StorageClass PersistentVolume Pod PersistentVolume Claim Pod Pod (AWS, gp2) (10Gi, ReadOnce) PersistentVolume Claim Infra provider / cluster builder: Infra admin: App dev: PersistentVolume
28. The Gateway API
29. The Gateway API • Not built-in yet; packaged as CRDs • >1 resource • gateway.networking.k8s.io ◦ GatewayClass/v1beta1 ◦ Gateway/v1beta1 ◦ HTTPRoute/v1beta1 ◦ TLSRoute/v1alpha1 - SNI routing ◦ GRPCRoute/v1alpha1 ◦ TCPRoute/v1alpha1 ◦ UDPRoute/v1alpha1
30. GatewayClass apiVersion: gateway.networking.k8s.io/v1beta1 kind: GatewayClass metadata: name: my-class spec: controllerName: gateway.envoyproxy.io/gatewayclass-controller
31. apiVersion: gateway.networking.k8s.io/v1beta1 kind: Gateway metadata: name: my-envoy-gateway spec: gatewayClassName: my-class listeners: - name: http protocol: HTTP port: 80 - name: https protocol: HTTPS port: 443 Gateway
32. HTTPRoute apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: http-log spec: parentRefs: [{name: my-gateway}] hostnames: ["www.example.com"] rules: - matches: - path: {value: /http-log, type: PathPrefix} backendRefs: - {group: "", kind: Service, name: http-log, port: 80, weight: 1}
33. HTTPRoute apiVersion: gateway.networking.k8s.io/v1beta1 kind: HTTPRoute metadata: name: http-log spec: parentRefs: [{name: my-gateway}] hostnames: ["www.example.com"] rules: - matches: - path: {value: /http-log, type: PathPrefix} filters: - {type: URLRewrite, urlRewrite: {path: {type: ReplacePrefixMatch, replacePrefixMatch: / }}} backendRefs: - {group: "", kind: Service, name: http-log, port: 80, weight: 1}
34. What else does it look like? • Heavily based on the Istio API • In turn, Istio implements the Gateway API ◦ Currently beta ◦ Will be default when gw-api hits v1 • Also implemented by the SMI Meshes (Linkerd2, Consul, Open Service Mesh, etc)
35. A Mesh API? • Gain resources to describe East-West (service mesh) • GAMMA group trying to get meshes to adopt the GW API, and conversely to get GW API to model mesh concerns (https://gateway-api.sigs.k8s.io/contributing/gamma/) • Istio 1.16: Kubernetes Gateway API Implementation Promoted to Beta Istio’s implementation of the Gateway API has been promoted to Beta. This is a significant step toward our goal of making the Gateway API the default API for traffic management in the future.
36. Reference Implementation: Envoy Gateway
37. What’s a standard? Nginx-ingress currently the de facto standard • Surely the most common, certainly when you discount cloud providers’ ingress • Only one mentioned in the main upstream docs
38. An Envoy-Based Gateway • But nginx isn’t very modern ◦ Reads its config from a file, not an API ◦ The operator hides this, and that’s fine; that’s its job ◦ But those reload events cause the drop of in-flight requests, which isn’t ok ◦ Plus other operational issues ◦ Hard to extend • Envoy is more modern, and designed for this kinda stuff ◦ xDS API ◦ It’s proven itself as Ingress, Sidecar, even GFE • A new gateway in town!
39. Another One? • Contour, Emissary (formerly Ambassador) agreed to rebase onto the EG code, but will keep their brands, add value
40. Demo!
41. 🍺 An offering to the demo gods...
42. 👀
43. An Adventure in Metrics • Pod: <EG Operator> ◦ Container <main>: no metrics port ◦ Container kube-rbac-proxy: https metrics port, just controller_runtime’s default stats • Pod: <Envoy Instance> ◦ Container Envoy: prom-format metrics on admin at localhost:19000 (unreachable)
44. Other Features • cert-manager has experimental support (hard to demo locally)
45. A Work-in-Progress • v0.3 targeting December ◦ Full compliance to the Gateway API ◦ Doesn’t seem to mean other basics, like metrics • To follow the project ◦ https://github.com/envoyproxy/gateway ◦ Envoy Slack #gateway-dev
46. Gateway API API Gateway
47. What Even is an API Gateway? You might think • TLS termination • Load Balancing • L7 Routing • WAF • Rate-limiting and quotas • Bot-blocking • OIDC auth • Caching • Body validation and transformation • Version and staging support • etc
48. What Even is an API Gateway? “Basic” features • TLS termination • Load Balancing • L7 Routing “API Gateway” features • WAF • Rate-limiting and quotas • Bot-blocking • OIDC auth • Caching • Body validation and transformation • Version and staging support • etc
49. Envoy (API) Gateway • Extensible code • Extensible API • New, clean codebase
50. Gateway API models API Gateways • Gain resources to describe API Gateway features ◦ Auth one in progress • On-going discussion about making the API extensible to model the different features in all the implementations, but in a consistent, first-class way • “GEP” - Gateway Enhancement Proposal (https://gateway-api.sigs.k8s.io/contributing/gep/) • Graduation path ◦ Vendor extension ◦ GW-API extension ◦ GW-API core
51. A Work-in-the-Future • Needs the API • Needs the Extensions • None being worked on yet (that I know of) • Except Coraza: a Golang implementation of mod_security
52. Recap
53. Where Next? • Release of Envoy Gateway 0.3 • Emissary, Contour rebasing eventually • Get Gateway API into upstream k8s • Extend the GW-API to model API-GW concerns ◦ Solve problems like modelling deploy of redis for global ratelimits • Build API-GW feature plugins • Gateway API v1? • Envoy Gateway 1.0?
54. Recap • Ingress API sucks • Gateway API doesn’t ◦ Ingress ◦ East-West ◦ API Gateway • Envoy Gateway exists. It hasn’t got far but you can try it at home. • Envoy Gateway will become an API Gateway ◦ That needs lots of work
55. Thanks! Slides Videos Demo code tetratelabs Questions @smarunich