Evolving the Kubernetes Ingress APIs to GA and Beyond

https://kccncna19.sched.com/event/UaYG/evolving-the-kubernetes-ingress-apis-to-ga-and-beyond-christopher-m-luciano-ibm-bowei-du-google

The Ingress API has existed as beta type since early 2016 release in Kubernetes 1.2. Since its inception, it has been fairly lightweight and additions remained infrequent in attempts to maintain portability within the Kubernetes ecosystem. In response, Ingress API implementations commonly leverage a different internal API or extend the Ingress API by heavily decorating the resource with annotations.

In this session, we will present the Ingress enhancements for the GA/v1 Ingress API and what factors went into these decisions. Furthermore, we'll explore several possible directions for what a v2 API could entail and walk through several examples including existing non-Kubernetes implementations

Thursday November 21, 2019 3:20pm - 3:55pm
Room 16AB - San Diego Convention Center Mezzanine Level
Customizing + Extending Kubernetes
Experience Level Intermediate (Mid-level experience)
Session Slides Included Yes

Transcript

2. Bowei Du <bowei@> Christopher M Luciano <cmluciano@> Evolving the Kubernetes Ingress APIs to GA and Beyond
3. Overview • Ingress GA • Current state • Changes towards GA • Evolution to next set of APIs • Goals • API design
4. Current State of Ingress • Provides basic k8s service backend mappings/rules and related TLSconfig • Majority of Ingress usage provided by off-the-shelf controllers like ingress-nginx, HAProxy, etc. • Current API is limited in scope to ensure massive portability • Side effect of many decorator annotations dangling off of Ingress metadata
5. v1 API changes Clean up the object model: IngressClass Tweaks/fix to the specification: • backend to defaultBackend • Path-based prefixes/regex • Hostname wildcards. Add flexibility that will be hard to change later: • Alternate backend types
6. backend -> defaultBackend - Rename backend to defaultBackend - Clarify use of this field as the explicit default - Open PR k/k#80179
7. Path based prefixes & regex • Current v1beta1 APIs assume that the path is a regex specified with the POSIX IEEE 1003.1 standard ◦ Does not match cloud-provider or nginx/haproxy today ◦ kubernetes/ingress-nginx#555 • v1 goals ◦ Explicitly state path match mode ◦ Support existing implementation-specific variance ◦ Portable prefix matching with options for future ideas
8. Path based prefixes & regex
9. PathType Examples PathType Path Request path Match? Exact /abc /abc Yes Exact /abc /cba No Prefix / any path Yes Prefix /abc/ /abc Yes, trailing slash doesn’t matter Prefix /aaa/bbb /aaa/bbb/ccc Yes, subpath match Prefix /aaa/bbb /aaa/bbbzzz No, no prefix match
10. Hostname wildcards • Goal ◦ *.foo.com matches app.foo.com • Validation ◦ * must appear as first DNS label ▪ Single label ▪ Cannot be Host == “*” *.foo.com bar.foo.com Matches based on shared suffix *.foo.com baz.bar.foo.com No match, wildcard only covers single label *.foo.com foo.com No match, wildcard only covers single label
11. IngressClass
12. Current backend types • Currently Ingress == L7 service sets ◦ Only Kubernetes Services are valid backends • Goal ◦ Support alternate backends like storage buckets
13. Alternate backend types • Existing Kubernetes Service types MUST remain supported by implementations • Custom services leverage new Resource field by way of IngressClass
14. IngressBackend examples Kubernetes Service Custom Resource
15. v1 API Timeline • Kubernetes Release ◦ v1.14 ▪ Ingress API in extensions/v1beta1 copied to networking.k8s.io/v1beta1 ▪ Mailing list announcement for Ingress extensions/v1beta1 deprecation ◦ v1.15 ▪ Updated all documentation and k/k repo code for networking.k8s.io/v1beta1 ▪ ingress-nginx v0.25.0 ▪ ingress-gce v1.7.0 ◦ v1.18 ▪ New networking.k8s.io/v1 APIs released ▪ Mailing list deprecation warning of Ingress networking.k8s.io/v1beta1 deprecation
16. v1 API Timeline • Kubernetes Release ◦ v1.19 ▪ Update ingress-nginx/gce for v1 APIs ▪ Remove ability to serve extensions/v1beta1 ◦ v1.2x ? ▪ Remove ability to serve networking.k8s.io/v1beta1
17. Evolving Landscape Cloud LBs GCP, AWS, Azure, ... Middle Proxies nginx, envoy, haproxy, ... Transparent “Proxies” sidecars, kube-proxy... Early resources design were “self-service” oriented: • Few tenants • Empowered dev owns whole deployment Now, we have: • Multiple team/roles interactions • Potentially multiple co-existing implementations
18. Goals for the evolution Better model the personas and roles involved with services and load-balancing. Support modern load-balancing features while maintaining portability (or maybe “predictability”) Have standard mechanisms for extension for API growth / implementation / vendor-specific behaviors.
19. Status Note: the API is in a proposal stage, so many things can change (Your voice as a user is key part of this process!) • Describe how goals map to API design • Give a sense of the overall object model • Highlight some interesting problems/approaches Many open questions.
20. Goals for evolution Better model the personas and roles involved with services and load-balancing. → Resource model, RBAC Support modern load-balancing features while maintaining portability (or maybe “predictability”) → Levels of support, specification and conformance Have standard mechanisms for extension for API growth / implementation / vendor-specific behaviors. → Resource model, polymorphism
21. Personas and Roles ‍♀ Infrastructure Provider Provides the infrastructure for cluster creation, e.g. cloud provider, internal PaaS team. ‍‍ Cluster Operator / NetOps / SRE Manages the cluster overall once its created. Responsible for overall policies, e.g. which services expose to Internet. ‍‍ Application Developer Builds the services and applications and defines traffic routing, services.
22. Modeling roles: Ingress Ingress Service Ingress Class ‍♀infrastructure provider ‍‍ application developer Ingress is a self-service model. IngressClass are created by infrastructure provider Application developer manages Ingress + Service; Ingress limited to simple L7 description.
23. ‍‍ cluster operator ‍♀ infrastructure provider ‍‍ application developer Modeling roles: Evolution Gateway Class Gateway *Route Service GatewayClass is created by the infrastructure provider (kinds of LB available) Gateway is an instantiation of a given LB. *Route (HTTPRoute) and Services are defined by the developer.
24. Gateway/Route schema Gateway Class Gateway *Route Service * * m n kind: GatewayClass name: internet-lb provider: acme.io/cloud parameters: apiGroup: acme.io/cloud kind: Parameters name: ... kind: GatewayClass name: private-lb provider: acme.io/cloud parameters: apiGroup: acme.io/cloud kind: GatewayParameters name: ... apiGroup: acme.io/cloud kind: Parameters public: true apiGroup: acme.io/cloud kind: Parameters public: false
25. Gateway/Route schema Gateway Class Gateway *Route Service * * m n kind: GatewayClass name: internet-lb ... kind: Gateway namespace: net-ops name: the-gateway class: internet-lb listeners: - port: 80 protocol: http routes: - kind: HTTPRoute name: my-app kind: HTTPRoute name: my-app rules: - path: /my-app ... gateways: - namespace: net-ops name: the-gateway kind: Service name: my-app
26. Roles and resources kind: GatewayClass name: internet-lb kind: Gateway name: my-public-app-1 kind: HTTPRoute name: my-private-app kind: Gateway name: my-app kind: HTTPRoute name: my-app kind: Service name: app-backend kind: HTTPRoute name: my-app Role: Infrastructure Provider Role: App developer Role: Cluster Operator / NetOps Team role separation
27. Roles and resources kind: GatewayClass name: internet-lb kind: GatewayClass name: private-lb kind: Gateway name: my-public-app-1 kind: Gateway name: my-private-app kind: HTTPRoute name: my-private-app kind: HTTPRoute name: private-service kind: Gateway name: my-app kind: Gateway name: private-service kind: HTTPRoute name: my-app kind: Service name: app-backend kind: HTTPRoute name: my-app kind: Service name: internal-svc kind: HTTPRoute name: my-app Role: Infrastructure Provider Role: App developer Role: Cluster Operator / NetOps Self-service Gateway
28. Roles and resources kind: GatewayClass name: internet-lb kind: GatewayClass name: private-lb kind: Gateway name: my-public-app-1 kind: Gateway name: my-private-app kind: HTTPRoute name: my-private-app kind: HTTPRoute name: private-service kind: Gateway name: my-app kind: Gateway name: private-service kind: HTTPRoute name: my-app kind: Service name: app-backend kind: HTTPRoute name: my-app kind: Service name: internal-svc kind: HTTPRoute name: my-app Role: Infrastructure Provider Role: App developer Role: Cluster Operator / NetOps kind: GatewayClass name: in-cluster-ngx kind: Gateway name: my-private-app kind: Gateway name: my-private-app kind: HTTPRoute name: private-service kind: Service name: test-svc Fully self-service
29. Design Challenges • Varied deployment patterns • Multiple cooperating resources + roles • Portability • Extensibility • Status
30. Gateway Deployment Models kind: GatewayClass name: in-cluster-ngx kind: HTTPRoute name: private-service kind: Service name: test-svc kind: Gateway name: my-private-app Controller then … configures a cloud LB / Gateway … instantiatiates a proxy instance in cluster / Gateway … merges Gateways together into a configuration, reloads configuration.
31. Designing for RBAC USER ROLE VERB RESOURCE Alice can act as a Cluster Operator with permission to Update the configuration of a Gateway. USER ROLE VERB RESOURCE
32. Designing for RBAC Resource boundaries should be split based on responsibilities. However, “handshake” required between Gateway/Route: • Protect who can use a given Gateway (“no Internet for you”) • Self-service attachment (Gateway ← Route) • Some users want to control who can export a *Route. Most natural modeling is explicit double-sided links.
33. Portability Custom API Extended API 100% portable IF supported Core API 100% portable Core MUST be supported. Extended Feature by feature. MAYBE supported, but MUST be portable. Part of API schema. Custom No guarantee for portability, No k8s API schema. gravity...
34. Portability Custom API Extended API 100% portable IF supported Core API 100% portable Enforcement by conformance tests. Extended feature definition requires self-contained conformance. Require all extended features be checkable statically. gravity...
35. Portability Custom API Extended API Core API Neato Feature ® Ingress GA Traffic splitting Regex? Rewrite
36. Extensions Extensibility mechanisms: annotations vs map[string]string vs Custom Resource (polymorphic “ObjectReference”) vs Raw Objects (inline JSON) meta: kind: Gateway routes: - apiGroup: networking.x.k8s.io kind: HTTPRoute name: my-application - apiGroup: acme.io kind: StorageBucket name: my-bucket
37. Status Long neglected aspect of LB APIs More complex due to multiple resource composition/references. Events: ephemeral “something changed” vs Status: “current state”
38. How to get involved API sketch is here: link Working group (coming soon, info will go out): - Bi-weekly meetings - SIG-NETWORK mailing list (link) - Slack channel - github.com/kubernetes-sigs/service-apis Help wanted: - Feedback on the proposal (users AND implementers) - Experimental implementations
39. Q&A
40. Who are we? Christopher M Luciano cmluciano@ Bowei Du bowei@