Share
Tools and Strategies for Making the Most of Kubernetes Access Control
Presented at KubeCon Atlanta 2025 together with Micah Hausler.
Location: Atlanta Convention Center, 285 Andrew Young International Blvd NW, Atlanta, GA 30313, USA
Sched link: https://sched.co/27FdC
Recording link: https://youtu.be/JBM0PRyDaPs?si=dg5DqsjUET8s-avH
Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/pull/5684
Abstract:
Have you ever struggled writing least-privilege access control policies for Kubernetes? Are you concerned about the wide permissions of installed Helm charts? Do you manage to regularly audit who has access to sensitive resources?
In this talk, Kubernetes contributors Micah and Lucas introduce you to open source tools that help you on your defense in depth journey for securing the Kubernetes API surface. They demonstrate how to right-size your RBAC rules semi-automatically, audit who can access sensitive resources, and check whether policy refactors are correct.
This talk is part of a journey to improve Kubernetes access control in core. However, to make this initiative successful, user feedback is needed throughout the process. You’ll learn about the planned Kubernetes Conditional Authorization feature, which will make authoring right-sized policies easier.
By the end of the talk, you will know how to get involved, and future directions for improved Kubernetes access control.
Lucas Käldström
Transcript
89 slides
-
#KubeCon #CloudNativeCon Tools and Strategies for Making the Most of
Kubernetes Access Control Lucas Käldström, Upbound Micah Hausler, AWS -
Who are we? Staff Engineer @ Upbound Formerly at Weaveworks
Former SIG Cluster Lifecycle co-lead CNCF Ambassador 2017-2024 Principal Engineer @ AWS Working on AWS EKS Kubernetes Security Response Committee SIG-Auth co-chair @luxas.dev & @micahhausler.com -
Outline 1. Introduction to Kubernetes Authorization 2. Kubernetes RBAC a.
Auto-discover RBAC rules needed with audit2rbac b. Query “what resources do I have access to?” using rakkess c. Query “who has access to sensitive resources?” using rakkess and kubectl-who-can 3. Conditional Authorization (Kubernetes Enhancement Proposal) a. Cedar Policy b. upbound/kubernetes-cedar-authorizer demo! 4. Conclusion: what’s next? @luxas.dev & @micahhausler.com -
Fast Safe Analyzable Expressive Correct RBAC VAP Analyzability wall Curated
Lifestyle For Unsplash+ -
Fast Safe Analyzable Expressive Correct RBAC Webhooks VAP Turing-completeness wall
Analyzability wall Note: Not configurable in all distributions! Curated Lifestyle For Unsplash+ -
Fast Safe Analyzable Expressive Correct RBAC Webhooks VAP Turing-completeness wall
Analyzability wall NEW? Note: Not configurable in all distributions! Curated Lifestyle For Unsplash+ -
Life of an API server request (pt. 1) Image from
“Usable Access Control in Cloud Management System” by Lucas Käldström @luxas.dev & @micahhausler.com -
Life of an API server request (pt. 1) Image from
“Usable Access Control in Cloud Management System” by Lucas Käldström -
Life of a write request (simplified) Authenticators 401 Webhook OIDC
CA @luxas.dev & @micahhausler.com -
Life of a write request (simplified) Authenticators Authorizers Metadata: API
Group Resource Type Namespace Name Username User Groups User UID User Extra 401 403 Webhook OIDC CA Webhook RBAC @luxas.dev & @micahhausler.com -
Mutating/Validating Admission Controllers 40X CEL Webhook Metadata: API Group Resource
Type Namespace Name Username User Groups User UID User Extra + Data: Payload/Body Previous Object Life of a write request (simplified) Authenticators Authorizers Metadata: API Group Resource Type Namespace Name Username User Groups User UID User Extra 401 403 Webhook OIDC CA Webhook Storage 200 RBAC @luxas.dev & @micahhausler.com -
Authenticators Authorizers Metadata: API Group Resource Type Namespace Name Username
User Groups User UID User Extra 401 403 Webhook OIDC CA Webhook RBAC Life of a read request (simplified) Storage 200 @luxas.dev & @micahhausler.com -
#KubeCon #CloudNativeCon Kubernetes RBAC RBAC = Role-Based Access Control RBAC
rules stored in the API server Privilege escalation prevention Only Allow roles, no Deny roles Operates only on request metadata, not objects -
Kubernetes RBAC Subjects RoleBinding admin RoleBinding deploy-workloads *:* OR Role
Bindings @luxas.dev & @micahhausler.com -
Kubernetes RBAC Subjects RoleBinding admin RoleBinding deploy-workloads Role admin Role
deploy-workloads 1:1 *:* OR Role Bindings Roles @luxas.dev & @micahhausler.com -
Kubernetes RBAC Subjects RoleBinding admin RoleBinding deploy-workloads Role admin Role
deploy-workloads 1:1 *:* Rule 1 Rule 2 OR OR (Rule 1 and Role deploy-workloads omitted for brevity) 1:* Role Bindings Roles Role Rules @luxas.dev & @micahhausler.com -
Kubernetes RBAC Subjects RoleBinding admin RoleBinding deploy-workloads Role admin Role
deploy-workloads 1:1 *:* Rule 1 Rule 2 OR OR (Rule 1 and Role deploy-workloads omitted for brevity) 1:* apiGroup resource namespace name AND Role Bindings Roles Role Rules Predicates @luxas.dev & @micahhausler.com -
Kubernetes RBAC Subjects RoleBinding admin RoleBinding deploy-workloads Role admin Role
deploy-workloads 1:1 *:* Rule 1 Rule 2 OR OR (Rule 1 and Role deploy-workloads omitted for brevity) 1:* apiGroup Obj 1 resource namespace name Obj 2 Obj 3 AND Role Bindings Roles Role Rules Predicates Objects @luxas.dev & @micahhausler.com -
Kubernetes RBAC Subjects RoleBinding admin RoleBinding deploy-workloads Role admin Role
deploy-workloads 1:1 *:* Rule 1 Rule 2 OR OR (Rule 1 and Role deploy-workloads omitted for brevity) 1:* apiGroup Obj 1 resource namespace name Obj 2 Obj 3 AND Role Bindings Roles Role Rules Predicates Objects “RBAC” “ABAC” -
How do I know which RBAC roles I need? When
in doubt, it is tempting to grant too wide permissions However, the audit log can tell you about unauthorized requests @luxas.dev & @micahhausler.com -
How do I know which RBAC roles I need? When
in doubt, it is tempting to grant too wide permissions However, the audit log can tell you about unauthorized requests 🙌 Jordan Liggitt’s audit2rbac tool can infer the roles needed from the audit log! Use carefully to not grant permissions to actually unauthorized requests App API server Audit log audit2rbac Read RBAC Rule 403 403 details -
Loads of RBAC rules… What do I have access to?
Check your own permissions using kubectl auth can-i <verb> <resource> @luxas.dev & @micahhausler.com -
Loads of RBAC rules… What do I have access to?
Check your own permissions using kubectl auth can-i <verb> <resource> However, to get a bigger picture, try out rakkess by Cornelius Weig Works through the (Self)SubjectAccessReview API Example output for “what can I do in namespace default?” Image Credit: rakkess -
Who has access to <insert sensitive resource>? A cluster administrator
must ensure that sensitive resources are properly protected. @luxas.dev & @micahhausler.com -
Who has access to <insert sensitive resource>? A cluster administrator
must ensure that sensitive resources are properly protected. In order to audit who (according to RBAC!) has access to what, rakkess or kubectl-who-can by Liz Rice/Aqua Security can be of help: Image Credit: kubectl-who-can demo @luxas.dev & @micahhausler.com -
#KubeCon #CloudNativeCon Conditional Authorization Brand-new Kubernetes Enhancement Proposal (KEP) Allow
for more granularity Unified UX between authorization and admission -
@luxas.dev & @micahhausler.com MSc thesis: Usable Access Control in Cloud
Management Systems Open access at: luxas/research -
Use-cases - Allow a principal to only read Secrets with
a given label @luxas.dev & @micahhausler.com -
Use-cases - Allow a principal to only read Secrets with
a given label - Allow a principal to update an object only when a sensitive field is unchanged @luxas.dev & @micahhausler.com -
Use-cases - Allow a principal to only read Secrets with
a given label - Allow a principal to update an object only when a sensitive field is unchanged - Allow a principal to create objects only with certain names @luxas.dev & @micahhausler.com -
Use-cases - Allow a principal to only read Secrets with
a given label - Allow a principal to update an object only when a sensitive field is unchanged - Allow a principal to create objects only with certain names - Allow a node agent to only access objects referring to them @luxas.dev & @micahhausler.com -
Use-cases - Allow a principal to only read Secrets with
a given label - Allow a principal to update an object only when a sensitive field is unchanged - Allow a principal to create objects only with certain names - Allow a node agent to only access objects referring to them - Allow a controller to only add/remove its own finalizer @luxas.dev & @micahhausler.com -
Use-cases - Allow a principal to only read Secrets with
a given label - Allow a principal to update an object only when a sensitive field is unchanged - Allow a principal to create objects only with certain names - Allow a node agent to only access objects referring to them - Allow a controller to only add/remove its own finalizer - Deny everyone except admins to use GPU adminAccess @luxas.dev & @micahhausler.com -
Mutating/Validating Admission Controllers 40X CEL Webhook Metadata: API Group Resource
Type Namespace Name Username User Groups User UID User Extra + Data: Payload/Body Previous Object Recall admission control for write requests Authenticators Authorizers Metadata: API Group Resource Type Namespace Name Username User Groups User UID User Extra 401 403 Webhook OIDC CA Webhook Storage 200 RBAC @luxas.dev & @micahhausler.com -
Validating Admission Control For a request to succeed, all admission
controllers must consider the object “valid” Admission control never gives permissions, but can only remove (deny policy) Webhook apiVersion: admission.k8s.io/v1 kind: AdmissionReview request: operation: CREATE name: foo-pod object: kind: Pod … response: allowed: true CEL -
Current problem: “Over-grant” in RBAC, deny in VAP RBAC Role
Allow in authorization RBAC Role Binding Allow in authorization CEL Policy Deny in admission create, update, delete gateways — object=* oldobject=* .class != ‘test-gateway’ .class == ‘test-gateway’ Kubernetes RBAC CEL Rule Amount of permissions for lucas Desired permissions -
Unified policies through Partial Evaluation What if it was possible
to declare in one place that “lucas can create gateways when class=’test-gateway’”? @luxas.dev & @micahhausler.com -
Unified policies through Partial Evaluation What if it was possible
to declare in one place that “lucas can create gateways when class=’test-gateway’”? In pseudo-code, it could look something like: “request.verb == ‘create’ && request.apiGroup == ‘gateway.networking.k8s.io’ && request.resource == ‘gateways’ && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation One trick is to use
partial evaluation. Mark request.object as “unknown”, but evaluate the expression as much as possible. If lucas does a kubectl create -f gateway.yaml, the following happens: “request.verb == ‘create’ && request.apiGroup == ‘gateway.networking.k8s.io’ && request.resource == ‘gateways’ && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation One trick is to use
partial evaluation. Mark request.object as “unknown”, but evaluate the expression as much as possible. If lucas does a kubectl create -f gateway.yaml, the following happens: “true && request.apiGroup == ‘gateway.networking.k8s.io’ && request.resource == ‘gateways’ && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation One trick is to use
partial evaluation. Mark request.object as “unknown”, but evaluate the expression as much as possible. If lucas does a kubectl create -f gateway.yaml, the following happens: “true && true && request.resource == ‘gateways’ && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation One trick is to use
partial evaluation. Mark request.object as “unknown”, but evaluate the expression as much as possible. If lucas does a kubectl create -f gateway.yaml, the following happens: “true && true && true && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation One trick is to use
partial evaluation. Mark request.object as “unknown”, but evaluate the expression as much as possible. If lucas does a kubectl create -f gateway.yaml, the following happens: “true && true && true && true && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation One trick is to use
partial evaluation. Mark request.object as “unknown”, but evaluate the expression as much as possible. If lucas does a kubectl create -f gateway.yaml, the following happens: “true && true && true && true && true && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation One trick is to use
partial evaluation. Mark request.object as “unknown”, but evaluate the expression as much as possible. If lucas does a kubectl create -f gateway.yaml, the following happens: “true && true && true && true && true && request.object.v1.spec.gatewayClassName == ’test-gateway’” Condition on the request object, defer evaluation to when decoded! -
Unified policies through Partial Evaluation Note that even though the
object is unknown, in some cases partial evaluation can evaluate to a concrete true or false: For example, if user hacker executes kubectl create -f gateway.yaml “request.verb == ‘create’ && request.apiGroup == ‘gateway.networking.k8s.io’ && request.resource == ‘gateways’ && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation Note that even though the
object is unknown, in some cases partial evaluation can evaluate to a concrete true or false: For example, if user hacker executes kubectl create -f gateway.yaml “true && request.apiGroup == ‘gateway.networking.k8s.io’ && request.resource == ‘gateways’ && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation Note that even though the
object is unknown, in some cases partial evaluation can evaluate to a concrete true or false: For example, if user hacker executes kubectl create -f gateway.yaml “true && true && request.resource == ‘gateways’ && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Unified policies through Partial Evaluation Note that even though the
object is unknown, in some cases partial evaluation can evaluate to a concrete true or false: For example, if user hacker executes kubectl create -f gateway.yaml “true && true && true && request.userInfo.username == “lucas” && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” -
Note that even though the object is unknown, in some
cases partial evaluation can evaluate to a concrete true or false: For example, if user hacker executes kubectl create -f gateway.yaml “true && true && true && false && has(request.object.v1) && request.object.v1.spec.gatewayClassName == ’test-gateway’” Unified policies through Partial Evaluation Expression is always false, no matter what the object is! Don’t decode, instead Deny -
Kubernetes Authorization Kubernetes considers an ordered list (chain) of authorizers.
Today, there are three decision modes: Allow, Deny, or NoOpinion. If Allow or Deny, enforce decision. If NoOpinion, consult next authorizer. @luxas.dev & @micahhausler.com -
Kubernetes Authorization with Conditions Kubernetes considers an ordered list (chain)
of authorizers. Today, there are three decision modes: Allow, Deny, or NoOpinion. If Allow or Deny, enforce decision. If NoOpinion, consult next authorizer. Conditional Authorization adds a new decision mode: Conditional, along with a set of conditions. The request can proceed to the admission stage if the decision is Allow or Conditional with at least one condition that can make the request get authorized. -
Conditional Authorization: Data model A condition has the following properties:
- condition: String with some predicate on the request/stored objects. - type: What language the condition is written in (e.g. CEL, OPA, or Cedar) - effect: Allow, Deny, or NoOpinion. How to treat “true”. - id: Authorizer-unique identifier (for reason and error messages) @luxas.dev & @micahhausler.com -
Conditional Authorization + Cedar policy language Authorization RequestInfo UserInfo Admission
Control Built-in condition enforcer* RequestInfo UserInfo Body New! Conditions Body Kubernetes luxas/conditional_authz_4 branch Authorizer Partial Evaluation Allow, Deny, or Conditions 403 403 Authentication Storage *If conditions are expressed in CEL, no need to ask the authorizer again Evaluate Conditions on New/Old Object @luxas.dev & @micahhausler.com -
Before KEP-4601, there was no way to say “authorize a
read only with labelSelector ingress=true”. KEP-4601 Authorize with Selectors RBAC Authorization CEL/Webhook Admission Metadata Data Reads Writes @luxas.dev & @micahhausler.com -
With KEP-4601, expressing “authorize a read only with labelSelector ingress=true”
is possible using a webhook authorizer (not builtin RBAC). KEP-4601 Authorize with Selectors RBAC Authorization CEL/Webhook Admission Metadata Data Reads Writes Field- and Label Selector Webhook Authorization @luxas.dev & @micahhausler.com -
Utilizing both KEPs together, makes it possible to express all
types of authorization using the same, unified interface for the user. KEP-4601 Authorize with Selectors Unified Policy Authoring Interface For example, using CEL or Cedar Metadata Data Reads Writes @luxas.dev & @micahhausler.com -
Open Source Authorization Engine Aims to be expressive, fast, safe,
and analyzable @luxas.dev & @micahhausler.com -
Maintains an encoding into decidable Mathematical Logic (SMT)* Open Source
Authorization Engine Aims to be expressive, fast, safe, and analyzable *Satisfiability Modulo Theories Logic. Announcement @luxas.dev & @micahhausler.com -
Open Source Authorization Engine Aims to be expressive, fast, safe,
and analyzable Supports RBAC, ReBAC and ABAC paradigms *Satisfiability Modulo Theories Logic. Announcement Maintains an encoding into decidable Mathematical Logic (SMT)* @luxas.dev & @micahhausler.com -
Open Source Authorization Engine Aims to be expressive, fast, safe,
and analyzable AWS has donated Cedar to the CNCF \o/ Supports RBAC, ReBAC and ABAC paradigms *Satisfiability Modulo Theories Logic. Announcement Maintains an encoding into decidable Mathematical Logic (SMT)* @luxas.dev & @micahhausler.com -
unified Kubernetes authorization using the Cedar Policy Language Demo time!
Available on github.com/upbound/kubernetes-cedar-authorizer @luxas.dev & @micahhausler.com -
Conditional Authorization + Cedar policy language Authorization RequestInfo UserInfo Admission
Control Built-in CEL condition enforcer* RequestInfo UserInfo Body New! Conditions Body Kubernetes luxas/conditional_authz_4 branch github.com/upbound/kubernetes-cedar-authorizer Partial Evaluation Allow, Deny, or CEL conditions 403 403 Policies Authentication Storage *If CEL condition support is integrated into core, use that, otherwise use the webhook @luxas.dev & @micahhausler.com -
Examples “Let user micahhausler only read and write secrets with
.type=tls” @luxas.dev & @micahhausler.com -
Examples “Let user micahhausler only read and write secrets with
.type=tls” @luxas.dev & @micahhausler.com -
What are the benefits of analyzable policies? ⇒ Compare permissiveness
of two policies (or sets of them) larger smaller @luxas.dev & @micahhausler.com -
What are the benefits of analyzable policies? ⇒ Compare permissiveness
of two policies (or sets of them) ⇒ Check for equality (help refactors) old new Uh oh! New, refactored policy not maybe as planned? -
What are the benefits of analyzable policies? ⇒ Compare permissiveness
of two policies (or sets of them) ⇒ Check for equality (help refactors) ⇒ Check for logical inconsistencies in a policy set (shadowing, no-ops) ← No effect ← Allow shadows allow ← Deny shadows allow Allow policy Deny policy @luxas.dev & @micahhausler.com -
#KubeCon #CloudNativeCon Conclusion Where to go from here? Join the
discussion, give feedback to us! -
Namespace: service-foo ReferenceGrant: Protecting object-to-object references All of the aforementioned
authorization policies have focused on “what can a principal do in the system?” Namespace: dns-admin HTTPRoute foo-route Secret foo-tls-cert @luxas.dev & @micahhausler.com -
Namespace: service-foo ReferenceGrant: Protecting object-to-object references All of the aforementioned
authorization policies have focused on “what can a principal do in the system?” Namespace: dns-admin HTTPRoute foo-route Secret foo-tls-cert @luxas.dev & @micahhausler.com -
Namespace: service-foo ReferenceGrant: Protecting object-to-object references All of the aforementioned
authorization policies have focused on “what can a principal do in the system?” Namespace: dns-admin HTTPRoute foo-route Secret foo-tls-cert ReferenceGrant allow-foo-cert @luxas.dev & @micahhausler.com -
Namespace: service-foo ReferenceGrant: Protecting object-to-object references All of the aforementioned
authorization policies have focused on “what can a principal do in the system?” A good writeup on the state of the union by Uwe Krüger here. HTTPRoute foo-route Namespace: dns-admin Secret foo-tls-cert ReferenceGrant allow-foo-cert -
Next Steps 1. Join #sig-auth-authorizers-dev in the Kubernetes Slack and/or
the SIG Auth biweekly meetings! @luxas.dev & @micahhausler.com -
Next Steps 1. Join #sig-auth-authorizers-dev in the Kubernetes Slack and/or
the SIG Auth biweekly meetings! 2. Comment on the KEP-5681 Conditional Authorization proposal! (Hopefully implemented in 1.36) @luxas.dev & @micahhausler.com -
Next Steps 1. Join #sig-auth-authorizers-dev in the Kubernetes Slack and/or
the SIG Auth biweekly meetings! 2. Comment on the KEP-5681 Conditional Authorization proposal! (Hopefully implemented in 1.36) 3. Try out the mentioned projects, give feedback and contribute @luxas.dev & @micahhausler.com -
Next Steps 1. Join #sig-auth-authorizers-dev in the Kubernetes Slack and/or
the SIG Auth biweekly meetings! 2. Comment on the KEP-5681 Conditional Authorization proposal! (Hopefully implemented in 1.36) 3. Try out the mentioned projects, give feedback and contribute 4. When the primitives are settled, work can proceed on the higher-level features RBAC++ and ReferenceGrant @luxas.dev & @micahhausler.com -
Thanks! Bluesky: @luxas.dev & @micahhausler.com LinkedIn: luxas & micahhausler CNCF
Slack(s): luxas & micahhausler