Share
End to End Message Authenticity in Cloud Native Systems
Presented at Cloud Native Rejekts London together with Micah Hausler
Schedule link: https://cfp.cloud-native.rejekts.io/cloud-native-rejekts-europe-london-2025/talk/U99JU3/
Recording: https://youtu.be/rJacyDygVi0
Location: 116 Pall Mall, London SW1Y 5ED, Storbritannien
Abstract:
OpenID Connect (OIDC) and mutual TLS are popular authentication mechanisms used widely in cloud native environments, and commonly as a basis for workload identity in SPIFFE. However, OIDC tokens are prone to interception, replay, and forwarding attacks and are unable to guarantee end-to-end request authenticity. Mutual TLS solves those problems at the transport layer, but is rarely used in browsers, and seldom fully end-to-end in microservices-oriented systems. HTTP Message Signatures is a new IETF specification that aims to solve credential replay, forwarding and end-to-end integrity attacks, and be broadly deployable.
This talk introduces the audience to HTTP Message Signatures and demonstrates its security benefits to authentication in cloud native, microservice-oriented, systems. Further, we’ll cover how the use of smart caching and replication allows this protocol to scale to millions of requests per second, and how this could be integrated with SPIFFE.
Lucas Käldström
Transcript
48 slides
-
End to End Message Authenticity in Cloud Native Systems Micah
Hausler, AWS & Lucas Käldström, Upbound @micahhausler.com & @luxas.dev -
Who are we? Staff Engineer @ Upbound Formerly at Weaveworks
Former SIG Cluster Lifecycle co-lead CNCF Ambassador 2017-2024 Principal Engineer @ AWS Working on AWS EKS Kubernetes Security Response Committee SIG-Auth Chair @micahhausler.com & @luxas.dev -
Outline 1. Why does end to end authenticity matter? a.
Attacks against common setups with intermediate proxies b. OpenID Connect (OIDC) token stealing 2. How is it possible to mitigate? a. HTTP Message Signatures (RFC 9421) 3. How to perform key distribution? a. OAuth 2.0 Demonstrating Proof of Possession (RFC 9449) b. OpenPubKey key distribution protocol c. Key database (e.g. out-of-band or GitHub) 4. Demo - https://github.com/micahhausler/httpsig-rejekts-demo 5. What’s next? a. WIMSE Service to Service Authentication & SPIFFE Demo Link @micahhausler.com & @luxas.dev -
(m)TLS in a microservices world TLS Server: www.acmeco.com HTTPS POST
/users Authorization: Bearer <id_token_A> {“name”: “lucas”} A @micahhausler.com & @luxas.dev -
(m)TLS in a microservices world TLS mTLS Server: www.acmeco.com HTTPS
POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev -
(m)TLS in a microservices world TLS mTLS Server: www.acmeco.com ✅
Authenticity: server cert & id token valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev -
(m)TLS in a microservices world TLS mTLS ✅ Authenticity: server
cert & id token valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted ✅ Authenticity: server & client cert valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev Server: www.acmeco.com -
TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the
correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway A Hijacked (or misconfigured) gateway: Token replay @micahhausler.com & @luxas.dev Server: www.acmeco.com -
TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the
correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway Seen tokens: id_token_A id_token_B id_token_C A Hijacked (or misconfigured) gateway: Token replay @micahhausler.com & @luxas.dev Server: www.acmeco.com -
Hijacked (or misconfigured) gateway: Token replay TLS mTLS Server: user-service
❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway Seen tokens: id_token_A id_token_B id_token_C HTTPS POST /users Authorization: Bearer <id_token_C> {“name”: “lucas”} A @micahhausler.com & @luxas.dev Server: www.acmeco.com -
Hijacked (or misconfigured) gateway: Payload change TLS mTLS Server: user-service
❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway A HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “micah”} @micahhausler.com & @luxas.dev Server: www.acmeco.com -
TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the
correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing ⇒ Every component in the request chain have to be fully trusted HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway A HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “micah”} Hijacked (or misconfigured) gateway: Payload change @micahhausler.com & @luxas.dev Server: www.acmeco.com -
Hijacked (or misconfigured) gateway: Summary TLS mTLS Server: user-service ❓Authenticity:
Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway A @micahhausler.com & @luxas.dev Server: www.acmeco.com -
OIDC ID token primer OIDC = OpenID Connect protocol. Very
widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify. -
OIDC ID token primer OIDC = OpenID Connect protocol. Very
widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify. -
OIDC ID token primer OIDC = OpenID Connect protocol. Very
widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login 2. Signed ID token Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify. -
OIDC ID token primer OIDC = OpenID Connect protocol. Very
widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login 2. Signed ID token Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification 3. Request with ID token Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify. -
OIDC ID token primer OIDC = OpenID Connect protocol. Very
widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token @micahhausler.com & @luxas.dev -
OIDC ID token primer OIDC = OpenID Connect protocol. Very
widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership @micahhausler.com & @luxas.dev -
OIDC ID token primer OIDC = OpenID Connect protocol. Very
widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership ❌ Symmetric credential => can be forwarded if leaked @micahhausler.com & @luxas.dev -
OIDC ID token primer OIDC = OpenID Connect protocol. Very
widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership ❌ Symmetric credential => can be forwarded if leaked ❌ Not bound to request contents => data can be modified by intermediaries @micahhausler.com & @luxas.dev -
HTTP Message Signatures: IETF RFC 9421 Idea: A client can
sign the security-sensitive content and headers with either a symmetric or asymmetric key, and the server can verify that the observed content actually matches what the client signed (and vice versa). TLS HTTPS POST /users Authorization: Bearer <id_token_A> Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} Server: www.acmeco.com A sign choose what data to sign (what is important?) -
HTTP Message Signatures: IETF RFC 9421 Idea: A client can
sign the security-sensitive content and headers with either a symmetric or asymmetric key, and the server can verify that the observed content actually matches what the client signed (and vice versa). TLS mTLS Server: user-service HTTPS POST /users Authorization: Bearer <id_token_A> Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} Server: www.acmeco.com Client: gateway A HTTPS POST /users Authorization: Bearer <id_token_A> Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} sign verify Content or header tampering => ❌ Unaltered => ✅ -
AWS sigv4: symmetric secret key + HMAC HTTP Message Signatures
RFC ≈ generalization of the AWS sigv4 protocol Client gets a access key ID + secret somehow (out of band) Both access key ID + secret are random strings generated by AWS IAM AWS client library uses the access key secret to sign critical HTTP request info AWS ARS verifies the request is correctly signed => user authenticated @micahhausler.com & @luxas.dev -
High-level: transitive trust 1. application trusts identity provider (configuration) 2.
identity provider attests the client (login) 3. identity provider binds a client identity to its client key => application can trust client identity and client key through identity provider @micahhausler.com & @luxas.dev -
Binding a key to an OIDC flow 1. Web login
page 2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com/.well-known/jwks.json for verification This is the “normal” OIDC flow 3. POST code private sign public verify Authenticate ID token using 5. Request with ID token 4. Get signed ID token -
Demonstrating Proof of Possession (DPoP): RFC 9449 1. Web login
page 2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com /.well-known/jwks.json for verification 5. Request with ID token with & Signature-Input: @body “authorization” Signature: dMT/A/76ehrdBTD/ 3. POST code and PoP 4. Get signed ID token, bound to private sign public verify Authenticate ID token using Can verify signature using private sign ✅ Ephemeral user key ❌ Changes needed in OIDC provider { “alg”: “ES256” }.{ "sub":"[email protected]", "iss":"https://server.example.com", "cnf": { "jkt":"<user public key>" } }. <signature> -
OpenPubKey: Make nonce meaningful 1. Web login page, set nonce
2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com/ .well-known/jwks.json for verification 5. Request with ID token with & Signature-Input: @body “authorization” Signature: dMT/A/76ehrdBTD/ 3. POST code 4. Get signed ID token, bound to thru nonce private sign public verify Authenticate ID token using Can verify signature using private sign ✅ Ephemeral user key ✅ Works with any OIDC provider { “alg”: “ES256” }.{ "sub":"[email protected]", "iss":"https://server.example.com", "nonce”: SHA-3({ “alg”: “ES256”, “upk”: <user-provided-key>, “rz”: crypto.random() }) }. <signature> -
GitHub is actually a public key database (!) In order
to clone private GitHub repositories, users can: - generate a private key locally - log into GitHub and register their public key - … and GitHub will accept usage of the private key, - AND expose public keys at https://github.com/<username>.keys (!) @micahhausler.com & @luxas.dev -
Demo: Signing Requests 1. TLS HTTPS GET /hello X-GitHub-Username: luxas
Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com luxas private sign 0. Log into Github and register public key -
Demo: Signing Requests 1. TLS 2. mTLS Server: demo-app HTTPS
GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com Client: gateway luxas private sign 0. Log into Github and register public key public verify HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… -
Demo: Signing Requests 1. TLS 2. mTLS Server: demo-app HTTPS
GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com Client: gateway luxas Content or header tampering => ❌ Unaltered => ✅ private sign 0. Log into Github and register public key 3. Download public key public verify HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… -
Live demo, you should participate and do this! 1. Generate
a new SSH key using e.g. ssh-keygen -t ecdsa -f test-key 2. Log into your GitHub, and register the public key (e.g. test-key.pub) 3. go install github.com/micahhausler/httpsig-rejekts-demo@latest 4. Run httpsig-rejekts-demo -key test-key -username <username> The program signs the HTTP content with your private key. 5. Execute the curl command the program outputs. Server downloads your public key from Github, and authenticates you! 6. If you change (like an intermediary) e.g. the signed content-type header, the server will notice the mismatch and reject the request -
Workload Identity with SPIFFE “Clients” can also be workloads, not
just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. @micahhausler.com & @luxas.dev -
Workload Identity with SPIFFE “Clients” can also be workloads, not
just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API @micahhausler.com & @luxas.dev -
Workload Identity with SPIFFE “Clients” can also be workloads, not
just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API 2. The gRPC API server, the agent, attests the workload @micahhausler.com & @luxas.dev -
Workload Identity with SPIFFE “Clients” can also be workloads, not
just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API 2. The gRPC API server, the agent, attests the workload 3. The agent gives the workload the OIDC tokens/x509 certs it’s bound to @micahhausler.com & @luxas.dev -
WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF
working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. @micahhausler.com & @luxas.dev -
WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF
working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs @micahhausler.com & @luxas.dev -
WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF
working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key @micahhausler.com & @luxas.dev -
WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF
working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key 3. By the authenticator trusting SPIFFE, it can also trust the workload’s key @micahhausler.com & @luxas.dev -
WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF
working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key 3. By the authenticator trusting SPIFFE, it can also trust the workload’s key Thanks to Evan and Pieter from SPIRL for answering questions about WIMSE! Get involved today! https://github.com/spiffe/spiffe/pull/327 @micahhausler.com & @luxas.dev -
Thanks! Bluesky: @luxas.dev LinkedIn: luxas CNCF Slack(s): luxas Bluesky: @micahhausler.com
LinkedIn: micahhausler CNCF Slack(s): micahhausler Credits: Icons by Flaticon