End to End Message Authenticity in Cloud Native Systems

Presented at Cloud Native Rejekts London together with Micah Hausler

Schedule link: https://cfp.cloud-native.rejekts.io/cloud-native-rejekts-europe-london-2025/talk/U99JU3/
Recording: https://youtu.be/rJacyDygVi0
Location: 116 Pall Mall, London SW1Y 5ED, Storbritannien

Abstract:
OpenID Connect (OIDC) and mutual TLS are popular authentication mechanisms used widely in cloud native environments, and commonly as a basis for workload identity in SPIFFE. However, OIDC tokens are prone to interception, replay, and forwarding attacks and are unable to guarantee end-to-end request authenticity. Mutual TLS solves those problems at the transport layer, but is rarely used in browsers, and seldom fully end-to-end in microservices-oriented systems. HTTP Message Signatures is a new IETF specification that aims to solve credential replay, forwarding and end-to-end integrity attacks, and be broadly deployable.

This talk introduces the audience to HTTP Message Signatures and demonstrates its security benefits to authentication in cloud native, microservice-oriented, systems. Further, we’ll cover how the use of smart caching and replication allows this protocol to scale to millions of requests per second, and how this could be integrated with SPIFFE.

Transcript

1. End to End Message Authenticity in Cloud Native Systems Micah Hausler, AWS & Lucas Käldström, Upbound @micahhausler.com & @luxas.dev
2. Who are we? Staff Engineer @ Upbound Formerly at Weaveworks Former SIG Cluster Lifecycle co-lead CNCF Ambassador 2017-2024 Principal Engineer @ AWS Working on AWS EKS Kubernetes Security Response Committee SIG-Auth Chair @micahhausler.com & @luxas.dev
3. Outline 1. Why does end to end authenticity matter? a. Attacks against common setups with intermediate proxies b. OpenID Connect (OIDC) token stealing 2. How is it possible to mitigate? a. HTTP Message Signatures (RFC 9421) 3. How to perform key distribution? a. OAuth 2.0 Demonstrating Proof of Possession (RFC 9449) b. OpenPubKey key distribution protocol c. Key database (e.g. out-of-band or GitHub) 4. Demo - https://github.com/micahhausler/httpsig-rejekts-demo 5. What’s next? a. WIMSE Service to Service Authentication & SPIFFE Demo Link @micahhausler.com & @luxas.dev
4. Why does end to end authenticity matter? @micahhausler.com & @luxas.dev
5. (m)TLS in a microservices world TLS Server: www.acmeco.com HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} A @micahhausler.com & @luxas.dev
6. (m)TLS in a microservices world TLS mTLS Server: www.acmeco.com HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev
7. (m)TLS in a microservices world TLS mTLS Server: www.acmeco.com ✅ Authenticity: server cert & id token valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev
8. (m)TLS in a microservices world TLS mTLS ✅ Authenticity: server cert & id token valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted ✅ Authenticity: server & client cert valid ✅ Integrity: data is signed ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} A Server: user-service Client: gateway @micahhausler.com & @luxas.dev Server: www.acmeco.com
9. TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway A Hijacked (or misconfigured) gateway: Token replay @micahhausler.com & @luxas.dev Server: www.acmeco.com
10. TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway Seen tokens: id_token_A id_token_B id_token_C A Hijacked (or misconfigured) gateway: Token replay @micahhausler.com & @luxas.dev Server: www.acmeco.com
11. Hijacked (or misconfigured) gateway: Token replay TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway Seen tokens: id_token_A id_token_B id_token_C HTTPS POST /users Authorization: Bearer <id_token_C> {“name”: “lucas”} A @micahhausler.com & @luxas.dev Server: www.acmeco.com
12. Hijacked (or misconfigured) gateway: Payload change TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway A HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “micah”} @micahhausler.com & @luxas.dev Server: www.acmeco.com
13. TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing ⇒ Every component in the request chain have to be fully trusted HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway A HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “micah”} Hijacked (or misconfigured) gateway: Payload change @micahhausler.com & @luxas.dev Server: www.acmeco.com
14. Hijacked (or misconfigured) gateway: Summary TLS mTLS Server: user-service ❓Authenticity: Hopefully the gateway forwarded the correct user’s ID token to user-service The client does not know the identity of the user service (ok) ❌ Integrity: The gateway could change the payload without user-service noticing ✅ Confidentiality: data is encrypted HTTPS POST /users Authorization: Bearer <id_token_A> {“name”: “lucas”} Client: gateway A @micahhausler.com & @luxas.dev Server: www.acmeco.com
15. OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify.
16. OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify.
17. OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login 2. Signed ID token Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify.
18. OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps 1. Web login 2. Signed ID token Identity Provider issuer: https://identity-provider.com 0. Download public keys for verification 3. Request with ID token Only the identity provider has the private key and can sign. Anyone can fetch the public key and verify.
19. OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token @micahhausler.com & @luxas.dev
20. OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership @micahhausler.com & @luxas.dev
21. OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership ❌ Symmetric credential => can be forwarded if leaked @micahhausler.com & @luxas.dev
22. OIDC ID token primer OIDC = OpenID Connect protocol. Very widely used for Single-Sign On (SSO) of users, and federating trust ⇒ User database/login can be centralized to one provider, but used by many apps ✅ Easy for an application (relying party) to validate the user’s ID token ✅ Easy to propagate extra information about the user, e.g. group membership ❌ Symmetric credential => can be forwarded if leaked ❌ Not bound to request contents => data can be modified by intermediaries @micahhausler.com & @luxas.dev
23. How can you mitigate these issues? @micahhausler.com & @luxas.dev
24. HTTP Message Signatures: IETF RFC 9421 Idea: A client can sign the security-sensitive content and headers with either a symmetric or asymmetric key, and the server can verify that the observed content actually matches what the client signed (and vice versa). TLS HTTPS POST /users Authorization: Bearer <id_token_A> Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} Server: www.acmeco.com A sign choose what data to sign (what is important?)
25. HTTP Message Signatures: IETF RFC 9421 Idea: A client can sign the security-sensitive content and headers with either a symmetric or asymmetric key, and the server can verify that the observed content actually matches what the client signed (and vice versa). TLS mTLS Server: user-service HTTPS POST /users Authorization: Bearer <id_token_A> Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} Server: www.acmeco.com Client: gateway A HTTPS POST /users Authorization: Bearer <id_token_A> Signature-Input: “authorization” @body Signature: dMT/A/76ehrdBTD/2Xx8… {“name”: “lucas”} sign verify Content or header tampering => ❌ Unaltered => ✅
26. How to distribute the keys? @micahhausler.com & @luxas.dev
27. AWS sigv4: symmetric secret key + HMAC HTTP Message Signatures RFC ≈ generalization of the AWS sigv4 protocol Client gets a access key ID + secret somehow (out of band) Both access key ID + secret are random strings generated by AWS IAM AWS client library uses the access key secret to sign critical HTTP request info AWS ARS verifies the request is correctly signed => user authenticated @micahhausler.com & @luxas.dev
28. High-level: transitive trust 1. application trusts identity provider (configuration) 2. identity provider attests the client (login) 3. identity provider binds a client identity to its client key => application can trust client identity and client key through identity provider @micahhausler.com & @luxas.dev
29. Binding a key to an OIDC flow 1. Web login page 2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com/.well-known/jwks.json for verification This is the “normal” OIDC flow 3. POST code private sign public verify Authenticate ID token using 5. Request with ID token 4. Get signed ID token
30. Demonstrating Proof of Possession (DPoP): RFC 9449 1. Web login page 2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com /.well-known/jwks.json for verification 5. Request with ID token with & Signature-Input: @body “authorization” Signature: dMT/A/76ehrdBTD/ 3. POST code and PoP 4. Get signed ID token, bound to private sign public verify Authenticate ID token using Can verify signature using private sign ✅ Ephemeral user key ❌ Changes needed in OIDC provider { “alg”: “ES256” }.{ "sub":"someone@example.com", "iss":"https://server.example.com", "cnf": { "jkt":"<user public key>" } }. <signature>
31. OpenPubKey: Make nonce meaningful 1. Web login page, set nonce 2. Redirect with code Identity Provider issuer: https://identity-provider.com 0. Download public keys from https://identity-provider.com/ .well-known/jwks.json for verification 5. Request with ID token with & Signature-Input: @body “authorization” Signature: dMT/A/76ehrdBTD/ 3. POST code 4. Get signed ID token, bound to thru nonce private sign public verify Authenticate ID token using Can verify signature using private sign ✅ Ephemeral user key ✅ Works with any OIDC provider { “alg”: “ES256” }.{ "sub":"someone@example.com", "iss":"https://server.example.com", "nonce”: SHA-3({ “alg”: “ES256”, “upk”: <user-provided-key>, “rz”: crypto.random() }) }. <signature>
32. GitHub is actually a public key database (!) In order to clone private GitHub repositories, users can: - generate a private key locally - log into GitHub and register their public key - … and GitHub will accept usage of the private key, - AND expose public keys at https://github.com/<username>.keys (!) @micahhausler.com & @luxas.dev
33. Demo: Signing Requests luxas 0. Log into Github and register public key
34. Demo: Signing Requests 1. TLS HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com luxas private sign 0. Log into Github and register public key
35. Demo: Signing Requests 1. TLS 2. mTLS Server: demo-app HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com Client: gateway luxas private sign 0. Log into Github and register public key public verify HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8…
36. Demo: Signing Requests 1. TLS 2. mTLS Server: demo-app HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8… Server: rejekts.dev.micahhausler.com Client: gateway luxas Content or header tampering => ❌ Unaltered => ✅ private sign 0. Log into Github and register public key 3. Download public key public verify HTTPS GET /hello X-GitHub-Username: luxas Signature-Input: @path x-github-username Signature: dMT/A/76ehrdBTD/2Xx8…
37. Live demo, you should participate and do this! 1. Generate a new SSH key using e.g. ssh-keygen -t ecdsa -f test-key 2. Log into your GitHub, and register the public key (e.g. test-key.pub) 3. go install github.com/micahhausler/httpsig-rejekts-demo@latest 4. Run httpsig-rejekts-demo -key test-key -username <username> The program signs the HTTP content with your private key. 5. Execute the curl command the program outputs. Server downloads your public key from Github, and authenticates you! 6. If you change (like an intermediary) e.g. the signed content-type header, the server will notice the mismatch and reject the request
38. What’s next? @micahhausler.com & @luxas.dev
39. Workload Identity with SPIFFE “Clients” can also be workloads, not just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. @micahhausler.com & @luxas.dev
40. Workload Identity with SPIFFE “Clients” can also be workloads, not just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API @micahhausler.com & @luxas.dev
41. Workload Identity with SPIFFE “Clients” can also be workloads, not just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API 2. The gRPC API server, the agent, attests the workload @micahhausler.com & @luxas.dev
42. Workload Identity with SPIFFE “Clients” can also be workloads, not just human users logging in thru a browser SPIFFE is the de-facto standard for distributing identities to workloads. SPIFFE is meant to solve the “bottom turtle” problem of trust: 1. When a workload starts up, it asks “who am I?” through a gRPC API 2. The gRPC API server, the agent, attests the workload 3. The agent gives the workload the OIDC tokens/x509 certs it’s bound to @micahhausler.com & @luxas.dev
43. WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. @micahhausler.com & @luxas.dev
44. WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs @micahhausler.com & @luxas.dev
45. WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key @micahhausler.com & @luxas.dev
46. WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key 3. By the authenticator trusting SPIFFE, it can also trust the workload’s key @micahhausler.com & @luxas.dev
47. WIMSE: Workload Identity in Multi System Environments WIMSE, an IETF working group, is working to add a new type of credential to SPIFFE, a Workload Identity Token. Like DPoP and OpenPubKey, the idea is that: 1. The workload gets a private key, used to sign challenges, tokens, and proofs 2. The issuer (SPIFFE) binds the workload identity to the public key 3. By the authenticator trusting SPIFFE, it can also trust the workload’s key Thanks to Evan and Pieter from SPIRL for answering questions about WIMSE! Get involved today! https://github.com/spiffe/spiffe/pull/327 @micahhausler.com & @luxas.dev
48. Thanks! Bluesky: @luxas.dev LinkedIn: luxas CNCF Slack(s): luxas Bluesky: @micahhausler.com LinkedIn: micahhausler CNCF Slack(s): micahhausler Credits: Icons by Flaticon