Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Open Infra Days Asia - Auditing in Kubernetes 101
Search
Nikhita Raghunath
September 11, 2021
0
32
Open Infra Days Asia - Auditing in Kubernetes 101
Talk about basics of auditing in Kubernetes, for Open Infra Days Asia 2021
Nikhita Raghunath
September 11, 2021
Tweet
Share
More Decks by Nikhita Raghunath
See All by Nikhita Raghunath
Getting started as an Open Source Contributor
nikhita
0
120
Auditing in Kubernetes 101
nikhita
0
150
Getting started with the Kubernetes Community - KubeCon Keynote
nikhita
1
100
How to Contribute to Kuberntes - Kubernetes Days India
nikhita
2
200
Extending the Kubernetes API 101
nikhita
3
460
Extending the Kubernetes API
nikhita
2
480
The Story Of $GOPATH
nikhita
4
570
Featured
See All Featured
Building Your Own Lightsaber
phodgson
103
6.1k
Code Reviewing Like a Champion
maltzj
520
39k
Being A Developer After 40
akosma
87
590k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
250
21k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
506
140k
The Pragmatic Product Professional
lauravandoore
31
6.3k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
191
16k
Adopting Sorbet at Scale
ufuk
73
9.1k
KATA
mclloyd
29
14k
Optimising Largest Contentful Paint
csswizardry
33
2.9k
BBQ
matthewcrist
85
9.3k
Become a Pro
speakerdeck
PRO
25
5k
Transcript
Auditing in Kubernetes 101 Nikhita Raghunath Staff Engineer, VMware
WHO AM I • Staff Engineer at VMware • Member
of the Kubernetes Steering Committee • Technical Lead for SIG Contributor Experience • CNCF Ambassador Github - nikhita Twitter - TheNikhita
SECRET CONTAINING PASSWORD IN YOUR CLUSTER
SECRET CONTAINING PASSWORD IN YOUR CLUSTER SECRET GOT UPDATED TO
MYSTERIOUS VALUE
SECRET CONTAINING PASSWORD IN YOUR CLUSTER SECRET GOT UPDATED TO
MYSTERIOUS VALUE LOGS
Logs from the Pod @TheNikhita
Logs from the Pod @TheNikhita
Logs from the Pod Events @TheNikhita
Logs from the Pod Events @TheNikhita
Logs from the Pod Events Apiserver Logs @TheNikhita
Logs from the Pod Events Apiserver Logs @TheNikhita
AUDIT LOGS!
{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "7684b057-7e2d-4188-a6ae-8fc51afd0c9d", "stage":
"ResponseComplete", "requestURI": "/api/v1/namespaces/default/secrets", "verb": "create", "user": { "username": "minikube-user", "groups": [ "system:masters", "system:authenticated" ] }, "sourceIPs": [ "X.Y.Z.1" ], "objectRef": { "resource": "secrets", "namespace": "default", "name": "mysecret", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 201 }, "requestReceivedTimestamp": "2020-02-12T18:06:04.577792Z", "stageTimestamp": "2020-02-12T18:06:04.584173Z", } AUDIT EVENTS @TheNikhita
WHAT HAPPENED "verb": "create", @TheNikhita
ON WHAT DID IT HAPPEN "objectRef": { "resource": "secrets", "namespace":
"default", "name": "mysecret", "apiVersion": "v1" }, @TheNikhita
WHEN DID IT HAPPEN "requestReceivedTimestamp": "2020-02-12T18:06:04.577792Z", "stageTimestamp": "2020-02-12T18:06:04.584173Z", @TheNikhita
WHO DID IT "user": { "username": "minikube-user", "groups": [ "system:masters",
"system:authenticated" ] }, @TheNikhita
WHERE WAS IT INITIATED "sourceIPs": [ "1.2.3.4" ], @TheNikhita
THAT’S A LOT OF LOGS!
LET’S CONTROL THE VERBOSITY
LET’S CONTROL THE VERBOSITY WHAT TO LOG WHEN TO LOG
LET’S CONTROL THE VERBOSITY WHAT TO LOG WHEN TO LOG
YAML
AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
WHEN TO LOG 1. RequestReceived - Audit handler receives request
@TheNikhita
WHEN TO LOG 1. RequestReceived - Audit handler receives request
2. ResponseStarted - For long running requests @TheNikhita
WHEN TO LOG 1. RequestReceived - Audit handler receives request
2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed @TheNikhita
WHEN TO LOG 1. RequestReceived - Audit handler receives request
2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed 4. Panic - Event generated when panic occurs @TheNikhita
RequestReceived ResponseComplete Response ResponseStarted Panic Request Response Kube APIserver
Request Kube APIserver
RequestReceived Request Kube APIserver
RequestReceived Response Request Kube APIserver
RequestReceived Response Panic Request Kube APIserver
RequestReceived Response ResponseStarted Request Kube APIserver
RequestReceived ResponseComplete Response ResponseStarted Request Kube APIserver
RequestReceived ResponseComplete Response ResponseStarted Request Response Kube APIserver
RequestReceived ResponseComplete Response ResponseStarted Panic Request Response Kube APIserver
WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:
Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets GROUP/VERSION RESOURCE VERBS @TheNikhita
WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:
Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
LEVELS 1. None - don’t log these requests @TheNikhita
LEVELS 1. None - don’t log these requests 2. Metadata
- only request metadata @TheNikhita
LEVELS 1. None - don’t log these requests 2. Metadata
- only request metadata 3. Request - ,, + request body @TheNikhita
LEVELS 1. None - don’t log these requests 2. Metadata
- only request metadata 3. Request - ,, + request body 4. RequestResponse - ,, + response body @TheNikhita
RECOMMENDATIONS FOR WRITING POLICIES
- level: Metadata resources: - group: "" resources: - secrets
- configmaps - group: authentication.k8s.io resources: - tokenreviews Only log at Metadata level for sensitive resources @TheNikhita
- level: None nonResourceURLs: - '/healthz*' - /version - '/swagger*'
Don’t log read-only URLs @TheNikhita
Log at RequestResponse level for critical resources Log at atleast
Metadata level for all resources @TheNikhita
rules: - level: RequestResponse resources: - group: "" resources: ["pods"]
- level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] @TheNikhita
rules: - level: RequestResponse resources: - group: "" resources: ["pods"]
- level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Evaluated in top-down order @TheNikhita
rules: - level: RequestResponse resources: - group: "" resources: ["pods"]
- level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Status calls can be large and high-volume @TheNikhita
More examples at https://github.com/kubernetes/kubernetes/blob/master/cl uster/gce/gci/configure-helper.sh @TheNikhita
WHERE DO THESE LOGS GO
BACKEND LOG WEBHOOK @TheNikhita
BACKEND LOG WEBHOOK • Writes events to disk • Sends
events to external API @TheNikhita
BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path
• Sends events to external API • --audit-webhook-config-file @TheNikhita
BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path
• Sends events to external API • --audit-webhook-config-file --audit-policy-file @TheNikhita
HOW ARE THESE LOGS SENT TO THE BACKEND
BATCHING BATCH BLOCKING BLOCKING-STRICT @TheNikhita
BATCHING BATCH BLOCKING BLOCKING-STRICT Buffers events & processes in batches
Blocks APIserver responses to process individual events Failure at RequestReceived stage leads to failure of whole call @TheNikhita
BATCHING BATCH BLOCKING BLOCKING-STRICT --audit-webhook-mode --audit-log-mode @TheNikhita
BATCHING BATCH BLOCKING BLOCKING-STRICT --audit-webhook-mode --audit-log-mode @TheNikhita
UPDATING AUDIT POLICY
UPDATING AUDIT POLICY RESTART OF APISERVER
UPDATING AUDIT POLICY RESTART OF APISERVER
UPDATING AUDIT POLICY
UPDATING AUDIT POLICY UPDATING A K8S RESOURCE
UPDATING AUDIT POLICY UPDATING A K8S RESOURCE
DYNAMIC AUDIT CONFIGURATION @TheNikhita
DYNAMIC AUDIT CONFIGURATION apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: mysink
spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://audit.app" @TheNikhita
SECURITY PERFORMANCE @TheNikhita
SECURITY PERFORMANCE Write access to feature = Read access to
all cluster data @TheNikhita
SECURITY PERFORMANCE Write access to feature = Read access to
all cluster data cluster-admin level privilege Increase in CPU/Memory Usage @TheNikhita
KEP #sig-auth slack channel on k8s slack @TheNikhita
None
LOG COLLECTOR PATTERNS
LOG COLLECTOR PATTERNS Audit Log File + Fluentd @TheNikhita
LOG COLLECTOR PATTERNS Audit Webhook File + Logstash @TheNikhita
LOG COLLECTOR PATTERNS Audit Webhook File + Falco @TheNikhita
HOW ARE AUDIT LOGS HELPFUL
UNDERSTANDING K8S INTERNALS Analysing system calls show how different components
interact @TheNikhita
DETECTING MISCONFIGURATIONS “Who deleted this resource?” @TheNikhita
TROUBLESHOOTING ISSUES Analysing calls which trigger HTTP errors @TheNikhita
PERFORMANCE ISSUES “Which app is generating lots of calls” @TheNikhita
CONCLUSION • Audit logs can give us a lot of
information of what goes on in our cluster • To control what should be logged, we write audit policies • Recommendations for writing audit policies • Different audit backends • Batching methods • Dynamic Audit Configuration • Log Collector Patterns
THANK YOU