Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Open Infra Days Asia - Auditing in Kubernetes 101
Search
Nikhita Raghunath
September 11, 2021
0
32
Open Infra Days Asia - Auditing in Kubernetes 101
Talk about basics of auditing in Kubernetes, for Open Infra Days Asia 2021
Nikhita Raghunath
September 11, 2021
Tweet
Share
More Decks by Nikhita Raghunath
See All by Nikhita Raghunath
Getting started as an Open Source Contributor
nikhita
0
120
Auditing in Kubernetes 101
nikhita
0
150
Getting started with the Kubernetes Community - KubeCon Keynote
nikhita
1
100
How to Contribute to Kuberntes - Kubernetes Days India
nikhita
2
200
Extending the Kubernetes API 101
nikhita
3
460
Extending the Kubernetes API
nikhita
2
480
The Story Of $GOPATH
nikhita
4
570
Featured
See All Featured
Designing Dashboards & Data Visualisations in Web Apps
destraynor
228
52k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
Designing the Hi-DPI Web
ddemaree
280
34k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
290
Optimising Largest Contentful Paint
csswizardry
33
2.9k
Art, The Web, and Tiny UX
lynnandtonic
296
20k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
9
680
Optimizing for Happiness
mojombo
376
69k
Designing Experiences People Love
moore
138
23k
Gamification - CAS2011
davidbonilla
80
5k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Navigating Team Friction
lara
183
14k
Transcript
Auditing in Kubernetes 101 Nikhita Raghunath Staff Engineer, VMware
WHO AM I • Staff Engineer at VMware • Member
of the Kubernetes Steering Committee • Technical Lead for SIG Contributor Experience • CNCF Ambassador Github - nikhita Twitter - TheNikhita
SECRET CONTAINING PASSWORD IN YOUR CLUSTER
SECRET CONTAINING PASSWORD IN YOUR CLUSTER SECRET GOT UPDATED TO
MYSTERIOUS VALUE
SECRET CONTAINING PASSWORD IN YOUR CLUSTER SECRET GOT UPDATED TO
MYSTERIOUS VALUE LOGS
Logs from the Pod @TheNikhita
Logs from the Pod @TheNikhita
Logs from the Pod Events @TheNikhita
Logs from the Pod Events @TheNikhita
Logs from the Pod Events Apiserver Logs @TheNikhita
Logs from the Pod Events Apiserver Logs @TheNikhita
AUDIT LOGS!
{ "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "7684b057-7e2d-4188-a6ae-8fc51afd0c9d", "stage":
"ResponseComplete", "requestURI": "/api/v1/namespaces/default/secrets", "verb": "create", "user": { "username": "minikube-user", "groups": [ "system:masters", "system:authenticated" ] }, "sourceIPs": [ "X.Y.Z.1" ], "objectRef": { "resource": "secrets", "namespace": "default", "name": "mysecret", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 201 }, "requestReceivedTimestamp": "2020-02-12T18:06:04.577792Z", "stageTimestamp": "2020-02-12T18:06:04.584173Z", } AUDIT EVENTS @TheNikhita
WHAT HAPPENED "verb": "create", @TheNikhita
ON WHAT DID IT HAPPEN "objectRef": { "resource": "secrets", "namespace":
"default", "name": "mysecret", "apiVersion": "v1" }, @TheNikhita
WHEN DID IT HAPPEN "requestReceivedTimestamp": "2020-02-12T18:06:04.577792Z", "stageTimestamp": "2020-02-12T18:06:04.584173Z", @TheNikhita
WHO DID IT "user": { "username": "minikube-user", "groups": [ "system:masters",
"system:authenticated" ] }, @TheNikhita
WHERE WAS IT INITIATED "sourceIPs": [ "1.2.3.4" ], @TheNikhita
THAT’S A LOT OF LOGS!
LET’S CONTROL THE VERBOSITY
LET’S CONTROL THE VERBOSITY WHAT TO LOG WHEN TO LOG
LET’S CONTROL THE VERBOSITY WHAT TO LOG WHEN TO LOG
YAML
AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata
omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
WHEN TO LOG 1. RequestReceived - Audit handler receives request
@TheNikhita
WHEN TO LOG 1. RequestReceived - Audit handler receives request
2. ResponseStarted - For long running requests @TheNikhita
WHEN TO LOG 1. RequestReceived - Audit handler receives request
2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed @TheNikhita
WHEN TO LOG 1. RequestReceived - Audit handler receives request
2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed 4. Panic - Event generated when panic occurs @TheNikhita
RequestReceived ResponseComplete Response ResponseStarted Panic Request Response Kube APIserver
Request Kube APIserver
RequestReceived Request Kube APIserver
RequestReceived Response Request Kube APIserver
RequestReceived Response Panic Request Kube APIserver
RequestReceived Response ResponseStarted Request Kube APIserver
RequestReceived ResponseComplete Response ResponseStarted Request Kube APIserver
RequestReceived ResponseComplete Response ResponseStarted Request Response Kube APIserver
RequestReceived ResponseComplete Response ResponseStarted Panic Request Response Kube APIserver
WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:
Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets GROUP/VERSION RESOURCE VERBS @TheNikhita
WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:
Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
LEVELS 1. None - don’t log these requests @TheNikhita
LEVELS 1. None - don’t log these requests 2. Metadata
- only request metadata @TheNikhita
LEVELS 1. None - don’t log these requests 2. Metadata
- only request metadata 3. Request - ,, + request body @TheNikhita
LEVELS 1. None - don’t log these requests 2. Metadata
- only request metadata 3. Request - ,, + request body 4. RequestResponse - ,, + response body @TheNikhita
RECOMMENDATIONS FOR WRITING POLICIES
- level: Metadata resources: - group: "" resources: - secrets
- configmaps - group: authentication.k8s.io resources: - tokenreviews Only log at Metadata level for sensitive resources @TheNikhita
- level: None nonResourceURLs: - '/healthz*' - /version - '/swagger*'
Don’t log read-only URLs @TheNikhita
Log at RequestResponse level for critical resources Log at atleast
Metadata level for all resources @TheNikhita
rules: - level: RequestResponse resources: - group: "" resources: ["pods"]
- level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] @TheNikhita
rules: - level: RequestResponse resources: - group: "" resources: ["pods"]
- level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Evaluated in top-down order @TheNikhita
rules: - level: RequestResponse resources: - group: "" resources: ["pods"]
- level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Status calls can be large and high-volume @TheNikhita
More examples at https://github.com/kubernetes/kubernetes/blob/master/cl uster/gce/gci/configure-helper.sh @TheNikhita
WHERE DO THESE LOGS GO
BACKEND LOG WEBHOOK @TheNikhita
BACKEND LOG WEBHOOK • Writes events to disk • Sends
events to external API @TheNikhita
BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path
• Sends events to external API • --audit-webhook-config-file @TheNikhita
BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path
• Sends events to external API • --audit-webhook-config-file --audit-policy-file @TheNikhita
HOW ARE THESE LOGS SENT TO THE BACKEND
BATCHING BATCH BLOCKING BLOCKING-STRICT @TheNikhita
BATCHING BATCH BLOCKING BLOCKING-STRICT Buffers events & processes in batches
Blocks APIserver responses to process individual events Failure at RequestReceived stage leads to failure of whole call @TheNikhita
BATCHING BATCH BLOCKING BLOCKING-STRICT --audit-webhook-mode --audit-log-mode @TheNikhita
BATCHING BATCH BLOCKING BLOCKING-STRICT --audit-webhook-mode --audit-log-mode @TheNikhita
UPDATING AUDIT POLICY
UPDATING AUDIT POLICY RESTART OF APISERVER
UPDATING AUDIT POLICY RESTART OF APISERVER
UPDATING AUDIT POLICY
UPDATING AUDIT POLICY UPDATING A K8S RESOURCE
UPDATING AUDIT POLICY UPDATING A K8S RESOURCE
DYNAMIC AUDIT CONFIGURATION @TheNikhita
DYNAMIC AUDIT CONFIGURATION apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: mysink
spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://audit.app" @TheNikhita
SECURITY PERFORMANCE @TheNikhita
SECURITY PERFORMANCE Write access to feature = Read access to
all cluster data @TheNikhita
SECURITY PERFORMANCE Write access to feature = Read access to
all cluster data cluster-admin level privilege Increase in CPU/Memory Usage @TheNikhita
KEP #sig-auth slack channel on k8s slack @TheNikhita
None
LOG COLLECTOR PATTERNS
LOG COLLECTOR PATTERNS Audit Log File + Fluentd @TheNikhita
LOG COLLECTOR PATTERNS Audit Webhook File + Logstash @TheNikhita
LOG COLLECTOR PATTERNS Audit Webhook File + Falco @TheNikhita
HOW ARE AUDIT LOGS HELPFUL
UNDERSTANDING K8S INTERNALS Analysing system calls show how different components
interact @TheNikhita
DETECTING MISCONFIGURATIONS “Who deleted this resource?” @TheNikhita
TROUBLESHOOTING ISSUES Analysing calls which trigger HTTP errors @TheNikhita
PERFORMANCE ISSUES “Which app is generating lots of calls” @TheNikhita
CONCLUSION • Audit logs can give us a lot of
information of what goes on in our cluster • To control what should be logged, we write audit policies • Recommendations for writing audit policies • Different audit backends • Batching methods • Dynamic Audit Configuration • Log Collector Patterns
THANK YOU