Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Open Infra Days Asia - Auditing in Kubernetes 101

Nikhita Raghunath
September 11, 2021
32

Open Infra Days Asia - Auditing in Kubernetes 101

Talk about basics of auditing in Kubernetes, for Open Infra Days Asia 2021

Nikhita Raghunath

September 11, 2021
Tweet

Transcript

  1. WHO AM I • Staff Engineer at VMware • Member

    of the Kubernetes Steering Committee • Technical Lead for SIG Contributor Experience • CNCF Ambassador Github - nikhita Twitter - TheNikhita
  2. { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "7684b057-7e2d-4188-a6ae-8fc51afd0c9d", "stage":

    "ResponseComplete", "requestURI": "/api/v1/namespaces/default/secrets", "verb": "create", "user": { "username": "minikube-user", "groups": [ "system:masters", "system:authenticated" ] }, "sourceIPs": [ "X.Y.Z.1" ], "objectRef": { "resource": "secrets", "namespace": "default", "name": "mysecret", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 201 }, "requestReceivedTimestamp": "2020-02-12T18:06:04.577792Z", "stageTimestamp": "2020-02-12T18:06:04.584173Z", } AUDIT EVENTS @TheNikhita
  3. ON WHAT DID IT HAPPEN "objectRef": { "resource": "secrets", "namespace":

    "default", "name": "mysecret", "apiVersion": "v1" }, @TheNikhita
  4. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  5. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  6. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  7. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  8. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests @TheNikhita
  9. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed @TheNikhita
  10. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed 4. Panic - Event generated when panic occurs @TheNikhita
  11. WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:

    Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets GROUP/VERSION RESOURCE VERBS @TheNikhita
  12. WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:

    Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  13. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata @TheNikhita
  14. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata 3. Request - ,, + request body @TheNikhita
  15. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata 3. Request - ,, + request body 4. RequestResponse - ,, + response body @TheNikhita
  16. - level: Metadata resources: - group: "" resources: - secrets

    - configmaps - group: authentication.k8s.io resources: - tokenreviews Only log at Metadata level for sensitive resources @TheNikhita
  17. Log at RequestResponse level for critical resources Log at atleast

    Metadata level for all resources @TheNikhita
  18. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] @TheNikhita
  19. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Evaluated in top-down order @TheNikhita
  20. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Status calls can be large and high-volume @TheNikhita
  21. BACKEND LOG WEBHOOK • Writes events to disk • Sends

    events to external API @TheNikhita
  22. BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path

    • Sends events to external API • --audit-webhook-config-file @TheNikhita
  23. BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path

    • Sends events to external API • --audit-webhook-config-file --audit-policy-file @TheNikhita
  24. BATCHING BATCH BLOCKING BLOCKING-STRICT Buffers events & processes in batches

    Blocks APIserver responses to process individual events Failure at RequestReceived stage leads to failure of whole call @TheNikhita
  25. DYNAMIC AUDIT CONFIGURATION apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: mysink

    spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://audit.app" @TheNikhita
  26. SECURITY PERFORMANCE Write access to feature = Read access to

    all cluster data cluster-admin level privilege Increase in CPU/Memory Usage @TheNikhita
  27. CONCLUSION • Audit logs can give us a lot of

    information of what goes on in our cluster • To control what should be logged, we write audit policies • Recommendations for writing audit policies • Different audit backends • Batching methods • Dynamic Audit Configuration • Log Collector Patterns