Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Open Infra Days Asia - Auditing in Kubernetes 101

Avatar for Nikhita Raghunath Nikhita Raghunath
September 11, 2021
35

Open Infra Days Asia - Auditing in Kubernetes 101

Talk about basics of auditing in Kubernetes, for Open Infra Days Asia 2021

Avatar for Nikhita Raghunath

Nikhita Raghunath

September 11, 2021
Tweet

Transcript

  1. WHO AM I • Staff Engineer at VMware • Member

    of the Kubernetes Steering Committee • Technical Lead for SIG Contributor Experience • CNCF Ambassador Github - nikhita Twitter - TheNikhita
  2. { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "7684b057-7e2d-4188-a6ae-8fc51afd0c9d", "stage":

    "ResponseComplete", "requestURI": "/api/v1/namespaces/default/secrets", "verb": "create", "user": { "username": "minikube-user", "groups": [ "system:masters", "system:authenticated" ] }, "sourceIPs": [ "X.Y.Z.1" ], "objectRef": { "resource": "secrets", "namespace": "default", "name": "mysecret", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 201 }, "requestReceivedTimestamp": "2020-02-12T18:06:04.577792Z", "stageTimestamp": "2020-02-12T18:06:04.584173Z", } AUDIT EVENTS @TheNikhita
  3. ON WHAT DID IT HAPPEN "objectRef": { "resource": "secrets", "namespace":

    "default", "name": "mysecret", "apiVersion": "v1" }, @TheNikhita
  4. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  5. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  6. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  7. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  8. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests @TheNikhita
  9. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed @TheNikhita
  10. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed 4. Panic - Event generated when panic occurs @TheNikhita
  11. WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:

    Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets GROUP/VERSION RESOURCE VERBS @TheNikhita
  12. WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:

    Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  13. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata @TheNikhita
  14. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata 3. Request - ,, + request body @TheNikhita
  15. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata 3. Request - ,, + request body 4. RequestResponse - ,, + response body @TheNikhita
  16. - level: Metadata resources: - group: "" resources: - secrets

    - configmaps - group: authentication.k8s.io resources: - tokenreviews Only log at Metadata level for sensitive resources @TheNikhita
  17. Log at RequestResponse level for critical resources Log at atleast

    Metadata level for all resources @TheNikhita
  18. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] @TheNikhita
  19. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Evaluated in top-down order @TheNikhita
  20. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Status calls can be large and high-volume @TheNikhita
  21. BACKEND LOG WEBHOOK • Writes events to disk • Sends

    events to external API @TheNikhita
  22. BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path

    • Sends events to external API • --audit-webhook-config-file @TheNikhita
  23. BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path

    • Sends events to external API • --audit-webhook-config-file --audit-policy-file @TheNikhita
  24. BATCHING BATCH BLOCKING BLOCKING-STRICT Buffers events & processes in batches

    Blocks APIserver responses to process individual events Failure at RequestReceived stage leads to failure of whole call @TheNikhita
  25. DYNAMIC AUDIT CONFIGURATION apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: mysink

    spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://audit.app" @TheNikhita
  26. SECURITY PERFORMANCE Write access to feature = Read access to

    all cluster data cluster-admin level privilege Increase in CPU/Memory Usage @TheNikhita
  27. CONCLUSION • Audit logs can give us a lot of

    information of what goes on in our cluster • To control what should be logged, we write audit policies • Recommendations for writing audit policies • Different audit backends • Batching methods • Dynamic Audit Configuration • Log Collector Patterns