Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Auditing in Kubernetes 101

Auditing in Kubernetes 101

Have you ever wondered who created particular changes in your cluster, when they created it or what resources were modified? All of such information about “what sequence of events lead to this scenario” can be obtained using the powerful audit logging feature. In this talk, we will first go over what audit logs are and how to leverage them to stay informed with what goes on in your cluster. Keeping both performance impact and accountability in mind, we will then walk through examples of policy configurations to enforce best security practices, detect misuse and make your cluster more compliant. We’ll also do a demo of setting up auditing on a cluster and inspecting the logs. Finally, we will see what future improvements are planned for this feature and how you can provide feedback and get involved.

Nikhita Raghunath

February 17, 2020
Tweet

More Decks by Nikhita Raghunath

Other Decks in Programming

Transcript

  1. WHO AM I • Software Engineer at Loodse • Member

    of the Kubernetes Steering Committee • Technical Lead for SIG Contributor Experience • CNCF Ambassador Github - nikhita Twitter - TheNikhita
  2. { "kind": "Event", "apiVersion": "audit.k8s.io/v1", "level": "Metadata", "auditID": "7684b057-7e2d-4188-a6ae-8fc51afd0c9d", "stage":

    "ResponseComplete", "requestURI": "/api/v1/namespaces/default/secrets", "verb": "create", "user": { "username": "minikube-user", "groups": [ "system:masters", "system:authenticated" ] }, "sourceIPs": [ "X.Y.Z.1" ], "objectRef": { "resource": "secrets", "namespace": "default", "name": "mysecret", "apiVersion": "v1" }, "responseStatus": { "metadata": {}, "code": 201 }, "requestReceivedTimestamp": "2020-02-12T18:06:04.577792Z", "stageTimestamp": "2020-02-12T18:06:04.584173Z", } AUDIT EVENTS @TheNikhita
  3. ON WHAT DID IT HAPPEN "objectRef": { "resource": "secrets", "namespace":

    "default", "name": "mysecret", "apiVersion": "v1" }, @TheNikhita
  4. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  5. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  6. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  7. AUDIT POLICY apiVersion: audit.k8s.io/v1 kind: Policy rules: - level: Metadata

    omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  8. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests @TheNikhita
  9. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed @TheNikhita
  10. WHEN TO LOG 1. RequestReceived - Audit handler receives request

    2. ResponseStarted - For long running requests 3. ResponseComplete - Response body completed 4. Panic - Event generated when panic occurs @TheNikhita
  11. WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:

    Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets GROUP/VERSION RESOURCE VERBS @TheNikhita
  12. WHAT TO LOG apiVersion: audit.k8s.io/v1 kind: Policy rules: - level:

    Metadata omitStages: - RequestReceived resources: - group: "" resources: - secrets @TheNikhita
  13. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata @TheNikhita
  14. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata 3. Request - ,, + request body @TheNikhita
  15. LEVELS 1. None - don’t log these requests 2. Metadata

    - only request metadata 3. Request - ,, + request body 4. RequestResponse - ,, + response body @TheNikhita
  16. - level: Metadata resources: - group: "" resources: - secrets

    - configmaps - group: authentication.k8s.io resources: - tokenreviews Only log at Metadata level for sensitive resources @TheNikhita
  17. Log at RequestResponse level for critical resources Log at atleast

    Metadata level for all resources @TheNikhita
  18. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] @TheNikhita
  19. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Evaluated in top-down order @TheNikhita
  20. rules: - level: RequestResponse resources: - group: "" resources: ["pods"]

    - level: Metadata resources: - group: "" resources: ["pods/log", "pods/status"] Status calls can be large and high-volume @TheNikhita
  21. BACKEND LOG WEBHOOK • Writes events to disk • Sends

    events to external API @TheNikhita
  22. BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path

    • Sends events to external API • --audit-webhook-config-file @TheNikhita
  23. BACKEND LOG WEBHOOK • Writes events to disk • --audit-log-path

    • Sends events to external API • --audit-webhook-config-file --audit-policy-file @TheNikhita
  24. BATCHING BATCH BLOCKING BLOCKING-STRICT Buffers events & processes in batches

    Blocks APIserver responses to process individual events Failure at RequestReceived stage leads to failure of whole call @TheNikhita
  25. DYNAMIC AUDIT CONFIGURATION apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: mysink

    spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://audit.app" @TheNikhita
  26. DYNAMIC AUDIT CONFIGURATION apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: mysink

    spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://audit.app" @TheNikhita
  27. DYNAMIC AUDIT CONFIGURATION apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: mysink

    spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://audit.app" @TheNikhita
  28. DYNAMIC AUDIT CONFIGURATION apiVersion: auditregistration.k8s.io/v1alpha1 kind: AuditSink metadata: name: mysink

    spec: policy: level: Metadata stages: - ResponseComplete webhook: throttle: qps: 10 burst: 15 clientConfig: url: "https://audit.app" @TheNikhita
  29. SECURITY PERFORMANCE Write access to feature = Read access to

    all cluster data cluster-admin level privilege @TheNikhita
  30. SECURITY PERFORMANCE Write access to feature = Read access to

    all cluster data cluster-admin level privilege Increase in CPU/Memory Usage Don’t use too many sinks @TheNikhita
  31. CONCLUSION • Audit logs can give us a lot of

    information of what goes on in our cluster • To control what should be logged, we write audit policies • Recommendations for writing audit policies • Different audit backends • Batching methods • Dynamic Audit Configuration • Log Collector Patterns