Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Enhance Kubernetes Security with Gatekeeper

moricho
April 07, 2021
950

Enhance Kubernetes Security with Gatekeeper

moricho

April 07, 2021
Tweet

Transcript

  1. 2 @_moricho_ (Morito Ikeda) Job - ~2020/10: BASE / AbemaTV

    / Cybozu... - 2020/10~: Mercari, Microservices Platform team (Platform Infra) - OSS: gVisor, Firecracker, ... Hobby - Camera, GoPro - Music (Bass, Drum)
  2. 3 Agenda • Platform Infra @ Mercari • Kubernetes Security

    • Enhance Platform Security with Gatekeeper • Promote DevSecOps • What's next
  3. 4 Microservice @ Mercari • Microservices are running on GKE

    ◦ Teams have ownership of their services • Provisioned with Terraform
  4. 5 Platform Infra @ Mercari • Enhance platform security, reliability,

    observability, … ◦ Developers can focus more on application logic ◦ Developer can work more productively • For example ◦ Enhance CI security ◦ Introduce HPA with External metrics / VPA ◦ Visualize Infra Cost ◦ etc • Enhancing kubernetes security is also our responsibility
  5. 6 Kubernetes Security • Container Image Scan (ex. Trivy) •

    Manifest Configuration (ex. Gatekeeper, Conftest) • Container Runtime (ex. gVisor) • etc Today, will focus on Manifest Configuration
  6. 7 Kubernetes Security What we need to be aware of

    when setting up manifests are: • Additional Linux Capabilities • Host Network • Host-path volumes • Privileged Containers • etc
  7. 8 Kubernetes Security • The number of developers and microservices

    are increasing ◦ Can’t put all security responsibility on the developers • It is our responsibility to ensure we have basic guardrails in our system. To resolve that, we introduced Gatekeeper
  8. 9 Enhance Platform Security with Gatekeeper What is Gatekeeper •

    General-purpose policy engine hosted by OPA (Open Policy Agent) • We can create policies in its own language called Rego ◦ For example ▪ Enforce pods to have resource limits ▪ Restrict the addition of Linux capabilities to containers ▪ Limit the container registries to only allowed certain ones
  9. 10 Enhance Platform Security with Gatekeeper What is Gatekeeper •

    Gatekeeper validates kubernetes resources based on policies ◦ Policies are deployed as CRD ◦ Gatekeeper is registered as Validation Webhook
  10. 11 Enhance Platform Security with Gatekeeper Security risks in containers

    and mitigations Policies that we implemented to enforce security best practices in our Kubernetes workloads: • Prevent adding extra capabilities • Restrict the usage of the host network • Prevent containers from running in privileged mode
  11. 12 Enhance Platform Security with Gatekeeper 1. Prevent adding extra

    capabilities • Docker has already blocked some capabilities such as SYS_ADMIN, NET_ADMIN, and so on ◦ but thoughtlessly adding other capabilities will have unexpected security risks ◦ Whitelist some components • Related CVE: CVE-2020-14386 ◦ Allows escape from a container to obtain root privileges ◦ This is related to CAP_NET_RAW ▪ Enables ARP (Address Resolution Protocol) spoofing attacks
  12. 13 Enhance Platform Security with Gatekeeper 1. Prevent adding extra

    capabilities • This policy checks PodSpec definitions in all Pod, Deployment, DaemonSet, and so on ◦ Checks `.spec.containers[].securityContext.capabilities.add`
  13. 14 Enhance Platform Security with Gatekeeper 2. Restrict the usage

    of the host network • Containers using the host network can do many things to jeopardize our cluster ◦ Allow processes in the pod to talk on the host’s loopback adapter ◦ Create port conflicts with other containers • Related CVE: CVE-2020-15257 ◦ containerd containers running in the host network namespaces with UID 0 allowed to gain the host root privileges
  14. 15 Enhance Platform Security with Gatekeeper 2. Restrict the usage

    of the host network • This policy checks PodSpec definitions in all Pod, Deployment, DaemonSet, and so on ◦ limits the value of `.spec.hostNetwork` to false
  15. 16 Enhance Platform Security with Gatekeeper 3. Prevent containers from

    running in privileged mode • Privileged mode allows container processes to access host devices ◦ Enables processes to behave like host processes (running outside a container) ◦ Allows bypassing almost all Kubernetes and Docker security controls
  16. 17 Enhance Platform Security with Gatekeeper 3. Prevent containers from

    running in privileged mode • This policy checks PodSpec definitions in all Pod, Deployment, DaemonSet, and so on ◦ limits the value of `.spec.containers[].securityContext.privileged` to false
  17. 18 Enhance Platform Security with Gatekeeper We have dozens of

    these Gatekeeper policies which are helping us in keeping security best practices • Related to ◦ host-path volumes ◦ host namespaces ◦ Istio ◦ node pool access ◦ etc
  18. 19 Promote DevSecOps Those works are based on the very

    concept of DevSecOps • Adds Continuous Security to DevOps • Spans the entire software development life cycle
  19. 20 Promote DevSecOps In our Gatekeeper example • Validation when

    deploying services to the Kubernetes in the development environment ◦ Automatic and continuous security checks during the deployment phase of the development cycle ◦ Can find and fix problems early on in your daily development ◦ Speeds up releases • Fundamental idea in DevSecOps: Shift-Left-Security
  20. 21 Promote DevSecOps Can also use Conftest in the CI

    level • It uses the Rego language for writing assertions
  21. 22 Promote DevSecOps • Asked the Security team to help

    us from the planning stage ◦ We proceeded with the plan under their review • The collaboration between our platform team and the security team is essential to promote DevSecOps
  22. 23 What's next • Not only security, Gatekeeper will become

    our platform wide guardrails ◦ Dedicated node-pool for a namespace ◦ Restrict developers from using alpha release features ◦ Force the developer to use a specific option for reliability or performance ▪ VPA: Enforces developers to specify a lower bound of request resources ▪ Enforce to write resource limits
  23. 24 Reference • Enhance Kubernetes Security with OPA Gatekeeper, mercari

    engineering • DevSecOps: What Is It and Why Is It Gaining Momentum in the Industry?, mercari engineering • open-policy-agent/gatekeeper, GitHub