Share
Embed
2018 Kubecon NA: How Symlinks Pwned Kubernetes
We go into technical details of the exploit and fix for CVE-2017-1002101
Michelle Au
December 11, 2018
Transcript
71 slides
-
How Symlinks Pwned K8s Michelle Au, Software Engineer, Google Jan
Šafránek, Software Engineer, Red Hat -
Reporting Github issue created 2017-11-30 PodSecurityPolicy can be sidestepped with
innocent emptyDir and subpath Here is a pod which would be allowed by fairly strict security policies, yet gives full control over node host by gaining access to docker socket: … -
Reporting Github issue created 2017-11-30 PodSecurityPolicy can be sidestepped with
innocent emptyDir and subpath Here is a pod which would be allowed by fairly strict security policies, yet gives full control over node host by gaining access to docker socket: … -
Reporting That’s not how it’s done! Follow https://kubernetes.io/docs/reference/issues-security/security/ • Responsibly
disclose to allow time to fix before public disclosure • [email protected] (optionally GPG encrypted) • Product Security Team handles the rest ◦ Evaluate impact ◦ Request CVE ◦ Coordinate development of fix, release, disclosure -
Volumes Background Node kind: Pod spec: containers: - name: my-container
volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} -
Volumes Background Node kind: Pod spec: containers: - name: my-container
volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} -
Volumes Background Node kind: Pod spec: containers: - name: my-container
volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Pod volume’s root path (host): /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume my- volume -
Volumes Background Node kind: Pod spec: containers: - name: my-container
volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Final host’s path: <pod volume’s root path> + <subpath> my- volume -
Volumes Background Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec:
containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} my- volume -
Volumes Background Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec:
containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker my- volume -
Volumes Background Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec:
containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Bind mount my- volume -
Vulnerability? Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers:
- name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Bind mount my- volume -
Vulnerability Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers:
- name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Created by user! Bind mount my- volume -
Vulnerability Node my- volume Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod
spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink -
Vulnerability Node my- volume Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod
spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS -
Vulnerability Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers:
- name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Symlink Root FS Bind mount my- volume -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath volumePath: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume subPath: data1 -
Naive Solution before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 volume subpath 1. Resolve all symlinks
2. Validate resolved path is inside volume 3. Give to CRI Subpath -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: / -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: / ❌ -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 ✅ -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅ -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath Race condition! User can still change anything to a symlink -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅ -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅ $ rmdir /mnt/my-volume/a/b/c $ ln -s / /mnt/my-volume/a/b/c -
Naive Solution 1. Resolve all symlinks 2. Validate resolved path
is inside volume 3. Give to CRI Subpath data1: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c data1: / ✅ $ rmdir /mnt/my-volume/a/b/c $ ln -s / /mnt/my-volume/a/b/c -
What Now? Need to “lock” directory between validation and CRI
• Windows: lock using CreateFile • Linux: bind mount -
Bind mount • Remount part of the file hierarchy somewhere
else • Independent on the original hierarchy • Atomic $ mount --bind \ /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c \ /var/lib/kubelet/safe/place -
Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved
path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place -
Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved
path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c -
Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved
path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c ✅ -
Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved
path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place $ mount --bind \ /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c \ /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0 safe place -
Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved
path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place $ docker -v /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0:/mnt/data -
Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved
path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place $ docker -v /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0:/mnt/data (was)$ -v /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1:/mnt/data -
Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved
path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place Safe -
Naive Bind-mount Solution 1. Resolve all symlinks 2. Validate resolved
path is inside volume 4. Give bind mount to CRI Subpath 3. Bind mount to safe place Race condition! User can still change anything to a symlink Race condition! User can still change anything to a symlink Safe -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD before: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 after: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/”) = 10 -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/”) = 10 openat(10, “a”, O_NOFOLLOW) = 11 -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/”) = 10 openat(10, “a”, O_NOFOLLOW) = 11 openat(11, “b”, O_NOFOLLOW) = 12 -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Goal: safely open /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c open(“/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/”) = 10 openat(10, “a”, O_NOFOLLOW) = 11 openat(11, “b”, O_NOFOLLOW) = 12 openat(12, “c”, O_NOFOLLOW) = 13 -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD $ ls -la /proc/<pidof kubelet>/fd/13 13 -> /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c -
$ ls -la /proc/<pidof kubelet>/fd/13 13 -> /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/a/b/c $ mount
--bind /proc/<pidof kubelet>/fd/13 \ /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0 Final Solution 1. Resolve all symlinks 2. Safely open FD to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD $ docker -v /var/lib/kubelet/pods/<uid>/volume-subpaths/<container name>/<volume name>/0:/mnt/data -
Final Solution 1. Resolve all symlinks 2. Safely open FD
to subpath, disallowing symlinks and validating path 5. Give bind mount to CRI Subpath 3. Bind mount opened FD 4. Close FD Safe (bind mount) Safe (no symlinks, inside container) -
Development github.com/kubernetes-security/kubernetes • Private, very limited access ◦ Access revoked
once fix released • Similar process as kubernetes/kubernetes ◦ Same test jobs ◦ Logs in private buckets ◦ Be careful with git pushes • Coordinated by Product Security Team -
Pre-Disclosure • Coordinated by Product Security Team • Kubernetes release
and branch managers • 3rd party Kubernetes vendors (“Private Distributors List”) ◦ Patch and test under embargo -
Public Disclosure 2018-03-12: CVE-2017-1002101 announced Kubernetes 1.7, 1.8, 1.9 quickly
patched and released Post-mortem document -
Secure Practices Now Don’t run containers as root user •
Use PodSecurityPolicy to require pods run as non-root • Caveat: containers still run as root gid ◦ K8s 1.10: RunAsGroup alpha feature Use PodSecurityPolicy to restrict volume access • Whitelist allowed volume types • Restrict both allowed HostPath prefixes and readOnly (K8s 1.11) -
Vulnerability Node Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers:
- name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Docker Container my- volume’ Symlink Root FS Bind mount my- volume -
Node Sandbox Future: Sandboxed Subpath? my- volume Final host’s path:
/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind: Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS -
Node Sandbox Future: Sandboxed Subpath? Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind:
Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS Subpath Handling my- volume -
Node Sandbox Future: Sandboxed Subpath? Final host’s path: /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~empty-dir/my-volume/data1 kind:
Pod spec: containers: - name: my-container volumeMounts: - name: my-volume mountPath: /mnt/data subPath: data1 volumes: - name: my-volume emptyDir: {} Symlink Root FS Container my- volume’ Subpath Handling my- volume -
Summary Follow the Kubernetes security disclosure process Be extra cautious
when handling untrusted paths • symlink race • time of check to time of use Set restrictive policies and use multiple security boundaries -
Get Involved! Kubernetes product security team Secure Container Isolation •
Overall: sig-node • Volumes: sig-storage • “Recent Advancements in Container Isolation” - today 1:45pm