systemd containers on OpenShift

"systemd in a container - what! why!?" We've got our Reasons, and I'll even explain them. But more interesting than the "why" is the "how", and that's what this talk is about. Come and learn about upcoming and already-delivered Kernel and Kubernetes security features that enable better container isolation and secure deployment of systemd-based workloads.

This is a talk about what happened when a handful of complete container newbies tried to port their massive, complex, "legacy" application to Kubernetes. In a single "monolithic" container. Based on systemd.

The container runtime shunned our application. Cloud engineers howled in dismay at our architecture decisions. Ultimately, like the hackers we are, we ignored their admonitions and doubled down. If the container runtime won't run our application, well, we'll just modify the container runtime!

And so we did. Our journey took us into the darkest corners of container runtimes, Kubernetes and systemd. And we have emerged to tell you the tale. There will be demos.

Attendees should expect to learn more about the security technologies that underpin Linux containers, including namespaces and cgroups, as well as the behaviour of systemd in containers.

Transcript

1. send in the chowns systemd containers on OpenShift Fraser Tweedale @hackuador January 15, 2022
2. Preliminaries CC-BY 4.0, except where otherwise noted Slides are available at speakerdeck.com/frasertweedale I will be available in the chatroom following the presentation
3. Agenda Containers and container standards Kubernetes and OpenShift FreeIPA: overview and use cases FreeIPA and systemd-based workloads on Kubernetes/OpenShift challenges, workarounds, solutions
4. What is a container? An process isolation and confinement abstraction Most commonly: OS-level virtualisation (shared kernel) e.g. FreeBSD jails, Solaris zones Container image defines filesystem contents
5. Containers on linux namespaces: pid, mount, network, cgroup, . . . (maybe) SELinux/AppArmor (maybe) restricted capabilities(7) or seccomp(2) profile
6. Container standards Open Container Initiative (OCI)1 Runtime Specification2 - low level runtime interface Linux, Solaris, Windows, VMs, . . . Implementations3: runc4 (reference implementation), crun5, Kata Containers6 1https://opencontainers.org 2https://github.com/opencontainers/runtime-spec 3https://github.com/opencontainers/runtime-spec/blob/main/implementations.md 4https://github.com/opencontainers/runc 5https://github.com/containers/crun 6https://katacontainers.io/
7. OCI Runtime Specification JSON configuration (example7) mounts, process and environment, lifecycle hooks, . . . Linux-specific: capabilities, namespaces, cgroup, sysctls, seccomp profile 7https://github.com/opencontainers/runtime-spec/blob/main/config.md#configuration-schema-example
8. Kubernetes and OpenShift
9. Kubernetes - container orchestration Abbreviation: “k8s” A container orchestration system Declarative configuration of container-based applications Integration with many cloud providers https://kubernetes.io/ https://github.com/kubernetes/
10. Kubernetes - terminology Container: isolated/confined process [tree] Pod: group (1+) of related Containers (e.g. HTTP app + database) Namespace: object and auth[nz] scope, such as for a team/project Node: a machine in the cluster; where Pods are executed
11. Kubernetes - more terminology Kubelet8: agent that executes Pods on Nodes Sandbox: isolation/confinement mechanism(s); one per Pod Container Runtime Interface (CRI)9: interface used by Kubelet to create/start/stop/destroy Sandboxes and Containers CRI-O10 containerd11 8https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ 9https://kubernetes.io/docs/concepts/architecture/cri/ 10https://cri-o.io/ 11https://containerd.io/
12. Kubernetes - Container Runtime Interface
13. Kubernetes - Container Runtime Interface - CRI-O
14. Kubernetes - Container Runtime Interface - CRI-O + runc
15. Kubernetes - Pod definition apiVersion: v1 kind: Pod metadata: name: fedora labels: app: fedora spec: containers: - name: fedora image : registry.fedoraproject.org/fedora :35- x86_64 command : [" sleep", "3600"] env : - name: DEBUG value: "1"
16. OpenShift13 a.k.a. OpenShift Container Platform (OCP) An enterprise-ready Kubernetes container platform Commercially supported by Red Hat Community “upstream” distribution: OKD12 Uses CRI-O and runc Latest stable release: 4.9 12https://www.okd.io/ 13https://openshift.com/
17. OpenShift - terminology All existing Kubernetes terminology, plus. . . Project: Extends the Namespace concept Security Context Constraint (SCC): policy affecting SELinux context, seccomp profile, capabilities, UID
18. OpenShift runtime environment (today) Sandboxes use SELinux, namespaces (cgroup, pid, mount, uts, network) Each Project gets assigned a unique UID range Containers run as a UID from that range Circumvent via RunAsUser and SCCs (bad idea)
19. FreeIPA
20. FreeIPA Open Source identity management solution Users, groups, services, authentication, access policies 389 DS (LDAP), MIT Kerberos, Apache, Dogtag PKI, SSSD, . . . Part of RHEL (commercial support) and Fedora (community support) https://www.freeipa.org/
21. FreeIPA on Kubernetes/OpenShift - use cases Identity services. . . for business applications running on the cluster for the cluster itself (API access, node access) for an entire organisation, hosted on their OpenShift cluster as a service, hosted and managed by a service provider
22. FreeIPA container Encapsulate the whole RHEL/Fedora-based system in a container PID 1 is systemd, which starts/manages all services We call this a monolithic container
23. Whyyyy?! Big engineering effort to rearchitect FreeIPA to be "cloud native" Ongoing costs as we support two different application architectures If we were starting from scratch today. . .
24. FreeIPA on OpenShift - challenges Unsurprisingly, there are many Main areas: runtime volumes and mounts ingress14,15 14https://frasertweedale.github.io/blog-redhat/posts/2021-11-18-k8s-tcp-udp-ingress.html 15https://frasertweedale.github.io/blog-redhat/posts/2020-12-08-k8s-srv-limitation.html
25. Challenges, workarounds and solutions
26. Runtime - user namespaces systemd and other components expect to run as root or other specific UID Solution: user_namespaces(7) Implemented in CRI-O, since OpenShift 4.7 Opt-in via Pod annotation Requires non-default cluster configuration Requires Pod to be admitted via anyuid (or similar) SCC16 16I am working on a way to avoid this
27. Runtime - user namespaces
28. Runtime - user namespaces apiVersion: v1 kind: Pod metadata: name: nginx labels: app: nginx annotations: io.openshift.builder: "true" io.kubernetes.cri-o.userns-mode: "auto:size=65536" spec: containers: - name: nginx image: quay.io/ftweedal/test -nginx:latest tty: true
29. Runtime - user namespaces - Kubernetes support KEP17-127: a long-running and ongoing discussion First proposal: https://github.com/kubernetes/enhancements/pull/1903 Second proposal: https://github.com/kubernetes/enhancements/pull/2101 Current proposal: https://github.com/kubernetes/enhancements/pull/3065 17Kubernetes Enhancement Proposal
30. Runtime - cgroups OpenShift creates a unique cgroup18 for each container cgroup namespace19 makes it the “root” namespace inside the container cgroupfs mounts it at /sys/fs/cgroup systemd needs write access. . . but doesn’t have it 18cgroups(7) 19cgroup_namespaces(7)
31. Runtime - cgroup ownership Solution: modify runtime to chown the cgroup to the container process UID But first: extend OCI Runtime Spec with semantics for cgroup ownership20 runc pull request21 Merged; release expected in OpenShift 4.11 or later 20https://github.com/opencontainers/runtime-spec/blob/main/config-linux.md#cgroup-ownership 21https://github.com/opencontainers/runc/pull/3057
32. Runtime - OCI cgroup ownership semantics chown container’s cgroup to host UID matching the process UID in container’s user namespace, if and only if. . . cgroups v2 in use, and container has its own cgroup namespace, and cgroupfs is mounted read/write
33. Runtime - OCI cgroup ownership semantics Only the cgroup directory itself, and the files mentioned in /sys/kernel/cgroup/delegate, should be chown’d: cgroup.procs cgroup.threads cgroup.subtree_control memory.oom.group22 22depends on kernel version
34. Runtime - cgroups v2 cgroups v2 is required for secure cgroup delegation it works, but is not yet the default cluster configuration it is on the roadmap
35. Runtime - cluster configuration (OCP 4.10) - 1/3 apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: name: enable -cgroupv2 -workers labels: machineconfiguration .openshift.io/role: worker spec: kernelArguments : - systemd. unified_cgroup_hierarchy =1 - cgroup_no_v1 ="all" - psi=1 ...
36. Runtime - cluster configuration (OCP 4.10) - 2/3 config: ignition: version: 3.1.0 storage: files : - path: /etc/subuid overwrite: true contents: source: data:text/plain;charset=utf -8; base64 , Y29 - path: /etc/subgid overwrite: true contents: source: data:text/plain;charset=utf -8; base64 , Y29 ...
37. Runtime - cluster configuration (OCP 4.10) - 3/3 systemd : units : - name: "rpm -overrides.service" enabled: true contents: | [Unit] Description=Install RPM overrides After=network -online.target rpm -ostreed.service [Service] ExecStart =/bin/sh -c ’rpm -q runc -1.0.3 -992. rhao || rpm-ostree override replace --reboot https :// ft Restart=on -failure [Install] WantedBy=multi -user.target
38. Demo
39. Links / resources Project main repo: https://github.com/freeipa/freeipa-openshift not much here yet, watch this space runc builds: https://ftweedal.fedorapeople.org/ Team blogs: https://frasertweedale.github.io/blog-redhat/tags/containers.html https://avisiedo.github.io/docs/ Demo: https://www.youtube.com/watch?v=OGAVvIJwmd0
40. Status and future Kubernetes: user namespaces support in an ongoing discussion OpenShift: systemd container in user namespace works, but experimental Official support is an open question We are hopeful, collaborating with OpenShift project and product management, looking for allies But we may end up having to rearchitect FreeIPA for the cloud
41. Elias Wicked Ales & Spirits https://www.facebook.com/wickedelias/posts/2967000120196980 Fair dealing for purpose of parody or satire © 2022 Red Hat, Inc. Except where otherwise noted this work is licensed under http://creativecommons.org/licenses/by/4.0/ Slides speakerdeck.com/frasertweedale Blog frasertweedale.github.io/blog-redhat Email ftweedal@redhat.com Twitter @hackuador