Reinventing Container Linux for the Wasm Era (and More) with System Extensions

Presented at Open Source Summit North America 2024, 17 April 2024.
Video recording: https://www.youtube.com/watch?v=OpMik3XSCi8&list=PLbzoR-pLrL6poagnac0dQuTXcmNvUHVOj&index=93

Abstract:
More than 10 years ago, CoreOS introduced the concept of a "container-optimized Linux", inspired by the ChromeOS concept of an immutable, image-based desktop operating system, that was designed for just one thing: deploying containers. Since then the Linux community has evolved tools for building distributions. In particular, systemd recently introduced system extensions (or sysexts) which have been embraced by a number of distros as an elegant way to combine components at provisioning time. After reviewing this background and explaining the necessary concepts, this session will look at how sysexts can be applied in the context of a Container Linux to build an immutable but configurable operating system adapted to a range of use cases. We will look in particular at how the Flatcar project has leveraged sysexts to replace torcx for platform-specific images, custom container runtimes, and versions tailored for deploying Wasm-based applications. The concepts, however, come from the multi-vendor UAPI group, and promise to be the foundation for how many modern Linux distros manage the balance between configurability and stability at scale.

Transcript

1. Reinventing Container Linux for the Wasm Era (and More) with System Extensions Andrew Randall Principal PM Manager Azure Core Linux
2. So you’re about to provision a new Linux server…
3. So you’re about to provision a new Linux server…
4. The Linux distro dichotomy Kernel + systemd Thousands of included packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux ✅ Flexible, works for just about any application
5. The Linux distro dichotomy Kernel + systemd Thousands of included packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux Flexible, works for just about any application Sources: Palo Alto Neworks, Security Magazine
6. The Linux distro dichotomy Kernel + systemd Thousands of included packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux ✅ Flexible, works for just about any application ❌ Large attack surface area ❌ Manageability ❌ Snowflakes / config drift
7. The Linux distro dichotomy Kernel + systemd Thousands of included packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux Kernel + systemd Minimal (10s/100s) collection of packages Container workloads loaded at runtime Immutable filesystem Special Purpose Linux ✅ Minimal attack surface area ✅ Manageability at scale ✅ Repeatable deployments
8. The Linux distro dichotomy Kernel + systemd Thousands of included packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux Kernel + systemd Minimal (10s/100s) collection of packages Container workloads loaded at runtime Immutable filesystem Special Purpose Linux Minimal attack surface area Manageability at scale Repeatable deployments
9. The Linux distro dichotomy Kernel + systemd Thousands of included packages Tens of thousands of additional optional packages Fully mutable filesystem General Purpose Linux Kernel + systemd Minimal (10s/100s) collection of packages Container workloads loaded at runtime Immutable filesystem Special Purpose Linux ✅ Minimal attack surface area ✅ Manageability at scale ✅ Repeatable deployments ❌ Inflexible - advanced knowledge required to modify base image
10. What if there were a better way…
11. Composable (Image-based) Linux Kernel + systemd Thousands of included packages Tens of thousands of additional optional packages Fully mutable filesystem Kernel + systemd OS extension layers loaded at boot time Container, Wasm modules, etc., loaded at runtime Immutable filesystem General Purpose Linux Composable Linux ✅ Minimal attack surface area ✅ Manageability at scale ✅ Repeatable deployments ✅ Easy to create custom OS flavors from composable system extension layers Kernel + systemd Minimal (10s/100s) collection of packages Container workloads loaded at runtime Immutable filesystem Special Purpose Linux
12. Anatomy of a System Extension (sysext) /usr /opt An overlay file system containing /usr & /opt Packaged as a disk image* Loaded at boot time by systemd-sysext * typically; can also be plain directory or btrfs subvolume https://www.freedesktop.org/software/systemd/man/latest/systemd-sysext.html sysext
13. Flatcar has embraced sysext /oem Torcx Replacement / Custom Container Runtimes OEM Partition Cluster API
14. Recent Applications in Flatcar Container Linux: 1) Torcx Replacement / Custom Container Runtimes ž torcx (from CoreOS) ž custom, tarball-based, complex, inflexible ž No behavior change for default (e.g. Docker, containerd) ž Easily add new runtimes (e.g. Podman) alongside or replacing standard ones sysext sysext sysext
15. Recent Applications in Flatcar Container Linux: 2) OEM Partition ž Separate partition fixed at build time for platform- specific tools/agents ž Not upgradeable without reprovisioning entire node ž Sysext for each target platform ž In-place upgrades /oem sysext sysext sysext
16. Recent Applications in Flatcar Container Linux: 3) Cluster API ž Custom worker node images combine OS + K8s control plane ž K8s + OS versions tied ž No in-place updates ž K8s control plane as sysext ž Stock distro images ž OS + K8s versions decoupled ž In-place updates sysext
17. Creating Sysexts: the Flatcar Sysext Bakery files to bake + config + metadata https://github.com/flatcar/sysext-bakery/blob/main/README.md bake.sh sysext image (.raw) sysext Kind of like docker build Kind of like your dockerfile
18. Publishing sysexts Create checksum upload sysext image + checksum + update conf to http endpoint (e.g. GitHub as part of build pipeline) Kind of like docker push sha256sum *.raw > SHA256SUMS CONF Create update conf file (optional)
19. Baked Goods, Ready to Consume https://github.com/flatcar/sysext-bakery/releases/tag/latest • docker • docker-compose • kubernetes • wasmcloud* • wasmtime* • cri-o (PR in progress) • k3s (PR in progress) * we’ll come to these later
20. A Brief Detour into Flatcar provisioning YAML Butane config (human readable) Butane transpiler JSON Ignition config (machine readable) This is where we want to specify the sysext(s) to use https://coreos.github.io/butane/
21. Provisioning Flatcar with a Sysext variant: flatcar version: 1.0.0 storage: files: - path: /opt/extensions/wasmtime/wasmtime-17.0.1-x86-64.raw contents: source: https://github.com/flatcar/sysext-bakery/releases/download/latest/wasmtime-17.0.1-x86-64.raw links: - target: /opt/extensions/wasmtime/wasmtime-17.0.1-x86-64.raw path: /etc/extensions/wasmtime.raw hard: false YAML
22. What about updates? OS-independent sysexts OS-dependent sysexts OS images • E.g. standalone go binary, no OS dependencies • systemd-sysupdate • simple semver based mechanism over https • Needs to update in lockstep with OS due to dependencies • Use OS update mechanism • Flatcar update server (Nebraska) extended to support sysexts • Sysexts part of OS image à updated as part of OS update
23. Configuring for Updates of OS-independent Sysexts variant: flatcar version: 1.0.0 storage: files: - path: /opt/extensions/wasmtime/wasmtime-17.0.1-x86-64.raw contents: source: https://github.com/flatcar/sysext-bakery/releases/download/latest/wasmtime-17.0.1-x86-64.raw - path: /etc/sysupdate.wasmtime.d/wasmtime.conf contents: source: https://github.com/flatcar/sysext-bakery/releases/download/latest/wasmtime.conf links: - target: /opt/extensions/wasmtime/wasmtime-17.0.1-x86-64.raw path: /etc/extensions/wasmtime.raw hard: false systemd: units: - name: systemd-sysupdate.timer enabled: true - name: systemd-sysupdate.service dropins: - name: wasmtime.conf contents: | [Service] ExecStartPre=/usr/lib/systemd/systemd-sysupdate -C wasmtime update - name: sysext.conf contents: | [Service] ExecStartPost=systemctl restart systemd-sysext YAML
24. What if I don’t want to pull the image at runtime? bake_flatcar_image.sh sysext(.raw) new flatcar image including sysext https://github.com/flatcar/sysext-bakery?tab=readme-ov-file#baking-sysexts-into-flatcar-os-images // Create a qemu image (latest stable) with pre-baked wasmcloud bake_flatcar_image.sh --fetch --vendor qemu_uefi wasmcloud:wasmcloud-0.82.0-x86-64.raw
25. Putting it all together: Wasm-Optimized Linux Wasm = Web Assembly By default, provably secure sandbox Most languages compile to it Runs on most OSes, architectures VERY small size, super fast start Wasm modules run in a Wasm runtime
26. Putting it all together: Wasm-Optimized Linux Kernel + systemd Core wasm utils loaded during init or baked into image Wasm modules loaded at runtime Immutable filesystem Wasm-Optimized Linux No Docker sysext active – no docker binaries in OS! Worth noting that if you disable a sysext, the binaries disappear from the OS file system. Might be important for e.g. compliance.
27. So many Wasm runtimes and tools to choose from Lunatic Modsurfer nerdctl runwasi SpiderLightning (slight) Spin wamr (wasm-micro- runtime) wasm3 WasmCloud WasmEdge Runtime wasmtime WaVe WaZero
28. More Wasm Goodness in Ralph’s Bakery https://github.com/squillace/sysext-bakery This is a great playground for all: feel free to submit additional sysexts here or upstream Flatcar sysext-bakery for mature projects
29. Takeaways • systemd-sysext is a promising new way to compose custom Linux distros • Immutable + minimal (with all the benefits that brings), but also flexible • Flatcar has already embraced as the way forward for enabling flexible deployments and customization • Great platform for production environments for Wasm and more
31. Get involved! CNCF Special Purpose OS Working Group tag-runtime.cncf.io/wgs/spos The Linux Userspace API (UAPI) Group uapi-group.org Flatcar Container Linux Project github.com/flatcar/Flatcar