The Future of Linux Application Distribution: OSTree, Flatpak & GNOME Software

Fragmentation is a key word when thinking about the current landscape of Linux applications both referring to desktop environments and embedded devices. We have incompatible package formats, different package-management systems, multi-arch root filesystem layouts and device-dependent ways of flashing whole operating systems. We need to address the needs of a new breed of storage and network constrained consumer products.

OSTree is a tool that allows creating, updating and deploying filesystems in an efficient and atomic manner so that the device is guaranteed to always be in a bootable state, regardless of potential issues happening during the upgrade. Through the use of hard links, hashes and delta upgrades the target device does not retrieve or store duplicate content.

Flatpak is a related technology that delivers applications that are not part of the OS. Building on proven technologies such as OSTree, D-Bus, systemd and new features from the Linux kernel, Flatpak is able to provide a way to package, deploy and run Linux applications in secure sandboxed environments. It introduces the concept of "runtime platforms" that flatpak applications depend upon to run, and which make it possible to run a flatpak created in any Linux environment in a completely different Linux environment.

GNOME Software is the user-facing application that enables the end user to install, remove and update a wide variety of software components such as new version of the OS, application and firmware updates. With its plugin-based architecture, GNOME Software is able to handle any type of underlying technology that a given system has to use.

The purpose of this presentation will be to provide an overview of OSTree, Flatpak and GNOME Software, and how those three technologies interact with each other. Together we think they will shape the future of application distribution for Linux.

This talk has been delivered on 2017, March the 16th, in Samsung Research UK, as a joint effort between Richard Hughes (Red Hat) and me (Endless).

Transcript

1. The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software Richard Hughes <rhughes@redhat.com> Mario Sánchez Prada <mario@endlessm.com> Samsung Research UK. Staines, 2017 March 16th
2. About Mario Computer Science Engineer by the University of Coruña Open Source developer, GNOME Foundation member Previously worked at Igalia and Samsung Research UK Currently at Endless Computers, working in the Desktop team focused on the development of the core platform Your neighbour in Staines-upon-Thames since 2013 The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
3. OSTree The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
4. What is OSTree? Git-like system for complete & bootable filesystems Disk efficient: de-duplication via SHA256SUM hashes, check outs files from the object store via hard links Network efficient: static deltas, summary file Reliable updates & rollback: atomicity, no inconsistencies Safe: GPG verification for commits and summary file Introspectable library and command line tools Multiple use cases: OS deployment, efficient OTA updates, continuous integration & QA, bundled applications... https://ostree.readthedocs.io The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
5. Atomic & incremental upgrades Git-like fetching via HTTP: simple setup Incremental downloads of objects, using pre-generated static deltas when available Automatic verification of fecthed objects and deltas Automatic creation of new deployments (+ 3-way merge) Atomic swapping of boot configurations via symlinks $ ls -l /ostree/ total 12K lrwxrwxrwx 1 root root 8 Oct 4 16:55 boot.0 -> boot.0.1 drwxr-xr-x 3 root root 4.0K Oct 4 16:55 boot.0.1 drwxr-xr-x 3 root root 4.0K Oct 4 16:55 deploy drwxr-xr-x 7 root root 4.0K Mar 12 12:59 repo The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
6. Some internal details Anatomy of an OSTree repository: Types of repositories: bare, bare-user, archive-z2 Objects (commits, dirtree, dirmeta, content) + refs The summary file OSTree deployments: Multiple deployments per OS, parallel installable Shared stateful data among deployments (/etc, /var) The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
7. Comparison with other systems OSTree vs APT/RPM Deploying full filesystem VS partial ones Truly atomic VS potentially broken intermediate stages No dependencies hell, no postinst/postrm hooks... OSTree vs image replication (flashing) Predictable like flashing, but much more efficient Only 2 persistent directories supported: /etc & /var Works on top of any filesystems supporting hard links Supports installing different versions of the OS in parallel The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
8. Who is using OSTree? Atomic project (Fedora, CentOS) GNOME Continuous Qt OTA updates Automotive Grade Linux (AGL) Endless OS Flatpak The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
9. Flatpak The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
10. What is Flatpak? A new way of distributing applications in Linux Sits on top of OSTree and bubblewrap (chroot on steroids) Cross-platform by design: runtimes and applications Reliable and secure: GPG signatures, sandboxing Allows having both user and system-wide installations Open Source project. Started by Red Hat, contributions from Endless, Collabora, Codethink, Intel, Kinvolk, Solus... Similar in some ways to Docker, but with the focus on end user applications instead of for containerized system-wide services. http://flatpak.org The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
11. A brief note on bubblewrap Allows running sandboxed applications in chroot-like environments as an unprivileged user Creates a mount namespace with / on a tmpfs Uses PR_SET_NO_NEW_PRIVS when cloning the process to limit what the binary can do after dropping privileges Implements a subset of the Kernel’s user namespaces feature to isolate processes Supported namespaces: CLONE_NEWUSER, CLONE_NEWIPC, CLONE_NEWPID, CLONE_NEWNET, CLONE_NEWUTS Allows passing a list of seccomp filters to limit syscalls The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
12. Bubblewrap example [fedoravm ~]$ bwrap --ro-bind /usr /usr --ro-bind /etc/resolv.conf /etc/resolv.conf \ --symlink usr/lib /lib --symlink usr/lib64 /lib64 --symlink usr/bin /bin \ --dir /tmp --proc /proc --dev /dev \ --unshare-pid --unshare-net \ --chdir / \ /bin/sh sh-4.3$ ls / bin dev etc lib lib64 proc tmp usr sh-4.3$ ls /dev/ console full null ptmx pts random shm stderr stdin stdout tty urandom zero sh-4.3$ ls -l /etc/ total 4 -rw-r--r-- 1 65534 65534 53 Mar 14 00:46 resolv.conf sh-4.3$ ifconfig lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 1 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 sh-4.3$ ps aux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND 1000 1 0.0 0.0 15472 160 ? S 01:28 0:00 bwrap --ro-bind /usr /usr --r 1000 2 0.0 0.1 122136 3608 ? S 01:28 0:00 /bin/sh 1000 8 0.0 0.1 150020 3544 ? R+ 01:29 0:00 ps aux The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
13. Anatomy of a Flatpak Runtime $ tree -L 3 /var/lib/flatpak/runtime/org.gnome.Platform/x86_64/3.22/active/ |-- deploy |-- files | |-- bin | | |-- [...] | | |-- basename | | |-- bash | | |-- [...] | |-- etc | | |-- [...] | | |-- ca-certificates.conf | | |-- dbus-1 | | |-- [...] | |-- lib | | |-- [...] | | |-- libglib-2.0.so.0.5000.2 | | |-- libGL.so -> libGL.so.1.0.0 | | |-- [...] | |-- lib64 | | ‘-- ld-linux-x86-64.so.2 -> /usr/lib/ld-linux-x86-64.so.2 | |-- [...] | |-- manifest-base-1.json | |-- manifest.json | |-- sbin -> bin | |-- share | | |-- [...] | | |-- applications | | |-- [...] | ‘-- var | |-- cache | |-- lib | ‘-- run ‘-- metadata The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
14. Anatomy of a Flatpak Application $ tree -L 3 /var/lib/flatpak/app/org.gnome.Todo/current/active/ /var/lib/flatpak/app/org.gnome.Todo/current/active/ |-- deploy |-- export | ‘-- share | |-- applications | |-- dbus-1 | ‘-- icons |-- files | |-- bin | | ‘-- gnome-todo | |-- lib | | |-- debug | | |-- evolution-data-server | | |-- girepository-1.0 | | |-- gnome-todo | | |-- goa-1.0 | | |-- libcamel-1.2.so -> libcamel-1.2.so.59.0.0 | | |-- [...] | | ‘-- systemd | |-- manifest.json | ‘-- share | |-- appdata | |-- applications | |-- dbus-1 | |-- GConf | |-- gir-1.0 | |-- glib-2.0 | |-- icons | |-- locale | |-- pixmaps | ‘-- runtime ‘-- metadata The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
15. Putting all together: inside a running flatpak app /.flatpak-info [...] /app /run/user/1000 /app/bin /run/user/1000/Xauthority /app/lib /run/user/1000/app /app/share /run/user/1000/app/org.gnome.Todo [...] /run/user/1000/bus /bin /run/user/1000/dconf /dev /run/user/1000/dconf/user /dev/console /run/user/1000/doc /dev/full /run/user/1000/flatpak-info /dev/null [...] [...] /sbin /etc /sys [...] /sys/block /home/mario [...] /home/mario/.config /tmp /home/mario/.local/share/flatpak /tmp/.X11-unix /home/mario/.var/app/org.gnome.Todo /tmp/.X11-unix/X99 /lib [...] /lib64 /usr /local /usr/bin /proc /usr/share /proc/1 /usr/share/applications /proc/1/attr [...] [...] /var /run /var/cache /run/build /var/config /run/build-runtime /var/config/user-dirs.dirs /run/host [...] /run/systemd /var/data /run/user/1000 /var/run [...] /var/tmp The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
16. Platform and SDK Runtimes Two types of runtimes: Platform runtime: just the bits needed to run apps SDK runtime: platform + the necessary tools and files for development purposes (e.g. headers, debug symbols...) Currently two main standard runtimes available: Freedesktop runtime: contains a set of essential libraries and services: D-Bus, GLib, PulseAudio, X11, Wayland GNOME runtime: based on the Freedesktop runtime, adds libraries like GTK+, GStreamer or GVFS on top. A KDE runtime is currently under development too: https://github.com/KDE/flatpak-kde-runtime The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
17. The Sandbox Limited access to the host system by default: No access to processes outside the sandbox (namespaces) No access to the network, session bus and devices Controlled execution of certain syscalls (seccomp filters) Read-only access to the runtime and app (bind mounts) Read-write access to $HOME/.var/app/$APPID Controlled access to resources (cgroups) No access to host services (e.g. X/Wayland, system bus...) Flatpak’s sandbox is very limiting by default, but there are ways of dealing with that to run real-word applications... The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
18. Escaping the Sandbox: fine-grained permissions Easiest way to work with the sandbox is to open “holes” in it: Grant access to UNIX domain sockets: X.org, Wayland, PulseAudio, System and Sesssion D-Bus... Grant access to specific devices: dri, kvm Grant access to see, use and/or own specific D-Bus names Share specific subsystems with the host (network, IPC) Fine-grained permissions for filesystem access Define extensions for runtimes or applications (e.g. l10n) Combining all this enables makes it possible to run apps in a more controlled way, but it’s not very secure. The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
19. The manifest file A Flatpak manifest file (metadata): [Application] name=org.gnome.Calculator runtime=org.gnome.Platform/x86_64/3.20 sdk=org.gnome.Sdk/x86_64/3.20 command=gnome-calculator [Context] shared=network;ipc; sockets=x11;wayland; filesystems=xdg-run/dconf;~/.config/dconf:ro; [Session Bus Policy] ca.desrt.dconf=talk [Environment] DCONF_USER_CONFIG_DIR=.config/dconf [Extension org.gnome.Calculator.Locale] directory=share/runtime/locale subdirectories=true [Extension org.gnome.Calculator.Debug] directory=lib/debug The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
20. Escaping the sandbox: Portals High-level APIs to allow sandboxed apps request access Out-of-process services running on the host system Sandboxed apps communicate via D-Bus Different types of portals for different needs: NetworkMonitor, OpenURI, Filechooser, Documents, Printing, Geolocation, Screenshots, Notifications, Proxy... Using portals is safe: They don’t expose sensitive information from the host Portal-initiated operations are interactive an cancellable Split in UI-less frontend + desktop-specific backends: Currently backends for GTK+, with KDE work-in-progress. GLib & GTK+ include support for several portals since 3.22 The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
21. Flatpak Portals: GTK+ backends $ flatpak run org.gnome.PortalTest The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
22. Building a flatpak apps { "id" : "org.gnome.Todo", "branch" : "stable", "runtime" : "org.gnome.Platform", "runtime-version" : "3.22", "sdk" : "org.gnome.Sdk", "build-options" : { "cflags" : "-O2 -g", "cxxflags" : "-O2 -g", "env" : { "V" : "1" } }, "command" : "gnome-todo", "modules" : [ { "name" : "gnome-online-accounts", "config-opts" : [ "--disable-telepathy", "--disable-documentation", "--disable-backend" ], "sources" : [ { "url" : "https://download.gnome.org/sources/gnome-online-accounts/3.22/gnome-online- "sha256" : "aacce93a71bf5e687a45ae0d00f31ea0625ddd8143235d6d8c64c4ec21bbfa33", "type" : "archive" } ] }, [...] ---> More depedencies here The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
23. Building a flatpak apps (II) [...] { "name" : "gnome-todo", "sources" : [ { "url" : "https://download.gnome.org/sources/gnome-todo/3.22/gnome-todo-3.22.1.tar.xz "sha256" : "cb80f64f5edeeac7b221146d2203bd1bebc49d275b7a41e7a5418f409d9c74af", "type" : "archive" } ] } ], "cleanup" : [ "/include", "/lib/pkgconfig", "/share/pkgconfig", "/share/aclocal", "/man", "/share/man", "/share/gtk-doc", "/share/vala", "*.la", "*.a" ], "finish-args" : [ "--share=ipc", "--socket=x11", "--socket=wayland", "--share=network", "--talk-name=org.gnome.OnlineAccounts", "--talk-name=org.gnome.evolution.dataserver.AddressBook9", "--talk-name=org.gnome.evolution.dataserver.Calendar7", "--talk-name=org.gnome.evolution.dataserver.Sources5", "--talk-name=org.gnome.evolution.dataserver.Subprocess.Backend.*", "--filesystem=xdg-run/dconf", "--filesystem=~/.config/dconf:ro", "--talk-name=ca.desrt.dconf", "--env=DCONF_USER_CONFIG_DIR=.config/dconf" ] } The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
24. Application distribution Publish your local repository: build-export Export your app to an OSTree (archive-z2) repository You could publish this repository now over HTTP Sign everything: build-sign, build-update-repo Important to GPG-sign the commits and the summary file Allows using unencrypted HTTP (faster downloads) Recommended to create a dedicated GPG key Push to a production public repository: e.g. rsync Simple requirements: static files served over HTTP! Push it to your public server once you’re happy Order your commands wisely (avoid race conditions) The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
25. Application distribution (II) Configure your public repository appropriately: build-update-repo -title=<title> build-update-repo -default-branch=<branch> Provide efficient updates: Enable HTTP keep-alive in the server (lots of files) Use OSTree’s static-deltas feature (good for big files) Run build-update-repo everytime an app changes Generate application metadata for software centers: Generate AppStream data for each application in your repo: build-update-repo will put it an appstream branch Make sure your apps must export an AppData XML file! The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
26. Flatpak filetypes: .flatpakrepo and .flatpakref Installing an application: gnome-recipes.flatpakref [Flatpak Ref] Title=GNOME Recipes Name=org.gnome.Recipes Branch=master Url=https://matthiasclasen.github.io/recipes-releases/repo IsRuntime=False GPGKey=mQENBFis[...]Kpp5G2YW RuntimeRepo=https://sdk.gnome.org/gnome.flatpakrepo Comment=GNOME loves to cook Configuring flatpak “repositories”: gnome.flatpakrepo [Flatpak Repo] Title=Gnome Stable Runtimes Url=http://sdk.gnome.org/repo/ Homepage=https://www.gnome.org/get-involved/ Comment=The standard Gnome runtime used by most gnome apps Description=GNOME runtimes are released with each major release and contain the main GNOME platform libraries. At the moment they only receive minor bug fixing and security updates, but should be considered ABI stable and frozen. Icon=https://www.gnome.org/wp-content/themes/gnome-grass/images/gnome-logo.png GPGKey=mQENBFUU[...]15w8jmY= The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
27. Installing a flatpak application in one click GNOME Software, flatpak and .flatpakref files in action: The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
28. References: » OSTree documentation: https://ostree.readthedocs.io/en/latest » Project Atomic: https://www.projectatomic.io » GNOME Continuous: https://wiki.gnome.org/Projects/GnomeContinuous » Qt OTA updates: https://doc.qt.io/QtOTA » Automotive Linux: https://automotivelinux.org » Endless OS: https://endlessos.com » Bubblewrap: https://github.com/projectatomic/bubblewrap » Flatpak documentation: https://docs.flatpak.org/en/latest » Flatpak portals: https://github.com/flatpak/xdg-desktop-portal » Flatpak portals (GTK): https://github.com/flatpak/xdg-desktop-portal-gtk » Alex Larsson’s blog: https://blogs.gnome.org/alexl » Christian Hergert’s talk on Scale15x: https://hergert.me/talks/Flatpak-Scale-15x.pdf The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
30. GNOME Software The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software
49. Thank you! The Future of Linux Application Distribution OSTree, Flatpak & GNOME Software