Container-related technologies supporting Gitpod

https://runtime.connpass.com/event/253798/

Transcript

1. Container-related technologies supporting Gitpod Container Runtime Meetup #4 Toru Komatsu(@utam0k) 1
2. Toru(うたもく), Engineer at Gitpod utam0k utam0k 2
3. Today you will learn Ideas for applying container-related tech to your services Not just using Kubernetes and Docker 3
4. 01 What is Gitpod? Table of contents 02 User workspace environments on Kubernetes pod 03 Faster image pulls with IPFS 04 Summary 05 Please give me your questions! 4
5. What is Gitpod? 01 5
6. https://gitpod.new Always ready to code. 6
7. Open Source We're free from big tech influence and integrate, not dictate. This makes our product development fast and close to our users & community. SaaS or Self-Hosted Use our SaaS solution running on the carbon neutral Google Cloud Platform or host Gitpod on your own cloud infrastructure using GKE, k3s, EKS or AKS. Secure by design Gitpod centralizes all source code and safely stores it in the cloud, never locally. Security is at the core of everything we do at Gitpod. 7
8. User workspace environments on Kubernetes pod 02 8
9. automated dev enviroments in Kubernetes Gitpod provides workspace pod​ workspace pod​ workspace pod​ workspace pod​ Node workspace pod​ 9
10. These are table stakes for developers. Granting root privileges to users? Isn't this dangerous? It can be done safely using rootless containers. no sudo, no apt-get, no Docker 10
11. a rough sketch Node DaemonSet workspace container How can we make it work? 11
12. a rough sketch Node DaemonSet workspace container ring0 ring1 PID 1 PID 2 User Namespace How can we make it work? ring2 PID 3 PID Namespace 12
13. a rough sketch Node DaemonSet workspace container ring0 ring1 PID 1 PID 2 User Namespace writeMapping(pid: 2) hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? ring2 PID 3 PID Namespace 13
14. workspace's user namespace node 0 → 10000 33333 → 43333 14
15. a rough sketch Node DaemonSet workspace container ring0 ring1 PID 1 PID 2 User Namespace writeMapping(pid: 2) hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? ring2 PID 3 PID Namespace 15
16. a rough sketch Node DaemonSet workspace container ring0 ring1 mount($mark, "/newroot/", "shiftfs") mount("proc", "/newroot/proc", "proc") … PID 1 PID 2 mount($containerRootFS, "shiftfs", "mark") User Namespace writeMapping(pid: 2) prepareUserNS() hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? ring2 PID 3 PID Namespace 16
17. workspace's user namespace node 0 → 10000 33333 → 43333 17
18. a rough sketch Node DaemonSet workspace container ring0 ring1 mount($mark, "/newroot/", "shiftfs") mount("proc", "/newroot/proc", "proc") … PID 1 PID 2 mount($containerRootFS, "shiftfs", "mark") User Namespace writeMapping(pid: 2) prepareUserNS() hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? ring2 PID 3 PID Namespace 18
19. a rough sketch Node DaemonSet workspace container ring0 ring1 ring2 mount($mark, "/newroot/", "shiftfs") mount("proc", "/newroot/proc", "proc") … PID 1 PID 2 PID 3 pivot_root("/newroot") mount($containerRootFS, "shiftfs", "mark") User Namespace PID Namespace writeMapping(pid: 2) prepareUserNS() hostPID := translatePID(pid: 2) write(/proc/$hostPID/uid_map) write(/proc/$hostPID/gid_map)​ How can we make it work? 19
20. Linux Namespace layering 20
21. Linux Namespace layering 21
22. Ok, now the developers have root privileges, can Docker work on a workspace? No, why? mount proc, capabilities… 22
23. mount proc with seccomp notify Node DaemonSet workspace container User Namespace PID Namespace How can we make it work? mount -t proc proc /proc ring1 23
24. mount proc with seccomp notify Node DaemonSet workspace container User Namespace PID Namespace How can we make it work? seccomp agent​ mount -t proc proc /proc mountProc($pid) ring1 24
25. mount proc with seccomp notify Node DaemonSet workspace container User Namespace PID Namespace How can we make it work? seccomp agent​ mount -t proc proc /proc mountProc($pid) hostPID := translatePID(pid: $pid) target := filepath.Join($WsRoot, "/proc") unix.Mount("proc", target, "proc") ring1 25
26. Dynamic/Static Resource limiting for pods CPU cpu.max cpu.stat Process pids.max IO io.max And more… 26
27. workspace pod​ workspace pod​ Node Daemons at each node govern the workspace of that node. It periodically monitors the resource usage of the workspace and updates the cgroup values to ensure efficient resource utilization. Dynamic/Static resource limiting DaemonSet ① Monitor ① Usage ③ Change ③ cgroup values containerd ② Ask a cgroup root 27
28. Faster image pulls with IPFS 03 28
29. Gitpod users can use custom images. This means that a variety of images are built and pulled, which is unpredictable from our developer's point of view. When starting up a new workspace, a lot of time is spent on image pulls. Why did Gitpod need to speed up image pulls? 29
30. IPFS(InterPlanetary File System) enables us P2P and content-addressable data sharing 30
31. P2P image distribution by IPFS When pulling, images can be retrieved from containerds without a registry such as the Docker Registry What is IPFS with containerd? https://medium.com/nttlabs/nerdctl-ipfs-975569520e3d 31
32. Manifest for image not using IPFS IPFS urls are added to the urls field Pull image once to use IPFS 32
33. enables us to ship many different things in one OCI image. registry-facade 33
34. enables us to ship many different things in one OCI image. registry-facade user's workspace image 34
35. user's workspace image web IDE desktop IDE workspacekit supervisor docker-up, runc-facade Put Gitpod's tools on the user's image to build a workspace enables us to ship many different things in one OCI image. registry-facade 35
36. sha256:63c395644fe1767284082… sha256:3353847b14ebfc8fd1fd3…  sha256:1bd9d3b7686a1c61c4d6d… sha256:a58c6b717b32ed3061fc6… sha256:b50a80767b4a9335fbb11… sha256:f82b95de3049e4a2fb6ab… sha256:dece7c53616378469e462… sha256:2bcb17063a1c100b4a6fd… sha256:e1caa81aa0d0cc390b573… sha256:3d92e3e7e06dce602725c… sha256:c186ba26dc778921dd494… sha256:7b193551031047df6cba1… sha256:1cf47bc524f71ccca20e9… sha256:08c01a0ec47e82ebe2bec… sha256:e9444ec2d0d74706e9f21… … sha256:695bc3e4c4edea7f91799… sha256:764d7a91d4d599a243d1a… Put Gitpod's tools on the user's image to build a workspace enables us to ship many different things in one OCI image. registry-facade 36
37. workspace manager kubelet containerd registry- facade registry workspace manager kubelet containerd registry- facade registry pull reg.gitpod.io/remote/<instanceId> GET /v2/remote/manifests/<instanceId> GetImageSpec(instanceId) GET /v2/workspace-image/manifest/latest GET /v2/workspacekit/manifest/some_version GET /v2/docker-up/manifest/some_version 37
38. workspace manager kubelet containerd registry- facade registry workspace manager kubelet containerd registry- facade registry pull reg.gitpod.io/remote/<instanceId> GET /v2/remote/manifests/<instanceId> GetImageSpec(instanceId) GET /v2/workspace-image/manifest/latest GET /v2/workspacekit/manifest/some_version GET /v2/docker-up/manifest/some_version gitpod's original component 38
39. workspace manager kubelet containerd registry- facade registry workspace manager kubelet containerd registry- facade registry pull reg.gitpod.io/remote/<instanceId> GET /v2/remote/manifests/<instanceId> GetImageSpec(instanceId) GET /v2/workspace-image/manifest/latest GET /v2/workspacekit/manifest/some_version GET /v2/docker-up/manifest/some_version OCI distribution spec 39
40. registry- facade registry registry- facade registry … IPFS IPFS containerd GET /v2/remote/manifests/<instanceId> containerd Redis Redis GET /2/workspace-image/manifest/latest 40
41. registry- facade registry registry- facade registry … IPFS IPFS containerd GET /v2/remote/manifests/<instanceId> containerd Redis Redis GET /2/workspace-image/manifest/latest Does each layers exists on IPFS? Reply the CID if it exists on IPFS 41
42. registry- facade registry registry- facade registry … IPFS IPFS containerd GET /v2/remote/manifests/<instanceId> containerd Redis Redis GET /2/workspace-image/manifest/latest modify the manifest for IPFS Reply the modified manifest GET Object from the urls field Does each layers exists on IPFS? Reply the CID if it exists on IPFS 42
43. Manifest for image not using IPFS IPFS urls are added to the urls field Pull image once to use IPFS 43
44. Fast Image Pulls Using IPFS And Opportunistic Caching @KubeCon US 2022 Christian Weichel & Manuel de Brito Fontes, Gitpod 44
45. Summary 04 45
46. Summary Thanks! ☑ ☑ ☑ ☑ ☑ ☑ User workspace environments on Kubernetes pod namespace - rings cgroup - dynamic resource limits seccomp notify - mount procfs Faster image pulls with IPFS contained on IPFS OCI Image spec Image pull intercept 46
47. Any questions? 05 47