(mnt) ◦ Process ID (pid) ◦ Network (net) ◦ Interprocess Communication (ipc) ◦ Unix Time-Sharing (uts) ◦ User ID (user) • Control groups (cgroup) • OverlayFS • … etc. → Let's find out what these are. How are containers isolated?
<NEWROOT> <COMMAND> chroot Sets an isolated root directory (a new root path) for a process and its children How are containers isolated? (1) Isolating root directory (chroot)
specified process. This process is running on “cgroup”, “user” namespaces of the init process. Isolated namespaces for this process Linux Namespace → A kernel feature to isolate system resources between processes How are containers isolated? (2) Linux Namespaces
a process(/bin/bash) with an isolated mount namespace(-m option). $ unshare -m /bin/bash # Run a process (/bin/bash) with isolated mount (-m) and IPC (-i) namespaces $ unshare -m -i /bin/bash How are containers isolated? (2) Linux Namespaces
big file tree, to make it accessible in Unix systems. # mount -t <type> <device> <dir> # e.g. Mount “tmpfs”(temporary file storage) into “/root/test” $ mount -t tmpfs tmpfs /root/test How are containers isolated? (3) Mount (mnt) namespace
$$ 2222 $ mkdir -p test && mount -t tmpfs tmpfs /root/test $ df | grep test tmpfs 2.0G 0 2.0G 0% /root/test $ exit $ df | grep test Show the current process ID Run `/bin/bash` with an isolated mount namespace(`-m` option) Exit `/bin/bash`(isolated mount namespace) and then check the file systems. The file system mounted into `/root/test` shouldn’t be visible. Mount tmpfs(temporary file storage) into `/root/test` Mount namespace Allows processes to have different mount points How are containers isolated? (3) Mount (mnt) namespace
inside the container, it looks like a virtual machine. But from the outside (host), it's just a process. How are containers isolated? (4) Process ID (pid) namespace
(-p option). $ unshare -f -p /bin/bash $ echo $$ # Show the current PID 1 Inside a container running on an isolated PID namespace, the first executed process (the entrypoint) always has the PID of 1. How are containers isolated? (4) Process ID (pid) namespace
IPC ◦ Shared memory(shm) ◦ Semaphores ◦ POSIX message queues(/proc/sys/fs/mqueue) • IPC objects are visible only to the processes on the same namespace How are containers isolated? (5) Inter-Process Communication (ipc) namespace
a network namespace named `test-ns` $ ip netns add test-ns $ ip netns list test-ns # Create a virtual ethernet interface pair (veth1, veth2) # Add `veth1` to `test-ns` namespace, and `veth2` to the network namespace of PID 1 $ ip link add veth1 netns test-ns type veth peer name veth2 netns 1 How are containers isolated? (6) Network (net) namespace
link list” to list network interfaces. # In `test-ns`, there are only `veth1` and loopback interfaces. $ ip netns exec test-ns ip link list 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 8: veth1@if7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 2a:aa:60:ee:27:d4 brd ff:ff:ff:ff:ff:ff link-netnsid 0 # On the host default network namespace, run the command “ip link list”. $ ip link list 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 (… omit …) 7: veth2@if8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether d2:a1:90:78:3c:4b brd ff:ff:ff:ff:ff:ff link-netnsid 0 How are containers isolated? (6) Network (net) namespace Isolates network interfaces, routing tables, and firewall rules.
container2 External Network Containers are running on an isolated network namespace, connected to the host via veth peers. The veths are connected to the “docker bridge” on the host. To communicate with the outside world, the containers have to go through the bridge. How are containers isolated? (6) Network (net) namespace
resources among multiple users. Multiple users are sharing the same machine, but we want to make them feel like they're using separate machines. How are containers isolated? (7) Unix Time-Sharing (uts) namespace
Unix Time-Sharing? How are containers isolated? (7) Unix Time-Sharing (uts) namespace Multiple users are sharing the same machine, but we want to make them feel like they're using separate machines. This comes from the concept of sharing computing resources among multiple users.
$ hostname hyojun $ exit $ hostname ubuntu Run /bin/bash on an isolated UTS namespace Change hostname to “hyojun” and then show the current hostname Exit bash, and then show the current hostname. = it keeps the original hostname(ubuntu) (The hostname was changed only in the process where the UTS namespace was isolated.) Show the current hostname hostname, domainname isolation How are containers isolated? (7) Unix Time-Sharing (uts) namespace
(Isolated user namespace) root(uid:0) hyojun(uid:1000) This user is root inside the container, but non-root on the host. How are containers isolated? (8) User ID (user) namespace
its containers, the users inside the containers can exercise the authority of the same uid on the host. Host Container (Non-isolated user namespace) root(uid:0) root(uid:0) How are containers isolated? (8) User ID (user) namespace
add a user to docker group so that non-root users can run docker This is the command you've run at least once after installing Docker. How are containers isolated? (8) User ID (user) namespace
add a user to docker group so that non-root users can run docker This is the command you've run at least once after installing Docker. WARNING! How are containers isolated? (8) User ID (user) namespace
bound host root directory through Docker. This is because the container root uid 0 has the same uid on the host. non-root$ docker run -ti -v /:/host ubuntu:18.04 /bin/bash If you bind the host's “/” root directory to the container… How are containers isolated? (8) User ID (user) namespace
Compatibility issues with external volumes or drivers that do not support user mapping. • The complexity of ensuring access rights for the files bound from the host, if the host uid and the container uid differ. • However, while the root on a unisolated user namespace has almost the same permissions as the host's root, it does not include all permissions. Why doesn't Docker isolate the user namespace? How are containers isolated? (8) User ID (user) namespace
good article to read on this topic: https://kinvolk.io/blog/2020/12/improving-kubernetes-and-container-security-with-user-namespaces/ How are containers isolated? (8) User ID (user) namespace
users to run the container runtime (e.g. Docker). • Make sure that the container's processes do not run as the root user. ◦ Assign a specific user and group to run processes • Do not bind any of the host’s important directories, to prevent containers from accessing them. Kubernetes provides security settings based on the same principles. https://kubernetes.io/docs/concepts/policy/pod-security-policy/#users-and-groups How are containers isolated? (8) User ID (user) namespace
among process groups • CPU • Memory • Network • Disk Limit CPU and memory usage... Prioritize network traffic, ... Provide statistics on usage, or etc. How are containers isolated? (9) Control group (cgroup)
environment. • Isolated environments are implemented through Linux namespaces. ◦ Mount (mnt) ◦ Process ID (pid) ◦ Network (net) ◦ Interprocess Communication (ipc) ◦ Unix Time-Sharing (uts) ◦ User ID (user) • Processes’ resource usage are limited through cgroups. How are containers isolated? Recap
usually managed by using the below types of workload resources. ◦ Job - Manages Pods that are executed once and terminated when the task is completed. ◦ ReplicaSet - Ensures that the specified number of Pods(replica) are running. ◦ DaemonSet - Ensures that only one pod running for each node ◦ StatefulSet - Manages Pods running stateful applications ◦ Deployment - Manages deployment of Pod, ReplicaSet updates A pod is the most basic and smallest unit created and managed by Kubernetes.
by the web server to an external log system https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent A case of running multiple containers in a pod
• One or more Sidecar Containers ◦ It serves supporting features for the primary container e.g. Monitoring, Logging, etc... When a pod consists of multiple containers... https://pixabay.com/photos/bmw-500-old-motorcycle-sidecar-4344066/ “Sidecar” attached to a motorcycle
• We learned that a container is a process that runs in an isolated environment. • The first process executed in the isolated PID namespace is pid 1. The state of the first process run in the container = Container’s life
all desired processes are running fine. The state of all processes running in the container ≠ Container’s life If there are multiple processes running inside a container...
command: ['sh', '-c', 'sleep 3600'] restartPolicy: Always (...) Don't worry, Kubernetes restarts the container according to the declared restartPolicy. • Always • OnFailure • Never When a specific container of a Kubernetes Pod is terminated
(terminated) Container 3 Retries with exponential back-off delay (10s, 20s, 40s,… up to 5 minutes) When a specific container of a Kubernetes Pod is terminated
the same node? (Containers in the same Pod always run in the same node) • Should the containers scale horizontally by the same number? (Pod are the units of scalability) • Should the containers be deployed together as a group?
• A pod is a group of containers. • If so... what’s shared and what’s isolated between containers in the same Pod? Isolation between containers in a Kubernetes Pod
This is the pod template spec: containers: - name: hello image: busybox command: ['sh', '-c', 'echo "first container" && sleep 3600'] - name: hello2 image: busybox command: ['sh', '-c', 'echo "second container" && sleep 3600'] restartPolicy: OnFailure # The pod template ends here This pod has 2 containers two-containers-pod.yml Isolation between containers in a Kubernetes Pod
shared between the containers in the pod → It’s possible to do IPC like using shared memory between containers. → The IP addresses and ports are shared between containers. It means that you have to be careful of port conflicts on containers in the same pod. Isolation between containers in a Kubernetes Pod
and Network namespaces. → The rest of the containers share these namespaces. This is to prevents issues in shared namespaces in the pod, even when a container running a user application terminates unexpectedly.
of Zombie process occurring in some containers, You can delegate the Zombie process reaping role to the Pause container by activating the Kubernetes “PID namespace sharing” option. • PID namespace sharing is enabled by default in Kubernetes v1.7. • However, from v1.8, it is disabled again due to compatibility issues with containers that depend on the init system. ◦ https://github.com/kubernetes/kubernetes/issues/48937 Reference link: https://www.ianlewis.org/en/almighty-pause-container
you can see the other container processes running on the same pod. PID 1 is always “pause” process when enabling PID namespace sharing. PID namespace sharing on Kubernetes
# Run a `pause` container $ docker run -d --ipc="shareable" --name pause k8s.gcr.io/pause:3.2 # Run a container executing `sleep` command on the ipc, net, pid namespaces of pause container $ docker run -ti --rm -d --name sleep-busybox \ --net=container:pause \ --ipc=container:pause \ --pid=container:pause \ busybox sleep 3600 # Run a container executing `ps` command on the ipc, net, pid namespaces of pause container. # We can see the other container’s processes(pause, sleep) because the pid namespace is shared. $ docker run --rm --name ps-busybox \ --net=container:pause \ --ipc=container:pause \ --pid=container:pause \ busybox ps # Let’s take a look the namespaces of “sleep-busybox” container $ ps -ef | grep sleep $ sudo lsns -p <PID>
name: practice spec: template: # This is the pod template spec: shareProcessNamespace: true containers: - name: sleep image: busybox command: ['sh', '-c', 'sleep 3600'] restartPolicy: OnFailure # The pod template ends here practice.yml $ kubectl apply -f practice.yml # Determine which node the pod is running on $ kubectl get pods -o wide # In the node, # let’s take a look at the `sleep` container’s namespaces node# $ ps -ef | grep sleep node# $ sudo lsns -p <PID>
The smallest unit that can be deployed in Kubernetes ▪ Pods are managed by various types of resources (Job, ReplicaSet, etc.) • A group of one or more containers ◦ running a single container ◦ running multiple containers ▪ Primary Container ▪ Sidecar Containers
container ◦ Even if the container is running, we cannot guarantee that all processes are running well. • When a container in a Kubernetes Pod is shut down, Kubelet restarts the container according to the restartPolicy. • Considerations to configure pod ◦ Should the containers run on the same node? ◦ Should the containers scale horizontally by the same number? ◦ Should the containers be deployed together as a group? Recap: Kubernetes Pod Concept
with Host → cgroup, user ◦ shared namespaces with the containers in the same pod → ipc, net ◦ Isolated namespaces for each container → mount, uts, pid ▪ pid namespace sharing is optional • The pause container? ◦ Creates and holds isolated IPC and Network namespace. ◦ Plays the role of reaping zombie processes when enabling PID namespace sharing. Recap: Kubernetes Pod Concept
devinjeon
hsawaji
izumin5210
layerx
fadis
notfounds
masaki12
mayuki
ikumatadokoro
mikik0
nekomatu
ivargrimstad
maaretp
jnunemaker
brettharned
eileencodes
erikaheidi
phodgson
brad_frost
bkeepers
chriscoyier
eitanlees
tammyeverts
csswizardry
destraynor
![OverlayFS storage driver $ docker inspect ubuntu | jq ".[].GraphDriver"](https://files.speakerdeck.com/presentations/cd02a5cc3a9a44b7aeea17524d663087/slide_17.jpg){kind=link}
