Container Tracer using OCI hooks on Kubernetes

1. R&D Center Base System Development Department Copyright 2019 Sony Corporation Container Tracer using OCI hooks on Kubernetes Kubernetes Invitational Meetup Tokyo #4 Kenta Tada R&D Center Sony Corporation
2. About me ⚫System Software Engineer, Sony ⚫OSS Contributor • runC • Docker • containerd and so on
3. Agenda ⚫OCI hooks recap ⚫Current status of Kubernetes integration ⚫Case Study • Tracer using OCI hooks for Tracer for ROS (Robot Operating System)
4. OCI hooks recap
5. OCI hooks ⚫OCI hooks are executed by container runtime ⚫You can set up the hooks at • Prestart • Poststart • Poststop ⚫From OCI runtime spec, the state of the container which includes container initial PID must be passed to hooks over stdin ⚫More hooks are currently proposed. • https://github.com/opencontainers/runtime-spec/pull/1008
6. Current status of Kubernetes integration
7. How to set up prestart hook in Kubernetes ⚫Kubernetes Pod Lifecycle and related hook ⚫Kubernetes did not provide prestart hook • https://github.com/kubernetes/kubernetes/issues/140 process lifetime process start process stop prestop poststart
8. How to operate OCI hooks from Kubernetes 1. Set up OCI hooks on high level runtime 2. Control OCI hooks per Pod using Kubernetes annotations
9. Current status of containerd 1. Set up OCI hooks on high level runtime • containerd is currently developing option for setting OCI hook – https://github.com/containerd/cri/pull/1248 – https://github.com/containerd/cri/issues/405 2. Control OCI hooks per Pod using Kubernetes annotations • containerd supported for passing annotations to low level runtime – https://github.com/containerd/cri/pull/1084 ◦ ×
10. Current status of CRI-O 1. Set up OCI hooks on high level runtime • CRI-O and Podman have already provided their own solution "oci- hooks“ 2. Control OCI hooks per Pod using Kubernetes annotations • CRI-O is currently developing passing annotations to low level runtime – https://github.com/cri-o/cri-o/issues/2402 ◦ ×
11. Case Study
12. Recap : Container Tracer ⚫We are developing an experimental tool to trace system calls inside containers without any debug tools in Pod. • https://speakerdeck.com/kentatada/debug-application-inside- kubernetes-using-linux-kernel-tools • https://github.com/KentaTada/oci-ftrace-syscall-analyzer ⚫ This tool uses OCI hooks to trace system calls from apps startup.
13. What is ROS (Robot Operating System)? ⚫ROS (Robot Operating System) provides libraries and tools to help software developers create robot applications. • http://wiki.ros.org/ ⚫ROS is the middleware. Hardware OS(Ex. Ubuntu) ROS Apps(Ex. path planning)
14. Tracer for ROS containers ⚫We are trying to launch ROS nodes on Kubernetes. • Ex. Using Docker with ROS –http://wiki.ros.org/docker ⚫We are also trying to use our container tracer to trace ROS nodes from apps startup.
15. Case Study : RT apps on the container environment ⚫RT apps on our rootless container failed to execute sched_setscheduler(2) during apps startup.
16. Logs
17. Tracer found two root causes ⚫Container side issue • With systemd, we needed to set up LimitRTPRIO in service file. –https://superuser.com/questions/403184/configuring-systemd-to- allow-daemon-to-set-rt-priority ⚫Kernel side issue • With systemd, we needed to consider CONFIG_RT_GROUP_SCHED. –https://blog.cybozu.io/entry/2018/06/22/080000 But these issues don’t depend on ROS.
18. Wrap up ⚫We cannot set up and control OCI hooks from Kubernetes for now. ⚫Container tracer is useful. • Contribute new facilities from our internal tool. –Ex. Linux capability checker –Ex. negative dentry snoop –https://qiita.com/kentaost/items/5f03ea32a2b2ef80270f • I want to discuss use cases of the container tracer among various industries.
19. SONYはソニー株式会社の登録商標または商標です。 各ソニー製品の商品名・サービス名はソニー株式会社またはグループ各社の登録商標または商標です。その他の製品および会社名は、各社の商号、登録商標または商標です。