Keeping it Real(Time)

Real-time in OpenStack is a thing now. Let's explore what this actually looks like to a user, and (in brief) how it works under the hood.

Since Juno, OpenStack projects like nova have been on a steady march towards adding the features and capabilities that users with NFV and HPC workloads really need from their cloud. The latest of these nova features builds upon earlier work to finally make real-time workloads in an OpenStack cloud a reality. In this talk, we take a peek under the nova hood to see how this feature works and how it ties into earlier work in this area (hint: KVM + libvirt do most of the heavy lifting). We then demonstrate why this matters and how you can use this feature and others like it in your own applications.

Presented at FOSDEM 2018

https://fosdem.org/2018/schedule/event/vai_keeping_it_realtime/

Transcript

1. Keeping It Real (Time) Enabling real-time compute in OpenStack ⸺ @stephenfin
2. A little bit of stage setting...
4. $ openstack server (create|delete|list|...) $ openstack network (create|delete|list|...) $ openstack image (create|delete|list|...) $ openstack volume (create|delete|list|...) ...
6. Um, what about NFV?
7. NFV History in OpenStack Before the OpenStack “Ocata” release, we already supported: • NUMA policies • CPU (thread) pinning policies • Hugepages • SR-IOV*
8. NFV History in OpenStack The OpenStack “Pike” and “Ocata” releases added two feature respectively: • Real time policy • Emulator threads policy
9. NFV History in OpenStack The OpenStack “Pike” and “Ocata” releases added two feature respectively: • Real time policy • Emulator threads policy
10. Prerequisites
11. Requirements Most configuration is done on the machine, but there are a few strict requirements. • Suitable hardware • OpenStack Pike or newer • Libvirt 1.2.13 or newer • Real-time kernel CentOS 7.4 was used for the demo
12. Host Configuration (Hardware) Disable the funkier CPU features • Hyper Threading (SMT) • Power management • Turbo Boost Essentially all the things you would do if benchmarking a system
13. Host Configuration (Software) Install dependencies • Real-time kernel • Real-time KVM module • Real-time tuned host profiles Enable hugepages to prevent page faults Isolate some cores
14. $ yum install -y kernel-rt.x86_64 kernel-rt-kvm.x86_64 $ yum install -y tuned-profiles-realtime tuned-profiles-nfv
15. $ yum install -y kernel-rt.x86_64 kernel-rt-kvm.x86_64 $ yum install -y tuned-profiles-realtime tuned-profiles-nfv # configure tuned profile, hugepages $ tuned-adm profile realtime-virtual-host $ cat /etc/default/grub | grep default_hugepagesz GRUB_CMDLINE_LINUX+="default_hugepagesz=1G"
16. $ yum install -y kernel-rt.x86_64 kernel-rt-kvm.x86_64 $ yum install -y tuned-profiles-realtime tuned-profiles-nfv # configure tuned profile, hugepages $ tuned-adm profile realtime-virtual-host $ cat /etc/default/grub | grep default_hugepagesz GRUB_CMDLINE_LINUX+="default_hugepagesz=1G" # configure nova $ cat /etc/nova/nova-cpu.conf | grep vcpu_pin_set vcpu_pin_set = <isolated CPUs>
17. Guest Configuration (Image) Requires many of the same dependencies • Real-time kernel • Real-time tuned guest profiles If you already have an application, use that
18. $ yum install -y kernel-rt.x86_64 $ yum install -y tuned-profiles-realtime tuned-profiles-nfv
19. $ yum install -y kernel-rt.x86_64 $ yum install -y tuned-profiles-realtime tuned-profiles-nfv # configure tuned profile, huge pages $ tuned-adm profile realtime-virtual-guest $ cat /etc/default/grub | grep default_hugepagesz GRUB_CMDLINE_LINUX+="default_hugepagesz=1G"
20. Guest Configuration (Flavor) Requires the following configuration options • CPU policy • CPU realtime policy • Mempages Optionally, you can also configure • Emulator thread policy • CPU thread policy
21. $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 \ rt1.small
22. $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 \ rt1.small $ openstack flavor set rt1.small \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:cpu_realtime=yes' \ --property 'hw:cpu_realtime_mask=^0-1' \ --property 'hw:mem_page_size=1GB'
23. $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 \ rt1.small $ openstack flavor set rt1.small \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:cpu_realtime=yes' \ --property 'hw:cpu_realtime_mask=^0-1' \ --property 'hw:mem_page_size=1GB'
24. $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 \ rt1.small $ openstack flavor set rt1.small \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:cpu_realtime=yes' \ --property 'hw:cpu_realtime_mask=^0-1' \ --property 'hw:mem_page_size=1GB'
25. $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 \ rt1.small $ openstack flavor set rt1.small \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:cpu_realtime=yes' \ --property 'hw:cpu_realtime_mask=^0-1' \ --property 'hw:mem_page_size=1GB'
26. $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 \ rt1.small $ openstack flavor set rt1.small \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:cpu_realtime=yes' \ --property 'hw:cpu_realtime_mask=^0-1' \ --property 'hw:mem_page_size=1GB'
27. $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 \ rt1.small $ openstack flavor set rt1.small \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:cpu_realtime=yes' \ --property 'hw:cpu_realtime_mask=^0-1' \ --property 'hw:mem_page_size=1GB' $ openstack server create --flavor rt1.small --image centos-rt
28. Under the hood
29. $ virsh dumpxml 1 | xpath /dev/stdin /domain/cputune
30. $ virsh dumpxml 1 | xpath /dev/stdin /domain/cputune <cputune> <shares>4096</shares> <vcpupin vcpu="0" cpuset="2" /> <vcpupin vcpu="1" cpuset="3" /> <vcpupin vcpu="2" cpuset="4" /> <vcpupin vcpu="3" cpuset="5" /> <emulatorpin cpuset="2-3" /> <vcpusched vcpus="2" scheduler="fifo" priority="1" /> <vcpusched vcpus="3" scheduler="fifo" priority="1" /> </cputune>
31. $ virsh dumpxml 1 | xpath /dev/stdin /domain/cputune <cputune> <shares>4096</shares> <vcpupin vcpu="0" cpuset="2" /> <vcpupin vcpu="1" cpuset="3" /> <vcpupin vcpu="2" cpuset="4" /> <vcpupin vcpu="3" cpuset="5" /> <emulatorpin cpuset="2-3" /> <vcpusched vcpus="2" scheduler="fifo" priority="1" /> <vcpusched vcpus="3" scheduler="fifo" priority="1" /> </cputune>
32. $ virsh dumpxml 1 | xpath /dev/stdin /domain/cputune <cputune> <shares>4096</shares> <vcpupin vcpu="0" cpuset="2" /> <vcpupin vcpu="1" cpuset="3" /> <vcpupin vcpu="2" cpuset="4" /> <vcpupin vcpu="3" cpuset="5" /> <emulatorpin cpuset="2-3" /> <vcpusched vcpus="2" scheduler="fifo" priority="1" /> <vcpusched vcpus="3" scheduler="fifo" priority="1" /> </cputune>
33. vcpupin The optional vcpupin element specifies which of host's physical CPUs the domain VCPU will be pinned to. If this is omitted, and attribute cpuset of element vcpu is not specified, the vCPU is pinned to all the physical CPUs by default. It contains two required attributes, the attribute vcpu specifies vcpu id, and the attribute cpuset is same as attribute cpuset of element vcpu. Since 0.9.0 Source: libvirt domain XML format (CPU Tuning)
34. vcpupin The optional vcpupin element specifies which of host's physical CPUs the domain VCPU will be pinned to. If this is omitted, and attribute cpuset of element vcpu is not specified, the vCPU is pinned to all the physical CPUs by default. It contains two required attributes, the attribute vcpu specifies vcpu id, and the attribute cpuset is same as attribute cpuset of element vcpu. Since 0.9.0 Source: libvirt domain XML format (CPU Tuning)
35. int virProcessSetAffinity(pid_t pid, virBitmapPtr map) { ... if (sched_setaffinity(pid, masklen, mask) < 0) { ... } ... } libvirt/src/util/virprocess.c
36. $ ps -e | grep qemu 27720 ? 00:00:04 qemu-kvm $ ps -Tp 27720 PID SPID TTY TIME CMD 27720 27720 ? 00:00:00 qemu-kvm 27720 27736 ? 00:00:00 qemu-kvm 27720 27774 ? 00:00:01 CPU 0/KVM 27720 27775 ? 00:00:00 CPU 1/KVM 27720 27776 ? 00:00:00 CPU 2/KVM 27720 27777 ? 00:00:00 CPU 3/KVM 27720 27803 ? 00:00:00 vnc_worker
37. $ taskset -p 27774 # CPU 0/KVM pid 27774's current affinity mask: 4 $ taskset -p 27775 # CPU 1/KVM pid 27775's current affinity mask: 8 $ taskset -p 27776 # CPU 2/KVM pid 27776's current affinity mask: 10 $ taskset -p 27777 # CPU 3/KVM pid 27777's current affinity mask: 20
38. $ virsh dumpxml 1 | xpath /dev/stdin /domain/cputune <cputune> <shares>4096</shares> <vcpupin vcpu="0" cpuset="2" /> <vcpupin vcpu="1" cpuset="3" /> <vcpupin vcpu="2" cpuset="4" /> <vcpupin vcpu="3" cpuset="5" /> <emulatorpin cpuset="2-3" /> <vcpusched vcpus="2" scheduler="fifo" priority="1" /> <vcpusched vcpus="3" scheduler="fifo" priority="1" /> </cputune>
39. $ virsh dumpxml 1 | xpath /dev/stdin /domain/cputune <cputune> <shares>4096</shares> <vcpupin vcpu="0" cpuset="2" /> <vcpupin vcpu="1" cpuset="3" /> <vcpupin vcpu="2" cpuset="4" /> <vcpupin vcpu="3" cpuset="5" /> <emulatorpin cpuset="2-3" /> <vcpusched vcpus="2" scheduler="fifo" priority="1" /> <vcpusched vcpus="3" scheduler="fifo" priority="1" /> </cputune>
40. vcpusched The optional vcpusched element specifies the scheduler type (values: batch, idle, fifo, rr) for particular vCPU threads (based on vcpus; leaving out vcpus sets the default). Valid vcpus values start at 0 through one less than the number of vCPU's defined for the domain. For real-time schedulers (fifo, rr), priority must be specified as well (and is ignored for non-real-time ones). The value range for the priority depends on the host kernel (usually 1-99). Since 1.2.13 Source: libvirt domain XML format (CPU Tuning)
41. vcpusched The optional vcpusched element specifies the scheduler type (values: batch, idle, fifo, rr) for particular vCPU threads (based on vcpus; leaving out vcpus sets the default). Valid vcpus values start at 0 through one less than the number of vCPU's defined for the domain. For real-time schedulers (fifo, rr), priority must be specified as well (and is ignored for non-real-time ones). The value range for the priority depends on the host kernel (usually 1-99). Since 1.2.13 Source: libvirt domain XML format (CPU Tuning)
42. int virProcessSetScheduler(pid_t pid, virProcessSchedPolicy policy, int priority) { ... if (sched_setscheduler(pid, pol, &param) < 0) { ... } ... } libvirt/src/util/virprocess.c
43. $ chrt -p 27774 # CPU 0/KVM pid 27774's current scheduling policy: SCHED_OTHER pid 27774's current scheduling priority: 0 $ chrt -p 27775 # CPU 1/KVM pid 27775's current scheduling policy: SCHED_OTHER pid 27775's current scheduling priority: 0 $ chrt -p 27776 # CPU 2/KVM pid 27776's current scheduling policy: SCHED_FIFO pid 27776's current scheduling priority: 1 $ chrt -p 27777 # CPU 3/KVM pid 27777's current scheduling policy: SCHED_FIFO pid 27777's current scheduling priority: 1
44. $ virsh dumpxml 1 | xpath /dev/stdin /domain/memoryBacking
45. $ virsh dumpxml 1 | xpath /dev/stdin /domain/memoryBacking <memoryBacking> <hugepages> <page size="1048576" unit="KiB" nodeset="0" /> </hugepages> <nosharepages /> <locked /> </memoryBacking>
46. $ ps -e | grep qemu 27720 ? 00:00:04 qemu-kvm $ grep huge /proc/*/numa_maps /proc/27720/numa_maps:7f3dc0000000 bind:0 ...
47. $ openstack server ssh rt-server --login centos
48. $ openstack server ssh rt-server --login centos # within the guest $ taskset -c 2 stress --cpu 4 & $ taskset -c 2 cyclictest -m -n -q -p95 -D 1h -h100 -i 200 \ > cyclictest.out $ cat cyclictest.out | tail -7 | head -3 # Min Latencies: 00006 # Avg Latencies: 00007 # Max Latencies: 00020
49. Wrap up
50. $ openstack flavor create --vcpus 4 --ram 4096 --disk 20 \ rt1.small $ openstack flavor set rt1.small \ --property 'hw:cpu_policy=dedicated' \ --property 'hw:cpu_realtime=yes' \ --property 'hw:cpu_realtime_mask=^0-1' \ --property 'hw:mem_page_size=1GB' $ openstack server create --flavor rt1.small --image centos-rt
51. Keeping It Real (Time) Enabling real-time compute in OpenStack ⸺ @stephenfin
52. References • libvirt domain XML format (CPU Tuning) — libvirt.org • taskset(1) — man7.org • sched_setaffinity(2) — man7.org • chrt(1) — man7.org • sched_setscheduler(2) — man7.org • Completely Fair Scheduler — doc.opensuse.org • Using and Understanding the Real-Time Cyclictest Benchmark — linuxfound.org • Deploying Real Time Openstack — that.guru
53. Credits Clocks photo by Ahmad Ossayli on Unsplash Clouds photo by Jason Wong on Unsplash