仮想化環境での利用者公平性

1. Ծ૝Խ؀ڥͰͷ ར༻ऀެฏੑ ઙా ୓໵ @syuu1228 12೥11݄20೔Ր༵೔
2. Ծ૝Խ؀ڥͷެฏੑʁ • KVMͷήετΛͨ͘͞Μ্ཱͪ͛ͨ࣌ɺϊʔυؒ ͷ௨৴ੑೳ΍CPUར༻࣌ؒͷެฏੑ͸Ͳͷఔ౓อ ͨΕ͍ͯΔͷ͔ • CPUར༻࣌ؒ • ௨৴ྔ • ϨΠςϯγ • VMลΓͷෛՙͷ͹Β͖ͭ 12೥11݄20೔Ր༵೔
3. ଌͬͯΈΑ͏ • ϨΠςϯγɾ௨৴ྔɾCPU࢖༻཰ͳͲͷ໘Ͱ Ͳͷఔ౓ެฏͳεέδϡʔϧ͕ߦΘΕΔͷ͔ • VM਺Λ࣮CPU਺Ҏ্ʹ૿΍ͨ࣌͠ʹԿ͕ى ͖Δ͔ • Ծ૝NICͱSR−IOVͰ͸ੑೳಛੑ͕ͲͷΑ͏ ʹҟͳΔ͔ 12೥11݄20೔Ր༵೔
4. ੑೳଌఆ؀ڥ • ̎୆ͷLinuxػɾ10G NIC • 1ʙ64୆ͷKVMήετɾ128ϓϩηεͷ netperf 1VM→VM͋ͨΓ128ϑϩʔ 2VM→VM͋ͨΓ64ϑϩʔ 4VM→VM͋ͨΓ32ϑϩʔ… 64VM→VM͋ͨΓ2ϑϩʔ • TCP Request/ResponseϞʔυ 1byteͷύέοτΛϐϯϙϯ VMϗετ ήετ ςετػ 10G NIC 10G NIC ήετ netperf netserver netperf 12೥11݄20೔Ր༵೔
5. ϋʔυʗιϑτͷεϖοΫ Distribution Ubuntu Server 12.10 Linux Kernel 3.5.0-18-generic QEMU-KVM 1.2.0 Netperf 2.5.0 CPU(VMϗετ) Intel Core i7 980 (3.33GHz) Memory(VMϗετ) 24GB CPU(ςετػ) Intel Core i7 860 (2.8GHz) Memory(ςετػ) 8GB NIC Intel 82599(ixgbe) ෺ཧ6ίΞɺ࿦ཧ12ίΞ 12೥11݄20೔Ր༵೔
6. ήετϚγϯͷઃఆ • Ծ૝CPUɿ1ͭ • Ծ૝NICͷઃఆ • macvtapͰbridge / vhost-net • SR-IOV • 512MBͷϝϞϦׂΓ౰ͯ 12೥11݄20೔Ր༵೔
7. ͓͞Β͍ɿSR-IOV • ෺ཧNIC͕VMʹରͯ͠Ծ૝NICΛ௚઀ఏڙɹIO͸ϋΠύʔόΠβʔΛհࡏͤͣʹߦΘΕΔ • ׂࠐΈ͚ͩ͸Ծ૝Խग़དྷ͍ͯͳ͍ͷͰKVMΛ௨ͯ͡సૹ͞ΕΔ • ࠷΋ੑೳ͕ߴ͍͕ϋʔυରԠ͕ඞཁʗήετͷύέοτΛϑΟϧλʔɾվม͢Δࣄ͸೉͍͠ NIC Χʔωϧ Χʔωϧ ήετ Ϣʔβ PF PFυϥΠό qemu kvm VF υϥΠό TCP/IP ελοΫ NIC2 ϑΥϫʔυ ׂΓࠐΈ ෺ཧׂΓࠐΈ ׂΓࠐΈ ׂࠐΈ ϋϯυϥ VF %."ύεεϧʔ ϋΠύʔόΠβ 12೥11݄20೔Ր༵೔
8. macvtap <interface type='direct'> <mac address='52:54:00:6b:28:01'/> <source dev='eth1' mode='vepa'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </interface> 12೥11݄20೔Ր༵೔
9. SR-IOV <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x04' slot='0x10' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </hostdev> 12೥11݄20೔Ր༵೔
10. ൺֱର৅ͱͯ͠ͷ ࣮ػੑೳଌఆ • ̎୆ͷLinuxػɾ10G NIC • KVMήετͰnetserver࣮ߦ • 128ϓϩηεͷnetperf • TCP Request/ResponseϞʔυ 1byteͷύέοτΛϐϯϙϯ VMϗετ ςετػ 10G NIC 10G NIC netperf netserver netperf netserver 12೥11݄20೔Ր༵೔
11. VM਺Λ1-64·Ͱ૿Ճ vhost-netͱSR-IOVൺֱ 12೥11݄20೔Ր༵೔
12. ඵؒτϥϯβΫγϣϯ਺ 0 1375.00 2750.00 4125.00 5500.00 1 2 4 8 16 32 64 vhost-net SR-IOV baremetal VM਺ transaction/sec ෺ཧίΞ਺ʹ͍ۙลΓͰ ੑೳ࠷ߴ 12೥11݄20೔Ր༵೔
13. ϨΠςϯγ 0 750.00 1500.00 2250.00 3000.00 1 2 4 8 16 32 64 vhost-net SR-IOV baremetal VM਺ ϨΠςϯγ஋ ෺ཧίΞ਺ʹ͍ۙลΓͰ ϨΠςϯγ࠷ద 12೥11݄20೔Ր༵೔
14. CPUෛՙ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 vhost-net SR-IOV baremetal VM਺ ύʔηϯςʔδ ෛՙΛVM਺෼ͷίΞ਺ʹ ͔͠෼ࢄͰ͖ͯͳ͍ 12೥11݄20೔Ր༵೔
15. VMؒͷภΓ ʢ ඵؒτϥϯβΫγϣϯ਺ʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 vhost-net SR-IOV VM਺ ύʔηϯςʔδ vhost-net < 10% SR-IOV < 30% 12೥11݄20೔Ր༵೔
16. VMؒͷภΓ ʢ ϨΠςϯγʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 vhost-net SR-IOV VM਺ ύʔηϯςʔδ vhost-net < 10% SR-IOV < 30% 12೥11݄20೔Ր༵೔
17. VMؒͷภΓ ʢ CPUෛՙʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 vhost-net SR-IOV VM਺ ύʔηϯςʔδ vhost-net < 10% SR-IOV < 20% 12೥11݄20೔Ր༵೔
18. ൺֱ݁Ռ • ෺ཧίΞ਺ʹ͍ۙVM਺ͩͱੑೳ͕Ұ൪ྑ͍Α͏ʹ ݟ͑Δ • ෺ཧίΞ਺ͷ10ഒͷVMΛ࣮ߦͯ͠΋ɺ͜ͷෛՙύλ ʔϯͰ͸ۃ୺ͳϨΠςϯγͷѱԽ΍ෛՙͷภΓͳͲ ͸ݟΒΕͳ͔ͬͨ • ෺ཧ؀ڥͱൺ΂͔ͯͳΓੑೳ͕ѱ͍ →ෛՙΛෳ਺ίΞ΁෼ࢄग़དྷ͍ͯͳ͍ࣄ͕ݪҼͷީ ิͱͯ͠ߟ͑ΒΕΔ 12೥11݄20೔Ր༵೔
19. Ծ૝؀ڥ্Ͱ ෳ਺ίΞ΁ωοτϫʔΫIO ෛՙ෼ࢄͤͨ࣌͞ͷੑೳ 12೥11݄20೔Ր༵೔
20. Ծ૝؀ڥ্ͰίΞ෼ࢄʁ • Ծ૝CPU਺Λ1-16·Ͱ૿΍ͯ͠ΈΔ • ίΞΛ૿΍͚ͨͩ͠Ͱ͸ωοτϫʔΫε λοΫ͕ฒྻͰ૸Βͳ͍ • ✕ RSSɿԾ૝NICɾVF͕ະରԠ • ◦ RPSɿήετଆLinuxͷઃఆͰ༗ޮԽ͠ ͯΈΔ 12೥11݄20೔Ր༵೔
21. RPS $ echo "f" > /sys/class/net/eth1/queues/rx-0/ rps_cpus $ echo 4096 > /sys/class/net/eth1/queues/ rx-0/rps_flow_cnt $ echo 32768 > /proc/sys/net/core/ rps_sock_flow_entries 12೥11݄20೔Ր༵೔
22. vCPU <vcpu placement='static'>16</vcpu> 12೥11݄20೔Ր༵೔
23. ඵؒτϥϯβΫγϣϯ਺ ʢvhost-netʣ 0 1375.00 2750.00 4125.00 5500.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 baremetal VM਺ transaction/sec VM =< 4ͳΒ΍΍վળ 12೥11݄20೔Ր༵೔
24. ඵؒτϥϯβΫγϣϯ਺ ʢvhost-net, RPSʣ 0 1375.00 2750.00 4125.00 5500.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 baremetal VM਺ transaction/sec VM =< 4ͳΒRPSͰ ੑೳվળ 12೥11݄20೔Ր༵೔
25. ඵؒτϥϯβΫγϣϯ਺ ʢSR-IOVʣ 0 1375.00 2750.00 4125.00 5500.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 baremetal VM਺ transaction/sec VM =< 4ͳΒ΍΍վળ 12೥11݄20೔Ր༵೔
26. ඵؒτϥϯβΫγϣϯ਺ ʢSR-IOV, RPSʣ 0 1375.00 2750.00 4125.00 5500.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 baremetal VM਺ transaction/sec VM =< 4ͳΒRPSͰ ੑೳվળ 12೥11݄20೔Ր༵೔
27. ϨΠςϯγ ʢvhost-netʣ 0 750.00 1500.00 2250.00 3000.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 baremetal VM਺ ϨΠςϯγ஋ tpsͱ͍͍ͩͨಉ͡܏޲ 12೥11݄20೔Ր༵೔
28. ϨΠςϯγ ʢvhost-net, RPSʣ 0 750.00 1500.00 2250.00 3000.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 baremetal VM਺ ϨΠςϯγ஋ tpsͱ͍͍ͩͨಉ͡܏޲ 12೥11݄20೔Ր༵೔
29. ϨΠςϯγ ʢSR-IOVʣ 0 750.00 1500.00 2250.00 3000.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 baremetal VM਺ ϨΠςϯγ஋ tpsͱ͍͍ͩͨಉ͡܏޲ 12೥11݄20೔Ր༵೔
30. ϨΠςϯγ ʢSR-IOV, RPSʣ 0 750.00 1500.00 2250.00 3000.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 baremetal VM਺ ϨΠςϯγ஋ tpsͱ͍͍ͩͨಉ͡܏޲ 12೥11݄20೔Ր༵೔
31. CPUෛՙ ʢvhost-netʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ VM =< 4ͳΒɺcpu = 1 ͷ࣌ΑΓෛՙΛ෼ࢄͰ ͖͍ͯΔʁ 12೥11݄20೔Ր༵೔
32. CPUෛՙ ʢvhost-net, RPSʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ RPSͰΑΓෛՙΛ෼ࢄ 12೥11݄20೔Ր༵೔
33. CPUෛՙ ʢSR-IOVʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ 12೥11݄20೔Ր༵೔
34. CPUෛՙ ʢSR-IOV, RPSʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ 12೥11݄20೔Ր༵೔
35. VMؒͷภΓ ʢ ඵؒτϥϯβΫγϣϯ਺ʗvhost-netʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ cpu >= 8, VM >=16ͷ࣌ʹ 15%Λ௒͑ΔภΓ 12೥11݄20೔Ր༵೔
36. VMؒͷภΓ ʢ ඵؒτϥϯβΫγϣϯ਺/vhost-net, RPSʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ cpu >= 8, VM >=16ͷ࣌ʹ 15%Λ௒͑ΔภΓ 12೥11݄20೔Ր༵೔
37. VMؒͷภΓ ʢ ඵؒτϥϯβΫγϣϯ਺ʗSR-IOVʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ ૯εϨου਺͕෺ཧCPU਺Λ ௒͑ͨลΓͰେ͖ͳภΓ 12೥11݄20೔Ր༵೔
38. VMؒͷภΓ ʢ ඵؒτϥϯβΫγϣϯ਺ʗSR-IOV, RPSʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ ૯εϨου਺͕෺ཧCPU਺Λ ௒͑ͨลΓͰେ͖ͳภΓ 12೥11݄20೔Ր༵೔
39. VMؒͷภΓ ʢ ϨΠςϯγʗvhost-netʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ tpsͱ͍͍ͩͨಉ͡܏޲ 12೥11݄20೔Ր༵೔
40. VMؒͷภΓ ʢ ϨΠςϯγʗvhost-net, RPSʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ 12೥11݄20೔Ր༵೔
41. VMؒͷภΓ ʢ ϨΠςϯγʗSR-IOVʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ ૯εϨου਺͕෺ཧCPU਺Λ ௒͑ͨลΓͰେ͖ͳภΓ 12೥11݄20೔Ր༵೔
42. VMؒͷภΓ ʢ ϨΠςϯγʗSR-IOV, RPSʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ ૯εϨου਺͕෺ཧCPU਺Λ ௒͑ͨลΓͰେ͖ͳภΓ 12೥11݄20೔Ր༵೔
43. VMؒͷภΓ ʢ CPUෛՙʗvhost-netʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ ͋·Γେ͖ͳภ Γ͸ݟΒΕͳ͍ 12೥11݄20೔Ր༵೔
44. VMؒͷภΓ ʢ CPUෛՙʗvhost-net, RPSʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 64 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ ͋·Γେ͖ͳภ Γ͸ݟΒΕͳ͍ 12೥11݄20೔Ր༵೔
45. VMؒͷภΓ ʢ CPUෛՙʗSR-IOVʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ VM਺ͷ্ঢͰ एׯ૿Ճ 12೥11݄20೔Ր༵೔
46. VMؒͷภΓ ʢ CPUෛՙʗSR-IOV, RPSʣ 0 25.00 50.00 75.00 100.00 1 2 4 8 16 32 cpu1 cpu2 cpu4 cpu8 cpu16 VM਺ ύʔηϯςʔδ VM਺ͷ্ঢͰ एׯ૿Ճ 12೥11݄20೔Ր༵೔
47. ൺֱ݁Ռ • Ծ૝ϚγϯͰ࣮ػੑೳͷ൒෼Ҏ্Λग़ͤͨͷ͸VM਺͕͘͝গͳ͘ vCPU਺͕͔ͳΓଟ͍࣌ͷΈ → ͜ͷ৚݅Ͱ͔͠RPSͰੑೳ͕ग़ͳ͍ • ૯vCPU਺Λଟͯ͘͠΋ҙ֎ͱੑೳ͸Լ͕Βͳ͍ • SR-IOVͷ৔߹ʹVF਺͕෺ཧίΞ਺Λ௒͑ͨลΓͰ͔ͳΓੑೳʹภ Γ͕ͰΔ • NIC্ͷΩϡʔʹରͯ͠ड৴ॲཧʹׂΓ౰ͯΔίΞ͕ෆ଍ͯ͠ ͜ͷΑ͏ͳঢ়گʹͳΔͱਪଌ • vhost-netͰ͸ͦͷΑ͏ͳݱ৅͕ݟΒΕͣɺ૯vCPU਺Λ૿΍ͯ͠΋ ൺֱతެฏʹॲཧ͞Ε͍ͯΔΑ͏ʹݟ͑Δ 12೥11݄20೔Ր༵೔
48. ϚϧνϢʔβ࣌ͷެ ฏੑ 12೥11݄20೔Ր༵೔
49. ϚϧνϢʔβ࣌ͷެฏੑ • σϑΥϧτεέδϡʔϦϯάͰ͸ɺશͯͷKVM ήετϓϩηε͕ެฏʹCPU࣌ؒΛׂΓ౰ͯΒ ΕΔ →ෳ਺ͷϢʔβ͕ෳ਺ͷVMΛىಈ͍ͯ͠Δ৔ ߹ɺϢʔβลΓͷϦιʔεׂ౰͕ෆެฏʹͳΔ • ϢʔβؒͰެฏʹεέδϡʔϦϯάͯ͠ཉ͍͠ 12೥11݄20೔Ր༵೔
50. cgroup • ϓϩηεάϧʔϓʹରͯ͠Ϧιʔε ʢCPUɺϝϞϦɺσΟεΫI/Oɺωοτ ϫʔΫI/Oʣͷར༻Λ੍ݶɾִ཭͢Δҝ ͷLinuxͷػೳ 12೥11݄20೔Ր༵೔
51. cgroup tool install $ apt-get install cgroup-bin 12೥11݄20೔Ր༵೔
52. $ ls /sys/fs/cgroup blkioɹϒϩοΫI/Oͷ੍ݶ cpuɹCPUϦιʔεͷ੍ݶ cpuacctɹCPUͷ૸ߦ࣌ؒͳͲͷ৘ใऩू cpusetɹCPUͷׂΓ෇͚ʹؔ͢Δ੍ݶ devicesɹσόΠε΁ͷΞΫηεڐՄɾڋ൱ freezerɹάϧʔϓʹଐ͢ΔϓϩηεͷҰ࣌ఀࢭɾ࠶։ memoryɹϝϞϦϦιʔεͷ੍ݶ net_clsɹૹग़ύέοτʹtc༻ͷλάΛઃఆ nsɹωʔϜεϖʔεʢίϯςφ༻ʣ 12೥11݄20೔Ր༵೔
53. cd /sys/fs/cgroup/cpu $ mkdir grp_a $ echo ‘12254’ > grp_a/tasks $ echo ‘2184’ > grp_a/tasks $ echo ‘512’ > grp_a/cpu.shares $ mkdir grp_b $ echo ‘9012’ > grp_b/tasks $ echo ‘1024’ > grp_b/cpu.shares άϧʔϓ࡞੒ άϧʔϓॴଐϓϩη εΛ௥ՃʢPIDʣ άϧʔϓ಺ͷλεΫͰ࢖༻ग़དྷ ΔCPU࣌ؒͷ૬ରత഑෼஋ 12೥11݄20೔Ր༵೔
54. libvirtdͱcgroup • /sys/fs/cgroup/cpu/libvirt/qemu/<VM໊>ʹ VMؔ࿈ϓϩηε͕ࣗಈొ࿥͞ΕΔ • /sys/fs/cgroup/cpu/libvirt/qemu/<VM໊>/ vcpuNʹvcpuϓϩηε͕ࣗಈొ࿥͞ΕΔ • libvirtܦ༝ͰcgroupͷϦιʔε੍ݶઃఆ ͕ग़དྷΔ 12೥11݄20೔Ր༵೔
55. virshͱcgroup $ virsh schedinfo vm0 Scheduler : posix cpu_shares : 1024 vcpu_period : 100000 vcpu_quota : -1 $ virsh schedinfo --set cpu_shares=512 vm0 12೥11݄20೔Ր༵೔
56. ̎ͭͷάϧʔϓʹެฏʹϦ ιʔεΛׂΓ౰ͯͯΈΔ • /sys/fs/cgroup/grp_a • VMΠϯελϯεɿ̒ݸ • cpu.sharesɿσϑΥϧτʢ̍̌̎̐ʣ • /sys/fs/cgroup/grp_b • VMΠϯελϯεɿ̍̌ݸ • cpu.sharesɿσϑΥϧτʢ̍̌̎̐ʣ 12೥11݄20೔Ր༵೔
57. άϧʔϓͷ࡞Γํ $ virsh start <VM໊> $ mkdir /sys/fs/cgroup/cpu/grp_a $ cat /sys/fs/cgroup/cpu/libvirt/qemu/<VM໊>/tasks > tmp $ cat /sys/fs/cgroup/cpu/libvirt/qemu/<VM໊>/ vcpu*/tasks >> tmp $ cat tmp > /sys/fs/cgroup/cpu/grp_a/tasks 12೥11݄20೔Ր༵೔
58. vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒ ʢcgroupະ࢖༻࣌ʣ 0 625000000 1250000000 1875000000 2500000000 ࣮ߦ࣌ؒ 12೥11݄20೔Ր༵೔
59. vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒ ʢcgroup࢖༻࣌ʣ 0 625000000 1250000000 1875000000 2500000000 ࣮ߦ࣌ؒ grp_a grp_b 12೥11݄20೔Ր༵೔
60. vhost-net/16VM࣌ͷωοτϫʔΫIO ʢcgroupະ࢖༻࣌ʣ 0 1375000 2750000 4125000 5500000 σʔλྔ 12೥11݄20೔Ր༵೔
61. vhost-net/16VM࣌ͷωοτϫʔΫIO ʢcgroup࢖༻࣌ʣ 0 1375000 2750000 4125000 5500000 σʔλྔ grp_a grp_b 12೥11݄20೔Ր༵೔
62. άϧʔϓ͝ͱͷ߹ܭ஋ൺֱ ʢcgroup࢖༻࣌ʣ 0 0.50 1.00 1.50 2.00 CPU Net grp_a grp_b 1.10 1.14 12೥11݄20೔Ր༵೔
63. ʦࢀߟʧάϧʔϓ͝ͱͷ߹ܭ஋ൺֱ ʢcgroupະ࢖༻࣌ʣ 0 0.50 1.00 1.50 2.00 CPU Net grp_a grp_b 1.64 1.65 12೥11݄20೔Ր༵೔
64. ൺֱ݁Ռ • άϧʔϓʹ෼͚Δ͚ͩͰσϑΥϧτͰάϧʔϓεέ δϡʔϦϯά͕ߦΘΕɺ࣮ଌ஋Ͱ΋άϧʔϓʹର͠ ͯެฏʹCPU͕഑෼͞Ε͍ͯΔࣄ͕֬ೝͰ͖Δ • CPUΛߜΔࣄͰ݁ՌతʹωοτϫʔΫIOΛߜΔࣄ ΋͋Δఔ౓ग़དྷ͍ͯΔ • Ͱ΋͋·Γݫີ͡Όͳ͍ʢ࣮ଌ14%͘Β͍ͷޡࠩʣ 12೥11݄20೔Ր༵೔
65. CFS bandwidth control • ͜ͷάϧʔϓ͸cpu.cfs_period_usͷ୯Ґ࣌ ؒதʹ࠷େcpu.cfs_quota_us͚࣮ͩޮग़དྷΔ • άϧʔϓ಺ͷϓϩηε͸period୯Ґ࣌ؒ தɺಉ࣌ʹ࣮ߦ͞Ε͏ΔͷͰquota͸period Λ௒͑͏Δʢ2CPUͰΊ͍͍ͬͺ͍૸ͬͨ Βquota = period * 2ʣ 12೥11݄20೔Ր༵೔
66. vhost-net/16VM࣌ͷCPU࣮ߦ࣌ؒ ʢquota=200000,period=50000ʣ 0 625000000 1250000000 1875000000 2500000000 ࣮ߦ࣌ؒ grp_a grp_b 12೥11݄20೔Ր༵೔
67. vhost-net/16VM࣌ͷωοτϫʔΫIO ʢquota=200000,period=50000ʣ 0 1375000 2750000 4125000 5500000 σʔλྔ grp_a grp_b 12೥11݄20೔Ր༵೔
68. άϧʔϓ͝ͱͷ߹ܭ஋ൺֱ ʢquota=200000,period=50000ʣ 0 0.50 1.00 1.50 2.00 CPU Net grp_a grp_b 1.01 0.63 12೥11݄20೔Ր༵೔
69. ݕূ݁Ռ • ͔֬ʹάϥϑ্͸៉ྷʹͳΔͷ͕ͩɺCPU͕ͲΜͳʹ ۭ͍͍ͯͯ΋quota·ͰߦͬͨΒઈର࣮ߦ͞Εͳ͍ • ͔ͱ͍࣮ͬͯߦ͖͠Εͳ͍ఔͷquotaΛ༩͑Δͱઃఆ͠ ͳ͍৔߹ͱมΘΒͳ͘ͳΔ • CPU΋ωοτϫʔΫIO΋ͪΐ͏ͲΑ͘ͽͬͨΓ1.0ͱ͍ ͏༁ʹ͸͍͔ͳ͔ͬͨ • period͕௕͗͢ΔͱVM͕௕͍࣮ؒߦ͞Εͳͯ͘ྑ͘ͳ ͍͜ͱ͕ى͖ͦ͏ 12೥11݄20೔Ր༵೔
70. net_cls $ tc class add dev virbr0 parent 10: classid 10:1 htb rate 24Mbit $ echo 0x100001 > /sys/fs/cgroup/net_cls/ grp_a/net_cls.classid ϓϩηε͔Βૹ৴͞ΕΔύέοτʹtcͷ λάΛ෇͚ΔࣄʹΑΓଳҬ੍ݶ͕ग़དྷΔ 12೥11݄20೔Ր༵೔
71. ·ͱΊ • KVMʹΑΔԾ૝Խ؀ڥͷϦιʔεׂΓ౰ͯͷެฏੑʹ͍ͭͯ ݕূͨ͠ • ࠓճͷ࣮ݧͷ৚݅ԼͰ͸ɺ࣮CPU਺Λେ͖͘௒͑ΔVM਺Ͱ ΋ۃ୺ͳϨΠςϯγͷѱԽ΍ෛՙͷภΓͳͲ͸ݟΒΕͳ͔ ͬͨ • ୠ͠ɺSR-IOVͷ৔߹ʹVF਺͕෺ཧίΞ਺Λ௒͑ͨลΓͰ͔ ͳΓੑೳʹภΓ͕ൃੜͨ͠ • cgroupΛ༻͍ͯϚϧνϢʔβ࣌ʹϢʔβؒͷCPUׂΓ౰ͯΛ ެฏʹग़དྷΔࣄΛ֬ೝͨ͠ 12೥11݄20೔Ր༵೔