Ruby 3x3 に向けたRubyのベンチマーク環境と成果 / Ruby Association Grant

Rubyアソシエーション開発助成成果報告会
https://rubyassociation.doorkeeper.jp/events/74355

Transcript

1. Ruby 3x3 ʹ޲͚ͨ RubyͷϕϯνϚʔΫ؀ڥͱ੒Ռ RubyΞιγΤʔγϣϯ։ൃॿ੒ ੒Ռใࠂձ @k0kubun
2. ࣗݾ঺հ GitHub, Twitter: k0kubun Rubyίϛολ: JITίϯύΠϥͷ։ൃ, ERBͷϝϯςφϯε ࡞ͬͨ: Hamlit, MItamae, md2key, xremap, LLRB
3. ࠓ೔࿩͢͜ͱ 1. ຊϓϩδΣΫτ͸ԿΛղܾ͔ͨ͠ 2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά 3. ϓϥάΠϯΞʔΩςΫνϟ 4. ϓϩδΣΫτऴྃޙͷ׆ಈ
4. 1. ຊϓϩδΣΫτ͸ԿΛղܾ͔ͨ͠
5. ͜Ε·ͰͷRubyͷϕϯνϚʔΫ؀ڥ
6. ϒϩοΫݺͼग़͠ͷΦʔόʔϔου໰୊ ඪ४ͷbenchmarkϥΠϒϥϦ΋benchmark-ips΋ɺܭଌ୯Ґ͕ϒ ϩοΫʹͳ͍ͬͯΔ ϒϩοΫݺͼग़͠͸ී௨ͷϝιουݺͼग़͠૬౰ͷΦʔόʔ ϔου͕͋Δ ͦΕΑΓ΋࠷దԽ͞Ε໋ͨྩͷܭଌ͸?
7. ෳ਺ͷRuby࣮૷ؒͰͷൺֱͷ೉͠͞ ϒϩοΫͰಈతʹܭଌํ๏͕ࢦఆՄೳͩͱɺͦͷϒϩοΫΛ ผͷRubyΠϯλϓϦλʹ౉ͯ͠ܭଌ͢Δͷ͸ࠔ೉ ͔͠͠Ruby 3x3ͷͨΊʹ͸ɺॲཧܥࣗମͷมߋͷޮՌΛධՁ ͍ͨ͠
8. ཚཱ͢ΔϕϯνϚʔΧʔ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦ಺ͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ
9. Ͱ͸Ͳ͏͢Δ͔?
10. ࠷΋͜ͷ໰୊ͷղܾʹ͍ۙπʔϧ ඪ४ͷbenchmarkϥΠϒϥϦ benchmark-ips RubyϦϙδτϦ಺ͷ benchmark/driver.rb RubyBenchͷ benchmark/driver.rb ͷϑΥʔΫ VladʹΑΔMJIT-benchmarksͷ࣮ߦΤϯδϯ
11. benchmark/driver.rb ͱ͸ ෳ਺RubyόΠφϦؒͰൺֱ͕Ͱ͖ɺ࣮ߦ࣌ؒͷଞʹϝϞϦ࢖༻ྔΛܭଌͰ ͖Δ ϒϩοΫͰ͸ͳ͘whileϧʔϓΛ࢖͏ͨΊɺܭଌͷΦʔόʔϔου͕গͳ͍ ͨͩ͠RubyϦϙδτϦ಺ͷεΫϦϓτʹಛԽͨ͠ॲཧ͕ೖ͍ͬͯΔ benchmark/driver.rb Λ࢖͍΍ͯ͘͢͠օͰ࢖͑͹͍͍ͷͰ͸? ͦΕ͕ benchmark_driver.gem
12. ࠾୒͞ΕͨϓϩδΣΫτ “ΑΓ౷ҰతͰ؆୯ͳRubyͷϕϯνϚʔΫ؀ڥͷ࣮ݱ" benchmark_driver.gem ͷػೳ֦ॆ Ruby 3x3 ʹ޲͚ͨϕϯνϚʔΫηοτͱCIͷ੔උ
13. benchmark_driver.gem ͷಛ௃
14. 1. ͋ΒΏΔϝτϦΫεΛ͋ΒΏΔํ๏Ͱग़ྗͰ͖Δ ޙड़͢ΔϓϥάΠϯػߏʹΑΓɺ༷ʑͳܭଌํ๏Λ༷ʑͳํ ๏ͰूܭՄೳ σϑΥϧτͰ͸benchmark-ips෩ͷܭଌɾग़ྗ ಉ͡ग़ྗͷ··ϝϞϦ࢖༻ྔΛܭଌͨ͠ΓɺOptcarrotͷfps Λܭଌͨ͠ΓͰ͖Δ
15. 2. ਫ਼౓ͷྑ͍ܭଌεΫϦϓτΛࣗಈͰੜ੒ whileϧʔϓΛ͢ΔεΫϦϓτΛಈతʹੜ੒͠ɺ͔ͭwhileϧʔϓ ʹ͔͔Δ࣌ؒ΋ܭଌͯࠩ͠͠Ҿ͘ ͦͷͨΊɺܭଌର৅͸ϒϩοΫͰ͸ͳ͘ඞͣจࣈྻͰ༩͑ΒΕΔ Rubyͷىಈ࣌ؒ΋ܭଌͯ͠͠·͏ΦϦδφϧͷbenchmark/ driver.rbͱҧ͍ɺੜ੒ͨ͠εΫϦϓτ಺Ͱ࣌ؒͷܭଌΛ͢Δ ىಈ࣌ؒͷϒϨͷӨڹΛड͚ʹ͍͘
16. 3. RubyʹՃ͑ɺYAMLͰ؆͔ܿͭॊೈʹܭଌ͕ߦͳ͑Δ ಈతͳϒϩοΫͰ͸ͳ͘จࣈྻͰ༩͑ΔҎ্ɺΘ͟Θ͟RubyͰϕ ϯνϚʔΫΛఆٛͤ͞Δඞཁ͕΋͸΍ͳ͍ YAMLͷγϯλοΫεͰϕϯνϚʔΫΛఆٛ͢Δͱ୹͘ॻ͚Δ .travis.yml ΍ .circleci/config.yml ͘Β͍ͷؾܰ͞Ͱஔ͍ͯ΋Β͍͍ͨ Optcarrotʹ͸طʹYAML͕഑ஔ͞Ε͍ͯΔ
17. 4. ෳ਺ͷRuby࣮૷ؒɺgemͷผόʔδϣϯؒͰൺֱͰ͖Δ RubyόΠφϦΛىಈͯ͠ܭଌ͢ΔͨΊɺෳ਺ͷRuby࣮૷ؒͰ ൺֱͰ͖ɺRuby 3x3ʹ࠷ద Ұ౓RubyΛىಈͨ͠Βͦ͜Ͱಉ͡gemͷෳ਺όʔδϣϯΛڞ ଘͤ͞Δ͜ͱ͸ෆՄೳ͕ͩɺbenchmark_driver.gemͰ͸ͦͷൺ ֱ͕Ͱ͖Δ
18. ࣮ࡍͷ࢖༻ྫ
19. Haml 4 vs Haml 5
20. ERB Ruby 2.4 vs Ruby 2.5
21. Optcarrot
22. VS JRuby
23. VS TruffleRuby
24. Docker Time.utc
25. Docker Time.utc w/ $TZ
26. Ruby 2.6 block.call
27. benchmark_driver.gemͷ։ൃ Ҏ֎ͷ੒Ռ෺
28. ϕϯνϚʔΫηοτͷ༻ҙ FluentdϕϯνϚʔΫͷࣗಈԽɾbenchmark_driverରԠ benchmark-driver/fluentd-benchmark MJIT-benchmarksͷbenchmark_driverରԠ benchmark-driver/mjit-benchmarks Watson1978͞ΜͷϕϯνϚʔΫηοτͷม׵෺ benchmark-driver/ruby-method-benchmarks
29. benchmark-driver.github.io
30. benchmark-driver.github.io GitHub pages ϕϯνϚʔΫαʔόʔ cronͰruby/ruby ΛϙʔϦϯά + ϕϯνϚʔΫ middleman build git push Travis ϕϯνϚʔΫ݁Ռͷ YAMLΛอଘ͢Δ GitHubϦϙδτϦ Πϯλʔωοτ ݁ՌͷYAMLΛ git push webhook ݟΕΔ आΓͯΔαʔόʔͷ࢓༷Ͱ ֎෦͔ΒΞΫηεͰ͖ͳ͍
31. benchmark-driver.github.io benchmark-driver/ruby-builder, skybench, sky2-infra… RubyBenchͩͱϓϩδΣΫτظؒதʹ͍͡Γ౗͢ͷ͕೉͍͠ ͱߟ͑ͨͨΊɺࢼݧతʹผϓϩδΣΫτͱͯ࢝͠ಈ ࣮ࡍʹ͸ϓϩδΣΫτऴྃޙ੠Λ͔͚ͨΒ͙͢৭ʑͳݖݶ Λ΋Β͑ͨ ৄࡉ͸ޙ൒Ͱ࿩͠·͢
32. 2. ϕϯνϚʔΫϝτϦΫεͷϞσϦϯά
33. ม਺ଟ͗͢໰୊ ܭଌର৅ͷεΫϦϓτୡ ࣄલ࣮ߦεΫϦϓτͷ૊߹ͤ ܭଌ͢ΔϝτϦΫεɺ͋Δ͍͸ܭଌํ๏ Rubyͷόʔδϣϯ gemͷόʔδϣϯ
34. ෳ਺ϝτϦΫεͷಉ࣌ܭଌ ܭଌ͢ΔεΫϦϓτͷҰཡͱRubyͷόʔδϣϯҰཡ͚ͩͰදʹͳ͍ͬͯ ͯɺෳ਺ͷϝτϦΫεΛࠞͥΔͱਏ͍ Ͱ΋user, systemͷCPU࣌ؒͱ࣮࣌ؒΛಉ࣌ʹग़͢Έ͍ͨͳͷ΋΍Γ͍ͨ (Ϟσϧ্͸ରԠ͚ͨ͠Ͳ࣮͸·ͩ͜Ε࣮૷ͯ͠ͳ͍) શͯͷϢʔεέʔεΛΧόʔͭͭ͠΋ഁ୼͍ͯ͠ͳ͍ϞσϧΛͲ͏ʹ͔ ࡞Εͳ͍͔?
35. v0.11~v0.12ݱࡏͷϞσϧ
36. v0.11~v0.12ݱࡏͷϞσϧ
37. v0.11~v0.12ݱࡏͷϞσϧ
38. ݕ౼ͨ͠ࣄ߲ HashͷΩʔ͸ඞͣΠϛϡʔλϒϧͳ΋ͷʹ͢Δ ͡Όͳ͍ͱOutputϓϥάΠϯ͕Hash͔Β஋Λࢀর͢Δ࣌ʹࠔΔ Outputͷ͠΍͢͞ΑΓRunnerͷ࣮૷͕ഁ୼͠ͳ͍ͷΛ༏ઌͨ͠ߏ଄ ಉ࣌ʹऔΒΕΔϝτϦΫε͕ͳΔ΂͍ۙ͘৔ॴʹདྷΔߏ଄ Output͸࠷ѱ࠷ޙʹ·ͱΊͯͲ͏ʹ͔Ͱ͖ΔͨΊ
39. 3. ϓϥάΠϯΞʔΩςΫνϟ
40. աڈͷϓϥάΠϯࣄྫ ϓϥάΠϯαϙʔτͷ͋Δ๻ͷ͜Ε·ͰͷϓϩδΣΫτ cookpad/barbeque itamae-kitchen/mitamae benchmark-driver/benchmark-driver
41. ڞ௨఺: ࣗ෼͔͠ϓϥάΠϯॻ͍ͯͳ͍ ͔͠΋ࣗݾධՁͰbenchmark-driverͷΠϯλʔϑΣʔε͕Ұ൪ ΍͹͍ ͦ΋ͦ΋͋·Γந৅Խʹ޲͍ͯͳ͍ཁ݅ͳؾ΋͍ͯ͠Δ ͳͷͰ࿩൒෼ʹฉ͍͓͍͍ͯͯͩ͘͞
42. ϓϥάΠϯγεςϜͷ֓ཁ
43. 2छྨͷϓϥάΠϯαϙʔτ RunnerϓϥάΠϯ: ܭଌํ๏ͱܭଌ͍ͨ͠ϝτϦΫεΛܾΊΔ OutputϓϥάΠϯ: ܭଌ݁ՌͷूܭΛߦͳ͏
44. RunnerϓϥάΠϯ Ruby API͔Β࣮ߦ͞ΕΔ͕ɺYAMLͷύʔε΋͜ͷRunnerϓ ϥάΠϯ͕ߦͳ͏ RunnerͷType͸benchmark-driverίϚϯυͰ༩͑ΒΕΔ͕ɺ YAMLͷதͰ΋ॻ͚ΔͷͰɺYAML಺ͰTypeΛݻఆͭͭ͠ಠ ࣗͷYAMLύʔαΛॻ͘ͱ޷͖์୊YAMLͷจ๏͕֦ுͰ͖ Δ
45. RunnerϓϥάΠϯ σϑΥϧτͷYAMLఆٛͷ΋ͷ: ips: ࣮ߦ࣌ؒ/࣮ߦճ਺Λܭଌ͢Δ once: ςετ༻ʹ1ճ͚ͩ૸ΒͤΔ time: ࣮ߦ࣌ؒ memory: Max Resident Set Size (ࠓ͸LinuxͷΈαϙʔτ)
46. RunnerϓϥάΠϯ ಠࣗͷYAMLఆٛͷ΋ͷ: recorded: record outputΛ࢖͏ͱੜ੒͞ΕΔYAMLͰɺಉ͡ܭ ଌ݁ՌΛԿ౓΋ग़ྗͰ͖Δ ruby_stdout: RubyίϚϯυͷstdoutΛύʔεͯ͠ϝτϦΫε ΛऔΔ
47. memory
48. memory
49. Optcarrot
50. Optcarrot
51. Runnerʹ͍ͭͯҙਤతʹαϙʔτΛ΍Ίͨػೳ Ruby͔Β࣮ߦ࣌ɺจࣈྻ͡Όͳͯ͘ϒϩοΫͰܭଌର৅Λ༩͑Δͷ΋α ϙʔτ͍ͯͨ͠ Ҡߦύε + ෳ਺όʔδϣϯ͕ࢦఆ͞Εͳ͚Ε͹evalͰ૸Βͤͨํ͕଎͍ ࣮͸࠷ॳ͸͜ͷػೳ͕Runnerͱͯ͠ϓϥάΠϯαϙʔτ͞Ε͍ͯͨ(3छ ྨͷϓϥάΠϯ͕͋ͬͨ) υϥΠόͱಉ໊͡લۭؒͰ૸ΒͤΔ෼࣮ߦ͕ΫϦʔϯ͡Όͳ͘ɺಉ͡εΫ ϦϓτΛ࣮ߦ͢Δʹ͸αϙʔτ͕ਏ͔ͬͨ͠ϝϦοτ΋ബ͍ͷͰ΍Ίͨ
52. v0.11~12ͷRunnerϓϥάΠϯͷॻ͖ํ YAMLΛந৅Խ͢ΔJobͱɺYAML͔ΒͦΕΛ࡞ΔJobParserΛఆٛ͢Δ #initializeʹ͸ɺCLIΦϓγϣϯΛ࣋ͭconfig, OutputΛந৅Խ͢Δ output, δϣϒͷϚτϦΫεΛ࡞Δcontexts͕༩͑ΒΕΔ #run ͸޷͖์୊࣮૷͍͍͕ͯ͠ɺoutputʹରͯ͠ϝιου܊Λܾ·ͬ ͨॱ൪ʹݺ͹ͳ͚Ε͹ͳΒͳ͍ #with_warmup, #with_benchmark, #with_job, #with_context, #report
53. OutputϓϥάΠϯ લड़ͨ͠ϞσϧΛड͚औͬͯूܭ͢Δ໾ׂ 1ͭܭଌ͢Δ͝ͱʹoutputʹ౉ͤΔΠϯλʔϑΣʔεͳͷͰɺ ݁Ռ͕ετϦʔϛϯάग़ྗͰ͖Δ શ෦ऴΘͬͨޙʹόϧΫͰग़ྗ͢ΔͨΊͷ؆୯ͳΠϯλʔ ϑΣʔε΋༻ҙ͍ͯ͠Δ
54. OutputϓϥάΠϯ ϏϧτΠϯͷϓϥάΠϯ compare: benchmark-ips෩ͷൺֱग़ྗ simple: benchmark/driver.rb ෩ग़ྗ markdown: GFMͷςʔϒϧग़ྗ record: ܭଌ݁ՌΛอଘͯ͠ผͷoutputʹྲྀ͢༻
55. OutputϓϥάΠϯ ඇެࣜϓϥάΠϯ gruff: gruff.gemͰάϥϑ͕ग़ྗͰ͖Δ rubybench: RubyBenchʹ݁ՌΛૹ৴
56. Markdown
57. gruff
58. ϓϥάΠϯʹؔ͢Δ஫ҙࣄ߲ v0ͷؒԿ౓΋ϓϥάΠϯΠϯλʔϑΣʔε͕มΘ͍ͬͯͯɺv1ʹ͢Δ·Ͱ͸ ֬ఆ͡Όͳ͍Ͱ͢ Runner Typeݻ༗ͷprelude͕άϩʔόϧͳcontexʹೖͬͯΔͷ͸Ҿ͖ണ͕ ͢༧ఆ ϓϥάΠϯΛ࡞ͬͯͩ͘͞Δํ͸ґଘΛ "benchmark_driver", "~> v0.X.Y" ʹ ౰໘͸͍͚ͯͨͩ͠Δͱॿ͔Γ·͢ ಉ͡ v0.X ͷؒ͸յ͞ͳ͍Α͏ʹ͍ͯ͠·͢
59. 4. ϓϩδΣΫτऴྃޙͷ׆ಈ
60. RubyBench
61. RubyBenchͱ͸ RubyͷύϑΥʔϚϯεΛ؂ࢹ͢ΔCI benchmark-driver.github.io ΑΓ΋େ෼લ͔Β͋Δ
62. RubyBenchͷΞʔΩςΫνϟ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …
63. ͜͜Λஔ͖׵͍͑ͨ RubyBench ϫʔΧʔ task-spooler → docker run ruby … GitHub fork for ruby/ruby RubyBench web Rails ΞϓϦέʔγϣϯ → Sidekiq GitHub ruby/ruby cronͰߋ৽ Webhook ssh tsp …
64. RubyBench΁ͷ౷߹ JITͱͷൺֱάϥϑ͸ޙ͔ΒRubyBenchʹ΋࣮૷͞Εͨ όάͬͯͨͷͰ௚ͨ͠Γͨ͠ benchmark_driver޲͚ͷϕϯνϚʔΫηοτ΍ਫ਼౓ͷྑ͍ܭଌػೳ͕ࠓޙ׆༻͠ ΍͍͢Α͏ɺϕϯνϚʔΧʔͷbenchmark_driver΁ͷஔ͖׵͑Λ΍͍ͬͯΔ طʹOptcarrot͸benchmark_driverͰܭଌ͞Ε͍ͯΔ ॏෳ͕ͪ͠ͳ࣮૷͕ϓϥάΠϯͰந৅ԽͰ͖͓ͯΓbenchmark_driverͷϓϥά Πϯػߏ͕׆༂͍ͯ͠Δ
65. RubyBench޲͚ʹɺϕϯνϚʔΫͷ֦ॆ΋΍͍͖ͬͯ·͢ VladͷMJIT޲͚ϕϯνϚʔΫूΛ௥Ճ༧ఆ Watson1978͞Μͷ֤ϝιου޲͚ϕϯνϚʔΫूΛม׵ͨ͠ ΋ͷ΋࢖Θ͍͔ͤͯͨͩ͘͞΋? https://github.com/benchmark-driver/ruby-method-benchmarks
66. ·ͱΊ benchmark_driver ศརͰ͢ͷͰɺੋඇ͝ར༻͍ͩ͘͞ Rubyͷߴ଎Խ΍RubyBenchʹطʹຊ੒Ռ͕׆༻͞Ε͍ͯ·͢
67. ँࣙ ຊϓϩδΣΫτ͸ɺ࡫ాߞҰ༷Λ࢝Ίͱͨ͠RubyΞιγΤʔ γϣϯ༷ͷ͝ڠྗʹΑ࣮ͬͯݱ͢Δ͜ͱ͕Ͱ͖·ͨ͠ɻ͜ͷ ৔ΛआΓͯޚྱਃ্͛͠·͢ɻ