Learn Ractor

RubyKaigi 2023 Day2

Transcript

1. Learn Ractor seki@ruby-lang.org
2. E-book version still available The dRuby Book is out of print, but... スタンプラリー対象外です。Not eligible for stamp rally 2
3. Agenda Concurrency is everywhere Introduction to Ractor case study 3
4. Agenda Concurrency is everywhere About me Concurrency is everywhere Introduction to Ractor case study 4
5. About me Masatoshi Seki, Ruby Core Committer (dRuby, Rinda, ERB), Programmer I started programming 40 years ago writing complex embedded systems for 30 years system with concurrency 40年くらいコード書いてて、30年くらいは複雑なシステム書いてるよ 5
6. Complex embedded systems dRuby in the real-world embedded systems. ΩϠϊϯϝσΟΧϧγεςϜζגࣜձࣾ
   ҩ༻ίϯϙʔωϯτٕज़։ൃηϯλʔɹؔকढ़ɹԂ઒ཾ໵
   4FQ ɹ - ref. RubyKaigi takeout 2021 ミニマム構成の製品を使って若者をRubyistにした事例発表 About us Masatoshi Seki, Ruby Core Committer (dRuby, Rinda, ERB), Programmer Tatsuya Sonokawa, Programmer Miwa Fukaya, Tochigi RubyKaigi Content Committee Chair, Software tester ਓؒυοΫޙͷ࠶ݕࠪͰࡱͬͯ΋ΒͬͨࣸਅͰ͢ɻେਓͳͷͰɻ 9 404
7. Complex embedded systems dRuby in the real-world embedded systems. ΩϠϊϯϝσΟΧϧγεςϜζגࣜձࣾ
   ҩ༻ίϯϙʔωϯτٕज़։ൃηϯλʔɹؔকढ़ɹԂ઒ཾ໵
   4FQ ɹ - ref. RubyKaigi takeout 2021 X-ray CT image of my gallstones About us Masatoshi Seki, Ruby Core Committer (dRuby, Rinda, ERB), Programmer Tatsuya Sonokawa, Programmer Miwa Fukaya, Tochigi RubyKaigi Content Committee Chair, Software tester ਓؒυοΫޙͷ࠶ݕࠪͰࡱͬͯ΋ΒͬͨࣸਅͰ͢ɻେਓͳͷͰɻ 9 404 Gallstones
8. Complex embedded systems Lots of processes, and explicit and implicit IPC 無数のプロセスとプロセス間通信がある。明示的だったり暗黙的だったりするよ 8
9. Concurrency is everywhere Concurrency exists at various levels of abstraction そこら中で時間的にオーバーラップして実行してる。ということはつまり... 9
10. Concurrency is everywhere Sequential execution is a special case むしろ、逐次実行は特殊なケースである 10
11. RubyKaigi takeout 2020 Rinda in the real-world embedded systems. seki@ruby-lang.org First printing from 2005 (Still available) dRuby ʹΑΔ
    ؔ
    কढ़
    ஶ
    ෼ࢄ ɾ Web ϓϩάϥϛϯά
    dRubyとRindaによる天文台のロボット望遠鏡（ガチ）の話 - 🤖🔭
12. RubyKaigi takeout 2020 自律して動作する装置／センサー群を統合するシステム。抽象度が高い感じする - Linda coordination model software block diagram 1 psz117 X f/2.5 20151207_yanagisawa.pdf ΑΓ OAOWFC䛾እほ 2015/12/07 奛㷔塭伖㈨埻WS 16 X
13. OpenMP 比較的低いレベルの例 13 int main(int argc, char *argv[]) { int i; #pragma omp parallel for for (i = 0; i < 10000; ++i) { /* ... */ } return 0; }
14. OpenMP 14 0...10000 int main(int argc, char *argv[]) { int i; #pragma omp parallel for for (i = 0; i < 10000; ++i) { /* ... */ } return 0; }
15. OpenMP 15 0...2000 2000...4000 4000...6000 6000...8000 8000...10000
16. Agenda Concurrency is everywhere Introduction to Ractor case study 16
17. Ractor parallel avoid race condition isolation 本当のスレッドと保護されたメモリー（オブジェクト）で 17
18. Thread to Ractor 単純なケースではThreadのように使えるよ 18 >> t = Thread.new(*arg) {|*arg| ... } ... >> t.value >> r = Ractor.new(*arg) {|*arg| ... } ... >> r.take
19. Thread to Ractor Ractorはtakeで値を取り出すのがちがうよ。 19 >> t = Thread.new(25) {|n| (1..n).inject(1, :*)} ... >> t.value => 15511210043330985984000000 >> r = Ractor.new(25) {|n| (1..n).inject(1, :*)} ... >> r.take => 15511210043330985984000000
20. avoid race conditions isolation block scope global variables shareable object 競合状態を嫌ってRactor間をまたぐオブジェクトに強い制限がある 20
21. isolation block scope Procだけど、外側の変数にはアクセスできないよ 21 >> hash = {'foo'=>'bar'} >> r = Ractor.new {hash['foo']} <internal:ractor>:271:in `new': can not isolate a Proc because it accesses outer variables (i). (ArgumentError)
22. isolation global variable グローバル変数にはアクセスできないぞ 22 >> $foo = 1 >> r = Ractor.new {$foo} #<Thread:0x0000000106ed7018 run> terminated with exception (report_on_exception is true): (irb):11:in `block in <top (required)>': can not access global variables $foo from non-main Ractors (Ractor::IsolationError)
23. pass by value deep copy 引数経由して渡せるよ。でもRactorをまたぐときに複製になる。dRubyみたいだ 23 >> hash = {'foo'=>'bar'} >> r = Ractor.new(hash) {|it| it['foo'] = 'baz'} ... >> r.take => "baz" >> hash => {"foo"=>"bar"} >> r = Ractor.new(hash) {|it| it.object_id} ... >> [hash.object_id, r.take] => [2070360, 2006160]
24. shareable Ractor.make_shareable make_shareableするとRactor間で共有されるぞ！でもfreezeされてしまうのだ DRbUndumpedみたい 24 >> hash = {'foo'=>'bar'} >> Ractor.make_shareable(hash) >> Ractor.new(hash) {|it| it.object_id}.take => 221520 >> hash.object_id => 221520 >> hash['foo'] = 'baz' (irb):16:in `<main>': can't modify frozen Hash: {"foo"=>"bar"} (FrozenError)
25. can't make shareable Queue, Monitor, TupleSpace 同期プリミティブは渡せない 25 >> Ractor.make_shareable(Queue.new) <internal:ractor>:820:in `make_shareable': can not make shareable object for #<Thread::Queue:0x0000000104c8b068> (Ractor::Error) >> Ractor.make_shareable(Monitor.new) <internal:ractor>:820:in `make_shareable': can not make shareable object for #<Monitor:0x0000000104e47398> (Ractor::Error) >> Ractor.make_shareable(Rinda::TupleSpace.new) <internal:ractor>:820:in `make_shareable': can not make shareable object for #<Monitor:0x00000001051e84c0> (Ractor::Error)
26. shareable object immutable class/module special object Ractor ENV 26
27. Ractor has two queues. takeしか使わなかった... 27 incoming queue Ractor#send / Ractor.receive outgoing queue Ractor.yield / Ractor#take
28. avoid race conditions Using Ractor is easy, but writing for Ractor is hard. straitjacket 拘束衣。オブジェクト指向プログラミングのように書くのはたいへんだと思う 28
29. PG Thought it wouldn't work, but it didn't 動かないと思うけど動かないもの 29 >> conn = PG.connect("postgres:///masaki") >> Ractor.make_shareable(conn) <internal:ractor>:820:in `make_shareable': can not make shareable object for #<PG::Connection:0x000000010469d8e0> (Ractor::Error) >> Ractor.new { PG.connect("postgres:///masaki").exec("select 1") } #<Thread:0x000000010a638048 run> terminated with exception (report_on_exception is true): /opt/homebrew/lib/ruby/gems/3.2.0/gems/pg-1.4.6/lib/pg/ connection.rb:70:in `conninfo_parse': ractor unsafe method called from not main ractor (Ractor::UnsafeError)
30. pp Thought it would work, but it didn't 動きそうで動かなかったもの。私の使い方が悪いのかもしれないです。 30 >> Ractor.new {pp 1} #<Thread:0x000000010a79f9e0 run> terminated with exception (report_on_exception is true): <internal:/opt/homebrew/lib/ruby/site_ruby/3.2.0/rubygems/ core_ext/kernel_require.rb>:40:in `require': can not access non-shareable objects in constant Kernel::RUBYGEMS_ACTIVATION_MONITOR by non-main ractor. (Ractor::IsolationError)
31. a situation suitable for Ractor Accelerate loops with parallelism numeric calculation in-memory database さっき説明したOpenMP見たい領域が得意そう。 31
32. There is an in-memory search engine 用意しておきました！ 32
33. Agenda Concurrency is everywhere Introduction to Ractor case study 33
34. RubyKaigi 2022 Create my own search engine. seki@ruby-lang.org ࣗ෼༻ͷݕࡧγεςϜΛ࡞Δ࿩ - There is an in-memory search engine dRubyについて話せなかったが、ポケモンカードの検索エンジンの話
35. Pokémon Trainer Energy Pokémon TCG Build a deck with 60 cards
36. Pokémon TCG similar deck search Heroku → Oracle Cloud Infrastructure VM : Arm x 4 core do not use containers OCIに移行したので、コンテナを使うのもやめた 36 2023
37. Weekly deck trend analysis 新機能！ 37 case 1
38. Hierarchical clustering Calculate distances for all combinations Merge closest node into cluster 38
39. Hierarchical clustering Calculate distances for all combinations Merge closest node into cluster 39
40. Hierarchical clustering Calculate distances for all combinations Merge closest node into cluster m C2 40
41. Hierarchical clustering Calculate distances for all combinations → mCn m C2 41 irb(main):001:0> Array.new(6).combination(2).size => 15 irb(main):002:0> Array.new(900).combination(2).size => 404550
42. Hierarchical clustering Calculate distances for all combinations Merge closest node into cluster 42
43. Hierarchical clustering Calculate distances for all combinations Merge closest node into cluster 43
44. Hierarchical clustering Calculate distances for all combinations Merge closest node into cluster 44
45. Hierarchical clustering Calculate distances for all combinations Merge closest node into cluster 45
46. Hierarchical clustering 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 ↑ Height vVkVkf-CfQ2Me-5wF5k5 pM2ySp-NqdVVy-SEXyyy Ffvkv5-urPxNL-VkFdvk x888Y4-ZEG7rl-8a8c88 pySESM-DfgV3e-EpSpEy QggnnQ-rTu8JY-iingnn cY88a8-41UAjT-88Gc4D 888Y8K-M5mgp6-Y4cxcD pyySSU-fpvTh7-Upy2XM cY48J8-P6fNIl-Kxac8J HHnQnL-P97pwl-PHgnng MSpSpy-vAWjQl-MySMpS fkFkFF-zaC6UD-v1kkbw pSypUM-wkyESS-p2yyM3 8YDcxc-feOc08-4GY884 kFk1Fv-0TdM4n-dv5kkV Yx8ax4-JDBmvc-cYG8c8 SSyXy2-N8v9C7-y2p3yp pXy3yy-9MrduS-yUpSy3 Fdv1k5-WkoeFa-FkvVkk 84888c-xCG2yN-Y8cGD8 wkVvFV-nyF8eL-FVkkfk xa8a88-dom7r2-x8Y4cx gnngiP-TQ1apH-nPnnQn nNL9gn-eoFUGm-nNnngQ fVwwVw-ubXExr-FFkkv5 QQn6ng-IKDkHa-nNiNnn Y8ccYa-GItA31-cY488x NHnnPg-jQhceA-nQnnLL yEUMRy-2mG9g3-pSSMyM Fkk1Vw-d4Wozk-FFvvFf 6QQg9n-AbEJHX-giPngn gN6nQn-ckH59n-9inQnL D88xY8-hpaIDr-8cxcYc LQggQn-hv20uI-HnnnQg YcK48x-cDdQ0X-xxGD8D nHL6Hg-knDcUp-6nQngQ SSMUpp-INYSgw-ppyySy nng96n-lBix3f-LQQLgg SMRSyy-GHEWTP-MyXpyE Vbkv1k-tOY61g-F5kvVk PnnnLn-PEaw1c-QLnngg pypSpy-c9cBp2-SSyypp XyyyyX-TrT2ek-SpyMpy LHHgHQ-mLxVMM-nngnn9 kVFkFb-AN5NWI-kvvfFb kv1v1f-vaYx04-kFfkkV x88J88-jBN0Jj-8Yc8Kx ySM3yy-gnhe17-XyUSyp 8xYD8D-npQXH8-KYYcxY Y4Y84x-MrJeFq-cDD488 nnnNgQ-VNO3JO-igNnNg 5FFk1k-hWutyR-vwVk5F Q6nLNN-H90bPb-nLginn pyySS2-H7DN4A-pUSyRy SEpEyy-hbxATA-SyyppM YKc8xY-R3feUE-8cc8a8 g69ngL-gWWKti-QinnnH nLgNQn-JVUT8W-HPHQLn D8Yx88-lrJIuu-JD84c4 kv55F5-3erdMp-VkFFwk v5VFf1-UKYbZn-VbVkkk Y48Y8a-RJ2LDC-8c8xcD pS2y2p-ZSaQvC-MSUyyy Sypyy2-K6PSF4-MppXXy K8c8YK-4jx3NW-8Dac48 XyRyyU-qrp4V9-pMRySM J888cc-lxoCeB-4DYJ8c giPgni-OJ09Yi-gngnnQ Y84cxx-3rJwVM-cYY84c ngiQgN-CnQ4gv-6nQngn kkkFVb-zoKBee-vFkFvk 6QgHnn-2VFveI-QnnngH 3yRSyy-D8i0T9-yy2ppM gnLnQQ-Cyt2rM-L6nnLN MUpSyX-xcRVC3-pyyRpS dkvkfk-77tnCm-FFVfFV F1kkFf-MVW9eT-vkvFkd kkkFkk-xYY2UD-kvw15V nngHng-OBmOQL-NQHLQL 8xKD84-DzKFZJ-4xK8cJ Q9ngLn-bjLzmZ-gQnQgQ NPnLgn-58T69t-nQnniH 8GD84D-DuK5jM-xYc84x Vk5vkk-bcQSA0-FkvFkv LLQnQn-cTWB0Y-gNQNnn nLnQnn-mtbSBC-ngQLHL 6ngniN-9Dj0L4-NninnQ iinngg-njYbYw-ngnNHQ ypUyy3-3rxk2j-p3pSy2 SpyyRM-Cq2rZQ-UyyMEy 55kvkF-p1hUSP-kFkFkb FkFVdF-Clq0WL-kvv1kd gnLngg-3NcLDa-Q9nPPN 1FVkvk-2dvvTK-kkdkVF Mypyyp-g0z6Cl-EMpSSy nnngQQ-QwK7at-9P9gnn cxY8G8-59CrtL-8c488x 9LNQg9-kuPro1-ggnQnn D888Dx-mAJVcL-KaYx8c LLLgnn-VdBYdx-gLnNnQ gHniQg-bmCra9-Lnggni Eyp23X-6lKVUo-MpppyS nPLgPN-2Vu2fH-gLgnQL x8Y48c-XDRyTC-KYaccc x8Y8Y8-LANoAw-88Jc8D acY88c-uS50Np-8xYxYc x88DYD-yXtIEy-8cJ88x 8Y4ccx-kwQSDk-48x8DG gngHng-dMAxgW-n6nQQ6 LPNnLn-G0fVYk-iQNnng NQLnnN-vDiFBq-nP9gNn Fvbkff-MmzaQA-kvVwVk c88Yc4-YeMfSt-8c48ca kkvdVV-OgMIQX-kFkFkk 48YKYc-HRXkZA-Ya8c88 nnQng6-DRu6oG-gnigin kFvkkv-iKWU1R-kbVvfk Fv5FFk-HQjlQO-kwkvkv LngLgN-Sdu3m1-nNLnQn SyMyy2-9owvQk-SypXMp kkvVfk-Q24Ze6-kkfFvF gHing9-jvXlft-QgLQni 8YaxDD-5uPjMv-c84xKc gLnQQn-oProA3-gLQgn9 gngQgg-bLqQCo-g6nQn6 8xca8Y-YGIUkZ-8x8Ycx nnn6NQ-oUJKe0-HQLngg D8GcY8-rW9jGo-Yxx8cK SypypS-g77hSn-MypUyp E2My2p-GEKcv4-pUySyR gQQNgL-oBkSWV-LQnPnn FkdVVv-O2oHwv-bbvkd5 88xxGa-Og85PD-c8Ya8D EMSyp2-2fQqfN-yyESUp YY84c8-RR4n4d-8c8xJx gnLnQg-euTM2D-gnnHQL xcc4YJ-8JSn76-Y8KDc8 Y8888c-GraJIz-x4Kcc4 c88x8Y-RdPWMA-88cD8Y k5FV1v-975h5I-FVFkkv LPNgQn-3q0i7b-6ngHNn nggnn9-4PJZ5K-gLQHng 8YJ88K-iGHqSA-xcxDc8 g9QngQ-trb93k-NnnHng pS3ypy-0GDmyP-yXMSUp Y8GJx8-c3qvQb-KY4Kcx bFkdkF-QDqqYK-vFkbbk QQgPnn-vVzh0f-gnNnLg ySyyR2-K40qf2-pXyMpp kkVFkF-2fCbID-vkdFFV x8Y8K8-t24oZW-acYx8c 8Y8Jc8-lxt7oi-x84cJY LLinnn-WGGLjk-LQnQgn EXpyMM-kVHZFw-MXyS2y 88xYDc-Utex6g-8cxx8c DKY884-G2Uobg-8aJcxx Eyp2yp-X8YjPR-3pSySU F5vVVv-acRcO1-FkFkdF cx8xJx-I5OlcH-888caY XRXSMy-00j6oU-pyppyX LnQnnH-mXhDzP-NgHngQ 5vvFkk-lOTpux-Vvkffk xKJcY8-k6OH37-8D8c8c VkvV5F-fZPXeQ-FkvvFV FvFwvw-PO6yND-kfbVFk vkb1kk-JjQzd4-VfFFfb p3SXEp-zAnOdf-yMSyXp 6nQgLg-796wEi-9innHL ngnNNP-YJpVX9-n9QnHN SXXppM-Aio17g-3MyMyE gQLgng-qJnyrq-nninQP 6nnQnn-ZbOkdT-LPLggN FkwvVk-0JXBtY-Vk1kv5 9n9nnQ-LG4gBy-L9LnQg dvkkFk-nu07S3-FFF1df L6Ln6i-ecLiWp-NgQQng p2pyMy-YUHLGz-pMpU2S 8cY48D-Kqr0g6-8cJ88D xD8c8K-1V2oBq-Y8aac8 1v1VkF-lwRkxD-FkVvkk cY8D8c-zdx6Xc-Y8cx8a a8KcKc-hzlU7N-8Dcx8Y y2pyMX-4GZa8x-MMyS2M yyppMX-i88khM-pS3SSU kkFk5k-pQFm92-kvvkFv bFFvfV-LdQ9FU-vk5kwF gnnHHP-3TnKB4-QngLQg ngnQnQ-Kr6ml6-LLgNnH x84YJc-BKoP5V-c8cx8D YcD84c-CKpkih-8K8DY4 8YDcx8-kAMY8T-D448cY kvwkFk-OCov4S-kwvVVf gnnnNg-lpf6hT-gLnQQg 88xYxc-3H7q7f-J8YaYY Xyypyp-qFmtto-My2ySE nQngPP-57dCZZ-NnnngL NLnnQg-WXipiX-LNgQin w5kbVk-XttGD8-vkkFkF 4cGx8G-S0wNBC-8Ycc4x 8GxcYx-OH2UUH-8cc48a SyXyyp-CI48IF-XMyMpy 6nN6nL-KBuiUY-nggQnL ax8cY8-6q8I5j-aKY4Yx F1vFvd-BqFbRu-FkfkFk XMpySy-Pdzy6z-yMEM2p Fwkbkk-uMrdbS-vVfFkk 2MypyM-gQ7P5G-XySEMy QLgQLn-1nKVTu-gQQQnH Spy2yM-4Nvzt6-URyyXU ypUpXM-ztrfJL-RyyypS 8ccJcY-kkMIDl-8Kc8a8 gLLggn-yvCK39-PnHQgn 8aac8D-zIwhY9-88GxYx xaca88-4E9ceZ-Yx8c48 ngnQQg-oxqaRX-nLH9L6 vkvkVF-RxL7MG-kVVv5v 8Y8D88-fJIb8F-xYx8c8 8x8x8K-82daju-xcD8Yx c8Jc8Y-GU8JRy-8a88cY D848cK-0jaJxg-J88YYc Vv5FkF-bOeyK8-Fkkf5f gQNnHn-WGS9K0-gN6Pnn bvkkwF-NhDuQr-kFvkVF kk1Vfk-4aWYnb-Fvbk1f kVFbFF-erSOuJ-wkk1kv HPHQ6n-9lHaAh-LPng6n 88YcaY-0qIYTk-xDY88J YY88Yx-AgFk80-8xDDDc Yc8c8K-nldoLK-888YKc UpypES-Ol9Ujw-yyyMyy YcacxD-KbQrek-8c88Yc D8J8Y8-SJKX5n-84YGc8 Fvk5dk-K6auXt-5VVV5F MyySXy-LHYR2U-p3pRMp vvkkwk-7TNI6g-vFFkfF RSMy3p-JMw8qI-yyySpS RyySSy-60b8OH-pXSSMy 8Y8888-AVb4Pb-DacxcD MyXpSE-PSA5X7-ypySSy gQQnni-IzZz4R-nPnnLn bkkkvV-j2TTzD-1Fkfvk gHnQLg-2LcmM7-nHnigP yUXMyU-RA8N0r-SMSpy3 LnnLgQ-CWkC2Y-niHnQH 888cc8-HWX2ML-8YcGcY k5kvFf-dxaYgv-VFvk5V xxc8Yc-q57E1T-JY88cD kVddkF-6pt4pn-VFfv1f gPgnnL-VbQ3lG-Q9nnQg FFvfkk-wDPW9d-FkfdFb S2yypX-QgNONa-MyRpyy Sp22Ep-mRnyTJ-yMy2Ep nnHgQQ-Pcwdh7-nHgigL FFkfdv-EJeIDW-5kFVkV 2ySpUM-fGliRn-pyMSSy iLiggn-fTlEhz-nnLQHP 3Spyyp-brGSW8-pUyMS2 3Spyyp-brGSW8-pUyMS2 nQnnnL-E6HSwW-gPnng6 ax8D88-0tZR3G-DcGYcc FFbkfk-Ahqclr-kkFvFV pypU3R-RHKaOf-3yMySy S2MMyp-tfWxGF-Upyyyp yyySyM-nM9cMj-Mp2EyX V1FdkF-A4ny3T-kkbVvd nNQLgQ-0Hvaqt-ngnngi fVvFFk-NUHTax-Fkkkkv kFkFf1-eRmTqv-vkv1kF yXMSyM-Qh65tM-yyUXSp yppMSX-yGCakE-2pyM2y fwVFVk-183UMQ-vkFFkv pSy2yS-UKDq4o-yMXyyS 88G4Yx-DwlHjt-c8cx8Y ypyyXy-pb3u0i-pySpyM Fkfk15-4XDphg-VbvVvF gNL6nN-f26zyi-ngQnQQ gQLggn-BTZkL7-QNHQin 2yyyMX-5YjiUW-XXppSX Fk1kvw-B62poY-FkvvFF MMXyyy-NSCi2H-pXpSER ggnLHn-LXeSkS-gPQQnQ 8ax8Yc-B1iwrH-K4D8cc kFbfF5-74ahY9-wkVVVv 88cY88-eFXdZr-xxYxcc K8xcxY-o5yMOo-c84YYD yyXSpM-Kcb8JW-MyppMy kvdVkF-5kFlwN-FFwfkF 888acc-8WUCbS-DxcYaD ngnQiH-QhNNz0-iPngng yM2pU2-QDc6cT-yyp2SU kkv5kV-7Pc53X-VfkFFk nLngQH-yer1jS-HggQLn UypMp2-3gaNFr-yMXp2S SpyyUM-4jH8rz-M2XMp2 kVkbwF-ajPl3H-vvkVdb VVkVFv-44HtbN-Fkkbdv kbkk5k-izx11l-FFFFvV UMSSSp-AJ7hpH-MREyyM G8caD8-jp4j8f-K4xYcx SyySyy-NuaovJ-MXXSpp yyEXpE-urWr81-EyyS2p gnLQLg-kDTow4-LLnngn NNQnHn-IDa7pf-QggNLn fVkVFF-tjWVdW-vVFkkd pyMypR-2T25Tq-RSyMpS pUXMyy-ja9d9w-SyMEUp gnQgnH-Ri9g32-LgHgNL 8DD8c8-OxZqL1-cYJacK kfkvVk-rPQHjL-b5kFb1 nNQnng-82kbpU-Hng9nN cYa888-p2GCPQ-88Y8cD xYx88c-E7tRMc-8cDaa8 pySyS2-qgNbnp-yMppME bk5kkk-o0af4U-kv5dFk Vk5Fkv-w4UgEO-FvvkdF kVkwFb-SxtreL-5Vkkvd Nnngnn-6o7P2u-ggPQ9Q cc8Kac-0EDf1N-Y8DxY8 VFFvFk-2NUr78-kwv51V SyEppM-Q7oNnr-pypUyy ngnQ6H-GtyUpN-nnnHnn LnninL-Gfjigu-PiLggQ nnLPQg-EHCVvo-NQgnnN nQLnQn-6Uj2Lk-gPnNQH nggQig-eA0RQN-HQnnnH 8Dc88Y-cw0ufi-cKc88D XyMyyp-UYPEdR-SX3SUp 8Ycca8-XkkqQg-aGY88K yUypSS-4WrxnC-XMppSy 4K8cx8-YqtVwW-cY88DY pM2pyR-kKwzxW-ppUSMy LnnnQi-hRaaWB-LnnP9g 88cc84-oeLGiw-KDcY8a FFdbkk-1iiZ6G-F5kv5k kfFvFv-8pxrcr-Fkkkkk ngHNgQ-LFjcFH-ngLLnQ QgLHgn-PhNwl4-gLQnNQ p2MyMM-A6l922-S22Spy 6ggnnL-3n7MKb-NnnnHQ 8cK8Yx-43UM4i-cY88c8 4YD8c8-udiY9H-K8c88c 9HPnQn-T5VVou-NgHnnn c8axx8-qVDXly-cxYcYD kkV5dF-vHHHJQ-Vfkvkk 5FkwdF-tf6KQ5-VkvFv5 ySpyyy-tNjCkC-ppp3My n6gQLn-DJ4wkP-gngnng nLnLgN-L3ErQj-Q9Hgng 66n6Ln-irXTVx-LngnQg a8YD88-7wjh7o-8ccca8 nQPgHH-N1XnV8-LLQLnn HnnLLN-kEtUgV-nHLngQ fkvkk1-uIqKay-k1kFvV SXXyM2-Zp7pPp-pRpp3y nggngg-6QpEdq-nQiHLN VF1fkk-gCRSjm-vkFVvk 8Y8aD8-PxTvBD-c88xJ8 kFkV1v-88YvWJ-kkkkFf c88YJ8-LzcBD4-xcc8ca pUppyp-w2d575-ypSyyp 8GYYcD-5rRgXA-8Gccx8 ypSMpM-ywy4eI-yy2pyS 8Yccc8-tYneHr-xDx888 9Qgn6N-riIe1q-nnngig vkVFkV-g7pgZ9-kkFwFv QnLLnn-Jl04DF-nnQngL kwdFFv-WWH1i6-wF5kkk 8DGcx8-lqWmWH-acDcY4 SpyEXy-rDPjYw-UyRppy 2EyMpS-6pzr09-ypMyyp dkkvkk-hViWbW-FwvkFw F5V5kV-F54puu-VvkdVv XyXyp3-xkCbUh-yyMSpM 4c8DY8-Ot68y0-ccYxGx nNQnQn-aCXGqU-9QgQQn VkFkvk-Hl7mGl-fkfk5k 2XySyp-yxhDQO-3SppUy MRypyS-2i4Url-pyypSS X2SyRU-LyWtXP-ySXpyy fVfkdv-RZJBcC-vFkVFb NnnnLQ-cFkMCg-nLgQin FFFkFv-RwtTEA-fkkFkf dkfkkk-R8qMjG-vFbfkk kvFFdF-UpHnMV-5k5kkk Fkkk5V-4oBPWI-VkkkvV ggHLnn-M0Pj8x-iHgn6Q yEpRp2-AvXSox-yySyyp gnnN9i-E039Jw-nLgggQ 6ngg6L-43MyEK-nggQnL RXXyp2-0pDXRa-yypSXM yX2yyp-VUSzza-pUypSM kvfFvk-GXYMh2-FVvwFF Ff5vkk-1HlTBb-F5kkkF y2XXSS-a7bZvT-ypEypM kVvF5V-GmUVQq-kdvvFk gnggn6-a2Ihzl-nQgiNQ LnnLQ9-xPq7uk-gLnNnN MXpyMy-AnJ1Pl-MSSppy QnnHnN-7Yhh3J-iLHnng HgLgHg-4OjrcS-nQnnnL yypS3p-rK8Xap-yyMRMS nNPQng-Fbu6qi-LgnNPL ngHnnn-qDHVrz-nngNQL x8x888-BdMNIF-D8Yacc NQniPn-0CeUUw-HgQngn Nnngnn-y9TBzW-Qin9Qn VkkkkV-RXSWMY-VkfFbv SpypEp-1S1DBJ-y3M2yS nnHLnL-oJk1fq-QiLgLg giQn9n-X9l3YG-ngNLQH gLinHL-lgIWYS-LPnNQn kkvwF1-cuoDmI-vkVkvk xY8488-iitdyJ-Kc8xac pXXyyS-QO24lS-pySUXy fVFFkk-k8iczo-1d5fkv NginNL-KTIcT4-nnnHQi cG888Y-7l8Nje-8DDK8c inNgng-LZEk1P-PgnQLn pySSRS-AQlvCZ-Upyyyy dkkF55-UPt79F-f5kv5k kFvfk5-v3CpqY-1bVFVk 6LQnng-M8eQmc-ngg9gn 88YcGY-iSyEBR-xcDDYc pyEEyy-rjtInL-ppSpMS nnnLLQ-xFzJFn-Lnngnn XypS2E-Pf3SLV-EpXyyy MyUyyp-qVn3L4-SSySpE c88c8c-tVBpGY-x8YYKG Fvkkkd-3fIdSd-kkk5Vd XyMMyp-1uGxEG-pySXEy k5fkvF-JVNG1E-kkkVbV 84Y8xJ-lx9wZe-Y8cxY8 yySyyy-JQPu4V-yRyUSp Y8Jcc8-qpCJbt-8D88YK fvv1Vk-UFvCIT-kFwkkf HNnnnQ-C0A9Nm-gHNQLn SypySM-VQTvjR-My2yyp 6PHngg-60tYV7-6QnnnL vkFkkF-iiI1X7-FFVkdv wkwdFv-cZuRig-VdFkkk 8YG8cY-z6ci46-8cD8Kc P9QnHg-kN8ED4-inLgng 8YY8Jc-o9uHiH-88xYDx kkdkF5-CIGccH-fF5kvV SUy3yM-OaMBcv-yypUpy ypMy2p-FFzmW2-ySpySy ypyXRS-L16SGq-SpyUpy Y8x8a8-8MNGCw-DccYx4 Ffv1vk-NxUbDV-fkVFFw FkfFkk-6vHnSh-VbvdVf F5k5kV-CciOJd-kvkVVF c8c4Dc-Teghco-c88YaD NQgnnn-3ibViR-L6ngnL 5Vkkkk-Kc0t7m-V5vFkv f5kkvk-35IYvc-kF5VdF pESSyp-gn0CgU-pyMRyS 8c888J-5hivVv-4acxYK yypMyX-MNe9fy-MpSUS3 FVVvvk-DJQmUN-kFkwVd yppyXp-3u6POe-SS2yyy 8KJaDa-Kj9bZ1-cDcY88 8Y4x84-bwtANd-axcx8J gQNQnn-xuRV4o-LnnngN VvwkFv-iy7fiU-wFkVbf ngiiQg-lOpYlt-ngLLnL ac8G88-JAl5JM-8DYx8c QPnnNL-9edpKq-HggnnL k1kkk1-n3MtXH-Fvv1kF fVvb1k-FvDpbQ-Fkk5fF c88cYG-pC3MhR-D8DYxa c88Gc8-hwd3Pl-8DY8a4 VdkkFv-ZYQbn4-fkVwkk 88YDx8-BGE0rQ-cc8484 2yppRS-BDzUOM-MMSEpy nQLQg6-rXiLHS-LLHggn vvVVFF-VATfWX-1Ffkkv cxac8c-PMljcj-88YccG MyXSyX-1GwVBR-S32ypp nnQgHH-OvNqkj-gNQngL cDc4YY-0JmWlW-Y88Ycx SUpSpy-vai0ka-XSyUyp cY8cxJ-xUWnJw-88DY48 ngQLg6-ZkMy0I-9nnnin NgLnNn-Lwmd5n-HngniQ xYx8ac-a5TU6t-Dc8c88 gNnnPn-ECzi8c-LnQngN SyXMM2-fSJmmA-SpMpMy kF5555-iF0iVE-vVkfVF FdkVvk-ij3jSg-fk51fF LgQgQn-9Jqh11-iQggQn FvkkvV-Ng42hI-FFFfkV bkfFVb-u7BzOm-fkVVfv ngQQnn-g3wfZb-igNPLP QgnNnN-qpdvJb-nHgnnH XMXypy-8mJmMd-SyRSXp kFVk5v-FTN4dQ-kVkkkf 6nQNLQ-qvq8TJ-LnnHgn XMSXy2-0O7Yzy-pyySpy nLgNgL-jK0iJz-QQnnng
47. Hierarchical clustering 0.0 0.1 0.2 0.3 0.4 vVkVkf-CfQ2Me-5wF5k5 pM2ySp-NqdVVy-SEXyyy Ffvkv5-urPxNL-VkFdvk x888Y4-ZEG7rl-8a8c88 pySESM-DfgV3e-EpSpEy QggnnQ-rTu8JY-iingnn cY88a8-41UAjT-88Gc4D 888Y8K-M5mgp6-Y4cxcD pyySSU-fpvTh7-Upy2XM cY48J8-P6fNIl-Kxac8J HHnQnL-P97pwl-PHgnng MSpSpy-vAWjQl-MySMpS fkFkFF-zaC6UD-v1kkbw pSypUM-wkyESS-p2yyM3 8YDcxc-feOc08-4GY884 kFk1Fv-0TdM4n-dv5kkV Yx8ax4-JDBmvc-cYG8c8 SSyXy2-N8v9C7-y2p3yp pXy3yy-9MrduS-yUpSy3 Fdv1k5-WkoeFa-FkvVkk 84888c-xCG2yN-Y8cGD8 wkVvFV-nyF8eL-FVkkfk xa8a88-dom7r2-x8Y4cx gnngiP-TQ1apH-nPnnQn nNL9gn-eoFUGm-nNnngQ fVwwVw-ubXExr-FFkkv5 QQn6ng-IKDkHa-nNiNnn Y8ccYa-GItA31-cY488x NHnnPg-jQhceA-nQnnLL yEUMRy-2mG9g3-pSSMyM Fkk1Vw-d4Wozk-FFvvFf 6QQg9n-AbEJHX-giPngn gN6nQn-ckH59n-9inQnL D88xY8-hpaIDr-8cxcYc LQggQn-hv20uI-HnnnQg YcK48x-cDdQ0X-xxGD8D nHL6Hg-knDcUp-6nQngQ SSMUpp-INYSgw-ppyySy nng96n-lBix3f-LQQLgg SMRSyy-GHEWTP-MyXpyE Vbkv1k-tOY61g-F5kvVk PnnnLn-PEaw1c-QLnngg pypSpy-c9cBp2-SSyypp XyyyyX-TrT2ek-SpyMpy LHHgHQ-mLxVMM-nngnn9 kVFkFb-AN5NWI-kvvfFb kv1v1f-vaYx04-kFfkkV x88J88-jBN0Jj-8Yc8Kx ySM3yy-gnhe17-XyUSyp 8xYD8D-npQXH8-KYYcxY Y4Y84x-MrJeFq-cDD488 nnnNgQ-VNO3JO-igNnNg 5FFk1k-hWutyR-vwVk5F Q6nLNN-H90bPb-nLginn pyySS2-H7DN4A-pUSyRy SEpEyy-hbxATA-SyyppM YKc8xY-R3feUE-8cc8a8 g69ngL-gWWKti-QinnnH nLgNQn-JVUT8W-HPHQLn D8Yx88-lrJIuu-JD84c4 kv55F5-3erdMp-VkFFwk v5VFf1-UKYbZn-VbVkkk Y48Y8a-RJ2LDC-8c8xcD pS2y2p-ZSaQvC-MSUyyy Sypyy2-K6PSF4-MppXXy K8c8YK-4jx3NW-8Dac48 XyRyyU-qrp4V9-pMRySM J888cc-lxoCeB-4DYJ8c giPgni-OJ09Yi-gngnnQ Y84cxx-3rJwVM-cYY84c ngiQgN-CnQ4gv-6nQngn kkkFVb-zoKBee-vFkFvk 6QgHnn-2VFveI-QnnngH 3yRSyy-D8i0T9-yy2ppM gnLnQQ-Cyt2rM-L6nnLN MUpSyX-xcRVC3-pyyRpS dkvkfk-77tnCm-FFVfFV F1kkFf-MVW9eT-vkvFkd kkkFkk-xYY2UD-kvw15V nngHng-OBmOQL-NQHLQL 8xKD84-DzKFZJ-4xK8cJ Q9ngLn-bjLzmZ-gQnQgQ NPnLgn-58T69t-nQnniH 8GD84D-DuK5jM-xYc84x Vk5vkk-bcQSA0-FkvFkv LLQnQn-cTWB0Y-gNQNnn nLnQnn-mtbSBC-ngQLHL ypUyy3-3rxk2j-p3pSy2 SpyyRM-Cq2rZQ-UyyMEy 55kvkF-p1hUSP-kFkFkb FkFVdF-Clq0WL-kvv1kd gnLngg-3NcLDa-Q9nPPN nnngQQ-QwK7at-9P9gnn cxY8G8-59CrtL-8c488x LLLgnn-VdBYdx-gLnNnQ gHniQg-bmCra9-Lnggni Eyp23X-6lKVUo-MpppyS x88DYD-yXtIEy-8cJ88x 8Y4ccx-kwQSDk-48x8DG LPNnLn-G0fVYk-iQNnng NQLnnN-vDiFBq-nP9gNn Fvbkff-MmzaQA-kvVwVk c88Yc4-YeMfSt-8c48ca kkvdVV-OgMIQX-kFkFkk 48YKYc-HRXkZA-Ya8c88 nnQng6-DRu6oG-gnigin 8YaxDD-5uPjMv-c84xKc gLnQQn-oProA3-gLQgn9 gngQgg-bLqQCo-g6nQn6 8xca8Y-YGIUkZ-8x8Ycx nnn6NQ-oUJKe0-HQLngg 88xxGa-Og85PD-c8Ya8D EMSyp2-2fQqfN-yyESUp YY84c8-RR4n4d-8c8xJx k5FV1v-975h5I-FVFkkv LPNgQn-3q0i7b-6ngHNn nggnn9-4PJZ5K-gLQHng QQgPnn-vVzh0f-gnNnLg ySyyR2-K40qf2-pXyMpp kkVFkF-2fCbID-vkdFFV x8Y8K8-t24oZW-acYx8c p3SXEp-zAnOdf-yMSyXp 6nQgLg-796wEi-9innHL yyppMX-i88khM-pS3SSU kkFk5k-pQFm92-kvvkFv F1vFvd-BqFbRu-FkfkFk XMpySy-Pdzy6z-yMEM2p 8x8x8K-82daju-xcD8Yx c8Jc8Y-GU8JRy-8a88cY D8J8Y8-SJKX5n-84YGc8 Fvk5dk-K6auXt-5VVV5F gPgnnL-VbQ3lG-Q9nnQg FFvfkk-wDPW9d-FkfdFb ax8D88-0tZR3G-DcGYcc FFbkfk-Ahqclr-kkFvFV pypU3R-RHKaOf-3yMySy 88cY88-eFXdZr-xxYxcc K8xcxY-o5yMOo-c84YYD yM2pU2-QDc6cT-yyp2SU gnLQLg-kDTow4-LLnngn 8DD8c8-OxZqL1-cYJacK nnLPQg-EHCVvo-NQgnnN 4K8cx8-YqtVwW-cY88DY FFdbkk-1iiZ6G-F5kv5k kfFvFv-8pxrcr-Fkkkkk 6ggnnL-3n7MKb-NnnnHQ 4YD8c8-udiY9H-K8c88c 9HPnQn-T5VVou-NgHnnn c8axx8-qVDXly-cxYcYD nLnLgN-L3ErQj-Q9Hgng 66n6Ln-irXTVx-LngnQg a8YD88-7wjh7o-8ccca8 nggngg-6QpEdq-nQiHLN 8Y8aD8-PxTvBD-c88xJ8 ypSMpM-ywy4eI-yy2pyS 9Qgn6N-riIe1q-nnngig 8DGcx8-lqWmWH-acDcY4 F5V5kV-F54puu-VvkdVv VkFkvk-Hl7mGl-fkfk5k fVfkdv-RZJBcC-vFkVFb kvFFdF-UpHnMV-5k5kkk gnnN9i-E039Jw-nLgggQ Ff5vkk-1HlTBb-F5kkkF HgLgHg-4OjrcS-nQnnnL SpypEp-1S1DBJ-y3M2yS nnHLnL-oJk1fq-QiLgLg
48. Hierarchical clustering 0.6 0.7 0.8 ↑ Height
49. Weekly deck trend analysis 適当な高さで切るとデッキアーキテクチャごとのクラスタになる！ 49
50. Pokémon TCG similar deck search 2023 50 card normalize map data/uniq_*.txt new deck metadata known deck sqlite3 sqlite3 My MacBook App deck similarity Search Initialize Crowler Web UI @deck @idf @norm @name @id_norm Make Vector Update Deck Build page periodic task 検索エンジンを流用して
51. Weekly deck trend analysis \ 51 card normalize map data/uniq_*.txt new decks known deck sqlite3 sqlite3 Rakefile deck similarity Initialize Crowler Clustering @deck @idf @norm @name @id_norm Make Vector hierarchical clustering クラスタリング n weeks
52. Weekly deck trend analysis \ 52 card normalize map data/uniq_*.txt new decks known deck sqlite3 sqlite3 Rakefile deck similarity Initialize Crowler Clustering @deck @idf @norm @name @id_norm Make Vector hierarchical clustering クラスタリング n weeks
53. Ractor-ize Using Ractor is easy, but writing for Ractor is hard. Refactoring was very hard Move calculation part to mix-in めちゃめちゃリファクタリングした... 53
54. n週間分のRactorをつくるよ 54 def weekly_analyze(world, deck_and_date) origin = Date.parse('2022-10-07') # ༵ۚ೔࢝·Γ weekly = deck_and_date.sort_by {|x| x[1]}.chunk {|x| (Date.parse(x[1]) - origin).to_i / 7 } result = weekly.map {|_, decks| { 'range' => Range.new(decks[0][1], decks[-1][1]), 'deck_count' => decks.size, 'cluster' => Cluster.new(world, decks.map {|x, d| x}) } } result.each do |x| x['cluster'].join end result end Ractor.new take
55. 55 2022-10-07 .. 2022-10-13 2022-10-14 .. 2022-10-20 2022-10-21 .. 2022-10-27 2022-11-07 .. 2022-11-03 2022-11-04 ... 2022-11-11
56. map { Ractor.new }.map {|r| r.take } Ractor起動！ 56 >> result = weekly.map {|w| Ractor.new(w) {|it| do_cluster(it)}} [ ]
57. map { Ractor.new }.map {|r| r.take } 終わったRactorもいるし、動いてるRactorもいるぞ 57 >> result = weekly.map {|w| Ractor.new(w) {|it| do_cluster(it)}} [ ]
58. map { Ractor.new }.map {|r| r.take } 気にせずtakeで待ち合わせ！ 58 >> result = weekly.map {|w| Ractor.new(w) {|it| do_cluster(it)}} [ ] >> result.map {|r| r.take}
59. map { Ractor.new }.map {|r| r.take } Linda本でもこういうイディオムあったような気がする 59 >> result = weekly.map {|w| Ractor.new(w) {|it| do_cluster(it)}} [ ] >> result.map {|r| r.take} [ ]
60. 28 Ractors, 8 cores (6 performance and 2 efficiency) 40%くらい速くなった。デッキ数が週ごとにばらつくため、多い週に律速した？ 60 w/o Ractor % /usr/bin/time -l rake 51.03 real 50.28 user 0.55 sys Ractor % /usr/bin/time -l rake 31.84 real 83.81 user 89.54 sys
61. Pokémon TCG similar deck search 61 card normalize map data/uniq_*.txt new deck metadata known deck sqlite3 sqlite3 My MacBook App deck similarity Search Initialize Crowler Web UI @deck @idf @norm @name @id_norm Make Vector Update Deck Build page periodic task 検索の高速化 case 2 n decks
62. Pokémon TCG similar deck search 62 card normalize map data/uniq_*.txt new deck metadata known deck sqlite3 sqlite3 My MacBook App deck similarity Search Initialize Crowler Web UI @deck @idf @norm @name @id_norm Make Vector Update Deck Build page periodic task 検索の高速化 n decks
63. Ractor-ize Using Ractor is easy, but writing for Ractor is hard. Refactoring was very hard Add decks while searching I want an appendable shared object 追記可能な共有可能なオブジェクトがほしいです！ 63
64. 変化がない領域だけRactorで検索する 64 if USING_RACTOR top = (_search_by_deck_core(@added_deck, v, n) + search_using_ractor(v, n)).max(n) else top = _search_by_deck_core(@deck, v, n) end search added decks (w/o Ractor) search static decks (Ractor)
65. 大きな配列を作るのを嫌ってイテレータを作った 65 def each_in(range) return enum_for(__method__, range) unless block_given? range.each {|x| yield(@deck[x])} end def _search_by_deck(v, n) ((0..@deck.size).step(@deck.size / @nproc) + [@deck.size]).each_cons(2).map {|s, e| Ractor.new(self, each_in(s...e), v, n) {|world, sub_decks, v, n| world._search_by_deck_core(sub_decks, v, n) } }.map {|r| r.take}.sum([]).max(n) end enum of subrange subrange iterator
66. 10Ractors, 46400 decks 66 0...5811 5911...11622 11622...17433 17433...23244 23244...29055
67. map { Ractor.new }.map {|r| r.take }.sum.max max(n)を集めてmax(n) 67 .map {|r| r.take} e.map {Ractor.new(r) {|it| it.max(n)}} [ ] [ ] .sum([]).max(n)
68. 10Ractors, 46400 decks だいぶ速くなった！ 68 M1 Pro MacBook w/o Ractor 0.272264 sec Ractor 0.070405 sec OCI ARM4 core w/o Ractor 0.369918735 sec Ractor 0.099914693 sec
69. Peak performance issue When tweeting, search slows down Multiple requests generate a lot of Ractor 宣伝ツイートすると検索がめっちゃ遅くなる 69
70. Agenda Concurrency is everywhere Introduction to Ractor case study 70
71. Pokémon Trainer Energy Pokémon TCG Build a deck with 60 cards