State of Namespace

RubyKaigi 2025 Day 1 talk about Namespace and its deevelopment/debugging

Transcript

1. State of Namespace Satoshi Tagomori (@tagomoris) 2025-04-16 RubyKaigi 2025
2. @tagomoris Satoshi Tagomori Maintainer/Founder: OSS: Fluentd, MessagePack, Norikra, Woothee, …. Event: ISUCON Service: Pathtraq
3. https://www. fl ickr.com/photos/takkanm/3978417669 Asakusa.rb
4. Product Manager, SAKURA Cloud (2024-08-01 ~)
5. SAKURA internet ࣾձΛࢧ͑Δ
   ύϒϦοΫΫϥ΢υɾେن໛ܭࢉࢿݯΠϯϑϥΛ
   Ұॹʹ࡞Γ·ͤΜ͔ʁ
   ソフトウェア開発、 インフラ基盤から フロントエンドまで 採 用 強化中! さくらインターネットではエン ジ ニア採 用 を強化しています さくらインターネットは新たなアイ デ アの創出に強い熱意と情熱を持って挑戦するお客様を は じ め、私たちとつな が りのあるす べ ての 人 たちのために、未来のある べ き姿を想い描きな が ら ―「やりたいこと」を「 で きる」に変える ― あらゆるア プ ローチを “インターネット” を通 じ て提供します。 詳しくはWebサイトにて、カジュアル 面 談もやってます 👉 www.sakura.ad.jp/lp/recruit-engineer/
6. https://sakura-tokyo.connpass.com/event/348223/
7. ✦ Namespace ✦ Timeline ✦ Class de fi nition ✦ Stories between Kaigis ✦ TODO Topics
8. ✦ Separating apps/libs into isolated spaces ✦ Load apps/libs in a space ✦ Hide changes from apps/libs in the space to others ✦ Run methods de fi ned in the space with the de fi nition Namespace
9. All classes/modules are shared in the entire Ruby process Before Namespace: Global Only Ruby Process Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7)
10. Collision: 2 versions of a library cause errors Before Namespace: Global Only Ruby Process Library Code DB::Client (v3) 😵 Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7)
11. Load apps/libs in a space Namespace Ruby Process Namespace Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7)
12. Hide changes by apps/libs in the namespace from others Namespace Ruby Process Namespace Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7) Library Code DB::Client (v3) 😀
13. Run methods de fi ned in the space with the de fi nition Namespace Ruby Process Namespace Application Code App::Func User Library Code DB::Client (v2) Library Code ActiveSupport (v7) Namespace Library Code DB::Client (v3) Namespace Application Code App::Func2 User Library Code ActiveSupport (v6) Application Code call call call
14. https://rubykaigi.org/2024/presentations/tagomoris.html#day1 Check the past Kaigi deck/video for the details!
15. Timeline
16. What did I do after RubyKaigi 2024?
17. DEBUG
18. RubyKaigi 2024 and Then (1) ✦ RubyKaigi 2024 ✦ Demo code ran or SEGV ✦ ‘make test-all’ stopped in the middle (SEGV) ✦ Aug, 2024 ✦ Demo code ran in success! ✦ ‘make test-all’ ran to the end w/ failures/errors
19. Namespace is still WIP, Mostly about just 1 SEGV. Stay tuned til Ruby 3.4! (or …?) Aug, 2024 My Idea at that time 1
20. ✦ Oct, 2024 ✦ ‘make test-all’ passed without failures/errors ✦ ‘make check’ with failures/errors ✦ “I have to give up on merging Namespace for Ruby 3.4” ✦ Jan, 2025 ✦ ‘make check’ passed without failures/errors ✦ ‘make exam’ with failures/errors RubyKaigi 2024 and Then (2)
21. –Matz “Satoshi is too busy recently.”
22. ✦ Apr, 2025 ✦ ‘make exam’ with failures/errors RubyKaigi 2025
23. ✦ Apr, 2025 ✦ ‘make exam’ with failures/errors ✦ Demo code dies with BUG 😱 RubyKaigi 2025
24. Class de fi nition and Namespace
25. ✦ struct RClass ✦ struct ✦ VALUE(Class object) ✦ struct rb_classext_struct (rb_classext_t) ✦ struct ✦ many other class de fi nitions ✦ struct RClass_and_rb_classext_t ✦ RClass + classext Ruby’s Class de fi nition RClass + rb_classext_t RClass rb_classext_t RClass_and_rb_classext_t
26. (Ruby 3.4) Information in RClass + rb_classext_t struct RClass { struct RBasic basic; VALUE super; struct rb_id_table *m_tbl; }; struct rb_classext_struct { VALUE *iv_ptr; struct rb_id_table *const_tbl; struct rb_id_table *callable_m_tbl; struct rb_id_table *cc_tbl; struct rb_id_table *cvc_tbl; size_t superclass_depth; VALUE *superclasses; struct rb_subclass_entry *subclasses; struct rb_subclass_entry *subclass_entry; struct rb_subclass_entry *module_subclass_entry; const VALUE origin_; const VALUE refined_class; union { class; singleton_class; } as; const VALUE includer; attr_index_t max_iv_count; unsigned char variation_count; bool permanent_classpath : 1; bool cloned : 1; VALUE classpath; };
27. (Namespace) Information in RClass + rb_classext_t struct RClass { struct RBasic basic; st_table *ns_classext_tbl; bool prime_classext_readable : 1; bool prime_classext_writable : 1; }; struct rb_classext_struct { const rb_namespace_t *ns; VALUE super; VALUE *iv_ptr; struct rb_id_table *m_tbl; struct rb_id_table *const_tbl; struct rb_id_table *callable_m_tbl; struct rb_id_table *cc_tbl; struct rb_id_table *cvc_tbl; size_t superclass_depth; VALUE *superclasses; struct rb_subclass_anchor *subclasses; rb_ns_subclasses_t *ns_super_subclasses; rb_ns_subclasses_t *ns_module_subclasses; const VALUE origin_; const VALUE refined_class; union { class; singleton_class; } as; const VALUE includer; attr_index_t max_iv_count; unsigned char variation_count; bool permanent_classpath : 1; bool cloned : 1; bool shared_const_tbl : 1; bool iclass_is_origin : 1; bool iclass_origin_shared_mtbl : 1; bool superclasses_owner : 1; bool superclasses_with_self : 1; VALUE classpath; };
28. (Namespace) Information in RClass + rb_classext_t struct RClass { struct RBasic basic; st_table *ns_classext_tbl; bool prime_classext_readable : 1; bool prime_classext_writable : 1; }; struct rb_classext_struct { const rb_namespace_t *ns; VALUE super; VALUE *iv_ptr; struct rb_id_table *m_tbl; struct rb_id_table *const_tbl; struct rb_id_table *callable_m_tbl; struct rb_id_table *cc_tbl; struct rb_id_table *cvc_tbl; size_t superclass_depth; VALUE *superclasses; struct rb_subclass_anchor *subclasses; rb_ns_subclasses_t *ns_super_subclasses; rb_ns_subclasses_t *ns_module_subclasses; const VALUE origin_; const VALUE refined_class; union { class; singleton_class; } as; const VALUE includer; attr_index_t max_iv_count; unsigned char variation_count; bool permanent_classpath : 1; bool cloned : 1; bool shared_const_tbl : 1; bool iclass_is_origin : 1; bool iclass_origin_shared_mtbl : 1; bool superclasses_owner : 1; bool superclasses_with_self : 1; VALUE classpath; }; NEW NEW Moved from RClass Changed data structure Moved from Object flag FL_USER*
29. ✦ Di ff erent Namespace, di ff erent class de fi nition ✦ classext per Namespace by shallow copying the values (at fi rst) Namespaces and classexts RClass ns_classext_tbl rb_classext_t for root/builtin namespace (prime) rb_classext_t for main namespace (non-prime) rb_classext_t for namespace X (non-prime) rb_classext for namespac Y (non-prime
30. ✦ Di ff erent Namespace, di ff erent class de fi nition ✦ classext per Namespace by shallow copying the values (at fi rst) Namespaces and classexts RClass ns_classext_tbl rb_classext_t for root/builtin namespace (prime) rb_classext_t for main namespace (non-prime) rb_classext_t for namespace X (non-prime) rb_classext for namespac Y (non-prime My Idea at that time 2
31. Stories between Kaigis (random order)
32. autoload autoload :Set, ‘set’ Set require ‘set’ Set
33. autoload autoload :Set, ‘set’ Set require ‘set’ Set const_tbl iv_ptr (__autoload__) rb_funcall(‘require’) Nothing to be done especially for Namespace! My Idea at that time 3
34. autoload with Namespaces autoload :Set, ‘set’ Set require ‘set’ Set in Namespace A in Namespace A in Namespace A in Namespace A
35. autoloading in caller ns (1) autoload :Set, ‘set’ in root ns Set in main ns require ‘set’ Set in main ns in main ns autoload entry consumed 1
36. autoloading in caller ns (2) autoload :Set, ‘set’ NameError Set in root ns Set in main ns in root ns in root ns require ‘set’ Set in main ns in main ns 1 2
37. autoloading in callee ns (1) autoload :Set, ‘set’ in root ns Set in main ns autoload entry consumed require ‘set’ Set in root ns in root ns
38. autoloading in callee ns (2) autoload :Set, ‘set’ in root ns Set in main ns NameError in main ns autoload entry consumed require ‘set’ Set in root ns in root ns
39. autoload in the right way autoload :Set, ‘set’ Set require ‘set’ Set Set autoload :Set, ‘set’ Set require ‘set’ in main ns trigger deep copying autoload_table in main ns in main ns in main ns in root ns in root ns in root ns in root ns
40. ✦ Top level constant “Ruby” introduced in Ruby 3.5 (trunk) ✦ Ruby::VERSION == RUBY_VERSION ✦ Tests crashed with SEGV on referring Ruby::VERSION ✦ After deleting and re-setting Ruby::VERSION ✦ Why??? Ruby::VERSION
41. ✦ Copying classext with shallow copying values including const_tbl ✦ keys and values of const_tbl are not copied Shallow copying const_tbl RClass ns_classext_tbl rb_classext_t (prime) const_tbl [A -> va] rb_classext_t (non-prime) const_tbl [A -> va] table shallow copy My Idea at that time 4
42. ✦ Deleting the constant A causes xfree(va) ✦ And va also exists in the const_tbl in the prime classext → SEGV! ✦ ASAN showed the root cause Deleting/Re-setting Constant on non-prime classext RClass ns_classext_tbl rb_classext_t (prime) const_tbl [A -> va] rb_classext_t (non-prime) const_tbl [] https://rubykaigi.org/2024/presentations/KJTsanaktsidis.html
43. ✦ Copying classext with deep copying const_tbl ✦ Values (rb_const_entry_t) are copied deeply to eliminate side-e ff ects ✦ Ruby::VERSION has test cases to delete/add its values, and it identi fi ed the bug deep copying const_tbl RClass ns_classext_tbl rb_classext_t (prime) const_tbl [A -> va] rb_classext_t (non-prime) const_tbl [A -> va’] table deep copy
44. ✦ ‘make exam’ randomly(?) crashed with “Killed: 9” (SIGKILL) ✦ on macOS ✦ without crash reports, without backtrace ✦ On Linux, ‘make exam’ also crashed with SEGV ✦ with broken crash reports, with broken backtrace “Killed: 9”
45. ✦ Identi fi ed the test case fi le: test/erb/test_erb.rb ✦ Wrote many debug prints (100 or more) ✦ required ‘cgi/escape’ — OK ✦ required ‘erb/escape’ — Crashed! ✦ Both are extensions (.so fi les) “Killed: 9” The Silver Bullet - printf 🤪
46. ✦ Extensions (.so) should be loaded in namespaces ✦ But dlopen() opens a fi le just once ✦ Second call on a fi le is just ignored 🥺 Loading Extensions in Namespace (1) util.so Namespace A Namespace B dlopen(“util.so") util_abc dlopen(“util.so")
47. ✦ 💡🤩 Copy the .so fi le with its basename and Namespace ✦ Di ff erent namespaces load di ff erent .so fi les Loading Extensions in Namespace (2) util.so Namespace A Namespace B dlopen(“a_util.so”) util_abc dlopen(“b_util.so”) util_abc My Idea at that time 5
48. ✦ Copy the .so fi le with its basename and Namespace ✦ “cgi/escape” in a → “a_escape.so” ✦ “erb/escape” in a → “a_escape.so” 🥶 ✦ What happens? when: 1. Overwriting the dlopen-ed .so fi le with others 2. Then calling dlopen() the .so fi le ✦ macOS: “Killed: 9” ✦ Linux: Nothing happened — then SEGV randomly Loading Extensions in Namespace (3)
49. ✦ Copy the .so fi le with its full feature name and Namespace ✦ feature name (fname): “cgi/escape”, “erb/escape” Loading Extensions in Namespace: Use Full Path cgi/escape Namespace A dlopen(“a_cgi+escape.so”) cgi_escape erb/escape erb_escape dlopen(“a_erb+escape.so”)
50. ✦ Stacks (push/pop) for managing namespaces ✦ One day: Too many pops on the stack — but why? ✦ rb_ensure() guards push/pop operations ✦ Bug #20655 ✦ Continuation (callcc, cont.call) triggers rb_ensure callback wrongly Bugs of MRI (1) cont.call triggers ensure in C VALUE rb_namespace_exec( const rb_namespace_t *ns, namespace_exec_func *func, VALUE arg) { rb_thread_t *th = GET_THREAD(); namespace_push(th, ns); return rb_ensure(func, arg, namespace_pop, (VALUE)th); } https://bugs.ruby-lang.org/issues/20655
51. ✦ VM_ASSERT(cme->owner == T_CLASS) failed in clearing method caches ✦ cme: callable method entry ✦ method: require, de fi ned_class: Kernel, owner: Kernel ✦ Bug #20767, caused by “Implicit Re fi nements” for namespace implementation ✦ Reproduced by 1. re fi ne Kernel#require (then using it) 2. alias #require to #original_require 3. rede fi ning Kernel#require Bugs of MRI (2) Invalid cme owner with re fi nement
52. ✦ ‘make check’ failed with SystemStackError ✦ Debug prints💡 — prism called itself again and again ✦ Using “—parser=parse.y” solved this problem ✦ But why? ✦ I found a comment by ko1 in bootstraptest/test_ractor.rb ✦ “prism parser with -O0 build consumes a lot of machine stack” ✦ Compiler option “-O3” solved this problem… but why? Bugs? of MRI (3) Don’t use compiler option “-O0”
53. ✦ Controlling method caches/entries ✦ cc_tbl, callable_m_tbl, gccct (global cc cache table) ✦ Rearchitecting subclasses data structure ✦ Identifying Namespaces to load .so/.rb fi les ✦ “Implicit Re fi nements” ✦ https://speakerdeck.com/tagomoris/namespace-now-and-then Other Major Things
54. ✦ Rearchitect current namespace management ✦ From stacks and threads To control frames (or cref) ✦ Namespace GC ✦ Collecting classext for GCed namespaces ✦ Deleting temp .so fi les TODO
55. TODO (cont.) Merging Namespace into Ruby 3.5 … or 4.0?
56. TODO (cont.) My Idea at THIS time Merging Namespace into Ruby 3.5 … or 4.0? See you next time!