Automatically generating types by running tests

2025/04/16 RubyKaigi Day1
https://rubykaigi.org/2025/presentations/sinsoku_listy.html#day1

Transcript

1. Automatically generating types by running tests RubyKaigi 2025 Day1
2. About me • Takumi Shotoku • @sinsokuʢGitHubʣ • @sinsoku_listyʢXʣ • Timee, Inc. and mov inc. • Omotesando.rb, Asakusa-bashi.rbs 2
4. Today's talk 1. Introducing rbs-trace 2. Implementation details 3. Future work 4
6. The API is very simple trace = RBS:"Trace.new # Enable tracing to call methods trace.enable # Call methods user = User.new("Yukihiro", "Matsumoto") user.say_hello # Disable tracing trace.disable # Save RBS type declarations as comments trace.save_comments 6
7. Insert RBS comments to Ruby files class User # @rbs (String, String) -" void def initialize(first_name, last_name) @first_name = first_name @last_name = last_name end # @rbs () -" String def full_name "#%@first_name} #%@last_name}" end # @rbs () -" void def say_hello puts "hi, #%full_name}." end end 7
8. Integration with RSpec Add the following code to spec_helper.rb. if ENV["RBS_TRACE"] RSpec.configure do |config| trace = RBS:#Trace.new config.before(:suite) { trace.enable } config.after(:suite) do trace.disable trace.save_comments end end end 8
9. Integration with Minitest Add the following code to test_helper.rb. if ENV["RBS_TRACE"] trace = RBS:"Trace.new trace.enable Minitest.after_run do trace.disable trace.save_comments end end 9
10. Then, simply run the tests $ RBS_TRACE=1 bin/rails test Automatically inserts RBS comments into Ruby files. 10
11. Questions • ❓ What is type generation from running tests? • ❓ Can it be used practically in Rails apps? • ❓ What about performance? 11
12. Tests and type declarations are aligned • No tests, no type declarations • Not enough tests, not enough type declarations • Tests are wrong, type declarations are wrong 12
13. Only executed methods RSpec.design User do describe "#foo" do it { .." } end end class User # @rbs () -" void def foo end def bar end end 13
14. Only arguments passed in tests RSpec.describe "Calculator" do describe ".sum" do context "when args are Integer" subject { Calculator.sum(1, 2) } it { is_expected.to eq 3 } end end end class Calculator # @rbs (Integer, Integer) -" Integer def self.sum(x, y) x.to_i + y.to_i end end 14
15. Two test cases can be generated RSpec.describe "Calculator" do describe ".sum" do context "when args are Integer" subject { Calculator.sum(1, 2) } it { is_expected.to eq 3 } end context "when args are String" subject { Calculator.sum("1", "2") } it { is_expected.to eq 3 } end end end class Calculator # @rbs (Integer|String, Integer|String) -" Integer def self.sum(x, y) x.to_i + y.to_i end end 15
16. Type declarations are generated as tests RSpec.describe "User" do describe "#initialize" do subject { User.new(nil) } it { is_expected.to be } end end class User # @rbs (nil) -" void def initialize(name) @name = name end end 16
17. Questions • ✅ What is type generation from running tests? • ❓ Can it be used practically in Rails apps? • ❓ What about performance? 17
18. It works in the following two Rails apps RED MINE flexible project management 18
19. Are type declarations correct? Looks good to me. 19
20. See for yourself • Redmine • https://github.com/sinsoku/redmine/pull/2 • Mastodon • https://github.com/sinsoku/mastodon/pull/4 20
21. Questions • ✅ What is type generation from running tests? • ✅ Can it be used practically in Rails apps? • ❓ What about performance? 21
22. Test execution time on local machine1 • Redmine - parallelize(workers: 1) • Simply run bin/rails test • Mastodon - using flatware • Run all tests in parallel 1 MacBook Pro (14-inch, 2021) M1 Max 64GB 22
23. Comparison results before after di! slower Redmine 2m14s 12m 7s 9m53s 5.43x Mastodon 45s 2m54s 2m 9s 3.87x 23
24. Organization's Rails apps jobs before after di! slower Timee, Inc.2 35 7m16s 10m40s 3m24s 1.47x mov inc.3 20 15m52s 22m16s 6m24s 1.40x 3 mov: Actions, RSpec, r7kamura/split-tests-by-timings 2 Timee: Actions, RSpec, https://github.com/mtsmfm/split-test 24
25. rbs-trace only needs to be run once. The performance trade-off is acceptable. 25
26. Questions • ✅ What is type generation from running tests? • ✅ Can it be used practically in Rails apps? • ✅ What about performance? If you have other questions, ask me after this talk! 26
27. Today's talk 1. Introducing rbs-trace 2. Implementation details 3. Future work 27
28. Core design 1. Record class name for arguments and return values 2. Perform this process on all method calls 3. Insert RBS comments above the method definition 28
29. Core design 1. Record class name for arguments and return values 2. Perform this process on all method calls 3. Insert RBS comments above the method definition This can be solved with TracePoint. 29
30. Record argument types with TracePoint class User def initialize(first_name, last_name) p("initialize") end end TracePoint.trace(:call) do |tp| p(defined_class: tp.defined_class, method_id: tp.method_id, parameters: tp.parameters) end User.new("Yukihiro", "Matsumoto") # {defined_class: User, method_id: :initialize, parameters: [[:req, :first_name], [:req, :last_name]]} # "initialize" 30
31. Record argument types with TracePoint class User def initialize(first_name, last_name) p("initialize") end end TracePoint.trace(:call) do |tp| tp.parameters.each do |_type, name| # [[:req, :first_name], [:req, :last_name]] value = tp.binding.local_variable_get(name) p(name:, value:, class_name: value.class) end end User.new("Yukihiro", "Matsumoto") # {name: :first_name, value: "Yukihiro", class: String} # {name: :last_name, value: "Matsumoto", class: String} # "initialize" 31
32. Record the return type with TracePoint class Calculator def self.sum(x, y) p("sum") x + y end end TracePoint.trace(:return) do |tp| p(return_value: tp.return_value, class: tp.return_value.class) end Calculator.sum(1, 2) # "sum" # {return_value: 3, class: Integer} 32
33. It looks like it can be implemented. And I tested it. 33
34. Issues • NoMethodError occurs in the class method • The class name becomes ActiveRecord:"Relation • Does not support void type • Does not work in parallel testing 34
35. The class method missing • Redmine has a class that extends BasicObject • BasicObject does not have a class method class A < BasicObject end obj = A.new p(class: obj.class) # undefined method 'class' for an instance of A (NoMethodError) 35
36. Solved with UnboundMethod class A < BasicObject end obj = A.new unbound_class = Object.instance_method(:class) p(class: unbound_class.bind_call(obj)) # {class: A} 36
37. Issues • ✅ NoMethodError occurs in the class method • The class name becomes ActiveRecord:"Relation • Does not support void type • Does not work in parallel testing 37
38. The name method is often overridden users = User.all users.class.name #"> "ActiveRecord:$Relation" users.class.to_s #"> "User:$ActiveRecord_Relation" 38
39. The name method is often overridden4 class X end X.method(:name) #"> #<Method: #<Class:X>(Module)#name()> users = User.all users.class.method(:name) #"> #<Method: #<Class:User:$ActiveRecord_Relation> # (ActiveRecord:$Delegation:$ClassSpecificRelation:$ClassMethods)#name() 4 https://github.com/rails/rails/blob/v8.0.2/activerecord/lib/active_record/relation/delegation.rb#L112- L114 39
40. Also resolved by UnboundMethod users = User.all unbound_name = Class.instance_method(:name) unbound_name.bind_call(users.class) #"> "User:$ActiveRecord_Relation" 40
41. How to get the correct class name UNBOUND_CLASS = Object.instance_method(:class) UNBOUND_NAME = Class.instance_method(:name) TracePoint.trace(:return) do |tp| klass = UNBOUND_CLASS.bind_call(tp.return_value) class_name = UNBOUND_NAME.bind_call(klass) end 41
42. Issues • ✅ NoMethodError occurs in the class method • ✅ The class name becomes ActiveRecord:"Relation • Does not support void type • Does not work in parallel testing 42
43. What is void type? class UsersController < ApplicationController def new user = User.new user.set_password render :ok end end class User # @rbs () -" String def set_password @password = SecureRandom.hex(20) end end 43
44. What is void type? class UsersController < ApplicationController def new user = User.new user.set_password render :ok end end class User # @rbs () -" void def set_password @password = SecureRandom.hex(20) end end 44
45. Use caller_locations class A def foo bar end def bar nil end end TracePoint.trace(:call) do |tp| p(method_id: tp.method_id, caller_locations:) end A.new.foo # {method_id: :foo, caller_locations: ["example.rb:3:in 'A#foo'", # "example.rb:13:in '<main>'"]} # {method_id: :bar, caller_locations: ["example.rb:6:in 'A#bar'", # "example.rb:3:in 'A#foo'", # "example.rb:13:in '<main>'"]} 45
46. Parse the caller's file with Prism 46
47. How to determine void type The conditions for void type are as follows. 1. Method call is directly under the def keyword 2. But exclude the last method call def example foo bar buz # the return value is used end 47
48. Example of parsing with Prism 48
49. Example of parsing with Prism 49
50. Example of parsing with Prism 50
51. Example of parsing with Prism 51
52. Example of parsing with Prism 52
53. Example of parsing with Prism 53
54. Example of parsing with Prism 54
55. Example of parsing with Prism 55
56. Issues • ✅ NoMethodError occurs in the class method • ✅ The class name becomes ActiveRecord:"Relation • ✅ Does not support void type • Does not work in parallel testing 56
61. Configure for parallel testing RSpec.configure do |config| trace = RBS:#Trace.new config.before(:suite) { trace.enable } config.after(:suite) do trace.disable trace.save_files(out_dir: "tmp/sig-#&ENV.fetch('TEST_ENV_NUMBER', nil)}") end end 61
62. Execute the following commands # Run parallel testings $ bin/flatware rspec -r ./spec/flatware_helper.rb # Merge RBS files into one $ bundle exec rbs-trace merge -$sig-dir='tmp/sig-*' > tmp/sig/merged.rbs # Insert RBS comments from the merged file $ bundle exec rbs-trace inline -$sig-dir=tmp/sig -$rb-dir=app -$rb-dir=lib 62
63. ! RBS files can be generated by CI 1. Add rbs-trace to Gemfile 2. Configure spec_helper.rb 3. Save RBS files on CI6 4. Download RBS files from CI 5. Use merge and inline commands locally 6 https://github.com/actions/upload-artifact ͳͲ 63
64. Issues • ✅ NoMethodError occurs in the class method • ✅ The class name becomes ActiveRecord:"Relation • ✅ Does not support void type • ✅ Does not work in parallel testing 64
65. Today's talk 1. Introducing rbs-trace 2. Implementation details 3. Future work 65
66. Future work 1. Support more types • Block arguments, Generics types, Interface types • ...etc 2. Update RBS comments each time tests are run 3. Save RBS files for gems not in gem_rbs_collection 66
67. Conclusion • It generally works and is practical • TracePoint, Prism, caller_locations are very useful • Read the code and the generated type declarations Let's generate type declarations! 67
68. Appendix https://github.com/sinsoku/redmine/pull/2 https://github.com/sinsoku/mastodon/pull/4 68