Embracing Ruby magic

In this talk, we discuss static analysis, DSLs and the intersection between the two. We then get into the new Ruby LSP add-on APIs that allow gems to handle different types of DSL from a static analysis perspective.

Transcript

1. Vinicius Stock Embracing Ruby magic Statically analyzing DSLs
2. Vinicius Stock X / GitHub: @vinistock Bsky: @vinistock.bsky.social Website: https://vinistock.com Sta ff engineer @ Ruby DX
3. Alexander Momchilov From C extension to pure C: Migrating RBS Day 2 16:20 Sub Hall Alexandre Terrasa Inline RBS comments for seamless type checking with Sorbet Day 3 14:00 Sub Hall
4. Apply for jobs at Shopify
5. What ’ s static analysis? A technique to predict what will happen in a program without executing it
6. Tools enabled by static analysis All LSP features • De fi nition • Hover • Completion • Signature help • Type error diagnostics • Code action refactors
7. Tools enabled by static analysis • Dead code detection • Unreachable code detection • Codebase visualizations
8. Why not execute the code? Executing code may have side e ff ects Executing code is slow Editor performance constraints are tight
9. DSLs (domain specific language) A way to extend Ruby itself
10. DSLs class Post belongs_to :user end
11. DSLs post.user post.user= # . . .
12. def belongs_to(entity) class_eval( < < ~RUBY) def # { entity} end def # { entity}=( # { entity}) end RUBY end
13. Invoking belongs_to mutates the class declaration Declaration DSL
14. class Post validate :all_is_well private def all_is_well end end
15. def validate(method_name) @validations < < method_name end
16. def save! self.class.validations.each do |method| send(method) end raise InvalidError if @errors.any? # . . . end
17. The meaning of the symbol is overloaded to represent a method Call site DSL
18. Analyzing DSLs In fl uence of runtime values and control fl ow
19. if some_runtime_value = = 5 define_method(var) do # . . . end end Analyzing DSLs
20. Analyzing DSLs Performance limitations In fl uence of runtime values and control fl ow
21. def create_method(var) define_method(var) do # . . . end end def create_hello create_method("hello") end
22. Analyzing DSLs Value of static analysis drops signi fi cantly with decreases to accuracy or performance
23. Ruby LSP https://shopify.github.io/ruby-lsp/
25. Ruby LSP We couldn’t handle all existing gem DSLs A way for gems to handle their own DSLs for static analysis
26. Ruby LSP add-ons API for other gems to enhance the Ruby LSP’s features Aimed at providing tool and framework speci fi c features
27. module MyGem class Addon < RubyLsp : : Addon def activate(global_state, queue) # Start background process # Handle settings end end end Basic structure
28. module MyGem class Addon < RubyLsp : : Addon def name "MyGem" end end end Basic structure
29. module MyGem class Addon < RubyLsp : : Addon def version "1.1.3" end end end Basic structure
30. module MyGem class Addon < RubyLsp : : Addon def deactivate # S h utdown any background # processes or perform cleanup end end end Basic structure
31. Listener traversal architecture AST traversal is performed by a visitor dispatcher Listeners can subscribe to node events to handle logic
32. class SomeListener def initialize(response_builder, dispatcher) @response_builder = response_builder # Register this listener to handle method defs dispatcher.register(self, :on_def_node_enter) end # Define what happens when we find a method def def on_def_node_enter(node) @response_builder < < node end end
33. # Top level entities: response building + execution # flow response_builder = [] dispatcher = Prism : : Dispatcher.new
34. # Top level entities: response building + execution # flow response_builder = [] dispatcher = Prism : : Dispatcher.new SomeListener.new(response_builder, dispatcher)
35. # Top level entities: response building + execution # flow response_builder = [] dispatcher = Prism : : Dispatcher.new SomeListener.new(response_builder, dispatcher) ast = Prism.parse("def foo; end").value dispatcher.visit(ast)
36. # Top level entities: response building + execution # flow response_builder = [] dispatcher = Prism : : Dispatcher.new SomeListener.new(response_builder, dispatcher) ast = Prism.parse("def foo; end").value dispatcher.visit(ast) puts response_builder # = > [#<Prism : : DefNode name="foo">]
37. Listener traversal architecture Good performance + separate concerns
38. Indexing enhancements The Ruby LSP way for handling declaration DSLs
39. class RailsEnhancement < RubyIndexer : : Enhancement # Found a method call def on_call_node_enter(node) end # Exiting that method call def on_call_node_leave(node) end end
40. # Enhancement API # Register a new method to a specific class/module @listener.add_method(…) # Register a new class or module # Advances stack so that the context will be inside # that new namespace @listener.add_module(…) @listener.add_class(…) # Pops name stack (leaves the current namespace) @listener.pop_namespace_stack
41. class Post belongs_to :user end
42. class RailsEnhancement < RubyIndexer : : Enhancement def on_call_node_enter(node) name = node.name return unless name = = :belongs_to end end
43. class RailsEnhancement < RubyIndexer : : Enhancement def on_call_node_enter(node) name = node.name return unless name = = :belongs_to first_arg = node.arguments&.arguments&.first unless first_arg.is_a?(Prism : : SymbolNode) return end end end
44. class RailsEnhancement < RubyIndexer : : Enhancement def on_call_node_enter(node) # . . . assoc = first_arg.value loc = node.location # def user; end @listener.add_method(assoc, loc, []) end end
45. class RailsEnhancement < RubyIndexer : : Enhancement def on_call_node_enter(node) # . . . sigs = [ Entry : : Signature.new( [Entry : : RequiredParameter.new( name: assoc )] ) ] # def user=(user); end @listener.add_method(" # { assoc}=", loc, sigs) end end
46. class Post # Ruby LSP’s indexer now knows # this is equivalent to: # # def user; end # def user=(user); end belongs_to :user end
47. Handling call site DSLs Handled on a per feature basis
48. module MyGem class Addon < RubyLsp : : Addon # Factory to register listeners def create_definition_listener( response_builder, uri, node_context, dispatcher) # Hook a new listener to the response builder # and dispatcher MyGem : : DefinitionListener.new( . . . ) end end end
49. class Post validate :all_is_well private def all_is_well end end
50. module MyGem class DefinitionListener def initialize(response_builder, index, node_context, dispatcher) @index = index @response_builder = response_builder @node_context = node_context dispatcher.register(self, :on_symbol_node_enter) end end end
51. module MyGem class DefinitionListener def on_symbol_node_enter(node) # Is the method call surrouding this # this symbol a validate call? surrounding_call = @node_context.call_node unless surrounding_call.name = = :validate return end end end end
52. module MyGem class DefinitionListener def on_symbol_node_enter(node) # . . . name = node.value return unless name end end end
53. module MyGem class DefinitionListener def on_symbol_node_enter(node) # . . . methods = @index.resolve_method( name, @node_context.fully_qualified_name ) return unless methods end end end
54. module MyGem class DefinitionListener def on_symbol_node_enter(node) # . . . methods.each do |m| range = range_from_location(m.location) @response_builder < < Location.new( uri: m.uri.to_s, range: range, ) end end end end
55. class Post validate :all_is_well private def all_is_well end end CTRL/CMD + click
56. Ruby LSP RuboCop Standard Reek Brakeman Rake Shoulda Fabrication Action Policy Rspec Rails Rubyfmt Tapioca
57. Ruby LSP RuboCop Standard Reek Brakeman Rake Fabrication Action Policy Rspec Rails Rubyfmt Tapioca Shoulda Model context protocol (MCP)
59. Static analysis in Ruby Ruby is very unique Looking at other gradual typing systems is not enough We need to experiment with new ideas
60. class Post belongs_to :user end Static analysis in Ruby
61. Static analysis in Ruby # : [T] (Symbol entity) - > void : declares { # def $entity: - > T # def $entity=(T $entity) - > void # } def belongs_to(entity) end class Post belongs_to :user # T: User end
62. Static analysis in Ruby # : (MethodSymbol) - > void def validate(method_name) end validate(:does_not_exist) Undefined method `does_not_exist` for User:Class
63. class UsersController < ApplicationController before_action :set_user, only: [:show] def show @user.valid_method end private def set_user @user = User.find(params[:id]) end end `valid_method` doesn’t exist for NilClass
64. class UsersController < ApplicationController before_action :set_user, only: [:show] # : () - > void : called_after set_user def show @user.valid_method end private def set_user @user = User.find(params[:id]) end end
65. Correctness is important, but not at the expense of usefulness Optimize for most common cases with escape hatches Better type checking experience
66. With some innovation in our type annotations, we might be able to account for Ruby’s dynamic nature Better type checking experience
67. Thank you