API for docs

RubyKaigi 2025, Matsuyama, Ehime

1. API for docs Soutaro Matsumoto
2. Soutaro Matsumoto • Ruby committer working for RBS/ Steep • Working at Timee • Started writing a series of articles for ιϑτ΢ΣΞσβΠϯ New
3. Soutaro Matsumoto • Ruby committer working for RBS/ Steep • Working at Timee • Started writing a series of articles for ιϑτ΢ΣΞσβΠϯ New
4. Steep & RBS updates • RBS 3.4 (RubyKaigi 2024) → 3.9 • Steep 1.6 (RubyKaigi 2024) → 1.10
6. Generics with default types rbs-3.6
7. Generics with default types rbs-3.6 steep:ignore annotation steep-1.7
8. Generics with default types rbs-3.6 steep:ignore annotation steep-1.7 Better project organization steep-1.9
9. Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 steep:ignore annotation steep-1.7 Better project organization steep-1.9
10. Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9
11. Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9 Memory footprint improvement steep-1.9 $ steep langserver --refork
12. Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9 Memory footprint improvement steep-1.9 $ steep langserver --refork Receiver type narrowing steep-1.10
13. Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9 Memory footprint improvement steep-1.9 $ steep langserver --refork Receiver type narrowing steep-1.10
14. Generics with default types rbs-3.6 resolve-type-names magic comment rbs-3.9 Deprecation annotation rbs-3.9, steep-1.10 steep:ignore annotation steep-1.7 Better project organization steep-1.9 Memory footprint improvement steep-1.9 $ steep langserver --refork Receiver type narrowing steep-1.10
15. Inline RBS declaration • Not yet... 🙇 • Working for RBS 4.0/Steep 2.0 with the feature • They will support inline RBS declaration directly, without rbs-inline gem
16. Documentation helper in Steep • Steep helps reading docs associated to the Ruby program components -- classes/modules/methods/interfaces/constants • Hover • Completion • Signature Help
17. Hover • Hover shows docs when you point the source code
18. Completion • Docs in the completion list
19. Signature Help • When you are writing the method arguments
20. Writing docs • You can put markdown text associated to RBS constructs
21. This is API • These features are implemented on the top of API that looks up docs with classes/modules/methods RBS fi les Internal data structure API Language Server
22. RDoc/YARD/... • RDoc and YARD generate human readable outputs
23. What is RDoc?
24. What is RDoc? ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end The syntax to annotate Ruby code
25. What is RDoc? ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end The syntax to annotate Ruby code The generated HTML fi les
26. What is RDoc? ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end The syntax to annotate Ruby code The generated HTML fi les The internal structure
27. ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end Input Internal structure Output RBS fi les
28. ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end Input Internal structure Output RBS fi les
29. RBS/Steep Implementation 1. The AST 2. The parser 3. The environment 4. The type checker 5. Displaying the docs RBS RBS RBS Steep Steep
30. The AST • The AST has comments associated to each RBS construct
31. The AST • The AST has comments associated to each RBS construct
32. The AST • The AST has comments associated to each RBS construct
33. The AST • The AST has comments associated to each RBS construct
34. The parser
35. The parser
36. The parser
37. The parser
38. ... The parser
39. ... The parser
40. ... The parser current_token.range
41. ... The parser current_token.range
42. ... The parser current_token.range
43. The environment • Environment stores all of the RBS declarations • Definition is the data structure that has methods and variables of a class/module
44. The environment • Environment stores all of the RBS declarations • Definition is the data structure that has methods and variables of a class/module
45. The environment • Environment stores all of the RBS declarations • Definition is the data structure that has methods and variables of a class/module
46. The environment • Environment stores all of the RBS declarations • Definition is the data structure that has methods and variables of a class/module
47. Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding
48. Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding
49. Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding
50. Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding
51. Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding
52. Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding
53. Type checker • The type checker identi fi es the type of each Ruby node • It also detects which method overload is called by a send node s = String.new() s.encoding
54. Displaying the docs { "method": "document/hover", "id": 1, "params": { "textDocument": { uri: "file://..." }, "position": { "line": 10, "character": 20 } } } Language server { "id": 1, "result": { "contents": "## 📚 String#encoding\n\nReturns the encoding" } } 1. Find the Ruby node associated at the position 2. Fetch the docs for the node 3. Format the docs for LSP
55. Problems • Steep starts type checking when you edit only comments in RBS fi les
56. Problems • Steep starts type checking when you edit only comments in RBS fi les
57. s = String.new() s.encoding
58. s = String.new() s.encoding
59. s = String.new() s.encoding singleton(String) type
60. s = String.new() s.encoding singleton(String) type String class name
61. s = String.new() s.encoding singleton(String) type String class name docs
62. s = String.new() s.encoding singleton(String) type String class name docs
63. s = String.new() s.encoding singleton(String) type String class name docs overload
64. s = String.new() s.encoding singleton(String) type String class name docs overload docs
65. Index • Introducing an index would solve the problem • RBS docs are registered to the index • The language server pulls the docs from the index using identi fi ers • The index may be reused after server restart Index Language server Query the docs by identi fi ers
67. String#encoding
68. String#encoding Array.new
69. Identifying overloads String#encoding Array#[]
70. Identifying overloads String#encoding Array#[]
71. foo = Foo.new foo.foo("hello") foo.foo(123)
72. foo = Foo.new foo.foo("hello") foo.foo(123)
73. foo = Foo.new foo.foo("hello") foo.foo(123)
74. Overload identi fi er • With line number of the overload? • Foo#foo@a.rbs:4 • Foo#foo@b.rbs:4
75. Overload identi fi er • With line number of the overload? • Foo#foo@a.rbs:4 • Foo#foo@b.rbs:4
76. Overload identi fi er • With line number of the overload? • Foo#foo@a.rbs:4 • Foo#foo@b.rbs:4
77. Overload identi fi er • The line number may change after comment update 😫 • Foo#foo@a.rbs:4 • Foo#foo@b.rbs:4
78. Overload identi fi er • The line number may change after comment update 😫 • Foo#foo@a.rbs:4 • Foo#foo@b.rbs:4
79. Overload identi fi er • The line number may change after comment update 😫 • Foo#foo@a.rbs:4 • Foo#foo@b.rbs:4
80. Overload identi fi er • With the index of the overload? • Foo#foo@0 • Foo#foo@1 • Overloads added with ... comes fi rst
81. Overload identi fi er • With the index of the overload? • Foo#foo@0 • Foo#foo@1 • Overloads added with ... comes fi rst
82. Overload identi fi er • With the index of the overload? • Foo#foo@0 • Foo#foo@1 • Overloads added with ... comes fi rst
83. Overload identi fi er • The fi le loading order is unspeci fi ed • Foo#foo@0 • Foo#foo@1 • Foo#foo@2 • (Foo#foo@2 always points to String case)
84. Overload identi fi er • The fi le loading order is unspeci fi ed • Foo#foo@0 • Foo#foo@1 • Foo#foo@2 • (Foo#foo@2 always points to String case)
85. Overload identi fi er • The fi le loading order is unspeci fi ed • Foo#foo@0 • Foo#foo@1 • Foo#foo@2 • (Foo#foo@2 always points to String case)
86. Annotate method names with types • Foo#foo@(String) -> void • Foo#foo@(Integer) -> void
87. JVM method descriptor • JVM method name contains parameter types and return type hello(Ljava/lang/String;I)Ljava/lang/String; String hello(String s, int i)
88. Method name normalization • Let's call it MethodType#normalize • Implementation agnostic • Stable after restarts (String name, id: ::Integer, email: String?) -> void (String, email: String?, id: Integer) -> void
89. Method name normalization • Let's call it MethodType#normalize • Implementation agnostic • Stable after restarts (String name, id: ::Integer, email: String?) -> void (String, email: String?, id: Integer) -> void Drop parameter names
90. Method name normalization • Let's call it MethodType#normalize • Implementation agnostic • Stable after restarts (String name, id: ::Integer, email: String?) -> void (String, email: String?, id: Integer) -> void Drop parameter names Sort keyword args
91. s = String.new() s.encoding Type checker Doc system
92. s = String.new() s.encoding Type checker Doc system
93. s = String.new() s.encoding Type checker Doc system
94. s = String.new() s.encoding singleton(String) type String class name Type checker Doc system
95. s = String.new() s.encoding singleton(String) type String class name Type checker Doc system
96. s = String.new() s.encoding singleton(String) type String class name Type checker Doc system
97. s = String.new() s.encoding singleton(String) type String class name String#encoding@() -> Encoding overload Type checker Doc system
98. s = String.new() s.encoding singleton(String) type String class name String#encoding@() -> Encoding overload Type checker Doc system
99. 🙇 Not fi nished yet
100. Can it be universal? • Making the index from other gems would be great idea • RDoc/YARD makes the index with their docs • Steep/ruby-lsp can use the index for their documentation features ## # Creates a new shape described by a +polyline+. # # If the +polyline+ does not end at the same point it started at the # first pointed is copied and placed at the end of the line. # # An ArgumentError is raised if the line crosses itself, but shapes may # be concave. def initialize polyline # ... end Input Universal index Output RBS fi les
101. Index content • Discussed the key of the index; assume we have a solution • How can we de fi ne the content of the index? • A data structure that can be shared with RBS/Steep, RDoc, YARD, which keeps the contents of all of them? • How about go-to-de fi nition?
102. Summary • The documentation needs APIs because it's used by IDE/editors to help programming • Steep already has implementation, but it is tightly coupled with type checker and causing problems • I plan to implement a doc system API (index) • The index may help other tools too (but I'm not sure if it's a good idea)