YARP

Building good Ruby tooling is dependent on having a good parser. Today, it is difficult to use the CRuby parser to build tooling because of its lack of public interface and documentation. This has led to people using external gems and generally fragmenting the community.

The Yet Another Ruby Parser project is building a universal Ruby parser that can be used by all Ruby implementations and tools. It is documented, error tolerant, and performant. It can be used without linking against CRuby, which means it can be easily used by other projects.

This talk is about the YARP project's motivation, design, implementation, and results. Come to learn about the future of parsing Ruby.

Transcript

1. Yet Another Ruby Parser
2. Kevin Newton @kddnewton
3. Yet Another Ruby Parser
4. Yet Another Ruby Parser
5. Yet Another Ruby Parser Can parse 100% of:
6. Yet Another Ruby Parser Can parse 100% of: Shopify/shopify
7. Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github
8. Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github ruby/ruby
9. Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github ruby/ruby The top 100 gems by download on rubygems.org
10. Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github ruby/ruby The top 100 gems by download on rubygems.org Experimental fork of CRuby
11. Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github ruby/ruby The top 100 gems by download on rubygems.org Experimental fork of CRuby Experimental feature in TruffleRuby
12. Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github ruby/ruby The top 100 gems by download on rubygems.org Experimental fork of CRuby Experimental feature in TruffleRuby Experimental branch in JRuby
13. Yet Another Ruby Parser
14. Yet Another Ruby Parser Motivations
15. Error tolerance Yet Another Ruby Parser Motivations
16. Error tolerance Portability Yet Another Ruby Parser Motivations
17. Error tolerance Portability Maintainability Yet Another Ruby Parser Motivations
18. Error tolerance Portability Maintainability Yet Another Ruby Parser Motivations Development
19. Error tolerance Portability Maintainability Challenges Yet Another Ruby Parser Motivations Development
20. Error tolerance Portability Maintainability Challenges Yet Another Ruby Parser Motivations Development Path forward
21. Error tolerance Portability Maintainability Challenges Adoption Yet Another Ruby Parser Motivations Development Path forward
22. Error tolerance Portability Maintainability Challenges Adoption Future work Yet Another Ruby Parser Motivations Development Path forward
23. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
24. Error tolerance Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
25. Error tolerance Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
26. Error tolerance $ irb irb(main):001:0> foo + bar + <> baz .../irb/workspace.rb:119:in `eval': (irb):1: syntax error, unexpected '<' (SyntaxError) foo + bar + <> baz ^ from .../exe/irb:11:in `<top (required)>' from .../bin/irb:25:in `load' from .../bin/irb:25:in `<main>' irb(main):002:0> Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
27. Error tolerance Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
28. Error tolerance Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
29. Automatically insert tokens Error tolerance Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
30. Automatically insert tokens Automatically insert nodes Error tolerance Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
31. Error tolerance Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
32. Automatically insert tokens Automatically insert nodes Context-based recovery Error tolerance Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
33. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end
34. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode
35. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode
36. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode
37. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode
38. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode
39. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode
40. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode
41. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode ArgumentsNode
42. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode
43. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode
44. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode CLASS
45. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode CLASS CLASS,DEF
46. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode CLASS CLASS,DEF CLASS,DEF,BLOCK
47. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode CLASS CLASS,DEF CLASS,DEF,BLOCK
48. Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode MissingNode ArgumentsNode ArgumentsNode CLASS CLASS,DEF CLASS,DEF,BLOCK
49. Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
50. Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
51. Portability CRuby Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
52. Portability parse.y CRuby Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
53. Portability parse.y ripper/parse.y CRuby Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
54. Portability parse.y ripper/parse.y CRuby Runtimes Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
55. Portability parse.y ripper/parse.y mruby CRuby Runtimes Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
56. Portability parse.y ripper/parse.y mruby JRuby CRuby Runtimes Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
57. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby CRuby Runtimes Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
58. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby CRuby Runtimes Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
59. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie CRuby Runtimes Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
60. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
61. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie parser CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
62. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie parser ruby_parser CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
63. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie parser ruby_parser tree-sitter-ruby CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
64. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie parser ruby_parser tree-sitter-ruby sorbet CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
65. Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie lib-ruby-parser parser ruby_parser tree-sitter-ruby sorbet CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
66. Portability kddnewton.com/parsing-ruby/ parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie lib-ruby-parser parser ruby_parser tree-sitter-ruby sorbet CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
67. Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
68. Portability syntax_tree Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
69. Portability syntax_tree rubyfmt Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
70. Portability syntax_tree rufo rubyfmt Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
71. Portability syntax_tree rufo rubyfmt syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
72. Portability syntax_tree rufo rubyfmt ruby-lsp syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
73. Portability syntax_tree rufo rubyfmt ruby-lsp rubocop syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
74. Portability syntax_tree rufo rubyfmt ruby-lsp rubocop standard syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
75. Portability syntax_tree rufo rubyfmt ruby-lsp unparser rubocop standard syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
76. Portability syntax_tree rufo rubyfmt ruby-lsp unparser ruby-next rubocop standard syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
77. Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next rubocop standard syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
78. Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
79. Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next debride rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
80. Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next flay debride rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
81. Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next flog flay debride rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
82. Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next flog flay debride fasterer rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
83. Portability syntax_tree rufo rubyfmt ruby-lsp sorbet solargraph unparser ruby-next flog flay debride fasterer rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
84. Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
85. No reliance on CRuby internals (VALUE, ID, etc.) Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
86. No reliance on CRuby internals (VALUE, ID, etc.) No reliance on any external packages or generators Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
87. No reliance on CRuby internals (VALUE, ID, etc.) No reliance on any external packages or generators Shared logic: unescaping strings, pack API, etc. Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
88. No reliance on CRuby internals (VALUE, ID, etc.) No reliance on any external packages or generators Shared logic: unescaping strings, pack API, etc. Serialization API Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
89. #include "yarp.h" #include "org_yarp_Parser.h" JNIEXPORT jbyteArray JNICALL Java_org_yarp_Parser_parseAndSerialize( JNIEnv *env, jclass clazz, jbyteArray source) { jsize size = (*env)->GetArrayLength(env, source); jbyte* bytes = (*env)->GetByteArrayElements(env, source, NULL); yp_buffer_t buffer; yp_buffer_init(&buffer); yp_parse_serialize((char*) bytes, size, &buffer); (*env)->ReleaseByteArrayElements(env, source, bytes, JNI_ABORT); jbyteArray serialized = (*env)->NewByteArray(env, buffer.length); jbyte *value = buffer.value; (*env)->SetByteArrayRegion(env, serialized, 0, buffer.length, value); yp_buffer_free(&buffer); return serialized; } Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
90. #include "yarp.h" #include "org_yarp_Parser.h" JNIEXPORT jbyteArray JNICALL Java_org_yarp_Parser_parseAndSerialize( JNIEnv *env, jclass clazz, jbyteArray source) { jsize size = (*env)->GetArrayLength(env, source); jbyte* bytes = (*env)->GetByteArrayElements(env, source, NULL); yp_buffer_t buffer; yp_buffer_init(&buffer); yp_parse_serialize((char*) bytes, size, &buffer); (*env)->ReleaseByteArrayElements(env, source, bytes, JNI_ABORT); jbyteArray serialized = (*env)->NewByteArray(env, buffer.length); jbyte *value = buffer.value; (*env)->SetByteArrayRegion(env, serialized, 0, buffer.length, value); yp_buffer_free(&buffer); return serialized; } Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
91. Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
92. Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
93. Readability Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
94. Readability Understandability Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
95. Readability Understandability Ease of contribution Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
96. Readability Understandability Ease of contribution Ease of change Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
97. Readability Understandability Ease of contribution Ease of change Documentation Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
98. Readability Understandability Ease of contribution Ease of change Documentation Test suites Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
99. Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
100. Is YARP more maintainable? Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
101. Is YARP more maintainable? Can we make YARP more maintainable? Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
102. Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
103. Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
104. Grammar ambiguities Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
105. Grammar ambiguities Unary/binary operators, spaces Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
106. Grammar ambiguities Unary/binary operators, spaces The `do` keyword Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
107. Grammar ambiguities Unary/binary operators, spaces The `do` keyword Newlines, comments, semicolons Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
108. Grammar ambiguities Unary/binary operators, spaces The `do` keyword Newlines, comments, semicolons Local variables and method calls Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
109. Grammar ambiguities Unary/binary operators, spaces The `do` keyword Newlines, comments, semicolons Local variables and method calls Encoding Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
110. Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
111. Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
112. Gem release Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
113. Gem release CRuby fork/compilation work Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
114. Gem release CRuby fork/compilation work JRuby exploring Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
115. Gem release CRuby fork/compilation work JRuby exploring TruffleRuby merged, actively working Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
116. Gem release CRuby fork/compilation work JRuby exploring TruffleRuby merged, actively working Other runtimes/tools Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
117. Gem release CRuby fork/compilation work JRuby exploring TruffleRuby merged, actively working Other runtimes/tools Compatibility with ripper Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
118. Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
119. Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
120. Error tolerance - forward scanning Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
121. class Foo def bar(baz) baz. end end Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
122. class Foo def bar(baz) baz. end end baz.end Missing `end` keyword Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
123. class Foo def bar(baz) baz. end end baz.end baz. Missing `end` keyword Missing call message Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
124. Error tolerance - scanning forward Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
125. Error tolerance - scanning forward Memory Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
126. Error tolerance - scanning forward Memory Arena allocation, improve locality Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
127. Error tolerance - scanning forward Memory Arena allocation, improve locality Reduce size of the tree Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
128. Error tolerance - scanning forward Memory Arena allocation, improve locality Reduce size of the tree More nodes, more specialization Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
129. Error tolerance - scanning forward Memory Arena allocation, improve locality Reduce size of the tree More nodes, more specialization Serialization - reduce size, variable length integers? Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work
130. Thank you!
131. Yet Another Ruby Parser github.com/Shopify/yarp Kevin Newton @kddnewton