Share
Embed
YARP
Building good Ruby tooling is dependent on having a good parser. Today, it is difficult to use the CRuby parser to build tooling because of its lack of public interface and documentation. This has led to people using external gems and generally fragmenting the community.
The Yet Another Ruby Parser project is building a universal Ruby parser that can be used by all Ruby implementations and tools. It is documented, error tolerant, and performant. It can be used without linking against CRuby, which means it can be easily used by other projects.
This talk is about the YARP project's motivation, design, implementation, and results. Come to learn about the future of parsing Ruby.
Kevin Newton
May 12, 2023
Transcript
131 slides
-
Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github
ruby/ruby The top 100 gems by download on rubygems.org -
Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github
ruby/ruby The top 100 gems by download on rubygems.org Experimental fork of CRuby -
Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github
ruby/ruby The top 100 gems by download on rubygems.org Experimental fork of CRuby Experimental feature in TruffleRuby -
Yet Another Ruby Parser Can parse 100% of: Shopify/shopify github/github
ruby/ruby The top 100 gems by download on rubygems.org Experimental fork of CRuby Experimental feature in TruffleRuby Experimental branch in JRuby -
Error tolerance Portability Maintainability Challenges Yet Another Ruby Parser Motivations
Development Path forward -
Error tolerance Portability Maintainability Challenges Adoption Yet Another Ruby Parser
Motivations Development Path forward -
Error tolerance Portability Maintainability Challenges Adoption Future work Yet Another
Ruby Parser Motivations Development Path forward -
Error tolerance Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Error tolerance Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Error tolerance $ irb irb(main):001:0> foo + bar + <>
baz .../irb/workspace.rb:119:in `eval': (irb):1: syntax error, unexpected '<' (SyntaxError) foo + bar + <> baz ^ from .../exe/irb:11:in `<top (required)>' from .../bin/irb:25:in `load' from .../bin/irb:25:in `<main>' irb(main):002:0> Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Error tolerance Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Automatically insert tokens Error tolerance Error tolerance · Portability ·
Maintainability · Challenges · Adoption · Future work -
Automatically insert tokens Automatically insert nodes Error tolerance Error tolerance
· Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Automatically insert tokens Automatically insert nodes Context-based recovery Error tolerance
Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode ArgumentsNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode CLASS -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode CLASS CLASS,DEF -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode CLASS CLASS,DEF CLASS,DEF,BLOCK -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode ArgumentsNode ArgumentsNode CLASS CLASS,DEF CLASS,DEF,BLOCK -
Error tolerance · Portability · Maintainability · Challenges · Adoption
· Future work class Foo def bar(baz) self.then { |value| value + baz * end end ClassNode StatementsNode DefNode CallNode SelfNode BlockNode CallNode CallNode MissingNode ArgumentsNode ArgumentsNode CLASS CLASS,DEF CLASS,DEF,BLOCK -
Portability Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Portability Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Portability CRuby Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Portability parse.y CRuby Error tolerance · Portability · Maintainability ·
Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y CRuby Error tolerance · Portability · Maintainability
· Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y CRuby Runtimes Error tolerance · Portability ·
Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby CRuby Runtimes Error tolerance · Portability
· Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby CRuby Runtimes Error tolerance ·
Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby CRuby Runtimes Error tolerance
· Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby CRuby Runtimes Error
tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie CRuby Runtimes
Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie CRuby Runtimes
Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie parser CRuby
Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie parser ruby_parser
CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie parser ruby_parser
tree-sitter-ruby CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie parser ruby_parser
tree-sitter-ruby sorbet CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie lib-ruby-parser parser
ruby_parser tree-sitter-ruby sorbet CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability kddnewton.com/parsing-ruby/ parse.y ripper/parse.y mruby JRuby TruffleRuby ruruby natalie lib-ruby-parser
parser ruby_parser tree-sitter-ruby sorbet CRuby Runtimes Tools Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Portability syntax_tree Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Portability syntax_tree rubyfmt Error tolerance · Portability · Maintainability ·
Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt Error tolerance · Portability · Maintainability
· Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt syntax_suggest Error tolerance · Portability ·
Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp syntax_suggest Error tolerance · Portability
· Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp rubocop syntax_suggest Error tolerance ·
Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp rubocop standard syntax_suggest Error tolerance
· Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp unparser rubocop standard syntax_suggest Error
tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp unparser ruby-next rubocop standard syntax_suggest
Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next rubocop standard
syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next rubocop standard
steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next debride rubocop
standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next flay debride
rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next flog flay
debride rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp solargraph unparser ruby-next flog flay
debride fasterer rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability syntax_tree rufo rubyfmt ruby-lsp sorbet solargraph unparser ruby-next flog
flay debride fasterer rubocop standard steep syntax_suggest Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Portability Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
No reliance on CRuby internals (VALUE, ID, etc.) Portability Error
tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
No reliance on CRuby internals (VALUE, ID, etc.) No reliance
on any external packages or generators Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
No reliance on CRuby internals (VALUE, ID, etc.) No reliance
on any external packages or generators Shared logic: unescaping strings, pack API, etc. Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
No reliance on CRuby internals (VALUE, ID, etc.) No reliance
on any external packages or generators Shared logic: unescaping strings, pack API, etc. Serialization API Portability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
#include "yarp.h" #include "org_yarp_Parser.h" JNIEXPORT jbyteArray JNICALL Java_org_yarp_Parser_parseAndSerialize( JNIEnv *env,
jclass clazz, jbyteArray source) { jsize size = (*env)->GetArrayLength(env, source); jbyte* bytes = (*env)->GetByteArrayElements(env, source, NULL); yp_buffer_t buffer; yp_buffer_init(&buffer); yp_parse_serialize((char*) bytes, size, &buffer); (*env)->ReleaseByteArrayElements(env, source, bytes, JNI_ABORT); jbyteArray serialized = (*env)->NewByteArray(env, buffer.length); jbyte *value = buffer.value; (*env)->SetByteArrayRegion(env, serialized, 0, buffer.length, value); yp_buffer_free(&buffer); return serialized; } Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
#include "yarp.h" #include "org_yarp_Parser.h" JNIEXPORT jbyteArray JNICALL Java_org_yarp_Parser_parseAndSerialize( JNIEnv *env,
jclass clazz, jbyteArray source) { jsize size = (*env)->GetArrayLength(env, source); jbyte* bytes = (*env)->GetByteArrayElements(env, source, NULL); yp_buffer_t buffer; yp_buffer_init(&buffer); yp_parse_serialize((char*) bytes, size, &buffer); (*env)->ReleaseByteArrayElements(env, source, bytes, JNI_ABORT); jbyteArray serialized = (*env)->NewByteArray(env, buffer.length); jbyte *value = buffer.value; (*env)->SetByteArrayRegion(env, serialized, 0, buffer.length, value); yp_buffer_free(&buffer); return serialized; } Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Maintainability Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Maintainability Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Readability Maintainability Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Readability Understandability Maintainability Error tolerance · Portability · Maintainability ·
Challenges · Adoption · Future work -
Readability Understandability Ease of contribution Maintainability Error tolerance · Portability
· Maintainability · Challenges · Adoption · Future work -
Readability Understandability Ease of contribution Ease of change Maintainability Error
tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Readability Understandability Ease of contribution Ease of change Documentation Maintainability
Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Readability Understandability Ease of contribution Ease of change Documentation Test
suites Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Maintainability Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Is YARP more maintainable? Maintainability Error tolerance · Portability ·
Maintainability · Challenges · Adoption · Future work -
Is YARP more maintainable? Can we make YARP more maintainable?
Maintainability Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Challenges Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Challenges Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Grammar ambiguities Challenges Error tolerance · Portability · Maintainability ·
Challenges · Adoption · Future work -
Grammar ambiguities Unary/binary operators, spaces Challenges Error tolerance · Portability
· Maintainability · Challenges · Adoption · Future work -
Grammar ambiguities Unary/binary operators, spaces The `do` keyword Challenges Error
tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Grammar ambiguities Unary/binary operators, spaces The `do` keyword Newlines, comments,
semicolons Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Grammar ambiguities Unary/binary operators, spaces The `do` keyword Newlines, comments,
semicolons Local variables and method calls Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Grammar ambiguities Unary/binary operators, spaces The `do` keyword Newlines, comments,
semicolons Local variables and method calls Encoding Challenges Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Adoption Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Adoption Error tolerance · Portability · Maintainability · Challenges ·
Adoption · Future work -
Gem release Adoption Error tolerance · Portability · Maintainability ·
Challenges · Adoption · Future work -
Gem release CRuby fork/compilation work Adoption Error tolerance · Portability
· Maintainability · Challenges · Adoption · Future work -
Gem release CRuby fork/compilation work JRuby exploring Adoption Error tolerance
· Portability · Maintainability · Challenges · Adoption · Future work -
Gem release CRuby fork/compilation work JRuby exploring TruffleRuby merged, actively
working Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Gem release CRuby fork/compilation work JRuby exploring TruffleRuby merged, actively
working Other runtimes/tools Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Gem release CRuby fork/compilation work JRuby exploring TruffleRuby merged, actively
working Other runtimes/tools Compatibility with ripper Adoption Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Future work Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Future work Error tolerance · Portability · Maintainability · Challenges
· Adoption · Future work -
Error tolerance - forward scanning Future work Error tolerance ·
Portability · Maintainability · Challenges · Adoption · Future work -
class Foo def bar(baz) baz. end end Error tolerance ·
Portability · Maintainability · Challenges · Adoption · Future work -
class Foo def bar(baz) baz. end end baz.end Missing `end`
keyword Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
class Foo def bar(baz) baz. end end baz.end baz. Missing
`end` keyword Missing call message Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance - scanning forward Future work Error tolerance ·
Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance - scanning forward Memory Future work Error tolerance
· Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance - scanning forward Memory Arena allocation, improve locality
Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance - scanning forward Memory Arena allocation, improve locality
Reduce size of the tree Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance - scanning forward Memory Arena allocation, improve locality
Reduce size of the tree More nodes, more specialization Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work -
Error tolerance - scanning forward Memory Arena allocation, improve locality
Reduce size of the tree More nodes, more specialization Serialization - reduce size, variable length integers? Future work Error tolerance · Portability · Maintainability · Challenges · Adoption · Future work