Parsing Ruby (RubyKaigi)

Since Ruby's inception, there have been many different projects that parse Ruby code. This includes everything from development tools to Ruby implementations themselves. This talk dives into the technical details and tradeoffs of how each of these tools parses and subsequently understands your applications. After, we'll discuss how you can do the same with your own projects using the Ripper standard library. You'll see just how far we can take this library toward building useful development tools.

Transcript

1. Parsing Ruby Kevin Newton https://kddnewton.com/parsing-ruby
2. The Early Days
3. twitter.com/kddnewton Parsing Ruby Ruby 0.06 1994-01-07 Fri Jan 7 15:23:20 1994 Yukihiro Matsumoto (matz at nws119) * baseline - version 0.06.
4. twitter.com/kddnewton Parsing Ruby Ruby 0.76 1995-05-19 Thu Jul 14 11:18:07 1994 Yukihiro Matsumoto (matz@ix-02) * parse.y: Dictを⽣成する構⽂を追加. こちらを{..}にした. * parse.y: 配列を⽣成する構⽂を[..]に変更した. 過去のRubyスクリプ トとの互換性が保てないが, Dictを⽣成する構⽂を導⼊するに当たり, perl5に合わせて(意識して), 変更する時期は今しかないと考えた. *BACKWARD INCOMPATIBILITY*
5. twitter.com/kddnewton Parsing Ruby Ruby 0.95 1995-12-21 Thu Nov 9 23:26:01 1995 Yukihiro Matsumoto <matz@caelum.co.jp> * parse.y (f_arglist): メソッド定義の引数を括弧で括らなくても良い ようにした． Mon Aug 7 12:47:41 1995 Yukihiro Matsumoto <matz@caelum.co.jp> * parse.y: resque -> rescue．恥ずかしいがtypoを残しておくわけには いかないよなあ．なんで今まで気がつかなかったのか…．
6. Ruby 1.x
7. twitter.com/kddnewton Parsing Ruby Ruby 1.0.961225 1996-12-25 Wed May 22 19:48:42 1996 Yukihiro Matsumoto <matz@caelum.co.jp> * parse.y (superclass): スーパークラスの指定⼦を`:'から`<'に変更． Wed Mar 27 10:02:44 1996 Yukihiro Matsumoto <matz@caelum.co.jp> * parse.y: 予約語の変更 continue -> next
8. twitter.com/kddnewton Parsing Ruby Ruby 1.0.971225 1997-12-25 Mon Apr 7 11:36:16 1997 Yukihiro Matsumoto <matz@caelum.co.jp> * parse.y (primary): syntax to access singleton class. Thu Apr 3 02:12:31 1997 Yukihiro Matsumoto <matz@caelum.co.jp> * parse.y (parse_regx): new option //[nes] to specify character code for regexp literals. Last speci fi ed code option is valid.
9. twitter.com/kddnewton Parsing Ruby Ruby 1.3.0 1998-12-24 • begin..rescue..else..end clauses • <<- indentable heredocs
 • :: method calls
10. twitter.com/kddnewton Parsing Ruby Ruby 1.2.0 1998-12-25 • heredocs
 • =begin to =end
 • true and false
 • BEGIN and END
 • %w
 • Top-level constant access
 • ||= and &&=
11. twitter.com/kddnewton Parsing Ruby Ruby 1.4.0 1999-08-13 • binary number literals
 • anonymous * in method de fi nitions
 • nested string interpolation
 • multibyte character identi fi ers
12. twitter.com/kddnewton Parsing Ruby Ruby 1.5.0 1999-12-07 • Compile-time string concatenation
13. twitter.com/kddnewton Parsing Ruby Ruby 1.6.0 2000-09-19 • rescue modi fi er
14. twitter.com/kddnewton Parsing Ruby nodeDump 0.1.0 2000-10-01 • C extension
 • Human-readable format
15. twitter.com/kddnewton Parsing Ruby ruth 0.0.1 2001-01-10 • C extension
 • Ruby under the hood
 • Ruby::Interpreter.parse
16. twitter.com/kddnewton Parsing Ruby Ruby 1.7.1 2001-06-01 • break and next now accept values
 • %w can escape spaces
 • rescue in singleton method bodies
17. twitter.com/kddnewton Parsing Ruby JRuby 2001-09-10 • Java port of Ruby 1.6
 • Rewrite actions in parse.y into Java
 • Rewrite standard library in Ruby
18. twitter.com/kddnewton Parsing Ruby ripper 0.0.1 2001-10-20 • Rewrite parse.y to dispatch parser events
 • “Ripper is still early-alpha version”
19. twitter.com/kddnewton Parsing Ruby MetaRuby 0.7.0 2002-10-09 • Ruby implemented in Ruby
 • Schema for Ruby AST
20. twitter.com/kddnewton Parsing Ruby Ruby 1.8.0 2003-08-04 • %W word lists
 • Dynamic symbols
 • Nested constant assignment
21. twitter.com/kddnewton Parsing Ruby ParseTree 1.0.0 2004-11-10 • C extension
 • s-expressions from NODE structs
22. twitter.com/kddnewton Parsing Ruby RubyNode 0.1.0 2006-06-05 • C extension
 • Hashes representing NODE struct fi elds
23. twitter.com/kddnewton Parsing Ruby Rubinius 2006-07-12 • sydney
 • Rewrote parse.y in Ruby
 • Rewrote standard library in Ruby
24. twitter.com/kddnewton Parsing Ruby Cardinal 2006-07-16 • Parrot VM
 • From scratch PGE grammar from Ruby EBNF
25. twitter.com/kddnewton Parsing Ruby IronRuby 2007-04-30 • Microsoft .NET port of Ruby
 • Rewrote parse.y, reused parts
26. twitter.com/kddnewton Parsing Ruby ruby_parser 1.0.0 2007-11-14 • Rewrite parse.y in Ruby, use racc
 • dawnscanner, debride, fasterer, fl ay, fl og, railroader, roodi
27. YARV
28. twitter.com/kddnewton Parsing Ruby Ruby 1.9.0 2007-12-25 • Bison
 • YARV
 • Ripper merged
 • Lambda literals
 • Symbol hash keys
29. twitter.com/kddnewton Parsing Ruby MacRuby 2008-03-13 • LLVM port using Objective-C
 • Mostly reuse parse.y
30. twitter.com/kddnewton Parsing Ruby Ruby 1.9.1 2009-01-30 • encoding pragma
 • call shorthand
 • positional arguments after splat
31. twitter.com/kddnewton Parsing Ruby Ruby Intermediate Language 2009-10-26 • Intermediate representation for powering semantic analysis
 • Used to implement type systems in OCaml
 • druby, rtc, rubydust
32. twitter.com/kddnewton Parsing Ruby LASER 0.0.1 2010-08-27 • Undergraduate thesis
 • Linter, type system, documentation generation
 • Used ripper internally
33. twitter.com/kddnewton Parsing Ruby Ruby 1.9.3 • JIS X 3017
 • ISO/IEC 30170:2012 2011-10-31
34. twitter.com/kddnewton Parsing Ruby Topaz • Ruby implementation targeting RPython
 • Rewrote parse.y making grammar rules into decorators 2012-04-07
35. Ruby 2.x
36. twitter.com/kddnewton Parsing Ruby Ruby 2.0.0 • Re fi nements
 • %i symbol lists
 • Keyword arguments 2013-02-24
37. twitter.com/kddnewton Parsing Ruby parser • Published gem, parser API
 • Well-documented
 • covered, deep-cover, erb-lint, fast, opal, packwerk, querly, rdl, reek, rubocop, rubrowser, ruby-lint, ruby-next, ruby_detective, rubycritic, seeing_is_believing, standard, steep, unparser, vernacular, yoda 2013-04-15
38. twitter.com/kddnewton Parsing Ruby Tru ffl eRuby • Originally forked JRuby
 • Graal dynamic compiler, Tru ffl e AST interpreter 2013-10-26
39. twitter.com/kddnewton Parsing Ruby Ruby 2.1.0 • Required keyword arguments
 • Rational and complex literals
 • Frozen string literal su ffi x 2013-12-25
40. twitter.com/kddnewton Parsing Ruby Ruby 2.2.0 • Dynamic symbol hash keys 2014-12-25
41. twitter.com/kddnewton Parsing Ruby Ruby 2.3.0 • Frozen string literal pragma
 • <<~ heredocs
 • &. lonely operator 2015-12-25
42. twitter.com/kddnewton Parsing Ruby Ruby 2.4.0 • Symbol#to_proc re fi nements
 • Top-level return
 • Multiple assignment in conditional 2016-12-25
43. twitter.com/kddnewton Parsing Ruby tree-sitter • Parser-generator library
 • vscode-ruby 2017-02-02
44. twitter.com/kddnewton Parsing Ruby typedruby • Type system in Rust, parser in C++
 • Grammar from Ruby 2.4, lexer from ruby_parser
 • Vendored in Sorbet 2017-02-26
45. twitter.com/kddnewton Parsing Ruby Ruby 2.5.0 • String interpolation re fi nements
 • rescue and ensure at the block level 2017-12-25
46. twitter.com/kddnewton Parsing Ruby Ruby 2.6.0 • RubyVM::AbstractSyntaxTree
 • Flip- fl op deprecated
 • Endless ranges
 • Non-ASCII constant names 2018-12-25
47. twitter.com/kddnewton Parsing Ruby Ruby 2.7.0 • Flip- fl op undeprecated • Method reference operator • Keyword argument warning • No other keywords syntax • Beginless range • Pattern matching • Numbered parameters • Rightward assignment • Argument forwarding 2019-12-25
48. Ruby 3.x
49. twitter.com/kddnewton Parsing Ruby Ruby 3.0.0 • Keyword arguments
 • Single-line “endless” methods
 • “Find pattern” pattern matching
 • shareable_constant_value pragma
 • in keyword pattern matching 2020-12-25
50. twitter.com/kddnewton Parsing Ruby Ruby 3.1.?? • Anonymous struct syntax
51. Lessons
52. twitter.com/kddnewton Parsing Ruby Lessons • Very di ffi cult to keep up with core
 • Small changes in syntax can have wide-reaching e ff ects
 • So much subtle logic in parse.y, very di ff i cult to extend
53. Options
54. twitter.com/kddnewton Parsing Ruby parser • Tons of community adoption
 • Well-documented
 • Not 100% compatible, stu ff may break
 • Doesn’t ship with/test with core
55. twitter.com/kddnewton Parsing Ruby RubyVM::AbstractSyntaxTree • Still too early to tell
 • Not implemented anywhere else
56. twitter.com/kddnewton Parsing Ruby ripper • Built into the parser generator
 • Well-tested in core
 • Ships with Ruby
 • No documentation
57. Language servers
58. twitter.com/kddnewton Parsing Ruby Language servers • castwide/solargraph (parser or RubyVM::AST) • mtsmfm/language_server-ruby (ripper)
 • rubyide/vscode-ruby (tree-sitter) • tomoasleep/yoda (parser)
59. Parsing Ruby Kevin Newton https://kddnewton.com/parsing-ruby