[RubyKaigi 2020] The why's and how's of transpiling Ruby

Transpiling is a source-to-source compiling. Why might we need it in Ruby? Compatibility and experiments.

Ruby is evolving fast nowadays. The latest MRI release introduced, for example, the pattern matching syntax. Unfortunately, not everyone is ready to use it yet: gems authors have to support older versions, Ruby implementations are lagging. And it's still experimental, which raises the question: how to evaluate proposals? By backporting them to older Rubies!

I want to discuss these problems and share the story of the Ruby transpiler — Ruby Next. A decent amount of Ruby hackery is guaranteed.

Transcript

1. Vladimir Dementyev
   Evil Martians
   The why’s and how’s
   of transpiling Ruby
   

   View Slide

2. palkan_tula
   palkan RubyKaigi‘20
   Ruby
   2
   

   View Slide

3. palkan_tula
   palkan RubyKaigi‘20
   Transpiler
   3
   Illustration from Ruby Weekly #477
   Source-to-source compiler
   

   View Slide

4. palkan_tula
   palkan RubyKaigi‘20
   The most popular transpiler
   4
   

   View Slide

5. palkan_tula
   palkan RubyKaigi‘20
   Yet another transpiler
   5
   

   View Slide

6. palkan_tula
   palkan RubyKaigi‘20
   6
   Ruby vs. Transpiling
   

   View Slide

7. palkan_tula
   palkan RubyKaigi‘20
   Ruby 3.02.8
   7
   

   View Slide

8. palkan_tula
   palkan RubyKaigi‘20
   Ruby is evolving faster
   than ever
   8
   

   View Slide

9. palkan_tula
   palkan RubyKaigi‘20
   Ruby 2.8
   9
   def greet(val) =
   case val
   in hello: hello if hello =~ /human/i
   ''
   in hello: 'martian'
   ''
   end
   greet(hello: 'martian') => greeting
   puts greeting # =>
   

   View Slide

10. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    10
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Pattern matching (2.7)
    

    View Slide

11. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    11
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Endless method (2.8)
    

    View Slide

12. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    12
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Rightward assignment (2.8)
    

    View Slide

13. palkan_tula
    palkan RubyKaigi‘20
    How to use upcoming
    features today?
    13
    

    View Slide

14. palkan_tula
    palkan RubyKaigi‘20
    Transpiler for Ruby
    14
    

    View Slide

15. palkan_tula
    palkan RubyKaigi‘20
    $ ruby -v

    2.5.3p105

    $ ruby -ruby-next -e "

    case {hello: 'こんにちは'}

    in hello: hello if hello =~ /[\p{Hiragana}]+/

    puts '私はウラジミールです'

    in hello: hello if hello =~ /[a-z]+/i

    puts 'My name is Vladimir'

    end"

    私はウラジミールです
    15
    tl;dr
    

    View Slide

16. palkan_tula
    palkan RubyKaigi‘20
    About me
    16
    github.com/palkan
    

    View Slide

17. palkan_tula
    palkan RubyKaigi‘20
    evilmartians.com
    17
    

    View Slide

18. palkan_tula
    palkan RubyKaigi‘20
    18
    evilmartians.com
    

    View Slide

19. palkan_tula
    palkan RubyKaigi‘20
    evilmartians.com
    19
    Ruby Next
    Transpiler for
    Ruby
    

    View Slide

20. palkan_tula
    palkan RubyKaigi‘20
    evl.ms/blog
    20
    

    View Slide

21. palkan_tula
    palkan RubyKaigi‘20
    Why transpiling Ruby?
    21
    

    View Slide

22. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Why transpiling?
    22
    

    View Slide

23. palkan_tula
    palkan RubyKaigi‘20
    stats.rubygems.org
    23
    

    View Slide

24. palkan_tula
    palkan RubyKaigi‘20
    Gem authors should stick
    to older versions
    24
    

    View Slide

25. palkan_tula
    palkan RubyKaigi‘20
    Story: Hanami::API
    25
    

    View Slide

26. palkan_tula
    palkan RubyKaigi‘20
    26
    Story: Hanami::API
    

    View Slide

27. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    Why transpiling?
    27
    

    View Slide

28. palkan_tula
    palkan RubyKaigi‘20
    JRuby 9.3 — 2.6
    TruffleRuby — 2.6
    mruby ~ 2.7 (no pattern matching)
    Opal, RubyMotion, Artichoke — ???
    Syntax support
    28
    

    View Slide

29. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    ...
    Why transpiling?
    29
    

    View Slide

30. palkan_tula
    palkan RubyKaigi‘20
    How to transpile Ruby?
    30
    

    View Slide

31. palkan_tula
    palkan RubyKaigi‘20
    Transpiler is a source-to-
    source compiler
    31
    

    View Slide

32. palkan_tula
    palkan RubyKaigi‘20
    Compilers
    32
    @pgurtovaya
    

    View Slide

33. palkan_tula
    palkan RubyKaigi‘20
    Parse
    Analyze/optimize
    Generate
    Transpiling
    33
    

    View Slide

34. palkan_tula
    palkan RubyKaigi‘20
    Transpiling
    34
    source code new source code
    AST new AST
    

    View Slide

35. palkan_tula
    palkan RubyKaigi‘20
    Ripper
    RubyVM::AbstractSyntaxTree
    Parser
    Source to AST
    35
    

    View Slide

36. palkan_tula
    palkan RubyKaigi‘20
    Written in pure Ruby
    Version-independent
    Bullet-proofed (e.g., by
    RuboCop)
    Parser
    36
    

    View Slide

37. palkan_tula
    palkan RubyKaigi‘20
    github.com/whitequark/parser
    37
    

    View Slide

38. palkan_tula
    palkan RubyKaigi‘20
    def transpile(source)
    rewriters.inject(source) do |src, rewriter|
    buffer = Parser ::Source ::Buffer.new("")
    buffer.source = src
    rewriter.new.rewrite(buffer, parse(src))
    end
    end
    38
    

    View Slide

39. palkan_tula
    palkan RubyKaigi‘20
    module Rewriters
    class ArgsForward < Base
    def on_forward_args(node)
    replace(node.loc.expression, "(* __rest __, & __block __")
    end
    def on_send(node)
    return super(node) unless node.children[2]&.type == :forwarded_args
    replace(node.children[2].loc.expression, "* __rest __, & __block __")
    end
    end
    end
    39
    Rewriter
    

    View Slide

40. palkan_tula
    palkan RubyKaigi‘20
    Ruby Next Transpiling
    40
    source code new source code
    AST in-place rewrite
    unparse bits of AST
    

    View Slide

41. palkan_tula
    palkan RubyKaigi‘20
    41
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    Example
    

    View Slide

42. palkan_tula
    palkan RubyKaigi‘20
    42
    def fizzbuzz(num)
    rems = [num % 3, num % 5]
    if rems == [0, 0]
    'FizzBuzz'
    elsif rems[0] == 0
    'Fizz'
    elsif rems[1] == 1
    'Buzz'
    else
    raise "NoMatchingPattern"
    end
    end
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    2.7 2.5
    "Transpile"
    by hand
    

    View Slide

43. palkan_tula
    palkan RubyKaigi‘20
    43
    def fizzbuzz(num)
    case; when ( __m __ = [(num % 3), (num % 5)]) && false
    when ( __p_1 __ = ( __m __.respond_to?(:deconstruct) &&
    ((( __m_arr __ = __m __.deconstruct) || true) && ((Array
    === __m_arr __) || Kernel.raise(TypeError, "#deconstruct
    must return Array"))))) && (( __p_2 __ = (2 ==
    __m_arr __.size)) && ((0 === __m_arr __[0]) && (0 ===
    __m_arr __[1]))) then 'FizzBuzz'
    when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[0]))
    then 'Fizz'
    when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[1]))
    then 'Buzz'; else; Kernel.raise(NoMatchingPatternError,
    __m __.inspect)
    end
    end
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    For humans (2.7) For machines (2.5)
    Transpile with
    Ruby Next
    

    View Slide

44. palkan_tula
    palkan RubyKaigi‘20
    Parse
    Analyze/optimize
    Generate
    Transpiling
    44
    

    View Slide

45. palkan_tula
    palkan RubyKaigi‘20
    module Rewriters
    class PatternMatching < Base
    def on_case_match(node)
    # ~800 LOC
    end
    end
    end
    45
    Pattern matching
    

    View Slide

46. palkan_tula
    palkan RubyKaigi‘20
    def call(val)
    status, headers, body = 200, {}, ""
    case val
    in [String => body]
    [status, headers, [body]]
    in [Integer => status]
    [status, headers, [body]]
    in [Integer, String] => response
    [response[0], headers, [response[1]]]
    in [Integer, Hash, String] => response
    headers.merge!(response[1])
    [response[0], headers, [response[2]]]
    end
    end
    46
    Comparison:
    transpiled (last pattern): 1162200.7 i/s
    baseline (last pattern): 799739.5 i/s - 1.45x slower
    Transpiled != Slow
    

    View Slide

47. palkan_tula
    palkan RubyKaigi‘20
    47
    

    View Slide

48. palkan_tula
    palkan RubyKaigi‘20
    How to integrate a
    transpiler into an
    interpreted language?
    48
    

    View Slide

49. palkan_tula
    palkan RubyKaigi‘20
    Gems: transpile at “build”/
    release time
    Apps/Scripts: transpile at
    runtime
    49
    

    View Slide

50. palkan_tula
    palkan RubyKaigi‘20
    $LOAD_PATH: where to search
    for “features”
    $LOADED_FEATURES: required
    files
    Two pillars of require
    50
    

    View Slide

51. palkan_tula
    palkan RubyKaigi‘20
    Add transpiled files to releases
    No additional runtime deps*
    Nextify gems
    51
    * polyfills might still be required
    

    View Slide

52. palkan_tula
    palkan RubyKaigi‘20
    $ ruby-next nextify ./lib -V

    Generated: ./lib/.rbnext/2.7/rubanok/rule.rb

    Generated: ./lib/.rbnext/2.8/rubanok/dsl/matching.rb
    52
    

    View Slide

53. palkan_tula
    palkan RubyKaigi‘20
    # lib/my_gem.rb
    require "ruby-next/language/setup"
    RubyNext ::Language.setup_gem_load_path
    53
    

    View Slide

54. palkan_tula
    palkan RubyKaigi‘20
    def setup_gem_load_path
    lib_dir = File.dirname(caller_locations(1, 1).first.path)
    current_index = $LOAD_PATH.index(lib_dir)
    next_dir = File.join(lib_dir, ".rbnext")
    if File.exist?(next_dir)
    $LOAD_PATH.insert current_index, next_dir
    current_index += 1
    end
    end
    54
    

    View Slide

55. palkan_tula
    palkan RubyKaigi‘20
    Hijack Kernel#require & co
    Reimplement require
    mechanism
    Runtime
    55
    

    View Slide

56. palkan_tula
    palkan RubyKaigi‘20
    module Kernel
    module_function
    alias_method :require_without_ruby_next, :require
    def require(path)
    RubyNext.require(path)
    rescue => e
    warn "Failed to require ' #{path}': #{e.message}"
    require_without_ruby_next(path)
    end
    end
    56
    Kernel#require
    *Don't try this at home
    

    View Slide

57. palkan_tula
    palkan RubyKaigi‘20
    module RubyNext
    module_function
    def require(path)
    realpath = resolve_feature_path(path)
    return false if $LOADED_FEATURES.include?(realpath)
    File.read(realpath)
    .then(&Language.:transform).then do |source|
    RubyVM ::InstructionSequence.compile(code, filepath).then(&:eval)
    $LOADED_FEATURES << path
    true
    end
    end
    end
    57
    RubyNext.require
    

    View Slide

58. palkan_tula
    palkan RubyKaigi‘20
    How to use a transpiler
    with compilable Rubies?
    58
    

    View Slide

59. palkan_tula
    palkan RubyKaigi‘20
    Transpile source files before
    compiling
    Replace target files with the
    transpiled versions at build time
    mruby
    59
    

    View Slide

60. palkan_tula
    palkan RubyKaigi‘20
    desc "transpile source code with ruby-next"
    task rbnext: [] do
    Dir.chdir(APP_ROOT) do
    sh "ruby-next nextify -V"
    end
    end
    Rake ::Task["compile"].enhance [:rbnext]
    60
    Nextify before compiling
    

    View Slide

61. palkan_tula
    palkan RubyKaigi‘20
    using(Module.new do
    refine MRuby ::Gem ::Specification do
    def setup_ruby_next!(next_dir: ".rbnext")
    lib_root = File.join(@dir, @mrblib_dir)
    next_root = Pathname.new(next_dir).absolute? ? next_dir : File.join(lib_root, next_dir)
    Dir.glob(" #{next_root}/**/*.rb").each do |next_file|
    orig_file = next_file.sub(next_root, lib_root)
    index = @rbfiles.index(orig_file) || raise "Source file not found for: #{next_file}"
    @rbfiles[index] = next_file
    end
    end
    end
    end)
    MRuby ::Gem ::Specification.new("acli") do |spec|
    # ...
    spec.setup_ruby_next!
    end
    61
    Update rbfiles
    

    View Slide

62. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    Evolution
    Why transpiling?
    62
    

    View Slide

63. palkan_tula
    palkan RubyKaigi‘20
    Recent Ruby versions added
    experimental features
    63
    

    View Slide

64. palkan_tula
    palkan RubyKaigi‘20
    We need an experiment
    64
    

    View Slide

65. palkan_tula
    palkan RubyKaigi‘20
    The best way to taste new
    features is to start using
    them every day
    65
    

    View Slide

66. palkan_tula
    palkan RubyKaigi‘20
    66
    Example: Hash shorthand
    

    View Slide

67. palkan_tula
    palkan RubyKaigi‘20
    67
    Hash shorthand
    

    View Slide

68. palkan_tula
    palkan RubyKaigi‘20
    $ ruby -v

    2.5.3p105

    $ ruby -ruby-next -e "

    event = "RubyKaigi"
    year = 2020
    p {event, year}

    "

    {event: "RubyKaigi", year: 2020}
    68
    Ruby Next 0.10.0
    

    View Slide

69. palkan_tula
    palkan RubyKaigi‘20
    Share your opinion,
    let's make future Ruby
    together!
    69
    

    View Slide

70. palkan_tula
    palkan RubyKaigi‘20
    github.com/ruby-next/ruby-next
    70
    

    View Slide

71. Thank you!
    evilmartians.com
    github.com/ruby-next
    evilmartians
    palkan_tula
    Vladimir Dementyev
    

    View Slide