$30 off During Our Annual Pro Sale. View Details »

[RubyKaigi 2020] The why's and how's of transpiling Ruby

[RubyKaigi 2020] The why's and how's of transpiling Ruby

Transpiling is a source-to-source compiling. Why might we need it in Ruby? Compatibility and experiments.

Ruby is evolving fast nowadays. The latest MRI release introduced, for example, the pattern matching syntax. Unfortunately, not everyone is ready to use it yet: gems authors have to support older versions, Ruby implementations are lagging. And it's still experimental, which raises the question: how to evaluate proposals? By backporting them to older Rubies!

I want to discuss these problems and share the story of the Ruby transpiler — Ruby Next. A decent amount of Ruby hackery is guaranteed.

Vladimir Dementyev

September 04, 2020
Tweet

More Decks by Vladimir Dementyev

Other Decks in Programming

Transcript

  1. Vladimir Dementyev
    Evil Martians
    The why’s and how’s
    of transpiling Ruby

    View Slide

  2. palkan_tula
    palkan RubyKaigi‘20
    Ruby
    2

    View Slide

  3. palkan_tula
    palkan RubyKaigi‘20
    Transpiler
    3
    Illustration from Ruby Weekly #477
    Source-to-source compiler

    View Slide

  4. palkan_tula
    palkan RubyKaigi‘20
    The most popular transpiler
    4

    View Slide

  5. palkan_tula
    palkan RubyKaigi‘20
    Yet another transpiler
    5

    View Slide

  6. palkan_tula
    palkan RubyKaigi‘20
    6
    Ruby vs. Transpiling

    View Slide

  7. palkan_tula
    palkan RubyKaigi‘20
    Ruby 3.02.8
    7

    View Slide

  8. palkan_tula
    palkan RubyKaigi‘20
    Ruby is evolving faster
    than ever
    8

    View Slide

  9. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    9
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>

    View Slide

  10. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    10
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Pattern matching (2.7)

    View Slide

  11. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    11
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Endless method (2.8)

    View Slide

  12. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    12
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Rightward assignment (2.8)

    View Slide

  13. palkan_tula
    palkan RubyKaigi‘20
    How to use upcoming
    features today?
    13

    View Slide

  14. palkan_tula
    palkan RubyKaigi‘20
    Transpiler for Ruby
    14

    View Slide

  15. palkan_tula
    palkan RubyKaigi‘20
    $ ruby -v

    2.5.3p105

    $ ruby -ruby-next -e "

    case {hello: 'こんにちは'}

    in hello: hello if hello =~ /[\p{Hiragana}]+/

    puts '私はウラジミールです'

    in hello: hello if hello =~ /[a-z]+/i

    puts 'My name is Vladimir'

    end"

    私はウラジミールです
    15
    tl;dr

    View Slide

  16. palkan_tula
    palkan RubyKaigi‘20
    About me
    16
    github.com/palkan

    View Slide

  17. palkan_tula
    palkan RubyKaigi‘20
    evilmartians.com
    17

    View Slide

  18. palkan_tula
    palkan RubyKaigi‘20
    18
    evilmartians.com

    View Slide

  19. palkan_tula
    palkan RubyKaigi‘20
    evilmartians.com
    19
    Ruby Next
    Transpiler for
    Ruby

    View Slide

  20. palkan_tula
    palkan RubyKaigi‘20
    evl.ms/blog
    20

    View Slide

  21. palkan_tula
    palkan RubyKaigi‘20
    Why transpiling Ruby?
    21

    View Slide

  22. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Why transpiling?
    22

    View Slide

  23. palkan_tula
    palkan RubyKaigi‘20
    stats.rubygems.org
    23

    View Slide

  24. palkan_tula
    palkan RubyKaigi‘20
    Gem authors should stick
    to older versions
    24

    View Slide

  25. palkan_tula
    palkan RubyKaigi‘20
    Story: Hanami::API
    25

    View Slide

  26. palkan_tula
    palkan RubyKaigi‘20
    26
    Story: Hanami::API

    View Slide

  27. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    Why transpiling?
    27

    View Slide

  28. palkan_tula
    palkan RubyKaigi‘20
    JRuby 9.3 — 2.6
    TruffleRuby — 2.6
    mruby ~ 2.7 (no pattern matching)
    Opal, RubyMotion, Artichoke — ???
    Syntax support
    28

    View Slide

  29. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    ...
    Why transpiling?
    29

    View Slide

  30. palkan_tula
    palkan RubyKaigi‘20
    How to transpile Ruby?
    30

    View Slide

  31. palkan_tula
    palkan RubyKaigi‘20
    Transpiler is a source-to-
    source compiler
    31

    View Slide

  32. palkan_tula
    palkan RubyKaigi‘20
    Compilers
    32
    @pgurtovaya

    View Slide

  33. palkan_tula
    palkan RubyKaigi‘20
    Parse
    Analyze/optimize
    Generate
    Transpiling
    33

    View Slide

  34. palkan_tula
    palkan RubyKaigi‘20
    Transpiling
    34
    source code new source code
    AST new AST

    View Slide

  35. palkan_tula
    palkan RubyKaigi‘20
    Ripper
    RubyVM::AbstractSyntaxTree
    Parser
    Source to AST
    35

    View Slide

  36. palkan_tula
    palkan RubyKaigi‘20
    Written in pure Ruby
    Version-independent
    Bullet-proofed (e.g., by
    RuboCop)
    Parser
    36

    View Slide

  37. palkan_tula
    palkan RubyKaigi‘20
    github.com/whitequark/parser
    37

    View Slide

  38. palkan_tula
    palkan RubyKaigi‘20
    def transpile(source)
    rewriters.inject(source) do |src, rewriter|
    buffer = Parser ::Source ::Buffer.new("")
    buffer.source = src
    rewriter.new.rewrite(buffer, parse(src))
    end
    end
    38

    View Slide

  39. palkan_tula
    palkan RubyKaigi‘20
    module Rewriters
    class ArgsForward < Base
    def on_forward_args(node)
    replace(node.loc.expression, "(* __rest __, & __block __")
    end
    def on_send(node)
    return super(node) unless node.children[2]&.type == :forwarded_args
    replace(node.children[2].loc.expression, "* __rest __, & __block __")
    end
    end
    end
    39
    Rewriter

    View Slide

  40. palkan_tula
    palkan RubyKaigi‘20
    Ruby Next Transpiling
    40
    source code new source code
    AST in-place rewrite
    unparse bits of AST

    View Slide

  41. palkan_tula
    palkan RubyKaigi‘20
    41
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    Example

    View Slide

  42. palkan_tula
    palkan RubyKaigi‘20
    42
    def fizzbuzz(num)
    rems = [num % 3, num % 5]
    if rems == [0, 0]
    'FizzBuzz'
    elsif rems[0] == 0
    'Fizz'
    elsif rems[1] == 1
    'Buzz'
    else
    raise "NoMatchingPattern"
    end
    end
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    2.7 2.5
    "Transpile"
    by hand

    View Slide

  43. palkan_tula
    palkan RubyKaigi‘20
    43
    def fizzbuzz(num)
    case; when ( __m __ = [(num % 3), (num % 5)]) && false
    when ( __p_1 __ = ( __m __.respond_to?(:deconstruct) &&
    ((( __m_arr __ = __m __.deconstruct) || true) && ((Array
    === __m_arr __) || Kernel.raise(TypeError, "#deconstruct
    must return Array"))))) && (( __p_2 __ = (2 ==
    __m_arr __.size)) && ((0 === __m_arr __[0]) && (0 ===
    __m_arr __[1]))) then 'FizzBuzz'
    when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[0]))
    then 'Fizz'
    when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[1]))
    then 'Buzz'; else; Kernel.raise(NoMatchingPatternError,
    __m __.inspect)
    end
    end
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    For humans (2.7) For machines (2.5)
    Transpile with
    Ruby Next

    View Slide

  44. palkan_tula
    palkan RubyKaigi‘20
    Parse
    Analyze/optimize
    Generate
    Transpiling
    44

    View Slide

  45. palkan_tula
    palkan RubyKaigi‘20
    module Rewriters
    class PatternMatching < Base
    def on_case_match(node)
    # ~800 LOC
    end
    end
    end
    45
    Pattern matching

    View Slide

  46. palkan_tula
    palkan RubyKaigi‘20
    def call(val)
    status, headers, body = 200, {}, ""
    case val
    in [String => body]
    [status, headers, [body]]
    in [Integer => status]
    [status, headers, [body]]
    in [Integer, String] => response
    [response[0], headers, [response[1]]]
    in [Integer, Hash, String] => response
    headers.merge!(response[1])
    [response[0], headers, [response[2]]]
    end
    end
    46
    Comparison:
    transpiled (last pattern): 1162200.7 i/s
    baseline (last pattern): 799739.5 i/s - 1.45x slower
    Transpiled != Slow

    View Slide

  47. palkan_tula
    palkan RubyKaigi‘20
    47

    View Slide

  48. palkan_tula
    palkan RubyKaigi‘20
    How to integrate a
    transpiler into an
    interpreted language?
    48

    View Slide

  49. palkan_tula
    palkan RubyKaigi‘20
    Gems: transpile at “build”/
    release time
    Apps/Scripts: transpile at
    runtime
    49

    View Slide

  50. palkan_tula
    palkan RubyKaigi‘20
    $LOAD_PATH: where to search
    for “features”
    $LOADED_FEATURES: required
    files
    Two pillars of require
    50

    View Slide

  51. palkan_tula
    palkan RubyKaigi‘20
    Add transpiled files to releases
    No additional runtime deps*
    Nextify gems
    51
    * polyfills might still be required

    View Slide

  52. palkan_tula
    palkan RubyKaigi‘20
    $ ruby-next nextify ./lib -V

    Generated: ./lib/.rbnext/2.7/rubanok/rule.rb

    Generated: ./lib/.rbnext/2.8/rubanok/dsl/matching.rb
    52

    View Slide

  53. palkan_tula
    palkan RubyKaigi‘20
    # lib/my_gem.rb
    require "ruby-next/language/setup"
    RubyNext ::Language.setup_gem_load_path
    53

    View Slide

  54. palkan_tula
    palkan RubyKaigi‘20
    def setup_gem_load_path
    lib_dir = File.dirname(caller_locations(1, 1).first.path)
    current_index = $LOAD_PATH.index(lib_dir)
    next_dir = File.join(lib_dir, ".rbnext")
    if File.exist?(next_dir)
    $LOAD_PATH.insert current_index, next_dir
    current_index += 1
    end
    end
    54

    View Slide

  55. palkan_tula
    palkan RubyKaigi‘20
    Hijack Kernel#require & co
    Reimplement require
    mechanism
    Runtime
    55

    View Slide

  56. palkan_tula
    palkan RubyKaigi‘20
    module Kernel
    module_function
    alias_method :require_without_ruby_next, :require
    def require(path)
    RubyNext.require(path)
    rescue => e
    warn "Failed to require ' #{path}': #{e.message}"
    require_without_ruby_next(path)
    end
    end
    56
    Kernel#require
    *Don't try this at home

    View Slide

  57. palkan_tula
    palkan RubyKaigi‘20
    module RubyNext
    module_function
    def require(path)
    realpath = resolve_feature_path(path)
    return false if $LOADED_FEATURES.include?(realpath)
    File.read(realpath)
    .then(&Language.:transform).then do |source|
    RubyVM ::InstructionSequence.compile(code, filepath).then(&:eval)
    $LOADED_FEATURES << path
    true
    end
    end
    end
    57
    RubyNext.require

    View Slide

  58. palkan_tula
    palkan RubyKaigi‘20
    How to use a transpiler
    with compilable Rubies?
    58

    View Slide

  59. palkan_tula
    palkan RubyKaigi‘20
    Transpile source files before
    compiling
    Replace target files with the
    transpiled versions at build time
    mruby
    59

    View Slide

  60. palkan_tula
    palkan RubyKaigi‘20
    desc "transpile source code with ruby-next"
    task rbnext: [] do
    Dir.chdir(APP_ROOT) do
    sh "ruby-next nextify -V"
    end
    end
    Rake ::Task["compile"].enhance [:rbnext]
    60
    Nextify before compiling

    View Slide

  61. palkan_tula
    palkan RubyKaigi‘20
    using(Module.new do
    refine MRuby ::Gem ::Specification do
    def setup_ruby_next!(next_dir: ".rbnext")
    lib_root = File.join(@dir, @mrblib_dir)
    next_root = Pathname.new(next_dir).absolute? ? next_dir : File.join(lib_root, next_dir)
    Dir.glob(" #{next_root}/**/*.rb").each do |next_file|
    orig_file = next_file.sub(next_root, lib_root)
    index = @rbfiles.index(orig_file) || raise "Source file not found for: #{next_file}"
    @rbfiles[index] = next_file
    end
    end
    end
    end)
    MRuby ::Gem ::Specification.new("acli") do |spec|
    # ...
    spec.setup_ruby_next!
    end
    61
    Update rbfiles

    View Slide

  62. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    Evolution
    Why transpiling?
    62

    View Slide

  63. palkan_tula
    palkan RubyKaigi‘20
    Recent Ruby versions added
    experimental features
    63

    View Slide

  64. palkan_tula
    palkan RubyKaigi‘20
    We need an experiment
    64

    View Slide

  65. palkan_tula
    palkan RubyKaigi‘20
    The best way to taste new
    features is to start using
    them every day
    65

    View Slide

  66. palkan_tula
    palkan RubyKaigi‘20
    66
    Example: Hash shorthand

    View Slide

  67. palkan_tula
    palkan RubyKaigi‘20
    67
    Hash shorthand

    View Slide

  68. palkan_tula
    palkan RubyKaigi‘20
    $ ruby -v

    2.5.3p105

    $ ruby -ruby-next -e "

    event = "RubyKaigi"
    year = 2020
    p {event, year}

    "

    {event: "RubyKaigi", year: 2020}
    68
    Ruby Next 0.10.0

    View Slide

  69. palkan_tula
    palkan RubyKaigi‘20
    Share your opinion,
    let's make future Ruby
    together!
    69

    View Slide

  70. palkan_tula
    palkan RubyKaigi‘20
    github.com/ruby-next/ruby-next
    70

    View Slide

  71. Thank you!
    evilmartians.com
    github.com/ruby-next
    evilmartians
    palkan_tula
    Vladimir Dementyev

    View Slide