[RubyKaigi 2020] The why's and how's of transpiling Ruby

Transcript

1. Vladimir Dementyev
   Evil Martians
   The why’s and how’s
   of transpiling Ruby
   

   View Slide

2. palkan_tula
   palkan RubyKaigi‘20
   Ruby
   2
   

   View Slide

3. palkan_tula
   palkan RubyKaigi‘20
   Transpiler
   3
   Illustration from Ruby Weekly #477
   Source-to-source compiler
   

   View Slide

4. palkan_tula
   palkan RubyKaigi‘20
   The most popular transpiler
   4
   

   View Slide

5. palkan_tula
   palkan RubyKaigi‘20
   Yet another transpiler
   5
   

   View Slide

6. palkan_tula
   palkan RubyKaigi‘20
   6
   Ruby vs. Transpiling
   

   View Slide

7. palkan_tula
   palkan RubyKaigi‘20
   Ruby 3.02.8
   7
   

   View Slide

8. palkan_tula
   palkan RubyKaigi‘20
   Ruby is evolving faster
   than ever
   8
   

   View Slide

9. palkan_tula
   palkan RubyKaigi‘20
   Ruby 2.8
   9
   def greet(val) =
   case val
   in hello: hello if hello =~ /human/i
   ''
   in hello: 'martian'
   ''
   end
   greet(hello: 'martian') => greeting
   puts greeting # =>
   

   View Slide

10. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    10
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Pattern matching (2.7)
    

    View Slide

11. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    11
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Endless method (2.8)
    

    View Slide

12. palkan_tula
    palkan RubyKaigi‘20
    Ruby 2.8
    12
    def greet(val) =
    case val
    in hello: hello if hello =~ /human/i
    ''
    in hello: 'martian'
    ''
    end
    greet(hello: 'martian') => greeting
    puts greeting # =>
    Rightward assignment (2.8)
    

    View Slide

13. palkan_tula
    palkan RubyKaigi‘20
    How to use upcoming
    features today?
    13
    

    View Slide

14. palkan_tula
    palkan RubyKaigi‘20
    Transpiler for Ruby
    14
    

    View Slide

15. palkan_tula
    palkan RubyKaigi‘20
    $ ruby -v

    2.5.3p105

    $ ruby -ruby-next -e "

    case {hello: 'こんにちは'}

    in hello: hello if hello =~ /[\p{Hiragana}]+/

    puts '私はウラジミールです'

    in hello: hello if hello =~ /[a-z]+/i

    puts 'My name is Vladimir'

    end"

    私はウラジミールです
    15
    tl;dr
    

    View Slide

16. palkan_tula
    palkan RubyKaigi‘20
    About me
    16
    github.com/palkan
    

    View Slide

17. palkan_tula
    palkan RubyKaigi‘20
    evilmartians.com
    17
    

    View Slide

18. palkan_tula
    palkan RubyKaigi‘20
    18
    evilmartians.com
    

    View Slide

19. palkan_tula
    palkan RubyKaigi‘20
    evilmartians.com
    19
    Ruby Next
    Transpiler for
    Ruby
    

    View Slide

20. palkan_tula
    palkan RubyKaigi‘20
    evl.ms/blog
    20
    

    View Slide

21. palkan_tula
    palkan RubyKaigi‘20
    Why transpiling Ruby?
    21
    

    View Slide

22. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Why transpiling?
    22
    

    View Slide

23. palkan_tula
    palkan RubyKaigi‘20
    stats.rubygems.org
    23
    

    View Slide

24. palkan_tula
    palkan RubyKaigi‘20
    Gem authors should stick
    to older versions
    24
    

    View Slide

25. palkan_tula
    palkan RubyKaigi‘20
    Story: Hanami::API
    25
    

    View Slide

26. palkan_tula
    palkan RubyKaigi‘20
    26
    Story: Hanami::API
    

    View Slide

27. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    Why transpiling?
    27
    

    View Slide

28. palkan_tula
    palkan RubyKaigi‘20
    JRuby 9.3 — 2.6
    TruffleRuby — 2.6
    mruby ~ 2.7 (no pattern matching)
    Opal, RubyMotion, Artichoke — ???
    Syntax support
    28
    

    View Slide

29. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    ...
    Why transpiling?
    29
    

    View Slide

30. palkan_tula
    palkan RubyKaigi‘20
    How to transpile Ruby?
    30
    

    View Slide

31. palkan_tula
    palkan RubyKaigi‘20
    Transpiler is a source-to-
    source compiler
    31
    

    View Slide

32. palkan_tula
    palkan RubyKaigi‘20
    Compilers
    32
    @pgurtovaya
    

    View Slide

33. palkan_tula
    palkan RubyKaigi‘20
    Parse
    Analyze/optimize
    Generate
    Transpiling
    33
    

    View Slide

34. palkan_tula
    palkan RubyKaigi‘20
    Transpiling
    34
    source code new source code
    AST new AST
    

    View Slide

35. palkan_tula
    palkan RubyKaigi‘20
    Ripper
    RubyVM::AbstractSyntaxTree
    Parser
    Source to AST
    35
    

    View Slide

36. palkan_tula
    palkan RubyKaigi‘20
    Written in pure Ruby
    Version-independent
    Bullet-proofed (e.g., by
    RuboCop)
    Parser
    36
    

    View Slide

37. palkan_tula
    palkan RubyKaigi‘20
    github.com/whitequark/parser
    37
    

    View Slide

38. palkan_tula
    palkan RubyKaigi‘20
    def transpile(source)
    rewriters.inject(source) do |src, rewriter|
    buffer = Parser ::Source ::Buffer.new("")
    buffer.source = src
    rewriter.new.rewrite(buffer, parse(src))
    end
    end
    38
    

    View Slide

39. palkan_tula
    palkan RubyKaigi‘20
    module Rewriters
    class ArgsForward < Base
    def on_forward_args(node)
    replace(node.loc.expression, "(* __rest __, & __block __")
    end
    def on_send(node)
    return super(node) unless node.children[2]&.type == :forwarded_args
    replace(node.children[2].loc.expression, "* __rest __, & __block __")
    end
    end
    end
    39
    Rewriter
    

    View Slide

40. palkan_tula
    palkan RubyKaigi‘20
    Ruby Next Transpiling
    40
    source code new source code
    AST in-place rewrite
    unparse bits of AST
    

    View Slide

41. palkan_tula
    palkan RubyKaigi‘20
    41
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    Example
    

    View Slide

42. palkan_tula
    palkan RubyKaigi‘20
    42
    def fizzbuzz(num)
    rems = [num % 3, num % 5]
    if rems == [0, 0]
    'FizzBuzz'
    elsif rems[0] == 0
    'Fizz'
    elsif rems[1] == 1
    'Buzz'
    else
    raise "NoMatchingPattern"
    end
    end
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    2.7 2.5
    "Transpile"
    by hand
    

    View Slide

43. palkan_tula
    palkan RubyKaigi‘20
    43
    def fizzbuzz(num)
    case; when ( __m __ = [(num % 3), (num % 5)]) && false
    when ( __p_1 __ = ( __m __.respond_to?(:deconstruct) &&
    ((( __m_arr __ = __m __.deconstruct) || true) && ((Array
    === __m_arr __) || Kernel.raise(TypeError, "#deconstruct
    must return Array"))))) && (( __p_2 __ = (2 ==
    __m_arr __.size)) && ((0 === __m_arr __[0]) && (0 ===
    __m_arr __[1]))) then 'FizzBuzz'
    when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[0]))
    then 'Fizz'
    when __p_1 __ && ( __p_2 __ && (0 === __m_arr __[1]))
    then 'Buzz'; else; Kernel.raise(NoMatchingPatternError,
    __m __.inspect)
    end
    end
    def fizzbuzz(num)
    case [num % 3, num % 5]
    in [0, 0] then 'FizzBuzz'
    in [0, _] then 'Fizz'
    in [_, 0] then 'Buzz'
    end
    end
    For humans (2.7) For machines (2.5)
    Transpile with
    Ruby Next
    

    View Slide

44. palkan_tula
    palkan RubyKaigi‘20
    Parse
    Analyze/optimize
    Generate
    Transpiling
    44
    

    View Slide

45. palkan_tula
    palkan RubyKaigi‘20
    module Rewriters
    class PatternMatching < Base
    def on_case_match(node)
    # ~800 LOC
    end
    end
    end
    45
    Pattern matching
    

    View Slide

46. palkan_tula
    palkan RubyKaigi‘20
    def call(val)
    status, headers, body = 200, {}, ""
    case val
    in [String => body]
    [status, headers, [body]]
    in [Integer => status]
    [status, headers, [body]]
    in [Integer, String] => response
    [response[0], headers, [response[1]]]
    in [Integer, Hash, String] => response
    headers.merge!(response[1])
    [response[0], headers, [response[2]]]
    end
    end
    46
    Comparison:
    transpiled (last pattern): 1162200.7 i/s
    baseline (last pattern): 799739.5 i/s - 1.45x slower
    Transpiled != Slow
    

    View Slide

47. palkan_tula
    palkan RubyKaigi‘20
    47
    

    View Slide

48. palkan_tula
    palkan RubyKaigi‘20
    How to integrate a
    transpiler into an
    interpreted language?
    48
    

    View Slide

49. palkan_tula
    palkan RubyKaigi‘20
    Gems: transpile at “build”/
    release time
    Apps/Scripts: transpile at
    runtime
    49
    

    View Slide

50. palkan_tula
    palkan RubyKaigi‘20
    $LOAD_PATH: where to search
    for “features”
    $LOADED_FEATURES: required
    files
    Two pillars of require
    50
    

    View Slide

51. palkan_tula
    palkan RubyKaigi‘20
    Add transpiled files to releases
    No additional runtime deps*
    Nextify gems
    51
    * polyfills might still be required
    

    View Slide

52. palkan_tula
    palkan RubyKaigi‘20
    $ ruby-next nextify ./lib -V

    Generated: ./lib/.rbnext/2.7/rubanok/rule.rb

    Generated: ./lib/.rbnext/2.8/rubanok/dsl/matching.rb
    52
    

    View Slide

53. palkan_tula
    palkan RubyKaigi‘20
    # lib/my_gem.rb
    require "ruby-next/language/setup"
    RubyNext ::Language.setup_gem_load_path
    53
    

    View Slide

54. palkan_tula
    palkan RubyKaigi‘20
    def setup_gem_load_path
    lib_dir = File.dirname(caller_locations(1, 1).first.path)
    current_index = $LOAD_PATH.index(lib_dir)
    next_dir = File.join(lib_dir, ".rbnext")
    if File.exist?(next_dir)
    $LOAD_PATH.insert current_index, next_dir
    current_index += 1
    end
    end
    54
    

    View Slide

55. palkan_tula
    palkan RubyKaigi‘20
    Hijack Kernel#require & co
    Reimplement require
    mechanism
    Runtime
    55
    

    View Slide

56. palkan_tula
    palkan RubyKaigi‘20
    module Kernel
    module_function
    alias_method :require_without_ruby_next, :require
    def require(path)
    RubyNext.require(path)
    rescue => e
    warn "Failed to require ' #{path}': #{e.message}"
    require_without_ruby_next(path)
    end
    end
    56
    Kernel#require
    *Don't try this at home
    

    View Slide

57. palkan_tula
    palkan RubyKaigi‘20
    module RubyNext
    module_function
    def require(path)
    realpath = resolve_feature_path(path)
    return false if $LOADED_FEATURES.include?(realpath)
    File.read(realpath)
    .then(&Language.:transform).then do |source|
    RubyVM ::InstructionSequence.compile(code, filepath).then(&:eval)
    $LOADED_FEATURES << path
    true
    end
    end
    end
    57
    RubyNext.require
    

    View Slide

58. palkan_tula
    palkan RubyKaigi‘20
    How to use a transpiler
    with compilable Rubies?
    58
    

    View Slide

59. palkan_tula
    palkan RubyKaigi‘20
    Transpile source files before
    compiling
    Replace target files with the
    transpiled versions at build time
    mruby
    59
    

    View Slide

60. palkan_tula
    palkan RubyKaigi‘20
    desc "transpile source code with ruby-next"
    task rbnext: [] do
    Dir.chdir(APP_ROOT) do
    sh "ruby-next nextify -V"
    end
    end
    Rake ::Task["compile"].enhance [:rbnext]
    60
    Nextify before compiling
    

    View Slide

61. palkan_tula
    palkan RubyKaigi‘20
    using(Module.new do
    refine MRuby ::Gem ::Specification do
    def setup_ruby_next!(next_dir: ".rbnext")
    lib_root = File.join(@dir, @mrblib_dir)
    next_root = Pathname.new(next_dir).absolute? ? next_dir : File.join(lib_root, next_dir)
    Dir.glob(" #{next_root}/**/*.rb").each do |next_file|
    orig_file = next_file.sub(next_root, lib_root)
    index = @rbfiles.index(orig_file) || raise "Source file not found for: #{next_file}"
    @rbfiles[index] = next_file
    end
    end
    end
    end)
    MRuby ::Gem ::Specification.new("acli") do |spec|
    # ...
    spec.setup_ruby_next!
    end
    61
    Update rbfiles
    

    View Slide

62. palkan_tula
    palkan RubyKaigi‘20
    Backporting
    Interoperability
    Evolution
    Why transpiling?
    62
    

    View Slide

63. palkan_tula
    palkan RubyKaigi‘20
    Recent Ruby versions added
    experimental features
    63
    

    View Slide

64. palkan_tula
    palkan RubyKaigi‘20
    We need an experiment
    64
    

    View Slide

65. palkan_tula
    palkan RubyKaigi‘20
    The best way to taste new
    features is to start using
    them every day
    65
    

    View Slide

66. palkan_tula
    palkan RubyKaigi‘20
    66
    Example: Hash shorthand
    

    View Slide

67. palkan_tula
    palkan RubyKaigi‘20
    67
    Hash shorthand
    

    View Slide

68. palkan_tula
    palkan RubyKaigi‘20
    $ ruby -v

    2.5.3p105

    $ ruby -ruby-next -e "

    event = "RubyKaigi"
    year = 2020
    p {event, year}

    "

    {event: "RubyKaigi", year: 2020}
    68
    Ruby Next 0.10.0
    

    View Slide

69. palkan_tula
    palkan RubyKaigi‘20
    Share your opinion,
    let's make future Ruby
    together!
    69
    

    View Slide

70. palkan_tula
    palkan RubyKaigi‘20
    github.com/ruby-next/ruby-next
    70
    

    View Slide

71. Thank you!
    evilmartians.com
    github.com/ruby-next
    evilmartians
    palkan_tula
    Vladimir Dementyev
    

    View Slide