Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Ruby Language Server

Ruby Language Server

Slides for my talk "Ruby Language Server" at RubyKaigi 2017 in Hiroshima, Japan

http://rubykaigi.org/2017/presentations/mtsmfm.html

Fumiaki MATSUSHIMA

September 19, 2017
Tweet

More Decks by Fumiaki MATSUSHIMA

Other Decks in Programming

Transcript

  1. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Fumiaki

    Matsushima GitHub, Twitter @mtsmfm Web Developer 3 / 116
  2. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Migrate

    Asakusa.rb meetup logs https://asakusarb.esa.io/ 13 / 116
  3. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Nothing

    on my laptop $ ruby -v ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16] $ rbenv -v zsh: command not found: rbenv 15 / 116
  4. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Do

    everything on Docker (basically) $ docker run ruby ruby -v ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux] $ docker-compose run ruby-2-4 bundle exec \ rake test $ docker-compose run ruby-2-3 bundle exec \ rake test 16 / 116
  5. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1.

    What is Language Server? 2. How can we implement Language Server in Ruby? 3. Introduction of language_server gem Today’s topics 20 / 116
  6. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1.

    What is Language Server? 1. What is Language Server? 2. How can we implement Language Server in Ruby? 3. Introduction of language_server gem 21 / 116
  7. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com What

    is Language Server? - A server which provides information to editors - ex. auto completion candidates, method definitions - How to communicate is defined as Language Server Protocol (LSP) - JSON-RPC based - Originally created by Microsoft and made open standard at 2016-06 22 / 116
  8. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Editor Position Completion items User execute “Trigger suggest” Server sends completion items 23 / 116
  9. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Editor Document (valid) User edits document (No op) 24 / 116
  10. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Editor Document (valid) Syntax Error User edits document Document (invalid) Server sends errors (No op) 25 / 116
  11. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Editor Document Formatted document User format document Server sends formatted document 26 / 116
  12. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Why

    do we need Language Server? - We have built language plugins for each editor - ex. a Ruby plugin for Vim, a Ruby plugin for Emacs, a Ruby plugin for VS code, … - We need to re-implement if we once implement core logic in editor specific language - Now we need to build just one Language Server 27 / 116
  13. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Traditional

    plugin Ruby Python PHP ... Vim Emacs VS Code Atom ... 28 / 116
  14. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Ruby

    Python PHP ... Vim ✔ Emacs ✔ VS Code ✔ Atom ✔ ... ... Traditional plugin 29 / 116
  15. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Ruby Python PHP ... Vim Emacs VS Code Atom ... 30 / 116
  16. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Ruby Python PHP ... Vim ✔ Emacs VS Code Atom ... 31 / 116
  17. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 2.

    How can we implement Language Server in Ruby? 1. What is Language Server? 2. How can we implement Language Server in Ruby? 3. Introduction of language_server gem 36 / 116
  18. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Creating

    Language Servers for Visual Studio Code https://code.visualstudio.com/docs/extensions/example-language-server 37 / 116
  19. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Editor - VS Code extension - Node.js - vscode-languageclient (npm) - Node.js - vscode- languageserver (npm) Implementation in the example 39 / 116
  20. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Ruby

    Python PHP ... Vim Emacs VS Code ✔ Atom ... In the example, we will build plugin for the editor... 40 / 116
  21. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Editor LSP Client (Plugin) Activate 42 / 116 Boot
  22. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Editor LSP Client (Plugin) Notify document is changed Notify document is changed Send error Show error Activate 43 / 116 Boot
  23. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Kinds

    of LSP Client - Editor specific library - Plugin author builds plugins depend on the library for each editor - https://github.com/Microsoft/vscode-languageserver-node - https://github.com/atom/atom-languageclient - Universal LSP Client plugin - Users install one plugin for all language - https://github.com/tomv564/LSP - https://github.com/autozimu/LanguageClient-neovim 45 / 116
  24. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com vscode-language

    -client vscode-ruby Ruby atom-language-client atom-ruby vscode-language -client vscode-python Python atom-language-client atom-python Using editor specific library Atom LSP Client lib VS Code LSP Client lib 46 / 116
  25. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Boot

    setting for Ruby Ruby Python Using universal LSP Client plugin Boot setting for Python Universal Client (VS code) Boot setting for Ruby Boot setting for Python Universal Client (VS code) 47 / 116
  26. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Pros

    and Cons - Editor specific library Author can provide editor specific configuration User don’t have to configure Author needs to build plugins for every editor User needs plugins for every language - Universal LSP Client plugin Author has to do “nothing” User needs one plugin for all language Author can’t provide editor specific configuration User must configure how to boot Language Server 48 / 116
  27. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com VS

    Code Language Server extension example let disposable = new LanguageClient( 'languageServerExample', 'Language Server Example', { module: context.asAbsolutePath(path.join('server', 'server.js')), transport: TransportKind.ipc }, { documentSelector: ['plaintext'], } ).start(); context.subscriptions.push(disposable); 49 / 116
  28. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com OK,

    let’s start with editor specific library for VSCode extension 50 / 116
  29. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server Editor - VS Code extension - Node.js - vscode-languageclient (npm) - Node.js - vscode- languageserver (npm) Implementation in the example 51 / 116
  30. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server requirements - Communicate with client using JSON-RPC 53 / 116
  31. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com JSON-RPC

    - A simple remote procedure call protocol - It uses JSON as data format - Transport agnostic - We can use socket, STDIO, HTTP, or other one as a transport layer 54 / 116
  32. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com {

    "jsonrpc": "2.0", "id": 4, "method": "textDocument/definition", "params": { "textDocument": { "uri": "file:///path/to/a.rb" }, "position": { "line": 1, "character": 15 } } } { "jsonrpc": "2.0", "id": 4, "result": [{ "uri": "file:///path/to/b.rb", "range": { "start": { "line": 6, "character": 0 }, "end": { "line": 36, "character": 0 } } } }] } JSON-RPC Request Response 55 / 116
  33. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server requirements - Communicate with client using JSON-RPC 56 / 116
  34. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server requirements - Communicate with client using JSON-RPC via client supported transport layer 57 / 116
  35. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com VS

    Code Language Server extension example let disposable = new LanguageClient( 'languageServerExample', 'Language Server Example', { module: context.asAbsolutePath(path.join('server', 'server.js')), transport: TransportKind.ipc }, { documentSelector: ['plaintext'], } ).start(); context.subscriptions.push(disposable); 58 / 116
  36. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server requirements - Communicate with client using JSON-RPC via client supported transport layer - vscode-languageclient supports: - STDIO - Node ipc - Named pipe - Socket file 59 / 116
  37. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language

    Server requirements - Communicate with client using JSON-RPC via client supported transport layer - vscode-languageclient supports: - STDIO - Node ipc - Named pipe - Socket file 63 / 116
  38. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com class

    Reader def read(&block) buffer = "" header_parsed = false content_length = nil while char = STDIN.getc buffer << char unless header_parsed if buffer[-4..-1] == "\r\n" * 2 content_length = buffer.match(/Content-Length: (\d+)/i)[1].to_i header_parsed = true buffer.clear end else if buffer.bytesize == content_length request = JSON.parse(buffer, symbolize_names: true) block.call(request) header_parsed = false buffer.clear end end end end end 64 / 116
  39. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com class

    Writer def write(response) response_str = response.merge(jsonrpc: "2.0").to_json headers = {"Content-Length" => response_str.bytesize} headers.each do |k, v| STDOUT.print "#{k}: #{v}\r\n" end STDOUT.print "\r\n" STDOUT.print response_str STDOUT.flush end end 65 / 116
  40. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com writer

    = Writer.new reader = Reader.new reader.read do |request| writer.write(id: request[:id], result: {}) End 66 / 116
  41. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com https://github.com/mtsmfm/language_server-protocol-ruby/blob/v

    0.3.0/lib/language_server/protocol/interface/initialize_result.rb https://github.com/Microsoft/language-server-proto col/blob/3.0.0/protocol.md Convert TS interface to Ruby class 74 / 116
  42. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 3.

    Introduction of language_server gem 1. What is Language Server? 2. How can we implement Language Server in Ruby? 3. Introduction of language_server gem 75 / 116
  43. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com language_server

    gem - A Language Server implementation for Ruby - In alpha stage - Pure Ruby - Syntax check - [WIP] Auto completion - [WIP] Jump to definition 76 / 116
  44. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Auto

    complete (top level) Syntax Check [WIP] Auto complete (instance level) [WIP] Jump to definition 77 / 116
  45. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com language_server

    gem overview AdHoc Definition Provider FileStore RubyWC Reader Completion Provider Writer Editor Linter AdHoc Rcodetools 80 / 116 Project
  46. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com textDocument

    /didChange textDocument/ publishDiagnostics Document Syntax Error Language Server Editor 82 / 116
  47. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1.

    Store document on memory (FileStore) 2. Run $ ruby -wc Syntax Error Document textDocument/didChange FileStore RubyWC Linter 83 / 116 textDocument/ publishDiagnostics Reader Writer
  48. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Syntax

    check $ ruby -wc foo.rb foo.rb:1: warning: assigned but unused variable - foo Syntax OK foo = 1 84 / 116
  49. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Syntax

    check _, err, _ = Open3.capture3("ruby -wc", stdin_data: @source) err.scan(/.+:(\d+):\s*(.+?)[,:]\s(.+)/).map do |line_num, type, message| Error.new(line_num: line_num.to_i - 1, message: message, type: type) end $ ruby -wc foo.rb foo.rb:1: warning: assigned but unused variable - foo Syntax OK 85 / 116
  50. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com textDocument

    /completion Position Completion items Language Server Editor 87 / 116
  51. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1.

    Read document from memory (FileStore) 2. Run rcodetools FileStore 88 / 116 textDocument/completion Rcodetools Completion items Position Completion Provider Reader Writer
  52. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com rcodetools

    - Collection of Ruby code manipulation tools - https://github.com/rcodetools/rcodetools - Written in pure Ruby (other than editor specific codes) - Dynamic analysis 89 / 116
  53. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com class

    Filter < ::Rcodetools::XMPCompletionFilter @candidates_with_description_flag = true def completion_code(*args) candidates_with_class(*args) rescue NewCodeError, RuntimeDataError, NoCandidates [nil, []] end end _, candidates = Filter.run(source, lineno: @line + 1, column: @character) candidates.map do |candidate| method_name, description = candidate.split(/\0/, 2) Protocol::Interfaces::CompletionItem.new( label: method_name, detail: description, kind: Protocol::Constants::CompletionItemKind::METHOD ) end 90 / 116
  54. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Dynamic

    analysis Can collect exact information Easier than static analysis - Ruby has powerful meta-programming features! Side-effect Can’t analyze instance level 91 / 116
  55. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Dynamic

    analysis problem - Side-effect - Analysing following code (accidently) creates foo.txt result = File.write('foo.txt', 'hi') result. 92 / 116
  56. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Dynamic

    analysis problem - Can’t analyze instance level - Non-reachable directly class Foo def initialize(requirement) @requirement = requirement end def foo 1. end end 93 / 116
  57. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com [WIP]

    Auto completion (instance level) - Static analysis - Constants only for now 94 / 116
  58. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 0.

    Analyze files under $BUNDLE_PATH statically on boot (Project) 1. Get current context for the position 2. Find available constants for the context FileStore 95 / 116 textDocument/completion Completion items Position Completion Provider AdHoc Project Reader Writer
  59. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com module

    Foo class Bar end End Ripper.sexp_raw(File.read('a.rb')) 97 / 116
  60. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com [

    :program, [:stmts_add, [:stmts_new], [ :module, [:const_ref, [:@const, "Foo", [1, 7]]], nil, [ :bodystmt, [ :stmts_add, [...], [ :class, [:const_ref, [:@const, "Bar", [2, 8]]], [...] ] ], ... ] ] ] ] 1 module Foo 2 class Bar 3 end 4 end 98 / 116
  61. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com class

    Parser < Ripper def on_const(name) Node::Constant.new(name: name, lineno: lineno, column: column) end def on_class(constant, superclass, children) result.classes << Node::Class.new( constant: constant, superclass: superclass, children: children, lineno: lineno, column: column ) end def on_module(constant, children) result.modules << Node::Module.new( constant: constant, children: children, lineno: lineno, column: column ) end End Event driven style 99 / 116
  62. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules

    = [ Module.new( lineno: 4, constant: Constant.new(name: 'Foo', namespaces: [], lineno: 1) ) ] result.classes = [ Class.new( lineno: 3, constant: Constant.new(name: 'Bar', namespaces: ['Foo'], lineno: 2) ) ] 1 module Foo 2 class Bar 3 end 4 end 100 / 116 ※ We’ll get column also
  63. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules

    #=> [<Module Foo#L1-10>, <Module Foo::Bar::Baz#L3-4>] result.classes #=> [<Class Foo::Bar#L2-5>] 1 module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 Ba_ 9 end 10 end Position (8, 7) 1. Get current context for the position 2. Find available constants for the context 101 / 116
  64. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules

    #=> [<Module Foo#L1-10>, <Module Foo::Bar::Baz#L3-4>] result.classes #=> [<Class Foo::Bar#L2-5>] 1 module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 Ba_ 9 end 10 end Position (8, 7) 1. Get current context for the position 2. Find available constants for the context 102 / 116
  65. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules

    #=> [<Module Foo#L1-10>, <Module Foo::Bar::Baz#L3-4>] result.classes #=> [<Class Foo::Bar#L2-5>] 1 module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 Ba_ 9 end 10 end Position (8, 7) 1. Get current context for the position 2. Find available constants for the context 103 / 116
  66. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com [WIP]

    Jump to definition - Static analysis - Currently class/module only 104 / 116
  67. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com textDocument

    /definition Position Location(s) Language Server Editor 105 / 116
  68. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 0.

    Analyze files under $BUNDLE_PATH statically on boot (Project) 1. Get current node for the position 2. Find module/class for the node FileStore 106 / 116 textDocument/definition Location(s) Position Definition Provider AdHoc Project Reader Writer
  69. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules

    #=> [<Class Foo#L1-6>, <Class Foo#L8-10>] result.classes #=> [<Class Foo::Bar#L2-5>] result.refs #=> [<VarRef Foo::Bar#L9(3..5)>] 1 class Foo 2 module Bar 3 def hi 4 end 5 end 6 end 7 8 class Foo 9 Bar 10 end Position (9, 3) 1. Get current context for the position 2. Find module/class for the node 107 / 116
  70. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules

    #=> [<Class Foo#L1-6>, <Class Foo#L8-10>] result.classes #=> [<Class Foo::Bar#L2-5>] result.refs #=> [<VarRef Foo::Bar#L9(3..5)>] 1 class Foo 2 module Bar 3 def hi 4 end 5 end 6 end 7 8 class Foo 9 Bar 10 end Position (9, 3) 1. Get current context for the position 2. Find module/class for the node 108 / 116
  71. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules

    #=> [<Class Foo#L1-6>, <Class Foo#L8-10>] result.classes #=> [<Class Foo::Bar#L2-5>] result.refs #=> [<VarRef Foo::Bar#L9(3..5)>] 1 class Foo 2 module Bar 3 def hi 4 end 5 end 6 end 7 8 class Foo 9 Bar 10 end Position (9, 3) 1. Get current context for the position 2. Find module/class for the node 109 / 116
  72. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Future

    prospects - Fully implement static analysis - Advanced syntax check - Jump to definition for Docker environment - Editor can’t read some files - Need to extend LSP (probably) - Show documentation - Integrate other tools - Syntax highlight etc... 110 / 116
  73. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1

    module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 Bar::Ba_ 9 end 10 end 111 / 116
  74. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1

    module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 self.class::Ba_ 9 end 10 end 112 / 116
  75. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Syntax

    check $ ruby -wc foo.rb foo.rb:2: syntax error, unexpected $undefined, expecting end-of-input if a == "\\n" ^ require "foo if a == "\\n" 113 / 116
  76. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Conclusion

    - Building Language Server is a common way to provide useful information to editor - I created language_server gem, an implementation of Language Server for Ruby - You can build your Language Server using language_server-protocol gem 114 / 116
  77. Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Let’s

    make our Ruby development experience better ! 116 / 116