Share
Embed
Ruby Language Server
Slides for my talk "Ruby Language Server" at RubyKaigi 2017 in Hiroshima, Japan
http://rubykaigi.org/2017/presentations/mtsmfm.html
Fumiaki MATSUSHIMA
September 19, 2017
Transcript
116 slides
-
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Fumiaki
Matsushima GitHub, Twitter @mtsmfm Web Developer 3 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com The
only handwriting mahjong font (Probably) 7 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com I
Mahjong Ruby 9 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com https://ninirb.github.io
An organizer of Nishinippori.rb (Arakawa-ku, Tokyo) 10 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Migrate
Asakusa.rb meetup logs https://asakusarb.esa.io/ 13 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com I
Mahjong Ruby Docker 14 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Nothing
on my laptop $ ruby -v ruby 2.0.0p648 (2015-12-16 revision 53162) [universal.x86_64-darwin16] $ rbenv -v zsh: command not found: rbenv 15 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Do
everything on Docker (basically) $ docker run ruby ruby -v ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux] $ docker-compose run ruby-2-4 bundle exec \ rake test $ docker-compose run ruby-2-3 bundle exec \ rake test 16 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com https://github.com/mtsmfm/yaichi
Access your containers via <container-name>.localhost 18 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com I
Mahjong Ruby Docker 19 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1.
What is Language Server? 2. How can we implement Language Server in Ruby? 3. Introduction of language_server gem Today’s topics 20 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1.
What is Language Server? 1. What is Language Server? 2. How can we implement Language Server in Ruby? 3. Introduction of language_server gem 21 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com What
is Language Server? - A server which provides information to editors - ex. auto completion candidates, method definitions - How to communicate is defined as Language Server Protocol (LSP) - JSON-RPC based - Originally created by Microsoft and made open standard at 2016-06 22 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Editor Position Completion items User execute “Trigger suggest” Server sends completion items 23 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Editor Document (valid) User edits document (No op) 24 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Editor Document (valid) Syntax Error User edits document Document (invalid) Server sends errors (No op) 25 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Editor Document Formatted document User format document Server sends formatted document 26 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Why
do we need Language Server? - We have built language plugins for each editor - ex. a Ruby plugin for Vim, a Ruby plugin for Emacs, a Ruby plugin for VS code, … - We need to re-implement if we once implement core logic in editor specific language - Now we need to build just one Language Server 27 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Traditional
plugin Ruby Python PHP ... Vim Emacs VS Code Atom ... 28 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Ruby
Python PHP ... Vim ✔ Emacs ✔ VS Code ✔ Atom ✔ ... ... Traditional plugin 29 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Ruby Python PHP ... Vim Emacs VS Code Atom ... 30 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Ruby Python PHP ... Vim ✔ Emacs VS Code Atom ... 31 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 2.
How can we implement Language Server in Ruby? 1. What is Language Server? 2. How can we implement Language Server in Ruby? 3. Introduction of language_server gem 36 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Creating
Language Servers for Visual Studio Code https://code.visualstudio.com/docs/extensions/example-language-server 37 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Editor - VS Code extension - Node.js - vscode-languageclient (npm) - Node.js - vscode- languageserver (npm) Implementation in the example 39 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Ruby
Python PHP ... Vim Emacs VS Code ✔ Atom ... In the example, we will build plugin for the editor... 40 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Still
we need to build plugin for each editor? 41 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Editor LSP Client (Plugin) Activate 42 / 116 Boot -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Editor LSP Client (Plugin) Notify document is changed Notify document is changed Send error Show error Activate 43 / 116 Boot -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com LSP
Client list http://langserver.org/ 44 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Kinds
of LSP Client - Editor specific library - Plugin author builds plugins depend on the library for each editor - https://github.com/Microsoft/vscode-languageserver-node - https://github.com/atom/atom-languageclient - Universal LSP Client plugin - Users install one plugin for all language - https://github.com/tomv564/LSP - https://github.com/autozimu/LanguageClient-neovim 45 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com vscode-language
-client vscode-ruby Ruby atom-language-client atom-ruby vscode-language -client vscode-python Python atom-language-client atom-python Using editor specific library Atom LSP Client lib VS Code LSP Client lib 46 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Boot
setting for Ruby Ruby Python Using universal LSP Client plugin Boot setting for Python Universal Client (VS code) Boot setting for Ruby Boot setting for Python Universal Client (VS code) 47 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Pros
and Cons - Editor specific library Author can provide editor specific configuration User don’t have to configure Author needs to build plugins for every editor User needs plugins for every language - Universal LSP Client plugin Author has to do “nothing” User needs one plugin for all language Author can’t provide editor specific configuration User must configure how to boot Language Server 48 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com VS
Code Language Server extension example let disposable = new LanguageClient( 'languageServerExample', 'Language Server Example', { module: context.asAbsolutePath(path.join('server', 'server.js')), transport: TransportKind.ipc }, { documentSelector: ['plaintext'], } ).start(); context.subscriptions.push(disposable); 49 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com OK,
let’s start with editor specific library for VSCode extension 50 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server Editor - VS Code extension - Node.js - vscode-languageclient (npm) - Node.js - vscode- languageserver (npm) Implementation in the example 51 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com How
about Ruby? 52 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server requirements - Communicate with client using JSON-RPC 53 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com JSON-RPC
- A simple remote procedure call protocol - It uses JSON as data format - Transport agnostic - We can use socket, STDIO, HTTP, or other one as a transport layer 54 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com {
"jsonrpc": "2.0", "id": 4, "method": "textDocument/definition", "params": { "textDocument": { "uri": "file:///path/to/a.rb" }, "position": { "line": 1, "character": 15 } } } { "jsonrpc": "2.0", "id": 4, "result": [{ "uri": "file:///path/to/b.rb", "range": { "start": { "line": 6, "character": 0 }, "end": { "line": 36, "character": 0 } } } }] } JSON-RPC Request Response 55 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server requirements - Communicate with client using JSON-RPC 56 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server requirements - Communicate with client using JSON-RPC via client supported transport layer 57 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com VS
Code Language Server extension example let disposable = new LanguageClient( 'languageServerExample', 'Language Server Example', { module: context.asAbsolutePath(path.join('server', 'server.js')), transport: TransportKind.ipc }, { documentSelector: ['plaintext'], } ).start(); context.subscriptions.push(disposable); 58 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server requirements - Communicate with client using JSON-RPC via client supported transport layer - vscode-languageclient supports: - STDIO - Node ipc - Named pipe - Socket file 59 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Existed
ruby gems support HTTP only 61 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Not
difficult to implement (tentative) one 62 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Language
Server requirements - Communicate with client using JSON-RPC via client supported transport layer - vscode-languageclient supports: - STDIO - Node ipc - Named pipe - Socket file 63 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com class
Reader def read(&block) buffer = "" header_parsed = false content_length = nil while char = STDIN.getc buffer << char unless header_parsed if buffer[-4..-1] == "\r\n" * 2 content_length = buffer.match(/Content-Length: (\d+)/i)[1].to_i header_parsed = true buffer.clear end else if buffer.bytesize == content_length request = JSON.parse(buffer, symbolize_names: true) block.call(request) header_parsed = false buffer.clear end end end end end 64 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com class
Writer def write(response) response_str = response.merge(jsonrpc: "2.0").to_json headers = {"Content-Length" => response_str.bytesize} headers.each do |k, v| STDOUT.print "#{k}: #{v}\r\n" end STDOUT.print "\r\n" STDOUT.print response_str STDOUT.flush end end 65 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com writer
= Writer.new reader = Reader.new reader.read do |request| writer.write(id: request[:id], result: {}) End 66 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com https://github.com/Microsoft/language-server-protocol/blob/3.0.0/protocol.md#initialize
Server can tell its capabilities 71 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com https://github.com/mtsmfm/language_server-protocol-ruby/blob/v
0.3.0/lib/language_server/protocol/interface/initialize_result.rb https://github.com/Microsoft/language-server-proto col/blob/3.0.0/protocol.md Convert TS interface to Ruby class 74 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 3.
Introduction of language_server gem 1. What is Language Server? 2. How can we implement Language Server in Ruby? 3. Introduction of language_server gem 75 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com language_server
gem - A Language Server implementation for Ruby - In alpha stage - Pure Ruby - Syntax check - [WIP] Auto completion - [WIP] Jump to definition 76 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Auto
complete (top level) Syntax Check [WIP] Auto complete (instance level) [WIP] Jump to definition 77 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com language_server
gem overview AdHoc Definition Provider FileStore RubyWC Reader Completion Provider Writer Editor Linter AdHoc Rcodetools 80 / 116 Project -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Syntax
check Error Warn 81 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com textDocument
/didChange textDocument/ publishDiagnostics Document Syntax Error Language Server Editor 82 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1.
Store document on memory (FileStore) 2. Run $ ruby -wc Syntax Error Document textDocument/didChange FileStore RubyWC Linter 83 / 116 textDocument/ publishDiagnostics Reader Writer -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Syntax
check $ ruby -wc foo.rb foo.rb:1: warning: assigned but unused variable - foo Syntax OK foo = 1 84 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Syntax
check _, err, _ = Open3.capture3("ruby -wc", stdin_data: @source) err.scan(/.+:(\d+):\s*(.+?)[,:]\s(.+)/).map do |line_num, type, message| Error.new(line_num: line_num.to_i - 1, message: message, type: type) end $ ruby -wc foo.rb foo.rb:1: warning: assigned but unused variable - foo Syntax OK 85 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Auto
completion (top level) 86 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com textDocument
/completion Position Completion items Language Server Editor 87 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1.
Read document from memory (FileStore) 2. Run rcodetools FileStore 88 / 116 textDocument/completion Rcodetools Completion items Position Completion Provider Reader Writer -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com rcodetools
- Collection of Ruby code manipulation tools - https://github.com/rcodetools/rcodetools - Written in pure Ruby (other than editor specific codes) - Dynamic analysis 89 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com class
Filter < ::Rcodetools::XMPCompletionFilter @candidates_with_description_flag = true def completion_code(*args) candidates_with_class(*args) rescue NewCodeError, RuntimeDataError, NoCandidates [nil, []] end end _, candidates = Filter.run(source, lineno: @line + 1, column: @character) candidates.map do |candidate| method_name, description = candidate.split(/\0/, 2) Protocol::Interfaces::CompletionItem.new( label: method_name, detail: description, kind: Protocol::Constants::CompletionItemKind::METHOD ) end 90 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Dynamic
analysis Can collect exact information Easier than static analysis - Ruby has powerful meta-programming features! Side-effect Can’t analyze instance level 91 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Dynamic
analysis problem - Side-effect - Analysing following code (accidently) creates foo.txt result = File.write('foo.txt', 'hi') result. 92 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Dynamic
analysis problem - Can’t analyze instance level - Non-reachable directly class Foo def initialize(requirement) @requirement = requirement end def foo 1. end end 93 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com [WIP]
Auto completion (instance level) - Static analysis - Constants only for now 94 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 0.
Analyze files under $BUNDLE_PATH statically on boot (Project) 1. Get current context for the position 2. Find available constants for the context FileStore 95 / 116 textDocument/completion Completion items Position Completion Provider AdHoc Project Reader Writer -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com module
Foo class Bar end End Ripper.sexp_raw(File.read('a.rb')) 97 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com [
:program, [:stmts_add, [:stmts_new], [ :module, [:const_ref, [:@const, "Foo", [1, 7]]], nil, [ :bodystmt, [ :stmts_add, [...], [ :class, [:const_ref, [:@const, "Bar", [2, 8]]], [...] ] ], ... ] ] ] ] 1 module Foo 2 class Bar 3 end 4 end 98 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com class
Parser < Ripper def on_const(name) Node::Constant.new(name: name, lineno: lineno, column: column) end def on_class(constant, superclass, children) result.classes << Node::Class.new( constant: constant, superclass: superclass, children: children, lineno: lineno, column: column ) end def on_module(constant, children) result.modules << Node::Module.new( constant: constant, children: children, lineno: lineno, column: column ) end End Event driven style 99 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules
= [ Module.new( lineno: 4, constant: Constant.new(name: 'Foo', namespaces: [], lineno: 1) ) ] result.classes = [ Class.new( lineno: 3, constant: Constant.new(name: 'Bar', namespaces: ['Foo'], lineno: 2) ) ] 1 module Foo 2 class Bar 3 end 4 end 100 / 116 ※ We’ll get column also -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules
#=> [<Module Foo#L1-10>, <Module Foo::Bar::Baz#L3-4>] result.classes #=> [<Class Foo::Bar#L2-5>] 1 module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 Ba_ 9 end 10 end Position (8, 7) 1. Get current context for the position 2. Find available constants for the context 101 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules
#=> [<Module Foo#L1-10>, <Module Foo::Bar::Baz#L3-4>] result.classes #=> [<Class Foo::Bar#L2-5>] 1 module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 Ba_ 9 end 10 end Position (8, 7) 1. Get current context for the position 2. Find available constants for the context 102 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules
#=> [<Module Foo#L1-10>, <Module Foo::Bar::Baz#L3-4>] result.classes #=> [<Class Foo::Bar#L2-5>] 1 module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 Ba_ 9 end 10 end Position (8, 7) 1. Get current context for the position 2. Find available constants for the context 103 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com [WIP]
Jump to definition - Static analysis - Currently class/module only 104 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com textDocument
/definition Position Location(s) Language Server Editor 105 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 0.
Analyze files under $BUNDLE_PATH statically on boot (Project) 1. Get current node for the position 2. Find module/class for the node FileStore 106 / 116 textDocument/definition Location(s) Position Definition Provider AdHoc Project Reader Writer -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules
#=> [<Class Foo#L1-6>, <Class Foo#L8-10>] result.classes #=> [<Class Foo::Bar#L2-5>] result.refs #=> [<VarRef Foo::Bar#L9(3..5)>] 1 class Foo 2 module Bar 3 def hi 4 end 5 end 6 end 7 8 class Foo 9 Bar 10 end Position (9, 3) 1. Get current context for the position 2. Find module/class for the node 107 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules
#=> [<Class Foo#L1-6>, <Class Foo#L8-10>] result.classes #=> [<Class Foo::Bar#L2-5>] result.refs #=> [<VarRef Foo::Bar#L9(3..5)>] 1 class Foo 2 module Bar 3 def hi 4 end 5 end 6 end 7 8 class Foo 9 Bar 10 end Position (9, 3) 1. Get current context for the position 2. Find module/class for the node 108 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com result.modules
#=> [<Class Foo#L1-6>, <Class Foo#L8-10>] result.classes #=> [<Class Foo::Bar#L2-5>] result.refs #=> [<VarRef Foo::Bar#L9(3..5)>] 1 class Foo 2 module Bar 3 def hi 4 end 5 end 6 end 7 8 class Foo 9 Bar 10 end Position (9, 3) 1. Get current context for the position 2. Find module/class for the node 109 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Future
prospects - Fully implement static analysis - Advanced syntax check - Jump to definition for Docker environment - Editor can’t read some files - Need to extend LSP (probably) - Show documentation - Integrate other tools - Syntax highlight etc... 110 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1
module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 Bar::Ba_ 9 end 10 end 111 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com 1
module Foo 2 class Bar 3 module Baz 4 end 5 end 6 7 def foo 8 self.class::Ba_ 9 end 10 end 112 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Syntax
check $ ruby -wc foo.rb foo.rb:2: syntax error, unexpected $undefined, expecting end-of-input if a == "\\n" ^ require "foo if a == "\\n" 113 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Conclusion
- Building Language Server is a common way to provide useful information to editor - I created language_server gem, an implementation of Language Server for Ruby - You can build your Language Server using language_server-protocol gem 114 / 116 -
Emoji artwork provided by EmojiOne Background pattern from subtlepatterns.com Let’s
make our Ruby development experience better ! 116 / 116