Upgrade to Pro — share decks privately, control downloads, hide ads and more …

IDE Development with Ruby

IDE Development with Ruby

Euruko 2021

Avatar for Soutaro Matsumoto

Soutaro Matsumoto

May 29, 2021
Tweet

More Decks by Soutaro Matsumoto

Other Decks in Programming

Transcript

  1. Soutaro Matsumoto • Working for Square from Tokyo • A

    Ruby core committer working for RBS • Develops a static type checker Steep • @soutaro (GitHub, Twitter)
  2. $ steep check # Type checking files: ............................................................................F................F.F.F.F.... .................................................... lib/rbs/definition_builder/ancestor_builder.rb:324:39:

    [error] Cannot pass a value of type `::RBS::AST::Members::Include` as an argument of type `::RBS::TypeName` │ ::RBS::AST::Members::Include <: ::RBS::TypeName │ ::RBS::AST::Members::Base <: ::RBS::TypeName │ ::Object <: ::RBS::TypeName │ ::BasicObject <: ::RBS::TypeName │ │ Diagnostic ID: Ruby::ArgumentTypeMismatch │ └ NoMixinFoundError.check!(member, env: env, member: member) ~~~~~~ lib/rbs/location.rb:188:12: [error] The branch is unreachable because the condition is exhaustive │ Diagnostic ID: Ruby::ElseOnExhaustiveCase │ └ raise ~~~~~ ...
  3. IDE Powered by Steep • Install Steep and VSCode extension

    • Diagnostics reporting, completion, navigations, hover
  4. IDE Features • Text editor • Syntax highlight • Folding

    • Diagnostics reporting • Hover • Navigations • Completion • Refactoring
  5. IDE Features • IDE features widely depend on the knowledge

    of the language • Syntax highlighting and folding requires the grammar • Diagnostics reporting, navigation, refactoring, ... are built on top of program analyses including type checking • (Build system and debugger depend on the language runtime)
  6. Program Analyses Levels Text analyses Syntactic analyses Semantic analyses •

    The input is sequence of characters • "Line is too long" line = io.gets() • The input is a syntax tree of Ruby program • Syntax highlighting, folding, basic linter features line = io.gets() line = io.gets() • Analyze based on the everything of program • Type checking (error detection), navigations, completion, refactoring, ... Assignment Method call Local variable ::IO String | nil (::IO#gets)
  7. IDE Development • IDE products need own program analyzers, on

    top of different APIs, in different languages • For RubyMine (Java) • For Visual Studio Code (TypeScript) • For Emacs (elisp) • For VIM (vimscript) • The analyzers provide essentially the same set of features IDE 1 UI Analyzer 1 IDE 2 UI Analyzer 2 IDE 3 UI Analyzer 3
  8. Language Server Protocol (LSP) • Supported by VSCode and many

    text editors • We can implement the server in Ruby! A Language Server is meant to provide the language-specific smarts and communicate with development tools over a protocol that enables inter-process communication. https://microsoft.github.io/language-server-protocol/ Microsoft, 2016
  9. IDE Development (revised) • Implement language specific features based on

    LSP • IDEs use the implementation VSCode Emacs VIM Analyzer LSP LSP LSP
  10. Language Servers for Ruby • Steep • Solargraph • Sorbet

    (C++) • vscode-ruby has it's own language server (TypeScript)
  11. Steep Architecture IDE Frontend Language Server Type Checker Steep LSP

    • IDE frontend starts Steep process • The communication is on pipe (stdio in Steep)
  12. You type in the editor line = io.getsa DidChangeTextDocument Notification

    { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } }
  13. You type in the editor line = io.getsa DidChangeTextDocument Notification

    { "method": "textDocument/didChange", "params": { "textDocument": { "uri": "/home/soutaro/src/foo/bar.rb", "version": 12 } , "contentChanges": [ { "range": { "start": { line: 0, character: 14 } , "end": { line: 0, character: 15 } } , "text": "a" } ] } } Steep updates the source code and starts type checking
  14. PublishDiagnostics Notification { "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics":

    [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error
  15. Editor shows an error line = io.getsa PublishDiagnostics Notification {

    "method": "textDocument/publishDiagnostics", "params": { "uri": "/home/soutaro/src/foo/bar.rb", "diagnostics": [ { "range": { "start": { "line": 0, "character": 10 } , "end": { "line": 0, "character": 15 } } , "message": "Type `::IO` does not have method `getsa`" } ] } } Steep detects a type error
  16. LSP events From IDE frontend to server From server to

    IDE frontend • textDocument/didChange • textDocument/didOpen • textDocument/didSave • textDocument/completion • textDocument/definition • textDocument/rename • workspace/symbol • textDocument/formatting • textDocument/foldingRange • textDocument/semanticTokens • textDocument/publishDiagnostics • window/showMessage • $/progress
  17. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end
  18. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop
  19. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification
  20. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code
  21. Simplified Source Code while event = client.receive() case event[:method] when

    "textDocument/didChange" checker.update_source_code(event[:params]) checker.type_check do |error| client.send_diagnostics(error) end when "textDocument/completion" ... end end Main loop When the event is didChange notification Updates the source code Type checks and reports the detected errors
  22. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get
  23. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } }
  24. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking
  25. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } Start type checking
  26. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking
  27. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...
  28. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } Start type checking Still type checking...
  29. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking...
  30. Problem • Running type checking on every single key hit

    blocks user's interaction line = io.get line = io.gets line = io.getsa { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 12 }, end: { line: 0, character: 13 } }, text: "t" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { start: { line: 0, character: 13 }, end: { line: 0, character: 14 } }, text: "s" } ] } } { method: "textDocument/didChange", params: { textDocument: { uri: "/home/soutaro/src/foo/bar.rb", version: 12 }, contentChanges: [ { range: { Start type checking Still type checking... Still type checking...
  31. Responsiveness • Returning response quickly to user's interaction • Tricks

    to make Steep analysis more responsive • Incremental type checking • Open files first • Drop unrelated code
  32. Incremental Type Checking • Type checking Ruby code affected by

    the change • Finishes much faster (when the project is big) line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class User attr_reader :name line = io.gets 1 changed file requires type checking 99 unchanged files can be skipped
  33. The fast path • When you change a Ruby file,

    it type checks only the file • Fast enough: Type checking a Ruby file takes = 500ms~1s • When you change RBS files, it runs full type checking • Slow: Type checking all code may take minutes • (Implementing incremental RBS validation improved, but still needs type checking all Ruby files)
  34. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  35. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  36. Open Files First • Open files have priority in the

    type checking queue for quick feedbacks line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get line = io.get class Conference attr_reader :talks class User attr_reader name: String attr_reader email: String attr_reader twitter: String def twitter_url: () -> String end user = User.load(payload) url = user.twitter_url "<a href='#{url}'>Twitter</a>" Ruby code opened RBS file you edit Not open files
  37. Drop unrelated code • Completion needs more responsiveness (<= 300ms)

    • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute?
  38. Drop unrelated code • Completion needs more responsiveness (<= 300ms)

    • Users wait for completion candidates when they use completion • Drop the unrelated method definitions to make the source code shorter def to_namespace namespace.append( self .name) end def alias ? kin end def absolute! self . class .new(namespace: namespace.absolute!, name: name) end def absolute? def to_namespace namespace.append(self.name) end def alias ? kin end def absolute! self.class.new(namespace: namespace.absolute!, name: name) end def absolute? namespace.absolute?
  39. Steep Architecture IDE Frontend Language Server & Type Checker Steep

    LSP • Many tricks for responsiveness in the type checker implementation • No clear boundary between language server and the type checker
  40. IDE Development with Ruby • IDE features need advanced program

    analyses like type checking • LSP allows development of IDE in any language • Steep is a static type checker implemented in Ruby with LSP support • Responsiveness is the key requirement • Shared some tricks to make Steep responsive
  41. • Language Server Protocol - https://microsoft.github.io/language-server- protocol/ • Steep -

    https://github.com/soutaro/steep • Solargraph - https://solargraph.org • Sorbet - https://sorbet.org • Visual Studio Code Ruby Extensions - https://github.com/rubyide/vscode- ruby • LanguageServer::Protocol Gem - https://github.com/mtsmfm/ language_server-protocol-ruby