Leveling Up Developer Tooling for the Modern Rails & Hotwire Era @ Ruby Türkiye, November 2024

The evolution of developer experience (DX) tooling has been a game-changer in how we build, debug, and enhance web applications.

This talk aims to illuminate the path toward enriching the Ruby on Rails ecosystem with advanced DX tools, focusing on the implementation of Language Server Protocols (LSP) for Stimulus and Turbo.

Drawing inspiration from the rapid advancements in JavaScript tooling, we explore the horizon of possibilities for Rails developers, including how advanced browser extensions and tools specifically designed for the Hotwire ecosystem can level up your developer experience.

Transcript

2. Marco Roth 👋 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth Full-Stack Developer & Open Source Contributor
3. Developer Tooling For The Modern Hotwire & Rails Era Marco Roth Full-Stack Developer & Open Source Contributor
4. Understand LSPs Building a simple LSP What ’ s Next for Tooling
5. CableReady Core Team Maintainer
6. I ❤ Open Source
7. Open Source Journey
8. rubyvideo.dev
12. https://rubyvideo.dev/organisations/ruby-turkiye X
13. Started to actively contribute in 2019
15. Had enough
18. X
19. Superpower ⚡
20. The future of modern Rails apps
21. I wanted to go back doing Full-Stack Rails
22. I want to help build the missing modern tooling
23. Went full-time on OSS from Jan 2022 - Aug 2023
24. To work on Hotwire and related projects
25. But, I noticed something
27. The JavaScript tooling and DX is awesome
28. We didn ’ t really have anything like it in the Ruby world
29. Or something quite as easy to setup and get started
31. I started to miss these tools while working in Ruby/Rails
32. No IDE
33. They felt clunky, heavyweight and cumbersome to use
34. Made me less productive
35. But this changed in the last few years
36. Editor Evolution
37. Language Server Protocol (LSP)
38. What is it and why do we want to use it?
39. Language Intelligence without the need for an IDE
40. Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
41. How does it work?
42. Your editor starts a background process per language
43. Communicates over STDIN/STDOUT or TCP/IP
44. JSON-RPC
45. Similar to HTTP
46. Content-Length: ...\r\n \r\n { "jsonrpc": "2.0", "id": 1, "method": "textDocument/completion", "params": { ... } }
47. Request Message Response Message Notification Message
48. Text Document Synchronization
49. textDocument/didOpen textDocument/didChange textDocument/didClose textDocument/didSave textDocument/publishDiagnostics ...
50. Stimulus LSP
51. What are we trying to solve?
52. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
53. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
54. <div |> </div>
55. <div |> data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet </div>
56. <div data-controller="|"> </div>
57. <div data-controller="user_profile"> </div>
58. <div data-controller="user-profile"> </div>
59. These mistakes happen so easily
60. I thought: "We can do better!"
61. Jun 23, 2021
66. Let ’ s take a look how it works
67. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
68. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
69. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
70. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
71. The server returns a JSON response
72. { id: 1, result: [ { "label": "TypeScript", "kind": 1, "data": 1 }, { "label": "JavaScript", "kind": 1, "data": 2 } ] }
74. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
75. Embedded Languages
76. The Stimulus grammar is not part of the HTML spec
77. So we need to extend the handling for HTML-like documents
78. Language services
79. Which language do you write it in?
82. Existing Data Providers Portability Documentation/Integration Existing Language Services
84. Data Providers
85. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
86. Let ’ s implement it
87. provideTags() { }
88. <div data-controller="hello"> ... </div>
89. provideTags() { }
90. provideTags() { return [] }
91. provideAttributes(tag: string) { }
92. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
93. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
94. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
95. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
96. Parse JavaScript
97. Parse Stimulus Controllers
98. class ControllerDefinition { readonly path: string methods: Array<string> = [] targets: Array<string> = [] classes: Array<string> = [] values: { [key: string]: Value } = {} }
99. Acorn
100. import { Parser as AcornParser } from "acorn" import { simple as walk } from "acorn-walk" class Parser { parseController(code: string, filename: string) { ... } }
101. f app/javascript/controllers/hello_controller.js
102. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
103. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
104. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
105. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
106. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
107. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
108. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
109. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
110. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
111. We do something similar for the Targets
112. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
113. <ControllerDefinition path="app/.../hello_controller.js" methods=[ "connect", "greet", "disconnect" ], targets=["name", "output"], classes=[], values={} >
114. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
115. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
116. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
117. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
118. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
120. <div data-controller="|"> </div>
121. provideValues(_tag: string, attribute: string) { }
122. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
123. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
125. https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
126. Jun 25, 2021
128. What else can we do?
129. Can ’ t AI do all of that?
130. Maybe
131. Different Use Cases
132. AI for "boilerplate" LSPs for reliability and accuracy
133. Limited Context Window Missing relation between relevant files
134. Diagnostics
140. What else can we do?
141. Code Actions to guide the user in the right direction
143. Fast forward to October 2023
144. Available Now Stimulus LSP
145. I asked the community for feedback
146. And got a lot of awesome feedback and ideas
147. Some of the naive approaches we used were starting to fall short
148. Especially the way we were parsing the JavaScript with Acorn
149. We assumed and hard-coded app/javascript/controllers/
150. The controller identifiers weren ’ t always accurate
151. import { application } from "controllers/application" import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading" eagerLoadControllersFrom("controllers", application)
153. The Language Server works
154. but it produced a lot of false positives and inaccurate results
155. Which is really annoying as a user
156. We addressed and fixed a lot of issues
157. Which technically ended up in a rewrite of the parser
158. Fast forward to May 2024
161. What ’ s next?
164. None of is this is exclusive to the LSP
165. The diagnostics could be used independently
166. Kind of like an independent tool as RuboCop
167. Coming Soon Stimulus Lint
168. Errors Diagnostics Best Practices Recommendations
169. $ stimulus-lint Stimulus Lint is inspecting 25 files ........................ 25 files inspected, 0 offenses detected, 0 offenses autocorrectable
170. <%= tag.div( data: { controller: "filter", filter_open_class: "border-white", filter_close_class: "hover:bg-gray-100 border-gray-300" } ) %> Full support for HTML+ERB and Rails-specific helpers
171. Diagnostics Code Actions Code Completion Refactoring Tools
172. Pre-Release Turbo LSP
174. Super Basic right now
175. Turbo also makes heavy use of ERB and view helpers
176. So it ’ s only really useful if we can provide intelligence for ERB
177. And that only works if we actually understand ERB
178. Stimulus LSP understands HTML in ERB documents
179. But it doesn ’ t understand Embedded Ruby that outputs HTML
182. Source: https://twitter.com/vinistock/status/1710280529889620038
183. ERB Support in Ruby LSP
184. RubyConf 2023 Hack Day
186. We quickly realized that we need something more powerful to make things work beyond the basics
187. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
188. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
189. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
190. Source: https://github.com/Shopify/ruby-lsp/pull/2235
191. Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
192. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
193. We lost the HTML-context in the process
194. <%= "Hello World" %> <div <%= attributes %>></div> <div id="<%= dom_id(post) %>"></div>
195. HTML with embedded Ruby (.html.erb, .rhtml, …)
196. ERB is versatile
197. a
198. Experiment HTML-aware ERB Parser
199. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP (maybe?) and more…
200. <article id="<%= dom_id(article) %>"></article> <input <% if true %> type="text" <% end %> /> <% @posts.each do |post| %> <h1><%= post.title %></h1> <% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>
201. Which made me think that it could be useful to have an HTML abstraction
202. <%= tag.p do %> Hello World! <% end %> <p> Hello World! </p> ==
203. == <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %> <div data-controller="hello" data-action="click->hello#greet" ></div>
204. Needs more exploration
205. This might also open to door to support Haml, Slim, Liquid, Phlex, Blade and other template languages
206. erb-languageservice html-templating-languageservice
207. If the parser can understand any ERB context and Rails-render calls it is super powerful
208. This could power advanced diagnostics
209. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
210. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
211. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
212. <div data-controller="hello"></div> <input data-hello-target="name" type="text">
213. Modern Rails apps are composed of multiple views/partials/components
214. <!-- main.html.erb --> <div data-controller="hello"> <%= render partial: "input" %> </div> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
215. But if the parser understands the Rails render logic
216. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
217. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
218. An advanced HTML-aware ERB parser could prove super helpful
219. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP
220. html_press gem phlexing gem deface gem better_html gem erb_lint gem Probably even more
221. HTML+ERB Formatter HTML+ERB Linter A new ERB Engine? Possibly even more?
222. Prism has a big effect on Ruby internals and the tooling landscape
224. An HTML-aware ERB parser could have a similar effect for HTML template tooling
225. In Progress Hotwire Browser Extension
230. Highlight Stimulus Controllers & Turbo Frames
231. Highlight Stimulus Controllers & Turbo Frames
232. Turbo Event Logging
233. Sneak Peak
237. Turbo Streams Debug Bar Pause Streams Processing Step through Streams Revert Turbo Stream Actions Show Diff of what an action changed and more…
238. Turbo Frames Reload Button R
239. Turbo Morphing Debug Turbo Morphing Visual Diffs Turbo Form Submission Debug Turbo Streams Header Debug ActionCable / Turbo Rails Cable Debug Turbo Frame Lazy Loading Debug Stimulus Controller Tree-View Panel … Ideas & Roadmap
240. It ’ s a work in progress
241. Let us know if you have any ideas, we are just getting started
242. Conclusion
243. Developer Experience and Developer Ergonomics matter.
244. This is the kind of tooling we are currently missing.
245. There is so much potential to level up even more
246. I am super excited for the future of these tools.
247. I believe that tools like these are part of the reason why a language/ framework keeps it ’ s relevancy
248. Or put differently
249. These tools are necessary and expected from a modern language and framework
250. We have been lacking behind
251. X
252. I want to keep building with Ruby and Rails
253. I want to help build the tools needed for the future of Rails applications.
254. WEEKLY
255. Hotwire Weekly • A new Hotwire-focused newsletter • Delivered Weekly • Explore what ’ s happening in the world of Hotwire • Progress and updates while we are building out hotwire.io • Been running since Oct 2023 • Over 1600+ subscribers and growing WEEKLY
257. t @Hotwire_Weekly M @Hotwire_Weekly@ruby.social Hotwire Weekly Sign up on hotwireweekly.com
258. A more productive Hotwire ecosystem.
259. A more unified Hotwire ecosystem.
260. Thank you 🙏 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth