Developer Tooling For The Modern Rails & Hotwire Era @ RailsHöck Switzerland, August 2024

Ruby on Rails Switzerland - Meetup August 2024

Organized by Renuo.ch - Hosted by Puzzle.ch

https://www.meetup.com/rubyonrails-ch/events/298503717

Transcript

1. Developer Tooling for the Modern Rails & Hotwire Era Marco Roth Pragmatic Rails - rails_api_logger Alessandro Rodi (Renuo) nochmal Matthias Viehweger (Puzzle ITC) Programatic Rails again - validation_errors Alessandro Rodi (Renuo) News Crawler via Langchain.RB and Gemini APIs Riccardo Carlesso (Google Could)
2. 👋 Marco Roth t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth Full-Stack Developer & Open Source Contributor
3. Developer Tooling For The Modern Hotwire & Rails Era Marco Roth Full-Stack Developer & Open Source Contributor
4. Understanding LSPs Building a simple LSP What ’ s Next for Tooling
5. CableReady Core Team Maintainer
6. I ❤ Open Source
8. Started to actively contribute in 2019
10. Had enough
13. X
14. Superpower ⚡
15. The future of modern Rails apps
16. I want to help build the modern tooling needed
17. Went full-time on OSS in January 2022
18. To work on Hotwire and related projects
19. But, I noticed something
20. The JavaScript tooling and DX is awesome
21. We didn ’ t really have anything like it in the Ruby world
23. I started to miss these tools while working in Ruby/Rails
24. No IDE
25. They felt clunky, heavyweight and cumbersome to use
26. Made me less productive
27. But this changed in the last few years
28. Editor Evolution
29. Language Server Protocol (LSP)
30. What is it and why do we want to use it?
31. Language Intelligence without the need for an IDE
32. Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
33. How does it work?
34. Your editor starts a one background process per language
35. Communicates over STDIN/STDOUT or TCP/IP
36. JSON-RPC
37. Similar to HTTP
38. Content-Length: ...\r\n \r\n { "jsonrpc": "2.0", "id": 1, "method": "textDocument/completion", "params": { ... } }
39. Stimulus LSP What are we trying to solve?
40. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
41. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
42. <div |> </div>
43. <div |> data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet </div>
44. <div data-controller="|"> </div>
45. <div data-controller="user-profile"> </div>
46. <div data-controller="user_profile"> </div>
47. These mistakes happen so easily
48. I thought: "We can do better!"
49. Jun 23, 2021
54. Let ’ s take a look how it works
55. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
56. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
57. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
58. { id: 1, method: "textDocument/completion", params: { "textDocument": { "uri": "file:///.../lsp-example/.../test.txt" }, "position": { "line": 1, "character": 7 }, "context": { "triggerKind": 1 } } }
59. { id: 1, result: [ { "label": "TypeScript", "kind": 1, "data": 1 }, { "label": "JavaScript", "kind": 1, "data": 2 } ] }
61. <div data-controller="hello"> <input data-hello-target="name" type="text"> <button data-action="click->hello#greet"> Greet </button> <span data-hello-target="output"></span> </div>
62. Embedded Languages
63. The Stimulus grammar is not part of the HTML spec
64. So we need to extend the handling for HTML
65. Language services
67. Data providers
68. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
69. Which language do you write it in?
72. Existing Data Providers Portability Documentation/Integration Existing Language Services
73. import { IHTMLDataProvider } from "vscode-html-languageservice" export class StimulusHTMLDataProvider implements IHTMLDataProvider { provideTags() { ... } provideAttributes(tag: string) { ... } provideValues(tag: string, attribute: string) { ... } }
74. Let ’ s implement it
75. provideTags() { }
76. provideTags() { return [] }
77. provideAttributes(tag: string) { }
78. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
79. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
80. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
81. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
82. Parse JavaScript
83. Parse Stimulus Controllers
84. class ControllerDefinition { readonly path: string methods: Array<string> = [] targets: Array<string> = [] classes: Array<string> = [] values: { [key: string]: Value } = {} }
85. Acorn
86. import { Parser as AcornParser } from "acorn" import { simple as walk } from "acorn-walk" class Parser { parseController(code: string, filename: string) { ... } }
87. f app/javascript/controllers/hello_controller.js
88. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
89. export default class extends Controller { static targets = ["name", "output"] connect() { ... } greet() { ... } disconnect() { ... } }
90. const pattern = "app/javascript/controllers/**/*_controller.js" const controllerFiles = await glob(pattern) controllerFiles.forEach(async path => { const code = await fs.readFile(path, "utf8") parser.parseController(code, path) })
91. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
92. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
93. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
94. parseController(code: string, filename: string) { const ast = this.parse(code) const controller = new ControllerDefinition(filename) walk(ast, { MethodDefinition(node) { if (node.kind === "method") { controller.methods.push(node.key.name) } }, }) }
95. We do something similar for the Targets
96. <ControllerDefinition path="app/.../hello_controller.js" methods=[ "connect", "greet", "disconnect" ], targets=["name", "output"], classes=[], values={} >
97. data-controller data-action data-[identifier]-target data-[identifier]-[class]-class data-[identifier]-[value]-value data-[identifier]-[outlet]-outlet
98. provideAttributes(_tag: string) { return [ { name: "data-controller" }, { name: "data-action" }, ] }
99. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
100. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
101. provideAttributes(_tag: string) { const targets = controllers.map(controller => `data-${controller.identifier}-target` ) return [ { name: "data-controller" }, { name: "data-action" }, ...targets, ] }
103. <div data-controller="|"> </div>
104. provideValues(_tag: string, attribute: string) { }
105. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
106. provideValues(_tag: string, attribute: string) { if (attribute === "data-controller") { return this.controllers.map(controller => controller.identifier ) } const match = attribute.match(/data-(.+)-target/) const controller = this.controllers.find(controller => controller.identifier == match[1] ) return controller.targets }
108. https://github.com/marcoroth/stimulus-lsp/commit/a8ec2cb4cd430ffe5ccc08c44249c12513ae3ce2
109. Jun 25, 2021
111. What else can we do?
112. Diagnostics
118. What else can we do?
119. Code Actions to guide the user in the right direction
121. Fast forward to October 2023
122. Available Now Stimulus LSP
123. I asked the community for feedback
124. And got a lot of awesome feedback and ideas
125. Some of the naive approaches we used were starting to fall short
126. Especially the way we were parsing the JavaScript with Acorn
127. We hard-coded app/javascript/controllers/
128. import { application } from "controllers/application" import { eagerLoadControllersFrom } from "@hotwired/stimulus-loading" eagerLoadControllersFrom("controllers", application)
130. The Language Server works
131. But produced a lot of false positives and inaccurate results
132. Which is really annoying as a user
133. We addressed and fixed a lot of issues
134. Which technically ended up in a rewrite of the parser
135. Fast forward to May 2024
138. What ’ s next?
141. None of is this is exclusive to the LSP
142. The diagnostics could be used independently
143. Kind of like RuboCop is an independent tool
144. Coming Soon Stimulus Lint
145. Errors Diagnostics Best Practices Recommendations
146. $ stimulus-lint Stimulus Lint is inspecting 25 files ........................ 25 files inspected, 0 offenses detected, 0 offenses autocorrectable
147. <%= tag.div( data: { controller: "filter", filter_open_class: "border-white", filter_close_class: "hover:bg-gray-100 border-gray-300" } ) %> Full support for HTML+ERB and Rails-specific helpers
148. Diagnostics Code Actions Code Completion Refactoring Tools
149. Pre-Release Turbo LSP
151. Turbo also makes heavy use of ERB and view helpers
152. So it ’ s only really useful if we can provide intelligence
153. And that only works if we actually understand ERB
154. Stimulus LSP understands HTML in ERB documents
155. But it doesn ’ t understand Embedded Ruby that outputs HTML
158. Source: https://twitter.com/vinistock/status/1710280529889620038
159. ERB Support in Ruby LSP
160. RubyConf 2023 Hack Day
162. We quickly realized that we need something more powerful to make things work beyond the basics
163. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
164. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
165. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
166. Source: https://github.com/Shopify/ruby-lsp/pull/2235
167. Source: https://railsatscale.com/2024-07-18-mastering-ruby-code-navigation-major-enhancements-in-ruby-lsp-2024
168. <div> <span><%= @user.firstname %></span> <b><%= @user.lastname %></b> </div>
169. We lost the HTML-context in the process
170. <%= "Hello World" %> <div <%= attributes %>></div> <div id="<%= dom_id(post) %>"></div>
171. Experiment HTML-aware ERB Parser
172. Stimulus LSP Turbo LSP Stimulus Lint Ruby LSP (maybe?)
173. <article id="<%= dom_id(article) %>"></article> <input <% if true %> type="text" <% end %> /> <% @posts.each do |post| %> <h1><%= post.title %></h1> <% end %> <%= content_tag(:p, "Hello world!") %> <%= tag.div tag.p("Hello world!") %> <%= tag.p do %> Hello world! <% end %> <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %>
174. Which made me think that it could be useful to have an HTML abstraction
175. In which the ERB view helpers also have an HTML tag node for Rails View Helpers
176. <%= tag.p do %> Hello World! <% end %> <p> Hello World! </p> ==
177. == <%= tag.div( data: { controller: "hello", action: "click->hello#greet" } ) %> <div data-controller="hello" data-action="click->hello#greet" ></div>
178. Hypertext Abstract Syntax Tree
179. { "type": "element", "tagName": "div", "properties": { "data-controller": "hello", "data-action": "click->hello#greet", }, "children": [] }
180. But we would have to adapt it to be compatible with ERB
181. Needs more exploration
182. This might also open to door to support Haml, Slim, Liquid, Phlex, Blade and other template languages
183. As long as we can transform the template syntax into HAST
184. Then we could build the whole logic for the Stimulus LSP on top of that
185. erb-languageservice html-templating-languageservice
186. If the parser can understand any ERB context and Rails-render calls it is super powerful
187. This could power advanced diagnostics
188. <div data-controller="hello"> <input data-hello-target="name" type="text"> </div>
189. <div data-controller="hello"></div> <input data-hello-target="name" type="text">
190. Modern Rails apps are composed of multiple views/partials/components
191. <!-- main.html.erb --> <div data-controller="hello"> <%= render partial: "input" %> </div> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
192. But if the parser understands the render logic
193. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
194. <!-- main.html.erb --> <div data-controller="hello"></div> <%= render partial: "input" %> <!-- _input.html.erb --> <input data-hello-target="name" type="text">
195. I think an advanced HTML-aware ERB parser could prove super helpful
196. Stimulus LSP Stimulus Lint Turbo LSP Ruby LSP
197. html_press gem phlexing gem deface gem better_html gem erb_lint gem Probably even more
198. HTML+ERB Formatter HTML+ERB Linter and more?
199. Prism has a big effect on Ruby internals and the tooling landscape
200. An HTML-aware ERB parser could have a similar effect for HTML template tooling
201. In Progress Hotwire Browser Extension
205. It ’ s a work in progress
207. Highlight Stimulus Controllers & Turbo Frames
208. Turbo Event Logging
209. Sneak Peak
212. Similarly for Stimulus Targets
214. Turbo Streams Debug Bar Pause Streams Processing Step through Streams Revert Turbo Stream Actions Show Diff of what an action changed and more…
215. Turbo Morphing Debug Turbo Morphing Visual Diffs Turbo Form Submission Debug Turbo Streams Header Debug ActionCable / Turbo Rails Cable Debug Turbo Frame Lazy Loading Debug Stimulus Controller Tree-View Panel … Ideas & Roadmap
216. Let us know if you have any ideas, we are just getting started
217. Conclusion
218. Developer Experience and Developer Ergonomics matter.
219. This is the kind of tooling we are currently missing.
220. There is so much potential to level up
221. I am super excited for the future of these tools.
222. I believe that tools like these are part of the reason why a language/framework keeps it ’ s relevancy
223. Or put differently
224. These tools are necessary and expected from a modern language and framework
225. X
226. I want to keep building with Ruby and Rails
227. I want to help build the tools needed for the future of Rails applications.
228. WEEKLY
229. Hotwire Weekly • A new Hotwire-focused newsletter • Delivered Weekly • Explore what ’ s happening in the world of Hotwire • Progress and updates while we are building out hotwire.io • Been running since Oct 2023 • Over 1400+ subscribers and growing WEEKLY
230. t @Hotwire_Weekly M @Hotwire_Weekly@ruby.social Hotwire Weekly Sign up on hotwireweekly.com
231. A more productive Hotwire ecosystem
232. A more unified Hotwire ecosystem
233. Feedback Ideas Thoughts Improvements Come fi nd me!
234. Thank you 🙏 t @marcoroth_ M @marcoroth@ruby.social g marcoroth.dev g @marcoroth