Gradually Modularizing your Monolith with Ruby Packs and Packwerk

Are you struggling with large amounts of complexity in your application? A big org with lots of cross-team dependencies? Too much coupling, causing bugs and incidents? Difficult time ramping up new engineers? Or do you just want to improve product optionality by having more reusable building blocks to create new features for your customers?

Join us to apply the packs ecosystem and packwerk to gradually modularize your application. We'll dive into decomposing your application, moving files into packs, and coming up with a technical and managerial plan to successfully roll this out to and get buy-in from the rest of your organization.

Transcript

1. Gradually Modularizing your Monolith with Ruby Packs and Packwerk Railsconf 2023 - Workshop Alex Evanczuk and Stephan Hagemann
2. 󰗞 I am Stephan Head of Product Infrastructure at Gusto https://ruby.social/@shageman stephanhagemann.com gradualmodularization.com 󰗞 I am Alex Product Infrastructure, Modularity at Gusto alexevanczuk@gmail.com
3. cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
4. Hundreds of engineers Millions of lines of code Thousands of commits per month Packwerk in production dev since 2020 Developing github.com/rubyatscale since 2022 cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Gusto Context
5. … like gems but with less overhead … and specifically built for working within large apps … and with an extensible ecosystem to gradually modularize over time cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Packages / Packs !?1!!
6. • Create a high-level structure within your application • Understand and manage app-internal dependencies • Reduce incidental complexity by removing dependency cycles • Create better-defined, domain-specific internal APIs • Reduce CI build time and cost • Untangle the ball-of-mud cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Why packages? Why gradual modularization?
7. leanpub.com/package-based-rails-applications/c/railsconf2023 cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Ruby/Rails Modularity Slack Server: tinyurl.com/rubyslack
8. Gradual Modularization: The Tools cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
9. Standing on the shoulders of giants! cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Packwerk Rubocop Sorbet
10. https://github.com/rubyatscale cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
11. Packs Rails Package Folders Packwerk Dependency checker Rubocop Packs Packs/TypedPublicApis cop Packs/DocumentedPublicApis cop Packs/ClassMethodsAsPublicApis cop Packs/RootNamespaceIsPackName cop Packwerk extensions Privacy checker Visibility checker Architecture checker Code Ownership Package-oriented ownership Use Packs CLI tools to manage and move files to packs Pack Stats Statistics reporting for packs and packwerk/rubocop usage Gradual Modularization - Tools cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
12. Where should you start? Gradual Modularization is a bit like choose your own adventure cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
13. Packs Rails Package Folders Packwerk Dependency checker Rubocop Packs Packs/TypedPublicApis cop Packs/DocumentedPublicApis cop Packs/ClassMethodsAsPublicApis cop Packs/RootNamespaceIsPackName cop Packwerk extensions Privacy checker Visibility checker Architecture checker Code Ownership Package-oriented ownership Use Packs CLI tools to manage and move files to packs Pack Stats Statistics reporting for packs and packwerk/rubocop usage Gradual Modularization - Tools cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
14. https://tinyurl.com/mod-tools-analysis ✅ ✅ cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
15. Packs Rails Package Folders Packwerk Dependency checker Rubocop Packs Packs/TypedPublicApis cop Packs/DocumentedPublicApis cop Packs/ClassMethodsAsPublicApis cop Packs/RootNamespaceIsPackName cop Packwerk extensions Privacy checker Visibility checker Architecture checker Code Ownership Package-oriented ownership Use Packs CLI tools to manage and move files to packs Pack Stats Statistics reporting for packs and packwerk/rubocop usage Gradual Modularization - Tools cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
16. Packs Rails Package Folders Packwerk Dependency checker Rubocop Packs Packs/TypedPublicApis cop Packs/DocumentedPublicApis cop Packs/ClassMethodsAsPublicApis cop Packs/RootNamespaceIsPackName cop Packwerk extensions Privacy checker Visibility checker Architecture checker Code Ownership Package-oriented ownership Use Packs CLI tools to manage and move files to packs Pack Stats Statistics reporting for packs and packwerk/rubocop usage Gradual Modularization - Tools cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
17. Let’s get practical! cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
18. Say hi to our helpers! cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
19. If you get stuck, send a message to our slack channel: #ws-gradually-modularizing-your-monolith-with-ruby-packs-and-packwerk cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
20. cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh Intro Install workshop Next Stage! (00 - 06) Understand the steps of the stage Execute all steps against mastodon Execute steps against your app Profit Good stage cadence > stage completion • Best effort • Didn’t finish? Continue during the next stage • We’ll be around after, if you want to discuss more
21. Step through script cd mastodon code ../railsconf-workshop/XX.sh # => run individual transforms manually Checkout the code for the stage cd mastodon git clean -fd && git checkout . && git checkout XX Run & read script cd mastodon code ../railsconf-workshop/XX.sh ../railsconf-workshop/XX.sh Options for getting through the workshop
22. Goal of this workshop: Let’s save mastodon some money on CI! cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
23. 00 Getting set up cd YOUR_WORKSPACE #ideally, next to your app git clone --depth=1 https://github.com/gradual-systems/mastodon.git git clone https://github.com/gradual-systems/railsconf-workshop.git cd mastodon; ../railsconf-workshop/00.sh
24. Stage 00 - part 1 echo " gem 'packs-rails' gem 'use_packs', group: %w(development test) gem 'visualize_packwerk', group: %w(development test) gem 'packwerk', github: 'gradual-systems/packwerk', branch: 'main', group: %w(development test) " >> Gemfile bundle install
25. Stage 00 - part 2 bundle binstubs bundler --force bundle binstub use_packs packwerk echo "pack_paths: - packs/* - . " > packs.yml echo "--require packs/rails/rspec" >> .rspec bin/packwerk init
26. Packwerk might crash on your app In packwerk.yml exclude: - {bin,node_modules,script,tmp,vendor}/**/* - <relative path to offending file> ⚠
27. 01 Extracting an admin pack
28. Admin interfaces… everyone’s got one?
29. How do you identify your admin files? #1 Learn your domain, speak to the experts, have a meeting, do some analysis, make some proposals, draft a solution, make sure to get it right, … #2 find . | grep admin
30. Stage 01 echo "Create admin pack" bin/packs create packs/admin bin/packs move packs/admin app/*/admin bin/packs move . packs/admin/app/javascript bin/packwerk update which dot && bin/packs visualize
31. Why not move JS? Because there is usually other stuff going on that is hard to see with ruby tooling Because JS is often “it’s own application” ⚠
32. Moving files typically just works… Except for when it doesn’t… #1 Explicit, manual requires #2 Autoloading failing to find stuff (metaprogramming) ⚠
33. bin/packs visualize #(for step 01)
34. 02 Improve admin pack dependencies
35. We can learn a lot from packwerk violations… Do we see if any other code should go in the admin pack?
36. Stage 02 echo "Fix some admin package dependencies" bin/packs move packs/admin app/controllers/api/*/admin bin/packwerk update which dot && bin/packs visualize
37. bin/packs visualize #(for step 02)
38. 03 Create the “messy middle” pack
39. What do we do with the fact that there are SO MANY violations from the admin pack and (relatively) few from the root pack??? Moving the few violating files into their own pack would tell us more about their entanglement
40. Stage 03 echo "Create pack for 'messy middle'" bin/packs create packs/messy_middle bin/packs move packs/messy_middle \ `../railsconf-workshop/03_find_files_to_root_violations.rb` bin/packwerk update which dot && bin/packs visualize
41. bin/packs visualize #(for step 03)
42. 04 Analyze and empty out the “messy middle”
43. Get yourself a clean dependency graph with this one weird trick…
44. We can move the messy middle into the root package!
45. Stage 04 echo "Merge 'messy middle' with root pack" bin/packs move . packs/messy_middle rm -rf packs/messy_middle bin/packwerk update which dot && bin/packs visualize
46. bin/packs visualize #(for step 04)
47. 05 Create user facing app pack
48. Alex and I have a long standing feud with code in the root pack…
49. If packages allow us to separate concepts within our domain, any domain code that is in the root package is entangled with the code that configures the app and makes it runnable… We like the root pack to only do that: application configuration and setup
50. What would we name the pack we extract out of the root pack? What is currently left there? The app the user sees! Let’s make that explicit
51. Stage 05 echo "Create user facing app pack" bin/packs create packs/user_facing_app bin/packs move packs/user_facing_app app bin/packs move . packs/user_facing_app/app/javascript bin/packwerk update which dot && bin/packs visualize
52. bin/packs visualize #(for step 05)
53. 06 Accept packwerk dependencies
54. These dependencies look pretty good! Let’s accept them.
55. Stage 06 echo "Accept packwerk dependencies" bin/packs add_dependency . packs/user_facing_app bin/packs add_dependency packs/admin packs/user_facing_app bin/packwerk update which dot && bin/packs visualize
56. bin/packs visualize #(for step 06)
57. There is a step 07…
58. Not yet… but we’re close 3559 tests 425 tests 158 tests Did we save mastodon some money on CI?
59. What about… Routes? Initializers? Other metaprogramming? Does packwerk see all dependencies? 😬 ❌ Dynamic method parameters ❌ Metaprogramming ❌ Monkey-patching ⚠
60. Did we save mastodon some money on CI? Not yet… but we’re close 3559 tests 425 tests 158 tests 425 + 158 = 583 tests to validate changes 158 tests to validate package 3559 + 425 + 158 = 4142 tests to validate changes At Gusto we save about 20% on CI doing this
61. 😬 There is no open-source library for this yet Care to contribute? → github.com/rubyatscale 😏 For now, this gist will do the basics 😁 https://tinyurl.com/conditionalpackbuilds How do you benefit from this?
62. Moving files around (even loads of them) isn’t scary! Moving files around leads to interesting insights! Packages give helpful context to our code! Packages can save money on CI! Recap and Summary
63. Packs Rails Package Folders Packwerk Dependency checker Rubocop Packs Packs/TypedPublicApis cop Packs/DocumentedPublicApis cop Packs/ClassMethodsAsPublicApis cop Packs/RootNamespaceIsPackName cop Packwerk extensions Privacy checker Visibility checker Architecture checker Code Ownership Package-oriented ownership Use Packs CLI tools to manage and move files to packs Pack Stats Statistics reporting for packs and packwerk/rubocop usage Gradual Modularization - Tools
64. Packs Rails Package Folders Packwerk Dependency checker Rubocop Packs Packs/TypedPublicApis cop Packs/DocumentedPublicApis cop Packs/ClassMethodsAsPublicApis cop Packs/RootNamespaceIsPackName cop Packwerk extensions Privacy checker Visibility checker Architecture checker Code Ownership Package-oriented ownership Use Packs CLI tools to manage and move files to packs Pack Stats Statistics reporting for packs and packwerk/rubocop usage Gradual Modularization - Tools
65. We hope this got you excited to start your modularization journey!
66. Thank you! 󰗞 I am Stephan 󰗞 I am Alex
67. Wait… we have more time?
68. Your own app! Step 07