[Saint P Ruby Meetup] Engine-ering Rails apps

Saint P Ruby: https://www.meetup.com/saintprug/

Rails applications tend to grow and turn into massive monoliths–that's a natural evolution of a Rails app, isn't it?

What happens next is you starting looking for an architectural solution to keep the codebase maintainable. Microservices? If you brave enough...

Rails ecosystem already has a right tool for the job: **engines**. With the help of engines, you can split your application into independent parts combined under the same _root_ application–the same way `rails` gem combines all its sub-frameworks, which are engines too, by the way.

Curious how to do that? Come to hear how we've _engine-ified_ our Rails monolith and what difficulties we faced along the way.

GitHub: https://github.com/palkan/engems

Transcript

1. Engine-ering Rails Applications Vladimir Dementyev, Saint-P Ruby 2019 Component A Component B Component C
3. We* are Doomed * Rails developers
4. Can you escape your destiny? V.Yurlov, for “The war of the worlds” by H.G.Wells
6. Components
7. Monolith
8. Hanami
9. Elixir
10. Shopify: Modular Monolith
11. SHOPIFY all others
12. Rails
13. The Book?
14. What we’ve done Or how we decided to go with engines
15. Co-living rentals service
16. Our mission • They had a property and lease managements system (admin panel) • They needed a community app for users • They wanted to keep everything in the same Rails app
17. Community • Events • Perks • Chat • Billing
18. Phase #1: Namespaces app/ controllers/ chat/… models/ chat/… jobs/ chat/…
19. Namespaces • Quick start • Fake isolation
20. Our mission evolves • There is a property and lease management application (admin panel) • We need a community app for users • We (they) want to keep everything in the same Rails app • And we (again, they) want to re-use the app’s code in the future for a side-project
21. Phase #2: Engines & Gems app/… engines/ chat/ app/ controllers/… … lib/… gems/…
22. The Modular Monolith: Rails Architecture
23. Engems Building Rails apps from engines and gems rails plugin new \ my_engine --mountable
24. Crash Course in Engines $ rails plugin new my_engine \ --mountable # or —full create create README.md create Rakefile create Gemfile … create my_engine.gemspec Engine is a gem
25. Crash Course in Engines my_engine/ app/ controllers/… … config/routes.rb lib/ my_engine/engine.rb my_engine.rb Added to paths
26. Crash Course in Engines # my_engine/lib/my_engine/engine.rb module MyEngine class Engine < ::Rails ::Engine # only in mountable engines isolate_namespace MyEngine end end
27. Crash Course in Engines # my_engine/config/routes.rb MyEngine ::Engine.routes.draw do get “/best_ruby_conference”, to: redirect("https: //spbrubyconf.ru") end
28. Crash Course in Engines # <root>/Gemfile gem "my_engine", path: "my_engine"
29. Crash Course in Engines # <root>/config/routes.rb Rails.application.routes.draw do mount MyEngine ::Engine => "/my_engine" end
30. $ rake routes Prefix Verb URI Pattern Controller#Action … my_engine /my_engine MyEngine ::Engine Routes for MyEngine ::Engine: best_ruby_conference GET /best_ruby_conference(:format) Crash Course in Engines
31. The end
34. Common Engines connect_by meet_by chat_by manage_by perks_by main app active-storage-proxy common-events
35. Dependencies • How to use non-Rubygems deps? • How to share common deps? • How to sync versions?
36. Path/Git problem # engines/my_engine/Gemfile gem "local-lib", path: " ../local-lib" gem "git-lib", github: "palkan/git-lib" # <root>/Gemfile gem "my_engine", path: "engines/my_engine" gem "local-lib", path: " ../local-lib" gem "git-lib", github: "palkan/git-lib"
37. eval_gemfile # engines/my_engine/Gemfile eval_gemfile "./Gemfile.runtime" # engines/my_engine/Gemfile.runtime gem "local-lib", path: " ../local-lib" gem "git-lib", github: "palkan/git-lib" # <root>/Gemfile gem "my_engine", path: "engines/my_engine" eval_gemfile "engines/my_engine/Gemfile.runtime"
38. eval_gemfile
39. Shared Gemfiles gemfiles/ profilers.gemfile rails.gemfile gem "stackprof", "0.2.12" gem "ruby-prof", "0.17.0" gem "memory_profiler", "0.9.12" gem "rails", “6.0.0.rc1”
40. Shared Gemfiles # engines/my_engine/Gemfile eval_gemfile " ../gemfiles/rails.runtime" eval_gemfile " ../gemfiles/profilers.runtime" # <root>/Gemfile eval_gemfile "gemfiles/rails.runtime" eval_gemfile "gemfiles/profilers.runtime"
41. Keep Versions N’Sync gem 'transdeps' A gem to find inconsistent dependency versions in component-based Ruby apps. NOT VERIFIED
42. DB vs. Engines • How to manage migrations? • How to write seeds? • How to namespace tables?
43. Migrations. Option #1 Install migrations: rails my_engine:install:migrations
44. Migrations. Option #2 “Mount“ migrations: # engines/my_engine/lib/my_engine/engine.rb initializer "my_engine.migrations" do |app| app.config.paths["db/migrate"].concat( config.paths["db/migrate"].expanded ) # For checking pending migrations ActiveRecord ::Migrator.migrations_paths += config.paths["db/migrate"].expanded.flatten end
45. Seeds # <root>/db/seed.rb ActiveRecord ::Base.transaction do ConnectBy ::Engine.load_seed PerksBy ::Engine.load_seed ChatBy ::Engine.load_seed MeetBy ::Engine.load_seed end
46. table_name_prefix class CreateConnectByInterestTags < ActiveRecord ::Migration include ConnectBy ::MigrationTablePrefix def change create_table :interest_tags do |t| t.string :name, null: false end end end
47. table_name_prefix module ConnectBy module MigrationTablePrefix def table_prefix ConnectBy.table_name_prefix end def table_name_options(config = ActiveRecord ::Base) { table_name_prefix: " #{table_prefix} #{config.table_name_prefix}", table_name_suffix: config.table_name_suffix } end end end
48. table_name_prefix # config/application.rb config.connect_by.table_name_prefix = "connect_"
49. Factories # engines/my_engine/lib/my_engine/engine.rb initializer "my_engine.factories" do |app| factories_path = root.join("spec", "factories") ActiveSupport.on_load(:factory_bot) do require "connect_by/ext/factory_bot_dsl" FactoryBot.definition_file_paths.unshift factories_path end end Custom load hook
50. Factories: Aliasing using ConnectBy ::FactoryBotDSL FactoryBot.define do # Uses ConnectBy.factory_name_prefix + "city" as the name factory :city do sequence(:name) { |n| Faker ::Address.city + " ( #{n})"} trait :private do visibility { :privately_visible } end end end Takes engine namespace into account
51. Factories: Aliasing # spec/support/factory_aliases.rb unless ConnectBy.factory_name_prefix.empty? RSpec.configure do |config| config.before(:suite) do FactoryBot.factories.map do |factory| next unless factory.name.to_s.starts_with?(ConnectBy.factory_name_prefix) FactoryBot.factories.register( factory.name.to_s.sub(/^ #{ConnectBy.factory_name_prefix}/, "").to_sym, factory ) end end end end
52. Factories: Aliasing • Use short (local) name when testing the engine • Use long (namespaced) when using in other engines
53. Shared Contexts my_engine/ lib/ my_engine/ testing/ shared_contexts/… shared_examples/… testing.rb
54. Shared Contexts # other_engine/spec/rails_helper/rb require "connect_by/testing/shared_contexts" require "connect_by/testing/shared_examples"
55. How to test engines?
56. Testing. Option #1 Using a full-featured Dummy app: spec/ dummy/ app/ controllers/ … config/ db/ test/ …
57. Testing. Option #2 gem 'combustion' A library to help you test your Rails Engines in a simple and effective manner, instead of creating a full Rails application in your spec or test folder.
58. Combustion begin Combustion.initialize! :active_record, :active_job do config.logger = Logger.new(nil) config.log_level = :fatal config.autoloader = :zeitwerk config.active_storage.service = :test config.active_job.queue_adapter = :test end rescue => e # Fail fast if application couldn't be loaded $stdout.puts "Failed to load the app: #{e.message}\n" \ " #{e.backtrace.take(5).join("\n")}" exit(1) end
59. Combustion spec/ internal/ config/ storage.yml db/ schema.rb
60. Combustion • Load only required Rails frameworks • No boilerplate, only the files you need • Automatically re-runs migrations for every test run
61. CI commands: engem: description: Run engine/gem build parameters: target: type: string steps: - run: name: "[ << parameters.target >>] bundle install" command: | .circleci/is-dirty << parameters.target >> || \ bundle exec bin/engem << parameters.target >> build Skip if no relevant changes
62. pattern = File.join( __dir __, " ../{engines,gems}/*/*.gemspec") gemspecs = Dir.glob(pattern).map do |gemspec_file| Gem ::Specification.load(gemspec_file) end names = gemspecs.each_with_object({}) do |gemspec, hash| hash[gemspec.name] = [] end # see next slide .circleci/is-dirty
63. tree = gemspecs.each_with_object(names) do |gemspec, hash| deps = Set.new(gemspec.dependencies.map(&:name)) + Set.new(gemspec.development_dependencies.map(&:name)) local_deps = deps & Set.new(names.keys) local_deps.each do |local_dep| hash[local_dep] << gemspec.name end end # invert tree to show which gem depends on what tree.each_with_object(Hash.new { |h, k| h[k] = Set.new }) do |(name, deps), index| deps.each { |dep| index[dep] << name } index end .circleci/is-dirty
64. .circleci/is-dirty def dirty_libraries changed_files = `git diff $(git merge-base origin/master HEAD) --name-only` .split("\n") raise "failed to get changed files" unless $ ?.success? changed_files.each_with_object(Set.new) do |file, changeset| if file =~ %r{^(engines|gems)/([^\/]+)} changeset << Regexp.last_match[2] end changeset end end
65. CI engines: executor: rails steps: - attach_workspace: at: . - engem: target: connect_by - engem: target: perks_by - engem: target: chat_by - engem: target: manage_by - engem: target: meet_by
66. Dev Tools • rails plugins new is too simple • Use generators: rails g engine
67. Generators # lib/generators/engine/engine_generator.rb class EngineGenerator < Rails ::Generators ::NamedBase source_root File.expand_path("templates", __dir __) def create_engine directory(".", "engines/ #{name}") end end
68. Dev Tools • rails plugins new is too simple • Use generators: rails g engine • Manage engines: bin/engem
69. bin/engem # run a specific test $ ./bin/engem connect_by rspec spec/models/connect_by/city.rb:5 # runs `bundle install`, `rubocop` and `rspec` by default $ ./bin/engem connect_by build # generate a migration $ ./bin/engem connect_by rails g migration <name> # you can run command for all engines/gems at once by using "all" name $ ./bin/engem all build
70. Engine Engine Main app
71. How to modify other engine’s entities?
72. Base & Behaviour • Base Rails classes within an engine MUST be configurable • They also MAY require to have a certain “interface”
73. Base & Behaviour module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end
74. module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end Base & Behaviour Configurable
75. Base & Behaviour # config/application.rb config.connect_by.application_controller = "MyApplicationController"
76. module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end Base & Behaviour “Interface”
77. class ApplicationController < ActionController ::API include ConnectBy ::ControllerBehaviour include JWTAuth include RavenContext if defined?(Raven) end Base & Behaviour
78. Modify Models • Prefer extensions over patches • Use load hooks (to support autoload/ reload)
79. Load Hooks # engines/connect_by/app/models/connect_by/city.rb module ConnectBy class City < ActiveRecord ::Base # ... ActiveSupport.run_load_hooks( "connect_by/city", self ) end end
80. Load Hooks # engines/meet_by/lib/meet_by/engine.rb initializer "meet_by.extensions" do ActiveSupport.on_load("connect_by/city") do include ::MeetBy ::Ext ::ConnectBy ::City end end
81. Load Hooks # engines/meet_by/app/models/ext/connect_by/city.rb module MeetBy module Ext module ConnectBy module City extend ActiveSupport ::Concern included do has_many :events, class_name: “MeetBy ::Events ::Member", inverse_of: :user, foreign_key: :user_id, dependent: :destroy end end end end end
82. How to communicate between engines?
83. Problem • Some “events” trigger actions in multiple engines • E.g., user registration triggers chat membership initialization, manager notifications, etc. • But user registration “lives” in `connect_by` and have no idea about chats, managers, whatever
84. Solution • Events • Events • Events
85. Tools • Hanami Events • Rails Events Store • Wisper (?) • dry-events (?)
87. Railsy RES
88. Railsy RES vs RES • Class-independent event types • Uses RES as interchangeable adapter • Better testing tools • Less verbose API and convention over configuration
89. Railsy RES class ProfileCompleted < Railsy ::Events ::Event # (optional) event identifier is used for transmitting events # to subscribers. # # By default, identifier is equal to `name.underscore.gsub('/', '.')`. self.identifier = "profile_completed" # Add attributes accessors attributes :user_id # Sync attributes only available for sync subscribers sync_attributes :user end
90. Railsy RES event = ProfileCompleted.new(user_id: user.id) # or with metadata event = ProfileCompleted.new( user_id: user.id, metadata: { ip: request.remote_ip } ) # then publish the event Railsy ::Events.publish(event)
91. Railsy RES initializer "my_engine.subscribe_to_events" do ActiveSupport.on_load "railsy-events" do |store| # async subscriber is invoked from background job, # enqueued after the current transaction commits store.subscribe MyEventHandler, to: ProfileCreated # anonymous handler (could only be synchronous) store.subscribe(to: ProfileCreated, sync: true) do |event| # do something end # subscribes to ProfileCreated automatically store.subscribe OnProfileCreated ::DoThat end end Convention over configuration
92. engines/ connect_by/ events/ connect_by/ users/ registered.rb chat_by/ subscribers/ connect_by/ users/ on_registered/ create_chat_account.rb Railsy RES
93. What we implement in the main app?
94. Main App • Authentication
95. Authentication module ConnectBy class ApplicationController < Engine.config.application_controller.constantize raise "Must include ConnectBy ::ControllerBehaviour" unless self < ConnectBy ::ControllerBehaviour raise "Must implement #current_user method" unless instance_methods.include?(:current_user) end end Engines don’t care about how do you obtain the current user
96. Main App • Authentication • Feature/system tests • Locales and mailers templates • Instrumentation and exception handling • Configuration
97. Gems Or stop putting everything into lib/ folder
98. Gems • Shared non-application specific code between engines • Isolated tests • Ability to share between applications in the future (e.g., GitHub Package Registry)
99. Gems gems/ common-rubocop/ common-testing/ common-graphql/ …
100. common-rubocop • Standard RuboCop configuration (based on `standard` gem) • RuboCop plugins (e.g., `rubocop- rspec`) • Custom cops
101. common-rubocop # .rubocop.yml inherit_gem: common-rubocop: config/base.yml
102. common-testing • RSpec tools bundle (`factory_bot`, `faker`, `test-prof`, formatters) • Common `spec_helper.rb` and `rails_helper.rb`
103. common-testing # rails_helper.rb require "common/testing/rails_configuration"
104. common-graphql • Base classes (object, mutation) • Additional scalar types • Batch loaders • Testing helpers
105. Why engines? And why not
106. Why engines? • Modular monolith instead of monolith monsters or micro-services hell • Code (and tests) isolation • Loose coupling (no more spaghetti logic)
107. Why engines? • Easy to upgrade frameworks (e.g., Rails) component by component • Easy to re-use logic in another app • …and we can migrate to micro-services in the future (if we’re brave enough)
108. Why not engines? • Not so good third-party gems support • Not so good support within Rails itself
109. Спасибо! Thanks!