RailsConf 2018 | The Future of Rails 6: Scalable by Default

We've all heard the phrase "Rails doesn't scale". Long running test suites and no standard for implementing multiple databases makes it hard scale monolithic Rails applications. Rails 6 will start making Rails scalable by default with parallel testing and improved support for using multiple databases. You'll no longer be forced to reinvent the wheel and create your own solution to these problems. In this talk we'll take a look why these improvements are important, how they work, and ways in which small ideas can quickly snowball into major changes. This is just the beginning of Rails 6.

Transcript

1. The Future of Rails 6 Scalable by Default
2. Eileen M. Uchitelle eileencodes.com @eileencodes
3. a
4. Core Team
5. “Rails doesn’t scale!”
7. Rails doesn’t scale
8. Rails doesn’t scale easily
9. Rails doesn’t scale easily (yet)
10. What does “scalable” mean?
11. Your test suite shouldn’t block development
12. Your application can handle data & traffic
13. The Future of Rails 6 Scalable by Default
14. Parallel Testing Speeding up your test suite
15. a
16. a
17. Forking - vs - Threads
18. Forking Processes with dRB
19. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end
20. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
21. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
22. # activesupport/lib/active_support/testing/parallelization module ActiveSupport::Testing::Parallelization […] def initialize(queue_size) @queue_size = queue_size @queue = Server.new @pool = [] @url = DRb.start_service("drbunix:", @queue).uri end […] end
23. # activesupport/lib/active_support/testing/parallelization module ActiveSupport::Testing::Parallelization […] def initialize(queue_size) @queue_size = queue_size @queue = Server.new @pool = [] @url = DRb.start_service("drbunix:", @queue).uri end […] end
24. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
25. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
26. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
27. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
28. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
29. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
30. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
31. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
32. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
33. module ActiveSupport::Testing::Parallelization […] def start […] end def shutdown @queue_size.times { @queue << nil } @pool.each { |pid| Process.waitpid(pid) } end […] end
34. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) end
35. test-database-0 test-database-1
36. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do DRb.stop_service 
 after_fork(worker) 
 queue = DRbObject.new_with_uri(@url) 
 while job = queue.pop klass = job[0] method = job[1] […]
37. module ActiveSupport::Testing::Parallelization […] def start @pool = @queue_size.times.map do |worker| fork do […]
 while job = queue.pop klass = job[0] method = job[1] reporter = job[2] result = Minitest.run_one_method(klass, method) queue.record(reporter, result) end run_cleanup(worker) […]
38. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2) parallelize_setup do |worker| ActiveRecord::Tasks::DatabaseTasks.create(:animals) ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.load_schema end parallelize_teardown do |worker| ActiveRecord::Tasks::DatabaseTasks.drop(:animals) end end
39. Threaded testing with Minitest
40. # test/test_helper.rb module ActiveSupport::TestCase parallelize(workers: 2, with: :threads) end
41. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
43. Connection ID: #03938752 Connection ID: #83321340
44. Connection ID: #83321340 Connection ID: #83321340
45. This is so weird. We seem to have an isolation problem…
47. Rails opens a connection
48. Puma opens a second connection Rails opens a connection
49. Puma opens a second connection Rails opens a connection
50. Test thread connection number 1 Test thread connection number 2
51. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
52. $ rails test
53. $ PARALLEL_WORKERS=14 rails test
54. def parallelize(workers: 2, with: :processes) workers = ENV["PARALLEL_WORKERS"].to_i if ENV["PARALLEL_WORKERS"] return if workers <= 1 executor = case with when :processes Testing::Parallelization.new(workers) when :threads Minitest::Parallel::Executor.new(workers) else raise ArgumentError, "#{with} is 
 not a supported parallelization executor." end self.lock_threads = false if defined?(self.lock_threads) && 
 with == :threads Minitest.parallel_executor = executor parallelize_me! end
55. Parallel Testing PR github.com/rails/rails/pull/31900
56. Your test suite shouldn’t block development
57. Your application can handle data & traffic
60. DB 1 DB 2 DB 3
61. Multiple Databases Making multi-db behavior easy
62. Primary
63. DB 1 DB 2 DB 3
64. Table id: integer name: string Table id: integer name: string Table id: integer name: string Table id: integer name: string Primary Animals
65. X Best practices are undocumented The State of Multi-db in Rails 5
66. # database.yml production: <<: *default database: db_production production_animals: <<: *default database: db_animals_production
67. # database.yml production: <<: *default database: db_production production_animals: <<: *default database: db_animals_production
68. # 3-tier database.yml production: primary: <<: *default database: db_production animals: <<: *default database: db_animals_production
69. db/animals_migrate ↳ 20180206210429_create_dogs.rb ↳ 20180206210437_create_cats.rb
70. ActiveRecord::Migrator.migrations_paths = "db/animals_migrate" ActiveRecord::Base.migrate
71. X Best practices are undocumented X Migrations don’t work The State of Multi-db in Rails 5
72. # app/models/animals_base.rb class AnimalsBase < ActiveRecord::Base self.abstract_class = true establish_connection :animals end
73. # app/models/dog.rb class Dog < AnimalsBase […] end # app/models/cat.rb class Cat < AnimalsBase […] end
74. X Best practices are undocumented X Migrations don’t work X Database tasks don’t exist The State of Multi-db in Rails 5
75. ActiveRecord::AdapterNotSpecified: 'production' database is not configured.
76. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning
77. # activerecord/lib/active_record/railtie.rb initializer "active_record.initialize_database" do ActiveSupport.on_load(:active_record) do self.configurations = Rails.application.config.database_configuration begin establish_connection rescue ActiveRecord::NoDatabaseError warn <<-end_warning
78. X Best practices are undocumented X Migrations don’t work X Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
79. Migrations Connections don’t work Rake tasks
81. Fixing migrations for multiple databases
82. path = Rails.root.to_s + "/db/animals_migrate" desc "Migrate the cluster db" task migrate: :environment do ActiveRecord::Migrator.migrations_paths = path ActiveRecord::Base.establish_connection(:animals) ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["db:schema:dump"].invoke end
83. # 3-tier database.yml production: primary: <<: *default database: db_production animals: <<: *default database: db_animals_production migrations_paths: db/animals_migrate
84. >> Dog.connection.migrations_paths => "db/animals_migrate"
85. Migrations Paths Refactoring PR github.com/rails/rails/pull/31727
86. X Best practices are undocumented ✔ Migrations don’t work X Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
87. Migrations Connections don’t work Rake tasks
88. Improving the database tasks for multi-dbs
89. rails db:drop rails db:create rails db:migrate
90. >> ActiveRecord::Base.configurations["production"] >> { "adapter"=>"mysql2", "database"=>"single_database_production" }
91. >> ActiveRecord::Base.configurations["production"] >> { "primary"=>{"adapter"=>"mysql2", "database"=>"multiple_databases_production"}, "animals"=>{"adapter"=>"mysql2", “database"=>"multiple_databases_production"} }
92. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
93. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
94. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
95. module ActiveRecord module DatabaseConfigurations class DatabaseConfig attr_reader :env_name, :spec_name, :config def initialize(env_name, spec_name, config) @env_name = env_name @spec_name = spec_name @config = config end end end end
96. >> db_config = DatabaseConfig.new("production", "animals", {"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }) >> #<ActiveRecord::DatabaseConfigurations::DatabaseCon fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>
97. >> db_config.env_name "production"
98. >> db_config.env_name "production" >> db_config.spec_name "animals"
99. >> db_config.env_name "production" >> db_config.spec_name "animals" >> db_config.config {…"database"=>"multiple_databases_producti on_animals"}
100. def self.db_configs(configs = configurations) configs.each_pair.flat_map do |env_name, config| walk_configs(env_name, "primary", config) end end
101. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
102. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
103. def self.configs_for(env, configs, &blk) env_with_configs = db_configs(configs).select do |db_config| db_config.env_name == env end if block_given? env_with_configs.each do |ewc| yield ewc.spec_name, ewc.config end else env_with_configs end end
104. >> DatabaseConfigurations.configs_for("production") >>[#<ActiveRecord::DatabaseConfigurations::Database Config:0x00007f994571edb0 @env_name="production", @spec_name="primary", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production"}>, #<ActiveRecord::DatabaseConfigurations::DatabaseCon fig:0x00007f994571edb0 @env_name="production", @spec_name="animals", @config={"adapter"=>"mysql2", "database"=>"multiple_databases_production_animals" }>]
105. task migrate: :load_config do ActiveRecord::Tasks::DatabaseTasks.migrate db_namespace["_dump"].invoke end
106. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config| ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
107. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config| ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
108. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config| ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
109. task migrate: :load_config do ActiveRecord::DatabaseConfigurations.configs_for( Rails.env ) do |_, config| ActiveRecord::Base.establish_connection(config) ActiveRecord::Tasks::DatabaseTasks.migrate end db_namespace["_dump"].invoke end
110. rails db:drop rails db:create rails db:migrate
111. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary rails db:migrate:animals
112. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
113. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
114. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
115. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
116. namespace :migrate ActiveRecord::Tasks::DatabaseTasks.for_each do |spec_name| desc "Migrate #{spec_name} database" task spec_name => :load_config do db_config = ActiveRecord::DatabaseConfigurations .config_for_env_and_spec(Rails.env, spec_name) ActiveRecord::Base.establish_connection( db_config.config ) ActiveRecord::Tasks::DatabaseTasks.migrate end end end end
117. rails db:drop:primary rails db:drop:animals rails db:create:primary rails db:create:animals rails db:migrate:primary rails db:migrate:animals
118. Database task improvements github.com/rails/rails/pull/32274 github.com/rails/rails/compare/ ab43b5405ea2...5ddcda6d5f56
119. X Best practices are undocumented ✔ Migrations don’t work ✔ Database tasks don’t exist X Default connection was broken The State of Multi-db in Rails 5
120. Connections don’t work Migrations Rake tasks
122. Thanks @tenderlove!
123. “Are you concerned that Rails will become bloated?”
124. What would Rails look like if we’d upstreamed these features 5 years ago?
125. a +
126. ' ( ) * + ,
128. Maybe Rails doesn’t scale…?
129. Your application isn’t special.
130. Forces you to write generic code
131. Train your future workforce
132. Give back to the community
133. ' ( ) * + ,
134. a♥
135. Scaling should make us happy
136. Rails isn’t dying, it’s maturing
140. “Rails does scale!”
141. The Future of Rails 6 Scalable by Default
142. Eileen M. Uchitelle eileencodes.com @eileencodes