[RubyConf 2022] Weaving & seaming mocks

Video: https://www.youtube.com/watch?v=ZFSS8_pFemc

https://www.rubyconfmini.com/program#Weaving-and-seaming-mocks

To mock or not mock is an important question, but let's leave it apart and admit that we, Rubyists, use mocks in our tests.

Mocking is a powerful technique, but even when used responsibly, it could lead to false positives in our tests (thus, bugs leaking to production): fake objects could diverge from their real counterparts.

In this talk, I'd like to discuss various approaches to keeping mocks in line with the actual implementation and present a brand new idea based on mock fixtures and contracts.

Transcript

2. 2 Ruby Devs ! Tests
3. How do you write tests?
4. palkan_tula palkan 4 Martin Fowler, Mocks Aren't Stubs
5. palkan_tula palkan 5 Classical def search(q) query = Query.build(q) User.find_by_sql(query.to_sql) end
6. palkan_tula palkan 5 Classical def search(q) query = Query.build(q) User.find_by_sql(query.to_sql) end user = create(:user, name: "Vova") expect(search("name=Vova")).to eq([user])
7. palkan_tula palkan Mockist 5 def search(q) query = Query.build(q) User.find_by_sql(query.to_sql) end expect(Query) .to receive(:build).with("name=x") .and_return(double(to_sql: :cond)) expect(User).to receive(:find_by_sql).with(:cond) .and_return(:users) expected(search("user=x")).to eq :users
8. palkan_tula palkan To mock or not to mock? Emily Samp @ RubyConf, 2021 6
9. palkan_tula palkan Classical 7
10. palkan_tula palkan 8 Mockist
11. palkan_tula palkan 9 Mocklassicism
12. palkan_tula palkan False positive 10 def search(q) query = Query.build(q) User.find_by_sql(query.to_sql) end expect(Query) .to receive(:build).with("name=x") .and_return(double(to_sql: :cond)) expect(User).to receive(:find_by_sql).with(:cond) .and_return(:users) expected(search("user=x")).to eq :users module Query - def self.build(query_str) - query_hash = Hash[query.split("=")] + def self.build(query_hash) # query has to sql end
13. palkan_tula palkan How to avoid false positives? 11
14. palkan_tula palkan 11 How to put seams? ?
15. palkan_tula palkan github.com/palkan 12
19. Keeping mocks in line with real objects 16
20. palkan_tula palkan 17 Case study: anyway_config class RubyConfConfig < Anyway::Config attr_config :city, :date coerce_types date: :date end github.com/palkan/anyway_config
21. 18 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby -r rubyconf_conf.rb -e "pp RubyConfConfig.new" #<RubyConfConfig config_name="rubyconf" env_prefix="RUBYCONF" values: city => "Providence" (type=env key=RUBYCONF_CITY), date => #<Date: 2022 11 15> (type=env key=RUBYCONF_DATE)> #<RubyConfConfig config_name="rubyconf" env_prefix="RUBYCONF" values: city => "Providence" (type=env key=RUBYCONF_CITY), date => #<Date: 2022 11 15> (type=env key=RUBYCONF_DATE)>
22. palkan_tula palkan 19 ENV parser class Anyway::Env def fetch(prefix) # Scans ENV and parses matching values some_hash end def fetch_with_trace(prefix) [fetch(prefix), traces[prefix]] end end
23. palkan_tula palkan 20 ENV loader class Anyway::Loaders::Env def call(env_prefix:, **_options) env = Anyway::Env.new env.fetch_with_trace(env_prefix) .then do |(data, trace)| Tracing.current_trace&.merge!(trace) data end end end •
24. 21 $ bundle exec rspec --tags env Run options: include {:env=>true} Randomized with seed 26894 ........ Finished in 0.00641 seconds (files took 0.27842 seconds to load) 8 examples, 0 failures Coverage is at 100.0%. Coverage report sent to Coveralls.
25. palkan_tula palkan 22 ENV loader describe Anyway::Loaders::Env do subject { described_class.call(env_prefix: "TESTO") } it "loads data from env" do with_env( "TESTO_A" => "x", "TESTO_DATA__KEY" => "value" ) do expect(subject) .to eq({"a" => "x", "data" => { "key" => "value"}}) end end end Aren't we testing ENV parser here, not loader?
26. palkan_tula palkan describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data, nil]) end it "loads data from Anyway::Env" do expect(subject) .to eq({"a" => "x", "data" => {"key" => "value"}}) end end describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data, nil]) end it "loads data from Anyway::Env" do expect(subject) .to eq({"a" => "x", "data" => {"key" => "value"}}) end end 23 Double-factoring
27. palkan_tula palkan describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data end it "loads data from Anyway::Env" do expect(subject) describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data end it "loads data from Anyway::Env" do expect(subject) 23 Double-factoring
28. Case #1. Undefined method 24
29. palkan_tula palkan 25 Refactor class Anyway::Env - def fetch(prefix) + def fetch(prefix, include_trace = false) @@ ... - def fetch_with_trace(prefix) - [fetch(prefix), traces[prefix]] - end end
30. palkan_tula palkan 26 describe Anyway::Loaders::Env do let(:env) { double("env") } before do allow(::Anyway::Env) .to receive(:new).and_return(env) allow(env) .to receive(:fetch_with_trace) .and_return([data, nil]) end # ... end Refactor
31. 27 $ bundle exec rspec --tags env Run options: include {:env=>true} Randomized with seed 26894 ........ Finished in 0.00594 seconds (files took 0.21516 seconds to load) 8 examples, 0 failures Coverage is at 100.0%. Coverage report sent to Coveralls.
32. 28 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby -r rubyconf_conf.rb -e "pp RubyConfConfig.new" #<RubyConfConfig config_name="rubyconf" env_prefix="RUBYCONF" values: city => "Providence" (type=env key=RUBYCONF_CITY), date => #<Date: 2022 11 15> (type=env key=RUBYCONF_DATE)> anyway_config/lib/anyway/loaders/env.rb:11:in `call': undefined method `fetch_with_trace' for #<Anyway::Env:0x00000001008948f0 @type_cast=Anyway::NoCast, @data={}, @traces={}> (NoMethodError) env.fetch_with_trace(env_prefix).then do |(conf, trace)| ^^^^^^^^^^^^^^^^^
33. palkan_tula palkan Double trouble — Tests are green ✅ — Coverage 100% # — Code doesn't work ❌ 29
35. palkan_tula palkan 31 github.com/rspec/rspec-mocks/issues/227
36. palkan_tula palkan 32 github.com/xaviershay/rspec-fire
37. palkan_tula palkan 33 describe Anyway::Loaders::Env do - let(:env) { double("env") } - let(:env) { instance_double("Anyway::Env") } Refactor
38. 34 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads data from Anyway::Env Failure/Error: allow(env_double).to receive(:fetch_with_trace).and_return([env, nil]) the Anyway::Env class does not implement the instance method: fetch_with_trace # ./spec/loaders/env_spec.rb:22:in `block (2 levels) in <compiled>'
39. palkan_tula palkan 35 double No strings attached
40. palkan_tula palkan verified double 35 double No strings attached Method existence
41. Case #2. Incorrect method signature 36
42. palkan_tula palkan 37 Refactor class Anyway::Env - def fetch(prefix, include_trace = false) + def fetch(prefix, include_trace: false) @@ ...
43. palkan_tula palkan 37 Refactor class Anyway::Env - def fetch(prefix, include_trace = false) + def fetch(prefix, include_trace: false) @@ ... describe Anyway::Loaders::Env do let(:env) { instance_double("env") } before do expect(env) .to receive(:fetch) .with("TESTO", true) .and_return([data, nil]) end # ... end
44. 38 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads data from Anyway::Env Failure/Error: env.fetch(env_prefix, true) ArgumentError: Wrong number of arguments. Expected 1, got 2. # ./lib/anyway/loaders/env.rb:11:in `call' # ./lib/anyway/loaders/base.rb:10:in `call' # ./spec/loaders/env_spec.rb:18:in
45. palkan_tula palkan 39 class MethodSignatureVerifier def initialize(signature, args=[]) # ... end def valid? missing_kw_args.empty? && invalid_kw_args.empty? && valid_non_kw_args? && arbitrary_kw_args? && unlimited_args? end end rspec/support/method_signature_verifier.rb
46. palkan_tula palkan 39 rspec/support/method_signature_verifier.rb class MethodSignature def initialize(method) # ... classify_parameters end def classify_parameters @method.parameters.each do |(type, name)| # ... end end end
47. palkan_tula palkan class MethodSignature def initialize(method) # ... classify_parameters end def classify_parameters @method.parameters.each do |(type, name)| # ... end end end 40 rspec/support/method_signature_verifier.rb
48. palkan_tula palkan 41 Method#parameters method_obj = Anyway::Env.instance_method(:fetch) method_obj.parameters #=> [ [:req, :prefix], [:key, :include_trace] ]
49. palkan_tula palkan 42 double verified double No strings attached Method existence Method parameters Method signature?
50. palkan_tula palkan Method signature — Parameters shape — Argument types — Return value type 43
51. palkan_tula palkan 44 Refactor class Anyway::Env + Parsed = Struct.new(:data, :trace) def fetch(prefix, include_trace: false) @@ ... - [data, trace] + Parsed.new(data, trace) end
52. palkan_tula palkan 44 Refactor class Anyway::Env + Parsed = Struct.new(:data, :trace) def fetch(prefix, include_trace: false) @@ ... - [data, trace] + Parsed.new(data, trace) end describe Anyway::Loaders::Env do let(:env) { instance_double("env") } before do expect(env) .to receive(:fetch) .with("TESTO", include_trace: true) .and_return([data, nil]) end # ... end
53. 45 $ bundle exec rspec --tags env Run options: include {:env=>true} Randomized with seed 43108 ........ Finished in 0.0234 seconds (files took 0.80197 seconds to load) 7 examples, 0 failures
54. 46 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby -r rubyconf_conf.rb -e "pp RubyConfConfig.new" #<RubyConfConfig config_name="rubyconf" env_prefix="RUBYCONF" values: city => "Providence" (type=env key=RUBYCONF_CITY), date => #<Date: 2022 11 15> (type=env key=RUBYCONF_DATE)> anyway_config/lib/anyway/tracing.rb:59:in `merge!': undefined method `trace?' for nil:NilClass (NoMethodError) from anyway_config/lib/anyway/loaders/env.rb:12:in `block in call' from <internal:kernel>:124:in `then' from anyway_config/lib/anyway/loaders/env.rb:11:in `call'
55. palkan_tula palkan “Test doubles are sweet for isolating your unit tests, but we lost something in the translation from typed languages. Ruby doesn't have a compiler that can verify the contracts being mocked out are indeed legit.” –rspec-fire's Readme 47
56. Mocks vs. types 48
57. palkan_tula palkan .rbs 49 double verified double No strings attached Method existence Method parameters
58. palkan_tula palkan typed double 49 double verified double No strings attached Method existence Method parameters Method signature
59. palkan_tula palkan typed_double — Intercept mocked calls — Type check them—that's it! 50
60. palkan_tula palkan 51 Interception RSpec::Mocks::VerifyingMethodDouble.prepend( Module.new do def proxy_method_invoked(obj, *args, &block) super.tap { TypedDouble.typecheck!(obj, *args) } end end )
61. palkan_tula palkan 52 RBS::Test
62. palkan_tula palkan 53 anyway_config.rbs module Anyway class Env class Parsed attr_reader data: Hash attr_reader trace: Tracing::Trace? end def fetch: (String prefix, ?include_trace: bool) -> Parsed end end
63. 54 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads data from Anyway::Env Failure/Error: raise RBS::Test::Tester::TypeError.new(errors) unless errors.empty? RBS::Test::Tester::TypeError: TypeError: [Anyway::Env#fetch] ReturnTypeError: expected `::Anyway::Env::Parsed` but returns `[{"a"=>"x", "data"=>{"key"=>"value"}}, nil]`
64. palkan_tula palkan What if... we don't have types % 55
65. On-the-fly type signatures generation 56
66. palkan_tula palkan Type generators — rbs prototype / tapioca — TypeProfiler — Tracing → signatures & 57
67. palkan_tula palkan On-the-fly types — Collect method calls made on real objects 58
68. palkan_tula palkan 59 Tracking calls TracePoint.trace(:call, :return) do |tp| next unless trackable?(tp.defined_class, tp.method_id) target, mid = tp.defined_class, tp.method_id if tp.event == :call method = tp.self.method(mid) args = [] kwargs = {} method.parameters.each do |(type, name)| val = tp.binding.local_variable_get(name) # ... end store[target][mid] << CallTrace.new(arguments: args, kwargs:) elsif tp.event == :return store[target][mid].last.return_value = tp.return_value end end
69. palkan_tula palkan 60 Tracking calls if tp.event == :call method = tp.self.method(mid) args = [] kwargs = {} method.parameters.each do |(type, name)| val = tp.binding.local_variable_get(name) # ... end
70. palkan_tula palkan On-the-fly types — Collect method calls made on real objects — Generate types from real call traces 61
71. palkan_tula palkan 62 SignatureGenerator class SignatureGenerator def to_rbs = [header, method_sigs, footer].join("\n") def args_sig(args) args.transpose.map do |arg_values| arg_values.map(&:class).uniq.map do "::#{_1.name}" end end.join(", ") end end
72. palkan_tula palkan 63 env.rbs module Anyway class Env def initialize: (?type_cast: (::Module)) -> (void) def fetch: ( ::String, ?include_trace: (::FalseClass | ::TrueClass) ) -> ::Anyway::Env::Parsed end end
73. palkan_tula palkan On-the-fly types — Collect method calls made on real objects — Generate types from real call traces — Identify tracing targets (mocked classes) & 64
74. Mock fixtures 65
75. palkan_tula palkan Fixturama evilmartians.com/chronicles/a-fixture-based-approach-to-interface-testing-in-rails 66
76. palkan_tula palkan 67 Fixturama github.com/nepalez/fixturama # fixtures/stubs/notifier.yml --- - class: Notifier chain: - create arguments: - :profileDeleted - <%= profile_id %> actions: - return: true - raise: ActiveRecord::RecordNotFound arguments: - "Profile with id: 1 not found" # for error message
77. palkan_tula palkan YAML fixtures — YAML != Ruby, hard to mock with non- primitive types — Existing mocks are not re-usable—a lot of refactoring 68
78. palkan_tula palkan 69 Mock context mock_context "Anyway::Env" do before do env_double = instance_double("Anyway::Env") allow(Anyway::Env).to receive(:new).and_return(env_double) data = {"a" => "x", "data" => {"key" => "value"}} allow(env_double) .to receive(:fetch).with("TESTO", any_args) .and_return([data, nil]) end end
79. palkan_tula palkan Mock context — Just a shared context — Evaluated within a "scratch" example group on initialization to collect information about stubbed classes and methods 70
80. palkan_tula palkan 71 Mock context def evaluate_context!(context_id, tracking) Class.new(RSpec::Core::ExampleGroup) do include_context(context_id) specify("true") { expect(true).to be(true) } after do RSpec::Mocks.space.proxies.values.each do tracking.register_from_proxy(_1) end end end.run end
81. palkan_tula palkan 72 Refactor describe Anyway::Loaders::Env do - let(:env) { instance_double(Anyway::Env) } + include_mock_context "Anyway::Env" @@ ...
82. palkan_tula palkan 73 RSpec post-check config.after(:suite) do TypedDouble.infer_types_from_calls!( CallsTracer.calls ) passed = MocksTracer.calls.all do TypedDouble.typecheck(_1) end unless passed exit(RSpec.configuration.failure_exit_code) end end
83. palkan_tula palkan 74 Mocked objects finder Mocked calls collector Real calls collector Type checker Types generator after(:suite) before(:suite) run time
84. Case #3. Non-matching behaviour 75
85. palkan_tula palkan 76 Refactor class Anyway::Env def fetch(prefix, **) + return if prefix.empty? @@ ...
86. palkan_tula palkan 76 Refactor class Anyway::Env def fetch(prefix, **) + return if prefix.empty? @@ ... mock_context "Anyway::Env" do before do # ... allow(env_double) .to receive(:fetch) .with("", any_args) .and_return( Anyway::Env::Parsed.new({}, nil)) end end
87. palkan_tula palkan 77 env.rbs module Anyway class Env def fetch: ( ::String, ?include_trace: (::FalseClass | ::TrueClass) ) -> ::Anyway::Env::Parsed? end end We cannot specify which string causes the return value to be nil
88. palkan_tula palkan Double contract — Stubs represent contracts — Tests using real objects MUST verify contracts (unit, end-to-end) 78
89. palkan_tula palkan 79 github.com/psyho/bogus github.com/robwold/compact
90. palkan_tula palkan Stubs → Contracts — Collect method stub expected arguments — Check that a real call with the matching arguments was made and its return type matches the mocked one 80
91. palkan_tula palkan 81 Stubs → Contracts allow(env_double).to receive(:fetch) .with("", any_args) .and_return(Anyway::Env::Parsed.new({})) Anyway::Env#fetch: ("", _) -> Anyway::Env::Parsed verification pattern
92. $ bundle exec rspec --tags env Mocks contract verifications are missing: No matching call found for: Anyway::Env#fetch: ("", _) -> Anyway::Env::Parsed Captured calls: ("", _) -> NilClass 82
93. palkan_tula palkan 83 Mocked objects finder Mocked calls collector Real calls collector Type checker Types generator Call patterns verifier after(:suite) before(:suite) run time
94. Are we there yet? 84
95. palkan_tula palkan Limitations — TracePoint could affect performance (~30 50% overhead) 85
96. palkan_tula palkan TP alternatives — Module#prepend — Source rewriting (Ruby Next ⏩) 86
97. palkan_tula palkan Limitations — TracePoint could affect performance — Parallel builds support is tricky — Verification patterns for non-value objects 87
98. To seam or not to seam? 88
99. palkan_tula palkan “Slow and reliable tests are in general much better than fast tests that break without reason (false positives) and don't catch actual breakage (false negatives).” –Jeremy Evans, Polished Ruby programming 89
100. palkan_tula palkan “If you track your test coverage, try for 100% coverage before integrations tests. Then keep writing integration tests until you sleep well at night.” –Active Interactor's Readme 90
101. palkan_tula palkan Keep it real — Integration tests are the best seams 91
102. palkan_tula palkan Keep it close to real — Integration tests are the best seams — Know your doubles 92
103. 93 $ stree search ~/dev/double_query.txt spec/**/*.rb spec/broadcast/redis_spec.rb:15 4: allow(Redis).to receive(:new) { redis_conn } spec/broadcast/redis_spec.rb:38 6: allow(redis_conn).to receive(:publish) spec/broadcast/nats_spec.rb:14 4: allow(NATS::Client).to receive(:new) { nats_conn } spec/broadcast/redis_spec.rb:52 6: allow(redis_conn).to receive(:publish) spec/broadcast/nats_spec.rb:15 4: allow(nats_conn).to receive(:connect) spec/broadcast/http_spec.rb:48 6: allow(AnyCable.logger).to receive(:error) spec/anycable_spec.rb:77 16: adapter = double("adapter", broadcast: nil) spec/broadcast/http_spec.rb:86 8: allow(adapter).to receive(:sleep) spec/broadcast/nats_spec.rb:38 6: allow(nats_conn).to receive(:publish) spec/broadcast/http_spec.rb:87 8: allow(AnyCable.logger).to receive(:error)
104. palkan_tula palkan 94 Syntax Tree Search CallNode[ receiver: NilClass, message: Ident[value: "double" | "instance_double"], arguments: ArgParen[ arguments: Args[ parts: [StringLiteral | VarRef[value: Const], BareAssocHash] ] ] ] | Command[ message: Ident[value: "double" | "instance_double"], arguments: Args[ parts: [StringLiteral | VarRef[value: Const], BareAssocHash] ] ] | # ... bit.ly/stree-doubles
105. palkan_tula palkan Keep it close to real 95 — Integration tests are the best seams — Know your doubles — Fixturize your doubles — Embrace types
106. palkan_tula palkan 96 Mock Suey github.com/test-prof/mock-suey gem "mock-suey"
107. Thanks! @palkan @palkan_tula evilmartians.com @evilmartians