Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[RubyConf 2022] Weaving & seaming mocks

[RubyConf 2022] Weaving & seaming mocks

https://www.rubyconfmini.com/program#Weaving-and-seaming-mocks

To mock or not mock is an important question, but let's leave it apart and admit that we, Rubyists, use mocks in our tests.

Mocking is a powerful technique, but even when used responsibly, it could lead to false positives in our tests (thus, bugs leaking to production): fake objects could diverge from their real counterparts.

In this talk, I'd like to discuss various approaches to keeping mocks in line with the actual implementation and present a brand new idea based on mock fixtures and contracts.

Vladimir Dementyev

November 16, 2022
Tweet

More Decks by Vladimir Dementyev

Other Decks in Programming

Transcript

  1. None
  2. 2 Ruby Devs ! Tests

  3. How do you write tests?

  4. palkan_tula palkan 4 Martin Fowler, Mocks Aren't Stubs

  5. palkan_tula palkan 5 Classical def search(q) query = Query.build(q) User.find_by_sql(query.to_sql)

    end
  6. palkan_tula palkan 5 Classical def search(q) query = Query.build(q) User.find_by_sql(query.to_sql)

    end user = create(:user, name: "Vova") expect(search("name=Vova")).to eq([user])
  7. palkan_tula palkan Mockist 5 def search(q) query = Query.build(q) User.find_by_sql(query.to_sql)

    end expect(Query) .to receive(:build).with("name=x") .and_return(double(to_sql: :cond)) expect(User).to receive(:find_by_sql).with(:cond) .and_return(:users) expected(search("user=x")).to eq :users
  8. palkan_tula palkan To mock or not to mock? Emily Samp

    @ RubyConf, 2021 6
  9. palkan_tula palkan Classical 7

  10. palkan_tula palkan 8 Mockist

  11. palkan_tula palkan 9 Mocklassicism

  12. palkan_tula palkan False positive 10 def search(q) query = Query.build(q)

    User.find_by_sql(query.to_sql) end expect(Query) .to receive(:build).with("name=x") .and_return(double(to_sql: :cond)) expect(User).to receive(:find_by_sql).with(:cond) .and_return(:users) expected(search("user=x")).to eq :users module Query - def self.build(query_str) - query_hash = Hash[query.split("=")] + def self.build(query_hash) # query has to sql end
  13. palkan_tula palkan How to avoid false positives? 11

  14. palkan_tula palkan 11 How to put seams? ?

  15. palkan_tula palkan github.com/palkan 12

  16. 13

  17. 14

  18. None
  19. Keeping mocks in line with real objects 16

  20. palkan_tula palkan 17 Case study: anyway_config class RubyConfConfig < Anyway::Config

    attr_config :city, :date coerce_types date: :date end github.com/palkan/anyway_config
  21. 18 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby

    -r rubyconf_conf.rb -e "pp RubyConfConfig.new" #<RubyConfConfig config_name="rubyconf" env_prefix="RUBYCONF" values: city => "Providence" (type=env key=RUBYCONF_CITY), date => #<Date: 2022 11 15> (type=env key=RUBYCONF_DATE)> #<RubyConfConfig config_name="rubyconf" env_prefix="RUBYCONF" values: city => "Providence" (type=env key=RUBYCONF_CITY), date => #<Date: 2022 11 15> (type=env key=RUBYCONF_DATE)>
  22. palkan_tula palkan 19 ENV parser class Anyway::Env def fetch(prefix) #

    Scans ENV and parses matching values some_hash end def fetch_with_trace(prefix) [fetch(prefix), traces[prefix]] end end
  23. palkan_tula palkan 20 ENV loader class Anyway::Loaders::Env def call(env_prefix:, **_options)

    env = Anyway::Env.new env.fetch_with_trace(env_prefix) .then do |(data, trace)| Tracing.current_trace&.merge!(trace) data end end end •
  24. 21 $ bundle exec rspec --tags env Run options: include

    {:env=>true} Randomized with seed 26894 ........ Finished in 0.00641 seconds (files took 0.27842 seconds to load) 8 examples, 0 failures Coverage is at 100.0%. Coverage report sent to Coveralls.
  25. palkan_tula palkan 22 ENV loader describe Anyway::Loaders::Env do subject {

    described_class.call(env_prefix: "TESTO") } it "loads data from env" do with_env( "TESTO_A" => "x", "TESTO_DATA__KEY" => "value" ) do expect(subject) .to eq({"a" => "x", "data" => { "key" => "value"}}) end end end Aren't we testing ENV parser here, not loader?
  26. palkan_tula palkan describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data)

    { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data, nil]) end it "loads data from Anyway::Env" do expect(subject) .to eq({"a" => "x", "data" => {"key" => "value"}}) end end describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data, nil]) end it "loads data from Anyway::Env" do expect(subject) .to eq({"a" => "x", "data" => {"key" => "value"}}) end end 23 Double-factoring
  27. palkan_tula palkan describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data)

    { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data end it "loads data from Anyway::Env" do expect(subject) describe Anyway::Loaders::Env do let(:env) { double("env") } let(:data) { {"a" => "x", "data" => {"key" => "value"}} } subject { described_class.call(env_prefix: "TESTO") } before do allow(::Anyway::Env).to receive(:new).and_return(env) allow(env).to receive(:fetch_with_trace).and_return([data end it "loads data from Anyway::Env" do expect(subject) 23 Double-factoring
  28. Case #1. Undefined method 24

  29. palkan_tula palkan 25 Refactor class Anyway::Env - def fetch(prefix) +

    def fetch(prefix, include_trace = false) @@ ... - def fetch_with_trace(prefix) - [fetch(prefix), traces[prefix]] - end end
  30. palkan_tula palkan 26 describe Anyway::Loaders::Env do let(:env) { double("env") }

    before do allow(::Anyway::Env) .to receive(:new).and_return(env) allow(env) .to receive(:fetch_with_trace) .and_return([data, nil]) end # ... end Refactor
  31. 27 $ bundle exec rspec --tags env Run options: include

    {:env=>true} Randomized with seed 26894 ........ Finished in 0.00594 seconds (files took 0.21516 seconds to load) 8 examples, 0 failures Coverage is at 100.0%. Coverage report sent to Coveralls.
  32. 28 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby

    -r rubyconf_conf.rb -e "pp RubyConfConfig.new" #<RubyConfConfig config_name="rubyconf" env_prefix="RUBYCONF" values: city => "Providence" (type=env key=RUBYCONF_CITY), date => #<Date: 2022 11 15> (type=env key=RUBYCONF_DATE)> anyway_config/lib/anyway/loaders/env.rb:11:in `call': undefined method `fetch_with_trace' for #<Anyway::Env:0x00000001008948f0 @type_cast=Anyway::NoCast, @data={}, @traces={}> (NoMethodError) env.fetch_with_trace(env_prefix).then do |(conf, trace)| ^^^^^^^^^^^^^^^^^
  33. palkan_tula palkan Double trouble — Tests are green ✅ —

    Coverage 100% # — Code doesn't work ❌ 29
  34. 30

  35. palkan_tula palkan 31 github.com/rspec/rspec-mocks/issues/227

  36. palkan_tula palkan 32 github.com/xaviershay/rspec-fire

  37. palkan_tula palkan 33 describe Anyway::Loaders::Env do - let(:env) { double("env")

    } - let(:env) { instance_double("Anyway::Env") } Refactor
  38. 34 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads

    data from Anyway::Env Failure/Error: allow(env_double).to receive(:fetch_with_trace).and_return([env, nil]) the Anyway::Env class does not implement the instance method: fetch_with_trace # ./spec/loaders/env_spec.rb:22:in `block (2 levels) in <compiled>'
  39. palkan_tula palkan 35 double No strings attached

  40. palkan_tula palkan verified double 35 double No strings attached Method

    existence
  41. Case #2. Incorrect method signature 36

  42. palkan_tula palkan 37 Refactor class Anyway::Env - def fetch(prefix, include_trace

    = false) + def fetch(prefix, include_trace: false) @@ ...
  43. palkan_tula palkan 37 Refactor class Anyway::Env - def fetch(prefix, include_trace

    = false) + def fetch(prefix, include_trace: false) @@ ... describe Anyway::Loaders::Env do let(:env) { instance_double("env") } before do expect(env) .to receive(:fetch) .with("TESTO", true) .and_return([data, nil]) end # ... end
  44. 38 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads

    data from Anyway::Env Failure/Error: env.fetch(env_prefix, true) ArgumentError: Wrong number of arguments. Expected 1, got 2. # ./lib/anyway/loaders/env.rb:11:in `call' # ./lib/anyway/loaders/base.rb:10:in `call' # ./spec/loaders/env_spec.rb:18:in
  45. palkan_tula palkan 39 class MethodSignatureVerifier def initialize(signature, args=[]) # ...

    end def valid? missing_kw_args.empty? && invalid_kw_args.empty? && valid_non_kw_args? && arbitrary_kw_args? && unlimited_args? end end rspec/support/method_signature_verifier.rb
  46. palkan_tula palkan 39 rspec/support/method_signature_verifier.rb class MethodSignature def initialize(method) # ...

    classify_parameters end def classify_parameters @method.parameters.each do |(type, name)| # ... end end end
  47. palkan_tula palkan class MethodSignature def initialize(method) # ... classify_parameters end

    def classify_parameters @method.parameters.each do |(type, name)| # ... end end end 40 rspec/support/method_signature_verifier.rb
  48. palkan_tula palkan 41 Method#parameters method_obj = Anyway::Env.instance_method(:fetch) method_obj.parameters #=> [

    [:req, :prefix], [:key, :include_trace] ]
  49. palkan_tula palkan 42 double verified double No strings attached Method

    existence Method parameters Method signature?
  50. palkan_tula palkan Method signature — Parameters shape — Argument types

    — Return value type 43
  51. palkan_tula palkan 44 Refactor class Anyway::Env + Parsed = Struct.new(:data,

    :trace) def fetch(prefix, include_trace: false) @@ ... - [data, trace] + Parsed.new(data, trace) end
  52. palkan_tula palkan 44 Refactor class Anyway::Env + Parsed = Struct.new(:data,

    :trace) def fetch(prefix, include_trace: false) @@ ... - [data, trace] + Parsed.new(data, trace) end describe Anyway::Loaders::Env do let(:env) { instance_double("env") } before do expect(env) .to receive(:fetch) .with("TESTO", include_trace: true) .and_return([data, nil]) end # ... end
  53. 45 $ bundle exec rspec --tags env Run options: include

    {:env=>true} Randomized with seed 43108 ........ Finished in 0.0234 seconds (files took 0.80197 seconds to load) 7 examples, 0 failures
  54. 46 $ RUBYCONF_CITY Providence RUBYCONF_DATE 2022 11 15 \ ruby

    -r rubyconf_conf.rb -e "pp RubyConfConfig.new" #<RubyConfConfig config_name="rubyconf" env_prefix="RUBYCONF" values: city => "Providence" (type=env key=RUBYCONF_CITY), date => #<Date: 2022 11 15> (type=env key=RUBYCONF_DATE)> anyway_config/lib/anyway/tracing.rb:59:in `merge!': undefined method `trace?' for nil:NilClass (NoMethodError) from anyway_config/lib/anyway/loaders/env.rb:12:in `block in call' from <internal:kernel>:124:in `then' from anyway_config/lib/anyway/loaders/env.rb:11:in `call'
  55. palkan_tula palkan “Test doubles are sweet for isolating your unit

    tests, but we lost something in the translation from typed languages. Ruby doesn't have a compiler that can verify the contracts being mocked out are indeed legit.” –rspec-fire's Readme 47
  56. Mocks vs. types 48

  57. palkan_tula palkan .rbs 49 double verified double No strings attached

    Method existence Method parameters
  58. palkan_tula palkan typed double 49 double verified double No strings

    attached Method existence Method parameters Method signature
  59. palkan_tula palkan typed_double — Intercept mocked calls — Type check

    them—that's it! 50
  60. palkan_tula palkan 51 Interception RSpec::Mocks::VerifyingMethodDouble.prepend( Module.new do def proxy_method_invoked(obj, *args,

    &block) super.tap { TypedDouble.typecheck!(obj, *args) } end end )
  61. palkan_tula palkan 52 RBS::Test

  62. palkan_tula palkan 53 anyway_config.rbs module Anyway class Env class Parsed

    attr_reader data: Hash attr_reader trace: Tracing::Trace? end def fetch: (String prefix, ?include_trace: bool) -> Parsed end end
  63. 54 $ bundle exec rspec --tags env 1) Anyway::Loaders::Env loads

    data from Anyway::Env Failure/Error: raise RBS::Test::Tester::TypeError.new(errors) unless errors.empty? RBS::Test::Tester::TypeError: TypeError: [Anyway::Env#fetch] ReturnTypeError: expected `::Anyway::Env::Parsed` but returns `[{"a"=>"x", "data"=>{"key"=>"value"}}, nil]`
  64. palkan_tula palkan What if... we don't have types % 55

  65. On-the-fly type signatures generation 56

  66. palkan_tula palkan Type generators — rbs prototype / tapioca —

    TypeProfiler — Tracing → signatures & 57
  67. palkan_tula palkan On-the-fly types — Collect method calls made on

    real objects 58
  68. palkan_tula palkan 59 Tracking calls TracePoint.trace(:call, :return) do |tp| next

    unless trackable?(tp.defined_class, tp.method_id) target, mid = tp.defined_class, tp.method_id if tp.event == :call method = tp.self.method(mid) args = [] kwargs = {} method.parameters.each do |(type, name)| val = tp.binding.local_variable_get(name) # ... end store[target][mid] << CallTrace.new(arguments: args, kwargs:) elsif tp.event == :return store[target][mid].last.return_value = tp.return_value end end
  69. palkan_tula palkan 60 Tracking calls if tp.event == :call method

    = tp.self.method(mid) args = [] kwargs = {} method.parameters.each do |(type, name)| val = tp.binding.local_variable_get(name) # ... end
  70. palkan_tula palkan On-the-fly types — Collect method calls made on

    real objects — Generate types from real call traces 61
  71. palkan_tula palkan 62 SignatureGenerator class SignatureGenerator def to_rbs = [header,

    method_sigs, footer].join("\n") def args_sig(args) args.transpose.map do |arg_values| arg_values.map(&:class).uniq.map do "::#{_1.name}" end end.join(", ") end end
  72. palkan_tula palkan 63 env.rbs module Anyway class Env def initialize:

    (?type_cast: (::Module)) -> (void) def fetch: ( ::String, ?include_trace: (::FalseClass | ::TrueClass) ) -> ::Anyway::Env::Parsed end end
  73. palkan_tula palkan On-the-fly types — Collect method calls made on

    real objects — Generate types from real call traces — Identify tracing targets (mocked classes) & 64
  74. Mock fixtures 65

  75. palkan_tula palkan Fixturama evilmartians.com/chronicles/a-fixture-based-approach-to-interface-testing-in-rails 66

  76. palkan_tula palkan 67 Fixturama github.com/nepalez/fixturama # fixtures/stubs/notifier.yml --- - class:

    Notifier chain: - create arguments: - :profileDeleted - <%= profile_id %> actions: - return: true - raise: ActiveRecord::RecordNotFound arguments: - "Profile with id: 1 not found" # for error message
  77. palkan_tula palkan YAML fixtures — YAML != Ruby, hard to

    mock with non- primitive types — Existing mocks are not re-usable—a lot of refactoring 68
  78. palkan_tula palkan 69 Mock context mock_context "Anyway::Env" do before do

    env_double = instance_double("Anyway::Env") allow(Anyway::Env).to receive(:new).and_return(env_double) data = {"a" => "x", "data" => {"key" => "value"}} allow(env_double) .to receive(:fetch).with("TESTO", any_args) .and_return([data, nil]) end end
  79. palkan_tula palkan Mock context — Just a shared context —

    Evaluated within a "scratch" example group on initialization to collect information about stubbed classes and methods 70
  80. palkan_tula palkan 71 Mock context def evaluate_context!(context_id, tracking) Class.new(RSpec::Core::ExampleGroup) do

    include_context(context_id) specify("true") { expect(true).to be(true) } after do RSpec::Mocks.space.proxies.values.each do tracking.register_from_proxy(_1) end end end.run end
  81. palkan_tula palkan 72 Refactor describe Anyway::Loaders::Env do - let(:env) {

    instance_double(Anyway::Env) } + include_mock_context "Anyway::Env" @@ ...
  82. palkan_tula palkan 73 RSpec post-check config.after(:suite) do TypedDouble.infer_types_from_calls!( CallsTracer.calls )

    passed = MocksTracer.calls.all do TypedDouble.typecheck(_1) end unless passed exit(RSpec.configuration.failure_exit_code) end end
  83. palkan_tula palkan 74 Mocked objects finder Mocked calls collector Real

    calls collector Type checker Types generator after(:suite) before(:suite) run time
  84. Case #3. Non-matching behaviour 75

  85. palkan_tula palkan 76 Refactor class Anyway::Env def fetch(prefix, **) +

    return if prefix.empty? @@ ...
  86. palkan_tula palkan 76 Refactor class Anyway::Env def fetch(prefix, **) +

    return if prefix.empty? @@ ... mock_context "Anyway::Env" do before do # ... allow(env_double) .to receive(:fetch) .with("", any_args) .and_return( Anyway::Env::Parsed.new({}, nil)) end end
  87. palkan_tula palkan 77 env.rbs module Anyway class Env def fetch:

    ( ::String, ?include_trace: (::FalseClass | ::TrueClass) ) -> ::Anyway::Env::Parsed? end end We cannot specify which string causes the return value to be nil
  88. palkan_tula palkan Double contract — Stubs represent contracts — Tests

    using real objects MUST verify contracts (unit, end-to-end) 78
  89. palkan_tula palkan 79 github.com/psyho/bogus github.com/robwold/compact

  90. palkan_tula palkan Stubs → Contracts — Collect method stub expected

    arguments — Check that a real call with the matching arguments was made and its return type matches the mocked one 80
  91. palkan_tula palkan 81 Stubs → Contracts allow(env_double).to receive(:fetch) .with("", any_args)

    .and_return(Anyway::Env::Parsed.new({})) Anyway::Env#fetch: ("", _) -> Anyway::Env::Parsed verification pattern
  92. $ bundle exec rspec --tags env Mocks contract verifications are

    missing: No matching call found for: Anyway::Env#fetch: ("", _) -> Anyway::Env::Parsed Captured calls: ("", _) -> NilClass 82
  93. palkan_tula palkan 83 Mocked objects finder Mocked calls collector Real

    calls collector Type checker Types generator Call patterns verifier after(:suite) before(:suite) run time
  94. Are we there yet? 84

  95. palkan_tula palkan Limitations — TracePoint could affect performance (~30 50%

    overhead) 85
  96. palkan_tula palkan TP alternatives — Module#prepend — Source rewriting (Ruby

    Next ⏩) 86
  97. palkan_tula palkan Limitations — TracePoint could affect performance — Parallel

    builds support is tricky — Verification patterns for non-value objects 87
  98. To seam or not to seam? 88

  99. palkan_tula palkan “Slow and reliable tests are in general much

    better than fast tests that break without reason (false positives) and don't catch actual breakage (false negatives).” –Jeremy Evans, Polished Ruby programming 89
  100. palkan_tula palkan “If you track your test coverage, try for

    100% coverage before integrations tests. Then keep writing integration tests until you sleep well at night.” –Active Interactor's Readme 90
  101. palkan_tula palkan Keep it real — Integration tests are the

    best seams 91
  102. palkan_tula palkan Keep it close to real — Integration tests

    are the best seams — Know your doubles 92
  103. 93 $ stree search ~/dev/double_query.txt spec/**/*.rb spec/broadcast/redis_spec.rb:15 4: allow(Redis).to receive(:new)

    { redis_conn } spec/broadcast/redis_spec.rb:38 6: allow(redis_conn).to receive(:publish) spec/broadcast/nats_spec.rb:14 4: allow(NATS::Client).to receive(:new) { nats_conn } spec/broadcast/redis_spec.rb:52 6: allow(redis_conn).to receive(:publish) spec/broadcast/nats_spec.rb:15 4: allow(nats_conn).to receive(:connect) spec/broadcast/http_spec.rb:48 6: allow(AnyCable.logger).to receive(:error) spec/anycable_spec.rb:77 16: adapter = double("adapter", broadcast: nil) spec/broadcast/http_spec.rb:86 8: allow(adapter).to receive(:sleep) spec/broadcast/nats_spec.rb:38 6: allow(nats_conn).to receive(:publish) spec/broadcast/http_spec.rb:87 8: allow(AnyCable.logger).to receive(:error)
  104. palkan_tula palkan 94 Syntax Tree Search CallNode[ receiver: NilClass, message:

    Ident[value: "double" | "instance_double"], arguments: ArgParen[ arguments: Args[ parts: [StringLiteral | VarRef[value: Const], BareAssocHash] ] ] ] | Command[ message: Ident[value: "double" | "instance_double"], arguments: Args[ parts: [StringLiteral | VarRef[value: Const], BareAssocHash] ] ] | # ... bit.ly/stree-doubles
  105. palkan_tula palkan Keep it close to real 95 — Integration

    tests are the best seams — Know your doubles — Fixturize your doubles — Embrace types
  106. palkan_tula palkan 96 Mock Suey github.com/test-prof/mock-suey gem "mock-suey"

  107. Thanks! @palkan @palkan_tula evilmartians.com @evilmartians