Upgrade to Pro — share decks privately, control downloads, hide ads and more …

reality is expensive: a better way of thinking about mock objects

reality is expensive: a better way of thinking about mock objects

Videos of this talk:
Rocky Mountain Ruby '12 — http://www.youtube.com/watch?v=TU3glG08BJI
Øredev 2012 — http://vimeo.com/53276460
SCNA 2012 — http://vimeo.com/54045166

This talk is an introduction to the different ways people use test doubles (be they mocks, fakes, stubs, or spies). Our goal will be to establish a more sophisticated means of communicating on the topic. This talk discusses our need to understand that reality is a variable under our control, the pros & cons of the different testing strategies, and the smells of test double abuse.

Justin Searls

November 10, 2012
Tweet

More Decks by Justin Searls

Other Decks in Programming

Transcript

  1. imagine: testing landing gear

    View full-size slide

  2. if our tests prove
    that our code works

    View full-size slide

  3. then "fake" means "cheat," right?

    View full-size slide

  4. we're all
    cheaters

    View full-size slide

  5. selenium
    A user who moves
    the mouse &
    keyboard exactly
    how you tell it to.

    View full-size slide

  6. selenium
    A user who moves
    the mouse &
    keyboard exactly
    how you tell it to.
    Over and over.

    View full-size slide

  7. selenium
    A user who moves
    the mouse &
    keyboard exactly
    how you tell it to.
    Over and over.
    Every time.

    View full-size slide

  8. test data
    Runner cleans
    the database on
    every test run.

    View full-size slide

  9. test data
    Runner cleans
    the database on
    every test run.
    Got years of data
    migrations?

    View full-size slide

  10. test data
    Runner cleans
    the database on
    every test run.
    Got years of data
    migrations?
    No problem!

    View full-size slide

  11. stamina
    Runtime is always
    started fresh for
    tests.

    View full-size slide

  12. stamina
    Runtime is always
    started fresh for
    tests.
    Subtle memory
    leak?

    View full-size slide

  13. stamina
    Runtime is always
    started fresh for
    tests.
    Subtle memory
    leak?
    Tests won't mind!

    View full-size slide

  14. web scale
    Tests run against
    a single-process
    runner or server.

    View full-size slide

  15. web scale
    Tests run against
    a single-process
    runner or server.
    Race conditions?

    View full-size slide

  16. web scale
    Tests run against
    a single-process
    runner or server.
    Race conditions?
    No deadlocks
    here!

    View full-size slide

  17. reality is expensive

    View full-size slide

  18. why not budget it?
    reality is expensive

    View full-size slide

  19. so, instead of asking

    View full-size slide

  20. is this test too fake?

    View full-size slide

  21. perhaps, we should ask

    View full-size slide

  22. how much reality
    does this test need?

    View full-size slide

  23. unknown fakeness
    we talked about

    View full-size slide

  24. known fakeness?
    but what about

    View full-size slide

  25. 1. identify fake things
    we already consciously
    do in our tests
    know thy fakeness

    View full-size slide

  26. 2. understand those
    fake things well enough
    to appraise their costs &
    benefits
    know thy fakeness

    View full-size slide

  27. 3. write more valuable
    tests by strategically
    deciding how much
    reality to give them
    know thy fakeness

    View full-size slide

  28. oft-overheard at conferences

    View full-size slide

  29. "Mocks are Good!"
    oft-overheard at conferences

    View full-size slide

  30. "Mocks are Good!"
    Bad!
    oft-overheard at conferences

    View full-size slide

  31. "Only Mock
    What You Own!"
    oft-overheard at conferences

    View full-size slide

  32. "Only Mock
    What You Own!"
    external systems!
    oft-overheard at conferences

    View full-size slide

  33. "Mock everything!"
    oft-overheard at conferences

    View full-size slide

  34. "Mock everything!"
    avoid over-mocking!
    oft-overheard at conferences

    View full-size slide

  35. nuance
    I'd some

    View full-size slide

  36. rigor
    I'd some

    View full-size slide

  37. thoughtfulness
    I'd some

    View full-size slide

  38. types of tests

    View full-size slide

  39. Dog App Tester
    Dog App
    end-to-end test

    View full-size slide

  40. DogFeederTest
    DogRepo
    Database
    DogFeeder
    integration test
    BoneRepo

    View full-size slide

  41. DogFeederTest
    DogRepo
    Fake Database
    DogFeeder
    "unit" test
    BoneRepo

    View full-size slide

  42. DogFeederTest
    Fake
    DogRepo DogFeeder
    isolation test
    Fake
    BoneRepo

    View full-size slide

  43. mock
    The popular term for a
    fake object that takes
    the place of a real object
    when a test is executed.
    n. \ˈmäk, ˈmȯk\

    View full-size slide

  44. test double
    The precise term for a
    fake object that takes
    the place of a real object
    when a test is executed.
    n. \ˈtest də-bəl\

    View full-size slide

  45. think "stunt double"

    View full-size slide

  46. our silly catch-all term
    test double

    View full-size slide

  47. an alternative implementation to
    stand-in for something you depend on
    fake
    test double

    View full-size slide

  48. replies to certain
    messages with
    responses that
    help you write tests
    fake
    test double
    stub

    View full-size slide

  49. ensures certain
    messages are received
    (and explodes upon
    receiving any
    unexpected messages)
    fake
    test double
    stub mock

    View full-size slide

  50. stealthily records
    all interactions,
    allowing you to
    make assertions
    about them later
    fake
    test double
    stub mock spy

    View full-size slide

  51. how should
    we use them?

    View full-size slide

  52. hold your fake horses!

    View full-size slide

  53. why should
    we use them?

    View full-size slide

  54. why do we
    test at all?

    View full-size slide

  55. ACCEPTANCE
    prove the app works as promised

    View full-size slide

  56. ACCEPTANCE
    prove the app works as promised

    View full-size slide

  57. SPECIFICATION
    examples of how the code behaves

    View full-size slide

  58. SPECIFICATION
    examples of how the code behaves

    View full-size slide

  59. REGRESSION
    prevent bugs from coming back

    View full-size slide

  60. REGRESSION
    prevent bugs from coming back

    View full-size slide

  61. DESIGN
    shape code by listening to tests

    View full-size slide

  62. DESIGN
    shape code by listening to tests

    View full-size slide

  63. CHARACTERIZE
    safely improve legacy code

    View full-size slide

  64. CHARACTERIZE
    safely improve legacy code

    View full-size slide

  65. this test provide?"
    ask, "what value will

    View full-size slide

  66. value-oriented test strategies

    View full-size slide

  67. prove it works

    View full-size slide

  68. prove it works
    minimize fakeness

    View full-size slide

  69. prove it works
    only write end-to-end
    & integration tests
    minimize fakeness

    View full-size slide

  70. value cost
    prove it works

    View full-size slide

  71. value cost
    prove it works
    • confidence
    that passing
    equals working

    View full-size slide

  72. value cost
    prove it works
    • confidence
    that passing
    equals working
    • fewer tests →
    less effort spent
    writing tests

    View full-size slide

  73. value cost
    prove it works
    • confidence
    that passing
    equals working
    • fewer tests →
    less effort spent
    writing tests
    • tests can't
    provide many
    design cues

    View full-size slide

  74. value cost
    prove it works
    • confidence
    that passing
    equals working
    • fewer tests →
    less effort spent
    writing tests
    • tests can't
    provide many
    design cues
    • slow feedback

    View full-size slide

  75. value cost
    prove it works
    • confidence
    that passing
    equals working
    • fewer tests →
    less effort spent
    writing tests
    • tests can't
    provide many
    design cues
    • slow feedback
    • high coverage
    is infeasible

    View full-size slide

  76. passage of time
    build duration

    View full-size slide

  77. passage of time
    build duration
    what we expect

    View full-size slide

  78. passage of time
    build duration

    View full-size slide

  79. passage of time
    build duration
    what actually happens
    once you account for
    growth of the system

    View full-size slide

  80. mocking boundaries

    View full-size slide

  81. mocking boundaries
    fake remote systems,
    never your app's code

    View full-size slide

  82. mocking boundaries
    fake remote systems,
    never your app's code
    primarily write unit
    & end-to-end tests

    View full-size slide

  83. value cost
    mocking boundaries

    View full-size slide

  84. value cost
    mocking boundaries
    • each object is
    exercised by
    many tests

    View full-size slide

  85. value cost
    mocking boundaries
    • each object is
    exercised by
    many tests
    • practical tests
    without deep
    knowledge of
    test doubles

    View full-size slide

  86. value cost
    mocking boundaries
    • each object is
    exercised by
    many tests
    • practical tests
    without deep
    knowledge of
    test doubles
    • tests give less
    feedback about
    interactions

    View full-size slide

  87. value cost
    mocking boundaries
    • each object is
    exercised by
    many tests
    • practical tests
    without deep
    knowledge of
    test doubles
    • tests give less
    feedback about
    interactions
    • one change →
    many test fixes

    View full-size slide

  88. value cost
    mocking boundaries
    • each object is
    exercised by
    many tests
    • practical tests
    without deep
    knowledge of
    test doubles
    • tests give less
    feedback about
    interactions
    • one change →
    many test fixes
    • end-to-end is
    extra redundant

    View full-size slide

  89. case-by-case

    View full-size slide

  90. case-by-case
    cater your testing approach
    to the needs of each situation

    View full-size slide

  91. value cost
    case-by-case

    View full-size slide

  92. value cost
    case-by-case
    • developers
    free to choose
    best approach

    View full-size slide

  93. value cost
    case-by-case
    • developers
    free to choose
    best approach
    • your team is
    likely already
    doing it!

    View full-size slide

  94. value cost
    case-by-case
    • developers
    free to choose
    best approach
    • your team is
    likely already
    doing it!
    • test doubles
    easily abused

    View full-size slide

  95. value cost
    case-by-case
    • developers
    free to choose
    best approach
    • your team is
    likely already
    doing it!
    • test doubles
    easily abused
    • no strategy →
    murky test value

    View full-size slide

  96. value cost
    case-by-case
    • developers
    free to choose
    best approach
    • your team is
    likely already
    doing it!
    • test doubles
    easily abused
    • no strategy →
    murky test value
    • "wanna mock
    that?" timesink

    View full-size slide

  97. the GOOS way

    View full-size slide

  98. the GOOS way
    isolation tests to drive design

    View full-size slide

  99. the GOOS way
    isolation tests to drive design
    end-to-end tests
    to prove it works

    View full-size slide

  100. value cost
    the GOOS way

    View full-size slide

  101. value cost
    the GOOS way
    • TDD gives rich
    feedback about
    interactions

    View full-size slide

  102. value cost
    the GOOS way
    • TDD gives rich
    feedback about
    interactions
    • consistent,
    fast, complete

    View full-size slide

  103. value cost
    the GOOS way
    • TDD gives rich
    feedback about
    interactions
    • consistent,
    fast, complete
    • easy to extract
    from app to libs

    View full-size slide

  104. value cost
    the GOOS way
    • TDD gives rich
    feedback about
    interactions
    • consistent,
    fast, complete
    • easy to extract
    from app to libs
    • requires
    discipline &
    practice

    View full-size slide

  105. value cost
    the GOOS way
    • TDD gives rich
    feedback about
    interactions
    • consistent,
    fast, complete
    • easy to extract
    from app to libs
    • requires
    discipline &
    practice
    • awkward "in-
    between" tests

    View full-size slide

  106. value cost
    the GOOS way
    • TDD gives rich
    feedback about
    interactions
    • consistent,
    fast, complete
    • easy to extract
    from app to libs
    • requires
    discipline &
    practice
    • awkward "in-
    between" tests
    • frameworks
    dislike isolation

    View full-size slide

  107. my favorite strategy

    View full-size slide

  108. 1. agree on literally
    any clear strategy
    my favorite strategy

    View full-size slide

  109. 1. agree on literally
    any clear strategy
    2. follow that strategy
    consistently
    my favorite strategy

    View full-size slide

  110. test double
    smells

    View full-size slide

  111. smell #
    1
    test doubles are used
    in integration tests

    View full-size slide

  112. perhaps
    test integrity was
    compromised to
    address painful
    setup or assertions

    View full-size slide

  113. suggestion
    respond to pain by
    changing your code,
    not your tests

    View full-size slide

  114. smell #2
    test replaces
    methods on the
    subject-under-test
    with test doubles

    View full-size slide

  115. perhaps
    subject has too many
    responsibilities

    View full-size slide

  116. suggestion
    extract a new object
    as a collaborator of
    the subject

    View full-size slide

  117. smell #3
    your objects extend
    framework classes
    that you don't own

    View full-size slide

  118. perhaps
    inherited behavior
    inhibits good test
    isolation

    View full-size slide

  119. suggestion
    avoid such extension
    or forgo isolation
    testing of those
    classes

    View full-size slide

  120. smell #4
    test contains lots of
    code just to configure
    test doubles

    View full-size slide

  121. perhaps
    the unit's number of
    collaborators or their
    collective surface
    area is too large

    View full-size slide

  122. suggestion
    reduce the count of
    methods on which
    the subject depends

    View full-size slide

  123. smell #5
    test doubles are used
    to stand-in for 3rd
    party APIs

    View full-size slide

  124. perhaps
    test pain isn't
    actionable, because
    design of 3rd party
    code can't be changed

    View full-size slide

  125. suggestion
    wrap third-party
    libraries in small
    abstractions you own

    View full-size slide

  126. blame the code
    before the test

    View full-size slide

  127. blame the code
    before the test
    double
    ^

    View full-size slide

  128. www.testdouble.com
    @andrewvida @jasonkarns
    @dmosher @joelhelbling
    @searls
    @kbaribeau @timwingfield @toddkaufman

    View full-size slide