Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Don't Let Your Tests Flake Out

Don't Let Your Tests Flake Out

The build’s red with a test failure. You re-run the tests and suddenly all is well. What’s going on?

While Ruby makes it easy to start testing your code, it’s also easy to write tests that fail in subtle, unexpected ways. Luckily, flaky tests share common causes, and there are ways to flush them out.

From order dependencies to static state, time comparisons to threading bugs, we’ll see what’s making your test suite unstable and how to get it rock solid again.

Jason R Clark

June 24, 2014
Tweet

More Decks by Jason R Clark

Other Decks in Technology

Transcript

  1. Globals 10 def test_isnt_evil refute $evil end def test_if_we_were_evil $evil

    = true assert $evil end false Tuesday, June 24, 14
  2. Globals 11 def test_isnt_evil refute $evil end def test_if_we_were_evil $evil

    = true assert $evil end true Tuesday, June 24, 14
  3. Globals 12 def test_if_we_were_evil $evil = true assert $evil end

    def test_isnt_evil refute $evil end true Tuesday, June 24, 14
  4. Globals 13 def test_if_we_were_evil $evil = true assert $evil end

    def test_isnt_evil refute $evil end true! Tuesday, June 24, 14
  5. 19 class ClassyTest < Minitest::Test @@classy = true def test_classy

    assert @@classy end end Class Variables Tuesday, June 24, 14
  6. 20 Class Variables class ClassyTest < Minitest::Test # ... def

    test_not_classy @@classy = false refute @@classy end end Tuesday, June 24, 14
  7. 24 class Object def boo puts "Boo" end end class

    Object remove_method(:boo) end Metaprogramming Tuesday, June 24, 14
  8. 24 class Object def boo puts "Boo" end end class

    Object remove_method(:boo) end class Object alias :hoo :boo end Metaprogramming Tuesday, June 24, 14
  9. 27 Threading class ThreadsAreCoolTest < Minitest::Test def test_coolness Thread.new do

    loop { $cool = true } end assert $cool end end Tuesday, June 24, 14
  10. Time Roundtripping 34 def test_time time = Time.now.to_f round_trip =

    Time.at(time).to_f assert_equal(time, round_trip) end Tuesday, June 24, 14
  11. Reset Consistently 42 def setup_and_teardown_agent(opts = {}, &block) define_method(:setup) do

    setup_agent(opts, &block) end define_method(:teardown) do teardown_agent end end Tuesday, June 24, 14
  12. https://github.com/travisjeffery/timecop travisjeffery/timecop 44 describe "some set of tests to mock"

    do before do Timecop.freeze(Time.local(1990)) end after do Timecop.return end it "should do blah blah blah" {} end Tuesday, June 24, 14
  13. Metaprogramming 47 class TransmogifierTest < Minitest::Test def test_modifies_class clazz =

    Class.new Transmogifier.go!(clazz) assert clazz.new.tentacles? end end Tuesday, June 24, 14
  14. 57 class SingleFailingTest < Minitest::Test def test_i_sometimes_fail 100.times do setup

    flaky_call() teardown end end end Loop Tuesday, June 24, 14
  15. 58 while true do bundle exec rake test || break

    done Loop Tuesday, June 24, 14
  16. Differences on the Build Box 59 OS Hardware Performance Paths,

    files, directories Gem versions Tuesday, June 24, 14
  17. Threading 61 class MultiThreadedWorker def go! Thread.new do # TODO:

    ONLY FOR DIAGNOSISING!!!!!!! sleep(0.1) work() end end end Tuesday, June 24, 14
  18. Threading 63 class MultiThreadedWorker attr_reader :thread def go! @thread =

    Thread.new do # ... end end end Tuesday, June 24, 14
  19. Threading 64 class MultiThreadedWorkerTest < Minitest::Test def test_worker worker =

    MultiThreadedWorker.new worker.go! worker.thread.join end end Tuesday, June 24, 14
  20. jasonrclark/hometown https://github.com/jasonrclark/hometown 79 class NoticeThreadTest < Minitest::Test def setup Thread.new

    { work! } end def test_notice_thread Thread.list.each do |thread| trace = Hometown.for(thread) p trace unless trace.nil? end end end Tuesday, June 24, 14
  21. jasonrclark/minitest-stately https://github.com/jasonrclark/minitest-stately 82 # Gemfile gem 'minitest-stately' # test_helper.rb #

    Watch for freshly started threads Minitest::Stately.watch("thread count") do Thread.list.count end Tuesday, June 24, 14