Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Observing Change: A Gold Master Test in Practice

Observing Change: A Gold Master Test in Practice

Tests try to observe change. But are some systems too big to observe them all? What if we need to test a function with a very complex output?

In this talk, we'll explore a Gold Master test– a special test for evaluating complicated legacy systems. We'll look at how this test takes an input, such as a production database, runs it through a transformative function, and then compares the output to an approved version of the output.

Testers of every experience level will leave this talk with a new technique for evaluating complex environments, and a broader conception of what a test can be.

Jake Worth

April 25, 2017
Tweet

More Decks by Jake Worth

Other Decks in Programming

Transcript

  1. Any code of your own that you haven't looked at

    for six or more months might as well have been written by someone else. — Eagleson's Law 2
  2. 12

  3. 17

  4. In nearly every legacy system, what the system does is

    more important than what it's supposed to do. — Michael C. Feathers, Working Effectively With Legacy Code, p. 186 18
  5. characterization test A characterization test is a test that characterizes

    the actual behavior of a piece of code.1 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 19
  6. characterization test process1 1.Use a piece of code in a

    test harness. 2.Write an assertion that you know will fail. 3.Let the failure tell you what the behavior is. 4.Change the test so that it expects the behavior the code produces. 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 20
  7. # some/crummy_spec.rb expect(something).to eq 2 => Failure/Error: expect(something).to eq 2

    expected: 2 got: 1 expect(something).to eq 1 => 1 example, 0 failures 21
  8. a heuristic for writing characterization tests1 (abridged) 1.Write tests where

    you will make changes. Write as many cases as you feel you need. 2.Look at the specific things you are going to change, and attempt to write tests for those. 3.Write additional tests on a case-by-case basis. 1 Michael C. Feathers, Working Effectively With Legacy Code, pg. 195. 22
  9. gold master test A gold master test is a regression

    test for complex, untested systems that asserts a consistent macro-level behavior. 23
  10. gold master test A gold master test is a regression

    test for complex, untested systems that asserts a consistent macro-level behavior. 28
  11. feature revisited • large production database • fed into a

    complex PostgreSQL function • output into a giant CSV file 30
  12. # lib/tasks/create_db.rake desc "Generate the gold master test database" task

    :create_db do destination = Rails.root.join('spec/fixtures/gold_master.sql') sh "pg_dump sanitized_utility_database <pg flags> #{destination}" end 33
  13. testing strategy this test can do two things: 1.first run:

    generate gold master 2.subsequent runs: compare current result to gold master 38
  14. # spec/features/gold_master_spec.rb gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) else gold_master

    = gold_master_file.read if gold_master != actual gold_master_file.write(actual) end expect(actual).to eq(gold_master) end 41
  15. # spec/features/gold_master_spec.rb describe 'Fan.shred!' do it 'produces a consistent result'

    do ApplicationRecord.connection.execute <<-SQL truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL actual = Fan.shred! gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(actual) else gold_master = gold_master_file.read if gold_master != actual gold_master_file.write(actual) end expect(actual).to eq(gold_master) end end end 42
  16. 46

  17. ideal application • [x] mature • [x] complex • [x]

    we expect minimal change to output 47
  18. assertion The homepage, given the same data, should not change,

    without us knowing why. https://github.com/hashrocket/hr-til/tree/gold- master-demo 48
  19. preparation revisited • get a production database dump • scrub

    sensitive information • dump as plaintext sql • check in sql 49
  20. scrub data -- db/sanitize_production.sql -- Sanitize devs update developers set

    username = concat('rocketeer', id), email = concat('rocketeer', id, '@hashrocket.com'), twitter_handle = concat('hashrocket', id), admin = false, slack_name = concat('rocketeer', id); -- Clear sessions delete from authem_sessions; -- Limit the posts delete from posts where id > 200; 50
  21. # spec/features/gold_master_spec.rb describe 'gold master' do before do ApplicationRecord.connection.execute <<-SQL

    truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL end end 51
  22. # spec/features/gold_master_spec.rb describe 'gold master' do before do ApplicationRecord.connection.execute <<-SQL

    truncate schema_migrations; #{Rails.root.join('spec/fixtures/gold_master.sql').read} SQL visit root_path end specify 'the homepage does not change' do page_html = page.html gold_master_file = Rails.root.join('spec/fixtures/gold_master.txt') if !gold_master_file.exist? gold_master_file.write(page_html) else gold_master = gold_master_file.read if gold_master != page_html gold_master_file.write(page_html) end expect(page_html).to eq(gold_master) end end end 54
  23. 55

  24. 56

  25. challenges • maintenance -> it's an investment • slower ->

    we can optimize data • implies correctness -> teaching opportunity 57
  26. 59

  27. further reading • Slides: https://speakerdeck.com/jwworth • Code samples: https://github.com/hashrocket/hr-til/tree/gold-master-demo •

    Image Source: Watermelon Kaboom!!, KT King, Flickr.com, https://www.flickr.com/photos/xtrah/5005443977, Accessed 18 Mar 2017. 1. Change Code Without Fear, Nada daVeiga, Dr. Dobb's, http://www.drdobbs.com/tools/change-code-without-fear/206105233, February 6, 2008. Accessed 26 February, 2017. 2. Detecting behavioural changes when refactoring a web-based legacy system, Peter Spegel, May 21, 2015. 3. Gold Master Testing, Bryan Helmkamp, Code Climate Blog, http://blog.codeclimate.com/blog/2014/02/20/gold-master-testing/, February 20, 2014. Accessed February 26, 2017. 4. Gold Master Testing, Jake Worth, Hashrocket Blog, https://hashrocket.com/blog/posts/gold-master-testing, November 8, 2016. 5. How I run Legacy Code Retreat, J. B. Rainsberger, Legacy Code Retreat, http://legacycoderetreat.typepad.com/blog/2011/11/how-i-run-legacy-code- retreat.html, November 21, 2011. Accessed 26 February, 2017. 6. How Not To Write Golden Master Tests, J. B. Rainsberger, The Code Whisperer, http://blog.thecodewhisperer.com/permalink/how-not-to-write-golden-master- tests. Accessed 22 March, 2017. 7. Using the Golden Master technique to test legacy code, Chris Melinn, https://chrismelinn.wordpress.com/2013/04/12/using-the-golden-master-technique-to- test-legacy-code/, 12 April 2013. Accessed 26 February, 2017. 8. Working Effectively With Characterization Tests - Part 2, Alberto Savoia, http://www.artima.com/weblogs/viewpost.jsp?thread=198674, March 13, 2007. Accessed February 26, 2017. 9. Working Effectively With Characterization Tests, Alberto Savoia, artima developer, http://www.artima.com/weblogs/viewpost.jsp?thread=198296, March 9, 2007. Accessed February 26, 2017. 10.Working Effectively With Legacy Code, Michael C. Feathers, 2005. 62