Testing Isn't Enough: Fighting Bugs with Hacks

This talk was given at RubyConf 2014.

Transcript

1. Testing Isn't Enough: Fighting Bugs with Hacks Paul Gross paul@braintreepayments.com @pgr0ss
2. Bugs are a serious problem in software
3. Mars Climate Orbiter September 23, 1999: Bug in calculation units caused the spacecraft to burn up in Mars atmosphere
4. World of Warcraft Plague September 13, 2005: Bug in a spell caused a pandemic through the virtual world
5. Knight Capital August 1, 2012: Bug in trading software lost $440 million
6. What's our strategy for dealing with bugs today?
7. Testing • Tests are great
8. Testing • Tests are great • Regardless of philosophy or strategy
9. Testing is not enough • Well tested software is not bug free
10. If debugging is the process of removing bugs, then programming must be the process of putting them in. — Edsger W. Dijkstra Program testing can be a very effective way to show the presence of bugs, but it is hopelessly inadequate for showing their absence. — Edsger W. Dijkstra
11. Mitigation • Fail fast when we know there is a problem
12. Mitigation • Fail fast when we know there is a problem • Structure our systems to reduce the severity of bugs
13. Real world example Operating rooms
15. Example Hospital management system
16. class ClinicalNoteController < ApplicationController def show @note = ClinicalNote.find_by!(id: params[:id]) end end
17. class ClinicalNoteController < ApplicationController def show @note = ClinicalNote.find_by!(id: params[:id], patient_id: params[:patient_id]) end end
18. class ClinicalNoteController < ApplicationController def show @note = ClinicalNoteService.find(params[:id], params[:patient_id]) end end
19. class ClinicalNoteController < ApplicationController def show @note = ClinicalNoteService.find(params[:patient_id], params[:id]) raise "incorrect patient" unless @note.patient_id == params[:patient_id] end end
20. Strategy • Figure out the invariants of a system
21. Strategy • Figure out the invariants of a system • Add runtime checks for these invariants
22. Strategy • Figure out the invariants of a system • Add runtime checks for these invariants • Raise and alert so the issue can be investigated and fixed
23. Example Braintree payment gateway
24. class CustomerController < ApplicationController def show @customer = Customer.find_by!(token: params[:token]) end end
25. @customer = @merchant.customers.find_by!(token: params[:token])
26. How can we prevent simple mistakes?
27. >> Customer.where(:token => 'Paul') RuntimeError: #finds must be scoped on Customer
28. >> Customer.where(:token => 'Paul') RuntimeError: #finds must be scoped on Customer >> merchant = Merchant.find(1) >> merchant.customers.where(:token => 'Paul') [#<Customer id: 1 ...
29. Allow bypassing >> Customer.allow_unscoped_find.where(:token => 'Paul')
30. What about code like this? Customer.find_by_sql(["SELECT * FROM customers WHERE token = ?", "Paul"])
31. URLs are scoped by merchant /merchants/<merchant_id>/customers/<token>
32. class ApplicationController < ActionController::Base around_filter :ensure_merchant_consistency def ensure_merchant_consistency(&block) @merchant = Merchant.find_by!(:public_id => params["merchant_id"]) MerchantConsistencyCheck.with_merchant(@merchant, &block) end end
33. class ApplicationController < ActionController::Base around_filter :ensure_merchant_consistency def ensure_merchant_consistency(&block) @merchant = Merchant.find_by!(:public_id => params["merchant_id"]) MerchantConsistencyCheck.with_merchant(@merchant, &block) end end ... Customer.find_by_sql(["SELECT * FROM customers WHERE token = ?", "Paul"]) ScopedFindHook::ScopeError: Customer cannot return objects scoped by the incorrect merchant. Got 6, expected 1.
34. Strategy • Focus on the validity of the data
35. Strategy • Focus on the validity of the data • Code can be fixed more easily than data
36. Example Recurring billing
37. Check data consistency if billing_period_start_date > billing_period_end_date raise "Subscription internal state is inconsistent -- " + "billing_period_start_date vs billing_period_end_date: #{inspect}" end
38. Check data consistency if billing_period_start_date > billing_period_end_date raise "Subscription internal state is inconsistent -- " + "billing_period_start_date vs billing_period_end_date: #{inspect}" end if _correct_next_billing_date != next_billing_date raise "Subscription internal state is inconsistent -- " + "next_billing_date incorrect: #{inspect}" end
39. Strategy • Write checks outside of the normal flow
40. Strategy • Write checks outside of the normal flow • Keep checking code simpler than real code
41. Example Settlement
42. Nagios check SELECT COUNT(*) FROM transactions WHERE status = 'submitted_for_settlement' AND updated_at < NOW() - INTERVAL '1 DAY'
43. Hacks for development
44. Example Safe migrations
45. def self.up add_index :transactions, :customer_last_name end
46. def self.up add_index :transactions, :customer_last_name end ActiveRecord::SafeSchemaStatements::UnsafeMigrationError: :add_index is NOT SAFE! Use safe_add_concurrent_index instead
47. def self.up safe_add_concurrent_index :transactions, :customer_last_name end
48. Allow bypassing def self.up unsafe_remove_column :transactions, :old_column end
49. Example Sanity specs
50. require 'spec_helper' describe 'Sanity Specs' do it "has a unique index on every public_id column" do indexes = ActiveRecord::Base.connection.select_values(<<-SQL) SELECT relname FROM pg_class WHERE relkind = 'i'" SQL ActiveRecord::Base.connection.tables.each do |table| indexes.grep(/index_#{table}_on.*public_id/).size.should eql(1), "#{table} does not have an index on public_id" end end end
51. Summary • Check for invariants
52. Summary • Check for invariants • Focus on data validity
53. Summary • Check for invariants • Focus on data validity • Write checks outside of the normal flow
54. Summary • Check for invariants • Focus on data validity • Write checks outside of the normal flow • Find your own examples
55. Questions? paul@braintreepayments.com @pgr0ss
56. Photo Credits Mars Climate Orbiter: Courtesy NASA/JPL-Caltech World of Warcraft: ©2004 Blizzard Entertainment, Inc. Bugs: flickr.com/photos/editor/11341534765 (CC BY 2.0) NYSE: flickr.com/photos/ilamont/5538510845 (CC BY 2.0) Testing: flickr.com/photos/sidelong/246816211 (CC BY 2.0) Kids don't float: flickr.com/photos/iluvcocacola/535258842 (CC BY-NC-ND 2.0) Pillars: flickr.com/photos/tom-carden/80262 (CC BY-NC-SA 2.0)
57. Photo Credits CDs: flickr.com/photos/swanksalot/2704017177 (CC BY-SA 2.0) Fence: flickr.com/photos/clonedmilkmen/2971765601 (CC BY- SA 2.0) Hacking: flickr.com/photos/adulau/9464930917 (CC BY-SA 2.0) Safety First: flickr.com/photos/krossbow/434064539 (CC BY 2.0)