Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing at Scale

Testing at Scale

Ash Christopher

June 13, 2013
Tweet

More Decks by Ash Christopher

Other Decks in Technology

Transcript

  1. TESTS SHOULD BE AS SIMPLE AS POSSIBLE Don’t write test

    code that needs it’s own tests! Be clear - not clever. DRY is great for code, just not for test code.
  2. TESTS SHOULD RUN AS QUICKLY AS POSSIBLE Encourage developers to

    run tests frequently. Sometimes running subset of tests may not be enough. Instill a culture of reliance on tests.
  3. TESTS SHOULD AVOID COUPLING WITH OTHER TESTS TestCases should not

    extend TestCases in other python modules. Tests should not use helper methods which exist in other python modules.
  4. TESTS SHOULD COMMUNICATE INTENT Tests are explicitly clear about what

    they are testing. Related test methods reside in the same TestCase. Tests should not rely on documentation to explain what they are doing. Developers reading the tests should not have to leave the TestCase code to determine what the test is doing.
  5. SET UP PRE-CONDITIONS Create the environment needed for the test

    to run. • Create model instances. • Set up the state required for the test. • Create mock objects. • Patch methods, functions and objects.
  6. PERFORM OPERATION UNDER TEST Run test using the environment established

    in the pre- conditions. • Modify state. • Record return values.
  7. MAKE ASSERTIONS Make assertions about the return values or side

    effects. • Check return values against expected values. • Test that the state has been modified as expected.
  8. DO ABSOLUTELY NOTHING ELSE • Don’t modify the test conditions

    and test assertions again. • Don’t extend your test further. • Don’t create test methods that chain together main tests. • Write a new test method, and copy the pre-conditions.
  9. BENEFITS OF FOLLOWING CANONICAL FORM Each test method tests exactly

    one path through the code under test. The set of TestCases will define a clear contract with the developer. For any test that fails, the cause will be easy to pinpoint because the expected results is clearly focused.
  10. DJANGO APP called factory_boy Third-party Django application. Based on popular

    Rails library called factory_girl. Based on your own Django models. Replacement for fixtures.
  11. Only define sub-factories for data that needs to be created.

    Model relations with null=True or blank=True then this isn’t generally a business need. ONLY CREATE DATA BASED ON BUSINESS NEEDS
  12. Factories should only be responsible for creating data, not setting

    up pre-conditions for tests. FACTORIES CREATE DATA ONLY
  13. COVERAGE.PY Treat the results of coverage.py as a litmus test.

    Use it to find paths in code that you missed in your tests.
  14. IN-MEMORY SQLite DATABASE Databases in memory are extremely fast. Generally,

    back-end agnostic. More complicated when using database backend specific code.
  15. NOSE + MULTIPROCESS Run test suite in multiple processes. Split

    tests, not TestCases so your tests need to be sure to ensure tearDown() happens.
  16. AVOID COMPLEXITY IN TESTS Harder for a reader of the

    test code to understand what is going on. The test code adds functionality which itself should be tested.
  17. TEST ONE THING Six assertions testing six different behaviors. Pre-conditions

    set four times. Should have split into separate test methods. Don’t do what you see here!
  18. User UserProfile 28 x Business Accounts Site 5x Business 1x

    Business (Personal) Currency Country Customer Vendor StandardAccount Account IncomeAccount ExpenseAccount Product PaymentTerm Tax (GST) Tax (PST) Tax (TEST) Invoice InvoiceItem (1) InvoiceItem (2) Bill BillItem (1) BillItem (2) MINIMIZE PRECONDITIONS NEEDED FOR THE TEST Loaded 2x test fixtures. Modified ownership for 6 businesses.
  19. MINIMIZE PRECONDITIONS NEEDED FOR THE TEST All this database record

    manipulation to test something that doesn’t even rely on the pre-conditions!
  20. DRY DOESN’T APPLY WHEN TESTING Shared helper methods tend to

    grow over time. Shared helper methods try to be generic enough for everyone to use. Impose a lookup burden on readers of the test code. Tend to become complex over time (do your tests need tests)?
  21. AVOID UNNECESSARY COUPLING IN YOUR TESTS When you refactor test

    code, you risk breaking unrelated tests. You no longer have direct control over your pre-conditions. Anyone can affect the confidence of your tests.
  22. DON’T DEFINE DOCSTRINGS IN TESTS Docstrings in test methods make

    it hard to track down tests in output. Use simple python comments instead.
  23. TEST METHOD NAMES DESCRIBE THE TEST Makes it easy to

    identify what the test is actually testing. Gives clues about what is wrong when tests fail.
  24. FIXTURES ARE HORRIBLE Fixtures are hard to maintain as your

    project’s data models change. Tend to get re-used in tests rather than generating fixtures for each TestCase.
  25. FACTORIES ARE WAY BETTER Factories generate data using the Django

    ORM. Give us hooks into data preparation. FactoryBoy lets us define data across relationships using familiar `double- underscore` notation. We can define associations in FactoryBoy to mimic our business logic.
  26. TESTS PASS, BUT SUITE FAILS Django TestCase’s run tests in

    a database transaction which gets rolled-back. History shows you can’t rely on developers to tearDown properly. * Only applies to the `default` database.
  27. MIXINS ARE NOT INCLUDED IN TEST DISCOVERY When a TestCase

    extends a BaseTestCase, it will run all tests, even the ones in the BaseTestCase. Leads to the same tests being run many times unnecessarily. Use a mixin - it will not be run by the test runner.
  28. USE nose AND django-nose Decorate your tests and TestCases with

    @attr(<attribute>). Segment your tests based on type of tests. Selectively choose which type of tests to run using -a<attribute> on the command line.
  29. CLEANUP IS A PAIN Can not guarantee that all the

    data is cleaned up - especially when using django signals. If we want to use multi-processing for tests, we need to be sure test methods can be run in any order. Over the entire test suite, did not give us enough gains to overcome the potential pitfalls. Using multi-processing for tests gave us much better wins.