[PyCon US 2016] To mock or not to mock, that is the question

Mocking is a very powerful testing concept that has some dangerous pitfalls. There are obvious use cases where mocks are an absolute requirement to be able to test a part of the app. Nevertheless sometimes apparently useful mocks can yield erroneous test results. This talk goes into deeper detail on the trade-off of using mocks in testing.

Transcript

1. T o
 
 or not to
 
 that is the question MO CK MO CK
2. @anabalica a treatise narrated by
 
 from Potato of Londontowne. ANA BALICA
3. Thou 
 shalt 
 write 
 tests. “ ” William Shakespeare
4. Chapter one
5. Mocks simulate
 the looks and behaviour 
 of real objects
6. REAL fig.1 fig.2 MOCK
7. mocks != stubs
8. ✓ Setup ✓ Test ✓ Verify state ✓ Teardown ✓ Setup ✓ Setup expectations ✓ Test ✓ Verify expectations ✓ Verify state ✓ Teardown Stubs Mocks
9. unittest.mock # Python 3.3 mock # Python 2.x
10. Mock()
11. Mock >>> from mock import Mock >>> m = Mock() >>> m.foo = 1 >>> m.foo 1 >>> m.bar <Mock name='mock.bar' id='4310136016'>
12. Mock() MagicMock()
13. __lt__ __gt__ __len__ __iter__ __bool__ __str__ __int__ __hash__ __exit__ __sizeof__
14. Mock() MagicMock() patch()
15. from mock import patch with patch('rainbow.Pony') as MockPony: MockPony.return_value = 42 Patching
16. Patching 'rainbow.Pony' # creatures.py class Pony: pass # rainbow.py from creatures import Pony pony = Pony()
17. Mock the object where it’s used, 
 not where it came from
18. Chapter two
19. Good mocks
20. with patch.dict('os.environ', {'ANDROID_ARGUMENT': ''}): pf = Platform() self.assertTrue(pf == ‘android') os.environ System calls
21. @mock.patch('sys.stdout', new_callable=six.StringIO) def test_print_live_refs_empty(self, stdout): trackref.print_live_refs() self.assertEqual(stdout.getvalue(), 'Live References\n\n\n') sys.stdout Streams
22. request.urlopen @patch('django.utils.six.moves.urllib.request.urlopen') def test_oembed_photo_request(self, urlopen): urlopen.return_value = self.dummy_response result = wagtail_oembed("http://www.youtube.com/watch/") self.assertEqual(result['type'], 'photo') Networking
23. @patch.object(DataLoader, '_get_file_contents') def test_parse_json_from_file(self, mock_def): mock_def.return_value = ('{"a": 1, "b": 2, "c": 3}', True) output = self._loader.load_from_file('dummy_json.txt') self.assertEqual(output, dict(a=1, b=2, c=3)) json.loads IO operations
24. @mock.patch('time.sleep') def test_500_retry(self, sleep_mock): self.set_http_response(status_code=500) # Create a bucket, a key and a file with self.assertRaises(BotoServerError): k.send_file(fail_file) Clocks, time, timezones time.sleep
25. with mock.patch('random.random', return_value=0.0): with self.assertChanges(get_timeline_size, before=10, after=5): backend.add(timeline, next(self.records)) random.random Unpredictable results
26. ✓ System calls ✓ Streams ✓ Networking ✓ IO operations ✓ Clocks, time, timezones ✓ Unpredictable results
27. ✓ Save time ✓ Make impossible possible ✓ Exclude external dependencies Why we like them
28. Chapter three
29. mocks Bad
30. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertTrue(saved_pony.age, 3) mock_save.assert_called_once() Problems?
31. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertTrue(saved_pony.age, 3) mock_save.assert_called_once() Problems?
32. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once() Problems?
33. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once() Problems?
34. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_twice() Problems?
35. with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.make_me_sandwich() Problems?
36. ¯\_(ϑ)_/¯
37. Solution 1 with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once_with()
38. Solution 2 with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.call_count, 1)
39. Failure with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.sandwich_count, 1)
40. Solution 3 Test Driven Development
41. Problems? with patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.call_count, 1)
42. Tests pass? SHIP IT!
43. Maybe 
 it’s incomplete?
45. Integration tests c = Client() response = c.post('/pony/', {'age': 1}) self.assertEqual(response.status_code, 201)
46. Unit tests Integration tests Mocks
47. Only mock types 
 that you own
48. Building on
 Third-Party
 Code
49. Adapter
 layer 3rd party API Application
 objects
50. Adapter
 layer 3rd party API Application
 objects Test this cluster Mock this
51. Conclusions
52. Mocks
 can be
 dangerous
53. Passing faulty tests give a 
 false sense of security
54. The end