[DjangoCon Europe 2016] To mock or not to mock, that is the question

Mocking is a very powerful testing concept that has some dangerous pitfalls. There are obvious use cases where mocks are an absolute requirement to be able to test a part of the app. Nevertheless sometimes apparently useful mocks can yield erroneous test results. This talk goes into deeper detail on the trade-off of using mocks in testing.

Transcript

1. T o
 
 or not to
 
 that is the question MO CK MO CK
2. @anabalica a treatise narrated by
 
 from Potato of Londontowne. ANA BALICA
3. Thou 
 shalt 
 write 
 tests. “ ” William Shakespeare
4. Chapter one
5. Mocks simulate
 the looks and behaviour 
 of real objects
6. REAL fig.1 fig.2 MOCK
7. mocks != stubs
8. unittest.mock # Python 3.3 mock # Python 2.x
9. Mock()
10. Mock >>> from mock import Mock >>> m = Mock() >>> m.foo = 1 >>> m.foo 1 >>> m.bar <Mock name='mock.bar' id='4310136016'>
11. Mock() MagicMock()
12. __lt__ __gt__ __len__ __iter__ __bool__ __str__ __int__ __hash__ __exit__ __sizeof__
13. Mock() MagicMock() patch()
14. Patching from mock import patch with patch('rainbow.Pony') as MockPony: MockPony.return_value = 42
15. Patching 'rainbow.Pony' # creatures.py class Pony: pass # rainbow.py from creatures import Pony pony = Pony()
16. Chapter two
17. Good mocks
18. System calls with mock.patch.dict("foo.os.environ", {"LOCAL_ENV": "production"}): conf = Configuration() self.assertEqual(conf.env, "production") os.environ
19. Streams @mock.patch('sys.stdout', new=StringIO) def test_silly_logger(self, mock_stdout): logger.print_refs() self.assertEqual(mock_stdout.getvalue(), 'Some refs') sys.stdout
20. HTTP requests /responses requests.get with mock.patch("foo.requests.get") as mock_get: mock_get.return_value = mock.Mock(status_code=200) status = is_it_down(“http://djangocon.eu") self.assertFalse(status)
21. IO operations m = mock.mock_open(read_data="DjangoCon \o/“) with mock.patch("foo.open", m, create=True): with open("bar.txt") as h: result = h.read() self.assertEqual(result, "DjangoCon \o/") open
22. Clocks, time, timezones from django.utils import timezone tz = pytz.timezone('Asia/Tokyo') dt = datetime.datetime(2016, 1, 1, tzinfo=tz) with mock.patch.object(timezone, 'now', return_value=dt): print timezone.now() timezone.now
23. Unpredictable results with mock.patch('foo.random.random', return_value=0.42): result = throw_dice() self.assertEqual(result, 0.42) random.random
24. ✓ System calls ✓ Streams ✓ HTTP requests ✓ IO operations ✓ Clocks, time, timezones ✓ Unpredictable results
25. Why we like them ✓ Save time ✓ Make impossible possible ✓ Exclude external dependencies
26. Chapter three
27. Bad mocks
28. with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertTrue(saved_pony.age, 3) mock_save.assert_called_once() Problems?
29. with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertTrue(saved_pony.age, 3) mock_save.assert_called_once() Problems?
30. Problems? with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once()
31. Problems? with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once()
32. Problems? with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_twice()
33. Problems? with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.make_me_a_sandwich()
34. ¯\_(ツ)_/¯
35. Solution 1 with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) mock_save.assert_called_once_with()
36. Solution 2 with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.call_count, 1)
37. Failure with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.sandwich_count, 1)
38. Solution 3 Test Driven Development
39. Problems? with mock.patch.object(Pony, 'save_base') as mock_save: form = PonyForm(instance=pony, data=data) self.assertTrue(form.is_valid()) saved_pony = form.save() self.assertEqual(saved_pony.age, 3) self.assertEqual(mock_save.call_count, 1)
40. Tests pass? SHIP IT!
41. Maybe 
 it’s incomplete?
43. c = Client() response = c.post('/pony/', {'age': 1}) self.assertEqual(response.status_code, 201) Functional tests
44. Unit tests Functional tests Mocks
45. Conclusions
46. Mocks
 can be
 dangerous
47. Passing faulty tests give a false sense of security
48. The end