Integration and Functional Testing with LiveServerTestCase, Selenium and More.

Talk given at DjangoCon US 2013. Chicago, IL.
The video: http://www.youtube.com/watch?v=BSYg9-sxSjM

1. ~ Julien Phalip Integration and Functional Testing with LiveServerTestCase, Selenium and More. September 5th, 2013 DJANGOCON US 2013
2. CONNECTED PERSONAL OBJECTS 5/2012 About me... ‣ Worked with Django since 2007 (version 0.96). ‣ Django core committer since 2011.
3. CONNECTED PERSONAL OBJECTS 5/2012 www.nurun.com
4. CONNECTED PERSONAL OBJECTS 5/2012 Definitions (subject to debate)
5. CONNECTED PERSONAL OBJECTS 5/2012 Definitions (subject to debate) ‣ Unit testing Ensure that small parts work in isolation.
6. CONNECTED PERSONAL OBJECTS 5/2012 Definitions (subject to debate) ‣ Unit testing Ensure that small parts work in isolation. ‣ Integration testing Ensure that those parts work well together.
7. CONNECTED PERSONAL OBJECTS 5/2012 Definitions (subject to debate) ‣ Unit testing Ensure that small parts work in isolation. ‣ Integration testing Ensure that those parts work well together. ‣ Functional testing Ensure that the application works, from a user’s perspective.
8. CONNECTED PERSONAL OBJECTS 5/2012 Testing in Django (the traditional way)
9. CONNECTED PERSONAL OBJECTS 5/2012 Testing in Django (the traditional way) from django.test import TestCase class MyTests(TestCase): def test_hello(self): r = self.client.get('/hello') self.assertTrue('Hello' in r.content)
10. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase
11. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase DJANGO THREAD
12. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase Application code DJANGO THREAD
13. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase TestCase Application code DJANGO THREAD
14. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase Database TestCase Application code DJANGO THREAD
15. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database TestCase Application code DJANGO THREAD
16. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database TestCase Application code MockRequest DJANGO THREAD
17. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database TestCase Application code MockRequest DJANGO THREAD
18. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database TestCase Application code MockRequest HttpResponse DJANGO THREAD
19. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database TestCase Application code MockRequest HttpResponse DJANGO THREAD
20. CONNECTED PERSONAL OBJECTS 5/2012 django.test.TestCase “Dummy” http client (self.client) Database TestCase Application code MockRequest HttpResponse DJANGO THREAD
21. CONNECTED PERSONAL OBJECTS 5/2012 Pretty good but incomplete...
22. CONNECTED PERSONAL OBJECTS 5/2012 Pretty good but incomplete... ‣ Does not exercise the full http protocol.
23. CONNECTED PERSONAL OBJECTS 5/2012 Pretty good but incomplete... ‣ Does not exercise the full http protocol. ‣ Does not allow to test the user interface.
24. And then came LiveServerTestCase...
25. And then came LiveServerTestCase...
26. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase
27. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase LiveServerTestCase DJANGO THREAD #1
28. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase LiveServerTestCase Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
29. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client LiveServerTestCase Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
30. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
31. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
32. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server
33. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
34. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
35. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
36. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
37. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
38. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase “Real” http client Database LiveServerTestCase http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
39. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase Database LiveServerTestCase http python-requests Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
40. CONNECTED PERSONAL OBJECTS 5/2012 django.test.LiveServerTestCase Database LiveServerTestCase http Application code DJANGO THREAD #2 DJANGO THREAD #1 Live http server wsgi
41. CONNECTED PERSONAL OBJECTS 5/2012 Some important details ‣ Live server listens to localhost:8081 by default ‣ Address can be overridden: ./manage.py test --liveserver=localhost:8082 ./manage.py test --liveserver=localhost:9000-9200 ‣ Address accessible from the test’s code: self.live_server_url
42. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests
43. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests ‣ Testing REST APIs:
44. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests ‣ Testing REST APIs: import requests class MyTests(LiveServerTestCase): def test_api_user(self): r = requests.get( self.live_server_url + '/api/user/1/') self.assertEqual(r.status_code, 200) self.assertEqual(r.json, {'username': 'john.doe', 'email': 'john@doe.com'})
45. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests
46. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests ‣ Testing OAUTH workflows:
47. CONNECTED PERSONAL OBJECTS 5/2012 Testing with python-requests ‣ Testing OAUTH workflows: import requests from requests_oauthlib import OAuth2 from oauthlib.oauth2 import BackendApplicationClient class MyTests(LiveServerTestCase): def test_oauth(self): backend = BackendApplicationClient('mybackend') oauth2 = OAuth2(client=backend, token=TOKEN) r = requests.get( self.live_server_url + '/oauth', auth=oauth2) self.assertEqual(r.status_code, 200) self.assertTrue('Hello' in r.content)
48. Browser automators
49. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators?
50. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer to real Web user environment.
51. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded.
52. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded. ‣ Javascript code & Ajax calls implicitly get executed.
53. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded. ‣ Javascript code & Ajax calls implicitly get executed. ‣ User interactions can be tested.
54. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded. ‣ Javascript code & Ajax calls implicitly get executed. ‣ User interactions can be tested. ‣ Browser compatibility can be tested.
55. CONNECTED PERSONAL OBJECTS 5/2012 Why use browser automators? ‣ Closer to real Web user environment. ‣ Media & static files (CSS/images/etc.) implicitly get loaded. ‣ Javascript code & Ajax calls implicitly get executed. ‣ User interactions can be tested. ‣ Browser compatibility can be tested. ‣ It’s fun!
56. CONNECTED PERSONAL OBJECTS 5/2012 Selenium
57. CONNECTED PERSONAL OBJECTS 5/2012 Selenium ‣ API to become W3C standard (currently in draft)
58. CONNECTED PERSONAL OBJECTS 5/2012 Selenium ‣ API to become W3C standard (currently in draft) ‣ Client libraries in Java, C#, .NET, Ruby, PHP, Perl, Javascript and Python.
59. CONNECTED PERSONAL OBJECTS 5/2012 Selenium ‣ API to become W3C standard (currently in draft) ‣ Client libraries in Java, C#, .NET, Ruby, PHP, Perl, Javascript and Python. ‣ Works across multiple browsers (Firefox, Chrome, IE, Safari, Opera)
60. CONNECTED PERSONAL OBJECTS 5/2012 Selenium ‣ API to become W3C standard (currently in draft) ‣ Client libraries in Java, C#, .NET, Ruby, PHP, Perl, Javascript and Python. ‣ Works across multiple browsers (Firefox, Chrome, IE, Safari, Opera) ‣ Easy to install: pip install selenium
61. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver import \ WebDriver class MyTests(LiveServerTestCase):
62. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver import \ WebDriver class MyTests(LiveServerTestCase): def setUp(self): self.driver = WebDriver() super(MyTests, self).setUp() def tearDown(self): self.driver.quit() super(MyTests, self).tearDown()
63. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver import \ WebDriver class MyTests(LiveServerTestCase): def setUp(self): self.driver = WebDriver() super(MyTests, self).setUp() def tearDown(self): self.driver.quit() super(MyTests, self).tearDown()
64. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver import \ WebDriver class MyTests(LiveServerTestCase): ...
65. CONNECTED PERSONAL OBJECTS 5/2012 from django.test import LiveServerTestCase from selenium.webdriver.firefox.webdriver import \ WebDriver class MyTests(LiveServerTestCase): def test_hello(self): self.driver.get( self.live_server_url + '/hello') e = self.driver.find_element_by_id('btn') e.click() ... ...
66. CONNECTED PERSONAL OBJECTS 5/2012 DEMO Django Core Test Suite
67. Tips and goodies
68. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle
69. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ https://github.com/bfirsh/needle
70. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ https://github.com/bfirsh/needle ‣ Based on Selenium.
71. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ https://github.com/bfirsh/needle ‣ Based on Selenium. ‣ Tests visuals by comparing screenshots.
72. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ Test the looks of an entire page...
73. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ Test the looks of an entire page...
74. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ Or just parts of it.
75. CONNECTED PERSONAL OBJECTS 5/2012 Testing visuals with Needle ‣ Or just parts of it.
76. CONNECTED PERSONAL OBJECTS 5/2012 from needle.cases import NeedleTestCase from from selenium.webdriver import firefox class MyTests(NeedleTestCase, LiveServerTestCase): driver_command_executor = firefox.ExtensionConnection( "127.0.0.1", firefox.FirefoxProfile()) def test_homepage_menu(self): self.driver.get(self.live_server_url) self.assertScreenshot('ul.menu', 'menu') Testing visuals with Needle
77. CONNECTED PERSONAL OBJECTS 5/2012 ====================================================================== FAIL: test_homepage_menu (myapp.tests.MyTests) ---------------------------------------------------------------------- Traceback (most recent call last): File "myapp/tests.py", line 115, in test_homepage_menu self.assertScreenshot('ul.menu', 'menu') File "/.virtualenvs/myproject/src/needle/needle/cases.py", line 99, in assertScreenshot pass File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/ python2.6/contextlib.py", line 23, in __exit__ self.gen.next() File "/.virtualenvs/myproject/src/needle/needle/cases.py", line 160, in compareScreenshot % (filename, distance)) AssertionError: The saved screenshot for 'menu' did not match the screenshot captured (by a distance of 26.17) Testing visuals with Needle
78. CONNECTED PERSONAL OBJECTS 5/2012 Test coverage
79. CONNECTED PERSONAL OBJECTS 5/2012 Test coverage ‣ Yes, it does work too.
80. CONNECTED PERSONAL OBJECTS 5/2012 Test coverage ‣ Yes, it does work too. $ coverage run manage.py test \ myapp.SeleniumTests $ coverage report
81. CONNECTED PERSONAL OBJECTS 5/2012 Headless mode for continuous integration
82. CONNECTED PERSONAL OBJECTS 5/2012 Headless mode for continuous integration ‣ Install a browser and fonts: $ sudo apt-get install -y firefox xfonts-100dpi xfonts-75dpi xfonts- scalable xfonts-cyrillic
83. CONNECTED PERSONAL OBJECTS 5/2012 Headless mode for continuous integration ‣ Install a browser and fonts: $ sudo apt-get install -y firefox xfonts-100dpi xfonts-75dpi xfonts- scalable xfonts-cyrillic ‣ Install Xvfb: $ sudo apt-get install -y xvfb
84. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Either with the Xvfb command: $ Xvfb -ac :99 2>/dev/null & $ DISPLAY=:99 ./manage.py test Headless mode for continuous integration
85. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Either with the Xvfb command: $ Xvfb -ac :99 2>/dev/null & $ DISPLAY=:99 ./manage.py test ‣ ... or with Jenkins: Xvfb plugin. Headless mode for continuous integration
86. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Either with the Xvfb command: $ Xvfb -ac :99 2>/dev/null & $ DISPLAY=:99 ./manage.py test ‣ ... or with Jenkins: Xvfb plugin. ‣ ... or with Travis: before_install: - "export DISPLAY=:99.0" - "sh -e /etc/init.d/xvfb start" Headless mode for continuous integration
87. CONNECTED PERSONAL OBJECTS 5/2012 ‣ ... or with PyVirtualDisplay: from pyvirtualdisplay import Display class MyTests(LiveServerTestCase) def setUp(self): self.display = Display( 'xvfb', visible=1,size=(1280, 1024)) self.display.start() def tearDown(self): self.display.stop() Headless mode for continuous integration
88. CONNECTED PERSONAL OBJECTS 5/2012 Sauce Labs
89. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Runs remotely in the cloud (no local browser needed). Sauce Labs
90. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Runs remotely in the cloud (no local browser needed). ‣ Gives access to multiple browsers (IE, FF, Opera, Chrome, Safari...) and multiple platforms (Windows, Linux, OSX, Android, iPad, iPhone). Sauce Labs
91. CONNECTED PERSONAL OBJECTS 5/2012 ‣ Runs remotely in the cloud (no local browser needed). ‣ Gives access to multiple browsers (IE, FF, Opera, Chrome, Safari...) and multiple platforms (Windows, Linux, OSX, Android, iPad, iPhone). ‣ Allows tests to be run in parallel. Sauce Labs
92. CONNECTED PERSONAL OBJECTS 5/2012
93. CONNECTED PERSONAL OBJECTS 5/2012
94. Not just Selenium...
95. CONNECTED PERSONAL OBJECTS 5/2012 Ghost.py ‣ http://jeanphix.me/Ghost.py/
96. CONNECTED PERSONAL OBJECTS 5/2012 Ghost.py from ghost import Ghost class MyTests(LiveServerTestCase): def setUp(self): self.ghost = Ghost() super(MyTests, self).setUp() def test_hello(self): self.ghost.open(self.live_server_url + '/hello') self.assertTrue('Hello' in self.ghost.content) ‣ http://jeanphix.me/Ghost.py/
97. CONNECTED PERSONAL OBJECTS 5/2012 CasperJS ‣ http://casperjs.org/ ‣ https://github.com/dobarkod/django-casper
98. CONNECTED PERSONAL OBJECTS 5/2012 CasperJS from casper.tests import CasperTestCase class MyTests(CasperTestCase): def test_something(self): self.assertTrue(self.casper('test.js')) ‣ http://casperjs.org/ ‣ https://github.com/dobarkod/django-casper
99. CONNECTED PERSONAL OBJECTS 5/2012 Splinter ‣ http://splinter.cobrateam.info
100. CONNECTED PERSONAL OBJECTS 5/2012 Splinter from splinter import Browser class MyTests(LiveServerTestCase): def setUp(self): self.browser = Browser() super(MyTests, self).setUp() def test_hello(self): self.browser.visit(self.live_server_url + '/hello') self.assertTrue( self.browser.is_text_present('Hello')) ‣ http://splinter.cobrateam.info
101. Epilogue...
102. CONNECTED PERSONAL OBJECTS 5/2012 A word of caution...
103. CONNECTED PERSONAL OBJECTS 5/2012 A word of caution... ‣ Selenium tests can be a little flaky.
104. CONNECTED PERSONAL OBJECTS 5/2012 A word of caution... ‣ Selenium tests can be a little flaky. ‣ Integration & functional tests are slow. Use them with moderation.
105. CONNECTED PERSONAL OBJECTS 5/2012 A word of caution... ‣ Selenium tests can be a little flaky. ‣ Integration & functional tests are slow. Use them with moderation. ‣ Use LiveServerTestCase or Selenium only for what the dummy client cannot already achieve.
106. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important.
107. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important. ‣ Increase your confidence in your code.
108. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important. ‣ Increase your confidence in your code. ‣ Increase your test coverage.
109. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important. ‣ Increase your confidence in your code. ‣ Increase your test coverage. ‣ Test the integration frontend/backend.
110. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important. ‣ Increase your confidence in your code. ‣ Increase your test coverage. ‣ Test the integration frontend/backend. ‣ Ensure the user interface works.
111. CONNECTED PERSONAL OBJECTS 5/2012 Integration & functional tests are important. ‣ Increase your confidence in your code. ‣ Increase your test coverage. ‣ Test the integration frontend/backend. ‣ Ensure the user interface works. ‣ Have fun in the process!
112. CONNECTED PERSONAL OBJECTS 5/2012 Photo credits ‣ http://www.flickr.com/photos/t0msk/3148160756/ ‣ http://www.flickr.com/photos/loozrboy/7311771718/ ‣ http://www.flickr.com/photos/johnonolan/5836211096/ ‣ http://www.flickr.com/photos/randomskk/3057595640/
113. @julienphalip http:/ /julienphalip.com http:/ /nurun.com Thank you!