Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Supercharge your Automated Testing with Compone...

Dan J
July 17, 2017

Supercharge your Automated Testing with Component and Integration Tests

These slides are for a talk I gave at AnDevCon DC 2017:
http://www.andevcon.com/dc2017/sessions#SuperchargeYourAutomatedTestingwithComponentandIntegrationTests

You can hear a video of the presentation here:
https://www.youtube.com/watch?v=vTzAFoQQcCo

------------------------------------

SUMMARY

You've probably heard of unit tests and UI tests, but what about component and integration tests?

Component and integration tests are critical layers in the test automation pyramid. They speed up your development and release cycles, reduce risk in your project, and they even provide hidden benefits like infrastructure monitoring!

This talk covers:
- why we write automated tests
- the differences between unit tests, component tests, integration tests and UI tests
- how to decide which kinds of tests to write
- the secret benefits of component and integration tests.

Themes in this talk are:
- don't be ideological, know when and why to write automated tests
- unit tests often leave gaps and need to be refactored whenever the implementation changes, so higher level tests are important ("the whole is greater than the sum of the parts")
- it's important to automate everything, this is a key skill in multiplying your impact as a developer.

Dan J

July 17, 2017
Tweet

More Decks by Dan J

Other Decks in Programming

Transcript

  1. Why write automated tests? • Deliver faster and higher quality

    • Your product is always ready to deploy • Capture institutional knowledge in tests • Automation hugely improves productivity
  2. 05 The Testing Pyramid Slower Higher level Better QA Harder

    to debug Faster More specific Less useful for QA Better for debugging
  3. The Testing Pyramid - Android • Android UI Tests, Real

    Servers • Android Tests, Real Servers • JVM/Robolectric Tests, Mock Data • Android Tests, Mock Data
  4. @Test public void testLoginSmokeTest() { TestAccountModel testAccount = testAccountManager.get(TestAccountType.blackbox); new

    LoginAndFtuxRobot() .skipWelcomeScreen() .username(testAccount.username) .password(testAccount.password) .login() .assertTermsAndConditionsShown() .acceptTermsAndConditions(); }
  5. @Test public void testLoginSmokeTest() { TestAccountModel testAccount = testAccountManager.get(TestAccountType.blackbox); new

    LoginAndFtuxRobot() .skipWelcomeScreen() .username(testAccount.username) .password(testAccount.password) .login() .assertTermsAndConditionsShown() .acceptTermsAndConditions(); }
  6. @Test public void testLoginSmokeTest() { TestAccountModel testAccount = testAccountManager.get(TestAccountType.blackbox); new

    LoginAndFtuxRobot() .skipWelcomeScreen() .username(testAccount.username) .password(testAccount.password) .login() .assertTermsAndConditionsShown() .acceptTermsAndConditions(); }
  7. @Test public void testLoginSmokeTest() { TestAccountModel testAccount = testAccountManager.get(TestAccountType.blackbox); new

    LoginAndFtuxRobot() .skipWelcomeScreen() .username(testAccount.username) .password(testAccount.password) .login() .assertTermsAndConditionsShown() .acceptTermsAndConditions(); }
  8. @Test public void testLoginSmokeTest() { TestAccountModel testAccount = testAccountManager.get(TestAccountType.blackbox); new

    LoginAndFtuxRobot() .skipWelcomeScreen() .username(testAccount.username) .password(testAccount.password) .login() .assertTermsAndConditionsShown() .acceptTermsAndConditions(); }
  9. Pros • Avoid tedious manual QA testing • Faster releases

    after code freeze • High confidence that basic features work Cons • Fail if your APIs/environment are flakey • Slow build/test cycle for developers • Hard to debug when something goes wrong 05 Black Box Tests
  10. • Need to find bugs early • Tests need to

    be super fast 05 Problem 2 – Developers Need Tests Too
  11. @Test public void testSafeCompareEqual() { assertThat(NumberUtils.safeCompare(null, null)).isEqualTo(0); assertThat(NumberUtils.safeCompare(21, 21)).isEqualTo(0); }

    @Test public void testSafeCompareLessThan() { // Right param is less than left param assertThat(NumberUtils.safeCompare(21, null)).isEqualTo(-1); assertThat(NumberUtils.safeCompare(23, 21)).isEqualTo(-1); } @Test public void testSafeCompareMoreThan() { // Left param is less than right param assertThat(NumberUtils.safeCompare(null, 21)).isEqualTo(1); assertThat(NumberUtils.safeCompare(21, 22)).isEqualTo(1); }
  12. @Test public void testSafeCompareEqual() { assertThat(NumberUtils.safeCompare(null, null)).isEqualTo(0); assertThat(NumberUtils.safeCompare(21, 21)).isEqualTo(0); }

    @Test public void testSafeCompareLessThan() { // Right param is less than left param assertThat(NumberUtils.safeCompare(21, null)).isEqualTo(-1); assertThat(NumberUtils.safeCompare(23, 21)).isEqualTo(-1); } @Test public void testSafeCompareMoreThan() { // Left param is less than right param assertThat(NumberUtils.safeCompare(null, 21)).isEqualTo(1); assertThat(NumberUtils.safeCompare(21, 22)).isEqualTo(1); }
  13. @Test public void testDecrypt() { byte[] decryptedData = decrypt(ENCRYPTED_DATA); String

    decryptedText = new String(decryptedData); assertThat("expectedUnencryptedText").isEqualTo(decryptedText); }
  14. @Captor ArgumentCaptor<Exception> exceptionCaptor; @Test public void testTransactionsErrorWithNullReferenceId() { TransactionsResponse event

    = new TransactionsResponse(new BaseError("error message")); fragment.receiveTransactions(event); assertThat(fragment.loadingMessage).isVisible(); assertThat(fragment.loadingMessage).hasText( "Unable to load transactions"); verify(analytics).trackError( "Unable to load transactions", "error message"); verify(loggingUtil).logHandledException(exceptionCaptor.capture()); assertThat(exceptionCaptor.getValue().getMessage()) .isEqualTo("Unable to load transactions"); }
  15. @Captor ArgumentCaptor<Exception> exceptionCaptor; @Test public void testTransactionsErrorWithNullReferenceId() { TransactionsResponse event

    = new TransactionsResponse(new BaseError("error message")); fragment.receiveTransactions(event); assertThat(fragment.loadingMessage).isVisible(); assertThat(fragment.loadingMessage).hasText( "Unable to load transactions"); verify(analytics).trackError( "Unable to load transactions", "error message"); verify(loggingUtil).logHandledException(exceptionCaptor.capture()); assertThat(exceptionCaptor.getValue().getMessage()) .isEqualTo("Unable to load transactions"); }
  16. @Captor ArgumentCaptor<Exception> exceptionCaptor; @Test public void testTransactionsErrorWithNullReferenceId() { TransactionsResponse event

    = new TransactionsResponse(new BaseError("error message")); fragment.receiveTransactions(event); assertThat(fragment.loadingMessage).isVisible(); assertThat(fragment.loadingMessage).hasText( "Unable to load transactions"); verify(analytics).trackError( "Unable to load transactions", "error message"); verify(loggingUtil).logHandledException(exceptionCaptor.capture()); assertThat(exceptionCaptor.getValue().getMessage()) .isEqualTo("Unable to load transactions"); }
  17. @Captor ArgumentCaptor<Exception> exceptionCaptor; @Test public void testTransactionsErrorWithNullReferenceId() { TransactionsResponse event

    = new TransactionsResponse(new BaseError("error message")); fragment.receiveTransactions(event); assertThat(fragment.loadingMessage).isVisible(); assertThat(fragment.loadingMessage).hasText( "Unable to load transactions"); verify(analytics).trackError( "Unable to load transactions", "error message"); verify(loggingUtil).logHandledException(exceptionCaptor.capture()); assertThat(exceptionCaptor.getValue().getMessage()) .isEqualTo("Unable to load transactions"); }
  18. @Captor ArgumentCaptor<Exception> exceptionCaptor; @Test public void testTransactionsErrorWithNullReferenceId() { TransactionsResponse event

    = new TransactionsResponse(new BaseError("error message")); fragment.receiveTransactions(event); assertThat(fragment.loadingMessage).isVisible(); assertThat(fragment.loadingMessage).hasText( "Unable to load transactions"); verify(analytics).trackError( "Unable to load transactions", "error message"); verify(loggingUtil).logHandledException(exceptionCaptor.capture()); assertThat(exceptionCaptor.getValue().getMessage()) .isEqualTo("Unable to load transactions"); }
  19. @Test public void testSecureWindowFlagSet() { int FLAG_SECURE = WindowManager.LayoutParams.FLAG_SECURE; ShadowWindow

    shadowWindow = shadowOf(activity.getWindow()); assertThat(shadowWindow.getFlag(FLAG_SECURE)).isTrue(); }
  20. @Mock public CardControlsFeature feature; @Test public void testLockedCardFeatureOn() { when(feature.isCardLockEnabled()).thenReturn(true);

    assertThat(cardLock).isVisible(); assertThat(cardLock).hasText("Card is Locked"); assertThat(rewards).isGone(); } @Test public void testLockedCardFeatureOff() { when(feature.isCardLockEnabled()).thenReturn(false); assertThat(cardLock).isGone(); assertThat(rewards).isVisible(); }
  21. @Mock public CardControlsFeature feature; @Test public void testLockedCardFeatureOn() { when(feature.isCardLockEnabled()).thenReturn(true);

    assertThat(cardLock).isVisible(); assertThat(cardLock).hasText(”This Card is Locked"); assertThat(rewards).isGone(); } @Test public void testLockedCardFeatureOff() { when(feature.isCardLockEnabled()).thenReturn(false); assertThat(cardLock).isGone(); assertThat(rewards).isVisible(); }
  22. @Mock public CardControlsFeature feature; @Test public void testLockedCardFeatureOn() { when(feature.isCardLockEnabled()).thenReturn(true);

    assertThat(cardLock).isVisible(); assertThat(cardLock).hasText(”This Card is Locked"); assertThat(rewards).isGone(); } @Test public void testLockedCardFeatureOff() { when(feature.isCardLockEnabled()).thenReturn(false); assertThat(cardLock).isGone(); assertThat(rewards).isVisible(); }
  23. @Mock public CardControlsFeature feature; @Test public void testLockedCardFeatureOn() { when(feature.isCardLockEnabled()).thenReturn(true);

    assertThat(cardLock).isVisible(); assertThat(cardLock).hasText(”This Card is Locked"); assertThat(rewards).isGone(); } @Test public void testLockedCardFeatureOff() { when(feature.isCardLockEnabled()).thenReturn(false); assertThat(cardLock).isGone(); assertThat(rewards).isVisible(); }
  24. @Mock public CardControlsFeature feature; @Test public void testLockedCardFeatureOn() { when(feature.isCardLockEnabled()).thenReturn(true);

    assertThat(cardLock).isVisible(); assertThat(cardLock).hasText(”This Card is Locked"); assertThat(rewards).isGone(); } @Test public void testLockedCardFeatureOff() { when(feature.isCardLockEnabled()).thenReturn(false); assertThat(cardLock).isGone(); assertThat(rewards).isVisible(); }
  25. Pros • Fast • Great for small components • Forces

    you to write clean, testable code Cons • Often need to be rewritten when refactoring components • Easy to miss things in the gaps between tests 05 Unit Tests
  26. • Often don’t guarantee component behaves correctly as a whole

    • Tests are tightly coupled 05 Problem 3 – Unit Tests Can’t Do Everything
  27. public class MyTwoAPICalls { private class Part1Callback { @Override public

    void onSuccessResponse(Call<Part1Response> call, Response<Part1Response> part1Response) { bus.post(new Part2Request()); } } @Subscribe public void busListener(Part2Request event) { remote.part2(event).enqueue(new Part2Callback()); } private class Part2Callback { @Override public void onSuccessResponse(Call<Part2Response> call, Response<Part1Response> part1Response) { // Do more stuff } } }
  28. public class MyTwoAPICalls { private class Part1Callback { @Override public

    void onSuccessResponse(Call<Part1Response> call, Response<Part1Response> part1Response) { bus.post(new Part2Request()); } } @Subscribe public void busListener(Part2Request event) { remote.part2(event).enqueue(new Part2Callback()); } private class Part2Callback { @Override public void onSuccessResponse(Call<Part2Response> call, Response<Part1Response> part1Response) { // Do more stuff } } }
  29. public class MyTwoAPICalls { private class Part1Callback { @Override public

    void onSuccessResponse(Call<Part1Response> call, Response<Part1Response> part1Response) { bus.post(new Part2Request()); } } @Subscribe public void busListener(Part2Request event) { remote.part2(event).enqueue(new Part2Callback()); } private class Part2Callback { @Override public void onSuccessResponse(Call<Part2Response> call, Response<Part1Response> part1Response) { // Do more stuff } } }
  30. public class MyTwoAPICalls { private class Part1Callback { @Override public

    void onSuccessResponse(Call<Part1Response> call, Response<Part1Response> part1Response) { bus.post(new Part2Request()); } } @Subscribe public void busListener(Part2Request event) { remote.part2(event).enqueue(new Part2Callback()); } private class Part2Callback { @Override public void onSuccessResponse(Call<Part2Response> call, Response<Part1Response> part1Response) { // Do more stuff } } }
  31. public class MyTwoAPICalls { private class Part1Callback { @Override public

    void onSuccessResponse(Call<Part1Response> call, Response<Part1Response> part1Response) { bus.post(new Part2Request()); } } @Subscribe public void busListener(Part2Request event) { remote.part2(event).enqueue(new Part2Callback()); } private class Part2Callback { @Override public void onSuccessResponse(Call<Part2Response> call, Response<Part1Response> part1Response) { // Do more stuff } } }
  32. @Test public void testServiceUnavailable503Json() { stubResponse(503, "service_unavailable_503.json"); bus.post(new Request()); assertBusEventReceived();

    assertEquals("SERVICE_UNAVAILABLE", response.getStatus()); assertEquals("We are undergoing system maintenance", response.getDescription()); }
  33. @Test public void testServiceUnavailable503Json() { stubResponse(503, "service_unavailable_503.json"); bus.post(new Request()); assertBusEventReceived();

    assertEquals("SERVICE_UNAVAILABLE", response.getStatus()); assertEquals("We are undergoing system maintenance", response.getDescription()); }
  34. • Difficult to test UI flows • Hermetic tests to

    avoid flakey tests 05 Component Test Playbook
  35. @Test public void testLoginSuccess() { TestAccountModel testAccount = testAccountManager.get(TestAccountType.blackbox); new

    LoginAndFtuxRobot() .skipWelcomeScreen() .username(testAccount.username) .password(testAccount.password) .login() .assertTermsAndConditionsShown(); }
  36. @Test public void testLoginSuccess() { stubberator.stubItAll(mockAccount); activityRule.launchActivity( new Intent(applicationContext, SplashActivity.class));

    new LoginAndFtuxRobot() .username("user") .password("pwd") .login() .assertTermsAndConditionsScreenShown(); }
  37. @Test public void testLoginSuccess() { stubberator.stubItAll(mockAccount); activityRule.launchActivity( new Intent(applicationContext, SplashActivity.class));

    new LoginAndFtuxRobot() .username("user") .password("pwd") .login() .assertTermsAndConditionsScreenShown(); }
  38. @Test public void testLoginSuccess() { stubberator.stubItAll(mockAccount); activityRule.launchActivity( new Intent(applicationContext, SplashActivity.class));

    new LoginAndFtuxRobot() .username("user") .password("pwd") .login() .assertTermsAndConditionsScreenShown(); }
  39. @Test public void testLoginIncorrectUsernameOrPassword() { stubberator.stubItAll(mockAccount); stubFor(StubMappings.login().willReturn(aResponse() .withStatus(401).withBody(gson.toJson(ERROR_RESPONSE)))); activityRule.launchActivity( new

    Intent(applicationContext, SplashActivity.class)); new LoginAndFtuxRobot(). .username("user") .password("pwd") .login() .assertTermsAndConditionsScreenShown(); }
  40. @Test public void testLoginIncorrectUsernameOrPassword() { stubberator.stubItAll(mockAccount); stubFor(StubMappings.login().willReturn(aResponse() .withStatus(401).withBody(gson.toJson(ERROR_RESPONSE)))); activityRule.launchActivity( new

    Intent(applicationContext, SplashActivity.class)); new LoginAndFtuxRobot(). .username("user") .password("pwd") .login() .assertInvalidUsernameOrPassword(); }
  41. Pros • Validate component correctness at its interfaces • No

    need to update tests when refactoring component internals • Component UI tests aren’t flakey • Screenshots work as documentation Cons • Slower than unit tests 05 Component & Component UI Tests
  42. • Your project slips if you do API integration testing

    too late • Test environment issues 05 Problem 4 – APIs Are Hard
  43. @Test public void testCardLockStatusUpdateSuccess() { Card card = loginAndGetCard(TestAccountType.card_lock_eligible); boolean

    wasCardOriginallyLocked = card.isCardLocked(); updateCardLockStatus(!wasCardOriginallyLocked, card); // Now change it back to the original lock status again updateCardLockStatus(wasCardOriginallyLocked, card); }
  44. @Test public void testCardLockStatusUpdateSuccess() { Card card = loginAndGetCard(TestAccountType.card_lock_eligible); boolean

    wasCardOriginallyLocked = card.isCardLocked(); updateCardLockStatus(!wasCardOriginallyLocked, card); // Now change it back to the original lock status again updateCardLockStatus(wasCardOriginallyLocked, card); }
  45. @Test public void testCardLockStatusUpdateSuccess() { Card card = loginAndGetCard(TestAccountType.card_lock_eligible); boolean

    wasCardOriginallyLocked = card.isCardLocked(); updateCardLockStatus(!wasCardOriginallyLocked, card); // Now change it back to the original lock status again updateCardLockStatus(wasCardOriginallyLocked, card); }
  46. @Test public void testCardLockStatusUpdateSuccess() { Card card = loginAndGetCard(TestAccountType.card_lock_eligible); boolean

    wasCardOriginallyLocked = card.isCardLocked(); updateCardLockStatus(!wasCardOriginallyLocked, card); // Now change it back to the original lock status again updateCardLockStatus(wasCardOriginallyLocked, card); }
  47. @Test public void testCardLockStatusUpdateSuccess() { Card card = loginAndGetCard(TestAccountType.card_lock_eligible); boolean

    wasCardOriginallyLocked = card.isCardLocked(); updateCardLockStatus(!wasCardOriginallyLocked, card); // Now change it back to the original lock status again updateCardLockStatus(wasCardOriginallyLocked, card); }
  48. • Check the API works as expected • Add useful

    assertions 05 Integration Test Playbook
  49. @Test public void testCardsSuccess() { loginAndGetCards(testAccount); assertSuccessResponse(response); List<Card> cards =

    response.getCards(); assertFalse("No cards were returned for " + testAccount, cards.isEmpty()); for (Card card : cards) { assertNotNull( "reference_id is missing on cards API, ” + ”transactions and receipt images will not be available", card.getReferenceId()); } }
  50. @Test public void testCardsSuccess() { loginAndGetCards(testAccount); assertSuccessResponse(response); List<Card> cards =

    response.getCards(); assertFalse("No cards were returned for " + testAccount, cards.isEmpty()); for (Card card : cards) { assertNotNull( "reference_id is missing on cards API, ” + ”transactions and receipt images will not be available", card.getReferenceId()); } }
  51. @Test public void testCardsSuccess() { loginAndGetCards(testAccount); assertSuccessResponse(response); List<Card> cards =

    response.getCards(); assertFalse("No cards were returned for " + testAccount, cards.isEmpty()); for (Card card : cards) { assertNotNull( "reference_id is missing on cards API, ” + ”transactions and receipt images will not be available", card.getReferenceId()); } }
  52. • Check the API works as expected • Add useful

    assertions • Use them for environment monitoring! 05 Integration Test Playbook
  53. Pros • Trivial to write if you have component tests

    • Automatically diagnose API issues • Easy to turn into API monitoring tools • Useful for validating older client versions still work Cons • Fail if your APIs/environment are flakey 05 Integration Tests
  54. Capital One Wallet Git Commits Pull Request PR Merged Assemble

    APKs and Test APKs Integration Tests Component Tests Component UI Tests Blackbox Tests ✓ ✓ ✓ ✓ ✓ ✓ Component UI Tests Unit Tests Component Tests ✓ ✓ ✓ Unit Tests ✓
  55. • Every bug needs an automated test (Production, QA, master)

    • Write tests at the same time as coding • Write a failing test before making it pass • Write tests as low in the testing pyramid as possible • Run tests before merging pull requests 05 Golden Rules
  56. 05 Capital One Wallet • Building a great app •

    Launch partner for FourSquare Pilgrim SDK • Top rated finance app by a US bank • First US bank with native Android contactless payments • Passionate about • Automated testing and continuous integration • Learning and sharing