Upgrade to Pro — share decks privately, control downloads, hide ads and more …

E2E2U: Slack’s journey to developer-owned end-t...

E2E2U: Slack’s journey to developer-owned end-to-end testing

Two years ago we presented on an approach that makes authoring reliable Android UI tests easier. To date, Slack android developers have written over 700 UI tests and all of them run as part of a merge-blocking (and, most importantly, reliable) pull-request check. Having conquered this level, we set our sights on the top of our test pyramid - automated e2e tests. This test suite was still being maintained by a single hero developer, who, through sheer grit and determination, didn’t let this important part of our release process die. Armed with experience in functional UI testing, we embarked on an adventure to make our end-to-end test suite a shared responsibility of the entire team. Our journey has taken us through the fields of dagger, the bog of espresso IdlingResources, the inferno of proguard and, perhaps the most challenging of them all, cultural aspects of quality assurance. We look forward to sharing where we are in our never-ending testing story.

Valera Zakharov

November 26, 2019
Tweet

More Decks by Valera Zakharov

Other Decks in Technology

Transcript

  1. NO

  2. 2015 ▸ calabash (ruby) ▸ xamarin (now appcenter.ms) ▸ cross

    platform https://www.everwilde.com/store/Calabash-Gourd-Seeds.html E2E TESTS
  3. FIRST E2E TESTS LOOKS LIKE THIS Feature: User Login Scenario:

    Unsuccessful user login Given the app has launched Then I wait for the "Login" button to appear When I enter "tstuser" into the "Username" field And I enter "qwerty" into the "Password" field And I touch "Login" Then I should see "Username you entered is incorrect" Scenario: Successful user login Given the app has launched Then I wait for the "Login" button to appear When I enter "testeruser" into the "Username" field And I enter "qwerty" into the "Password" field And I touch "Login" Then I should see "Hey testeruser!" https://www.toptal.com/mobile/android-ios-ui-testing-calabash
  4. FIRST E2E TESTS WHAT IS AN END-TO-END TEST ▸ covers

    important user journeys ▸ talks to the backend https://aws.amazon.com/solutions/case-studies/slack/
  5. FIRST E2E TESTS PROBLEMS ▸ blackbox = sign in every

    time ▸ unstable ▸ decoupled from dev workflow ▸ no synchronization with the app under test ▸ really slow http://ruffledfeathersandspilledmilk.com/wp-content/uploads/2013/11/Bella-272.jpg
  6. 2017 public void setUp() throws Exception { super.setUp(); User user

    = User.builder()…/* setters */…build(); when(mockUsersDataProvider.getUser(USER_ID)) .thenReturn(Observable.just(user)); when(mockUserPresenceManager.isUserActive(USER_ID, true)) .thenReturn(true); when(mockDndInfoDataProvider.getDndInfo(userId)) .thenReturn(Observable.just(new DndInfo(/* params */))); Channel channel = Channel.builder()…/* setters */…build(); when(mockChannelsDataProvider.getMessagingChannelObservable(CHANNEL_ID)) .thenReturn(Observable.just(channel)); when(mockChannelNameHelper.getDisplayNameObservable(eq(channel))) .thenReturn(Observable.just(CHANNEL_NAME)); launchActivity(); } slackPowerPack = new SlackPowerPack(context.getApplicationContext()); slackPowerPack.users() .addUser(User.builder()…/* setters */…build()) .addUser(User.builder()…/* setters */…build()) .pack(); slackPowerPack.channels() .addChannel(Channel.builder()…/* setters */…build()) .addChannel(Channel.builder()…/* setters */…build()) .pack(); POWER PACK
  7. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 +VALERA

    ESPRESSO +KEVIN POWER PACK E2E TO ESPRESSO 2017 2018
  8. E2E TO ESPRESSO ROBOTS @Test fun signInWithValidCredentialsAndSignOut() { signInRobot .signIn(signedInUser)

    .verifyToolbarTeamName(signedInUser.teamDomain) .tapOverflowSettingsOption() .signOutOfCurrentTeam() } public ToolbarRobot signIn(TestUser testUser, boolean waitForHomeActivity) { tapSignInButton(); tapSignInManuallyButton(); enterTeamURL(testUser.teamDomain); tapNextButtonOnTeamURL(); enterEmailAddress(testUser.email); tapNextButtonOnEnterEmailAddress(); enterPassword(testUser.password); tapNextButtonOnEnterPassword(); if (waitForHomeActivity) { // Wait until activity transition between FirstSignInActivity to HomeActivity onView(isRoot()).perform(waitForActivity(HomeActivity.class, MAX_TIMEOUT)); } return new ToolbarRobot(); } https://jakewharton.com/testing-robots/
  9. E2E TO ESPRESSO KOTLIN ROBOTS? toolbarRobot .tapOverflowSettingsOption() // transition to

    settings .tapAdvancedSettings() // transition to advanced settings .tapResetLocalStorage() // transition back to home activity .verifyToolbarTeamName(signedInUser.teamDomain) toolbar { overflow(SETTINGS) } settings { setting(ADVANCED) advanced { tapResetLocalStorage() } } toolbar { verify { toolbarTeamName(signedInUser.teamDomain) } } fun toolbar(func: ToolbarRobot.() -> Unit) = ToolbarRobot().apply { func() } ¯\_(ツ)_/¯
  10. E2E TO ESPRESSO E2E TESTS CAN BE TARGETED public abstract

    class SignedInBaseTest { @Override protected void createObjectGraph() { slackApp = ApplicationProvider.getApplicationContext(); appComponent = EndToEndAppComponentBuilder.get(slackApp, getMdmComponent()); appComponent.endpointsHelper().setApiUrl(getApiUrl()); slackApp.resetAppComponent(appComponent); appComponent.inject(this); accountManager = appComponent.accountManager(); signInDataProvider = appComponent.signInDataProvider(); setupUserAndTeamData(); Account account = accountManager.getAccountWithTeamId(signedInUser.teamId); EndToEndUserComponent userComponent = appComponent.endToEndUserComponent() .loggedInUser(LoggedInUser.create(account.userId(), account.teamId(), account.enterpriseId(), account.authToken())) .idlingResourceModule(new IdlingResourceModule(IdlingResourceTestProvider.END_TO_END_IDLING_COUNTER_PROVIDER)) .build(); slackApp.addUserGraph(signedInUser.teamId, userComponent); inject(userComponent); } POINT APP AT THE GIVEN BACKEND SET UP SIGNED IN STATE
  11. E2E TO ESPRESSO SYNCHRONIZE WITH IDLINGRESOURCES ▸ OkHttp ▸ RxIdler

    ▸ IdlingExecutors https://github.com/JakeWharton/okhttp-idling-resource https://github.com/square/RxIdler
  12. E2E TO ESPRESSO IDLING EXECUTORS public final class WrappedCallable<T> implements

    Callable<T> { private final Callable<T> delegate; private final CountingIdlingResource idlingResource; public WrappedCallable(Callable<T> delegate, CountingIdlingResource idlingResource) { this.delegate = delegate; this.idlingResource = idlingResource; } @Override public T call() throws Exception { Timber.v("%s(IdlingResource).increment", idlingResource.getName()); idlingResource.increment(); T call; try { call = delegate.call(); } finally { Timber.v("%s(IdlingResource).decrement", idlingResource.getName()); idlingResource.decrement(); } return call; } }
  13. E2E TO ESPRESSO IDLING EXECUTORS @Provides @UserScopeSingleton static EventDispatcher provideMSEventDispatcher(EventHandlerFactory

    eventHandlerFactory, PersistentStore persistentStore, JsonInflater jsonInflater, @Named(EspressoModule.EVENT_DISPATCHER_IDLING_RESOURCE) CountingIdlingResource idlingResource) { PausableThreadPoolExecutor msEventsExecutor = PausableThreadPoolExecutorImpl.newSingleThreadExecutor( new NamedThreadFactory("ms-event-dispatcher-")); if (BuildConfig.DEBUG) { msEventsExecutor = new IdlingPausableThreadPoolExecutor(msEventsExecutor, idlingResource); } return new EventDispatcherImpl(eventHandlerFactory, msEventsExecutor, persistentStore, jsonInflater); }
  14. E2E TO ESPRESSO PLUGGING DIRECTLY INTO PROD CODE internal fun

    filter(constraint: CharSequence?): List<UniversalResult> { countingIdlingResource?.increment() return universalResultDataProviderLazy.get().getResults(constraint.toString(), options) .onErrorReturn { emptyList } .blockingGet() } … override fun publishResults(constraint: CharSequence?, results: FilterResults) { // This method is called by a delayed handler message, so previous results might emit // one after another. We want to ignore these backpressure updates and only take the latest. // See Filter.RequestHandler for implementation details. if (constraint == latestConstraint) { @Suppress("UNCHECKED_CAST") // We suppress this cast warning because we know it's always going to be a list of UniversalResults. displayList = results.values?.let { results.values as List<UniversalResult> } ?: emptyList notifyDataSetChanged() } countingIdlingResource?.decrement() }
  15. E2E TO ESPRESSO A SPRINKLING OF HACKS IN ROBOTS ▸

    Activity transitions ▸ Hacky “synchronization” with waitForElement fun enterChannelName(channelName: String): CreateWorkspaceRobot { waitForElementDisplayed(allOf(withId(R.id.header), withText(R.string.what_is_a_project))) waitAndTypeText(allOf(withId(R.id.input_field), withHint(R.string.project_name_example), isDisplayed()), channelName) return this } // Reduce flakiness with transition between FirstSignInActivity to HomeActivity onView(isRoot()).perform(waitForActivity(HomeActivity.class, MAX_TIMEOUT));
  16. E2E TO ESPRESSO PROMISING RESULTS ▸ vastly improved stability ▸

    speed up of 7x ▸ ability for devs to debug locally & contribute
  17. E2E TO ESPRESSO BUT STILL NOT OWNED BY DEVS ▸

    not enough stability for pre-merge ▸ no official handover
  18. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 +VALERA

    ESPRESSO +KEVIN POWER PACK E2E TO ESPRESSO PERF TESTING E2E2U 2017 2018
  19. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT ./gradlew app:assembleInternalRelease app:assembleInternalReleaseAndroidTest

    -PtestBuildTypeRelease=true 18:08:57 gcloud: 18:08:57 app: /mnt/tmp/android-e2e-full/TEST-ONLY-app-internal-x86-release.apk 18:08:57 test: /mnt/tmp/android-e2e-full/TEST-ONLY-app-internal-release-androidTest.apk
  20. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT ▸ We need

    to build an apk that ▸ is minified ▸ can be used to run instrumentation tests ▸ has code coverage turned on staging { initWith release signingConfig signingConfigs.debug // Enable code coverage for staging build variant testCoverageEnabled true proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro', 'proguard-staging-rules.pro' matchingFallbacks = ['release'] }
  21. E2E2U RUN E2E TESTS AGAINST RELEASE VARIANT if (BuildConfig.DEBUG) {

    enableStrictMode(); ThreadUtils.enableThreadChecks(); } if (buildConfig.isDebug() && !buildConfig.isStaging()) { enableStrictMode(); ThreadUtils.enableThreadChecks(); }
  22. E2E2U STABILITY: RUN AGAINST PRODUCTION BACKEND ▸ staging env =

    somewhat isolated production ▸ much more stable than dev environments ▸ downside: no test apis ▸ can’t easily manipulate it for testing (e.g. creating users, etc) https://slack.engineering/how-slack-built-shared-channels-8d42c895b19f
  23. E2E2U STABILITY: LIVING WITH SHARED STATE ▸ problem: multiple tests

    running in parallel using the same team/user ▸ shared state lives “in the cloud” ▸ not possible/practical to set up new test fixture for every test ▸ solution: provision many users and pick randomly from the set for each test
  24. E2E2U STABILITY: LIVING WITH SHARED STATE private void setUpRandomUsers(int extraUserCount)

    { final TestUser.Generator generator = getTestUserGenerator(); signedInUser = generator.generate(); secondaryUser = generator.generate(); for (int i = 0; i < extraUserCount; i++) { userCollection[i] = generator.generate(); } } class DefaultGeneratorImpl { val excluded = mutableSetOf<String>() override fun generate(): TestUser { val user = random(*excluded.toTypedArray()) excluded += user.userId return user } fun random(vararg avoidUsersWithIds: String): TestUser { // attempt (up to maxUsers times) to generate a random user that does not collide with the provided list of users. // we'd have to be really unlucky, but it is theoretically possible that we won't get it, therefore // we handle that case by throwing a RE. for (x in 0 until maxUsers) { val i = Random.nextInt(from = 3, until = idMap.size) val userId = idMap[i] ?: throw NoSuchElementException("No user id for index $i") if (userId !in avoidUsersWithIds) { return TestUser(teamDomain, “$baseUserNamePrefix$i", “[email protected]", password, userId, teamId) } } throw IllegalStateException("Could not generate random user different from $avoidUsersWithIds") }
  25. E2E2U STABILITY: THIS ONE EASY TRICK WILL MAKE YOUR TESTS

    NOT FLAKY https://github.com/TestArmada/flank gcloud app: $APK_PATH test: $TEST_APK_PATH use-orchestrator: true flaky-test-attempts: 4
  26. SLACK LAUNCHED 2013 2015 E2E TESTS 2019 +JASON 2016 +VALERA

    ESPRESSO +KEVIN POWER PACK E2E TO ESPRESSO PERF TESTING E2E2U TODAY 2017 2018
  27. TODAY AND DAGGER public abstract class SignedInBaseTest { @Override protected

    void createObjectGraph() { slackApp = ApplicationProvider.getApplicationContext(); appComponent = EndToEndAppComponentBuilder.get(slackApp, getMdmComponent()); appComponent.endpointsHelper().setApiUrl(getApiUrl()); slackApp.resetAppComponent(appComponent); appComponent.inject(this); accountManager = appComponent.accountManager(); signInDataProvider = appComponent.signInDataProvider(); setupUserAndTeamData(); Account account = accountManager.getAccountWithTeamId(signedInUser.teamId); EndToEndUserComponent userComponent = appComponent.endToEndUserComponent() .loggedInUser(LoggedInUser.create(account.userId(), account.teamId(), account.enterpriseId(), account.authToken())) .idlingResourceModule(new IdlingResourceModule(IdlingResourceTestProvider.END_TO_END_IDLING_COUNTER_PROVIDER)) .build(); slackApp.addUserGraph(signedInUser.teamId, userComponent); inject(userComponent); }
  28. TODAY BUT WE’VE COME A LONG WAY Not so long

    ago Hours of failure triage before every release Today Ship it on green (almost)
  29. YES