Share
Embed
DevFestFL 2017 - Espresso: A Screenshot is Worth 1,000 Words
Speaker: https://twitter.com/handstandsam
Event: https://devfestflorida.org/schedule/
Sam Edwards
November 11, 2017
Transcript
112 slides
-
Sam Edwards @HandstandSam • Android GDE • Architected and leading
Espresso efforts for Capital One Wallet • Opinions in this talk are my own. http://handstandsam.com -
Audience POLL: • Testing? • Espresso? • Espresso with CI?
• Screenshots? -
Outline • What is Espresso? • Screenshots - Why? How?
When? The Cost? • How to get “Screenshots for Free” • Tips and Tricks -
What is Espresso? • A library provided by Google for
Android UI testing • Built on Android Test Instrumentation • Aware of the idle state and therefore FAST • Easy to read and write -
NEVER Thread.sleep() • AsyncTask Threadpool ◦ Leverage for Networking and
RxJava • Idling Resources - Use CountingIdlingResource ◦ increment(); and decrement(); https://developer.android.com/reference/android/support/test/espresso/contrib/CountingIdlingResource.html -
Oops, Your Test Failed android.support.test.espresso.NoMatchingViewException: No views in hierarchy found
matching: with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=160.0, child-count=1} | +--->TextView{id=2131230720, res-name=greeting, visibility=VISIBLE, width=656, height=960, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=32.0, y=32.0, text=Hello world!, input-type=0, ime-target=false, has-links=false} | +-->ActionBarContainer{id=16909231, res-name=action_bar_container, visibility=VISIBLE, width=720, height=112, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=48.0, child-count=2} -
Ahh, I See Why android.support.test.espresso.NoMatchingViewException: No views in hierarchy found
matching: with text: “DONE!” View Hierarchy: +>DecorView{id=-1, visibility=VISIBLE, width=720, height=1280, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=3} | +->ActionBarOverlayLayout{id=16909230, res-name=decor_content_parent, visibility=VISIBLE, width=720, height=1184, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, has-input-connection=false, x=0.0, y=0.0, child-count=2} | +-->FrameLayout{id=16908290, res-name=content, visibility=VISIBLE, width=720, height=1024, has-focus=false, has-focusable=false, has-window-focus=true, is-clickable=false, is-enabled=true, is-focused=false, is-focusable=false, is-layout-requested=false, is-selected=false, root-is-layout-requested=false, -
Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable
throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } }); -
Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable
throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } }); -
Screenshots on FAILURES Espresso.setFailureHandler(new FailureHandler() { @Override public void handle(Throwable
throwable, Matcher<View> matcher) { ScreenshotHelper.takeScreenshotForcefully("test_failed"); try { new DefaultFailureHandler(applicationContext).handle(throwable, matcher); } catch (Exception e) { logger.error(e.getMessage(), e); throw new RuntimeException(e); } } }); -
1. See What is Being Tested 2. Diagnose Failures 3.
Share Your Tests -
NO ONE Sees The Value of Your Tests… YET •
Your Boss • Product Owners • Designers • Manual Testers -
That’s because they can’t tell what your tests are doing
and don’t find any use for them. To them it’s just console output. -
User Interfaces are VISUAL • A User Interface is something
very hard to represent in a non-visual way. • Exercise: ◦ Imagine you need to tell someone who is blind what the Mona Lisa painting looks like. What would you say? https://en.wikipedia.org/wiki/Mona_Lisa -
Test Runners and HTML Reports • Run Instrumentation Commands •
Collect Logs • Collect Screenshots • Generate HTML Reports -
Spoon Test Runner java -jar spoon-runner-1.7.0-jar-with-dependencies.jar \ --apk example-app.apk \
--test-apk example-tests.apk • Run instrumentation tests in parallel • Shard tests across multiple devices • Beautiful HTML reports -
Falcon & (Falcon Spoon Compat) FalconSpoon.screenshot(activity, tag); FalconSpoon.screenshot(activity, tag, testClassName,
testMethodName); https://github.com/jraska/Falcon/ -
UiAutomator File outputFile = Spoon.screenshot(activity, tag, testClass, testMethod); uiDevice.takeScreenshot(outputFile); •
Works inside and outside of your app • API 18+ (Jelly Bean MR2) https://developer.android.com/topic/libraries/testing-support-library/in dex.html#UIAutomator -
Other Alternatives • Firebase Test Lab’s “Screenshotter” ◦ https://firebase.google.com/docs/test-lab/test-screenshots •
Fastlane’s “Screengrab” ◦ https://github.com/fastlane/fastlane/tree/master/screengrab -
Typical Espresso Test @RunWith(AndroidJUnit4.class) public class LoginTest { @Rule public
ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ onView(withId(R.id.username)).perform(typeText("sam")); onView(withText("LOGIN")).perform(click()); onView(withId(R.id.home_view_pager)).check(matches(isDisplayed())); } } -
Typical Espresso Test With Screenshots @RunWith(AndroidJUnit4.class) public class LoginTest {
@Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ onView(withId(R.id.username)).perform(typeText("sam")); takeScreenshot("entered_username"); takeScreenshot("clicking_login"); onView(withText("LOGIN")).perform(click()); takeScreenshot("homescreen_shown"); onView(withId(R.id.home_view_pager)).check(matches(isDisplayed())); } } -
Robot Testing Pattern View Presenter Model Test What Test Test
Test Test Test Test -
Robot Testing Pattern View Presenter Model Test What Test Test
Test Test Test Test X X X X X X X View Changed -
Robot Testing Pattern View Presenter Model Test What Test Test
Test Test Test Test Robot -
Robot Testing Pattern View Presenter Model Test What Test Test
Test Test Test Test Robot X View Changed -
Espresso Test With a Robot @RunWith(AndroidJUnit4.class) public class LoginTest {
@Rule public ActivityTestRule<LoginActivity> activityRule = new ActivityTestRule(LoginActivity.class); @Test public void testLogin(){ new LoginRobot().username("sam").login().assertHomeScreenShown(); } } -
Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final
Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; } -
Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final
Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; } -
Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final
Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; } -
Example Robot static final Matcher<View> VIEW_MATCHER_USERNAME_EDIT_TEXT = withId(R.id.username); static final
Matcher<View> VIEW_MATCHER_LOGIN_BUTTON = withText("LOGIN"); static final Matcher<View> VIEW_MATCHER_HOME_SCREEN = withId(R.id.home_view_pager); public LoginRobot username(String username) { onView(VIEW_MATCHER_USERNAME_EDIT_TEXT).perform(clearText(), typeText(username), closeSoftKeyboard()); takeScreenshot("username_entered"); return this; } public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; } -
When to Take Screenshots • Assertions ◦ Assertion(s) ◦ SCREENSHOT
• Actions (Alters screen state) ◦ Displayed Assertion ◦ SCREENSHOT ◦ Action public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; } -
When to Take Screenshots • Assertions ◦ Assertion(s) ◦ SCREENSHOT
• Actions (Alters screen state) ◦ Displayed Assertion ◦ SCREENSHOT ◦ Action public LoginRobot login() { onView(VIEW_MATCHER_LOGIN_BUTTON).check(matches(isDisplayed())); takeScreenshot("logging_in"); onView(VIEW_MATCHER_LOGIN_BUTTON).perform(click()); return this; } public LoginRobot assertHomeScreenShown() { onView(VIEW_MATCHER_HOME_SCREEN).check(matches(isDisplayed())); takeScreenshot("home_screen_shown"); return this; } -
When Should I Take Screenshots? • Well, it depends… but
ALWAYS on FAILURE • To generate reports for product and design teams • On small/low-res emulators with mdpi screen density for smaller screenshots. -
Use Screenshots Wisely • Programmatically configurable • Change config without
recompiling ◦ Leverage Instrumentation Args ▪ --e screenshots=true ▪ --e screenshots=false -
Easy Screenshots • Screenshots on Failure for FREE • Settings:
ALL, FAILURE_ONLY or NONE ◦ Can be changed: ▪ Programmatically ▪ Via instrumentation args • Configurable ◦ Reporting Framework ◦ Screenshot Implementation -
“Can’t write to sdcard” error • Use @GrantPermissionRule from latest
Android Test Support Library • Declare in app’s Manifest: ◦ READ_EXTERNAL_STORAGE and WRITE_EXTERNAL_STORAGE -
Target Audience for Espresso “Espresso is targeted at developers, who
believe that automated testing is an integral part of the development lifecycle. While it can be used for black-box testing, Espresso’s full power is unlocked by those who are familiar with the codebase under test.” -
Execution Time Overhead in Minutes 384x640 mdpi Nexus 5 Nexus
6P Falcon +16.53% +62.31% +114.39% UiAutomator +28.93% +116.15% +168.87% -
Report Size Overhead in MB 384x640 mdpi Nexus 5 Nexus
6P Falcon +17.35% +66.41% +90.13% UiAutomator +18.41% +64.77% +89.19% -
Get Currently RESUMED Activity private static Activity getCurrentActivityInstance() { final
Activity[] activity = new Activity[1]; getInstrumentation().runOnMainSync(new Runnable() { public void run() { Collection resumedActivities = ActivityLifecycleMonitorRegistry.getInstance() .getActivitiesInStage(RESUMED); if (resumedActivities.iterator().hasNext()) { Activity currentActivity = (Activity) resumedActivities.iterator().next(); activity[0] = currentActivity; logger.trace("Activity obtained of type: " + currentActivity.getClass().getName()); } } }); return activity[0]; } -
Get Test Method & Class from Current Stack Trace public
static StackTraceElement getTestMethodStackTraceElement() { StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); for (int i = 4; i < stackTrace.length; i++) { StackTraceElement stackTraceElement = stackTrace[i]; try { Method method = Class.forName(stackTraceElement.getClassName()).getMethod( stackTraceElement.getMethodName()); Annotation annotation = method.getAnnotation(Test.class); if (annotation != null) { return stackTraceElement; } } catch (Exception e) { // That's fine, will look at the next stack trace element } } return stackTrace[4]; } -
Get Test Method & Class from Current Stack Trace public
static StackTraceElement getTestMethodStackTraceElement() { StackTraceElement[] stackTrace = Thread.currentThread().getStackTrace(); for (int i = 4; i < stackTrace.length; i++) { StackTraceElement stackTraceElement = stackTrace[i]; try { Method method = Class.forName(stackTraceElement.getClassName()).getMethod( stackTraceElement.getMethodName()); Annotation annotation = method.getAnnotation(Test.class); if (annotation != null) { return stackTraceElement; } } catch (Exception e) { // That's fine, will look at the next stack trace element } } return stackTrace[4]; } -
Screenshot-tests-for-android by Facebook public class MyTests { @Test public void
doScreenshot() { View view = mLayoutInflater.inflate(R.layout.my_layout, null, false); ViewHelpers.setupView(view) .setExactWidthDp(300) .layout(); Screenshot.snap(view) .record(); } } -
Robot Testing Pattern • A Test Architecture Pattern for Maintainability
• Learned about it from Jake Wharton’s talk: ◦ VIDEO: https://realm.io/news/kau-jake-wharton-testing-robots/ ◦ SLIDES: https://speakerdeck.com/jakewharton/testing-robots-kotlin-night-may-2016 -
Create a Custom Test Runner • Control test runner behavior
without recompiling • Turn on/off all screenshots with a command-line parameter ◦ --e screenshots=true ◦ --e screenshots=false -
Define Your “LOW BAR” Device • Android Emulator Virtual Device
• 384x640 • mdpi - 160 dots per inch • Able to run in continuous integration -
Test Butler by LinkedIn “Reliable Android testing, at your service.”
• Stabilizes the Android emulator ◦ Disables animations ◦ Disables crash & ANR dialogs ◦ Locks the keyguard, WiFi radio, and CPU to ensure they don't go to sleep unexpectedly while tests are running. • Handles changing global emulator settings and holds relevant permissions so your app doesn't have to. ◦ Enable/disable WiFi ◦ Change device orientation ◦ Set location services mode ◦ Set application locale https://github.com/linkedin/test-butler -
Create a Custom Test Runner • Control test runner behavior
without recompiling • Turn on/off all screenshots with a command-line parameter ◦ --e screenshots=true ◦ --e screenshots=false -
Create a Custom Test Runner public class MyAndroidJUnitRunner extends AndroidJUnitRunner
{ @Override public void onCreate(Bundle arguments) { // process you parameters here. super.onCreate(arguments); } } defaultConfig { testInstrumentationRunner "com.handstandsam.MyAndroidJUnitRunner" } -
Create a Custom Test Runner public class MyAndroidJUnitRunner extends AndroidJUnitRunner
{ @Override public void onCreate(Bundle arguments) { // process you parameters here. super.onCreate(arguments); } } defaultConfig { testInstrumentationRunner "com.handstandsam.MyAndroidJUnitRunner" } -
GIFs • Generated by default • Processing overhead ◦ 11
Seconds for 21 Screenshots -> • Disable using “--no-animations”