Upgrade to Pro — share decks privately, control downloads, hide ads and more …

UI Testing over the years

UI Testing over the years

UI testing has been around since iOS 4, but it never really caught on. I've asked various members of the iOS community about their experience and tips.
Join me on a trip of different frameworks, strategies and tools to find a solution that fits your app. You will hear about KIF, EarlGrey, XCU and various open source projects to help you write better tests, to parallelize them or even to remind you to write tests.

This talk was hand-crafted and recorded for App Builders 2020. Video: https://vimeo.com/416980950

Peter Steinberger

May 11, 2020
Tweet

More Decks by Peter Steinberger

Other Decks in Technology

Transcript

  1. Scope — Manual Tests — UI Tests — Snapshot Tests

    — Integration Tests — Unit Tests UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  2. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  3. What framework(s) do you use for UI testing? — XCUITest

    — KIF — EarlGrey (v1/v2) — Xamarin.UITest — Appium — Calabash — Other UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  4. How reliable are your UI tests? — ! — "

    — # — $ — % https://twitter.com/steipete/status/1255021187064893440
  5. Do you distribute your tests? — No (One test at

    a time) — Parallel Simulators — Distribute across Machines — Distribute across machines + Parallel Simulators UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  6. At Funding Circle we have been writing UI tests since

    the beginning. We have a few hundreds currently and they focus on the frequent paths. They take about 30 min to run (not parallelised yet). We still use KIF We suffered the usual pain points (hard to maintain, flaky, etc), but we have always managed to keep them under control. Every PR has to come with the relevant automated tests in order to be merged. — Edu Caselles | Funding Circle Group https://twitter.com/CasellesEdu/status/1255185884045082624
  7. KIF: Keep It Functional — Written in 2011 by Square

    — Lots of private API — Busy waiting — Uses accessiblityLabel UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  8. KIF: Keep It Functional — Written in 2011 by Square

    — Lots of private API — Busy waiting — Uses accessiblityLabel to query views UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  9. ! Create your own UITouch extension UITouch { public init!(in

    view: UIView!) public init!(at point: CGPoint, in view: UIView!) open func setLocationInWindow(_ location: CGPoint) open func setPhaseAndUpdateTimestamp(_ phase: UITouch.Phase) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  10. ! Accessibility 101 — accessiblityLabel — accessiblityIdentifier - value, traits,

    frame, hint - container, containerSpace UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  11. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  12. UIAutomation — Added in iOS 4 (2010) — Removed in

    iOS 10 (2016) — JavaScript-based ! UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  13. var testName = "Test 1"; var target = UIATarget.localTarget(); var

    app = target.frontMostApp(); var window = app.mainWindow(); UIALogger.logStart(testName); // select the elements UIALogger.logMessage("Select the first tab"); var tabBar = app.tabBar(); var selectedTabName = tabBar.selectedButton().name(); if (selectedTabName != "First") { tabBar.buttons()["First"].tap(); } // tap on the text fiels UIALogger.logMessage("Tap on the text field now"); var recipeName = "Unusually Long Name for a Recipe"; window.textFields()[0].setValue(recipeName); target.delay(2); // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard"); app.keyboard().buttons()["return"].tap(); var textValue = window.staticTexts()["RecipeName"].value(); if (textValue === recipeName){ UIALogger.logPass(testName); } else { app.logElementTree(); UIALogger.logFail(testName); } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  14. var testName = "Test 1" var target = UIATarget.localTarget() var

    app = target.frontMostApp() var window = app.mainWindow() UIALogger.logStart(testName) // select the elements UIALogger.logMessage("Select the first tab") var tabBar = app.tabBar() var selectedTabName = tabBar.selectedButton().name() if (selectedTabName != "First") { tabBar.buttons()["First"].tap() } // tap on the text fiels UIALogger.logMessage("Tap on the text field now") var recipeName = "Unusually Long Name for a Recipe" window.textFields()[0].setValue(recipeName) target.delay(2) // tap on the text fiels UIALogger.logMessage("Dismiss the keyboard") app.keyboard().buttons()["return"].tap() var textValue = window.staticTexts()["RecipeName"].value() if (textValue == recipeName) { UIALogger.logPass(testName) } else { app.logElementTree() UIALogger.logFail(testName) } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  15. func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool

    { UIApplication.shared.keyWindow?.layer.speed = 100 return true } https://pspdfkit.com/blog/2016/running-ui-tests-with-ludicrous-speed/
  16. Busy Waiting testWithViewController(pdfController) { pdfController.tapOnAnnotation(fileLinkAnnotation) // Verify that our web

    view controller is visible waitForCondition(pdfController.isControllerClassVisible(PSPDFWebViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) if #available(iOS 9, *) { pdfController.tapOnAnnotation(webLinkAnnotation) waitForCondition(pdfController.isControllerClassVisible(SFSafariViewController.self)) pdfController.dismissViewControllerAnimated(false, completion: nil) } } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  17. Boolean CTTRunLoopRunUntil(Boolean(^fulfilled_)(), Boolean polling_, CFTimeInterval timeout_) { // Loop Observer

    Callback __block Boolean fulfilled = NO; void (^beforeWaiting) (CFRunLoopObserverRef observer, CFRunLoopActivity activity) = ^(CFRunLoopObserverRef observer, CFRunLoopActivity activity) { assert(!fulfilled); //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_(); if(fulfilled) { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()); } else if(polling_) { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()); } }; CFRunLoopObserverRef observer = CFRunLoopObserverCreateWithHandler(NULL, kCFRunLoopBeforeWaiting, true, 0, beforeWaiting); CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); // Run! CFRunLoopRunInMode(kCFRunLoopDefaultMode, timeout_, false); CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, kCFRunLoopDefaultMode); CFRelease(observer); return fulfilled; } https://bou.io/CTTRunLoopRunUntil.html
  18. func CTTRunLoopRunUntil(_ fulfilled_: @escaping () -> Bool, _ polling_: Bool,

    _ timeout_: CFTimeInterval) -> Bool { // Loop Observer Callback var fulfilled = false let beforeWaiting: ((_ observer: CFRunLoopObserver?, _ activity: CFRunLoopActivity) -> Void)? = { observer, activity in assert(!fulfilled) //RunLoop should be stopped after condition is fulfilled. // Check Condition fulfilled = fulfilled_() if fulfilled { // Condition fulfilled: stop RunLoop now. CFRunLoopStop(CFRunLoopGetCurrent()) } else if polling_ { // Condition not fulfilled, and we are polling: prevent RunLoop from waiting and continue looping. CFRunLoopWakeUp(CFRunLoopGetCurrent()) } } let observer = CFRunLoopObserverCreateWithHandler(nil, CFOptionFlags(CFRunLoopActivity.beforeWaiting.rawValue), true, CFIndex(0), beforeWaiting) CFRunLoopAddObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) // Run! CFRunLoopRunInMode(CFRunLoopMode.defaultMode, timeout_, false) CFRunLoopRemoveObserver(CFRunLoopGetCurrent(), observer, CFRunLoopMode.defaultMode) return fulfilled } UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  19. ! Square UI Testing over the years — App Builders

    2020 | Peter Steinberger — @steipete
  20. We have over 8000 tests consisting of a mixture of

    Unit and UI Tests in our codebase that run on devices housed in an on-site lab. A test suite of around 200 tests run with every PR. If a test fails, it is automatically rerun to account for flakiness. We're still doing manual testing to account for hard to automate paths: I doubt those devices take out their airpods while watching Tiger King. — Carola Nitz | Netflix https://twitter.com/CaroN
  21. At Twitch we have about 350 iOS UI tests, taking

    ~29 min with parallel Simulators on Jenkins. Running them on devices has been both flaky and slower every time we tried. We retry failed tests because we’ve always struggled with flakiness. Our UI tests are for the most part end-to-end integration tests that hit the server. Just because the UI tests are passing doesn’t mean the app has no bugs. But it’s been really valuable as a “first line of defense” to catch crashes earlier and keep master relatively healthy, both important in a team of 18 engineers. — Javier Soto | Twitch https://twitter.com/Javi/status/1255123189690068993
  22. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  23. I love this one tweet calling them indeterministic; croissants are

    "flaky" Our UI tests also cover a lot of networking, which doesn't help. But on the other hand we implicitly test a lot more with that — a blessing and a curse. A pull request from me yesterday ran 854 "features". 1 failure. Took 35 minutes. Tests were split across 13 nodes. — Bas Broek | XING https://twitter.com/basthomas/status/1255044115512782848
  24. We have a very large UI test suite. Over time

    this allowed us to cut our release cycle down to 1 week. Besides reducing manual verification, it proved to be extremely helpful on large under the hood refactorings. — Tomas Camin | Subito https://twitter.com/tomascamin/status/1255048867378343937
  25. We've tried many different implementations of UI tests, including writing

    custom frameworks. Our suite would take from 4 to 6 hours. Even the most basic UI test would sometimes fail because it could not find a text field, or a label. — Eneko Alonso | Mindbody https://twitter.com/eneko/status/1255499202400002064
  26. XCUI Best Practices // use accessibility identifiers let resultView =

    app.otherElements["view_result"] // always wait before asserting let viewExists = resultView.waitForExistence(timeout: 10) XCTAssert(viewExists) // write helpers (Page Object Pattern) class SearchPageObject { func type(query: String) -> Self func tapSend() -> ResultPage } https://medium.com/@danielcarlosce/some-good-practices-for-xcuitest-807bfe6b720d, https://medium.com/capital-one-tech/robot-pattern-testing-for-xcuitest-4c2f0c40b4ad, https:// blog.novoda.com/ui-testing-part-3/
  27. Prevent Keyboard Tutorials // rdar://34634970 if let defaults = UserDefaults(suiteName:

    "com.apple.Preferences") { // Since iOS 11 defaults.set(true, forKey:"DidShowGestureKeyboardIntroduction") // iPhone since iOS 13 defaults.set(true, forKey:"DidShowContinuousPathIntroduction") } https://github.com/google/EarlGrey/issues/633
  28. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  29. Tracking Busy Resources — Grand Central Dispatch — NSTimer /

    performSelector:afterDelay: — Layout Pass — View Appear/Disappear — Keyboard Transitions — CA/UI Animations — View Controller Transitions — WebView/Network Requests — Gesture Recognition — UIScrollView — Systemwide User Interaction UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  30. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  31. UI Testing over the years — App Builders 2020 |

    Peter Steinberger — @steipete
  32. Interposing C functions struct dyld_interpose_tuple { const void* replacement; const

    void* replacee; }; extern void dyld_dynamic_interpose(const struct mach_header* mh, const struct dyld_interpose_tuple array[], size_t count); https://github.com/opensource-apple/dyld/blob/master/include/mach-o/dyld_priv.h
  33. Interposing Grand Central Dispatch __attribute__((used)) __attribute__((section("__DATA,__interpose"))) static const dyld_interpose_tuple grey_static_interpose_tuples[]

    = { { grey_dispatch_after, dispatch_after }, { grey_dispatch_async, dispatch_async }, { grey_dispatch_sync, dispatch_sync }, { grey_dispatch_after_f, dispatch_after_f }, { grey_dispatch_async_f, dispatch_async_f }, { grey_dispatch_sync_f, dispatch_sync_f } }; UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  34. ✅ SwiftUI UI Testing over the years — App Builders

    2020 | Peter Steinberger — @steipete
  35. Recap — KIF — accessibilityIdentifier + Inspector — UIAutomation —

    Accelerating Animations layer.speed — Busy Waiting — Running Single Tests -only-testing — Parallel Testing — EarlGrey 1/2 — dyld_dynamic_interpose — SwiftUI UI Testing over the years — App Builders 2020 | Peter Steinberger — @steipete
  36. ! Tooling UI Testing over the years — App Builders

    2020 | Peter Steinberger — @steipete
  37. ! Thanks UI Testing over the years — App Builders

    2020 | Peter Steinberger — @steipete