Appiumテストの失敗原因調査のコツと実践

https://appiummeetuptokyo.connpass.com/event/353600/ のスライドですー

Transcript

1. Appiumテストの失敗原因調査のコツ と実践 MagicPod Inc. 2025 1
2. Contents 1. Introduction 2. The Challenges of Running Mobile E2E Tests 3. Takeaway #1: Building a Strong Foundation in Appium 4. Takeaway #2: Enhancing Your Debugging Experience 5. Takeaway #3: Narrow Down the Cause of Failures 6. Case study: Debugging a WebDriverAgent Issue 7. Conclusion MagicPod Inc. 2025 2
3. Masayuki Wakizaka (脇坂 雅幸) Engineering manager at MagicPod Inc. We provide an E2E test automation platform using Selenium/Appium. I've been working at quality engineering domain for 10 years. E2E test automation engineer and test manager at SHIFT Inc. Software engineer at MagicPod Inc. One of Appium contributers A husband, a tennis player, sometimes a traveller I'm exieted to participate in Selenium/Appium Conf 2025! MagicPod Inc. 2025 3
4. 2. The Challenges of Running Mobile E2E Tests MagicPod Inc. 2025 4
5. Difficulty in Diagnosing Test Failures Flakiness: A race condition, intermittent network/device/server issue etc Integrations: Depends on test script, AUT, DUT, Appium, third party framework, network, machine resources, etc Performance: Abuse of XPath locators, server resource/load issue, platform difference etc MagicPod Inc. 2025 5
6. 3. Takeaway #1: Building a Strong Foundation in Appium MagicPod Inc. 2025 6
7. Appium An OSS framework that enables E2E test automation Implements W3C protocol that extends Selenium Works in client-server model. Provides several programming languages for test script. MagicPod Inc. 2025 7
8. 3.1. Appium Architecture MagicPod Inc. 2025 8
9. 3.1. Appium Architecture MagicPod Inc. 2025 9
10. 3.2. How Appium Works? MagicPod Inc. 2025 10
11. 3.2. How Appium Works? - WebDriverAgent MagicPod Inc. 2025 11
12. 3.2. How Appium Works? - XCTest Framework MagicPod Inc. 2025 12
13. 3.2. How Appium Works? - Finding an Element MagicPod Inc. 2025 13
14. 3.2. How Appium Works? - HTTP Request to Appium Server MagicPod Inc. 2025 14
15. 3.2. How Appium Works? - Proxy the Request MagicPod Inc. 2025 15
16. 3.2. How Appium Works? - Query the Element by XCTest MagicPod Inc. 2025 16
17. 3.2. How Appium Works? - The Response MagicPod Inc. 2025 17
18. 3.3. Other Dependences - SDK iOS: Xcode, xcrun simctl and xcrun devicectl Android: Android SDK (especially adb) MagicPod Inc. 2025 18
19. 3.3. Other Dependences - XCUITest Driver Packages XCUITest packages Description appium-xcuitest-driver The main package of XCUITest driver appium-webdriveragent A WebDriver server used to remote control iOS devices appium-remote-debugger Remote Debugger that enables WebView testing appium-ios-simulator A package for iOS simulator appium-ios-device A package for iOS real device node-simctl A wrapper for xcrun simctl command MagicPod Inc. 2025 19
20. 4. Takeaway #2: Enhancing Your Debugging Experience MagicPod Inc. 2025 20
21. 4. Takeaway #2: Enhancing Your Debugging Experience 1. Understand the Appium server log 2. Understand the Device log 3. Get the page source 4. Save screenshots or movie MagicPod Inc. 2025 21
22. 4.1. Appium Server Logs Appium server log is the log output generated by the Appium server when running tests. Interaction between your test script, appium server and device Command invocations on Appium server It helps you to narrow down the cause of your test failure. MagicPod Inc. 2025 22
23. 4.1. Appium Server Logs - Session Creation Success When creating a session, it conducts a series of complicated settings to run E2E tests It often helps to check if the Appium server recieves your intended desired capabilities [HTTP] --> POST /session {"capabilities":{"alwaysMatch":{... ... [XCUITestDriver@b2e2] Creating session with W3C capabilities: { "alwaysMatch": { "platformName": "iOS", "appium:platformVersion": "18.3", "appium:deviceName": "iPhone 12", "appium:automationName": "XCUITest", "appium:newCommandTimeout": 21600, "appium:app": "/Users/mwakizaka/..." }, ... [e8b04301][HTTP] <-- POST /session 200 7319 ms - 539 Full log: https://gist.github.com/mwakizaka/61c3a9722e911705ea248f7399fd15a8#file-session-creation-xcuitest-log 23
24. 4.1. Appium Server Logs - Session Creation Failure Failed to create a session due to a provisioning profiles issue (Full log) [HTTP] --> POST /session {"capabilities":{"alwaysMatch"... ... [b2ca6bf5][XCUITestDriver@8ff1] Beginning test with command 'xcodebuild build-for-testing \ test-without-building -project /path/to/WebDriverAgent.xcodeproj -scheme WebDriverAgentRunner \ -derivedDataPath /path/to/appium-xcuitest-driver/node_modules -destination id=<read device id> \ IPHONEOS_DEPLOYMENT_TARGET=16.4 DEVELOPMENT_TEAM=<TEAM ID> CODE_SIGN_IDENTITY=iPhone Developer \ GCC_TREAT_WARNINGS_AS_ERRORS=0 COMPILER_INDEX_STORE_ENABLE=NO' in directory '/path/to/appium-webdriveragent' ... [b2ca6bf5][Xcode] /path/to/WebDriverAgent.xcodeproj: error: No profiles for \ 'com.facebook.WebDriverAgentRunner.xctrunner' were found: Xcode couldn't find any iOS App \ Development provisioning profiles matching 'com.facebook.WebDriverAgentRunner.xctrunner'. \ Automatic signing is disabled and unable to generate a profile. To enable automatic signing, \ pass -allowProvisioningUpdates to xcodebuild. ... [b2ca6bf5][XCUITestDriver@8ff1] Unable to launch WebDriverAgent. Original error: xcodebuild \ failed with code 70. This usually indicates an issue with the local Xcode setup or \ WebDriverAgent project configuration or the driver-to-platform version mismatch. \ Consider checking the WebDriverAgent configuration guide for real iOS devices at \ https://github.com/appium/appium-xcuitest-driver/blob/master/docs/real-device-config.md.. \ Make sure you follow the tutorial at https://appium.github.io/appium-xcuitest-driver/latest \ /preparation/real-device-config/ ... [b2ca6bf5][HTTP] <-- POST /session 500 9574 ms - 1580 MagicPod Inc. 2025 24
25. 4.2. Device Logs Device logs refer to logs collected directly from the mobile device/emulator/simulator. They contain system level information including app crashes, system event, debugging messages from the device OS. a.k.a, syslog and logcat for iOS and Android, respectively. Device logs is useful when you investigate issues beyond the Appium's responsibility. e.g. App crashes, gesture recognition issues, network errors cf: appiumpro Eddition 55: Using Mobile Execution Commands to Continuously Stream Device Logs with Appium 25
26. 4.2. Device Logs - iOS Simulator 2025-01-20 13:33:39.267 Df testmanagerd[70867:14da78e] [com.apple.dt.xctest:Default] Requesting screenshot. # Component Description 1 2025-01-20 13:33:39.267 Timestamp (Date, Time, Microseconds) 2 Df log level 3 testmanagerd Sub system (e.g. App name, process name, bundle id) 4 [70867:14da78e] Process ID (PID) and Thread ID (TID) 5 [com.apple.dt.xctest:Default] Log category (e.g., General, Network, UI, etc.) 6 Requesting screenshot. The actual log message MagicPod Inc. 2025 26
27. 4.3. Taking the Page Source The page source describes the XML reprensentation of the current screen of AUT. It helps your debugging especially when you take it before finishing running E2E test. It may capture the screen where the element of you element does not appear. MagicPod Inc. 2025 27
28. 4.3. Taking the Page Source - iOS <?xml version="1.0" encoding="UTF-8"?> <AppiumAUT> ... <XCUIElementTypeTable ... > <XCUIElementTypeCell ... > ... <XCUIElementTypeStaticText ... name="Activity Indicators" ... /> <XCUIElementTypeStaticText ... name="ActivityIndicatorViewController" ... /> ... </XCUIElementTypeCell> ... </XCUIElementTypeTable> ... </AppiumAUT> cf: Full page source 28
29. 4.3. Taking the Page Source - Note It can be often an expensive and time-consuming operation especially on iOS. The scope is different between iOS and Android. iOS: It includes elements outside the viewport. Android (UIA2): It does not include elements outside the viewport. cf: https://github.com/appium/appium/discussions/19044 MagicPod Inc. 2025 29
30. 4.4. Taking Screenshots or Recording a Movie Although it depends on your need, but I recommend recording a movie. If you go with screenshots, I recommend taking a screenshot every after interacting with your AUT. Some cloud providers such as SauceLabs and Browserstack support recording movies during the test run by default. Screenshots and movies may not be available when some settings such as FLAG_SECURE is enabled MagicPod Inc. 2025 30
31. 5. Takeaway #3: Narrow Down the Cause of Failures MagicPod Inc. 2025 31
32. 5.1. CI/CD along with Version Control Every component of your product and test may contribute to an E2E test failure. You should run E2E test as frequently as possible. -> It helps you to narrow down the changes which caused a test failure. You should avoid conducting large changes at once. Appium project adopts the semantic versioning. When they bump up the major version, it includes a breaking change. You should read carefully the release note. They sometimes drop support for older iOS versions. cf: Xcode/iOS Version Support 32
33. 5.2. Appium Inspector A GUI assistant tool for Appium, providing visual inspection of the AUT. It works like an REPL. You don't have to implement your test script for debugging. You can try reproducing your test failure using Appium Inspector Note that you need to launch an Appium server beforehand. specify --allow-cors flag when using Appium Inspector on the web MagicPod Inc. 2025 33
34. 5.2. Appium Inspector - Small Demo MagicPod Inc. 2025 34
35. 5.3. When You Cannot Find an Element on the Page Source Make sure if the element is accessbile via accessibility feature VoiceOver on iOS TalkBack on Android You can also use IDE tools Accessibility Inspector on Xcode Layout Inspector on Android Studio MagicPod Inc. 2025 35
36. 6. Case study - Debugging a WebDriverAgent Issue MagicPod Inc. 2025 36
37. The Issue Elements on iOS half sheet couldn't be found. MagicPod Inc. 2025 37
38. The Expected Result Elements on iOS half sheet should be found. MagicPod Inc. 2025 38
39. The Page Source? I couldn't confirm the existence of half sheet on the page source. We expected there should have been an element like <XCUIElementTypeButton type="XCUIElementTypeButton" name="footer" label="Purchase" .../> Previously, we could interact with the elements. cf: Page Source 39
40. Is It Accessible via Accessibility Inspector? Mysteriously, it exists. An Appium issue? MagicPod Inc. 2025 40
41. Narrowing Down the Suspicious Changes The test failure occurred on iOS. It's important to check XCUITest driver versions. I suspected WDA caused the issue because it is WDA's role. XCUITest dependencies Last successful version First failed version appium-xcuitest-driver 5.16.1 7.24.18 appium-webdriveragent 5.15.9 8.9.1 appium-remote-debugger 10.2.1 12.1.1 appium-ios-simulator 5.5.2 6.1.11 appium-ios-device 2.7.11 2.7.23 node-simctl 7.3.13 7.5.3 MagicPod Inc. 2025 41
42. Which Version of WDA Caused the Issue? I suspected WDA 6.0.0 based on the XCUITest driver's release note MagicPod Inc. 2025 42
43. What's the Breaking Change in WDA 6.0.0? WDA < 6 : Appium users don't have to be aware of the active application when handling system dialogs (such as permission dialogs) WDA >=6 : Appium users have to be aware of it. Refer to Interact with dialogs managed by com.apple.springboard for more info To preserve the behaviour prior to version 6, we use respectSystemAlerts setting since XCUITest driver 7.17.0 MagicPod Inc. 2025 43
44. Reproducing the Issue I confirmed that the issue started to occur at version WDA 6.0.0 by comparing it with WDA 5.15.9 BTW, I had a difficult time to reproduce the issue because I missed to specify appium:bundlId capability (Oh my laziness..) cf: Interact with dialogs managed by com.apple.springboard 44
45. The Resolution Although I'm not good at WDA implementation, I managed to submit a pull request. By this PR, respectSystemAlerts setting takes into account of half sheet. The reviewers are very friendly and responsive. You can also create a ticket in the Appium project and request the maintainers to fix it if you encounter an Appium issue. MagicPod Inc. 2025 45
46. The Result You can now find elements on half sheet using respectSystemAlerts setting. MagicPod Inc. 2025 46
47. 7. Conclusion It is often difficult to locate the cause of failure in mobile E2E testing. Although we don't have a silver bullet, there are some lead bullets. Building the basic understanding on Appium, the structure and its dependences will improve your guess for which dependency contributes to test failures. Enhancing Your Debugging Experience helps you to dive into test failures Narrowing down the cause of failures is the first step for resolving the test failures MagicPod Inc. 2025 47
48. 8. Appendix MagicPod Inc. 2025 48
49. 3.3. UIA2 driver packages UIA2 packages Description appium-uiautomator2- driver The main package appium-uiautomator2- server A WebDriver server used to remote control Android devices appium-android-driver The super class of UIA2 driver io.appium.settings Android app responsible for tweaking Android device settings appium-adb A wrapper for adb command appium-chromedriver A package for WebView testing MagicPod Inc. 2025 49
50. 4.1. Appium Server Logs - Server Startup It displays the Appium server version, your driver versions and plugin versions. The information often enables you to narrow down causes of your test failure. % appium [Appium] Welcome to Appium v2.11.4 (REV d3f61992f4f64bf4b600933d38b8fe3f61838b38) ... [Appium] Appium REST http interface listener started on http://0.0.0.0:4723 [Appium] You can provide the following URLs in your client code to connect to this server: http://127.0.0.1:4723/ (only accessible from the same host) http://192.168.10.136:4723/ [Appium] Available drivers: [Appium] - xcuitest@7.32.0 (automationName 'XCUITest') [Appium] - uiautomator2@3.10.0 (automationName 'UiAutomator2') [Appium] Available plugins: [Appium] - relaxed-caps@1.0.6 [Appium] No plugins activated. Use the --use-plugins flag with names of plugins to activate Full log: https://gist.github.com/mwakizaka/61c3a9722e911705ea248f7399fd15a8#file-appium-server-start-up-log 50
51. 4.1. Appium Server Logs - Finding an element Indicates the locator strategy/value as the request body and the succesful response. [e8b04301][HTTP] --> POST /session/e8b04301-7bbe-400f-b1bf-060cb0d7ca5a/element {"using":"accessibility id","value":"Buttons"} [e8b04301][XCUITestDriver@b2e2] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id, css selector [e8b04301][XCUITestDriver@b2e2] Proxying [POST /element] to [POST http://127.0.0.1:8100/session/39553AB5-EA91-46FA-A116-5EBA2E44C89F/element] with body: {"using":"accessibility id","value":"Buttons"} [e8b04301][XCUITestDriver@b2e2] Got response with status 200: {"value":{"ELEMENT":"31000000-0000-0000-E852-010000000000", "element-6066-11e4-a52e-4f735466cecf":"31000000-0000-0000-E852-010000000000"}, "sessionId":"39553AB5-EA91-46FA-A116-5EBA2E44C89F"} [e8b04301][XCUITestDriver@b2e2] Responding to client with driver.findElement() result: {"element-6066-11e4-a52e-4f735466cecf":"31000000-0000-0000-E852-010000000000", "ELEMENT":"31000000-0000-0000-E852-010000000000"} [e8b04301][HTTP] <-- POST /session/e8b04301-7bbe-400f-b1bf-060cb0d7ca5a/element 200 236 ms - 137 Full log: https://gist.github.com/mwakizaka/61c3a9722e911705ea248f7399fd15a8#file-finding-an-element-log 51
52. 4.1. Appium Server Logs - No element found The end point returns 404 response if no element found [441e6d73][HTTP] --> POST /session/441e6d73-a6a3-4dcc-8779-f8473965edc6/element {"using":"accessibility id","value":"Buttons"} [441e6d73][XCUITestDriver@1338] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id, css selector [441e6d73][XCUITestDriver@1338] Proxying [POST /element] to [POST http://127.0.0.1:8100/session/8AB4FAC1-5AAC-45FD-BFED-E20AB9EFCF77/element] with body: {"using":"accessibility id","value":"Buttons"} [441e6d73][XCUITestDriver@1338] Got response with status 404: {"value":{"error":"no such element","message":"unable to find an element using 'accessibility id', value 'Buttons'","traceback":"... [441e6d73][W3C] Matched W3C error code 'no such element' to NoSuchElementError [441e6d73][XCUITestDriver@1338] Encountered internal error running command: NoSuchElementError: An element could not be located on the page using the given search parameters. [441e6d73][HTTP] <-- POST /session/441e6d73-a6a3-4dcc-8779-f8473965edc6/element 404 137 ms - 1325 Full log: https://gist.github.com/mwakizaka/61c3a9722e911705ea248f7399fd15a8#file-element-not-found-log 52
53. 4.1. Other tips - appium server Use Appium server debug log with the timestamp Specify debug for --log-level flag Use --log-timestamp flag MagicPod Inc. 2025 53
54. 4.1. Other tips - XCUITest driver Capabilities for increasing logs on iOS appium:showXcodeLog : Whether to display the output of the Xcode command appium:showIOSLog : Whether to show logs from a device in the appium logs appium:simulatorLogLevel : Customizes the log level for iOS simulator logs appium:iosSimulatorLogsPredicate : --predicate flag to filter iOS simulator logs Logging for Safari browser: appium:safariLogAllCommunication appium:enablePerformanceLogging , appium:showSafariConsoleLog , appium:showSafariNetworkLog cf: appium-xcuitest-driver capabilities 54
55. 4.1. Other tips - UiAutomator2 driver Capabilities for increasing/tweaking logs on Android appium:logcatFormat : Customizes the logcat print format, where format is one of: brief process tag thread raw time threadtime long. threadtime is the default value. appium:showChromedriverLog : Forwards chromedriver log to Appium server if enabled. cf: appium-uiautomator2-driver capabilities 55
56. 4.2. Device Logs - Taking a screenshot 2025-01-20 13:33:39.266 Df WebDriverAgentRunner-Runner[70888:14cf08e] (WebDriverAgentLib) The device under test does not support HEIC image encoding. Falling back to PNG 2025-01-20 13:33:39.267 Df testmanagerd[70867:14da78e] [com.apple.dt.xctest:Default] Requesting screenshot. Full log: https://gist.github.com/mwakizaka/71c8508ff1a850986c4bf12146ccb626#file-ios-sim-device-log 56
57. 4.2. Device Logs - Tapping on the screen 2025-01-20 13:33:56.730 Df testmanagerd[70867:14cede6] [com.apple.dt.xctest:Default] Recap event action playback completed 2025-01-20 13:33:56.730 Df testmanagerd[70867:14da8e2] [com.apple.dt.xctest:Default] Waiting up to 5.0s for TouchEventsCompleted to confirm <XCSynthesizedEventRecord 'W3C Touch Action display 0'> Path 1: Touch down at 338.0, 38.0, offset=0.00s Touch up at 338.0, 38.0, offset=0.10s. Full log: https://gist.github.com/mwakizaka/71c8508ff1a850986c4bf12146ccb626#file-ios-sim-tap-event-log 57
58. 4.2. Device Logs - An AUT crash 2025-01-20 13:33:56.595 Df siriusproto[70904:14cf29e] (CoreFoundation) *** Terminating app due to uncaught exception 'NSRangeException', reason: '*** -[__NSArray0 objectAtIndex:]: index 0 beyond bounds for empty array' *** First throw call stack: ( 0 CoreFoundation 0x0000000180491128 __exceptionPreprocess + 172 1 libobjc.A.dylib 0x000000018008412c objc_exception_throw + 56 2 CoreFoundation 0x0000000180365bb4 -[__NSArray0 objectEnumerator] + 0 3 AVKit 0x00000001943bbe40 -[AVMobileChromelessContentTabsViewController _activeContentTab] + 80 4 AVKit 0x00000001943bbb9c -[AVMobileChromelessContentTabsViewController scrollViewDidEndDecelerating:] + 220 5 <application name> 0x00000001001e8354 -[RNSScreen traverseForScrollView:] + 236 6 CoreFoundation 0x00000001803f5cd8 __NSARRAY_IS_CALLING_OUT_TO_A_BLOCK__ + 16 7 CoreFoundation 0x00000001804d8b70 -[__NSSingleObjectArrayI enumerateObje Full log: https://gist.github.com/mwakizaka/71c8508ff1a850986c4bf12146ccb626#file-ios-sim-aut-crash-log 58
59. 4.2. Interpret the Device Logs - iOS real devices Feb 28 14:39:53 iPhone-11 testmanagerd[584] <Notice>: Requesting screenshot. Component Description Feb 28 14:39:53 Timestamp (Month, Day, Time in HH:MM:SS) iPhone-11 Device name (hostname) testmanagerd[584] Process name (testmanagerd) and Process ID (PID: 584) <Notice> Log level (e.g., <Error>, <Warning>, <Notice>, <Info>, etc.) Requesting screenshot. The actual log message MagicPod Inc. 2025 59
60. 4.2. Interpret the Device Logs - Android 02-21 12:17:42.860 17753 17782 I appium : CaptureScreenshot command Component Description 02-21 Date (MM-DD format) 12:17:42.860 Timestamp (HH:MM:SS.sss, with milliseconds) 17753 Process ID (PID) 17782 Thread ID (TID) I Log Level (I for Info) appium Tag (Identifies the source component, e.g., "appium") CaptureScreenshot command Log message MagicPod Inc. 2025 60
61. 4.2. Appium Server Logs VS Device Logs Appium server logs are often useful when identifying issues on test scenario and its initial setup. i.e. Unable to find an element, missing configuration on the DUT, etc. Device logs are often useful when identifying issues on AUT, DUT and third party framework. e.g. App crashes, gesture recognition issues, network errors MagicPod Inc. 2025 61
62. 4.3. How to record a movie You need to use Start Recording Screen and Stop Recording Screen APIs This API may not also work when some settings such as FLAG_SECURE is enabled The examples using WebDriverIO // Android const options = { videoSize: '1280x720', bitRate: '1000000', forceRestart: true, timeLimit: '300' }; await driver.startRecordingScreen(options); await driver.saveRecordingScreen('<path/to/movie.mp4>'); // iOS const options = { videoType: 'mpeg4', videoFps: 5, videoQuality: 'low', forceRestart: true, timeLimit: '300' }; await driver.startRecordingScreen(options); await driver.saveRecordingScreen('<path/to/movie.mp4>'); cf: https://github.com/appium/appium/issues/11276 62
63. 5.3. Example of inaccessible elements MagicPod Inc. 2025 63
64. 5.3. When you find an inaccessible element? The AUT needs to be updated iOS: Assign accessibilityIdentifier property to the element Android: Assign contentDescription attribute to the element Be aware of that the value will be read aloud by VoiceOver / TalkBack MagicPod Inc. 2025 64
65. EOF MagicPod Inc. 2025 65