Leveraging AI in Test Automation Products

The slide for AI and Software Testing Night, held on February 1, 2024.

Transcript

1. Leveraging AI in Test Automation Products AI & Software Testing Night
2. Nozomi Ito ( 伊藤 望) Founder/CEO of MagicPod Inc. Organizer of SeConf Tokyo and SeleniumJP Author of several test automation books X: @nozomi_ito2 About me
3. Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.
4. Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.
5. No-code, cloud-based, end-to-end test automation platform for web & mobile apps
7. Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.
8. AI features of MagicPod 2. AI Use in MagicPod ① Human-readable UI element captions ② Self-healing
9. UI element caption calculation by AI 2. AI Use in MagicPod - ①Human-readable UI element captions Human-readable caption ・Calculated by AI ・Not used by the test runner Selenium/Appium locator ・Calculated from UI tree ・Used by the test runner
10. Information used for caption calculation 2. AI Use in MagicPod - ①Human-readable UI element captions Information on the UI tree: attributes, texts, tag names, etc. Position relation between elements
11. Information used for caption calculation 2. AI Use in MagicPod - ①Human-readable UI element captions OCR (Recognize characters on the image) Object recognition by the deep learning engine
12. What is self-healing? 2. AI Use in MagicPod - ②Self-healing When a test fails to find a UI element due to a change in the application’s page structure, AI modifies the script according to the page change.
13. Click “register” Test script internal name updated! What happens if it is executed? Example of self-healing 2. AI Use in MagicPod - ②Self-healing name=register name=registerButton
14. Example of self-healing 2. AI Use in MagicPod - ②Self-healing The AI engine updates script and the test passes The user is notified of the update detail
15. 2 types of self-healing 2. AI Use in MagicPod - ②Self-healing A. Multiple locators approach B. Single locator approach Adopted by MagicPod
16. A. Multiple locators approach 2. AI Use in MagicPod - ②Self-healing Suitable for record-and-playback test automation tools When recording a test: Record an element by multiple attributes Such as id, name, index When the test fails: When failing to identify the element by an attribute, try other recorded attributes
17. A. Multiple locators approach 2. AI Use in MagicPod - ②Self-healing ex.) Testim (testim.io) *based on our own research Update weight when changing the locator
18. B. Single locator approach 2. AI Use in MagicPod - ②Self-healing Adopted by MagicPod When creating a test: Store locators and the full UI tree (HTML) When the test fails: Compare the run-time UI tree and the recorded UI tree, and use a sufficiently similar element if it exists
19. B. Single locator approach 2. AI Use in MagicPod - ②Self-healing ex.) MagicPod Creation-time UI tree Run-time UI tree Creation-time locator New locator
20. B. Single locator approach 2. AI Use in MagicPod - ②Self-healing Factors for determining elements similarity Element attribute, position, parent-child relationship, etc. OCR result (MagicPod-only) Object recognition result by the deep learning engine (MagicPod-only) Advantages The same simple locator as Selenium/Appium
21. Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.
22. 3. Use of Generative AI Very useful for code test automation tools such as Selenium, Playwright Not so useful for no-code test automation tools at this time Conversation with generative AI requires more time than just recording user operation Is AI useful for test scenario generation?
23. 3. Use of Generative AI 3 ideas for generative AI collaboration ① Summarize test cases ② Generate change comments ③ (Detect random IDs)
24. 3. Use of Generative AI - ①Summarize test cases Motivations: Understanding each test case requires large efforts for a new member It is not easy to keep human-written summary up-to-date Generate summary by generative AI
25. 3. Use of Generative AI - ①Summarize test cases Experiment Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Human-readable text The script "Check that signup works correctly" automates end-to-end testing for user signup on a hotel website: 1. It ensures logout if the user is already logged in. 2. Fills out the signup form with unique details like email, password, name, etc. 3. Submits the form and verifies entered data's correctness on the confirmation page. 4. Confirms successful signup by logging out and logging back in, checking for the "MyPage" header's presence. Test script Summary MagicPod public API Generative AI
26. 3. Use of Generative AI - ①Summarize test cases Test script
27. 3. Use of Generative AI - ①Summarize test cases Human-readable text Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Title: Check that singup works correctly
28. The following is a browser E2E test automation script. The script name is "[test script name]". Could you create a summary by which I can grasp what kind of test the script does? =============================================== [Human-readable test script ] 3. Use of Generative AI - ①Summarize test cases Prompt for generative AI
29. The script "Check that signup works correctly" automates end-to-end testing for user signup on a hotel website: 1. It ensures logout if the user is already logged in. 2. Fills out the signup form with unique details like email, password, name, etc. 3. Submits the form and verifies entered data's correctness on the confirmation page. 4. Confirms successful signup by logging out and logging back in, checking for the "MyPage" header's presence. 3. Use of Generative AI - ①Summarize test cases Generated summary
30. 3. Use of Generative AI - ②Generate change comments Generate change comments by AI Motivations: Set better change comment than “Autosaved” when a change is saved automatically
31. Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Updated the script by modifying the user's name, address, and telephone number inputs to use new values for improved clarity and specificity in test data. 3. Use of Generative AI - ②Generate change comments Experiment Human-readable text (old) Test script (old) change comment Generative AI Test script (new) Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Human-readable text (new)
32. 3. Use of Generative AI - ②Generate change comments Change detail ......
33. 3. Use of Generative AI - ②Generate change comments Prompt for generative AI The following 2 browser E2E test automation scripts represent the script before change and the script after the change when an engineer modified the script. Before Change ---------------- [Full human-readable script before the change] ---------------- After change ---------------- [Full human-readable script after the change] ---------------- Please create a change comment in one sentence.
34. Updated the script by modifying the user's name, address, and telephone number inputs to use new values for improved clarity and specificity in test data. 3. Use of Generative AI - ②Generate change comments Generated change comments Updated user profile information for signup test to reflect new user details and address. Variation 2 Variation 1
35. Motivations: We don’t want to use random IDs in locators since they are dynamic, cryptic, and vulnerable to change Random ID detection is not always easy 3. Use of Generative AI - ③Detect random IDs Smarter locators not using random IDs
36. 3bb5fe17-285c-4aa7-828f-f54816ed1bdf user001 prefix_2bd5ff24-185c-4ab7-128f-f54616ededda second_section magicPod mantine-zbve6blpq test_combo gtm-noscript m-large 23_aKvs-b8bW2Vg3fwHozO 13LGdX8RMStbBE9w-t0gZ1 css-r6dqg1 prefix_3bb5fe17-285c-4aa7-828f-f54816ed1bdf code-snippet related-container data-v-0c1f62df privateStripeFrame456 aras-toolbar container hs-analytics aras-header aras-es-toolbar-input o3ofqfx9qfqo sc-ilJxFg bkOQwZ gh-header-actions mt-0 mb-3 mb-md-2 ml-1 flex-md-order-1 flex-shrink-0 d-flex flex-items-center gap-1 3. Use of Generative AI - ③Detect random IDs Experiment all ids & classes HTML Generative AI 3bb5fe17-285c-4aa7-828f-f54816ed1bdf prefix_2bd5ff24-185c-4ab7-128f-f54616ededda mantine-zbve6blpq 23_aKvs-b8bW2Vg3fwHozO 13LGdX8RMStbBE9w-t0gZ1 css-r6dqg1 prefix_3bb5fe17-285c-4aa7-828f-f54816ed1bdf data-v-0c1f62df o3ofqfx9qfqo sc-ilJxFg bkOQwZ random ids & classes Locator calculation logic
37. The followings are newline-separated list of id attributes in a HTML. Among them, which ones contain randomly generated texts? "Randomly generated" means, not decided by human, but generated by system and vulnerable to change. Please return the result as a newline-separated list. Note that we don't regard number sequences/prefixes/suffixes as "Randomly generated" texts. [New-line separated id list] 3. Use of Generative AI - ③Detect random IDs Prompt for generative AI
38. 3. Use of Generative AI - ③Detect random IDs Result Generative AI can detect many random IDs. Detected ones are almost always correct But frequently it fails to detect some random IDs Besides, it takes 6 seconds to check just 30 IDs It is difficult to integrate the logic into the MagicPod product
39. Agenda Quick Overview of MagicPod 1. AI Use in MagicPod 2. Use of Generative AI 3. Our Final Goal 4.
40. 4. Our Final Goal Final goal of MagicPod MagicPod can execute tests just from the test case for manual tester Creating a test case completely from scratch is further next goal Log in with user@example.com / pass001
41. Thank you!