MagicPod - ①Human-readable UI element captions Human-readable caption ・Calculated by AI ・Not used by the test runner Selenium/Appium locator ・Calculated from UI tree ・Used by the test runner
②Self-healing Suitable for record-and-playback test automation tools When recording a test: Record an element by multiple attributes Such as id, name, index When the test fails: When failing to identify the element by an attribute, try other recorded attributes
②Self-healing Adopted by MagicPod When creating a test: Store locators and the full UI tree (HTML) When the test fails: Compare the run-time UI tree and the recorded UI tree, and use a sufficiently similar element if it exists
②Self-healing Factors for determining elements similarity Element attribute, position, parent-child relationship, etc. OCR result (MagicPod-only) Object recognition result by the deep learning engine (MagicPod-only) Advantages The same simple locator as Selenium/Appium
automation tools such as Selenium, Playwright Not so useful for no-code test automation tools at this time Conversation with generative AI requires more time than just recording user operation Is AI useful for test scenario generation?
Understanding each test case requires large efforts for a new member It is not easy to keep human-written summary up-to-date Generate summary by generative AI
Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Human-readable text The script "Check that signup works correctly" automates end-to-end testing for user signup on a hotel website: 1. It ensures logout if the user is already logged in. 2. Fills out the signup form with unique details like email, password, name, etc. 3. Submits the form and verifies entered data's correctness on the confirmation page. 4. Confirms successful signup by logging out and logging back in, checking for the "MyPage" header's presence. Test script Summary MagicPod public API Generative AI
text Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Title: Check that singup works correctly
script name is "[test script name]". Could you create a summary by which I can grasp what kind of test the script does? =============================================== [Human-readable test script ] 3. Use of Generative AI - ①Summarize test cases Prompt for generative AI
for user signup on a hotel website: 1. It ensures logout if the user is already logged in. 2. Fills out the signup form with unique details like email, password, name, etc. 3. Submits the form and verifies entered data's correctness on the confirmation page. 4. Confirms successful signup by logging out and logging back in, checking for the "MyPage" header's presence. 3. Use of Generative AI - ①Summarize test cases Generated summary
logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Updated the script by modifying the user's name, address, and telephone number inputs to use new values for improved clarity and specificity in test data. 3. Use of Generative AI - ②Generate change comments Experiment Human-readable text (old) Test script (old) change comment Generative AI Test script (new) Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html // Log out once if being logged in If UI element "Logout" button is visible Click "Logout" button Navigate to URL https://hotel.testplanisphere.dev/en-US/signup.html End branch // Sing up by using an unique email address Store unique value generated based on the current time to variable UNIQUE_KEY Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Input ${PASSWORD} to "Password(confirmation)" input area Input Test user to "Name" input area Input Tokyo, Japan to "Address" input area Input 00011122233 to "Tel" input area Select I do not answer. from "Gender" combo box Input 2000-07-04 to "Date of birth" input area Check "Receive notification" checkbox Click "Sign up" button // Assert values set on the previous page are correctly displayed Assert value of UI element Email value equals test-user${UNIQUE_KEY}@example.com Assert value of UI element Name value equals Test user Assert value of UI element Membership value equals Premium Assert value of UI element Address value equals Tokyo, Japan Assert value of UI element Tel value equals 00011122233 Assert value of UI element Gender value equals not answered Assert value of UI element Date of birth value equals July 4, 2000 Assert value of UI element Notification value equals received // Confirm that the user is successfully created and log-in by the user succeeds Click "Logout" button Click Login Input test-user${UNIQUE_KEY}@example.com to "Email" input area Input ${PASSWORD} to "Password" input area Click "Login" button Assert UI element "MyPage" header exists Human-readable text (new)
for generative AI The following 2 browser E2E test automation scripts represent the script before change and the script after the change when an engineer modified the script. Before Change ---------------- [Full human-readable script before the change] ---------------- After change ---------------- [Full human-readable script after the change] ---------------- Please create a change comment in one sentence.
telephone number inputs to use new values for improved clarity and specificity in test data. 3. Use of Generative AI - ②Generate change comments Generated change comments Updated user profile information for signup test to reflect new user details and address. Variation 2 Variation 1
since they are dynamic, cryptic, and vulnerable to change Random ID detection is not always easy 3. Use of Generative AI - ③Detect random IDs Smarter locators not using random IDs
HTML. Among them, which ones contain randomly generated texts? "Randomly generated" means, not decided by human, but generated by system and vulnerable to change. Please return the result as a newline-separated list. Note that we don't regard number sequences/prefixes/suffixes as "Randomly generated" texts. [New-line separated id list] 3. Use of Generative AI - ③Detect random IDs Prompt for generative AI
Generative AI can detect many random IDs. Detected ones are almost always correct But frequently it fails to detect some random IDs Besides, it takes 6 seconds to check just 30 IDs It is difficult to integrate the logic into the MagicPod product
execute tests just from the test case for manual tester Creating a test case completely from scratch is further next goal Log in with [email protected] / pass001
magicpod
cyberagentdevelopers
yuu26
miholovesq
furuton
hide212131
ar_tama
tsujiba
koujikozaki
gimupop
honnibal
marktimemedia
scottboms
mza
brad_frost
chriscoyier
tanoku
irinanazarova
philnash
holman
smashingmag
