Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Explore the world of test automation through in...

atom
January 26, 2025

Explore the world of test automation through insightful real-world case studies

atom

January 26, 2025
Tweet

More Decks by atom

Other Decks in Technology

Transcript

  1. Arrangility Sdn. Bhd. Explore the world of test automation through

    insightful real-world case studies CEO Tomotaka Asagi
  2. © 2025 Arrangility Sdn. Bhd. AGENDA 2 1. Introduction 2.

    Technology Used 3. Case 1: Start by Clearly Stating Your Goal 4. Case 2: Dealing with Operational Issues 5. Summary
  3. © 2025 Arrangility Sdn. Bhd. Self-introduction Tomotaka Asagi CEO 3

    • Passion: Automated Testing • Hobbies: Photography, Driving • Work History: Sales → Development → Testing
  4. © 2025 Arrangility Sdn. Bhd. Specializes in Automated Software Testing

    & QA Consulting 1. Introduction: My Company 5
  5. © 2025 Arrangility Sdn. Bhd. 1. Introduction: My Group Company

    6 Japan Quality Co., Ltd./ DAD Lab (Overseas verification base) 16F, Da Nang Software Park, 02 Quang Trung, Thach Thang Ward, Hai Chau District, Da Nang, VIETNAM ARRANGILITY SDN. BHD. Unit LG-02 & LG-03, Ground Floor, CoPlace 5 Block 3710, Persiaran APEC, Cyber 8, 63000 Cyberjaya Selangor Darul Ehsan, MALAYSIA Human Crest Co., Ltd.(HQ) 26F Yokohama Landmark Tower, 2-2-1, Minatomirai, Nishi-ku, Yokohama City, Kanagawa, JAPAN 220-8126 Japan・Vietnam・Malaysia
  6. © 2025 Arrangility Sdn. Bhd. 8 • Product/Service Used Automated

    regression testing is efficient but incurs significant costs due to frequent UI changes, updates to the test execution environment, network stability issues, etc. "Lynx" is a regression test automation service. Provides a one-stop service from construction to operation of test automation systems. Continuously runs tests to discover and notify you of problems. Checks response time and display distortion. Multi-browser compatibility Early detection of problems and Significant reduction in operating costs Easy installation with no hassle. Maintenance handled by us
  7. © 2025 Arrangility Sdn. Bhd. • Scope • Scope of

    Automated Testing: Currently limited to End-to-End (E2E) testing that interacts with the UI layer. Unit tests and other types are not included • No Blame Assigned: This is not about blaming anyone 10 UI Service Unit
  8. © 2025 Arrangility Sdn. Bhd. • Question that client normally

    asked? • Where to begin: ◦ I want to do it, but where should I start? • How to start: ◦ What technology do you use? ◦ What kind of tests are you running? • What is the challenges to continue: ◦ Too many failures ◦ Why? Crash? 11
  9. © 2025 Arrangility Sdn. Bhd. 2. Common tools & technology

    • Test Automation Framework ◦ Selenium(Selenium Grid) ◦ Appium ◦ Cypress ◦ Playwrite • Testing Framework ◦ JUnit ◦ Cucumber • Test Execution Environment ◦ Real Machine ◦ Workspaces ◦ Docker • CI Environment ◦ Jenkins ◦ GitLab • Scripting Language ◦ Java • Test Case Management ◦ Excel ◦ (Test Framework) • Others ◦ OpenCV(Image Comparison) 14
  10. © 2025 Arrangility Sdn. Bhd. 2. General Configuration Diagram 15

    WorkSpaces Node Hub EC2 Availability Zone ap-northeast-1a Availability Zone ap-northeast-1c LynxDashboard LynxRun GitLab Excel Management Test Case Test Target DockerCompose
  11. 3. Case 1 : Decide your goal 17 Execute Reporting

    Resource Team Culture Skill Set Test Automation Solution GOAL HERE
  12. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Overview •

    Business-to-Business (B2B) Web Services Company • Test target System ◦ System A (FE: AngularJS , BE: PHP) ◦ System B (FE: AngularJS , BE: Java) • Staging Environment 18
  13. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Issues •

    Issues ◦ Concerns regarding product quality, such as the lack of regression testing (involves over 1,000 cases...) ◦ System issues are sometimes only detected through user reports • Request ◦ Want to verify if any components have ceased functioning following the update ◦ Want to identify any existing issues ◦ Want to run daily automated tests 19 Goal → Risk Reduction
  14. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Clarify the

    Goal How testing should be automated? 20
  15. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Test Pyramids

    21 Manual Testing UI Service Unit 2009 Mike Cohn, Succeeding with Agile: Software Development Using Scrum
  16. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Clarify the

    Goal What to be tested: Core function 22
  17. © 2025 Arrangility Sdn. Bhd. 3. Case 1: How I

    Managed the Situation 23 Screen Name Function Name Class Function Risk Manual costs (cost to do manually) Automation cost (cost to automate) Whole Multi-user operations Function Low Low Medium Whole Automatic version upgrade Function Low Low Medium HOME Screen User management Data Medium Low Low HOME Screen Login information Display Low Low Low Login Screen Login/Logout Login High Low Low HOME Screen Notice Display Low Low Low HOME Screen Work memo Display Low Low Low HOME Screen Help Display Low Low Low Whole Smartphone compatibility Function High High Medium Product Management Product information list Display High Low Low Product Management Product search (product list) Search Low Low Low Product Management CSV import Import High High Medium
  18. © 2025 Arrangility Sdn. Bhd. Screen Name Function Name Class

    Function Risk Manual costs (cost to do manually) Automation cost (cost to automate) Test Case Test Case No. Mall Setting Inventory ratio setting Setting High High Low inventoryRatioSetting_testCase inventoryRatioSetting_mallLink_testCcase 2 Product Management CSV import Import High High Medium CSVImport_registration_testCase CSVImport_edit_testCase 2 Linkage by product and mall Display High High Medium productAndMallUnitLink_product_testCase productAndMallUnitLink_stock_testCase productAndMallUnitLink_mall_testCase productAndMallUnitLink_stock_mallLink_testCase productAndMallUnitLink_mall_mallLink_testCase 5 Log Mall update log Display High High Medium mallUpdateLog_log_testCase mallUpdateLog_csvExport_testCase mallUpdateLog_search_testCase 3 Mall Setting Mall update time setting Setting High High Medium mallUpdateTimeSetting_testCase 1 System Setting Low stock email notification Setting High High High? lowStockEmailNotification_testCase lowStockEmailNotification_canSent_mallLink_testCase lowStockEmailNotification_cannotSent_mallLink_testCase 3 Log sales log Display High High High? salesLog_log_testCase salesLog_csvExport_testCase salesLog_search_testCase 3 Mall Setting Order cancellation setting Data High High High? orderCancellationSetting_testCase orderCancellationSetting_start_mallLink_testCase orderCancellationSetting_stop_mallLink_testCase 3 Mall Setting FBA multichannel setting Setting High High High? fbaMultichannelSetting_testCase 1 Mall Setting API setting Setting High High High? apiSetting_doneSetting_testCase 1 3. Case 1: How I Managed the Situation 24
  19. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Result •

    Test case number ◦ System A: 20 cases ◦ System B: 77 cases • Time until the first test case is executed regularly: 3 weeks • Test execution frequency: Once a day • When a test fails: Send summary and screenshot to Slack • Completion of all test cases: 2 months 25
  20. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Changes •

    Scrum was implemented • Automation of unit tests has been initiated • Now able to refactor • Future cautions ◦ Avoid becoming an anti-pattern 26 https://testing.googleblog.com/2020/11/fixing-test-hourglass.html
  21. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Keep in

    mind 27 For quality, ALWAYS do the right software testing
  22. © 2025 Arrangility Sdn. Bhd. 3. Case 1: Summary •

    Do’s ◦ Test independency ◦ Test idempotency ▪ Preparation(Condition necessary for testing, e.g. test data, etc.) ▪ Cleaning up(Deleting unnecessary data) • Don’ts ◦ Automate manual test cases ◦ Automate regression test cases → Do not aim automate as goal 28
  23. © 2025 Arrangility Sdn. Bhd. 4. Case 2 31 CASIO

    COMPUTER CO., LTD. CASIO WATCHES
  24. © 2025 Arrangility Sdn. Bhd. 4. Case 2: Overview 「CASIO

    WATCHES」 conduct automated test focuses on regression checks, to confirm that basic functions are operating normally. • Goal ◦ Early detection of defects during development, reduction of rework, reduction of development cost, and reduction of testing man-hours ◦ Overall check after production release, early detection of market defects, ensuring market quality • Target ◦ Regression testing in development environment (regression testing) ◦ Regression testing in production environment (regression testing) 32
  25. © 2025 Arrangility Sdn. Bhd. 4. Case 2: Overview •

    Target App ◦ Production app ◦ Development app ◦ (just before release) • Test Content ◦ Functional test ◦ 284 cases/suite • Text Execution Frequency ◦ Run twice a day(12 AM, 12 PM) 33
  26. 4. Case 2: Automated Testing Overall Configuration 34 1,200 Execution/day

    Production Env. Production Env. Development Env. Development Env.
  27. © 2025 Arrangility Sdn. Bhd. 4. Case 2: Issues 1.

    Difficulty in managing test failures 2. Test is too unstable 35
  28. © 2025 Arrangility Sdn. Bhd. 4. Case 2: Issues Faced

    in Automated Test Operations Automated test & Problem Solving 36
  29. © 2025 Arrangility Sdn. Bhd. 4. Case 2: First Issue

    in Automated Test Operations • Issue: Test failures lead to lengthy investigation and resolution time ◦ Each case execution time takes 10-30 minutes → Rerun → Crashes midway... And crashes again during correction checks • Cause ◦ Automated screen-by-screen test cases • Countermeasure ◦ Redesign and split cases for automated testing. ◦ Increased from 48 → 284 cases (Final : 500 cases) • Effect ◦ Significantly reduces investigation time for test failures ◦ Test are now stable 37 Too Long!
  30. © 2025 Arrangility Sdn. Bhd. 4. Case 2: Second Issues

    in Automated Test Operations • Issue: Tests fail due to minor screen changes ◦ Frequent modification of test scripts, leading to instability (There were always a lot of unstable parts) • Cause ◦ Long XPATH changes during scrolling ◦ Accuracy issues with OpenCV image matching • Countermeasure ◦ Add an ID (recognized by Appium) • Effect ◦ Test are now stable ◦ Enhanced collaboration with the development team 38
  31. © 2025 Arrangility Sdn. Bhd. 4. Case 2: Key Points

    for Continuation 1. Assign a Responsible Person 2. Address Failed Tests Promptly: Establish a policy within 24 hours 3. Ensure Test Stability: Remove any unstable tests 4. Continuously Refactor Tests: Reinforce the goal and value of your tests 5. Seek Team Collaboration: Engage developers and other team members 39
  32. © 2025 Arrangility Sdn. Bhd. 5. Summary of E2E automated

    testing • Focus on basic tests • Start immediately • Refactor as needed • Collaborate effectively 41 UI Service Unit