QA Processes Supporting High-Frequency Releases

Speaker: Koichiro Takashima
Title: QA Processes Supporting High-Frequency Releases

Transcript

1. CONFIDENTIAL © PayPay 2024/9/26 QA Processes Supporting High-Frequency Releases 
 1 Koichiro Takashima 
 PayPay QA Team 

2. CONFIDENTIAL © PayPay Self introduction 高嶋 康一郎 (Koichiro Takashima) 2023~ PayPay - QA Engineering Manager - Working on QA of one of the product lines. 2014~ Rakuten - Test Engineer, Manager - Worked on various types of platform (e.g. Game console, Smartphone app, Web app) 2012~ SIer - Application Engineer - Application development in SIer (mainly C#) Career Experienced services, roles PayPay, Online Payment service, Flea market app, Internet supermarket, e-book, e-magazine, Messaging app, Game console app etc., QA team management, QA environment management, Test device management Hobbys Travel, Gadgets, Cooking, Bread making by いらすとや
3. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction 2. A Processes Supporting High-Frequency Releases 3. Collaboration of a Multinational Team 4. Our Future Plans
4. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction 2. A Processes Supporting High-Frequency Releases 3. Collaboration of a Multinational Team 4. Our Future Plans
5. CONFIDENTIAL © PayPay PayPay QA Team Introduction 5
6. CONFIDENTIAL © PayPay Introduction of the PayPay QA Core App & Growth Financial Services & Billpayments Payments & MiniApp QA Service/QA Tech (Kaizen, Standalization, Tools) Merchant O2O Core App & Growth QA Financial Services & Billpayments QA Payments & MiniApp QA Merchant O2O QA Total 130+ members! QA Team Performance and Stability Assurance (PSA)
7. CONFIDENTIAL © PayPay Introduction of the PayPay QA PayPay QA's History Manual & Automation separate team • Outsourcing and managed by dev Establishment of “In-house QA Team” Manual & Automation integration • Have 1 common regression test suite • Automation of the regression test PSA (Performance Test) Integration • Established PayPay India QA Team
8. CONFIDENTIAL © PayPay Introduction of the PayPay QA Minimize Manual Maximize Auto Manual & Auto members Auto Script 1 Auto Script 2 Auto Script X Project Test Run & Report Coverage Complementary Maintenance Product Line Manual & Auto QA Test case 1 Test Case 2 Test Case X Integrated Process Auto Regression Test Suite Manual Regression Test Suite Coverage NOT shared Maintenance Manual QA Test case 1 Test Case 2 Test Case X Maintenance Auto QA Auto Script 1 Auto Script 2 Auto Script X Project Test Run & Report Test Run & Report X X Coverage NOT shared QA Legacy Challenge NOW Integrated Regression Test Suite
9. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction 2. A Processes Supporting High-Frequency Releases 3. Collaboration of a Multinational Team 4. Our Future Plans
10. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases Release Note 4.60.0 - iOS / Android / Web - 2024.09.04 Release Release Note 4.61.0 - iOS / Android / Web - 2024.09.11 Release Release Note 4.62.0 - iOS / Android / Web - 2024.09.18 Release ... Weekly Release!
11. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 11 How? By いらすとや
12. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 12 What are the blockers of high frequency release?
13. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 13 Regression test New feature test Time consuming factors - Manual test execution - Unoptimized process - Test estimation - Reporting by いらすとや
14. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 14 Regression test New feature test Time consuming factors - Manual test execution - Unoptimized process - Test estimation - Reporting by いらすとや
15. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 15 Automation
16. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases -70%! Manual effort reduction by Automation! • On the Regression Test. • Reduced 70% of manual test effort. • Execution can be done within a couple of days.
17. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 17 Process
18. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases Project A Feature test Regression test Release Project C Feature test Regression test Release Project B Feature test Regression test Release Regression test on each project…?
19. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases Project A Feature test Regression test Release Project C Feature test Regression test Release Project B Feature test Regression test Release Regression test on each project…? - Need extra effort = cost - Quality risk (branch management)
20. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases PayPay QA Regression Test • Separate from new feature test • Regular execution (Weekly) • Fixed members and resources
21. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases New feature A New feature E QA QA Regression Test (ver. 4.55.0) Regression Test (ver. 4.56.0) Regression Test (ver. 4.57.0) New feature B QA New feature C QA New feature D QA - New feature A - New feature B - New feature C - New feature D - New feature E
22. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases New feature A New feature E QA QA New feature F (Agile) QA Regression Test (ver. 4.55.0) Regression Test (ver. 4.56.0) Regression Test (ver. 4.57.0) New feature B QA New feature C QA New feature D QA - New feature A - New feature B - New feature C - New feature D - New feature E - New feature F
23. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases New feature A New feature E QA QA New feature F (Agile) QA Regression Test (ver. 4.55.0) Regression Test (ver. 4.56.0) Regression Test (ver. 4.57.0) New feature B QA New feature C QA New feature D QA - New feature A - New feature B - New feature C - New feature D - New feature E - New feature F STG QA
24. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases New feature A New feature E QA QA New feature F (Agile) QA Regression Test (ver. 4.55.0) Regression Test (ver. 4.56.0) Regression Test (ver. 4.57.0) New feature B QA New feature C QA New feature D QA - New feature A - New feature B - New feature C - New feature D - New feature E - New feature F STG QA - For Feature Test - Each module version is based on each feature test - For Regression Test - Each module version is same as PROD
25. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 25 Regression test New feature test Time consuming factors - Duplicate test - Manual test - Test estimation - Reporting by いらすとや
26. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 26 Test estimation
27. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases Test estimation Need at least several days (2~7 days) Test estimation tasks - Listing up test items - How many test cases - Number of test device - Resource allocation - Buffer So many tasks! By いらすとや
28. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases Test estimation Bottom-up estimation Analogical estimation 2~7 days less than a day by unsplash by unsplash
29. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases 29 Reporting
30. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases QA sign-off report (Release judgement report) Manually need to gather data from different places. Tool A - Project info - Requirement - Specs Tool B - Test cases Tool C - Bug tickets By いらすとや
31. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases QA sign-off report: Centralized reporting tool & Automation Atlassian tools - Project info - Requirement - Specs - Test cases - Bug tickets Integrated tools 2~3 days > A couple of hours Automated report creation tool (Google Extension) By いらすとや By Google
32. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases Automation Test Report: Cetralized Reporting Tool Kibana (All test results status and Report) ReportPortal (Screenshots, Logs. For analysis.)
33. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction 2. A Processes Supporting High-Frequency Releases 3. Collaboration of a Multinational Team 4. Our Future Plans
34. CONFIDENTIAL © PayPay Collaboration and Culture of a Multinational Team Engineers from all over the world By unsplash • India • Japan • Taiwan • Indonesia • China • Vietnam • US • Russia • Mexico • Korea • Germany • Bangladesh
35. CONFIDENTIAL © PayPay Collaboration and Culture of a Multinational Team Japanese Speaker Bilingual, Multi-lingual English Speaker How we are communicating? By いらすとや By いらすとや By いらすとや
36. CONFIDENTIAL © PayPay Collaboration and Culture of a Multinational Team Kiara (Slack) AI Interpreter
37. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction 2. A Processes Supporting High-Frequency Releases 3. Collaboration of a Multinational Team 4. Our Future Plans
38. CONFIDENTIAL © PayPay Future Plans Advanced Testing Technologies Visual Testing 
 CI/CD, DevOps 
 AI Utilization 
 By unsplash
39. CONFIDENTIAL © PayPay Future Plans Automation Expansion Increasing Automation coverage 
 SDET 
 by unsplash By いらすとや
40. CONFIDENTIAL © PayPay Future Plans - PayPay SDET 40 SDET (Software Development Engineer in Test): A specialized role combining software development and testing skills to design, develop, and implement automated tests, ensuring robust and high-quality software delivery. Recent Contributions: • Impact Analysis Tool: Analyzes change impacts. • DevTestPro: This tool automates CI testing, enabling early bug detection and supporting Shift Left.. • Pipeline Optimization: Enhances testing pipeline efficiency. Benefits of Expanding the SDET Team: • Accelerate Automation: Speed up automation and support new tool development. • Enhance Testing Coverage: Improve back-end and integration testing. • Develop Frameworks: Create and refine testing frameworks. • Early Automation: Collaborate for early bug detection. • Tool Development: Implement new tools and APIs. Key Advantage: • Increased Efficiency: Boost automation capabilities and overall project quality. by OpenArt　
41. CONFIDENTIAL © PayPay We are hiring! PayPay Groupでは新しい仲間を募集しています。ご興味のある方はこちらからご応募ください（現在、 PayPay 証券、PayPayカードで採用中です） PayPay Group is now looking for new members. If you are interested, please apply here (We are currently hiring for PayPay Securities and PayPay Card) PayPay Securities PayPay Card