Building Sustainable Quality Development

Speaker: Yusaku Watanabe, Shiori Saito
Title: Building Sustainable Quality Development_Integrating Quality Culture and Test Efficiency

Transcript

1. CONFIDENTIAL © PayPay 2025/4/22 Building Sustainable Quality Development 
 Integrating Quality Culture and Test Efficiency 
 1 Yusaku / Shiori 
 PayPay QA Team 

2. CONFIDENTIAL © PayPay Self introduction Yusaku Watanabe Career Shiori (Saito) Trigubenko 2021~ PayPay - QA Manager 2024~ QA Manager (Payments&MiniApp) ~ 2023 Manual QA (QA Operation) 2020~ Cognizant Japan - QA Test Analyst - Worked on AI voice assistant test 2019~ Startup - QA Automation Engineer - Worked on Game console UI/API automation Career 2021~ PayPay - QA Manager 　　　(Financial services & Billpayments) 2019~ Software company - QA Manager - Worked on testing Industrial Equipment 2012~ QA Vender company - QA Engineer - Worked on testing Audio Equipment
3. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction + Recap from QA Night #1 2. Regression Test Optimization and Automation Expansion 3. Expansion of test scope and Shift-left activity 4. Tech Blog / Hiring Info Sharing
4. CONFIDENTIAL © PayPay PayPay QA Team Introduction 4
5. CONFIDENTIAL © PayPay Introduction of the PayPay QA Core App & Growth Product Line Financial Services & Billpayments Product Line Payments & MiniApp Product Line QA Service/QA Tech (Kaizen, Standalization, Tools) Merchant O2O Product Line Core App & Growth QA Financial Services & Billpayments QA Payments & MiniApp QA Merchant O2O QA Total 100+ members! QA Team
6. CONFIDENTIAL © PayPay Recap of QA Night #1 PayPay QA's History Manual & Automation separate team • Outsourcing and managed by dev Establishment of “In-house QA Team” Manual & Automation integration • Have 1 common regression test suite • Automation of the regression test PayPay India QA Integration • Established PayPay India QA Team
7. CONFIDENTIAL © PayPay Recap of QA Night #1 Minimize Manual Maximize Auto Manual & Auto members Auto Script 1 Auto Script 2 Auto Script X Project Test Run & Report Coverage Complementary Maintenance Product Line Manual & Auto QA Test case 1 Test Case 2 Test Case X Integrated Process Auto Regression Test Suite Manual Regression Test Suite Coverage NOT shared Maintenance Manual QA Test case 1 Test Case 2 Test Case X Maintenance Auto QA Auto Script 1 Auto Script 2 Auto Script X Project Test Run & Report Test Run & Report X X Coverage NOT shared QA Legacy Challenge NOW Integrated Regression Test Suite
8. CONFIDENTIAL © PayPay Recap of QA Night #1 -70%! Manual effort reduction by Automation! • On the Regression Test. • Reduced 70% of manual test effort. • Execution can be done within a couple of days.
9. CONFIDENTIAL © PayPay PayPay QA keeps improving for the best product quality… QA Night #2 9
10. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction + Recap from QA Night #1 2. Regression Test Optimization and Automation Expansion 3. Expansion of test scope and Shift-left activity 4. Tech Blog / Hiring Info Sharing
11. CONFIDENTIAL © PayPay Regression Test Optimization and Automation Expansion 11
12. CONFIDENTIAL © PayPay Regression Test Optimization and Automation Expansion 課題 • 日々、機能追加によりリグレッションテストの量が拡大し、実施工数も増加している。 アクション • Manual工数の削減 • Automationの範囲拡大 (70% → 90%) • In-sprint Automation
13. CONFIDENTIAL © PayPay Regression Test Optimization and Automation Expansion • Manual工数の削減 ◦ テストケース実施優先度の再検討 ▪ 従来 : • 機能の重要度で実施優先度を決定 ▪ 改善 : • 機能の重要度 + 過去のテスト結果 + 過去の不具合で実施優先度を決 定 (Data driven approach) ◦ テスト管理ツールの活用 → 不具合分析の効率up ◦ 更にテストシナリオの実施優先度を下げる ◦ 過去半年間、テストシナリオから検出された不具合の Severityから の実施優先度を下げる ◦ Major, Fatalの不具合が発生したテストシナリオについては優先度 を上げる • 削減後、テスト・不具合の結果をトレースしテスト品質を測定
14. CONFIDENTIAL © PayPay Regression Test Optimization and Automation Expansion • Automationの範囲拡大 ◦ Manual テストシナリオのAutomation再検討 ▪ 自動化の強化 • 自動化できなかった機能の自動化検討 ◦ 「決済コードのスキャン」をコード決済をアプリ上で疑似的に再現 し、自動化に成功。 ▪ Manual テストシナリオの分解 • シナリオの中で自動化できる部分を抽出する ◦ 本人確認のフローにおいて、カメラ部分は Manual、UI・テキスト部 分はAutiomation
15. CONFIDENTIAL © PayPay QA Processes Supporting High-Frequency Releases New feature A New feature E QA QA New feature F (Agile) QA Regression Test (ver. 4.55.0) Regression Test (ver. 4.56.0) Regression Test (ver. 4.57.0) New feature B QA New feature C QA New feature D QA - New feature A - New feature B - New feature C - New feature D - New feature E - New feature F STG QA • In-sprint Automation の徹底 ◦ Regression test 初回から自動化でテスト を開始する
16. CONFIDENTIAL © PayPay Regression Test Optimization and Automation Expansion • 3ヶ月で自動化率を約20%増加。 • 機能追加時もスムーズに自動化を行い、自動化率の減少を最低限に留めた。
17. CONFIDENTIAL © PayPay Regression Test Optimization and Automation Expansion • 3ヶ月でManualで実施するケース数を約50%削減。 • テスト品質も維持。 (不具合検出率・市場流出不具合をトレースし確認 )
18. CONFIDENTIAL © PayPay Regression Test Optimization and Automation Expansion • QA 全体の成果 ◦ 自動化率 70% → 90%達成 ◦ 生産性は40％以上向上 ◦ リグレッションテストの総所要工数を 60％以上削減 • これから ◦ 定期的に最適化の対応を行い、最適な Regression testでテストし続ける ◦ Automation rate を更に上げる
19. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction + Recap from QA Night #1 2. Regression Test Optimization and Automation Expansion 3. Expansion of test scope and Shift-left activity 4. Tech Blog / Hiring Info Sharing
20. CONFIDENTIAL © PayPay Initiative of Micro Service E2E Testing 20
21. CONFIDENTIAL © PayPay Initiative of Micro Service E2E Testing Tests to verify the user interface of an application Test carried out at API Gateway layer which communicates with client Test the entire backend of the application, which includes databases, servers, business logic, APIs, and system integration Validate new API/code changes by writing service-level tests within the same service QA QA DEV DEV
22. CONFIDENTIAL © PayPay Initiative of Micro Service E2E Testing Tests to verify the user interface of an application Test carried out at API Gateway layer which communicates with client Test the entire backend of the application, which includes databases, servers, business logic, APIs, and system integration Validate new API/code changes by writing service-level tests within the same service QA QA DEV DEV QA • Focus on edge cases • Cover the use cases that are either missing or not feasible through API gateway testing
23. CONFIDENTIAL © PayPay Initiative of Micro Service E2E Testing 23 Feature Test • Early-stage QA involvement helped in detecting critical issues before E2E, reducing rework and cost. • Collaboration with Dev & Arch teams led to improvements in service implementation and architecture reliability. Microservice E2E Test Regression Test UI Test API Gateway layer Test UI Test + API Test (To ensure existing features should not be broken by every week’s release)
24. CONFIDENTIAL © PayPay Early Automation (Shift-Left) 24
25. CONFIDENTIAL © PayPay How we used to work … #1 All the testing has started once all the modules (Microservice, Gateway API, and front end) are ready, QA starts testing
26. CONFIDENTIAL © PayPay How we used to work … #2 Blocker issue Schedule delays… • Issue 1: If there is a blocker issue on 1~3 days of QA Phase, it impacts on project delivery schedules… • Issue 2: Most of the scenarios need to be tested manually in the feature test which is very expensive
27. CONFIDENTIAL © PayPay Early Automation (Shift-Left) Why not start to involve QA testing even before full STG Ready?
28. CONFIDENTIAL © PayPay Early Automation (Shift-Left) - New activity Shift-left Catch the potential critical issues before starting end to end test Automation scripts cover the compatibility test cases (Different OS versions, Device sizes, Languages) on behalf of manual testing.
29. CONFIDENTIAL © PayPay Success Keys of Microservice E2E testing + Early Automation Success Keys • Shift-Left Testing : Ensures quality is embedded throughout the entire SDLC by involving QA teams early. • Collaboration : Fosters collaboration between developers and testers to enhance product development. • Test Automation : Leverages automation to detect defects sooner, reducing rework and accelerating delivery. • Structured SOPs : Streamlining testing efforts with established procedures to improve efficiency and reliability. • Continuous Improvement : Adapts and enhances practices as the project progresses to support the Shift-Left approach. • Outcomes : Enables faster feedback loops and results in higher-quality software.
30. CONFIDENTIAL © PayPay Improvement & Next Actions for PayPay QA Shift-left #1: Develop a more efficient resource planning #2: Address the increasing workload of automation maintenance
31. CONFIDENTIAL © PayPay Agenda 1. PayPay QA Introduction + Recap from QA Night #1 2. Regression Test Optimization and Automation Expansion 3. Expansion of test scope and Shift-left activity 4. Tech Blog / Hiring Info Sharing
32. CONFIDENTIAL © PayPay Rockstars in PayPay QA! Tech Talks vol.47－The KAIZEN Project Undertaken by All QA Members EN) https://insideout.paypay.ne.jp/en/2025/02/27/techtalks-vol47-en/ JP) https://insideout.paypay.ne.jp/2025/02/27/techtalks-vol47-jp/ Tech Talks vol.48 – Auto launch of PayPay app from Native Camera EN) https://insideout.paypay.ne.jp/en/2025/04/02/techtalks-vol48-en/ JP) https://insideout.paypay.ne.jp/2025/04/02/techtalks-vol48-jp/
33. CONFIDENTIAL © PayPay We are hiring! PayPay Groupでは新しい仲間を募集しています。ご興味のある方はこちらからご応募ください PayPay Group is now looking for new members. If you are interested, please apply here PayPay
34. CONFIDENTIAL © PayPay 34 Thank you!