Upgrade to Pro — share decks privately, control downloads, hide ads and more …

How we continuously delivery LINE TODAY App wit...

How we continuously delivery LINE TODAY App with high agility and high quality

Kuan-Wei Lin
LINE Taiwan Senior Quality Engineer
https://linedevday.linecorp.com/jp/2019/sessions/F2-4

LINE DevDay 2019

November 21, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. 2019 DevDay How We Continuously Delivery LINE TODAY App With

    High Agility and High Quality > Kuan-Wei Lin > LINE Taiwan Senior Quality Engineer
  2. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal

    Deploy External Deploy UI Tests Integration / API Tests Release Parallel Tasks Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android
  3. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal

    Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Automation Parallel Tasks Device Lab Parallel Tasks iOS Android
  4. Automation Pull Request Code Review Build Unit Tests Integration /

    API Tests Parallel Tasks External Deploy UI Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android CI/CD Puzzle Internal Deploy xcodebuild test -only-testing:Tests ./gradlew assemble testUnitTest
  5. Automation Parallel Tasks Internal Deploy External Deploy UI Tests Release

    Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Integration / API Tests Code Review Build Unit Tests CI/CD Puzzle Pull Request xcodebuild test -only-testing:UITests -destination* ./gradlew connectedAndroidTest
  6. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal

    Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Parallel Tasks Automation
  7. CI/CD Puzzle Pull Request Code Review Build Unit Tests Internal

    Deploy External Deploy UI Tests Integration / API Tests Release Regression Tests Exploratory Tests Device Lab Parallel Tasks iOS Android Parallel Tasks Automation
  8. Unit Tests UI Tests > What Is Good Code Coverage

    To Have? > How To Have Reliable UI Automation Tests?
  9. Agenda > UI Automation Tests Guideline > Unit Tests &

    Code Coverage Strategy Unit Tests UI Tests
  10. Testing Spectrum Integration / API Tests UI Tests Regression Tests

    Unit Tests Unit End To End Resource • API automation testing • Load testing • Database integration tests • Native test frameworks • Device lab • UI automation guideline • Agile testing • Exploratory testing • Logic & UI separation • Test double (Stub/Mock/Fake) • TDD (Test-Driven Development) • Code coverage strategy Automation Manual
  11. Why Flaky Tests Are Blockers? 1st 2nd 3rd 4th 5th

    6th 7th 8th 9th 10th Test A Test B Test C … … Result Automation Runs
  12. Why Flaky Tests Are Blockers? 1st 2nd 3rd 4th 5th

    6th 7th 8th 9th 10th Test A Test B Test C … … Result Automation Runs
  13. 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th

    Test A ? Test B ? Test C ? … 100 More UI Tests? Result ? Why Flaky Tests Are Blockers? Automation Runs
  14. > Known issues of test frameworks (XCUITest / Espresso) >

    Different device dependency > Cross App testing instability > Parallel testing instability > IDE run v.s. Command line run > And more… External Flaky Factors > UI may change in every release > Test environment instability > Test data instability > Dynamic UI elements design > Poorly written test code Internal Flaky Factors UI Automation Testing Vulnerability
  15. Be Aware of Maintenance Cost and Flaky Tests Before Starting

    UI Automation Tests UI Automation Testing Is Smoke Testing
  16. Smoke Cases • Most common user scenarios • Mature functionality

    • Suitable to automate Regression Cases • Features which are keep developing • Hard / High cost to automate UI Automation Tests Regression Tests Automation Manual Acceptance Criteria
  17. 0% 20% 40% 60% 80% 0 400 800 1200 1600

    App Versions 3.0 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 Manual Regression Tests UI Automation tests Ratio 12.9% 11% 20.1% 19.3% 30.5% 38.2% 43% 51.5% 50.1% 50.4% 914 950 1,279 1,519 1,186 1,074 1,054 536 622 400
  18. API Server API + MVC API Client Models Controllers View

    User action Update Notify Update Notify / Parsing Update Unit Tests UI Tests SERVER CLIENT Request Response
  19. API Server Models Notify UIViewController View Controller API Client Notify

    / Parsing Update ? SERVER CLIENT iOS MVC Request Response Unit Tests UI Tests
  20. API Server iOS MVVM Models Notify UIViewController View Controller ViewModel

    Binding API Client Update Notify / Parsing SERVER CLIENT Request Response Unit Tests UI Tests
  21. Models UIViewController View Controller ViewModel API Client Unit Tests UI

    Tests Manager Helper DB Animation Cell View iOS Code Coverage Formula
  22. Code Coverage Functionalities ViewModel Model API Client Manager Helper… UI

    ViewController View Cell Animation… Unit Tests UI Tests
  23. Code Coverage UI Not in UI Tests Functionalities ViewModel Model

    API Client Manager Helper… UI in UI Tests ViewController View Cell Animation… Unit Tests UI Tests
  24. Code Coverage New Functionalities New UI Functionalities ViewModel Model API

    Client Manager Helper… Unit Tests UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
  25. Code Coverage New UI Not in UI Tests New UI

    in UI Tests Functionalities ViewModel Model API Client Manager Helper… New Functionalities Unit Tests UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
  26. Code Coverage System Callback, Extension, Utility… Functionalities ViewModel Model API

    Client Manager Helper… New Functionalities Unit Tests UI Tests New UI Not in UI Tests New UI in UI Tests UI Not in UI Tests UI in UI Tests ViewController View Cell Animation…
  27. Code Coverage New UI in UI Tests Functionalities ViewModel Model

    API Client Manager Helper… System Callback, Extension, Utility… New Functionalities Unit Tests UI Tests UI in UI Tests ViewController View Cell Animation… New UI Not in UI Tests UI Not in UI Tests
  28. Code Coverage ?% Unit Tests UI Tests System Callback, Extension,

    Utility… New UI Not in UI Tests UI Not in UI Tests
  29. Unit Integration User Interface Fatal Issues Functional Issues UI Issues

    Regression Tests Exploratory Tests Automation Manual Edge case Issues
  30. CI/CD Pipeline | Testing Spectrum In Progress Other Projects Now

    > UI automation tests guideline > Unit tests & code coverage strategy • Web Application • … LINE TODAY LINE LINE SDK LINE MUSIC (Taiwan) > Automated pipeline