Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reduce flaky test cases with Gradle retry plugi...

Reduce flaky test cases with Gradle retry plugin and Allure report

Elena Alejo (LINE Corporation, DSE Center SET)

LINE Developer Meetup #64 "Development and Test"
https://line.connpass.com/event/179721/

LINE Developers

July 08, 2020
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. SET team (Software Engineer in Test) 3 • We consider

    ourselves DEVELOPERS • Create testing tools for Developers based on their necessities • Experiment • Coach into Agile Testing I am a SET in LINE * Created by Avatar created by Avatars in pixels
  2. Bi-weekly meeting 5 • We do bi-weekly meeting with LINE

    Shop, is our time for improve testing at LINE Shop Goal Look for opportunities to improve testing Do experiments and demo them Create a backlog for the next two weeks, everybody is welcome to do any tasks Facilitate the creation of the test cases Testing good practices Out of scope SET team do all the testing At LINE, SET/Dev ratio is quite low, so we focus on provide Testing infrastructure to Developers, so they can make their own tests
  3. 7 Sometimes Jenkins Job fails randomly It could be great

    if we can retry the test cases Next execution works without changing anything!
  4. Flaky test cases • Give us different results under same

    conditions • Give us false negative, test fails consistently but code is ok Due • Test code quality is bad • Test dependencies with external system 10
  5. Gradle Retry plugin • Configure Retries adding Gradle retry plugin

    • You can configure the plugin to stop, if test is failing for more than x times 12 Retrying allow us to reduce the Noise of false negative test cases
  6. Gradle Retry plugin • If a test fails, it will

    be retried up to 3 times Output: Builds percentage of failure drop We are building trust! 13
  7. 15 Job Execution is green again! Yeap, but test Report

    is messy, repetitions makes harder to understand the report Also we did not Fix any issue, just put a band aid to the job execution! 4 retries Flaky test cases are still there!!
  8. Allure is a test reporting tool “Allure Framework is a

    flexible lightweight multi-language test report tool that not only shows a very concise representation of what have been tested in a neat web report form, but allows everyone participating in the development process to extract maximum of useful information from everyday execution of tests” 17
  9. Next bi-weekly meeting WFH! 18 Zoom photos: “Every Type of

    Zoom call Participant, Illustrated by Cats (https://medium.com/tenderlymag/eve ry-type-of-zoom-call-participant-illustrated-by-cats-c8a6592195b1)
  10. 21 Cool I like the colors! If a test case

    is retried it is not adding noise to the report I like it too! How I know if a test is Retried?, can I filter so I investigate if the test is flaky You cannot filter By retried, let’s customize our report!
  11. Next bi-weekly meeting WFH! 24 Inspired by “Every Type of

    Zoom call Participant, Illustrated by Cats (https://medium.com/tenderlymag/every- type-of-zoom-call-participant-illustrated-by-cats-c8a6592195b1)
  12. 28 cool, so coverImageUrlOrBlank() is a flaky test case, because

    it failed and passed under same conditions Let’s mark it as flaky! But what happens if We access to an old test report data? As Jenkins deletes the build info every X Executions we will lose report data. Let’s fix it!
  13. 34 cool, so we are not going to lost data

    Anymore!! We can create a lot of metrics! I cannot wait to create more metrics! Finally!
  14. Conclusions (Bi-weekly meeting) Pros • It’s nice to have a

    space for chat about testing • Getting in touch with Developers allow us to know their necessities • Developers point of view is great for grow as SET • Collaboration with Dev helps us to understand its infrastructure 36
  15. Conclusions (Bi-weekly meeting) Cons • Limitations, as we are not

    part of the development cycle we miss opportunities to improve testing • Difficult to get more developers time for testing 37
  16. Conclusions (Allure) Pros • Open source project • Report’s UI

    is nice • Integration with several test engines (JUnit, Pytest…) • Able to integrate with Jenkins and other CI tools • Historic and metrics are a good starting point 38
  17. Conclusions (Allure) Cons • A bit difficult to set up

    in a large project (once you understand how it works is easier) • Once you have more then 10.000 test cases, loading is slow • Historic works if you run test cases few times a day • Lack of documentations (up to date, examples...) • I have to read source code to be able to create customized plugins • Architecture is not best one (not easy to host it) • Community is big but not 100% supportive 39
  18. It’s allure worth it? • It’s a report that can

    be used for multiple users • It’s a gateway to ES metrics • I can customize it easily 40 Allure can be used with most of LINE test projects! YES!