Pull Request-Driven E2E Testing with Kubernetes

Hiroaki Ozono
LINE Fukuoka Test Automation Team Manager
https://linedevday.linecorp.com/2020/ja/sessions/8307
https://linedevday.linecorp.com/2020/en/sessions/8307

Transcript

2. Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
3. Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
4. Session Overview E2E test
5. Basic E2E Test PR - 1 Pull Request-Driven Test Session Overview Today I would like to talk about; Production E2E Test Server Server Server Server Data Server Server Server Server Data Server Server Server Server Data E2E Test Staging PR - 2 Server Server Server Server Data E2E Test PR - 3 Server Server Server Server Data E2E Test
6. Session Overview Today I would like to talk about; Engineering E2E test Role SET* *Software Engineer in Test
7. Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
8. Introduction Automation test team Technical stacks - Selenium for web app - Appium for mobile app - We make own execution and reporting system Automation Test Team Overview - 6 members - Mainly E2E testing. Not only web app, but also mobile app. Image: https://linefukuoka.co.jp/ja/career/engineer/interview_set
9. Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Test › Tasks we do › Features › Pull Reqest-Driven E2E test › Why we tackle it › Architecture › Improvements › Summary
10. Basic E2E Test Tasks we do - In the case of LINE Creators Market project
11. Basic E2E Test Tasks we do - In the case of LINE Creators Market project For Two Yeas 2 Test cases 100+ Environments 2
12. Basic E2E Test Features › The E2E Test can detect a very little change which cannot be found by human › It helps to improve the quality of the application by incorporating it into the iterative Regression test
13. Basic E2E Test Features › Stateful datasets › Low parallelism and long execution time
14. Basic E2E Test Features › Stateful datasets › Low parallelism and long execution time Parallelism Staging Server Server Server Server Data Tester Test B Test A Developer
15. Basic E2E Test Features › Stateful datasets › Low parallelism and long execution time Parallelism Execution time Staging Server Server Server Server Data Tester Test B Test A Developer Test 1 Test 2 Test 3 Test 4
16. Basic E2E Test Features › Late feedback to the developers › Sometimes, it is too late that finding a bug in the staging environment › When has your PR been merged??? › Are problems always simple? Might be NO in many cases › Stateful datasets › Low parallelism and long execution time
17. Basic E2E Test Features › Late feedback to the developers › Sometimes, it is too late that finding a bug in the staging environment › When has your PR been merged??? › Are problems always simple? Might be NO in many cases › Deploy services on Kubernetes each pull requests › Based on pull request codes › For E2E test › Stateful datasets › Low parallelism and long execution time
18. About SET Role Engineering E2E test Role SET
19. SET? DX? Definitions › SET › “Originally software engineers with deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.” › Ref. https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html › Not Only Scripting E2E test cases
20. SET? DX? Definitions › SET › “Originally software engineers with deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.” › Ref. https://testing.googleblog.com/2016/03/from-qa-to-engineering-productivity.html › Not Only Scripting E2E test cases › DX › Developer eXperience › Helps developers as a developer
21. SET? DX? The target to contribute Test Automation Engineer *Developer eXperience Application Quality DX
22. SET? DX? The target to contribute *Developer eXperience Application Quality SET DX
23. SET? DX? The target to contribute *Developer eXperience Application Quality SET DX Test Automation Engineer
24. Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Test › Tasks we do › Features › Pull Reqest-Driven E2E test › Architecture › Improvements › Summary
25. Pull Request-Driven E2E Test An actual flows Engineering E2E test Role SET
26. Pull Request-Driven E2E Test Same as Pull Request Lifecycle Deploy Test Delete Build Review Test Merge Create Pull Request
27. Pull Request-Driven E2E Test Same as Pull Request Lifecycle Review Test Merge Create Pull Request - Modify codes - Make a branch - Push codes to the Github - Review from colleagues - Do the Unit tests / Integration tests with CI services - Merge code modification to the target branch
28. Pull Request-Driven E2E Test Same as Pull Request Lifecycle - Build docker images - Publish docker images to the docker registry - Register the domain to the DNS server - Deploy services to the Kubernetes cluster - Run E2E tests on services deployed in Kubernetes - Delete deployed services when pull request is merged Deploy Test Delete Build
29. Pull Request-Driven E2E Test Overview Drone / CircleCI Github Docker Registory Virtual Browser E2E Test Github App Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data
30. Pull Request-Driven E2E Test Overview Drone / CircleCI Github Docker Registory Github App Build
31. Pull Request-Driven E2E Test Overview Drone / CircleCI Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Deploy
32. Pull Request-Driven E2E Test Overview Virtual Browser E2E Test Github App Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Test
33. Pull Request-Driven E2E Test Overview Drone / CircleCI Kubernetes Cluster PR - 1 Server Server Server Server Data PR - 2 Server Server Server Server Data PR - 3 Server Server Server Server Data Delete
34. Routing Manifests Pull Request-Driven E2E Test
35. How to Control Kubernetes Manifests › Controlling / Managing Kubernetes manifests is one of the important things › Inject variables dynamically › e.g. pull request number, commit hash
36. How to Control Kubernetes Manifests › Controlling / Managing Kubernetes manifests is one of the important things › Inject variables dynamically › e.g. pull request number, commit hash › Kustomize? Helm? ksonnet? › In our usage, anything is fine › We chose Kustomize › Since 1.14, Kubectl supports the management of Kubernetes objects using a kustomization file
37. Inject Kustomize: https://kustomize.io/ Kustomize file Kubernetes manifests How to Control Kubernetes Manifests
38. Inject Kustomize: https://kustomize.io/ Kustomize file Kubernetes manifests How to Control Kubernetes Manifests
39. How to Control the Routing Test Instance Virtual Browser Load Balancer Virtual Browser Test Instance User *.e2e-test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx
40. Kubernetes Cluster Test Instance NGINX Ingress Controller Virtual Browser Load Balancer Virtual Browser Test Instance User Namespace pr-1 *.e2e-test.example.com Service Service Service Pod Service Service Pod pr-1.e2e- test.example.com Namespace pr-2 Service Service Service Pod Service Service Pod pr-2.e2e- test.example.com Namespace pr-3 Service Service Service Pod Service Service Pod pr-3.e2e- test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx How to Control the Routing
41. Github App Label Command
42. Pull Request-Driven E2E Test › Stateful datasets › Issues in parallel execution and execution time › Ensure the independence of each pull request
43. Pull Request-Driven E2E Test › Late feedback to the developers › it is too late that finding a bug in the staging environment › Running E2E tests on Pull Request has made the feedback earlier › Reduce the number of bugs that are getting into the Staging environment › Also, it can be used for reviewing pull requests as a PREVIEW › Stateful datasets › Issues in parallel execution and execution time › Ensure the independence of each pull request
44. Use as Preview Usually On local Server Server Server Server Data Developer Github Access Pull Build & Launch
45. Use as Preview Usually On local Server Server Server Server Data Developer Github Access Pull Build & Launch PR-Driven Test On K8s Server Server Server Server Data Developer Github Access Just Access
46. Ref. https://linecorp.com/pdf/en/LINE_STYLE_BOOK_EN.pdf
47. Ref. https://linecorp.com/pdf/en/LINE_STYLE_BOOK_EN.pdf
48. Keep Going to Improve Perfectly stateless Completely parallelize Build - Test Less than 5min
49. Before PR - 1 Server Server Server Server Data Test A Test C Test B After PR - 1 - 1 Server Server Server Server Data PR - 1 - 2 Server Server Server Server Data PR - 1 - 3 Server Server Server Server Data Test A Test B Test C Keep Going to Improve
50. Namespace pr-1 Kubernetes Cluster Github App Service Service Service Pod Service Service Pod Test Job 1 Service Service Service Pod Service Service Pod Service Service Service Pod Service Service Pod Test Job 2 Test Job 3 Deploy&Test Deploy&Test Deploy&Test Deploy Trace ID Trace ID Trace ID Keep Going to Improve
51. Kubernetes Cluster Namespace pr-1 Load Balancer User *.e2e-test.example.com NGINX Ingress Controller: https://github.com/kubernetes/ingress-nginx Preview Service Service Service Pod Service Service Pod NGINX Ingress Controller pr-1.e2e- test.example.com Keep Going to Improve Service Pod Pod Test Job 1 Service Pod Pod Service Pod Pod Test Job 2 Test Job 3 Deploy&Test Deploy&Test Deploy&Test
52. Agenda › Session Overview › Introduction › Automation Test Team › Basic E2E Testing › Tasks we do › Features › Pull Reqest-Driven E2E testing › Why we tackle it › Architecture › Improvements › Summary
53. Summary Engineering E2E test Role SET › The role change from test automation engineer to SET › Not only scripting test cases › Being a part of service development to improve DX
54. Summary Engineering E2E test Role SET › Improving the E2E test environment with Kubernetes › Deployed services for each pull requests and created an E2E test environments with fast feedback
55. Thank you