Continuous Integration for Android using Jenkins

This talk explains how Jenkins can be used to improve product quality, stability and delivery speed by embracing continuous integration. We’ll walk through how Android developers and testers can benefit from using Jenkins, and then give some pro tips for more advanced capabilities, including how to speed up your tests by running them in parallel across multiple Jenkins slaves.

It was presented at the Android Summit 2016: http://androidsummit.org/

Dan Jarvis: http://careers.stackoverflow.com/dj
Sam Edwards: https://twitter.com/handstandsam

Transcript

1. CONTINUOUS INTEGRATION FOR ANDROID USING JENKINS
2. 05 About Us – Dan Jarvis & Sam Edwards • Tech Leads for Capital One Wallet on Android • Top rated finance app by a US bank • First US bank with native Android contactless payments • Each has 12+ years professional experience • Passionate about automation and knowledge sharing @HandstandSam
3. • Continuous Integration (CI) fundamentals • Productivity tips • Capital One Wallet case study • Jenkins as a platform • Pro tips - how to run tests in parallel 05 Agenda
4. CONTINUOUS INTEGRATION FUNDAMENTALS
5. What is CI?
6. Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early. ”
7. Why do CI?
9. Why? • Deliver faster and higher quality • Your product is always ready to deploy • Capture institutional knowledge in tests • Automation hugely improves productivity • Multiply your impact!
10. What is Jenkins?
11. A server that executes jobs based on triggers. ”
12. Jenkins Terminology •Jobs •Plugins •Master, slaves and executors
13. Jenkins Jobs – “A Job is essentially a script”
14. Jenkins Jobs - Configuration
15. Jenkins Jobs – Execution & History
18. Jenkins Plugins • Plugins can be written to extend Jenkins capabilities Default Install Plugins Added
19. 05 Android Emulator Plugin
20. Jenkins Master, Slaves and Executors • Jobs are configured on the master node • The master dispatches jobs to run on executors on slave nodes (which could be on the same machine as the master)
21. What Makes Jenkins so Great? • Easy, installation, configuration & a web based interface • Open source https://github.com/jenkinsci/jenkins • >1000 plugins • Distributed and scalable • Long history and widespread adoption • Enterprise integrations
22. PRODUCTIVITY TIPS
23. Real-time Notifications
26. Spoon - https://github.com/square/spoon
30. Screenshots on Assertion Failure Spoon.screenshot(activity,"test_failure");
31. Spoon & Screenshots
32. More on Spoon, Espresso and Screenshots http://droidcon.nyc/ http://www.devfestdc.org/
33. How to Run Spoon
34. Custom Links to Spoon Results
35. Running Specific Instrumentation Tests (docs) Run By Instrumentation Argument(s) Code Method -e class com.FooTest#test1 Class -e class com.FooTest Standard Annotations -e size small -e size medium -e size large @SmallTest @MediumTest @LargeTest Custom Annotations -e annotation -e notAnnotation @MyCustomAnnotation Package –e package
36. Android Instrumentation Runner (docs)
37. Spoon Arguments
38. CASE STUDY WALLET
40. The Testing Pyramid - Wallet Examples • Android Espresso Tests, Real Servers • Android Tests, Real Servers • JVM Tests, Mock Data • Android Component Tests, Mock Data Black Box Tests Integration Tests Component Tests Unit Tests
41. Auto-run Tests for Every PR
42. GitHub Pull Request Builder Plugin
43. Capital One Wallet for Android on Jenkins Today Git Commit Pull Request (PR) Opened or Updated Unit Tests on PR Branch ✓ PR Merged Assemble APKs and Test APKs Integration Tests Unit Tests Component Tests Blackbox Tests ✓ ✓ ✓ ✓ ✓ ✓
44. JENKINS AS A PLATFORM
45. Linked Jobs & Triggers
46. Newbie – One Job Compile/Build Integration Tests Unit Tests Component Tests Blackbox Tests
47. Better – Four Jobs Run Component Tests Run Unit Tests Compile/Build Compile/Build Compile/Build Run Blackbox Tests Run Integration Tests Compile/Build
48. Better – Four Jobs Run Component Tests Run Unit Tests Compile/Build Compile/Build Compile/Build Run Blackbox Tests Run Integration Tests Compile/Build Limitations: • Not great if you want to repeatedly re-run the tests • Wastes time/money if you have less than four slaves/executors
49. Better – Five Jobs Run Component Tests Run Unit Tests Compile/Build Run Blackbox Tests Run Integration Tests
50. Configuration - Triggering
51. Better – Five Jobs Run Component Tests Run Unit Tests Compile/Build Run Blackbox Tests Run Integration Tests
52. Configuration – Copy Build Artifacts From Other Job
53. Continuous Delivery
54. Building a Release Candidate Compile/Build ✓ Manual Testing and Upload Alpha
55. What if I told you every build that passes your automated tests is a release candidate? ”
56. Continuous Delivery for Clients Compile/Build Integration Tests Unit Tests Component Tests Blackbox Tests ✓ ✓ ✓ ✓ ✓ Alpha
57. Continuous Delivery for All
58. Server Unit Tests Continuous Delivery for Servers ✓ ✓ Server Integration Tests ✓
59. Your automated client tests are now an API. ”
60. iOS Stable Integration Tests Server Unit Tests Continuous Delivery for Servers ✓ Server Integration Tests Android Stable Integration Tests ✓ ✓ ✓ ✓
61. Continuous Delivery for Servers 1. Server developer runs unit tests locally
62. Continuous Delivery for Servers 1. Server developer runs unit tests locally 2. On pull request, unit tests and all server and client integration tests run on a transient server instance
63. Continuous Delivery for Servers 1. Server developer runs unit tests locally 2. On pull request, unit tests and all server and client integration tests run on a transient server instance 3. On merge, server code gets automatically deployed to shared QA region
64. Pipeline Views
65. Pipeline Jenkins Pipelines: https://jenkins.io/doc/pipeline/
67. Manage Scripts in Separate Repository Benefits: • Versioned with great tooling to view history
68. Manage Scripts in Separate Repository Benefits: • Versioned with great tooling to view history • Great when you have to update multiple or similar scripts like a library version number
69. Manage Scripts in Separate Repository Benefits: • Versioned with great tooling to view history • Great when you have to update multiple or similar scripts like a library version number • Great when re-building Jenkins
70. Manage Scripts in Separate Repository Benefits: • Versioned with great tooling to view history • Great when you have to update multiple or similar scripts like a library version number • Great when re-building Jenkins
71. PRO TIPS RUNNING TESTS IN PARALLEL
72. Recap Run Component Tests Run Unit Tests Compile/Build Run Blackbox Tests Run Integration Tests
73. • Split them across devices and/or executors …or both! 05 Parallel Tests
74. Sharding Your Tests Run Component Tests Run Unit Tests Compile/Build Run Blackbox Tests Run Integration Tests Blackbox Tests [0] Blackbox Tests [1] Blackbox Tests [2] Blackbox Tests [3]
75. Parallel Tests Attempt #1
78. • The main job uses up an executor while waiting for Phases to complete (which each use their own executor) • Phases need to do their own partitioning of the test suite • Test results are not aggregated back into a single report 05 Problems with MultiJob Plugin
79. 05 Android Instrumentation Sharding (docs)
80. Matrix Project Plugin
82. 05 Configuring The Matrix
83. 05 Tell The Job Which Test Shard To Run
84. • Easier to pass sharding index to build executors • Starts parallel jobs for all combinations of configurations 05 Benefits of Matrix Plugin
85. Supercharging Your Matrix
86. 05 Android Emulator Plugin • Built in support for Multi-configuration (matrix) jobs!
87. 05 Android Emulator Plugin
88. THE END
89. ENCORE
90. 05 Android Emulator PROS • Multiple screen sizes are easy • FREE! • Fast (10x faster install speed then normal devices) CONS • “Snapshot” feature is unreliable • Workaround: Re-create the AVD every time
91. 05 Genymotion Emulator PROS • Multiple screen sizes are easy • Fast • Snapshot feature works pretty well CONS • Requires Licenses • Can be difficult to manage with sharding
92. 05 Real Devices PROS • They are REAL Devices • There are things like Google Test Cloud, Test Droid and Amazon Device Farm CONS • You have to buy all the devices & hook them up • OR you have to pay a lot for cloud based use of devices which can be slow
93. 05 Jenkins Architecture
94. THAT’S ALL FOLKS