Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Journey to painless releases: Continuous delive...

Journey to painless releases: Continuous delivery for Philips Hue Android

Video: https://www.youtube.com/watch?v=VkFcJZWtufA

Wind back one year ago:
- releasing took us 10 weeks
- our app was plagued by regression
- developers felt completely detached from our users

Clearly our traditional strategy of heavily testing each release (up-front quality) wasn't working. So we decided to flip things around: why not try to reduce the cost of errors instead? Hence we decided to move our Philips Hue Android app to continuous deployment.

In this talk you will learn that if releasing is painful, you should actually release even more often. Step by step I'll explain how we moved the Philips Hue Android app to continuous deployment, what our current release process/CI looks like, what changes we made to our overall testing approach, how we empowered developers and testers, and how we made our release process completely painless.

Jeroen Mols

April 23, 2019
Tweet

More Decks by Jeroen Mols

Other Decks in Programming

Transcript

  1. @MOLSJEROEN FRUSTRATIONS Issues were found extremely late Developers felt detached

    from our users Constant context switches between releases Releasing very painful Friction between developers and testers
  2. @MOLSJEROEN GOALS Speed up development Get a grip on quality

    Feedback between developers and users
  3. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING

    BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTIN
  4. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING TESTING

    BETA DEVELOPMENT TESTING TESTING BETA DEVELOPMENT TESTING TESTING BETA
  5. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEVELOPMENT TESTING BETA

    TESTING DEVELOPMENT TESTING BETA TESTING DEVELOPMENT TESTING BETA TESTING
  6. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? BETA DEVELOPMENT TEST

    TEST BETA DEVELOPMENT TEST TEST BETA DEVELOPMENT TEST TEST
  7. @MOLSJEROEN WHAT DO WE WANT TO ACHIEVE? DEV & TEST

    BETA DEV & TEST BETA DEV & TEST BETA
  8. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE VERIFY develop master release feature 1-5 weeks
  9. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature 1-5 weeks
  10. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 1-5 weeks
  11. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE MASTER VERIFY develop master release feature STABILIZE 3 weeks 1-5 weeks
  12. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT PULL REQUEST QUALIFY

    MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks
  13. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks
  14. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY develop master release feature 3 weeks 1-5 weeks SHIP
  15. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks
  16. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks
  17. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  18. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  19. @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull

    request Merge
 conflicts Hard to
 verify
  20. @MOLSJEROEN Large pull request Hard to
 review Long lived
 pull

    request Merge
 conflicts Overhead: 
 regression, 
 dependencies,… Hard to
 verify
  21. @MOLSJEROEN MERGING/INTEGRATING FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE

    PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 1-5 weeks 3 weeks 3 weeks QUALIFY
  22. @MOLSJEROEN QUALIFICATION: SPEED UP BUILDS Update all dependencies to latest

    version Follow best practices Modularize Butterknife reflect Gradle build cache and build scans Monitor and keep on experimenting
  23. @MOLSJEROEN QUALIFICATION: SPEED UP CI Git hook instead of polling

    Git cache Preload gradle wrapper in docker container Unit tests over instrumentation tests Run subset of tests Parallel tasks
  24. @MOLSJEROEN QUALIFICATION: SPEED UP TESTS Legacy “unit tests” written as

    instrumentation Converted 1272 tests Build speed decreased by 3m 18s -9% code coverage Removed CI bottleneck Make room for new Espresso tests
  25. "IF YOU MERGE EVERY DAY, SUDDENLY YOU NEVER GET TO

    THE POINT WHERE YOU HAVE HUGE CONFLICTS THAT ARE HARD TO RESOLVE." Linus Torvalds
  26. @MOLSJEROEN RECAP: SPEEDING UP INTEGRATION Blazing fast builds A state

    of the art CI Do verification on develop Many small branches => Integration speed up from 1 - 5 weeks to 0,5 - 2days
  27. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE PROMOTE MASTER STABILIZE VERIFY SHIP develop master release feature MERGE BACK 0,5-2 days 3 weeks 3 weeks
  28. @MOLSJEROEN BUILD VARIANTS buildTypes { debug { signingConfig signingConfigs.debug ext.enableCrashlytics

    = false } release { minifyEnabled true proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro' signingConfig signingConfigs.release } uitest { // provide implementation for idling resources } } flavorDimensions "MODE" productFlavors { daily { applicationId 'com.philips.lighting.hue2.daily' dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDaily buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDaily } store { dimension "MODE" buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyApp buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeStore } } variantFilter { variant -> def names = variant.flavors*.name if (names.contains("store") && variant.buildType.name == 'debug') { variant.setIgnore(true) } if (names.contains("store") && variant.buildType.name == 'uitest') { variant.setIgnore(true) } }
  29. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  30. @MOLSJEROEN IDLING RESOURCES: UI TEST VARIANT object EspressoIdlingResource { private

    val idling = BaseIdlingResource() fun increment(code: String, reason: String) = idling.increment(code, reason) fun decrement(code: String, reason: String) = idling.decrement(code, reason) }
  31. @MOLSJEROEN IDLING RESOURCES: DEBUG AND RELEASE VARIANT object EspressoIdlingResource {

    fun increment(code: String, reason: String) { /* NOP */ } fun decrement(code: String, reason: String) { /* NOP */ } }
  32. @MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling

    = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }
  33. @MOLSJEROEN IDLING RESOURCES: UNIFIED object EspressoIdlingResource { private val idling

    = BaseIdlingResource() fun increment(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { idling.increment(code, reason) } fun decrement(code: String, reason: String) = if (RuntimeBehavior.isFeatureEnabled(IDLING_RESOURCES)) { mCountingIdlingResource.decrement(code reason) } }
  34. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  35. @MOLSJEROEN BUILD VARIANTS Prod API keys Proguard Test settings Idling

    resource s Leak canary Prod firmware Develop
 Portal Release Release test Debug Espresso
  36. @MOLSJEROEN GRADLE CONFIGURATION buildTypes { debug { signingConfig signingConfigs.debug applicationIdSuffix

    '.daily' ext.enableCrashlytics = false buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppDebug buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeDebug } release { signingConfig signingConfigs.release minifyEnabled true proguardFiles 'proguard-rules.pro' buildConfigField "String", "HOCKEY_APP_ID", Keys.hockeyAppRelease buildConfigField "String", "AMPLITUDE_APP_ID", Keys.amplitudeRelease }
  37. @MOLSJEROEN REDUCE COMPLEXITY: SUMMARY Drop unnecessary branches No branches from

    branches Minimize build variants Feature flags for dynamic configuration => Remove all accidental complexity
  38. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY SHIP master release feature MERGE BACK 0,5-2 days 6 weeks*
  39. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH DEVELOP MENT BETA PULL REQUEST

    QUALIFY MERGE PROMOTE RELEASE STABILIZE VERIFY RELEASE master release feature MERGE BACK 0,5-2 days ~4 weeks
  40. @MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) {

    ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }
  41. @MOLSJEROEN FEATURE FLAGS: NEW FEATURES override fun onViewCreated(view: View) {

    ... if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.POWER_ON_BEHAVIOR)) { power_on_behavior.visibility = View.VISIBLE } else { power_on_behavior.visibility = View.GONE } ... }
  42. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }
  43. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() }
  44. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { var code

    = NewLogic() code.with(UnitTests()) // ... code.doMagic() }
  45. @MOLSJEROEN FEATURE FLAGS: EXISTING CODE fun withLegacy() { if (RuntimeBehavior.isFeatureEnabled(FeatureFlag.LOGIC_REFACTOR))

    { var code = NewLogic() code.with(UnitTests()) // ... code.doMagic() } else { var code = LegacyLogic() code.with(OutUnitTests()) // ... code.doMagic() } }
  46. @MOLSJEROEN AUTOMATE TESTING Library Feature Feature Feature Feature Feature Feature

    Feature Feature Feature App Library Library Lib Lib Library Library Lib
  47. @MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app

    Espresso tests Run all automated tests of the modules that are impacted Manual verification if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing
  48. @MOLSJEROEN TEST STRATEGY Pull requests: Run all automated level1 app

    Espresso tests Run all automated tests of the modules that are impacted Manual verification if high impact Release: Run all automated Espresso tests Run all automated feature and library tests Deploy: Manual exploratory testing
  49. @MOLSJEROEN IMPACT ANALYSIS SIDE EFFECTS POSSIBLE IN 
 OTHER PARTS

    OF CODE? (REGRESSION) ALL SIDE EFFECTS CONTAINED BEHIND A FEATURE FLAG? LOW IMPACT LOW IMPACT HIGH IMPACT Yes Yes No Not sure Not sure No
  50. @MOLSJEROEN STABILIZE MASTER: SUMMARY Use feature toggles Static code analysis

    Heavily automate testing Merge master before checks Requalify pull requests after each merge Impact assessment => Able to drop release branch
  51. @MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE

    FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP
  52. @MOLSJEROEN DEVELOPMENT PROCESS 0,5-2 days 2 weeks FEATURE BRANCH FEATURE

    FLAG DEVELOP PULL REQUEST MERGE QUALIFY
 ++ VERIFY IMPACT master feature BETA MANUAL RELEASE SHIP
  53. @MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to Google Play Internal Test

    Track') { steps { script { def releaseNotes = readFile 'app/build/release_notes.txt' androidApkUpload( googleCredentialsId: 'Google Play Android Developer', trackName: 'internal', apkFilesPattern: '**/*-release.apk', deobfuscationFilesPattern: '**/mapping.txt', recentChangeList: [[language: 'en-US', text: releaseNotes]] ) } } }
  54. @MOLSJEROEN RELEASE APP (DELIVERY) stage('Deploy to HockeyApp') { steps {

    sh "pipelines/scripts/hockeyapp/deploy_app_debug.sh" } }
  55. @MOLSJEROEN AUTOMATE DELIVERY: SUMMARY Integrate dependencies automatically Automate upload to

    Google play/hockey app Release directly from master Hotfixes from separate branch
  56. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST

    MERGE QUALIFY
 ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*
  57. @MOLSJEROEN 1. BI-WEEKLY RELEASES DEV & TEST BETA DEV &

    TEST BETA DEV & TEST BETA 2 weeks DEV & TEST BETA
  58. @MOLSJEROEN SUMMARY Focus on whitebox testing Gain trust and give

    trust back Don’t blow your trust! Rely on automated testing Closer collaboration between testers and developers Root cause analyses
  59. @MOLSJEROEN DEVELOPMENT PROCESS FEATURE BRANCH FEATURE FLAG DEVELOP PULL REQUEST

    MERGE QUALIFY
 ++ master RELEASE VERIFY feature IMPACT 0,5-2 days 2 hours BETA SHIP 1 week*