companies save money World’s largest inventory Flights, hotels, trains, accommodation All in one place, no more chasing receipts Search, book, pay, invoice, share and report — all on the same platform. Save money Cheaper options, low commissions, more data to optimise, Expensify integration
squad team ❌ Hold a release until desired feature is completed (no bug fixing) ❌ Multi-squad problems: ◦ Separate features block each other ◦ Delays and uncertainty ◦ Who owns QA for the single binary? is the feature done? Work on a feature Ship new app version Yes No
Merged changes are automatically tested There’s a CI server running those tests Continuous Integration Changes merged into the main branch are automatically deployed to a production-like environment Release to production is manual Continuous Delivery Changes merged into the main branch are automatically deployed to production No human intervention Continuous Deployment Source: https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment
merge changes Merged changes are automatically tested There’s a CI server running those tests Continuous Integration Changes merged into the main branch are automatically deployed to a production-like environment Release to production is manual Continuous Delivery Changes merged into the main branch are automatically deployed to production No human intervention Continuous Deployment Source: https://www.atlassian.com/continuous-delivery/principles/continuous-integration-vs-delivery-vs-deployment
are not instant: review process can take days and even reject the update Updates are not mandatory: users may never update to a new release Release as a single binary Apps are distributed as one single binary: we can’t update only parts of it To remove feature’s code you need to ship a new version of the whole project Rollback App stores do not allow rollbacks Any new bug means • New release for the whole app needs to be done (single binary) • It won’t be fixed today (we don’t control distribution environments)
train Mobile release train: Checklist [ ] Assign release captains for Android and iOS [ ] Check which feature flags will need enabling [ ] Force upgrade after the release? [ ] Test cases for new features are added to Test Checklist [ ] Check all translations are in Phrase [ ] Release notes created and translated [ ] Changelog updated [ ] Release train announced in Slack [ ] Clone Amplitude release dashboard
Train “departs” at 15.00 o’clock strictly Release branch cut from master New changes to master not allowed into the train from this point QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: Departure
Code freeze release_version = prompt(text: "Please provide a version number for the new release:") create_branch(from: "master", release_version: release_version) # Generate builds create_staging_build(branch_name: git_branch) create_production_build(branch_name: git_branch) UI.success "🍻 STAGING build and PRODUCTION build generated for current branch" end Mobile release train: Departure
Release Captain drives this phase Manual* testing of production build. Core & feature tests shared in spreadsheet Feature flags checked here Only fixes accepted at this stage QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint * Working towards automation in Q1 2021 Mobile release train: QA
QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Allowed into the release branch if critical bugs are found during QA Non-critical bug fixes will board the next release train Needs agreement between Release Captain and PM Bug fix Mobile release train: bug fixing
Semi-automated process Release notes handled by PM Validation process in stores can take up to 2-3 days QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: store submission
1% > 2% > 5% > 20% > … > 100% to closely monitor adoption issues Staged rollout can be halted but never rolled back Hotfixes allowed for critical bugs -> 100% QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Hotfix Mobile release train: staged rollout
Rollout reaches 100% Release tagged on Github Release branch merged back to master QA phase Day 10 - 17.00 Store submission Staged rollout phase Day 15 100% release New sprint Mobile release train: Release
branch to master" lane :finish_release do # Get the release branch release_branch = sh("git branch -r | grep 'release' | tail -1 | tr -d ' ' | cut -d'/' -f2-").chomp # Check out release_branch and pull changes sh("git checkout #{release_branch}") # Get current version current_version = get_version_name # Tag release and push sh("git tag -a v#{current_version} -m '' && git push origin v#{current_version}") # Create Pull request open_pr_release(version: current_version) UI.success "🍻 Success!" end end Mobile release train: Finish a release
stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Release dashboard is created automatically ➔ Publish release rollout status reports automatically on Slack ➔ Halt rollout if error rate goes over a threshold ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
stores Monitor release rollout & track errors Release train checklist & QA Communicate train status Tooling: working towards automation ➔ Triggered automatically from Bitrise at specific times ➔ Start and finish release trains at fixed times ➔ Automatically create new release builds when hotfixes are merged into release branches and submit them ➔ Automatically run SAST/DAST tool nightly ➔ Release dashboard is created automatically ➔ Publish release rollout status reports automatically on Slack ➔ Halt rollout if error rate goes over a threshold ➔ All train status changes are announced automatically ➔ Capacity to query: what’s the rollout status? Error rate? ➔ Capacity to action: halt release, generate stage build,... ➔ Automate the different checks starting with scripts. Ex, active feature flags, release changelog, etc. ➔ Currently evaluating platforms for E2E testing automation. Goal: nightly E2E tests running on cloud devices
lgvalle
nkforwork
tkikuc
murabayashi
maaretp
ikumatadokoro
tsucchinoko
qnighy
kno3a87
mot_techtalk
har1101
biacco42
oheyadam
sonatard
jrom
thoeni
thekraken
soudai
aki_iinuma
chriscoyier
stephaniewalter
jacobian
sugarenia
cromwellryan
lauravandoore
