Real-world Continuous Delivery: Learn, Adapt, Improve (ConFoo 2020)

REAL WORLD CD LEARN, ADAPT & IMPROVE MICHIEL ROOK @MICHIELTCS

LARGE SEMI-GOV IN NL

@michieltcs MYSQL ELASTIC INTERNET

@michieltcs TOPIC TEAMS

@michieltcs FIRE TEAM

@michieltcs PACKAGE/DEP UPDATES

@michieltcs BUG TRIAGE

@michieltcs RELEASE ROLLOUT

@michieltcs A TYPICAL RELEASE...

@michieltcs RELEASE CHECKLIST

@michieltcs @michieltcs Create release Test release Deploy release every 2
weeks Thursday (sprint end) Monday (sprint start) Tuesday

@michieltcs 1. BRANCH RC 2. DEPLOY TO ACC 3. SIGNAL
TESTER(S) 4. SIGNAL PO 5. FIX ON BRANCH 6. GET APPROVAL 7. DEPLOY TO PROD 8. VERIFY PROD

@michieltcs 2-3 DAYS MANUAL WORK

@michieltcs ... HOT FIXES?

@michieltcs GOALS

@michieltcs REDUCE COST

@michieltcs FAST FEEDBACK

@michieltcs REDUCE TOIL

@michieltcs REDUCE TOIL the kind of work tied to running
a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value, and that scales linearly as a service grows

@michieltcs PUT PO/BIZ BACK IN DRIVER'S SEAT

@michieltcs WHY CD THEN?

@michieltcs MAKE THINGS SMALL

@michieltcs BIG STEPS

@michieltcs FAIL BIG

@michieltcs SMALL STEPS

@michieltcs FAIL SMALL

@michieltcs IF IT HURTS DO IT MORE OFTEN

@michieltcs $ = REALIZED VALUE CREDITS TO @FGOULDING

@michieltcs @michieltcs HIGH VS. LOW PERFORMING TEAMS Source: Accelerate -
The Science Of DevOps

APPROACH

@michieltcs 3 PHASES

@michieltcs DEFER SPLITTING THE MONOLITH

@michieltcs INCREASE TEMPO & AUTOMATION

@michieltcs SINGLE PIPELINE

@michieltcs IMPROVEMENTS & TESTING

PHASE 1: INCREASING RELEASE CADENCE

@michieltcs WEEKLY RELEASE

@michieltcs AUTOMATION

@michieltcs @michieltcs Build / Package Automated Tests Acceptance Environment Production
Environment

@michieltcs pipeline {  stages {  stage('Run tests') {  sh "gradle
check"  }    stage('Build docker image') {  sh "docker build -t jobservice:${env.BUILD_NUMBER} ."  sh "docker push jobservice:${env.BUILD_NUMBER}"  }    stage('Deploy staging') {  sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}  -i staging deploy.yml"  }    stage('Deploy production') {  sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}  -i prod deploy.yml"  }  }  PIPELINE AS CODE image: registry.local/runner:latest    stages:  - test  - images  - deploy    phpunit:  stage: test  script:  - bin/phpunit    images:  stage: images  script:  - make images publish  only:  - master    deploy:  stage: deploy  script:  - make deploy  only:  - master

@michieltcs WAY OF WORKING

@michieltcs PAIR PROGRAMMING

BOY SCOUT RULE

@michieltcs NO MORE RELEASE BRANCHES

@michieltcs @michieltcs

PHASE 2: SINGLE PIPELINE

@michieltcs @michieltcs Build / Package Automated Tests Acceptance Environment Production
Environment

@michieltcs CONTINUOUS DEPLOYMENT

@michieltcs ZERO DOWNTIME DEPLOYS

@michieltcs ROLLING DEPLOYS

@michieltcs @michieltcs LOAD BALANCER ROLLING UPDATE SERVICE V1.0 SERVICE V1.0
SERVICE V1.0 @michieltcs

SERVICE V1.0 SERVICE V1.1 @michieltcs

SERVICE V1.1 @michieltcs

@michieltcs PIPELINE FAILURES

@michieltcs FLAKY TESTS

@michieltcs TIMEOUTS

@michieltcs NETWORK STABILITY ISSUES

@michieltcs EXTERNAL DEPS IN TESTS

@michieltcs BRITTLE INFRASTRUCTURE

@michieltcs IF YOU CAN'T TRUST IT - REMOVE IT

STOP THE LINE

@michieltcs EXTREME FEEDBACK

@michieltcs RELEASE TIMING?

FEATURE TOGGLES

@michieltcs DECOUPLE DEPLOYMENTS FROM RELEASES

@michieltcs CAUTION: FEATURE TOGGLE DEBT

PHASE 3: IMPROVEMENTS & TESTING

@michieltcs PIPELINE SPEED

@michieltcs BUY A BIGGER BOX

@michieltcs CLEVER PIPELINE IMPROVEMENTS: PARALLELIZE

@michieltcs TESTING IMPROVEMENTS

@michieltcs @michieltcs UNIT TESTS E2E / VISUAL TESTS INTEGRATION TESTS
LOTS OF MANUAL TESTING E2E TESTS

@michieltcs QUALITY ENGINEERING

@michieltcs AUTOMATION

@michieltcs @michieltcs CONTRACT TESTING

@michieltcs @michieltcs UI TESTING

@michieltcs @michieltcs VISUAL TESTING

@michieltcs @michieltcs DEPENDENCY SCANNING

@michieltcs @michieltcs UNIT TESTS UI / VISUAL TESTS Exploratory testing
& user feedback Monitoring & alerting INTEGRATION / CONTRACT TESTS COST SPEED

RESULTS & TAKEAWAYS

@michieltcs LEARNINGS

@michieltcs PIPELINE SPEED & STABILITY

@michieltcs TEST THE PIPELINE

@michieltcs WAY OF WORKING & MINDSET

@michieltcs VISUAL TESTING TOOLS

@michieltcs SOME STATS

@michieltcs ~4000 TESTS PER DEPLOYMENT

@michieltcs ~10 MINUTES PER PIPELINE RUN

@michieltcs ~1.5 HOURS FROM DEV TO PROD

@michieltcs IN 6 MONTHS: MORE THAN 5000 DEPLOYMENTS

@michieltcs @michieltcs THANK YOU FOR LISTENING! @michieltcs / [email protected] www.michielrook.nl

Real-world Continuous Delivery: Learn, Adapt, I...

Real-world Continuous Delivery: Learn, Adapt, Improve (ConFoo 2020)

More Decks by Michiel Rook

Other Decks in Programming

Featured

Transcript