The Road to Continuous Deployment: a case study (IPC Berlin 2017)

It’s a situation many of us are familiar with: a large legacy, monolithic application, limited or no tests, slow and manual release process, low velocity, no confidence… A lot of refactoring is required, but management keeps pushing for new features. How to proceed? Using examples and lessons learned from a real-world case, I’ll show you how to replace a legacy application with a modern service-oriented architecture and build a continuous integration and deployment pipeline to deliver value from the first sprint. On the way, we’ll take a look at the process, automated testing, monitoring, master/trunk based development and various tips and best practices.

Transcript

1. THE ROAD TO CONTINUOUS DEPLOYMENT - A CASE STUDY MICHIEL ROOK @michieltcs
2. ▸ Developer, consultant, trainer, speaker ▸ @michieltcs
4. @michieltcs
5. @michieltcs Large aging monolith
6. @michieltcs Large aging monolith Generates income
7. @michieltcs Large aging monolith Slow & complex Generates income
8. @michieltcs Large aging monolith Slow & complex Generates income Technical debt
9. FRONTEND MYSQL DB BACKEND LOAD BALANCERS / VARNISH JOB SITE JOB SITE JOB SITE FRONTEND FRONTEND FRONTEND BACKEND BACKEND BACKEND MEMCACHE FTP EXT. SERVICES SOLR @michieltcs
10. @michieltcs
11. @michieltcs Manual releases
12. @michieltcs Manual releases Fragile tests
13. @michieltcs Manual releases Frequent issues Fragile tests
14. @michieltcs Manual releases Frequent issues Fragile tests Low velocity
15. GOALS @michieltcs
16. GOALS REDUCE
 ISSUES 1 @michieltcs
17. GOALS REDUCE
 ISSUES REDUCE
 LEAD TIME 1 2 @michieltcs
18. GOALS REDUCE
 ISSUES REDUCE
 LEAD TIME INCREASE PRODUCTIVITY 1 2 3 @michieltcs
19. GOALS REDUCE
 ISSUES REDUCE
 LEAD TIME INCREASE PRODUCTIVITY INCREASE MOTIVATION 1 2 3 4 @michieltcs
20. REFACTOR? REBUILD?
21. APPROACH ▸ API first ▸ Services per domain object (job, jobseeker, ...) ▸ Migrate individual pages @michieltcs
22. ORIGINAL MONOLITH DB @michieltcs
23. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH DB DB DB @michieltcs
24. ORIGINAL MONOLITH PROXY SERVICE ORIGINAL MONOLITH ORIGINAL MONOLITH SERVICE SERVICE SERVICE PROXY DB DB DB DB DB DB @michieltcs
25. ARCHITECTURE FRONTENDS ARE SERVICES @michieltcs
26. ARCHITECTURE FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS @michieltcs
27. ARCHITECTURE FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS ACCESS LEGACY DB'S @michieltcs
28. ARCHITECTURE FRONTENDS ARE SERVICES SERVICES BEHIND LOAD BALANCERS ACCESS LEGACY DB'S SERVICES IN CONTAINERS @michieltcs
29. PROCESS
30. CONTINUOUS EVERYTHING
31. CD?
32. DEV BUILD / TEST CONTINUOUS INTEGRATION @michieltcs
33. DEV BUILD / TEST ACCEPTANCE PRODUCTION CONTINUOUS DELIVERY @michieltcs
34. DEV BUILD / TEST STAGING / ACCEPTANCE PRODUCTION CONTINUOUS DEPLOYMENT @michieltcs
35. EVERY COMMIT GOES TO PRODUCTION
36. IF IT HURTS DO IT MORE OFTEN
37. SMALL STEPS @michieltcs
38. SMALL STEPS EARLY FEEDBACK @michieltcs
39. SMALL STEPS EARLY FEEDBACK REDUCE TIME TO RECOVER @michieltcs
40. SMALL STEPS EXPERIMENTS! EARLY FEEDBACK REDUCE TIME TO RECOVER @michieltcs
41. ONLY COMMIT TO MASTER
42. NO BRANCHES
43. NO BRANCHES REALLY.
46. BRANCHES ARE CHEAP?
47. DELAYING INTEGRATION
48. CONFLICTS
49. FEATURE TOGGLES
50. DECOUPLE DEPLOYMENTS FROM RELEASES
51. @michieltcs
52. FEATURE TOGGLE DEBT
53. CODE REVIEWS?
54. PAIR PROGRAMMING
55. CONTINUOUS INLINE CODE REVIEW
56. BOY SCOUT RULE
57. GATES
58. 100% CODE COVFEFE *
59. 100% CODE COVERAGE*
60. DEVOPS
61. DEVSECOPS
62. DEVSECQAOPS
63. DEVSECQABIZOPS
64. DEVSECQABIZNETOPS
65. DEVSECQABIZSYSNETOPS
66. DEVTESTSECQABIZSYSNETOPS
67. DEV*OPS
68. TEAM BUILDS IT
69. TEAM RUNS IT
70. TEAM IMPROVES IT
71. TEAM IS RESPONSIBLE
72. BUILD PIPELINE
73. AUTOMATE REPEATABLE THINGS
74. CONTINUOUS TESTING
75. CONFIDENCE
76. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI TESTS @michieltcs
77. UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE UI TESTS public function testJobCannotBeFound() {
 $jobRepository = $this->prophesize(JobRepository::class);
 $jobRepository->getById(EXPECTED_JOB_ID)
 ->shouldBeCalled()
 ->willReturn(false);
 
 $jobService = new JobService($jobRepository->reveal());
 
 $this->assertFalse($jobService->getById(EXPECTED_JOB_ID));
 } DEFENSE IN DEPTH @michieltcs
78. UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI TESTS public function testFindJob() {
 $expectedJob = $this->loadFixture('active_job.yml');
 $actualJob = $this->repository->getById($expectedJob->getId());
 
 self::assertInstanceOf(Job::class, $actualJob);
 self::assertEquals($expectedJob->getId(), $actualJob->getId());
 } DEFENSE IN DEPTH @michieltcs
79. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI TESTS Scenario: Link to related job
 Given a job exists
 And there are related jobs available
 When that job is viewed
 Then a list of related jobs is shown
 And each related job links to the detail page of the related job @michieltcs
80. DEFENSE IN DEPTH UNIT TESTS INTEGRATION
 TESTS ACCEPTANCE
 TESTS UI TESTS @michieltcs
81. UNIT TESTS INTEGRATION TESTS ACCEPTANCE TESTS UI TESTS SMOKE
 TESTS Cost Speed Exploratory
 testing Monitoring @michieltcs
82. GOOD ENOUGH
83. PIPELINE SPEED = KEY
84. PIPELINE AS CODE node {
 stage('Run tests') {
 sh "phpunit"
 sh "behat"
 }
 
 stage('Build docker image') {
 sh "docker build -t jobservice:${env.BUILD_NUMBER} ."
 sh "docker push jobservice:${env.BUILD_NUMBER}"
 }
 
 stage('Deploy staging') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i staging deploy.yml"
 }
 
 stage('Deploy production') {
 sh "ansible-playbook -e BUILD=${env.BUILD_NUMBER}
 -i prod deploy.yml"
 }
 } @michieltcs
85. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER @michieltcs
86. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker pull @michieltcs
87. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker run @michieltcs
88. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER wait_for: port=8080 delay=5 timeout=15 @michieltcs
89. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER uri:
 url: http://localhost:8080/health
 status_code: 200
 timeout: 30 @michieltcs
90. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
91. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER template: src=haproxy.cfg.j2
 dest=/etc/haproxy/haproxy.cfg service: name=haproxy state=reloaded @michieltcs
92. PULL IMAGE START NEW CONTAINER WAIT FOR PORT SMOKE TESTS / HEALTH CHECKS ADD NEW CONTAINER TO LB REMOVE OLD CONTAINER FROM LB STOP OLD CONTAINER docker stop docker rm @michieltcs
93. @michieltcs
94. FEEDBACK!
95. RESULTS
96. Team acceptance @michieltcs
97. Team acceptance New technology @michieltcs
98. Team acceptance New technology Docker stability @michieltcs
99. Team acceptance New technology Docker stability Pipeline stability @michieltcs
100. Feature toggle cap @michieltcs
101. Feature toggle cap Business alignment @michieltcs
102. Feature toggle cap Business alignment Focus on replacing legacy @michieltcs
103. Build per service
 < 10 min. 1 @michieltcs
104. Build per service
 < 10 min. 1 50+ deploys per day 2 @michieltcs
105. Build per service
 < 10 min. 1 50+ deploys per day Reduced number of issues 2 3 @michieltcs
106. Build per service
 < 10 min. 1 50+ deploys per day Reduced number of issues Improved page load times 2 3 4 @michieltcs
107. Improved metrics & audience statistics 5 @michieltcs
108. Improved metrics & audience statistics 5 Learning new technology 6 @michieltcs
109. Improved metrics & audience statistics 5 Learning new technology Increased confidence, velocity & fun 6 7 @michieltcs
110. LITERATURE https://trunkbaseddevelopment.com/
111. THANK YOU! @michieltcs / mrook@fourscouts.nl www.michielrook.nl