Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Easing into continuous deployment
Search
Chris Keathley
July 28, 2017
Programming
1
280
Easing into continuous deployment
How we moved our team from static deployments into continuous deployment.
Chris Keathley
July 28, 2017
Tweet
Share
More Decks by Chris Keathley
See All by Chris Keathley
Solid code isn't flexible
keathley
4
940
Building Adaptive Systems
keathley
34
2k
Contracts for building reliable systems
keathley
5
700
Kafka, the hard parts
keathley
2
1.4k
Building Resilient Elixir Systems
keathley
6
1.9k
Consistent, Distributed Elixir
keathley
5
1.4k
Telling stories with data visualization
keathley
0
510
Leveling up your git skills
keathley
0
640
Generative Testing in Elixir
keathley
0
420
Other Decks in Programming
See All in Programming
AWS初心者ってどうやってAWSを学ぶ?〜アプリエンジニアがやってよかったアーキテクチャ学習方法〜
yamanashi_ren01
0
190
Architectures with Lightweight Stores: New Rules and Options
manfredsteyer
PRO
0
100
AWSでゲームサーバーを運用! Amazon GameLiftのお話
iriikeita
0
200
Cloudflare Workers x AWS Lambdaの組み合わせユースケース / Cloudflare Workers x AWS Lambda Combination Use Case
seike460
PRO
2
310
小さな開発会社を作った理由
polidog
0
1.9k
英語
s_shimotori
1
220
今こそ始める、CDKコンストラクトライブラリ開発 ― 入門から実践まで
tmokmss
1
930
はしめてのプログラミングとロボット制御
watawatavoltage
0
290
最近追加した型の紹介とその振り返り
aki19035vc
0
170
継続的な活動で築く地方エンジニアの道
myamashii
2
350
ぼっちを避けて楽しむためのアノテコノテ / Various Tips and Tricks to Avoid Loneliness and Have Fun
nrslib
3
1.7k
わかりやすい正解を捨てて、コトに向き合う - スクラムフェス金沢2024 スポンサーセッション
yusukekokubo
0
170
Featured
See All Featured
Clear Off the Table
cherdarchuk
89
320k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
228
16k
A Philosophy of Restraint
colly
200
16k
Build your cross-platform service in a week with App Engine
jlugia
227
17k
The Mythical Team-Month
searls
217
43k
Producing Creativity
orderedlist
PRO
340
39k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
360
22k
Web development in the modern age
philhawksworth
203
10k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
36
9.1k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
129
32k
4 Signs Your Business is Dying
shpigford
178
21k
Creatively Recalculating Your Daily Design Routine
revolveconf
214
11k
Transcript
Continuous Deployment Chris Keathley / @ChrisKeathley /
[email protected]
I work with a distributed team
I work with a distributed team
None
None
None
None
None
Warehouse
Warehouse API
Warehouse API Apps
The problem
Slow Iteration Cycle Deployment Deployment 2 weeks
Slow Iteration Cycle Deployment Deployment Deployment 2 weeks 2 weeks
Slow Iteration Cycle Deployment Deployment Deployment 3 weeks
Slow Iteration Cycle Deployment Deployment Deployment Hopefully someday
Large PRs
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Unsure about state of the application
Rollbacks are a scam
None
None
Data Migration
Data Migration ?
Don’t do this
always Move forward
always Move forward
always Move forward
The goal should never be to roll back a deployment
The goal is to minimize the damage done by any
given deployment
There are bugs in your system
Solutions
We needed to deploy more often
So we did
Automated Deployment
What do you deploy?
Commit Sha
None
Jars
Artifacts
Git Tags
Containers
Your App
Your App Server
None
Master Branch How we merge our code PR
CI Github Registry Container Slack PR Notification
CI Kubernetes Deploy Auto-deploy Green builds of master
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
Service B
CI Kubernetes Deploy Auto-deploy Green builds of master Service B
CI Kubernetes Deploy Auto-deploy Green builds of master Service B
Service A
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
CI Kubernetes Deploy Auto-deploy Green builds of master Service A
Service B
Tests Metrics &
Integration Tests + Property Tests
Integration Tests TEst App DB Service
Modeling Users as FSMs logged_out logged_in login logout vote
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Property Tests Add Todo Edit Todo Delete Todo
Generate Commands
Generated Commands [{:add_todo, “Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo,
"", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Generate Commands
Generate Commands
Generate Commands
Generate Commands
Generate Commands
Generated Commands [{:add_todo, “Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo,
"", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Generated Commands [{:add_todo, “Test Todo”, 1}, {:delete_todo, "", 2}] [{:add_todo,
“Test Todo”, 1}, {:edit_todo, "Edited", 2}, {:delete_todo, "", 1}, {:add_todo, “New Todo", 3}, {:delete_todo, "", 2} {:edit_todo, “Edited Todo”, 2}]
Prometheus Service A Grafana Service B Service C
Prometheus Service A Grafana Service B Service C Slack
# Alert for any instance that have a 95th percentile
> 200ms. ALERT APIHighRequestLatency IF api_http_request_latencies_second{quantile="0.95"} > 0.2 FOR 5m ANNOTATIONS { summary = "High request latency on {{ $labels.instance }}", description = "{{ $labels.instance }} has a median request latency above 1s (current value: {{ $value }}s)", }
Track “Business” Metrics
None
Feature releases and flags
None
Features aren’t all or nothing
Features != Deployments
Deployment
Deployment Features
Deployment Features
User
User staff?(user) == true
User staff?(user) == false
User staff?(user) == false
defmodule MyApp.FeatureFlags do alias MyApp.User def foo_enabled?(%User{staff: is_staff}), do: is_staff
def foo_enabled?(_), do: false def bar_enabled?(%User{staff: is_staff}), do: is_staff def bar_enabled?(_), do: false end
Browser Feature Service
Feature Service Feature Service Feature Service
Feature Service Feature Service Feature Service
Feature Service Feature Service Feature Service
You have updates ready! Reset
None
With larger Traffic numbers you could use percentages
Alchemy
“Transmute lead code into gold in production”
Prior Art: https://github.com/github/scientist
Users_Controller DB User.all
DB User.all UserService.all
User.all UserService.all ==
def index(conn) do users = old_query() render(conn, "index.json", users: users)
end
def index(conn) do users = experiment("users-query") |> control(&old_query/0) |> candidate(&new_query/0)
|> run render(conn, "index.json", users: users) end
def index(conn) do users = experiment("users-query") |> control(&old_query/0) |> candidate(&new_query/0)
|> candidate(&fancy_query/0) |> run render(conn, "index.json", users: users) end
1) Shuffles test order 2) Runs Each test in parallel
3) exports the data Alchemy
DB User.all UserService.all Control Candidate Control UserController
None
1) Do the results match? 2) How long does each
test take to return? Measure
No more cutovers
DB User.all UserService.all
DB User.all UserService.all User service
Migrations
http://blog.datomic.com/2017/01/the-ten-rules-of-schema-growth.html
DB Schema App Application Coupling
Your application knows about your schema
Lets remove a column
Lets remove a column 1) all application code needs to
stop using that column
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting 4) Remove the column
Lets remove a column 1) all application code needs to
stop using that column 2) Update all ETL processes 3) Update Reporting 4) Remove the column Split all of these up
Lets Add a column
Lets Add a column 1) Add the column
Lets Add a column 1) Add the column 2) Eventually
start using it
Prefer Additive Migrations
CI Kubernetes Deploy Auto-deploy Green builds of master
CI Kubernetes Deploy Auto-deploy Green builds of master Migration
CI Kubernetes Deploy Auto-deploy Green builds of master Migration DB
Chat-Ops
None
Chat is…
Chat is… Centralized
Chat is… Centralized Transparent
Chat is… Centralized Transparent Open
Try to do operational tasks in chat
None
defmodule Hedwig.Responders.Ping do use Hedwig.Responder @usage """ hedwig: ping -
Responds with 'pong' """ respond ~r/ping$/i, msg do reply msg, "pong" end end
None
Generate grafana graphs
None
Deploy
None
Team Building
Conclusion
These are tools at our disposal
Deploy more often, safely
Thanks Chris Keathley / @ChrisKeathley /
[email protected]