Deploying distributed software services to the cloud without breaking a sweat

From CodePaLOUsa 2011.

Transcript

4. # finger $(whoami) Login: susan Name: Susan Potter Directory: /home/susan Shell: /bin/zsh On since Mon 29 Sep 1997 21:18 (GMT) on tty1 from :0 No mail on me@susanpotter.net Plan: twitter: @SusanPotter github: mbbx6spp
5. # finger $(whoami) Login: susan Name: Susan Potter Directory: /home/susan Shell: /bin/zsh On since Mon 29 Sep 1997 21:18 (GMT) on tty1 from :0 No mail on me@susanpotter.net Plan: twitter: @SusanPotter github: mbbx6spp
6. # finger $(whoami) Login: susan Name: Susan Potter Directory: /home/susan Shell: /bin/zsh On since Mon 29 Sep 1997 21:18 (GMT) on tty1 from :0 No mail on me@susanpotter.net Plan: twitter: @SusanPotter github: mbbx6spp
7. # finger $(whoami) Login: susan Name: Susan Potter Directory: /home/susan Shell: /bin/zsh On since Mon 29 Sep 1997 21:18 (GMT) on tty1 from :0 No mail on me@susanpotter.net Plan: twitter: @SusanPotter github: mbbx6spp
8. # finger $(whoami) Login: susan Name: Susan Potter Directory: /home/susan Shell: /bin/zsh On since Mon 29 Sep 1997 21:18 (GMT) on tty1 from :0 No mail on me@susanpotter.net Plan: twitter: @SusanPotter github: mbbx6spp
9. Scope of Talk • Approaches
10. Scope of Talk • Approaches • Best Practices
11. Scope of Talk • Approaches • Best Practices • Pitfalls
12. Scope of Talk • Approaches • Best Practices • Pitfalls • Possibilities
13. Scope of Talk • Approaches • Best Practices • Pitfalls • Possibilities • (not) Chef vs Puppet
14. Scope of Talk • Approaches • Best Practices • Pitfalls • Possibilities • (not) Chef vs Puppet • (not) Why Cloud?
15. Scope of Talk • Approaches • Best Practices • Pitfalls • Possibilities • (not) Chef vs Puppet • (not) Why Cloud? • (not) Why DevOps?
16. Scope of Talk • Approaches • Best Practices • Pitfalls • Possibilities • (not) Chef vs Puppet • (not) Why Cloud? • (not) Why DevOps? • (not) Which Delivery Model?
17. Scope of Talk • Approaches • Best Practices • Pitfalls • Possibilities • (not) Chef vs Puppet • (not) Why Cloud? • (not) Why DevOps? • (not) Which Delivery Model? • (not) Release Management
18. Scope of Talk • Approaches • Best Practices • Pitfalls • Possibilities • (not) Chef vs Puppet • (not) Why Cloud? • (not) Why DevOps? • (not) Which Delivery Model? • (not) Release Management • (not) Monitoring
19. Cloud buzzzzz
20. Cloud: Delivery Models [1/2] Software (as a Service) Platform (as a Service) Infrastructure (as a Service)
21. Cloud: Delivery Models [1/2] Software (as a Service) Platform (as a Service) Infrastructure (as a Service)
22. Cloud: Delivery Models [1/2] Software (as a Service) Platform (as a Service) Infrastructure (as a Service)
23. Cloud: Delivery Models [1/2] Software (as a Service) Platform (as a Service) Infrastructure (as a Service)
24. Cloud: Delivery Models [2/2]
25. Cloud: Characteristics • Instant on-demand
26. Cloud: Characteristics • Instant on-demand • Virtualized performance, reliability
27. Cloud: Characteristics • Instant on-demand • Managed by others • Virtualized performance, reliability
28. Cloud: Characteristics • Instant on-demand • Managed by others • Virtualized performance, reliability • Lack control predictability, reliability, quality
29. Cloud: Characteristics • Instant on-demand • Managed by others • Pay as you go • Virtualized performance, reliability • Lack control predictability, reliability, quality
30. Cloud: Characteristics • Instant on-demand • Managed by others • Pay as you go • Virtualized performance, reliability • Lack control predictability, reliability, quality • Pay as you go!
31. DevOps Is it a command?
32. DevOps: Definition [1/2] • Share responsibility across organizational boundaries
33. DevOps: Definition [1/2] • Share responsibility across organizational boundaries • Invest in people by reducing finger pointing [togetherness] and human error [automation]
34. DevOps: Definition [1/2] • Share responsibility across organizational boundaries • Invest in people by reducing finger pointing [togetherness] and human error [automation] • Manage infrastructure not priority queues of production issues
35. DevOps: Definition [1/2] • Share responsibility across organizational boundaries • Invest in people by reducing finger pointing [togetherness] and human error [automation] • Manage infrastructure not priority queues of production issues • Make infrastructure predictable repeatable, testable, deterministic
36. DevOps: Definition [2/2]
37. Deployment Pipeline commit -> production
38. Deployment Pipeline: Prerequisites • Design for cloud e.g. decentralized, layered, parallelized, collaborating single purpose services, async I/O
39. Deployment Pipeline: Prerequisites • Design for cloud e.g. decentralized, layered, parallelized, collaborating single purpose services, async I/O • Always-ready codebase buildable, testable, deployable
40. Deployment Pipeline: Prerequisites • Design for cloud e.g. decentralized, layered, parallelized, collaborating single purpose services, async I/O • Always-ready codebase buildable, testable, deployable • Managed infrastructure read: SCM and consistent distribution to target nodes
41. Deployment Pipeline: Prerequisites • Design for cloud e.g. decentralized, layered, parallelized, collaborating single purpose services, async I/O • Always-ready codebase buildable, testable, deployable • Managed infrastructure read: SCM and consistent distribution to target nodes • Expect [system] failure handle failures sensibly, policies for timeouts, etc
42. Deployment Pipeline: Prerequisites • Design for cloud e.g. decentralized, layered, parallelized, collaborating single purpose services, async I/O • Always-ready codebase buildable, testable, deployable • Managed infrastructure read: SCM and consistent distribution to target nodes • Expect [system] failure handle failures sensibly, policies for timeouts, etc • Test early and often! outside-in development helps
43. Deployment Pipeline: Prerequisites • Design for cloud e.g. decentralized, layered, parallelized, collaborating single purpose services, async I/O • Always-ready codebase buildable, testable, deployable • Managed infrastructure read: SCM and consistent distribution to target nodes • Expect [system] failure handle failures sensibly, policies for timeouts, etc • Test early and often! outside-in development helps • Build from the ground up layer infrastructure, inject configuration at boot/load time
44. Deployment: Common Bottlenecks • Automation build, provision, configure, integrate Figure: http://www.flickr.com/people/laenulfean/
45. Deployment: Common Bottlenecks • Automation build, provision, configure, integrate • Distribution binaries, assets, configuration Figure: http://www.flickr.com/people/laenulfean/
46. Deployment: Common Bottlenecks • Automation build, provision, configure, integrate • Distribution binaries, assets, configuration • Timeframe restricted window of time Figure: http://www.flickr.com/people/laenulfean/
47. Deployment: Common Bottlenecks • Automation build, provision, configure, integrate • Distribution binaries, assets, configuration • Timeframe restricted window of time • Data schema updates, data migrations Figure: http://www.flickr.com/people/laenulfean/
48. Solution Approaches Solve problems, don’t use tools!
49. Automation Approaches Figure: http://www.flickr.com/people/krazydad/ • Full stack server-driven e.g. Chef/Knife, Puppet Master
50. Automation Approaches Figure: http://www.flickr.com/people/krazydad/ • Full stack server-driven e.g. Chef/Knife, Puppet Master • Full stack client e.g. Chef Solo
51. Automation Approaches Figure: http://www.flickr.com/people/krazydad/ • Full stack server-driven e.g. Chef/Knife, Puppet Master • Full stack client e.g. Chef Solo • Application-tier client e.g. Capistrano, Vlad the Deployer
52. Automation Approaches Figure: http://www.flickr.com/people/krazydad/ • Full stack server-driven e.g. Chef/Knife, Puppet Master • Full stack client e.g. Chef Solo • Application-tier client e.g. Capistrano, Vlad the Deployer • Command & control e.g. Vertibrae (inactive), Nanite
53. Distribution Approaches Figure: http://www.flickr.com/people/nsalt • Shared filesystem less security and reliability in community/public or across zones/regions
54. Distribution Approaches Figure: http://www.flickr.com/people/nsalt • Shared filesystem less security and reliability in community/public or across zones/regions • Pull from source control higher time variance as target nodes increase
55. Distribution Approaches Figure: http://www.flickr.com/people/nsalt • Shared filesystem less security and reliability in community/public or across zones/regions • Pull from source control higher time variance as target nodes increase • Bittorrent or similar e.g. Twitter’s Murder
56. Timeframe Approaches Figure: http://www.flickr.com/people/athenicsword • Hot upgrades e.g. Erlang/OTP appup/code_change/3
57. Timeframe Approaches Figure: http://www.flickr.com/people/athenicsword • Hot upgrades e.g. Erlang/OTP appup/code_change/3 • Rolling upgrades Software design considerations
58. Timeframe Approaches Figure: http://www.flickr.com/people/athenicsword • Hot upgrades e.g. Erlang/OTP appup/code_change/3 • Rolling upgrades Software design considerations • Environment replacement Flip a switch, acceptance <-> production
59. Data Approaches Figure: http://www.flickr.com/people/shanghaidaddy • Scriptable schema Alternative: NoSQL, but. . .
60. Data Approaches Figure: http://www.flickr.com/people/shanghaidaddy • Scriptable schema Alternative: NoSQL, but. . . • Automated migrations e.g. Rails’ Migrations
61. Data Approaches Figure: http://www.flickr.com/people/shanghaidaddy • Scriptable schema Alternative: NoSQL, but. . . • Automated migrations e.g. Rails’ Migrations • Sanity checks e.g. cherry picked, consistency checks
62. Common Pitfalls & Workarounds
63. Organizational Culture Figure: http://www.flickr.com/people/lucgaloppin/
64. Organizational Culture [Workaround] Figure: http://www.flickr.com/photos/42682395@N04/
65. Feature Branching Figure: http://www.flickr.com/photos/33939293@N02/
66. Feature Branching [Workaround]
67. Feature Branching [Workaround] DON’T!
68. Live by the meter. Die by the meter. Figure: WARNING
69. Live by the meter. Die by the meter. [Workarounds] Figure: http://www.flickr.com/people/dirkjankraan/
70. Great Expectations Figure: http://www.flickr.com/people/atoxinsocks/
71. Great Expectations [Workaround] Figure: http://www.flickr.com/people/dirkjankraan/
72. Note: Design & Idempotency Figure: A Twitter dialog between me and @jlouis666
73. Possibilities Where.next?
74. Possibilities
75. Possibilities • Dynamic resource allocation allocate based on load, time of day, day of week/month
76. Possibilities • Dynamic resource allocation allocate based on load, time of day, day of week/month • Canary deployments (e.g. A/B testing)
77. Possibilities • Dynamic resource allocation allocate based on load, time of day, day of week/month • Canary deployments (e.g. A/B testing) • Multi-region or multi-provider relocate based on time of day, failover
78. Questions? Figure: http://www.flickr.com/photos/42682395@N04/
79. Questions? Figure: http://www.flickr.com/photos/42682395@N04/ @SusanPotter