Cloudstack design decisions

Cloud operations at scale

Transcript

1. CLOUDSTACK DESIGN DECISIONS CLOUD OPERATIONS AT SCALE
2. SHORT BIO Pierre-Yves Ritschard CTO @ exoscale - The safe home for your cloud applications Open Source Developer - pithos, cyanite, riemann, collectd, openbsd Architect of several cloud platforms - paper.li Recovering Operations Engineer
3. Simple and efficient cloud hosting platform Full compatibility with automation tools Hosted in a safe jurisdiction
4. CLOUD BUILDING BLOCKS service infrastructure software people
5. SERVICE SIMPLICITY AND SCALABILITY Cloudstack based Basic networking Local storage KVM hypervisor: SmartOS inspired
6. CLOUDSTACK Great extensibility, easy to plug into.
7. BASIC NETWORKING One IP per VM. Security groups are hypervisor controlled layer 2 firewall rules. Provides all the flexibility of a traditionnal firewall, completely API controlled.
8. LOCAL STORAGE Fast I/O, persistent disks.
9. KVM HYPERVISOR Best in class hypervisor. Diskless and netboot approach. Avoids resource waste, facilitates upgrades.
10. INFRASTRUCTURE THE GOOD CITIZEN CONTRACT Configuration management Visibility Build factory Remote execution
11. THE GOOD CITIZEN CONTRACT new machines have roles role defines converged configuration as sum of components each component has an expected normal state and reports it no local intervention needed
12. CONFIGURATION MANAGEMENT code is a great way to define infrastructure ensures homogeneity ability to iterate fast great source of change tracking avoids fear of change
13. OVER 3000 COMMITS
14. CONFIGURATION MANAGEMENT: PUPPET battle tested tool simple declarative DSL to express configuration fits our component approach well
15. VISIBILITY FROM THE MAP TO THE TERRITORY logs metrics alerts
16. WHY FOCUS ON VISIBILITY distributed systems with lots of moving parts, high node volatility
17. LOGS all application and system logs sent over the wire logstash disects and extracts metadata elasticsearch indexes for easy retrieval simple correlation
19. METRICS all application and system metrics sent over the wire by collectd graphite's carbon aggregates and produces appropriate roll- ups if it moves, graph it. if it doesn't, graph it if it starts moving.
21. ALERTS unbounded stream of log and metric data passive approach bodes well with node volatility riemann takes decisions based on stream content ability to extract meaningful information
22. BUILD FACTORY continuous integration package repositories
23. CONTINUOUS INTEGRATION over 60 build jobs ties into our code hosting platform handled by jenkins
24. PACKAGE REPOSITORIES generates valid and signed Debian repositories ensures fast upgrades simplifies configuration management
25. REMOTE EXECUTION a simple pubsub system recurrent commands stored as scenarios command line, HTTP and IRC interaction
26. A SIMPLE PUBSUB SYSTEM each node runs an agent responsible for carrying out commands. commands are sent to groups of nodes (by predicates such as role).
27. RECURRENT COMMANDS STORED AS SCENARIOS intricate workflows can be expressed through a simple DSL
28. COMMAND LINE, HTTP AND IRC INTERACTION most of our production environment can be controlled through our chatroom
29. SOFTWARE FILLING IN THE GAPS Customer management Real-time metering and billing Integrated console A few other things
30. CUSTOMER MANAGEMENT Keeping track of our users Support services (ticket management, coupons, emails)
31. REAL-TIME METERING AND BILLING can't be tied to a cloudstack only solution cloudstack emits useful data ties into our customer management
32. INTEGRATED CONSOLE integrated experience across our services hides complexity and cloudstack specifics exposes exoscale specific features
34. A FEW OTHER THINGS pithos cyanite fleet collectd add-ons
35. PEOPLE EFFICIENT WORK. QUIET NIGHTS Small SRE team Avoiding deploy anxiety
36. SMALL SRE TEAM Our platform must be simple to operate, additional moving parts must provide business value or help operations
37. AVOIDING DEPLOY ANXIETY Our software and infrastructure helps ensure we have good tools to ensure quiet nights and easily caught errors
38. LOOKING BACK Cloudstack is a solid foundation for a IAAS platform There's a bit more to it than just installing cloudstack Building a sustainable and scalable platform on top of cloudstack is possible
39. QUESTIONS ?