Service Discovery and Configuration Management

A talk given at Loadays, all about the need for two speeds of configuration - modelled and emergent. Includes an example of using Puppet along with Consul.

Transcript

1. Service Discovery and Configuration Management Puppet Labs Gareth Rushgrove Two speeds of configuration
2. @garethr
3. Gareth Rushgrove
4. Gareth Rushgrove
5. This talk
6. A quick introduction to Service Discovery Gareth Rushgrove
7. Why it’s useful to have two speeds of configuration Gareth Rushgrove
8. An example of using Consul alongside Puppet Gareth Rushgrove
9. Some brief thoughts about change control and autonomous systems Gareth Rushgrove
10. What is Service Discovery?
11. Gareth Rushgrove
12. Service discovery is a key component of most distributed systems and service oriented architectures. Gareth Rushgrove http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/
13. The problem seems simple at first: How do clients determine the IP and port for a service that exist on multiple hosts? Gareth Rushgrove http://jasonwilder.com/blog/2014/02/04/service-discovery-in-the-cloud/
14. Etcd, Consul, Zookeeper Gareth Rushgrove
15. Hadoop uses Zookeeper Socketplane uses Consul CoreOS uses etcd Cloud Foundry uses Consul Cloud Foundry also uses etcd Kubernetes uses etcd Gareth Rushgrove
16. Gareth Rushgrove Thanks to etcd for the graphics
17. Gareth Rushgrove Thanks to etcd for the graphics
18. Gareth Rushgrove Thanks to etcd for the graphics
19. Gareth Rushgrove Jepsen
20. Why now?
21. Every organisation wants to move quicker Gareth Rushgrove
22. Provisioning times are decreasing rapidly Gareth Rushgrove
23. The number of applications, and instances of those applications, is increasing rapidly Gareth Rushgrove
24. The number of systems administrators is probably going to stay the same. Sorry. Gareth Rushgrove
25. Technology moving towards a cluster centric view rather than a host centric view Gareth Rushgrove
26. Hard-coding everything doesn’t scale Gareth Rushgrove
27. Some relevant technology you may have heard of Gareth Rushgrove
28. Gareth Rushgrove
29. Gareth Rushgrove
30. Gareth Rushgrove
31. Gareth Rushgrove
32. Two speeds of configuration
33. Configuration management tools tend to focus on upfront modelling, managing configuration we know and want to control over time Gareth Rushgrove
34. For example the version of a package or application, user permissions, security settings, system constrains Gareth Rushgrove
35. Service discovery tools handle emergent configuration which is a property of the running system Gareth Rushgrove
36. For example load balancer and proxy backends, network and firewall routes as applications come and go Gareth Rushgrove
37. Wait, but I want to define some of my network configuration upfront Gareth Rushgrove
38. Hence two speeds, model driven and emergent Gareth Rushgrove
39. How to mix configuration management and service discovery
40. We’ll use Consul for this example Gareth Rushgrove
41. We’ll also use Puppet Gareth Rushgrove
42. First lets install Consul everywhere Gareth Rushgrove
43. Gareth Rushgrove
44. That’s it Gareth Rushgrove
45. Gareth Rushgrove KyleAnderson/consul
46. We can configure servers and clients using Hiera Gareth Rushgrove
47. Gareth Rushgrove
48. Gareth Rushgrove
49. Consul nodes need to register with a server Gareth Rushgrove
50. Gareth Rushgrove
51. Populating the Consul service registry Gareth Rushgrove
52. Gareth Rushgrove
53. Gareth Rushgrove Consul dashboard
54. Create your own types Gareth Rushgrove
55. Gareth Rushgrove
56. Gareth Rushgrove
57. Using information from Consul in Puppet Gareth Rushgrove
58. Gareth Rushgrove
59. Gareth Rushgrove
60. Gareth Rushgrove
61. Gareth Rushgrove lynxman/hiera_consul
62. Trigger Puppet when something pertinent changes Gareth Rushgrove
63. Gareth Rushgrove
64. Gareth Rushgrove Demo time
65. Same example possible with a bit of hacking for etcd Gareth Rushgrove
66. Gareth Rushgrove garethr/hiera-etcd
67. Gareth Rushgrove KyleAnderson/etcd
68. Advantages Changes happen quickly, but without losing the ability to address configuration drift Gareth Rushgrove
69. Advantages Reduce complexity of watchers, all changes lead to a Puppet run Gareth Rushgrove
70. Advantages All changes to configuration, however triggered, are tracked in one place Gareth Rushgrove
71. Advantages Define all watchers in code Gareth Rushgrove
72. Advantages Reduce duplication, same code runs for new and existing instances Gareth Rushgrove
73. Bonus distributed configuration Gareth Rushgrove
74. Replace lots of identical local files on disk with distributed key/values Gareth Rushgrove
75. Gareth Rushgrove
76. Gareth Rushgrove
77. Gareth Rushgrove garethr/key_value_config
78. Conclusions
79. Change control and autonomous systems Gareth Rushgrove
80. The importance of building trust in automation Gareth Rushgrove
81. Two speeds of configuration Gareth Rushgrove
82. Questions? And thanks for listening