Consul 0.6 and Consul-Template

Consul is a tool for service discovery, configuration, and monitoring of modern microservices architectures that is broadly deployed and used. In this talk, we will give a brief overview of Consul and the problems it solves and then discuss new features introduced in Consul 0.6. Consul also has a large ecosystem of tooling that makes it easy to integrate into existing infrastructure. One of these tools is consul-template and we will discuss it’s use and new features.

Transcript

1. HASHICORP
2. HASHICORP Armon Dadgar @armon
3. HASHICORP
4. HASHICORP
5. CONSUL HASHICORP Enable SOA / Microservices Datacenter Runtime
6. HASHICORP SOA PRIMER Autonomous Limited Scope Loose Coupling
7. HASHICORP ORDER PROCESSING WEB APP ORDER HISTORY FORECASTING
8. HASHICORP ORDER PROCESSING WEB APP DISCOVERY Which nodes are part of "order processing"?
9. HASHICORP ORDER PROCESSING WEB APP LOAD BALANCING How to ensure request leveling across providers? NODE 1 NODE 2 NODE N
10. HASHICORP ORDER PROCESSING WEB APP ANTI-PATTERN Load Balancer is a Single Point of Failure (SPOF) NODE 1 NODE 2 NODE N LOAD BALANCER
11. HASHICORP ORDER PROCESSING WEB APP HEALTH CHECKING How to avoid routing to unhealthy hosts? NODE 1 NODE 2 NODE 3 LOAD BALANCER
12. HASHICORP WEB APP CONFIGURATION How to efficiently push dynamic configuration? WEB 1 WEB 2 WEB N maintenance: false feature_a: true role: "web"
13. HASHICORP SERVICE DISCOVERY LOAD BALANCING HEALTH CHECKING KEY-VALUE CONFIGURATION 4 BASIC PROBLEMS
14. CONSUL HASHICORP Consul 0.6 Network Tomography Prepared Queries MemDB Enhanced ACLs TCP & Docker Health Checks
15. HASHICORP Network Tomography Model Underlying Network EsFmate RTT Nearest Neighbor RouFng
16. HASHICORP Network Tomography
17. HASHICORP Network Tomography
18. HASHICORP Network Tomography
19. Terminal HASHICORP $ consul rtt node-10-0-1-8 Estimated node-10-0-1-8 <-> node-10-0-1-6 rtt: 0.781 ms (using LAN coordinates)$ $ sleep 30 $ consul rtt node-10-0-1-8 Estimated node-10-0-1-8 <-> node-10-0-1-6 rtt: 0.719 ms (using LAN coordinates)
20. Terminal HASHICORP $ curl localhost:8500/v1/catalog/nodes?near=node-78r16zb3q | jq '.[].Node' "node-78r16zb3q" "node-10-0-4-190" "node-10-0-1-7" "node-10-0-4-240" $ curl localhost:8500/v1/catalog/service/vault? near=node-78r16zb3q | jq '.[].Node' "node-10-0-1-71" "node-10-0-3-119" "node-10-0-3-249"
21. CONSUL HASHICORP Service Discovery DNS Interface HTTP API Smart Clients Consul-Template
22. Terminal HASHICORP $ dig vault.service.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> vault.service.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9406 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;vault.service.consul. IN A ;; ANSWER SECTION: vault.service.consul. 0 IN A 10.0.1.71 vault.service.consul. 0 IN A 10.0.3.119 vault.service.consul. 0 IN A 10.0.3.249 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:16 UTC 2015 ;; MSG SIZE rcvd: 146
23. Terminal HASHICORP $ dig beta.vault.service.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> beta.vault.service.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2613 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;beta.vault.service.consul. IN A ;; ANSWER SECTION: beta.vault.service.consul. 0 IN A 10.0.3.119 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:20 UTC 2015 ;; MSG SIZE rcvd: 126
24. HASHICORP { "Name": "vault-with-failover", "Service": { "Service": "vault", "Failover": { "NearestN": 3 }, "Tags": ["!beta"] }, "DNS": { "TTL": "5s" } }
25. Terminal HASHICORP $ dig vault-with-failover.query.consul ; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> vault-with- failover.query.consul ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1039 ;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;vault-with-failover.query.consul. IN A ;; ANSWER SECTION: vault-with-failover.query.consul. 0 IN A 10.0.1.71 vault-with-failover.query.consul. 0 IN A 10.0.3.249 ;; Query time: 2 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Fri Dec 11 02:09:30 UTC 2015 ;; MSG SIZE rcvd: 150
26. HASHICORP Prepared Queries Complex Queries Datacenter Failover ACL IntegraFon Change Behavior via API
27. HASHICORP Read Path Optimization LMDB (C) B-Tree API (Go) Go <= 1.4, Mark & Sweep LMDB Off-Heap Serialization on boundary
28. HASHICORP Read Path Optimization MemDB (Go) Immutable Radix API (Go) Go >= 1.5, Soft Realtime MemDB On-Heap Pure Go
29. HASHICORP MemDB MulF-Version Concurrency Control (MVCC) TransacFon Support Rich Indexing Avoids SerializaFon Cost
30. HASHICORP Enhanced ACLs (0.4) Key / Value Store (0.5) Service RegistraFon (0.6) Service Discovery (0.6) User Events (0.6) Keyring Updates
31. HASHICORP Enhanced ACLs User API WEB APP Database Catalog API X
32. CONSUL HASHICORP HTTP Check TTL Check Script Check +TCP Check +Docker Check
33. HASHICORP { "check": { "id": "ssh", "name": "SSH TCP on port 22", "tcp": "localhost:22", "interval": "10s", "timeout": "1s" } }
34. HASHICORP { "check": { "id": "mem-util", "name": "Memory utilization", "docker_container_id": "f972c95ebf0e", "shell": "/bin/bash", "script": "/usr/local/bin/check_mem.py", "interval": "10s" } }
35. CONSUL HASHICORP Consul 0.7 Performance Stability Security
36. HASHICORP Consul Eco-System Consul Template Consul Replicate envconsul consul-cli fabio …
37. HASHICORP Consul Template Dynamically render templates Reload on Change Integrate with Everything* Real Time
38. HASHICORP backend frontend maxconn {{ key "frontend/maxconn" }} balance roundrobin{{range service "app.frontend"}} service {{.ID}} {{.Address}}:{{.Port}}{{end}} backend frontend maxconn 256 balance roundrobin server web1 10.0.1.100:80 server web2 10.0.2.200:80
39. HASHICORP {{ with $service := range services }} backend {{$service.Name}} maxconn {{ key (printf “service/%s/maxconn”, $service.Name) }} balance roundrobin{{range service $service.Name}} service {{.ID}} {{.Address}}:{{.Port}}{{end}} 1) Query for all services (1 Query) 2) Query maxconn for each service (N Queries) 3) Query the instances of each service (N Queries) 4) Total of 2*N+1 Queries
40. HASHICORP Consul 2*N+1 Queries M(2*N+1) Queries N = Num Services M = Instances of CT
41. HASHICORP Number of Services Number of Instances Total Queries 10 5 105 25 10 510 50 20 2020 100 30 6030
42. HASHICORP Query De-Duplication Reduce Load Improve QoS
43. HASHICORP Consul 2*N+1 Queries Leader
44. HASHICORP Consul 2*N+1 Queries Leader 1 Write
45. HASHICORP Consul 2*N+1 Queries Leader 1 Write 1 Query Followers
46. HASHICORP Consul 2*N+1 Queries Leader 1 Write 1 Query Followers M+2*N Queries 1 Write N = Num Services M = Instances of CT
47. HASHICORP Number of Services Number of Instances Previous Queries New Queries 10 5 105 25 25 10 510 60 50 20 2020 120 100 30 6030 230
48. HASHICORP 1-2 Order of Magnitude Reduction
49. CONSUL HASHICORP Client Access Patterns Distributed Database Best Practices
50. CONSUL HASHICORP TLS & Encryption ACLs Stale Queries Query De-Duplication CPU & IO Capacity Cluster Health
51. CONSUL HASHICORP Static Configuration Client Usage Dynamic State
52. CONSUL HASHICORP Telemetry Statsd Statsite
53. HASHICORP
54. HASHICORP Best practice for all systems
55. CONSUL HASHICORP Consul 0.6 Consul Template Best Practices & Telemetry
56. HASHICORP Thanks! Q/A