Running Consul at scale: Service discovery in the cloud

Datadog had 400 VMs in AWS, was ingesting millions of metrics per second, and was having pain around service discovery and quick configuration changes. Darron Froese discusses how Datadog integrated Consul into its environment. Eighteen months later, Datadog’s 1,000+ VMs receiving millions more metrics per second were using Consul to facilitate 60-second cluster-wide configuration changes and make service discovery simpler and more flexible. Darron outlines mistakes made and lessons learned as well as some tips for successful implementation in your own environment.

This presentation was given at Velocity New York on September 21, 2016.

More information and a link to a recording of the talk will be available here:

https://blog.froese.org/2016/09/21/velocity-running-consul-at-scale/

Transcript

1. C O N S U L @ V E L O C I T Y R U N N I N G C O N S U L @ S C A L E S E R V I C E D I S C O V E RY I N T H E C L O U D
2. D A R R O N F R O E S E D A R R O N @ F R O E S E . O R G - @ D A R R O N
3. W H AT I ’ M G O I N G T O C O V E R T O D AY I N A S T O RY F O R M A T • Consul is awesome. • Consul had some sharp corners and rough spots. • Consul is a great addition to your infrastructure NOW - here’s how.
4. L AT E 2 0 1 4 • 4 year old codebase. • Cutting apart our monolith. • Rapid growth across the board. • Having config management and service discovery pain.
5. S E R V I C E D I S C O V E RY WA S A H Y B R I D • Chef searches. 30 minutes to update. • Large numbers of manually managed IP addresses. • There was nothing really wrong with it - but it was getting harder to manage.
6. D I S T R I B U T E D S Y S T E M S “ M O S Y S T E M S . M O P R O B L E M S . ” - T H E N O T O R I O U S B . I . G .
7. O V E R A L L P L A N N O V E M B E R 2 0 1 4
8. W H AT I S C O N S U L ?
9. R A F T C O N S E N S U S H T T P : / / T H E S E C R E T L I V E S O F D A TA . C O M / R A F T /
10. C A N I T H E L P D ATA D O G ? W E W E R E N ’ T S U R E .
11. S TA G I N G • ~100 nodes in total. • 3 x m3.medium server nodes
 4GB of RAM - 3 ECU - 1 cpu core - SSD drives.
12. P H A S E 1 P L A N • Initial deploy • Small amount of services. • Minimal KV usage • How will it act? • Consul 0.4.1.
13. B E F O R E P R O D “ M O N I T O R F I R S T ” H T T P S : / / B L O G . F R O E S E . O R G / P R E S E N TA T I O N S /
14. S H I P I T I T ’ S P R O B A B LY F I N E
15. D E P L O Y E D T O P R O D L A T E D E C E M B E R 2 0 1 4 .
16. Y O L O T O T H E M A X T H I S WA S N O T • Not in the critical path. • An outage with Consul could NOT take us down. • Our decision to actually depend on Consul would come later - when it had proven itself.
17. P R O D • 5 x m3.large server nodes
 7.5GB of RAM - 6.5 ECU
 2 cpu cores - SSD drives. • Rapidly required us to spin up 2 more server nodes - it wasn’t stable at 3.
18. I T S TA B I L I Z E D A N D A L L WA S W E L L
19. D ATA D O G S E R V I C E O N E O F T H E F I R S T T H I N G S W E A D D E D .
20. D ATA D O G S E R V I C E
21. D ATA D O G S E R V I C E O N E O F T H E F I R S T T H I N G S W E A D D E D .
22. G I T 2 C O N S U L S T R O N G LY C O N S I S T E N T 
 K E Y VA L U E S T O R E A VA I L A B L E 
 O N L O C A L H O S T W I T H A N H T T P Q U E RY. H T T P S : / / G I T H U B . C O M / C I M P R E S S - M C P / G I T 2 C O N S U L
23. C O N S U L - C O N F I G G I T 2 C O N S U L
24. G I T 2 C O N S U L + C O N S U L - C O N F I G H O W I T W O R K S
25. M O R E A N D M O R E U S E U P A N D T O T H E R I G H T
26. L E A D E R S H I P T R A N S I T I O N S P R E T T Y C O M M O N - M O S T LY H A R M L E S S ( I N L O W D O S E S )
27. M AY 2 0 1 5 N O D E S + +
28. S E R V I C E R E G I S T R AT I O N W E ’ R E G E T T I N G S E R I O U S N O W
29. S E R V I C E D I S C O V E RY C U R L / H T T P L O O K U P
30. S E R V I C E D I S C O V E RY D N S L O O K U P
31. W O U L D I T F L A P ? I N A N D O U T O F T H E S E R V I C E C A TA L O G
32. N O . I T D I D N O T F L A P I N A N D O U T O F T H E S E R V I C E C A TA L O G
33. U S I N G D N S W O R R I E D A B O U T S P E E D
34. D N S M A S Q F R O N T E D C O N S U L’ S D N S R E S O LV E R
35. C O N S U L _ D N S _ B A C K U P ( T H E H O S T S F I L E ) C O N S U L - T E M P L A T E
36. N O T S U C C E S S F U L E V E N I N S TA G I N G
37. U S E T H E K V S T O R E T O D I S T R I B U T E . B U I LT O N T H E S E R V E R N O D E S
38. I T W O R K S R E A L LY W E L L
39. W I T H O U T R AT E L I M I T I N G I T WA S A B I T H A I RY
40. N O L E A D E R S H I P T R A N S I T I O N S N O N E A T A L L
41. T H E V E RY N E X T D AY “ L E T ’ S C L E A N T H I S U P ”
42. R E A D - P R E S S U R E C A U S I N G L E A D E R S H I P T R A N S I T I O N S
43. C O N S U L I S N E W T H E E D G E WA S A L I T T L E B L O O D Y T H E R E WA S V E RY L I T T L E R E A L W O R L D I N F O R M A T I O N A B O U T I T.
44. L O T S O F S M A L L K E Y S D O N ’ T D O T H I S - W I T H C O N S U L
45. O N C E W E U N D E R S T O O D W E U P S I Z E D O U R V M S
46. T H I N G S Q U I E T E D D O W N I N S TA L L E D L A R G E R S E R V E R N O D E S
47. H O W D I D I T W O R K ? T H E D N S M A S Q T H I N G …
48. D N S M A S Q H O N O R S T H E 
 C O N S U L T T L A S K S O N C E E V E RY 1 0 S E C O N D S
49. I T R E S P O N D S Q U I C K LY W E L L
50. C L U S T E R W I D E S TAT S D N S M A S Q A N D T H E M A G I C A L H O S T S F I L E H T T P S : / / G I T H U B . C O M / D A R R O N / G O S H E
51. P E O P L E W E R E A B I T S C A R E D W E R E N ’ T S U R E I F W E W E R E G O I N G T O C O N T I N U E
52. N O D E S W E R E G O I N G D E A F T H E Y D R O P P E D O U T O F T H E C A TA L O G
53. B U T I C O U L D F I N A L LY D U P L I C AT E T H E P R O B L E M I WA S V I S I T I N G T H E O F F I C E & H E A R D S O M E G R U M B L I N G
54. H A S H I C O R P L E N T A H A N D H U G E T H A N K S T O J A M E S A N D A R M O N F O R A L L T H E I R H E L P !
55. 2 D E A D L O C K S F I X E D B U T T H E K E Y WA S Q U A K E R E L A T E D .
56. A N D A L L WA S R I G H T F O R T H E M O S T PA R T
57. O C T O B E R 2 0 1 5 N O D E S + + + - M O S T LY S TA B L E - B U T C A U T I O U S
58. C O N S U L - C O N F I G H E L P E D A S W E G R E W M A D E R E T I R I N G & S WA P P I N G S O M E S E R V I C E S E A S I E R
59. W H E N T H E L E A D E R S H I P T R A N S I T I O N S G R E W S U P E R S I Z E D O U R S E R V E R S
60. C 3 . 2 X L A R G E D I D T H E T R I C K A T 1 0 0 0 + N O D E S - I T ’ S L I K E W E ( A L M O S T ) T U R N E D T H E M O F F.
61. 2 S M A L L O U TA G E S L A S T Y E A R . O N E F O R 3 M I N U T E S B E C A U S E O F A PA C K A G I N G P R O B L E M O N E F O R A N H O U R T H A T WA S D O C U M E N TA T I O N A N D “ B R O A D C A S T I N P U T T O A L L PA N E S ” R E L A T E D
62. W H AT S H O U L D Y O U K N O W ? W H A T H A V E W E L E A R N E D ?
63. C O N S U L I S A W E S O M E I T ’ S Y O U R D A TA C E N T E R ’ S B A C K B O N E
64. M O N I T O R I N G I S E S S E N T I A L J U S T D O I T
65. U P G R A D E T O 0 . 6 . X T O N S O F F I X E S A N D U P G R A D E S
66. U S E S L E S S M E M O RY TA S T E S G R E A T, L E S S F I L L I N G !
67. C O N S U L L O V E S C P U F E E D I T A L L T H E C P U S
68. N E T W O R K B A N D W I D T H C A P S A R E R E A L
69. S O M E E X A M P L E S I Z I N G • m3.large ~300 nodes • c3.xlarge ~500 nodes • c3.2xlarge ~800-2000 nodes (provided you’re guarding cpu and network) • As always YMMV.
70. E M B R A C E FA I L U R E B U I L D F O R I T - A D D R E T R I E S - B A C K O F F - C I R C U I T B R E A K E R S M A K E S Y O U R W H O L E S Y S T E M M O R E R E S I L I E N T
71. WAT C H Y O U R R E A D V E L O C I T Y D O N ’ T D D O S Y O U R S E L F
72. U S E F E W E R A N D L A R G E R K E Y S R A T H E R T H A N L O T S O F S M A L L K E Y S E S P E C I A L LY I F Y O U ’ R E R E A D I N G A L O T O F T H E M A T O N C E
73. O L D P R E F I X WAT C H L O T S O F T I N Y K E Y S
74. N E W H O T N E S S 1 L A R G E R K E Y
75. L O C K D O W N PA R T S O F T H E K V S T O R E F E E D I N D A TA F R O M T H E O U T S I D E A C L S A R E Y O U R F R I E N D
76. C O N S U L WAT C H E S A R E P O W E R F U L M A K E S U R E T H E Y O N LY F I R E W H E N Y O U WA N T T H E M T O H T T P S : / / G I T H U B . C O M / D A R R O N / S I F T E R
77. I F O U T P U T I S N ’ T U N I Q U E D O N ’ T B U I L D C O N F I G O N E V E RY N O D E U S E T H E K V S T O R E T O M O V E T H O S E F I L E S A R O U N D
78. K V E X P R E S S U S E T H E K V S T O R E T O T R A N S P O R T C O N F I G U R A T I O N F I L E S I N & O U T B O T H D I R E C T I O N S
79. M A I N F E AT U R E S • 10MB Go binary • Uploads and downloads files under 512KB • Emits Dogstatsd metrics and Datadog Events • Files sent == files delivered • Doesn’t re-upload or re-deliver. • Runs commands after delivery
80. I T ’ S S U P E R FA S T < 5 0 0 M S T O D E L I V E R A F I L E T O 1 0 0 0 N O D E S
81. H T T P S : / / G I T H U B . C O M / D ATA D O G / K V E X P R E S S
82. T H A N K S ! D A R R O N @ F R O E S E . O R G @ D A R R O N G I T H U B . C O M / D A R R O N C O N S U L @ V E L O C I T Y R U N N I N G C O N S U L @ S C A L E S E R V I C E D I S C O V E RY I N T H E C L O U D