Running Consul at Scale—Journey from RFC to Production

We had many VMs in AWS - were ingesting millions of metrics per second - and were having pain around service discovery and quick configuration changes. This is the story of how we integrated Consul into our environment, what it helped us with, mistakes we made and some tips for successful implementation in your own environment. 10 months later, our growing cluster was using Consul to facilitate 60 second cluster-wide configuration changes and make service discovery simpler and more flexible.

This presentation was given at SREcon16 in Santa Clara, California on April 8, 2016.

More information and a link to a recording of the talk is available here:

https://blog.froese.org/2016/04/08/srecon-running-consul-at-scale/

Transcript

1. C O N S U L @ S R E C O N R U N N I N G C O N S U L @ S C A L E J O U R N E Y F R O M R F C T O P R O D U C T I O N
2. D A R R O N F R O E S E D A R R O N @ F R O E S E . O R G - @ D A R R O N
3. W H E R E W E R E W E ? W I N T E R 2 0 1 4 @ D A TA D O G
4. L AT E 2 0 1 4 • 4 year old codebase. • Cutting apart our monolith. • Rapid growth across the board. • Having config management and service discovery pain.
5. O L D P R A C T I C E S W E R E F R AY I N G W E C O U L D N ’ T D O I T T H E S A M E WA Y A N Y M O R E . H T T P : / / J A S O N W I L D E R . C O M / B L O G / 2 0 1 4 / 0 2 / 0 4 / S E R V I C E - D I S C O V E RY- I N - T H E - C L O U D /
6. S E R V I C E D I S C O V E RY WA S A H Y B R I D • Chef searches. 30 minutes to update. • Large numbers of manually managed IP addresses. • There was nothing really wrong with it - but it was getting harder to manage.
7. D I S T R I B U T E D S Y S T E M S “ M O S Y S T E M S . M O P R O B L E M S . ” - T H E N O T O R I O U S B . I . G .
8. M I D 2 0 1 4 B A C K I N G S T O R E F O R D O C K E R C O N TA I N E R S
9. O V E R A L L P L A N N O V E M B E R 2 0 1 4
10. W H AT I S C O N S U L ?
11. R A F T C O N S E N S U S H T T P : / / T H E S E C R E T L I V E S O F D A TA . C O M / R A F T /
12. C A N I T H E L P D ATA D O G ? W E W E R E N ’ T S U R E .
13. S TA G I N G • ~100 nodes in total. • 3 x m3.medium server nodes
 4GB of RAM - 3 ECU - 1 cpu core - SSD drives.
14. P H A S E 1 P L A N • Initial deploy • Small amount of services. • Minimal KV usage • How will it act? • Consul 0.4.1.
15. B E F O R E P R O D “ M O N I T O R F I R S T ” H T T P S : / / B L O G . F R O E S E . O R G / P R E S E N TA T I O N S /
16. S H I P I T I T ’ S P R O B A B LY F I N E
17. D E P L O Y E D T O P R O D L A T E D E C E M B E R 2 0 1 4 .
18. Y O L O T O T H E M A X T H I S WA S N O T • Not in the critical path. • An outage with Consul could NOT take us down. • Our decision to actually depend on Consul would come later - when it had proven itself.
19. P R O D • 5 x m3.large server nodes
 7.5GB of RAM - 6.5 ECU
 2 cpu cores - SSD drives. • Rapidly required us to spin up 2 more server nodes - it wasn’t stable at 3.
20. I T S TA B I L I Z E D A N D A L L WA S W E L L
21. D ATA D O G S E R V I C E O N E O F T H E F I R S T T H I N G S W E A D D E D .
22. D ATA D O G S E R V I C E
23. D ATA D O G S E R V I C E O N E O F T H E F I R S T T H I N G S W E A D D E D .
24. C O N S U L E X E C T O Y E D W I T H A N D D I S A B L E D
25. G I T 2 C O N S U L S T R O N G LY C O N S I S T E N T 
 K E Y VA L U E S T O R E A VA I L A B L E 
 O N L O C A L H O S T W I T H A N H T T P Q U E RY. H T T P S : / / G I T H U B . C O M / C I M P R E S S - M C P / G I T 2 C O N S U L
26. C O N S U L - C O N F I G G I T 2 C O N S U L
27. G I T 2 C O N S U L + C O N S U L - C O N F I G H O W I T W O R K S
28. M O R E A N D M O R E U S E U P A N D T O T H E R I G H T
29. L E A D E R S H I P T R A N S I T I O N S P R E T T Y C O M M O N - M O S T LY H A R M L E S S ( I N L O W D O S E S )
30. M AY 2 0 1 5 N O D E S + +
31. S E R V I C E R E G I S T R AT I O N W E ’ R E G E T T I N G S E R I O U S N O W
32. S E R V I C E D I S C O V E RY C U R L / H T T P L O O K U P
33. S E R V I C E D I S C O V E RY D N S L O O K U P
34. W O U L D I T F L A P ? I N A N D O U T O F T H E S E R V I C E C A TA L O G
35. N O . I T D I D N O T F L A P I N A N D O U T O F T H E S E R V I C E C A TA L O G
36. U S I N G D N S W O R R I E D A B O U T S P E E D
37. D N S M A S Q F R O N T E D C O N S U L’ S D N S R E S O LV E R
38. C O N S U L _ D N S _ B A C K U P ( T H E H O S T S F I L E ) C O N S U L - T E M P L A T E
39. N O T S U C C E S S F U L E V E N I N S TA G I N G
40. U S E T H E K V S T O R E T O D I S T R I B U T E . B U I LT O N T H E S E R V E R N O D E S
41. I T W O R K S R E A L LY W E L L
42. W I T H O U T R AT E L I M I T I N G I T WA S A B I T H A I RY
43. N O L E A D E R S H I P T R A N S I T I O N S N O N E A T A L L
44. T H E V E RY N E X T D AY “ L E T ’ S C L E A N T H I S U P ”
45. R E A D - P R E S S U R E C A U S I N G L E A D E R S H I P T R A N S I T I O N S
46. C O N S U L I S N E W T H E E D G E WA S A L I T T L E B L O O D Y T H E R E WA S V E RY L I T T L E R E A L W O R L D I N F O R M A T I O N A B O U T I T.
47. L O T S O F S M A L L K E Y S D O N ’ T D O T H I S - W I T H C O N S U L 0 . 5 . X
48. O N C E W E U N D E R S T O O D W E U P S I Z E D O U R V M S
49. T H I N G S Q U I E T E D D O W N I N S TA L L E D L A R G E R S E R V E R N O D E S
50. H O W D I D I T W O R K ? T H E D N S M A S Q T H I N G …
51. D N S M A S Q H O N O R S T H E C O N S U L T T L A S K S O N C E E V E RY 1 0 S E C O N D S
52. I T R E S P O N D S Q U I C K LY W E L L
53. C L U S T E R W I D E S TAT S D N S M A S Q A N D T H E M A G I C A L H O S T S F I L E H T T P S : / / G I T H U B . C O M / D A R R O N / G O S H E
54. D U R I N G A FA I L O V E R
55. P E O P L E W E R E A B I T S C A R E D W E R E N ’ T S U R E I F W E W E R E G O I N G T O C O N T I N U E
56. N O D E S W E R E G O I N G D E A F T H E Y D R O P P E D O U T O F T H E C A TA L O G
57. B U T I C O U L D F I N A L LY D U P L I C AT E T H E P R O B L E M I WA S V I S I T I N G T H E O F F I C E & H E A R D S O M E G R U M B L I N G
58. H A S H I C O R P L E N T A H A N D H U G E T H A N K S T O J A M E S A N D A R M O N F O R A L L T H E I R H E L P !
59. 2 D E A D L O C K S F I X E D B U T T H E K E Y WA S Q U A K E R E L A T E D .
60. A N D A L L WA S R I G H T F O R T H E M O S T PA R T
61. O C T O B E R 2 0 1 5 N O D E S + + + - M O S T LY S TA B L E - B U T C A U T I O U S
62. C O N S U L - C O N F I G H E L P E D A S W E G R E W M A D E R E T I R I N G & S WA P P I N G S O M E S E R V I C E S E A S I E R
63. S TA R T E D U S I N G E V E N T S L I K E C O N S U L E X E C B U T W I T H S E C U R I T Y + +
64. C O N S U L L O C K R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S R U N N I N G WA I T I N G WA I T I N G
65. C O N S U L L O C K R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S C R A S H E S R U N N I N G WA I T I N G X
66. C O N S U L L O C K R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S
67. W H E N T H E L E A D E R S H I P T R A N S I T I O N S G R E W S U P E R S I Z E D O U R S E R V E R S
68. C 3 . 2 X L A R G E D I D T H E T R I C K A T 1 0 0 0 + N O D E S - I T ’ S L I K E W E ( A L M O S T ) T U R N E D T H E M O F F.
69. 2 S M A L L O U TA G E S L A S T Y E A R . O N E F O R 3 M I N U T E S B E C A U S E O F A PA C K A G I N G P R O B L E M O N E F O R A N H O U R T H A T WA S D O C U M E N TA T I O N A N D “ B R O A D C A S T I N P U T T O A L L PA N E S ” R E L A T E D
70. W H AT S H O U L D Y O U K N O W ? W H A T H A V E W E L E A R N E D ?
71. C O N S U L I S A W E S O M E I T ’ S Y O U R D A TA C E N T E R ’ S B A C K B O N E
72. M O N I T O R I N G I S E S S E N T I A L J U S T D O I T
73. U P G R A D E T O 0 . 6 . X T O N S O F F I X E S A N D U P G R A D E S
74. U S E S L E S S M E M O RY TA S T E S G R E A T, L E S S F I L L I N G !
75. C O N S U L L O V E S C P U F E E D I T A L L T H E C P U S
76. S O M E E X A M P L E S I Z I N G • m3.large ~300 nodes • c3.xlarge ~500 nodes • c3.2xlarge ~800 nodes • As always YMMV. • 0.6 is more efficient - might be able to use smaller nodes.
77. E M B R A C E FA I L U R E B U I L D F O R I T - A D D R E T R I E S - B A C K O F F - C I R C U I T B R E A K E R S M A K E S Y O U R W H O L E S Y S T E M M O R E R E S I L I E N T
78. WAT C H Y O U R R E A D V E L O C I T Y D O N ’ T D D O S Y O U R S E L F
79. U S E F E W E R A N D L A R G E R K E Y S R A T H E R T H A N L O T S O F S M A L L K E Y S E S P E C I A L LY I F Y O U ’ R E R E A D I N G A L O T O F T H E M A T O N C E
80. L O C K D O W N PA R T S O F T H E K V S T O R E F E E D I N D A TA F R O M T H E O U T S I D E A C L S A R E Y O U R F R I E N D
81. C O N S U L WAT C H E S A R E P O W E R F U L M A K E S U R E T H E Y O N LY F I R E W H E N Y O U WA N T T H E M T O H T T P S : / / G I T H U B . C O M / D A R R O N / S I F T E R
82. I F O U T P U T I S N ’ T U N I Q U E D O N ’ T B U I L D C O N F I G O N E V E RY N O D E U S E T H E K V S T O R E T O M O V E T H O S E F I L E S A R O U N D
83. O N E M O R E T H I N G T H A T ’ S M Y L A S T T I P F O R T O D A Y B U T W E H A V E
84. K V E X P R E S S U S E T H E K V S T O R E T O T R A N S P O R T C O N F I G U R A T I O N F I L E S I N & O U T B O T H D I R E C T I O N S
85. M A I N F E AT U R E S • 10MB Go binary • Uploads and downloads files under 512KB • Emits Dogstatsd metrics and Datadog Events • Files sent == files delivered • Doesn’t re-upload or re-deliver • Very safe • Runs commands after delivery
86. I T ’ S S U P E R FA S T < 5 0 0 M S T O D E L I V E R A F I L E T O 1 0 0 0 N O D E S
87. S P E E D I N A N D O U T
88. C A N P O S T D I F F S W H E N F I L E S U P D A T E
89. M E T R I C S F O R A L L M E A S U R E A L L T H E T H I N G S
90. H T T P S : / / G I T H U B . C O M / D ATA D O G / K V E X P R E S S
91. S T O P. D E M O T I M E .
92. Q U E S T I O N S ?
93. T H A N K S ! D A R R O N @ F R O E S E . O R G @ D A R R O N G I T H U B . C O M / D A R R O N C O N S U L @ S R E C O N R U N N I N G C O N S U L @ S C A L E J O U R N E Y F R O M R F C T O P R O D U C T I O N