Service Discovery in the Cloud

We had many VMs in AWS - were ingesting millions of metrics per second - and were having pain around service discovery and quick configuration changes. This is the story of how we integrated Consul into our environment, what it helped us with, mistakes we made and some tips for successful implementation in your own environment. 10 months later, our growing cluster was using Consul to facilitate 60 second cluster-wide configuration changes and make service discovery simpler and more flexible.

This presentation was given at Scale14x in Pasadena, California on January 23, 2016.

More information and a link to a recording of the talk is available here: https://blog.froese.org/2016/01/23/service-discovery-in-the-cloud-with-consul/

Transcript

1. C O N S U L @ S C A L E 1 4 X S E R V I C E D I S C O V E RY I N T H E C L O U D
2. D A R R O N F R O E S E D A R R O N @ F R O E S E . O R G - @ D A R R O N
3. W H AT I S S E R V I C E D I S C O V E RY ? T W O M A I N C O M P O N E N T S
4. S E R V I C E R E G I S T R AT I O N I P R O V I D E T H I S S E R V I C E A T T H I S I P A D D R E S S A N D P O R T. S E R V I C E D I S C O V E RY C A N Y O U T E L L M E W H E R E T O C O N N E C T T O T H I S S E R V I C E ?
5. W H E R E W E R E W E ? W I N T E R 2 0 1 4 @ D A TA D O G
6. L AT E 2 0 1 4 • 4 year old codebase. • Cutting apart our monolith. • Rapid growth across the board.
7. O L D P R A C T I C E S W E R E F R AY I N G W E C O U L D N ’ T D O I T T H E S A M E WA Y A N Y M O R E . H T T P : / / J A S O N W I L D E R . C O M / B L O G / 2 0 1 4 / 0 2 / 0 4 / S E R V I C E - D I S C O V E RY- I N - T H E - C L O U D /
8. D I S C O V E RY WA S A H Y B R I D • Chef searches. 30 minutes to update. • Large numbers of manually managed IP addresses. • There was nothing really wrong with it - but it was getting harder to manage.
9. D I S T R I B U T E D S Y S T E M S “ M O S Y S T E M S . M O P R O B L E M S . ” - T H E N O T O R I O U S B . I . G .
10. M I D 2 0 1 4 B A C K I N G S T O R E F O R D O C K E R C O N TA I N E R S
11. O V E R A L L P L A N N O V E M B E R 2 0 1 4
12. W H AT I S C O N S U L ?
13. R A F T C O N S E N S U S H T T P : / / T H E S E C R E T L I V E S O F D A TA . C O M / R A F T /
14. C A N I T H E L P D ATA D O G ? W E W E R E N ’ T S U R E .
15. S TA G I N G • ~100 nodes in total. • 3 x m3.medium server nodes
 4GB of RAM - 3 ECU - 1 cpu core - SSD drives.
16. P H A S E 1 P L A N • Initial deploy • Small amount of services. • Minimal KV usage • How will it act? • Consul 0.4.1.
17. B E F O R E P R O D “ M O N I T O R F I R S T ” H T T P S : / / B L O G . F R O E S E . O R G / P R E S E N TA T I O N S /
18. S H I P I T I T ’ S P R O B A B LY F I N E
19. D E P L O Y E D T O P R O D L A T E D E C E M B E R 2 0 1 4 .
20. P R O D • 5 x m3.large server nodes
 7.5GB of RAM - 6.5 ECU
 2 cpu cores - SSD drives. • Rapidly required us to spin up 2 more server nodes - it wasn’t stable at 3.
21. I T S TA B I L I Z E D A N D A L L WA S W E L L
22. D ATA D O G S E R V I C E O N E O F T H E F I R S T T H I N G S W E A D D E D .
23. D ATA D O G S E R V I C E
24. D ATA D O G S E R V I C E O N E O F T H E F I R S T T H I N G S W E A D D E D .
25. C O N S U L E X E C T O Y E D W I T H A N D D I S A B L E D
26. G I T 2 C O N S U L S T R O N G LY C O N S I S T E N T 
 K E Y VA L U E S T O R E A VA I L A B L E 
 O N L O C A L H O S T W I T H A N H T T P Q U E RY. H T T P S : / / G I T H U B . C O M / C I M P R E S S - M C P / G I T 2 C O N S U L
27. C O N S U L - C O N F I G G I T 2 C O N S U L
28. G I T 2 C O N S U L + C O N S U L - C O N F I G H O W I T W O R K S
29. M O R E A N D M O R E U S E U P A N D T O T H E R I G H T
30. L E A D E R S H I P T R A N S I T I O N S P R E T T Y C O M M O N - M O S T LY H A R M L E S S
31. M AY 2 0 1 5 A L M O S T 6 0 0 N O D E S
32. S E R V I C E R E G I S T R AT I O N W E ’ R E G E T T I N G S E R I O U S N O W
33. S E R V I C E D I S C O V E RY C U R L / H T T P L O O K U P
34. S E R V I C E D I S C O V E RY D N S L O O K U P
35. W O U L D I T F L A P ? I N A N D O U T O F T H E S E R V I C E C A TA L O G
36. N O . I T D I D N O T F L A P I N A N D O U T O F T H E S E R V I C E C A TA L O G
37. U S I N G D N S W O R R I E D A B O U T S P E E D
38. D N S M A S Q F R O N T E D C O N S U L’ S D N S R E S O LV E R
39. C O N S U L _ D N S _ B A C K U P ( T H E H O S T S F I L E ) C O N S U L - T E M P L A T E
40. N O T S U C C E S S F U L E V E N I N S TA G I N G
41. U S E T H E K V S T O R E T O D I S T R I B U T E . B U I LT O N T H E S E R V E R N O D E S
42. I T W O R K S R E A L LY W E L L
43. W I T H O U T R AT E L I M I T I N G I T WA S A B I T H A I RY
44. N O L E A D E R S H I P T R A N S I T I O N S N O N E A T A L L
45. T H E V E RY N E X T D AY “ L E T ’ S C L E A N T H I S U P ”
46. R E A D - P R E S S U R E C A U S I N G L E A D E R S H I P T R A N S I T I O N S
47. C O N S U L I S N E W T H E E D G E WA S A L I T T L E B L O O D Y T H E R E WA S V E RY L I T T L E R E A L W O R L D I N F O R M A T I O N A B O U T I T.
48. L O T S O F S M A L L K E Y S D O N ’ T D O T H I S - W I T H C O N S U L 0 . 5 . X
49. O N C E W E U N D E R S T O O D W E U P S I Z E D O U R V M S
50. T H I N G S Q U I E T E D D O W N I N S TA L L E D L A R G E R S E R V E R N O D E S
51. H O W D I D I T W O R K ? T H E D N S M A S Q T H I N G …
52. D N S M A S Q H O N O R S T H E C O N S U L T T L A S K S O N C E E V E RY 1 0 S E C O N D S
53. I T R E S P O N D S Q U I C K LY W E L L
54. H O W D I D I T W O R K ? D N S M A S Q A N D T H E M A G I C A L H O S T S F I L E
55. P E O P L E W E R E A B I T S C A R E D W E R E N ’ T S U R E I F W E W E R E G O I N G T O C O N T I N U E
56. N O D E S W E R E G O I N G D E A F T H E Y D R O P P E D O U T O F T H E C A TA L O G
57. B U T I C O U L D F I N A L LY D U P L I C AT E T H E P R O B L E M I WA S V I S I T I N G T H E O F F I C E & H E A R D S O M E G R U M B L I N G
58. H A S H I C O R P L E N T A H A N D H U G E T H A N K S T O J A M E S A N D A R M O N F O R A L L T H E I R H E L P !
59. 2 D E A D L O C K S F I X E D B U T T H E K E Y WA S Q U A K E R E L A T E D .
60. A N D A L L WA S R I G H T F O R T H E M O S T PA R T
61. O C T O B E R 2 0 1 5 8 5 0 N O D E S - M O S T LY S TA B L E - B U T C A U T I O U S
62. C O N S U L - C O N F I G H E L P E D A S W E G R E W M A D E R E T I R I N G & S WA P P I N G S O M E S E R V I C E S E A S I E R
63. S TA R T E D U S I N G E V E N T S L I K E C O N S U L E X E C B U T W I T H S E C U R I T Y + +
64. C O N S U L L O C K R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S R U N N I N G WA I T I N G WA I T I N G
65. C O N S U L L O C K R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S C R A S H E S R U N N I N G WA I T I N G X
66. C O N S U L L O C K R U N 1 O F N P R O C E S S E S - W I T H H O T S PA R E S
67. W H E N T H E L E A D E R S H I P T R A N S I T I O N S G R E W S U P E R S I Z E D O U R S E R V E R S
68. C 3 . 2 X L A R G E D I D T H E T R I C K A T 1 2 0 0 N O D E S - I T ’ S L I K E W E ( A L M O S T ) T U R N E D T H E M O F F.
69. 2 S M A L L O U TA G E S L A S T Y E A R . O N E F O R 3 M I N U T E S B E C A U S E O F A PA C K A G I N G P R O B L E M O N E F O R A N H O U R T H A T WA S D O C U M E N TA T I O N A N D “ B R O A D C A S T I N P U T T O A L L PA N E S ” R E L A T E D
70. B T W - W E ’ R E H I R I N G H T T P : / / J O B S . D ATA D O G H Q . C O M /
71. J A N U A RY 2 0 1 6 W H A T H A V E W E L E A R N E D ?
72. C O N S U L I S A W E S O M E I T ’ S Y O U R D A TA C E N T E R ’ S B A C K B O N E
73. M O N I T O R I N G I S E S S E N T I A L J U S T D O I T
74. U P G R A D E T O 0 . 6 . X T O N S O F F I X E S A N D U P G R A D E S
75. U S E S L E S S M E M O RY TA S T E S G R E A T, L E S S F I L L I N G !
76. C O N S U L L O V E S C P U F E E D I T A L L T H E C P U S
77. S O M E E X A M P L E S I Z I N G • m3.large ~300 nodes • c3.xlarge ~500 nodes • c3.2xlarge ~800 nodes • As always YMMV. • 0.6 is more efficient - might be able to be smaller nodes.
78. E M B R A C E FA I L U R E B U I L D F O R I T - A D D R E T R I E S - B A C K O F F - C I R C U I T B R E A K E R S M A K E S Y O U R W H O L E S Y S T E M M O R E R E S I L I E N T
79. WAT C H Y O U R R E A D V E L O C I T Y D O N ’ T D D O S Y O U R S E L F
80. U S E F E W E R A N D L A R G E R K E Y S R A T H E R T H A N L O T S O F S M A L L K E Y S E S P E C I A L LY I F Y O U ’ R E R E A D I N G A L O T O F T H E M A T O N C E
81. L O C K D O W N PA R T S O F T H E K V S T O R E F E E D I N D A TA F R O M T H E O U T S I D E A C L S A R E Y O U R F R I E N D
82. C O N S U L WAT C H E S A R E P O W E R F U L M A K E S U R E T H E Y O N LY F I R E W H E N Y O U WA N T T H E M T O H T T P S : / / G I T H U B . C O M / D A R R O N / S I F T E R
83. I F O U T P U T I S N ’ T U N I Q U E D O N ’ T B U I L D C O N F I G O N E V E RY N O D E U S E T H E K V S T O R E T O M O V E T H O S E F I L E S A R O U N D
84. O N E M O R E T H I N G T H A T ’ S M Y L A S T T I P F O R T O D A Y B U T W E H A V E
85. K V E X P R E S S U S E T H E K V S T O R E T O T R A N S P O R T C O N F I G U R A T I O N F I L E S I N & O U T B O T H D I R E C T I O N S
86. M A I N F E AT U R E S • 10MB Go binary • Uploads and downloads files under 512KB • Emits Dogstatsd metrics and Datadog Events • Files sent == files delivered • Doesn’t re-upload or re-deliver • Very safe • Runs commands after delivery
87. I T ’ S S U P E R FA S T < 5 0 0 M S T O D E L I V E R A F I L E T O 1 0 0 0 N O D E S
88. S P E E D I N A N D O U T
89. C A N P O S T D I F F S W H E N F I L E S U P D A T E
90. M E T R I C S F O R A L L M E A S U R E A L L T H E T H I N G S
91. V E RY Q U I C K D E M O
92. H T T P S : / / G I T H U B . C O M / D ATA D O G / K V E X P R E S S
93. Q U E S T I O N S ?
94. C O N S U L @ S C A L E 1 4 X S E R V I C E D I S C O V E RY I N T H E C L O U D T H A N K S ! D A R R O N @ F R O E S E . O R G @ D A R R O N G I T H U B . C O M / D A R R O N