Containerizing your monolith

My presentation @ LISA19

Transcript

1. Containerizing your monolith Jano González (@janogonzalez)
2. Introduction
3. What’s SoundCloud
4. > 200M Tracks > 20M Creators Many SoundCloud Rappers What’s SoundCloud
5. 2017-2018 Migration Keep moving Motivation
6. What’s the balance? Motivation
7. Our migration to microservices The monolith Containerizing the monolith Conclusion Contents
8. Our migration to microservices
9. $ rails new soundcloud How we started
10. Around 2012
11. Around 2012
12. Around 2012
13. Around 2012
14. After
15. After
16. But what about deployment?
17. Component Environment Zone Deployment Our abstractions
18. Deployment The process
19. The monolith
20. Track User Playlist The monolith Core entities
21. 360 Chef provisioned bare-metal machines Rails 2.3 Capistrano deployment The monolith The technology
22. The monolith The architecture Public API Internal API Public API Strangler Internal API caching/ strangler Another BFF Many microservices Public Web Workers
23. The monolith The components Public API MoshiMoshi (Internal API) Public Web Assets MoshiMoshi Comments (Internal API) Workers Cron Shell Migration
24. The monolith The hosts component statsd-exporter mtail statsd passenger- exporter …
25. Utilization Deployment Lack of confidence The monolith The issues
26. Containerizing the monolith
27. Throw it into Kubernetes
28. Congrats, your monolith is a microservice now
29. Thank You!
30. 1.5 Engineers 1 year until the last bit was cleaned up The project
31. The first milestone
32. Docker development container Tests on GoCD The first milestone
33. The proof of concept
34. First staging component The proof of concept Does it even work?
35. The proof of concept PR O BLEM ! Init script Nginx Passenger Passenger Process Rails App ??? Where are my env variables?
36. The proof of concept Env variables with The Perl Hack™ SO LU TIO N
37. Productionizing
38. Deployment Monitoring Logs Productionizing Does it run where it matters?
39. Productionizing Anatomy of a traffic serving pod component statsd statsd-exporter passenger- exporter mtail twemproxy twemproxy-cu twemproxy- exporter init
40. Don’t choke service discovery Be allocatable Productionizing Sizing the pods
41. CPU units for main container Passenger processes Productionizing Sizing the pods 3 16
42. Productionizing PR O BLEM ! Stdout v/s the log metrics exporter Component STDOUT Log aggregator File??? Mtail
43. Productionizing Mtail with The Rotatelogs Hack™ SO LU TIO N
44. Productionizing Public API
45. Orchestration Productionizing Public API
46. Productionizing Public API PR O BLEM ! DNS latency and our excessive usage Component Service Service Service
47. Productionizing Public API CoreDNS and the DNS Hack™ SO LU TIO N
48. Productionizing Internal API
49. Highest throughput Productionizing Internal API
50. Productionizing Internal API PR O BLEM ! Latency was too high
51. Productionizing Internal API Optimize GC and make cheaper SQL queries SO LU TIO N
52. Productionizing Internal API PR O BLEM ! Errors spikes during deployment
53. Productionizing Internal API The preStop Trick™ SO LU TIO N
54. Productionizing Internal API PR O BLEM ! Errors spikes during deployment (still???)
55. Productionizing Internal API The Pre Start Trick™ SO LU TIO N
56. The rest
57. • Workers • Cron jobs • Shell / Migration hosts • Cleanup! The rest
58. Finishing
59. Current status
60. On-prem Cloud Current status Number of pods ~1000 ~140
61. On-prem RPS Cloud RPS Current status Traffic 25K 3K
62. Current status Many deploys
63. Conclusions
64. One Infrastructure One Delivery Process Conclusions What we solved
65. Step by step Controlled rollouts Managing expectations Conclusions How we did it
66. Improved utilization Increased confidence Enabling new initiatives Conclusions Benefits
67. Assess current progress Evaluate costs and benefits Conclusions Should you do it?
68. Thank You!
69. @janogonzalez https://soundcloud.com/janogonzalez