AWS SQS queues & Kubernetes Autoscaling Pitfalls Stories

Talk at the Cloud Native Computing Foundation meetup @dcard.tw

1. AWS SQS queues & Kubernetes Autoscaling Pitfalls Stories Cloud Native Foundation meetup @dcard.tw @eric_khun
2. Make it work, Make it right, Make it fast kent beck (agile manifesto - extreme programming)
3. Make it work, Make it right, Make it fast kent beck (agile manifesto - extreme programming)
4. Make it work, Make it right, Make it fast kent beck (agile manifesto - extreme programming)
5. Buffer
7. Buffer • 80 employees , 12 time zones, all remote
8. Quick intro
10. Main pipelines flow
11. it can look like ... golang Talk @Maicoin :
14. How do we send posts to social medias?
15. A bit of history... 2010 -> 2012: Joel (founder/ceo) 1 cronjob on a Linode server $20/mo 512 mb of RAM 2012 -> 2017 : Sunil (ex-CTO) Crons running on AWS ElasticBeanstalk / supervisord 2017 -> now: Kubernetes / CronJob controller
16. AWS Elastic Beanstalk: Kubernetes:
17. At what scale? ~ 3 million SQS messages per hour
18. Different patterns for many queues
19. Are our workers (consumers of the SQS queues ) efficients?
20. Are our workers efficients?
21. Are our workers efficients?
22. Empty messages? > Workers tries to pull messages from SQS, but receive “nothing” to process
23. Number of empty messages per queue
24. Sum of empty messages on all queues
26. 1,000,000 API calls to AWS costs 0.40$ We have 7,2B calls/month for “empty messages” It costs ~$25k/year > Me:
28. AWS SQS Doc
30. Or in the AWS console
31. Results?
32. empty messages
33. AWS
35. $120 > $50 saved daily > $2000 / month > $25,000 / year (it’s USD, not TWD)
36. Paid for querying “nothing”
37. (for the past 8 years )
38. Benefits - Saving money - Less CPU usage (less empty requests) - Less throttling (misleading) - Less containers > Better resources allocation: memory/cpu request
39. Why did that happen?
40. Default options
42. Never questioning what’s working decently or the way it’s been always done
43. What could have helped? Infra as code (explicit options / standardization) SLI/SLOs (keep re-evaluating what’s important) AWS architecture reviews (taging/recommendations from aws solutions architects)
44. Make it work, Make it right, Make it fast
45. Make it work, Make it right, Make it fast
46. Do you remember?
50. Need to analytics on Twitter/FB/IG/LKD… on millions on posts faster
51. workers consuming time
53. What’s the problem?
54. Resources allocated and not doing anything most of the time Developer trying to put find compromises on the number of workers
55. How to solve it?
56. Autoscaling! (with Keda.sh) Supported by IBM / Redhat / Microsoft
58. Results
60. But notice anything?
61. Before autoscaling
62. After autoscaling
63. After autoscaling
64. What’s happening?
65. Downscaling
66. Why?
67. delete pod lifecycle
68. what went wrong - Workers didn’t manage SIGTERM sent by k8s - Kept processing messages - Messages were halfway processed and killed - Messages were sent back to the the queue again - Less workers because of downscaling
69. solution - When receiving SIGTERM stop processing new messages - Set a graceful period long enough to process the current message if (SIGTERM) { // finish current processing and stop receiving new messages }
72. And it can also help with sqs empty messages
73. Make it work, Make it right, Make it fast
74. Make it work, Make it right, Make it fast
75. Thanks!
76. Questions? monitory.io taiwangoldcard.com travelhustlers.co ✈