Introducing Vamana

As presented on Chennai AWS Meetup - October 2015

Event - http://www.meetup.com/Chennai-Amazon-Web-Services-Meetup/events/226182749/

1. Introducing Vamana j.mp/to-vamana
2. About Me • Dev/Ops @ Indix • Hindu Mythology Fan • OSS Contributor • ashwanthkumar.in
3. init Auto scalars are init systems in your data center
4. Enter AWS
5. • Scales massively • Has Cool Off capabilities to avoid scale storms • Auto Rebalancing across AZs / Subnets • Integration with ELB Amazon ASG - Good Parts
6. Amazon ASG - Limitations • Can support only 1 launch configuration actively ◦ Single Instance type ◦ Single life cycle - Spot / OD • Auto Scaling Tightly coupled with only Cloudwatch • Alarms can be triggered (automatically) but only based on a single metric • Limited stat functions - avg, sum, min, max and # data samples
7. Enter Vamana http://github.com/ashwanthkumar/vamana2
8. Vamana Architecture Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
9. Vamana for Hadoop1 Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push MR Demand and Supply Metrics Get MR Demand And Supply Metrics Set the computed “Desired” Value
10. Demand vs Supply Metrics for Hadoop • We collect supply metrics from the Cluster Summary table ◦ map_supply ◦ reduce_supply • Demand metrics are collected as cumulative sum of map & reduce tasks of all Running jobs ◦ map_demand ◦ reduce_demand https://github.com/ashwanthkumar/hadoop-as-publisher
11. Demand vs Supply Metrics for Hadoop
12. Vamana - Configuration
13. After Vamana - Demand vs Supply
14. After Vamana - Cost Reduction Savings ~ $ 30 per day
15. Vamana Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
16. Vamana - Pluggable App Scalar Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
17. Vamana - Pluggable Metric Store Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
18. Vamana - Pluggable Auto scalar Vamana - Collect Demand and Supply metrics from CloudWatch / Custom Metrics System - Use the Scalar Implementation to compute the new desired capacity. - Update the ASG with the new “Desired”. Push Demand and Supply Metrics Get Demand And Supply Metrics Set the computed “Desired” Value
19. Questions? Thank you! https://github.com/ashwanthkumar/vamana2
20. Meta Implementation Details and other notes
21. Metric Collector
22. App Scalar
23. Hadoop Scalar
24. Auto scalar
25. • We’ve lots of datapipelines running on various versions of Hadoop • Each of them have their own usage pattern ◦ A Staging cluster has only workloads for 3-4 hours a day ◦ Production cluster has workloads 24x7 • We started having Scale Up and Scale Down stages in our Datapipelines • Initially it helped but started breaking when ◦ Pipeline fails before completion and the cluster is not scaled down ◦ Every new pipeline created had to have a scale up and scale down stage ◦ More than 1 pipeline started sharing the cluster Hadoop @ Indix