Two different ways to export AWS CloudWatch Metrics

Transcript

1. Two different ways to export AWS CloudWatch Metrics 7th Aug, APJ Open Mic August 2024 - Japan Edition Sales Engineer - Datadog Kento Kimura
2. Self-Introduction • Div : Technical Solutions / Sales Engineering • Role : Pre-sales Engineering w/ knowledge of Cloud Architect • Certs: 12x AWS Certified, 11x Google Cloud Certified, 13x Azure Certified • Awards: 2023 Japan AWS Jr.Champion → AWS Community Builder(2024) 2022-24 JapanAWS All Certifications Engineer Google Cloud Partner Top Engineer 2023-24 Kento Kimura (木村健人) Datadog Japan G.K. History Datacenter Service Manager → Pre-sales of Public Cloud → Pre-sales of Observability tool Community AWS Community Builders / Jagu'e'r(Japan Google Cloud Usergroup for Enterprise) Partner Top Engineer 2023 Partner Top Engineer 2024
3. Agenda 3 Why export CloudWatch metrics? CloudWatch metrics API CloudWatch Metric Streams Summary
4. 4 Why export CloudWatch Metrics?
5. What are metrics? Metrics are data about the performance of your systems. By default, many services provide free metrics for resources . For example: EC2 - CPU Utilization, Disk Read/Write, Network In/Out EBS - Volume Read/Write, BurstBalance Metrics Data: Numerical values with data points → Lightweight data in monitoring. [ 3.7, 17:50:01 ], [ 8.5, 17:50:12 ] 5 “ “
6. Use case for CloudWatch metrics Monitor... • Performance: Response time for each resource associated with the system • Capacity: Excess of each resource associated with the system • Errors: Normalcy of each resource associated with the system Visualize... • Query: Explore data using the query function • Dashboard: Decide on a visualization scheme and review it regularly 6
7. Why export CloudWatch metrics If you want more visibility than CloudWatch can provide... • Monitor metrics of AWS resources with 3rd Party(Partner) monitoring solutions. ◦ Correlate with other Monitoring information ◦ Easily visualize metrics data • Store metrics data longer than 15 months. ◦ Store high resolution data(with no roll up) in S3 as data lake *CloudWatch stores metrics data for 15 months with rollup after 15 days, resolution roll up form 1 min to 1 hour. 7
8. Two different ways 8 CloudWatch Metric Streams CloudWatch metrics API polling GetMetricData Response with data Streaming data CloudWatch has APIs for its backend, so we can get metrics through periodic polling. This metrics export is a pull architecture. CloudWatch launched streaming feature with Amazon Data Firehose in 2022. This metrics export is a push architecture.
9. 9 CloudWatch Metrics API
10. Two different ways - Number One 10 CloudWatch Metric Streams CloudWatch metrics API polling GetMetricData Response with data Streaming data CloudWatch has APIs for its backend, so we can get metrics through periodic polling. This metrics export is a pull architecture. CloudWatch launched streaming feature with Amazon Firehose in 2022. This metrics export is a push architecture.
11. Points of CloudWatch Metrics API 11 CloudWatch metrics API polling Response with data CloudWatch has APIs for its backend, so we can get metrics through periodic polling. This metrics export is a pull architecture. • Need to understand API specifications and implement polling apps/tools. ◦ Using AWS SDK is good to implement • Response data need to be received and converted to the format for apps/tools. • The cost depends on how often you call APIs and how many data points are called. • Difficult to get data in real time. GetMetricData
12. Pros & Cons Pros • Easy setup: partners often have polling capabilities. • Easy cost estimation: only based on CloudWatch Metrics API calling. • Cross region & account: flexible to poll data from multiple region & account. Cons • Latency: 5-10 mins latency to be available via APIs. • Trade-of(real-time, cost): The cost depends on polling frequency. • Using IAM assume role: Understand how IAM works and appropriate permissions. • Quota: 500 metrics/req, 100,800 data points/req If you don't need real-time metrics reporting, exporting metrics may be the cheapest and easiest way to go. 12
13. 13 CloudWatch Metric Streams
14. Two different ways - Number Tow 14 CloudWatch Metric Streams CloudWatch metrics API polling GetMetricData Response with data Streaming data CloudWatch has APIs for its backend, so we can get metrics through periodic polling. This metrics export is a pull architecture. CloudWatch launched streaming feature with Amazon Data Firehose in 2022. This metrics export is a push architecture.
15. Points of CloudWatch Metric Streams 15 CloudWatch Metric Streams Streaming data CloudWatch launched streaming feature with Amazon Data Firehose in 2022. This metrics export is a push architecture. • Need to deploy Metric Streams and Amazon Firehose to stream data. ◦ Destinations are depends on Firehose • Data can be converted by Firehose. • The cost depends on how many data points are called and how much data is streamed. • Destinations must be capable of receiving stream data.
16. Pros & Cons Pros • Scalable without problems: no need to worry about metrics and data points increases • Easily set up and stream: OOTBs for partner services and S3 Cons • Near real time: Not actually real time, 2-3 mins delay. • Regional resources: both Metric Streams and Data Firehose are regional • Not all metrics: cannot stream metrics that report with more than two hours delay. ◦ e.g. Billing estimated charge, S3 bucket size Best way to consider scalability and efficiency, but beware of cost. 16
17. 17 Key takeaways
18. Key takeaways • Both ways are different architectures and have Pros & Cons • Consider based on scalability, efficiency, cost. ◦ Metric Streams are more scalable. ◦ API calls are often cheaper when the collection frequency is 5-10 minutes or more. • Remember this presentation when you want to export CloudWatch Metrics 18
19. Thank you