Prometheusでデータの水平分割を試みる/Let's split prometheus data

Transcript

1. Prometheus で データ分割を試みる Prometheus Meetup Tokyo #3
2. profile: name: Wataru Matsui org: [ zlab.co.jp ] twitter: @watawuwu
3. Agenda • Motivation • Non-goal • How to respond to increase • How to scale out • Configuration • Browse and Aelrts • Issue
4. Motivation
5. Address growing data
6. Non-goal
7. × Highly available ◦ Data redundancy × Long term storage
8. How to respond to increase in memory and storage usage
9. • Reduce data retention • Prolong scrape interval • Reduce unnecessary metrics • Scale up • Scale out • Remote Write/Storage
10. How to scale out without remote storage
11. Prometheus is easy to scale out Pod Prometheus Pod Pod Pod Prometheus
12. Configuration
13. A. Per scrape rule • For popular settings in Kubernetes ◦ Container Metrics(cAdvisor) ◦ Node Metrics ◦ Application Metrics cAdvisor Node Application
14. A. Per scrape rule • Application Metrics can be easily split into multiple scrape rules App A App B - job_name: 'app-xxx' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_app_xxx_scrape] action: keep
15. B. Per metrics(Not Time series) • Same scrape target, but decide whether to scrape by metrics name metric_relabel_configs: - source_labels: [__name__] action: drop regex: 'container_fs' cAdvisor cAdvisor
16. C. Per label hash • Switch targets by label hash - source_labels: [__address__] modulus: ${shard_total} target_label: __tmp_hash action: hashmod - source_labels: [__tmp_hash] regex: ${shard_num} action: keep cAdvisor cAdvisor addr: 10.26.80.18 addr: 10.26.80.19 shard_num: 0 shard_num: 1 shard_total: 2
17. Browse And Alerts
18. A. Aggregate using Remote read API remote_read: - url: http://prometheus-01:9090/api/v1/read read_recent: true - url: http://prometheus-02:9090/api/v1/read read_recent: true - url: http://prometheus-03:9090/api/v1/read read_recent: true
19. B. Aggregate using Thanos Querier Ruler Sidecar Sidecar Sidecar
20. Issue
21. × Not autoscale × Complicated by redundancy × Can't resharding or rebalancing
22. Thanks!