Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
InfluxDB + Prometheus
Search
Paul Dix
August 18, 2017
Programming
2
3.1k
InfluxDB + Prometheus
Slides from my Promcon2017 talk on integrating InfluxDB and Prometheus.
Paul Dix
August 18, 2017
Tweet
Share
More Decks by Paul Dix
See All by Paul Dix
InfluxDB IOx Project Update - 2021-02-10
pauldix
0
200
InfluxDB IOx data lifecycle and object store persistence
pauldix
1
560
InfluxDB 2.0 and Flux
pauldix
1
670
Flux and InfluxDB 2.0
pauldix
1
1.3k
Querying Prometheus with Flux
pauldix
1
790
Flux (#fluxlang): a new (time series) data scripting language
pauldix
7
5k
At Scale, Everything is Hard
pauldix
2
650
IFQL and the future of InfluxData
pauldix
2
1.3k
Time series & monitoring with InfluxDB and the TICK stack
pauldix
0
410
Other Decks in Programming
See All in Programming
macOS でできる リアルタイム動画像処理
biacco42
9
2.4k
OnlineTestConf: Test Automation Friend or Foe
maaretp
0
110
광고 소재 심사 과정에 AI를 도입하여 광고 서비스 생산성 향상시키기
kakao
PRO
0
170
「今のプロジェクトいろいろ大変なんですよ、app/services とかもあって……」/After Kaigi on Rails 2024 LT Night
junk0612
5
2.2k
タクシーアプリ『GO』のリアルタイムデータ分析基盤における機械学習サービスの活用
mot_techtalk
4
1.5k
イベント駆動で成長して委員会
happymana
1
330
Jakarta EE meets AI
ivargrimstad
0
600
リアーキテクチャxDDD 1年間の取り組みと進化
hsawaji
1
220
受け取る人から提供する人になるということ
little_rubyist
0
250
アジャイルを支えるテストアーキテクチャ設計/Test Architecting for Agile
goyoki
9
3.3k
Streams APIとTCPフロー制御 / Web Streams API and TCP flow control
tasshi
2
350
Enabling DevOps and Team Topologies Through Architecture: Architecting for Fast Flow
cer
PRO
0
340
Featured
See All Featured
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
506
140k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
Art, The Web, and Tiny UX
lynnandtonic
297
20k
The Cult of Friendly URLs
andyhume
78
6k
Building an army of robots
kneath
302
43k
Practical Orchestrator
shlominoach
186
10k
Music & Morning Musume
bryan
46
6.2k
Become a Pro
speakerdeck
PRO
25
5k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
232
17k
Raft: Consensus for Rubyists
vanstee
136
6.6k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
44
2.2k
Transcript
InfluxDB + Prometheus Paul Dix CTO @ InfluxData @pauldix paul@influxdb.com
Open Source Time Series Database
• OSS - MIT License • Written in Go •
SQL-ish query language • Time Series Merge Tree storage engine & inverted index • Commercial Enterprise - HA + Scale Out Clustering
Data Model Measurement cpu,host=serverA,num=1,region=west idle=1.667,system=2342.2 1492214400000000000 Tags Fields nanosecond epoch
float64, int64, bool, string uint64 in next release
Example Query select percentile(90, value) from cpu where time >
now() - 12h and “region” = ‘west’ group by time(10m), host
Process: • OSS - MIT License • Written in Go
• Process, monitor, alert, act/execute • TICK script • Streaming & Batch • Store data back into InfluxDB • User Defined Functions • Service Discovery & Pull
Collect: • OSS - MIT License • Written in Go
• Agent deployed across infrastructure • Input plugins - system, docker, postgres, mysql, cassandra, elastic, hadoop, redis, nginx, apache, etc. • Output plugins - InfluxDB, Graphite, Kafka, etc.
Why is a competitor at Promcon?
What is Prometheus?
Prometheus Server
Alert Manager
Exposition Format
Client Libraries • Supported • Go • Java • Python
• Ruby • Community • C++ • C# • Node.js • PHP • etc…
PromQL
API /api/v1/series /api/v1/label/<name>/values
Remote Read/Write API
Prometheus is more than just the server
Influx + Prometheus today
We’re embracing pull & push
Prometheus Scrape Targets Kapacitor
https://github.com/prometheus/prometheus/issues/2911 https://github.com/influxdata/prometheus_metric_normalizer
Remote Read/Write
Prometheus Config # Remote write configuration (for Graphite, OpenTSDB, or
InfluxDB). remote_write: - url: "http://localhost:9201/write" # Remote read configuration (for InfluxDB only at the moment). remote_read: - url: "http://localhost:9201/read"
metric -> measurement labels -> tags value -> only field
(float64)
Remote Read Protobuf over HTTP
Reads message Query { int64 start_timestamp_ms = 1; int64 end_timestamp_ms
= 2; repeated LabelMatcher matchers = 3; } enum MatchType { EQUAL = 0; NOT_EQUAL = 1; REGEX_MATCH = 2; REGEX_NO_MATCH = 3; } message LabelMatcher { MatchType type = 1; string name = 2; string value = 3; }
Remote Read JSON over HTTP
Unnecessary Overhead
Can we push down processing?
Influx + Prometheus future
First, pull remote gateway into InfluxDB!
Influx Data Model tags + value + time
Like Prometheus message Sample { double value = 1; int64
timestamp_ms = 2; } message LabelPair { string name = 1; string value = 2; } message TimeSeries { repeated LabelPair labels = 1; // Sorted by time, oldest sample first. repeated Sample samples = 2; }
metric is __name__ label
But we’ll continue to support int64, bool, string, and uint64
IFQL - (Influx Functional Query Language) select(database:"foo",where:{'metric'="requests"}) .range(start:-4h) .window(period:10m) .sum()
.interpolate(start:-4h,every:10m,value:{mean($)})
Decouple QL from Processing from Storage
None
DAG { "operations": [ { "id": "select", "kind": "select", "spec":
{ "database": "mydb" } }, { "id": "range", "kind": "range", "spec": { "start": "-4h", "stop": "now" } }, { "id": "sum", "kind": "sum" } ], "edges": [ { "parent": "select", "child": "range" }, { "parent": "range", "child": "sum" } ] } }
Pushdown Predicates
Opportunities to improve storage/remote?
Bulk Write
Long term storage and query for Prometheus SaaS & On-Premise
Thank you @pauldix paul@influxdb.com