Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
InfluxDB + Prometheus
Search
Paul Dix
August 18, 2017
Programming
2
3.1k
InfluxDB + Prometheus
Slides from my Promcon2017 talk on integrating InfluxDB and Prometheus.
Paul Dix
August 18, 2017
Tweet
Share
More Decks by Paul Dix
See All by Paul Dix
InfluxDB IOx Project Update - 2021-02-10
pauldix
0
200
InfluxDB IOx data lifecycle and object store persistence
pauldix
1
530
InfluxDB 2.0 and Flux
pauldix
1
640
Flux and InfluxDB 2.0
pauldix
1
1.2k
Querying Prometheus with Flux
pauldix
1
760
Flux (#fluxlang): a new (time series) data scripting language
pauldix
7
5k
At Scale, Everything is Hard
pauldix
2
640
IFQL and the future of InfluxData
pauldix
2
1.3k
Time series & monitoring with InfluxDB and the TICK stack
pauldix
0
390
Other Decks in Programming
See All in Programming
Amazon Neptuneで始める初めてのグラフDB ー グラフDBを使う意味を考える ー
satoshi256kbyte
2
260
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
0
120
Debugging: All you need to know (for simultaneous interpreting)
jmatsu
2
800
LangChainの現在とv0.3にむけて
os1ma
4
920
仮想ファイルシステムを導入して開発環境のストレージ課題を解消する
segadevtech
2
550
意外とフォントが大事だった話 / Font Issues on Internationalization
fumi23
0
110
いまから追い上げる、Jetpack Compose トレーニング
nyafunta9858
0
590
Using Livebook to build and deploy internal tools @ ElixirConf 2024
hugobarauna
0
250
状態管理ライブラリZustandの導入から運用まで
k1tikurisu
3
470
Method Swizzlingを行うライブラリにおけるマルチモジュール設計
yoshikma
0
120
o1モデルのプロンプトエンジニアリングって?
ktc_wada
0
260
GoのIteratorに詳しくなってしまう
inatonix
1
200
Featured
See All Featured
Bash Introduction
62gerente
608
210k
Visualization
eitanlees
142
15k
No one is an island. Learnings from fostering a developers community.
thoeni
18
2.9k
Keith and Marios Guide to Fast Websites
keithpitt
408
22k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
29
2.6k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
26
2k
The Language of Interfaces
destraynor
153
23k
Navigating Team Friction
lara
183
13k
Side Projects
sachag
451
42k
Build your cross-platform service in a week with App Engine
jlugia
228
18k
What's new in Ruby 2.0
geeforr
340
31k
Intergalactic Javascript Robots from Outer Space
tanoku
268
26k
Transcript
InfluxDB + Prometheus Paul Dix CTO @ InfluxData @pauldix paul@influxdb.com
Open Source Time Series Database
• OSS - MIT License • Written in Go •
SQL-ish query language • Time Series Merge Tree storage engine & inverted index • Commercial Enterprise - HA + Scale Out Clustering
Data Model Measurement cpu,host=serverA,num=1,region=west idle=1.667,system=2342.2 1492214400000000000 Tags Fields nanosecond epoch
float64, int64, bool, string uint64 in next release
Example Query select percentile(90, value) from cpu where time >
now() - 12h and “region” = ‘west’ group by time(10m), host
Process: • OSS - MIT License • Written in Go
• Process, monitor, alert, act/execute • TICK script • Streaming & Batch • Store data back into InfluxDB • User Defined Functions • Service Discovery & Pull
Collect: • OSS - MIT License • Written in Go
• Agent deployed across infrastructure • Input plugins - system, docker, postgres, mysql, cassandra, elastic, hadoop, redis, nginx, apache, etc. • Output plugins - InfluxDB, Graphite, Kafka, etc.
Why is a competitor at Promcon?
What is Prometheus?
Prometheus Server
Alert Manager
Exposition Format
Client Libraries • Supported • Go • Java • Python
• Ruby • Community • C++ • C# • Node.js • PHP • etc…
PromQL
API /api/v1/series /api/v1/label/<name>/values
Remote Read/Write API
Prometheus is more than just the server
Influx + Prometheus today
We’re embracing pull & push
Prometheus Scrape Targets Kapacitor
https://github.com/prometheus/prometheus/issues/2911 https://github.com/influxdata/prometheus_metric_normalizer
Remote Read/Write
Prometheus Config # Remote write configuration (for Graphite, OpenTSDB, or
InfluxDB). remote_write: - url: "http://localhost:9201/write" # Remote read configuration (for InfluxDB only at the moment). remote_read: - url: "http://localhost:9201/read"
metric -> measurement labels -> tags value -> only field
(float64)
Remote Read Protobuf over HTTP
Reads message Query { int64 start_timestamp_ms = 1; int64 end_timestamp_ms
= 2; repeated LabelMatcher matchers = 3; } enum MatchType { EQUAL = 0; NOT_EQUAL = 1; REGEX_MATCH = 2; REGEX_NO_MATCH = 3; } message LabelMatcher { MatchType type = 1; string name = 2; string value = 3; }
Remote Read JSON over HTTP
Unnecessary Overhead
Can we push down processing?
Influx + Prometheus future
First, pull remote gateway into InfluxDB!
Influx Data Model tags + value + time
Like Prometheus message Sample { double value = 1; int64
timestamp_ms = 2; } message LabelPair { string name = 1; string value = 2; } message TimeSeries { repeated LabelPair labels = 1; // Sorted by time, oldest sample first. repeated Sample samples = 2; }
metric is __name__ label
But we’ll continue to support int64, bool, string, and uint64
IFQL - (Influx Functional Query Language) select(database:"foo",where:{'metric'="requests"}) .range(start:-4h) .window(period:10m) .sum()
.interpolate(start:-4h,every:10m,value:{mean($)})
Decouple QL from Processing from Storage
None
DAG { "operations": [ { "id": "select", "kind": "select", "spec":
{ "database": "mydb" } }, { "id": "range", "kind": "range", "spec": { "start": "-4h", "stop": "now" } }, { "id": "sum", "kind": "sum" } ], "edges": [ { "parent": "select", "child": "range" }, { "parent": "range", "child": "sum" } ] } }
Pushdown Predicates
Opportunities to improve storage/remote?
Bulk Write
Long term storage and query for Prometheus SaaS & On-Premise
Thank you @pauldix paul@influxdb.com