Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Querying Prometheus with Flux

Paul Dix
August 09, 2018

Querying Prometheus with Flux

Talk given at PromCon 2018 where I introduce Flux (#fluxlang) and show how it can be used to query Prometheus servers.

Paul Dix

August 09, 2018
Tweet

More Decks by Paul Dix

Other Decks in Technology

Transcript

  1. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system")
  2. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Comments
  3. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Functions
  4. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: r => r._measurement == "cpu" and r._field == "usage_system") Pipe forward operator
  5. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Named Arguments
  6. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") String Literal
  7. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Duration Literal (relative time)
  8. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:”2018-08-09T14:00:00Z“) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Time Literal
  9. // get all data from the telegraf db from(db:"telegraf") //

    filter that by the last hour |> range(start:-1h) // filter further by series with a specific measurement and field |> filter(fn: (r) => r._measurement == "cpu" and r._field == "usage_system") Anonymous Function
  10. Operators + == != ( ) - < !~ [

    ] * > =~ { } / <= = , : % >= <- . |>
  11. Types • int • uint • float64 • string •

    duration • time • regex • array • object • function • namespace • table • table stream
  12. fromProm() |> range(start:-1m) |> filter(fn: (r) => r.__name__ == “node_cpu_seconds_total”

    and r.mode == “idle” and r.cpu == “0”) |> keep(columns: [“name”, “cpu”, “host”, “mode”, “_value”, “_time”])
  13. fromProm() |> range(start:-1m) |> filter(fn: (r) => r.__name__ == “node_cpu_seconds_total”

    and r.mode in [“idle”, “user”] and r.cpu == “0”) |> keep(columns: [“name”, “cpu”, “host”, “mode”, “_value”, “_time”])
  14. fromProm() |> range(start:-30s) |> filter(fn: (r) => r.__name__ == “node_cpu_seconds_total”

    and r.mode == “idle” and r.cpu =~ /0|1/) |> count() |> keep(columns: [“name”, “cpu”, “host”, “mode”, “_value”, “_time”])
  15. Nested range vectors fromProm(host:”http://localhost:9090") |> filter(fn: (r) => r.__name__ ==

    "node_disk_written_bytes_total") |> range(start:-1h) // transform into non-negative derivative values |> derivative() // break those out into tables for each 10 minute block of time |> window(every:10m) // get the max rate of change in each 10 minute window |> max() // and put everything back into a single table |> window(every:inf) // and now let’s convert to KB |> map(fn: (r) => r._value / 1024.0)
  16. Multiple Servers dc1 = fromProm(host:”http://prom.dc1.local:9090") |> filter(fn: (r) => r.__name__

    == “node_network_receive_bytes_total”) |> range(start:-1h) |> insertGroupKey(key: “dc”, value: “1”) dc2 = fromProm(host:”http://prom.dc2.local:9090") |> filter(fn: (r) => r.__name__ == “node_network_receive_bytes_total”) |> range(start:-1h) |> insertGroupKey(key: “dc”, value: “2”) dc1 |> union(streams: [dc2]) |> limit(n: 2) |> derivative() |> group(columns: [“dc”]) |> sum()
  17. Work with data from many sources • from() // influx

    • fromProm() • fromMySQL() • fromCSV() • fromS3() • …
  18. Defining Functions windowAgg = (every, fn, <-stream) => { return

    stream |> window(every: every) |> fn() |> window(every:inf) } fromProm(query: `node_cpu_seconds_total{cpu=“0”,mode=“idle”}` |> range(start: -1m) |> windowAgg(every:20s, fn: min)
  19. Packages & Namespaces package “flux-helpers” windowAgg = (every, fn, <-stream)

    => { return stream |> window(every: every) |> fn() |> window(every:inf) } // in a new script import helpers “github.com/pauldix/flux-helpers" fromProm(query: `node_cpu_seconds_total{cpu=“0”,mode=“idle”}` |> range(start: -1m) |> helpers.windowAgg(every:20s, fn: min)
  20. Project Status • Everything in this talk is prototype (as

    of 2018-08-09) • Proposed Final Language Spec • Release flux, fluxd, InfluxDB 1.7, InfluxDB 2.0 alpha • Iterate with community to finalize spec • Optimizations! • https://github.com/influxdata/flux