Mixing metrics and logs with Grafana + Influx

Grafana’s new Explore area is adding support for both metric and logs display for the Influx datasource. This allows you to quickly access your metrics, and as part of troubleshooting, bring up related logs. We’ll also look at the latest support for Flux inside Grafana.

Transcript

1. Mixing metrics and logs with Grafana + Influx David Kaltschmidt @davkals InfluxDays June 2019
2. I’m David Working on Explore, Prometheus, and Loki at Grafana Labs Previously: Unifying Metrics/Logs/Traces at Kausal, Weavescope david@grafana.com Twitter: @davkals
3. Grafana Dashboarding solution Observability platform
4. Unified way to look at data from different sources Logos of datasources
5. Feb 2019: Major release Grafana 6 with Explore and better panel editing
6. Latest release: Grafana 6.2 with Bar Gauge, and lazy loading
7. Grafana + Influx
8. Dashboard with Influx <= 1.7 - Graphical query builder - InfluxQL - Database authentication via username/password
9. New: Explore with Influx <= 1.7 - New section for ad-hoc queries - Same query builder
10. New: Explore with Influx <= 1.7 - Table view to look at raw values
11. Log aggregation with Influx
12. Troubleshooting journey
13. Log aggregation with Influx - Central place to store your application logs - Revisit logs after apps/services/hosts have crashed - Basic analytics support with query language - Influx is not designed for logs, but may be “good enough” - Running a second system for logs adds operational complexity
14. host1 InfluxDB host2 - Measurement: “logs” - Fields: - “message” - “details” - Tags: - “datacenter” - “host” - “app” Modelling log streams app1 app2 db1 db2 app1 dc1
15. Getting logs into Influx (1/3): HTTP API and client libraries
16. Getting logs into Influx (2/3): Telegraf logs input plugin
17. Getting logs into Influx (3/3): Fluentd https://docs.fluentd.org/how-to-guides/syslog-influxdb
18. New: Log viewer - New view in Explore (master, and soon 6.3) - Works with Influx 1.7 - Tag-based filtering for log streams - Will work with v2 shortly
19. Log viewer: Ad-hoc value distributions, log levels, deduplication
20. Explore: Split view to look at metrics and logs side-by-side
21. Demo
22. Grafana + Influx 2.0 and Flux
23. Flux datasource for Influx >= 2.0 - External datasource (for now), latest version: 5.3.1 - Flux expression editor - Supports token authentication (Influx v2 only) - Install via Grafana CLI: grafana-cli plugins install grafana-influxdb-flux-datasource Repo for filing issues: https://github.com/grafana/influxdb-flux-datasource
24. Try it out (1/3) $> docker run --name influxdb -p 9999:9999 quay.io/influxdb/influxdb:2.0.0-alpha
25. Try it out (2/3) $> grafana-cli plugins install grafana-influxdb-flux-datasource $> export INFLUX_TOKEN=<TOKEN> $> telegraf --config http://localhost:9999/api/v2/telegrafs/<CONFIG_ID>
26. Try it out (3/3)
27. Outlook - Unifying Flux and Influx datasources - Adding version drop-down to datasource config page - Explore and panels render editors based on version - Datasource uses appropriate API paths and authentication mechanism - Logs in Flux datasource - Improve tag filtering UX
28. Thank you. UX feedback to david@grafana.com @davkals