Time series & monitoring with InfluxDB and the TICK stack

Intro talk to time series data, InfluxDB, and the TICK stack

Transcript

1. Time series & monitoring with InfluxDB & the TICK stack Paul Dix paul@influxdb.com @pauldix
2. Modern engine for metrics & events (time series data)
3. What is time series data?
4. Stock trades and quotes
5. Metrics
6. Analytics
7. Events
8. Sensor data
9. Two kinds of time series data…
10. Regular time series t0 t1 t2 t3 t4 t6 t7 Samples at regular intervals
11. Irregular time series t0 t1 t2 t3 t4 t6 t7 Events whenever they come in
12. Working with Time Series collect, store, visualize, process
13. Products Built on the Open-Source “TICK stack”
14. Store: • OSS - MIT License • Written in Go • SQL-ish query language • Time Series Merge Tree storage engine & inverted index • Retention Policies • Continuous Queries
15. API - 2 endpoints
16. API - 2 endpoints /write?db=mydb&rp=foo Write: HTTP POST
17. API - 2 endpoints /write?db=mydb&rp=foo /query?db=mydb&rp=foo Write: HTTP POST Read: HTTP GET
18. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257 BODY:
19. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257 BODY: Line Protocol
20. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257 BODY: Measurement name
21. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257 BODY: Tags (should be sorted by key, value)
22. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257 BODY: Tags (values are always strings)
23. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=23422.0 1422568543702900257 BODY: Fields
24. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 some_measurement value=23422.0,context=“asdf jkl” 1422568543702900257 BODY: Unlimited Fields
25. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 some_measurement value=23422 1422568543702900257 BODY: Fields (types: int64, float64, string, bool)
26. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=0.0 1422568543702900257 BODY: Fields (floats must have decimal)
27. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=0.0 1422568543702900257 BODY: Fields (types must remain consistent)
28. Writing Data /write?db=mydb&rp=foo cpu_load,host=server01,region=us-west value=0.64 cpu_load,host=server02,region=us-west value=0.55 1422568543702900257 network,direction=in,host=server01,region=us-west value=0.0 1422568543702900257 BODY: Timestamp (nanosecond epoch)
29. Influx CLI
30. Create a database CREATE DATABASE foo
31. Create a retention policy CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT]
32. Create a retention policy CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT] CREATE RETENTION POLICY high_precision ON mydb DURATION 7d REPLICATION 3 DEFAULT
33. Create a retention policy CREATE RETENTION POLICY <rp-name> ON <db-name> DURATION <duration> REPLICATION <n> [DEFAULT] CREATE RETENTION POLICY high_precision ON mydb DURATION 7d REPLICATION 3 DEFAULT Writes will go into this RP unless otherwise specified
34. Discovery
35. Inverted index of measurements and tags
36. Discovery SHOW MEASUREMENTs
37. Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA'
38. Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS
39. Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS SHOW TAG KEYS from CPU
40. Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS SHOW TAG KEYS from CPU SHOW TAG VALUES from CPU WITH KEY = 'region'
41. Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS SHOW TAG KEYS from CPU SHOW TAG VALUES from CPU WITH KEY = 'region' SHOW SERIES
42. Discovery SHOW MEASUREMENTs SHOW MEASUREMENTS where host = 'serverA' SHOW TAG KEYS SHOW TAG KEYS from CPU SHOW TAG VALUES from CPU WITH KEY = 'region' SHOW SERIES SHOW SERIES where service = 'redis'
43. Queries
44. SQL-ish select * from some_series where time > now() - 1h
45. Aggregates select percentile(90, value) from cpu where time > now() - 1d group by time(10m)
46. Aggregates select percentile(90, value) from cpu where time > now() - 1d group by time(10m), region Group by a tag
47. Where against Regex (field) select value from some_log_series where value =~ /.*ERROR.*/ and time > "2014-03-01" and time < "2014-03-03"
48. Where against Regex (tag) select value from some_log_series where host =~ /.*asdf.*/ and time > "2014-03-01" and time < “2014-03-03" group by host
49. Functions min max percentile first last stddev mean count sum median distinct count(distinct) difference moving_average
50. Continuous queries CREATE CONTINUOUS QUERY "10m_event_count" ON mydb BEGIN SELECT count(value) INTO "6_months".events FROM events GROUP BY time(10m) END;
51. Collect: • OSS - MIT License • Written in Go • Agent deployed across infrastructure • Input plugins - system, docker, postgres, mysql, cassandra, elastic, hadoop, redis, nginx, apache, etc. • Output plugins - InfluxDB, Graphite, Kafka, etc.
52. Visualize: • OSS - AGPL License • Written in Go, React, dygraph • UI for administering TICK stack • Ad-hoc exploration and visualization • Create monitoring and alerting rules in Kapacitor • Query builder, TICK script editor, and more
57. Process: • OSS - MIT License • Written in Go • Process, monitor, alert, act/execute • TICK script • Streaming & Batch • Store data back into InfluxDB • User Defined Functions • Service Discovery & Pull
58. Thank you. Paul Dix paul@influxdb.com @pauldix