DevopsdaysIndia 2018: Loki, Prometheus but for logs

Transcript

1. Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia 2018
2. putadent gouthamve
3. QQQ
4. Outline • Quick Grafana intro • Full observability • Loki • The future
5. Grafana intro
6. Grafana From Dashboarding solution To Observability platform
7. Unified way to look at data from different sources Logos of datasources
8. Full Observability Picture credits: Peter Bourgon
9. Troubleshooting journey
10. Instrumenting an app
11. App • Classic 3-tiered app • Deployed in Kubernetes • It’s running, but how is it doing? Load balancers App servers DB servers
12. Add instrumentation • Prometheus for metrics • Loki for logs • Jaeger for traces
13. Change app • Introduce a bug • Let’s debug this with our new tools
14. Demo: http://localhost:3000/explore
15. Loki: Grafana logging in detail
16. Goal: Keeping it simple https://twitter.com/alicegoldfuss/status/981947777256079360
17. Logs should be cheap! Existing Solutions are hard to scale and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals
18. Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}
19. Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}
20. Service Discovery for Grafana Logging • Prometheus-style service discovery of logging targets • Labels are indexed as metadata, e.g.: {job=”app1”} • Relabeling rules
21. Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging datasource
22. Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource
23. Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier
24. Loki: Write Path Your jobs Distributor Ingester Index Chunks
25. Distributor Distributor Ingester Use consistent hashing to assign a timestream to an ingester.
26. Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line
27. Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks
28. Loki: Read Path Ingester Index Chunks Querier
29. Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3 Ceph / Minio FILES Distributor Ingester Querier
30. Loki: Status • Alpha software • Focus on correctness, ease-of-use first, performance later • Onboard paying customers in Q1 2019 • Beyond Kubernetes
31. • Cheap but not free :) • Grep + Parsing in browser • Can’t do complicated queries Loki: Caveats
32. What we’re working on
33. Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable, edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)
34. Integrate Tracing • Associate traces with logs and metrics • Labels FTW • Aiming for Q2 2019
35. Logging UI needs to be refined still behind feature flag, feedback welcome: @davkals or david@grafana.com UX improvements on logs and metrics views Performance improvements
36. One last thing...
37. https://www.grafanacon.org/2019/
38. Tack for listening Questions to goutham@grafana.com @putadent