Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
DevopsdaysIndia 2018: Loki, Prometheus but for ...
Search
Goutham Veeramachaneni
December 09, 2018
Programming
1
2k
DevopsdaysIndia 2018: Loki, Prometheus but for logs
Goutham Veeramachaneni
December 09, 2018
Tweet
Share
More Decks by Goutham Veeramachaneni
See All by Goutham Veeramachaneni
Storage in 2.0
gouthamve
1
110
Other Decks in Programming
See All in Programming
Dev ContainersとGitHub Codespacesの素敵な関係
ymd65536
1
120
Java ジェネリクス入門 2024
nagise
0
600
カスタムしながら理解するGraphQL Connection
yanagii
1
1.2k
Sidekiqで実現する 長時間非同期処理の中断と再開 / Pausing and Resuming Long-Running Asynchronous Jobs with Sidekiq
hypermkt
6
2.6k
アジャイルを支えるテストアーキテクチャ設計/Test Architecting for Agile
goyoki
7
2.7k
Progressive Web Apps für Desktop und Mobile mit Angular (Hands-on)
christianliebel
PRO
0
110
offers_20241022_imakiire.pdf
imakurusu
2
360
Piniaの現状と今後
waka292
5
1.4k
Modern Angular: Renovation for Your Applications
manfredsteyer
PRO
0
200
プロジェクト新規参入者のリードタイム短縮の観点から見る、品質の高いコードとアーキテクチャを保つメリット
d_endo
1
1k
LLM生成文章の精度評価自動化とプロンプトチューニングの効率化について
layerx
PRO
2
130
詳細解説! ArrayListの仕組みと実装
yujisoftware
0
470
Featured
See All Featured
A better future with KSS
kneath
238
17k
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
31
2.7k
How to Ace a Technical Interview
jacobian
275
23k
A designer walks into a library…
pauljervisheath
202
24k
Producing Creativity
orderedlist
PRO
341
39k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
250
21k
StorybookのUI Testing Handbookを読んだ
zakiyama
26
5.2k
Bash Introduction
62gerente
608
210k
The World Runs on Bad Software
bkeepers
PRO
65
11k
Intergalactic Javascript Robots from Outer Space
tanoku
268
27k
Practical Orchestrator
shlominoach
186
10k
Transcript
Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia
2018
putadent gouthamve
QQQ
Outline • Quick Grafana intro • Full observability • Loki
• The future
Grafana intro
Grafana From Dashboarding solution To Observability platform
Unified way to look at data from different sources Logos
of datasources
Full Observability Picture credits: Peter Bourgon
Troubleshooting journey
Instrumenting an app
App • Classic 3-tiered app • Deployed in Kubernetes •
It’s running, but how is it doing? Load balancers App servers DB servers
Add instrumentation • Prometheus for metrics • Loki for logs
• Jaeger for traces
Change app • Introduce a bug • Let’s debug this
with our new tools
Demo: http://localhost:3000/explore
Loki: Grafana logging in detail
Goal: Keeping it simple https://twitter.com/alicegoldfuss/status/981947777256079360
Logs should be cheap! Existing Solutions are hard to scale
and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals
Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}
Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}
Service Discovery for Grafana Logging • Prometheus-style service discovery of
logging targets • Labels are indexed as metadata, e.g.: {job=”app1”} • Relabeling rules
Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging
datasource
Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource
Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier
Loki: Write Path Your jobs Distributor Ingester Index Chunks
Distributor Distributor Ingester Use consistent hashing to assign a timestream
to an ingester.
Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line
Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks
Loki: Read Path Ingester Index Chunks Querier
Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3
Ceph / Minio FILES Distributor Ingester Querier
Loki: Status • Alpha software • Focus on correctness, ease-of-use
first, performance later • Onboard paying customers in Q1 2019 • Beyond Kubernetes
• Cheap but not free :) • Grep + Parsing
in browser • Can’t do complicated queries Loki: Caveats
What we’re working on
Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable,
edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)
Integrate Tracing • Associate traces with logs and metrics •
Labels FTW • Aiming for Q2 2019
Logging UI needs to be refined still behind feature flag,
feedback welcome: @davkals or
[email protected]
UX improvements on logs and metrics views Performance improvements
One last thing...
https://www.grafanacon.org/2019/
Tack for listening Questions to
[email protected]
@putadent