Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
DevopsdaysIndia 2018: Loki, Prometheus but for ...
Search
Goutham Veeramachaneni
December 09, 2018
Programming
1
2.1k
DevopsdaysIndia 2018: Loki, Prometheus but for logs
Goutham Veeramachaneni
December 09, 2018
Tweet
Share
More Decks by Goutham Veeramachaneni
See All by Goutham Veeramachaneni
Storage in 2.0
gouthamve
1
120
Other Decks in Programming
See All in Programming
ワンバイナリWebサービスのススメ
mackee
10
7.5k
try-catchを使わないエラーハンドリング!? PHPでResult型の考え方を取り入れてみよう
kajitack
3
340
ts-morph実践:型を利用するcodemodのテクニック
ypresto
1
540
クラシルリワードにおける iOSアプリ開発の取り組み
funzin
1
810
CRUD から CQRS へ ~ 分離が可能にする柔軟性
tkawae
0
230
抽象データ型について学んだ
ryounasso
0
210
ワイがおすすめする新潟の食 / 20250530phpconf-niigata-eve
kasacchiful
0
260
AIにコードを生成するコードを作らせて、再現性を担保しよう! / Let AI generate code to ensure reproducibility
yamachu
7
6.1k
バリデーションライブラリ徹底比較
nayuta999999
1
440
複数アプリケーションを育てていくための共通化戦略
irof
3
750
primeNumberでのRBS導入の現在 && RBS::Traceでinline RBSを拡充してみた
mnmandahalf
0
260
Javaのルールをねじ曲げろ!禁断の操作とその代償から学ぶメタプログラミング入門 / A Guide to Metaprogramming: Lessons from Forbidden Techniques and Their Price
nrslib
1
290
Featured
See All Featured
Build your cross-platform service in a week with App Engine
jlugia
231
18k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.5k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Making the Leap to Tech Lead
cromwellryan
134
9.3k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.3k
Visualization
eitanlees
146
16k
GitHub's CSS Performance
jonrohan
1031
460k
Why You Should Never Use an ORM
jnunemaker
PRO
56
9.4k
Building Better People: How to give real-time feedback that sticks.
wjessup
368
19k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
123
52k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Rails Girls Zürich Keynote
gr2m
94
13k
Transcript
Loki Like Prometheus, but for logs Goutham Veeramachaneni @gouthamve DevOpsDaysIndia
2018
putadent gouthamve
QQQ
Outline • Quick Grafana intro • Full observability • Loki
• The future
Grafana intro
Grafana From Dashboarding solution To Observability platform
Unified way to look at data from different sources Logos
of datasources
Full Observability Picture credits: Peter Bourgon
Troubleshooting journey
Instrumenting an app
App • Classic 3-tiered app • Deployed in Kubernetes •
It’s running, but how is it doing? Load balancers App servers DB servers
Add instrumentation • Prometheus for metrics • Loki for logs
• Jaeger for traces
Change app • Introduce a bug • Let’s debug this
with our new tools
Demo: http://localhost:3000/explore
Loki: Grafana logging in detail
Goal: Keeping it simple https://twitter.com/alicegoldfuss/status/981947777256079360
Logs should be cheap! Existing Solutions are hard to scale
and operate You don’t need full text search / indexing Metrics, Logs and Traces need to work together More Goals
Logging for Kubernetes {job=”app1”} {job=”app3”} {job=”app2”}
Logging for Kubernetes (2) {job=”app1”} {job=”app3”} {job=”app2”}
Service Discovery for Grafana Logging • Prometheus-style service discovery of
logging targets • Labels are indexed as metadata, e.g.: {job=”app1”} • Relabeling rules
Logging architecture {job=”app1”} {job=”app2”} Node Logging agent Logging service Logging
datasource
Logging architecture {job=”app1”} {job=”app2”} Node Promtail Loki Logging datasource
Loki: Architecture Your jobs Distributor Ingester Index Chunks Querier
Loki: Write Path Your jobs Distributor Ingester Index Chunks
Distributor Distributor Ingester Use consistent hashing to assign a timestream
to an ingester.
Loki: Write Path Ingester {job=”app1”, instance=”...”,..} {job=”app2”, instance=”...”,..} Log line
Loki: Write Path Ingester {job=”app2”, instance=”...”,..} Index Chunks
Loki: Read Path Ingester Index Chunks Querier
Loki: Scalability Index Chunks Bigtable DynamoDB Cassandra BOLTDB GCS S3
Ceph / Minio FILES Distributor Ingester Querier
Loki: Status • Alpha software • Focus on correctness, ease-of-use
first, performance later • Onboard paying customers in Q1 2019 • Beyond Kubernetes
• Cheap but not free :) • Grep + Parsing
in browser • Can’t do complicated queries Loki: Caveats
What we’re working on
Enable Explore UI (BETA: Prometheus) Behind feature flag. To enable,
edit Grafana config ini file [explore] enabled = true Set up a datasource that supports Explore, e.g., Prometheus. Will be released in 6.0 (Feb 2019)
Integrate Tracing • Associate traces with logs and metrics •
Labels FTW • Aiming for Q2 2019
Logging UI needs to be refined still behind feature flag,
feedback welcome: @davkals or
[email protected]
UX improvements on logs and metrics views Performance improvements
One last thing...
https://www.grafanacon.org/2019/
Tack for listening Questions to
[email protected]
@putadent