Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Dashboards as a Hobby
Search
Sohei Iwahori
April 24, 2024
Technology
0
880
Building Dashboards as a Hobby
趣味のダッシュボード開発とDashboards as codeまわりのお話
Sohei Iwahori
April 24, 2024
Tweet
Share
More Decks by Sohei Iwahori
See All by Sohei Iwahori
SREのためのeBPF活用ステップアップガイド
egmc
3
2.3k
eBPFと周辺技術を利用してPHPアプリケーションコードを変更しない可視化をやってみる
egmc
1
52
PHPアプリケーションにおけるeBPFの使い所
egmc
1
1.3k
Runbookに何を書き、どのようにアラートを振り分けるか?
egmc
2
4.1k
プロダクション環境の信頼性を損ねず観測する技術
egmc
8
9.6k
practices-for-making-alerts-actionable.pdf
egmc
3
8.2k
Other Decks in Technology
See All in Technology
JavaScript 研修
recruitengineers
PRO
3
180
MySQL HeatWave:サービス概要のご紹介
oracle4engineer
PRO
4
1.7k
AIエージェントの開発に必須な「コンテキスト・エンジニアリング」とは何か──プロンプト・エンジニアリングとの違いを手がかりに考える
masayamoriofficial
0
400
サービスロボット最前線:ugoが挑むPhysical AI活用
kmatsuiugo
0
190
[CVPR2025論文読み会] Linguistics-aware Masked Image Modelingfor Self-supervised Scene Text Recognition
s_aiueo32
0
210
Yahoo!広告ビジネス基盤におけるバックエンド開発
lycorptech_jp
PRO
1
280
kintone開発チームの紹介
cybozuinsideout
PRO
0
73k
DeNA での思い出 / Memories at DeNA
orgachem
PRO
3
1.6k
浸透しなさいRFC 5322&7208
hinono
0
120
新規案件の立ち上げ専門チームから見たAI駆動開発の始め方
shuyakinjo
0
130
Devinを使ったモバイルアプリ開発 / Mobile app development with Devin
yanzm
0
190
実践アプリケーション設計 ②トランザクションスクリプトへの対応
recruitengineers
PRO
3
220
Featured
See All Featured
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.6k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.8k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
23
1.4k
BBQ
matthewcrist
89
9.8k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
20k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Rebuilding a faster, lazier Slack
samanthasiow
83
9.1k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Making Projects Easy
brettharned
117
6.3k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
780
Transcript
Building Dashboards as a Hobby 2024/04/24 Sohei Iwahori (GREE, Inc.)
who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ
/ Monitoring Unit Leader / Senior Lead Engineer » ओʹήʔϜͷΠϯϑϥͱࢹγεςϜ » σʔλιʔεओʹPrometheus
࠷ۙͷΈ » systemd_exporterʹsystemd_resolved ͷϝτϦΫεΛՃ͢ΔPR͕Ϛʔδ ͞Εͨ(1݄) » ͕৽όʔδϣϯ͕ͳ͔ͳ͔ϦϦʔε ͞Εͳ͍ » ޙͷϑϦ
࠷ۙͷΈ
agenda » μογϡϘʔυͷຽओԽ » झຯͷμογϡϘʔυ࡞Ͱ͏πʔϧ » ࡞ͬͨμογϡϘʔυͷհ » ·ͱΊ
μογϡϘʔυͷຽओԽ
GrafanaͷμογϡϘʔυͷخ͠͞ » ࢥ͍͍ͭͨμογϡϘʔυΛUI্͔Β࡞Ͱ͖Δ » τϥΠΞϯυΤϥʔͷ͞ » RRDtoolͰٯϙʔϥϯυه๏ͰάϥϑΛॻ͔࣌͘Βѹతͳརศੑ
RRDTool -> Grafana + PromQL
झຯͷμογϡϘʔυ࡞Ͱ ͏πʔϧ
μογϡϘʔυ։ൃͷΤίγεςϜ » Grafana as code ৭ʑ͋Δɾɾ1 » Grafana Terraform provider
» Grafana Ansible collection » Grizzly » Grafana Crossplane provider » Kubernetes Grafana Operator » ͦͷ΄͔ScenesͳͲ 1 A complete guide to managing Grafana as code: tools, tips, and tricks Ishan Jain.
ͲΕʹ͢Δ͔ » ਖ਼ͦΜͳʹ͜Ε͕͍͍ʂΈ͍ͨͳͷͳ͍ؾ͕͢ΔͷͰϒϩάͷ௨Γطଘͷ ίʔυࢿ࢈ʹ߹ΘͤͯͰ » τϥΠΞϯυΤϥʔखಈͰɺظతʹ࡞ΓࠐΈ͍ͨ߹ʹίʔυԽ » ͱ͍͑JSONܗࣜͳΒ݁ہϕʔεΛखͰ࡞ΔɺͰ » ݸਓͰ͓खܰʹΔʹखಈ࡞+Ұ෦GrizzlyΛར༻
Grizzly » https://github.com/grafana/grizzly » Grafana։ൃͷcliπʔϧʢgrrίϚϯυʣ » ݱࡏDashbaordཧ͚ͩͰͳ͘σʔλιʔεɺGrafana Cloud prometheus ͷrules/alertsɺSynthetic
MonitoringͷཧʹରԠ » v0.4.1࣌ » جຊͷه๏YAMLɺJSONɺJsonnetʢ+Grafonnetʣ͔ͭ͑Δ
Grizzyͱͷग़ձ͍ » 2020ࠒ0.0.1ͷalpha࣌ͰJsonnet(+Grafonnet)Ͱͷར༻ΨΠυ͕͋Γɺ͍࢝Ίͨ » લड़ͷϦΞϧλΠϜөػೳͳͲ͍উख͕ྑ͔ͬͨͷͰ͜Εͳ͔ͳ͔͍͍ͷͰͱ ࢥ͍ͬͯͨ » جຊGrafonnetͰදݱ͠ɺgrafanaDashboards:: ԼʹμογϡϘʔυΦϒδΣΫτ Λஔͯ͠apply͢ΔͱμογϡϘʔυʹө͞ΕΔ
» grr watchͰϑΝΠϧͷมߋݕͯ͠ଈ࠲ʹremoteͷGrafanaΠϯελϯεʹөͳͲɺ ։ൃମݧ͕Α͔ͬͨ » ͦͯ͠2021-02-27 v0.1.0͕ϦϦʔε͞Εͨ
Grizzy 0.1.0 » ٸʹه๏͕k8sελΠϧͷYAMLʹͳͬͯhidden element͕ඇਪͱͳͬͨɾɾ
Jsonnet + GrafonnetͰҾ͖ଓ͖͑·͢ (0.4.1ݱࡏ) » dashboardఆٛΛspecʹೖΕΔܗͰJsonnetͷ··ͰOK local g = import
'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet'; local dashboard = g.dashboard.new("my dahsboard"); ... { apiVersion: 'grizzly.grafana.com/v1alpha1', kind: 'Dashboard', metadata: { name: "dashboard name", folder: "dashboard folder" }, spec: dashboard }
grr serve(from 0.4.0) » ϩʔΧϧͰGrafanaཱͯͯϦϞʔτͷGrafanaΛ มߋ͢Δ͜ͱͳ͘μογϡϘʔυฤू͕Ͱ͖Δ » ϦΫΤετϓϩΩγ͞ΕΔͷͰϦϞʔτͷ σʔλιʔε͕͑Δ »
UI্Ͱอଘ͢ΔͱϩʔΧϧͷμογϡϘʔυ yaml͕ߋ৽͞ΕΔʢ!?ʣ » ͱΓ͋͑ͣܗΛ࡞ͬͯɺμογϡϘʔυΛखಈ ฤूͭͭ͠όʔδϣϯཧ͢ΔɺͳͲ͕Ͱ͖· ͢ɺศར
GrizzlyͷΠϚΠνͳ » Τϥʔϝοηʔδ͕ෆ » GrafanaͷΫϨσϯγϟϧ͕ෆ͍ͯ͠Δঢ়ଶͰ invalid memory address ͱ͔ੜͬΆ͍Τϥʔ͕ग़ͨΓ »
എܠͱͯ͠࠷ۙ։ൃׂ͕ͱεϐʔσΟʔʹਐΜͰͦ͏ɺ໘༷·ͩ҆ఆ͠ ͯͳ͍ » υΩϡϝϯτ͕؆ૉͰॳखͷ͍ํ͕Α͘Θ͔Βͳ͍ » ຊޠใ૿͍͖͍ͯͨ͠Ͱ͢Ͷ
࡞ͬͨμογϡϘʔυͷ հ
࡞ͬͨμογϡϘʔυͷհ » systemd_resolved » sysload » process dashbaord with treemap
࡞ͬͨμογϡϘʔυͷհ » systemd_resolvedʢ͜Εͱʣ » sysload » process dashbaord with treemapʢ͜ΕͷΛগ͠ʣ
systemd_resolved https://grafana.com/grafana/dashboards/15816-systemd-resolved-exporter/
systemd_resolved » ಄Ͱͨ͠Έͷճऩ » systemtd_resolvedͷϝτϦΫεΛऔΔͨΊʹಠཱͨ͠exporterΛ2022ʹॻ͍ͨ » https://github.com/egmc/systemd_resolved_exporter » ࣌Λܦͯcommunityϝϯςͱͳͬͨsystemd_exporterʹಉ༷ͷ࣮Λߦ͏PRΛ ೖΕͯϚʔδ͞Εͨ
» σϞ༻ʹμογϡϘʔυ࡞͚ͬͨͲ·ͩʢsystemd_exporterͷํͰʣ͑ͳ ͍ɺϦϦʔε͞ΕͨΒྑ͔ͬͨΒͬͯΈ͍ͯͩ͘͞ɾɾ
process dashbaord with treemap https://grafana.com/grafana/dashboards/13882-process-exporter-dashboard-with-treemap/
process dashbaord with treemap » DatadogͷμογϡϘʔυʹ͋ΔϝϞϦϚοϓతͳλΠϧදࣔΛߦ͏ෳ߹άϥϑ͕Α͔ͬͨͷͰਅࣅͯ࡞ͬͨ » process-exporeter2ͷϝτϦΫεΛදࣔ͢Δ » /proc
ҎԼͷใΛͱʹ » ҙʹάϧʔϐϯάͨ͠ϓϩηε͝ͱͷϝϞϦ/cpu/io/uptimeͳͲ » ݸਓతʹಛʹ͍͍ͨͷϝϞϦͷ༻ঢ়گͱͦͷมԽ » ͳ͔ͳ͔ศརʹ͑ͨͷͰࣗࣾͷڥʹ࣋ͪࠐΜͰ͍ͬͯΔ 2 process-exporter(https://github.com/ncabatoff/process-exporter)
ϝϞϦϦʔΫͷՄࢹԽͱ͔
OOMൃੜલͷঢ়گ֬ೝͱ͔
μογϡϘʔυ ͬͱڞ༗͞Εͯ΄͍͠ » ެࣜαΠτͷํগ͠վળ͞Ε͖ͯͨ » ΧςΰϦ·ΘΓ͏ͪΐ͍΄͍͠ » ͱ͍͑ݟ͚ͭΔͷେมͳͷͰڞ༗ɺ ൃ৴͕૿͑Δͱخ͍͠ »
͍͖ͬͯ·͠ΐ͏
·ͱΊ
·ͱΊ » μογϡϘʔυ։ൃָ͍͠ » ͕ɺ͋·Γ։ൃɺڞ༗ࣄྫ͕ྲྀΕͯ͜ͳ͍ʢؾ͕͢ΔʣɺͷͰڞ༗͍͖ͯ͠ ·͠ΐ͏ » πʔϧ৭ʑ͋Δ͕ɺݸਓతʹ·ͣखಈ࡞Ͱ » Grizzlyෳͷ࡞๏Ͱ͑ΔͷͰ͕͠ΒΈ͕ͳ͚Ε͓͢͢ΊͰ͢
Thank you for listening