Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Dashboards as a Hobby
Search
Sohei Iwahori
April 24, 2024
Technology
0
860
Building Dashboards as a Hobby
趣味のダッシュボード開発とDashboards as codeまわりのお話
Sohei Iwahori
April 24, 2024
Tweet
Share
More Decks by Sohei Iwahori
See All by Sohei Iwahori
eBPFと周辺技術を利用してPHPアプリケーションコードを変更しない可視化をやってみる
egmc
1
45
PHPアプリケーションにおけるeBPFの使い所
egmc
1
1.3k
Runbookに何を書き、どのようにアラートを振り分けるか?
egmc
2
4k
プロダクション環境の信頼性を損ねず観測する技術
egmc
8
9.6k
practices-for-making-alerts-actionable.pdf
egmc
3
8.1k
Other Decks in Technology
See All in Technology
doda開発 生成AI元年宣言!自家製AIエージェントから始める生産性改革 / doda Development Declaration of the First Year of Generated AI! Productivity Reforms Starting with Home-grown AI Agents
techtekt
0
200
登壇ネタの見つけ方 / How to find talk topics
pinkumohikan
3
250
OAuth/OpenID Connectで実現するMCPのセキュアなアクセス管理
kuralab
5
840
CI/CDとタスク共有で加速するVibe Coding
tnbe21
0
230
(非公式) AWS Summit Japan と 海浜幕張 の歩き方 2025年版
coosuke
PRO
1
330
“社内”だけで完結していた私が、AWS Community Builder になるまで
nagisa53
1
150
UIテスト自動化サポート- Testbed for XCUIAutomation practice
notoroid
0
110
キャディでのApache Iceberg, Trino採用事例 -Apache Iceberg and Trino Usecase in CADDi--
caddi_eng
0
170
Observability infrastructure behind the trillion-messages scale Kafka platform
lycorptech_jp
PRO
0
130
Microsoft Build 2025 技術/製品動向 for Microsoft Startup Tech Community
torumakabe
1
200
In Praise of "Normal" Engineers (LDX3)
charity
2
1.2k
成立するElixirの再束縛(再代入)可という選択
kubell_hr
0
840
Featured
See All Featured
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
233
17k
YesSQL, Process and Tooling at Scale
rocio
173
14k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
161
15k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
A designer walks into a library…
pauljervisheath
206
24k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
228
22k
Building Applications with DynamoDB
mza
95
6.5k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
Git: the NoSQL Database
bkeepers
PRO
430
65k
GitHub's CSS Performance
jonrohan
1031
460k
Transcript
Building Dashboards as a Hobby 2024/04/24 Sohei Iwahori (GREE, Inc.)
who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ
/ Monitoring Unit Leader / Senior Lead Engineer » ओʹήʔϜͷΠϯϑϥͱࢹγεςϜ » σʔλιʔεओʹPrometheus
࠷ۙͷΈ » systemd_exporterʹsystemd_resolved ͷϝτϦΫεΛՃ͢ΔPR͕Ϛʔδ ͞Εͨ(1݄) » ͕৽όʔδϣϯ͕ͳ͔ͳ͔ϦϦʔε ͞Εͳ͍ » ޙͷϑϦ
࠷ۙͷΈ
agenda » μογϡϘʔυͷຽओԽ » झຯͷμογϡϘʔυ࡞Ͱ͏πʔϧ » ࡞ͬͨμογϡϘʔυͷհ » ·ͱΊ
μογϡϘʔυͷຽओԽ
GrafanaͷμογϡϘʔυͷخ͠͞ » ࢥ͍͍ͭͨμογϡϘʔυΛUI্͔Β࡞Ͱ͖Δ » τϥΠΞϯυΤϥʔͷ͞ » RRDtoolͰٯϙʔϥϯυه๏ͰάϥϑΛॻ͔࣌͘Βѹతͳརศੑ
RRDTool -> Grafana + PromQL
झຯͷμογϡϘʔυ࡞Ͱ ͏πʔϧ
μογϡϘʔυ։ൃͷΤίγεςϜ » Grafana as code ৭ʑ͋Δɾɾ1 » Grafana Terraform provider
» Grafana Ansible collection » Grizzly » Grafana Crossplane provider » Kubernetes Grafana Operator » ͦͷ΄͔ScenesͳͲ 1 A complete guide to managing Grafana as code: tools, tips, and tricks Ishan Jain.
ͲΕʹ͢Δ͔ » ਖ਼ͦΜͳʹ͜Ε͕͍͍ʂΈ͍ͨͳͷͳ͍ؾ͕͢ΔͷͰϒϩάͷ௨Γطଘͷ ίʔυࢿ࢈ʹ߹ΘͤͯͰ » τϥΠΞϯυΤϥʔखಈͰɺظతʹ࡞ΓࠐΈ͍ͨ߹ʹίʔυԽ » ͱ͍͑JSONܗࣜͳΒ݁ہϕʔεΛखͰ࡞ΔɺͰ » ݸਓͰ͓खܰʹΔʹखಈ࡞+Ұ෦GrizzlyΛར༻
Grizzly » https://github.com/grafana/grizzly » Grafana։ൃͷcliπʔϧʢgrrίϚϯυʣ » ݱࡏDashbaordཧ͚ͩͰͳ͘σʔλιʔεɺGrafana Cloud prometheus ͷrules/alertsɺSynthetic
MonitoringͷཧʹରԠ » v0.4.1࣌ » جຊͷه๏YAMLɺJSONɺJsonnetʢ+Grafonnetʣ͔ͭ͑Δ
Grizzyͱͷग़ձ͍ » 2020ࠒ0.0.1ͷalpha࣌ͰJsonnet(+Grafonnet)Ͱͷར༻ΨΠυ͕͋Γɺ͍࢝Ίͨ » લड़ͷϦΞϧλΠϜөػೳͳͲ͍উख͕ྑ͔ͬͨͷͰ͜Εͳ͔ͳ͔͍͍ͷͰͱ ࢥ͍ͬͯͨ » جຊGrafonnetͰදݱ͠ɺgrafanaDashboards:: ԼʹμογϡϘʔυΦϒδΣΫτ Λஔͯ͠apply͢ΔͱμογϡϘʔυʹө͞ΕΔ
» grr watchͰϑΝΠϧͷมߋݕͯ͠ଈ࠲ʹremoteͷGrafanaΠϯελϯεʹөͳͲɺ ։ൃମݧ͕Α͔ͬͨ » ͦͯ͠2021-02-27 v0.1.0͕ϦϦʔε͞Εͨ
Grizzy 0.1.0 » ٸʹه๏͕k8sελΠϧͷYAMLʹͳͬͯhidden element͕ඇਪͱͳͬͨɾɾ
Jsonnet + GrafonnetͰҾ͖ଓ͖͑·͢ (0.4.1ݱࡏ) » dashboardఆٛΛspecʹೖΕΔܗͰJsonnetͷ··ͰOK local g = import
'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet'; local dashboard = g.dashboard.new("my dahsboard"); ... { apiVersion: 'grizzly.grafana.com/v1alpha1', kind: 'Dashboard', metadata: { name: "dashboard name", folder: "dashboard folder" }, spec: dashboard }
grr serve(from 0.4.0) » ϩʔΧϧͰGrafanaཱͯͯϦϞʔτͷGrafanaΛ มߋ͢Δ͜ͱͳ͘μογϡϘʔυฤू͕Ͱ͖Δ » ϦΫΤετϓϩΩγ͞ΕΔͷͰϦϞʔτͷ σʔλιʔε͕͑Δ »
UI্Ͱอଘ͢ΔͱϩʔΧϧͷμογϡϘʔυ yaml͕ߋ৽͞ΕΔʢ!?ʣ » ͱΓ͋͑ͣܗΛ࡞ͬͯɺμογϡϘʔυΛखಈ ฤूͭͭ͠όʔδϣϯཧ͢ΔɺͳͲ͕Ͱ͖· ͢ɺศར
GrizzlyͷΠϚΠνͳ » Τϥʔϝοηʔδ͕ෆ » GrafanaͷΫϨσϯγϟϧ͕ෆ͍ͯ͠Δঢ়ଶͰ invalid memory address ͱ͔ੜͬΆ͍Τϥʔ͕ग़ͨΓ »
എܠͱͯ͠࠷ۙ։ൃׂ͕ͱεϐʔσΟʔʹਐΜͰͦ͏ɺ໘༷·ͩ҆ఆ͠ ͯͳ͍ » υΩϡϝϯτ͕؆ૉͰॳखͷ͍ํ͕Α͘Θ͔Βͳ͍ » ຊޠใ૿͍͖͍ͯͨ͠Ͱ͢Ͷ
࡞ͬͨμογϡϘʔυͷ հ
࡞ͬͨμογϡϘʔυͷհ » systemd_resolved » sysload » process dashbaord with treemap
࡞ͬͨμογϡϘʔυͷհ » systemd_resolvedʢ͜Εͱʣ » sysload » process dashbaord with treemapʢ͜ΕͷΛগ͠ʣ
systemd_resolved https://grafana.com/grafana/dashboards/15816-systemd-resolved-exporter/
systemd_resolved » ಄Ͱͨ͠Έͷճऩ » systemtd_resolvedͷϝτϦΫεΛऔΔͨΊʹಠཱͨ͠exporterΛ2022ʹॻ͍ͨ » https://github.com/egmc/systemd_resolved_exporter » ࣌Λܦͯcommunityϝϯςͱͳͬͨsystemd_exporterʹಉ༷ͷ࣮Λߦ͏PRΛ ೖΕͯϚʔδ͞Εͨ
» σϞ༻ʹμογϡϘʔυ࡞͚ͬͨͲ·ͩʢsystemd_exporterͷํͰʣ͑ͳ ͍ɺϦϦʔε͞ΕͨΒྑ͔ͬͨΒͬͯΈ͍ͯͩ͘͞ɾɾ
process dashbaord with treemap https://grafana.com/grafana/dashboards/13882-process-exporter-dashboard-with-treemap/
process dashbaord with treemap » DatadogͷμογϡϘʔυʹ͋ΔϝϞϦϚοϓతͳλΠϧදࣔΛߦ͏ෳ߹άϥϑ͕Α͔ͬͨͷͰਅࣅͯ࡞ͬͨ » process-exporeter2ͷϝτϦΫεΛදࣔ͢Δ » /proc
ҎԼͷใΛͱʹ » ҙʹάϧʔϐϯάͨ͠ϓϩηε͝ͱͷϝϞϦ/cpu/io/uptimeͳͲ » ݸਓతʹಛʹ͍͍ͨͷϝϞϦͷ༻ঢ়گͱͦͷมԽ » ͳ͔ͳ͔ศརʹ͑ͨͷͰࣗࣾͷڥʹ࣋ͪࠐΜͰ͍ͬͯΔ 2 process-exporter(https://github.com/ncabatoff/process-exporter)
ϝϞϦϦʔΫͷՄࢹԽͱ͔
OOMൃੜલͷঢ়گ֬ೝͱ͔
μογϡϘʔυ ͬͱڞ༗͞Εͯ΄͍͠ » ެࣜαΠτͷํগ͠վળ͞Ε͖ͯͨ » ΧςΰϦ·ΘΓ͏ͪΐ͍΄͍͠ » ͱ͍͑ݟ͚ͭΔͷେมͳͷͰڞ༗ɺ ൃ৴͕૿͑Δͱخ͍͠ »
͍͖ͬͯ·͠ΐ͏
·ͱΊ
·ͱΊ » μογϡϘʔυ։ൃָ͍͠ » ͕ɺ͋·Γ։ൃɺڞ༗ࣄྫ͕ྲྀΕͯ͜ͳ͍ʢؾ͕͢ΔʣɺͷͰڞ༗͍͖ͯ͠ ·͠ΐ͏ » πʔϧ৭ʑ͋Δ͕ɺݸਓతʹ·ͣखಈ࡞Ͱ » Grizzlyෳͷ࡞๏Ͱ͑ΔͷͰ͕͠ΒΈ͕ͳ͚Ε͓͢͢ΊͰ͢
Thank you for listening