Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Building Dashboards as a Hobby
Search
Sohei Iwahori
April 24, 2024
Technology
0
370
Building Dashboards as a Hobby
趣味のダッシュボード開発とDashboards as codeまわりのお話
Sohei Iwahori
April 24, 2024
Tweet
Share
More Decks by Sohei Iwahori
See All by Sohei Iwahori
Runbookに何を書き、どのようにアラートを振り分けるか?
egmc
2
2.4k
プロダクション環境の信頼性を損ねず観測する技術
egmc
8
9.2k
practices-for-making-alerts-actionable.pdf
egmc
3
7.5k
Other Decks in Technology
See All in Technology
データベース03: 関係データモデル
trycycle
0
100
MLOpsの「壁」を乗り越える、LINEヤフーの Data Quality as Code
lycorptech_jp
PRO
8
630
LayerXにおけるLLMプロダクト開発の今までとこれから
layerx
PRO
4
740
一生覚えておきたい「システム開発=コミュニケーション」〜初めての実務案件振り返りLT〜
maimyyym
2
330
ルーターでプレゼンする
puhitaku
1
3.3k
Rustで「プリズモイダル法」を利用して「土量計算」をガチでやる
nokonoko1203
1
310
中年男性がメインフレームから クラウドへキャリアシフトしてみた
uechishingo
0
210
IaCジェネレーターとBedrockで詳細設計書を生成してみた
tsukasa_ishimaru
4
890
Gradle Build Scanを使ってビルドのことを知ろう potatotips #87
tomorrowkey
2
160
2024春 注目のWeb系 OSS & SaaS 3選
makies
0
180
[新卒向け研修資料] テスト文字列に「うんこ」と入れるな(2024年版)
infiniteloop_inc
5
18k
Além do else! Categorizando Pokemóns com Pattern Matching no JavaScript
wmsbill
0
710
Featured
See All Featured
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
245
20k
How STYLIGHT went responsive
nonsquared
92
4.8k
No one is an island. Learnings from fostering a developers community.
thoeni
16
2.1k
ParisWeb 2013: Learning to Love: Crash Course in Emotional UX Design
dotmariusz
104
6.6k
Fireside Chat
paigeccino
22
2.6k
Agile that works and the tools we love
rasmusluckow
325
20k
The Language of Interfaces
destraynor
151
23k
What’s in a name? Adding method to the madness
productmarketing
PRO
17
2.7k
Music & Morning Musume
bryan
41
5.6k
Pencils Down: Stop Designing & Start Developing
hursman
117
11k
Being A Developer After 40
akosma
66
580k
VelocityConf: Rendering Performance Case Studies
addyosmani
321
23k
Transcript
Building Dashboards as a Hobby 2024/04/24 Sohei Iwahori (GREE, Inc.)
who? » Sohei Iwahori (@egmc) » GREE, Inc. » Πϯϑϥ
/ Monitoring Unit Leader / Senior Lead Engineer » ओʹήʔϜͷΠϯϑϥͱࢹγεςϜ » σʔλιʔεओʹPrometheus
࠷ۙͷΈ » systemd_exporterʹsystemd_resolved ͷϝτϦΫεΛՃ͢ΔPR͕Ϛʔδ ͞Εͨ(1݄) » ͕৽όʔδϣϯ͕ͳ͔ͳ͔ϦϦʔε ͞Εͳ͍ » ޙͷϑϦ
࠷ۙͷΈ
agenda » μογϡϘʔυͷຽओԽ » झຯͷμογϡϘʔυ࡞Ͱ͏πʔϧ » ࡞ͬͨμογϡϘʔυͷհ » ·ͱΊ
μογϡϘʔυͷຽओԽ
GrafanaͷμογϡϘʔυͷخ͠͞ » ࢥ͍͍ͭͨμογϡϘʔυΛUI্͔Β࡞Ͱ͖Δ » τϥΠΞϯυΤϥʔͷ͞ » RRDtoolͰٯϙʔϥϯυه๏ͰάϥϑΛॻ͔࣌͘Βѹతͳརศੑ
RRDTool -> Grafana + PromQL
झຯͷμογϡϘʔυ࡞Ͱ ͏πʔϧ
μογϡϘʔυ։ൃͷΤίγεςϜ » Grafana as code ৭ʑ͋Δɾɾ1 » Grafana Terraform provider
» Grafana Ansible collection » Grizzly » Grafana Crossplane provider » Kubernetes Grafana Operator » ͦͷ΄͔ScenesͳͲ 1 A complete guide to managing Grafana as code: tools, tips, and tricks Ishan Jain.
ͲΕʹ͢Δ͔ » ਖ਼ͦΜͳʹ͜Ε͕͍͍ʂΈ͍ͨͳͷͳ͍ؾ͕͢ΔͷͰϒϩάͷ௨Γطଘͷ ίʔυࢿ࢈ʹ߹ΘͤͯͰ » τϥΠΞϯυΤϥʔखಈͰɺظతʹ࡞ΓࠐΈ͍ͨ߹ʹίʔυԽ » ͱ͍͑JSONܗࣜͳΒ݁ہϕʔεΛखͰ࡞ΔɺͰ » ݸਓͰ͓खܰʹΔʹखಈ࡞+Ұ෦GrizzlyΛར༻
Grizzly » https://github.com/grafana/grizzly » Grafana։ൃͷcliπʔϧʢgrrίϚϯυʣ » ݱࡏDashbaordཧ͚ͩͰͳ͘σʔλιʔεɺGrafana Cloud prometheus ͷrules/alertsɺSynthetic
MonitoringͷཧʹରԠ » v0.4.1࣌ » جຊͷه๏YAMLɺJSONɺJsonnetʢ+Grafonnetʣ͔ͭ͑Δ
Grizzyͱͷग़ձ͍ » 2020ࠒ0.0.1ͷalpha࣌ͰJsonnet(+Grafonnet)Ͱͷར༻ΨΠυ͕͋Γɺ͍࢝Ίͨ » લड़ͷϦΞϧλΠϜөػೳͳͲ͍উख͕ྑ͔ͬͨͷͰ͜Εͳ͔ͳ͔͍͍ͷͰͱ ࢥ͍ͬͯͨ » جຊGrafonnetͰදݱ͠ɺgrafanaDashboards:: ԼʹμογϡϘʔυΦϒδΣΫτ Λஔͯ͠apply͢ΔͱμογϡϘʔυʹө͞ΕΔ
» grr watchͰϑΝΠϧͷมߋݕͯ͠ଈ࠲ʹremoteͷGrafanaΠϯελϯεʹөͳͲɺ ։ൃମݧ͕Α͔ͬͨ » ͦͯ͠2021-02-27 v0.1.0͕ϦϦʔε͞Εͨ
Grizzy 0.1.0 » ٸʹه๏͕k8sελΠϧͷYAMLʹͳͬͯhidden element͕ඇਪͱͳͬͨɾɾ
Jsonnet + GrafonnetͰҾ͖ଓ͖͑·͢ (0.4.1ݱࡏ) » dashboardఆٛΛspecʹೖΕΔܗͰJsonnetͷ··ͰOK local g = import
'github.com/grafana/grafonnet/gen/grafonnet-latest/main.libsonnet'; local dashboard = g.dashboard.new("my dahsboard"); ... { apiVersion: 'grizzly.grafana.com/v1alpha1', kind: 'Dashboard', metadata: { name: "dashboard name", folder: "dashboard folder" }, spec: dashboard }
grr serve(from 0.4.0) » ϩʔΧϧͰGrafanaཱͯͯϦϞʔτͷGrafanaΛ มߋ͢Δ͜ͱͳ͘μογϡϘʔυฤू͕Ͱ͖Δ » ϦΫΤετϓϩΩγ͞ΕΔͷͰϦϞʔτͷ σʔλιʔε͕͑Δ »
UI্Ͱอଘ͢ΔͱϩʔΧϧͷμογϡϘʔυ yaml͕ߋ৽͞ΕΔʢ!?ʣ » ͱΓ͋͑ͣܗΛ࡞ͬͯɺμογϡϘʔυΛखಈ ฤूͭͭ͠όʔδϣϯཧ͢ΔɺͳͲ͕Ͱ͖· ͢ɺศར
GrizzlyͷΠϚΠνͳ » Τϥʔϝοηʔδ͕ෆ » GrafanaͷΫϨσϯγϟϧ͕ෆ͍ͯ͠Δঢ়ଶͰ invalid memory address ͱ͔ੜͬΆ͍Τϥʔ͕ग़ͨΓ »
എܠͱͯ͠࠷ۙ։ൃׂ͕ͱεϐʔσΟʔʹਐΜͰͦ͏ɺ໘༷·ͩ҆ఆ͠ ͯͳ͍ » υΩϡϝϯτ͕؆ૉͰॳखͷ͍ํ͕Α͘Θ͔Βͳ͍ » ຊޠใ૿͍͖͍ͯͨ͠Ͱ͢Ͷ
࡞ͬͨμογϡϘʔυͷ հ
࡞ͬͨμογϡϘʔυͷհ » systemd_resolved » sysload » process dashbaord with treemap
࡞ͬͨμογϡϘʔυͷհ » systemd_resolvedʢ͜Εͱʣ » sysload » process dashbaord with treemapʢ͜ΕͷΛগ͠ʣ
systemd_resolved https://grafana.com/grafana/dashboards/15816-systemd-resolved-exporter/
systemd_resolved » ಄Ͱͨ͠Έͷճऩ » systemtd_resolvedͷϝτϦΫεΛऔΔͨΊʹಠཱͨ͠exporterΛ2022ʹॻ͍ͨ » https://github.com/egmc/systemd_resolved_exporter » ࣌Λܦͯcommunityϝϯςͱͳͬͨsystemd_exporterʹಉ༷ͷ࣮Λߦ͏PRΛ ೖΕͯϚʔδ͞Εͨ
» σϞ༻ʹμογϡϘʔυ࡞͚ͬͨͲ·ͩʢsystemd_exporterͷํͰʣ͑ͳ ͍ɺϦϦʔε͞ΕͨΒྑ͔ͬͨΒͬͯΈ͍ͯͩ͘͞ɾɾ
process dashbaord with treemap https://grafana.com/grafana/dashboards/13882-process-exporter-dashboard-with-treemap/
process dashbaord with treemap » DatadogͷμογϡϘʔυʹ͋ΔϝϞϦϚοϓతͳλΠϧදࣔΛߦ͏ෳ߹άϥϑ͕Α͔ͬͨͷͰਅࣅͯ࡞ͬͨ » process-exporeter2ͷϝτϦΫεΛදࣔ͢Δ » /proc
ҎԼͷใΛͱʹ » ҙʹάϧʔϐϯάͨ͠ϓϩηε͝ͱͷϝϞϦ/cpu/io/uptimeͳͲ » ݸਓతʹಛʹ͍͍ͨͷϝϞϦͷ༻ঢ়گͱͦͷมԽ » ͳ͔ͳ͔ศརʹ͑ͨͷͰࣗࣾͷڥʹ࣋ͪࠐΜͰ͍ͬͯΔ 2 process-exporter(https://github.com/ncabatoff/process-exporter)
ϝϞϦϦʔΫͷՄࢹԽͱ͔
OOMൃੜલͷঢ়گ֬ೝͱ͔
μογϡϘʔυ ͬͱڞ༗͞Εͯ΄͍͠ » ެࣜαΠτͷํগ͠վળ͞Ε͖ͯͨ » ΧςΰϦ·ΘΓ͏ͪΐ͍΄͍͠ » ͱ͍͑ݟ͚ͭΔͷେมͳͷͰڞ༗ɺ ൃ৴͕૿͑Δͱخ͍͠ »
͍͖ͬͯ·͠ΐ͏
·ͱΊ
·ͱΊ » μογϡϘʔυ։ൃָ͍͠ » ͕ɺ͋·Γ։ൃɺڞ༗ࣄྫ͕ྲྀΕͯ͜ͳ͍ʢؾ͕͢ΔʣɺͷͰڞ༗͍͖ͯ͠ ·͠ΐ͏ » πʔϧ৭ʑ͋Δ͕ɺݸਓతʹ·ͣखಈ࡞Ͱ » Grizzlyෳͷ࡞๏Ͱ͑ΔͷͰ͕͠ΒΈ͕ͳ͚Ε͓͢͢ΊͰ͢
Thank you for listening