Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Monitoring Containerized Elixir
Search
さっちゃん
September 07, 2019
Programming
1
1k
Monitoring Containerized Elixir
#ElixirConfJP #CastleConf
さっちゃん
September 07, 2019
Tweet
Share
More Decks by さっちゃん
See All by さっちゃん
みんなのオブザーバビリティプラットフォームを作ってるんだがパフォーマンスがやばい #mackerelio #srenext
ne_sachirou
0
1.5k
作ってよかったgraceful shutdownライブラリ #kyotogo
ne_sachirou
0
1.3k
path 依存型って何?
ne_sachirou
0
690
野生の onbording と onbording 設計 #kyototechtalk
ne_sachirou
0
650
メトリックはいかにして見え續ける樣になったか #devio2022
ne_sachirou
0
97
名實一致
ne_sachirou
0
680
まかれるあなとみあ ―Mackerel のしくみを理解する 30 分― @ Hatena Engineer Seminar #16
ne_sachirou
0
3.2k
tacit programming : Point-free, Concatenatives & J
ne_sachirou
0
1k
Let's create stateful systems, by Elixir
ne_sachirou
1
970
Other Decks in Programming
See All in Programming
Чего вы не знали о строках в Python – Василий Рябов, PythoNN
sobolevn
0
150
iOSアプリの信頼性を向上させる取り組み/ios-app-improve-reliability
shino8rayu9
0
130
猫と暮らすネットワークカメラ生活🐈 ~Vision frameworkでペットを愛でよう~ / iOSDC Japan 2025
yutailang0119
0
210
そのpreloadは必要?見過ごされたpreloadが技術的負債として爆発した日
mugitti9
2
2.8k
メモリ不足との戦い〜大量データを扱うアプリでの実践例〜
kwzr
1
740
iOSエンジニア向けの英語学習アプリを作る!
yukawashouhei
0
170
iOS 17で追加されたSubscriptionStoreView を利用して5分でサブスク実装チャレンジ
natmark
0
480
Pythonスレッドとは結局何なのか? CPython実装から見るNoGIL時代の変化
curekoshimizu
4
1.2k
クラシルを支える技術と組織
rakutek
0
190
Model Pollution
hschwentner
1
180
なぜGoのジェネリクスはこの形なのか? Featherweight Goが明かす設計の核心
ryotaros
7
1k
AIを活用したレシート読み取り機能の開発から得られた実践知 / AI Receipt Scan Practice
rockname
2
1.5k
Featured
See All Featured
Intergalactic Javascript Robots from Outer Space
tanoku
273
27k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Scaling GitHub
holman
463
140k
How STYLIGHT went responsive
nonsquared
100
5.8k
Learning to Love Humans: Emotional Interface Design
aarron
274
40k
What's in a price? How to price your products and services
michaelherold
246
12k
Building Adaptive Systems
keathley
43
2.8k
Become a Pro
speakerdeck
PRO
29
5.5k
Java REST API Framework Comparison - PWX 2021
mraible
33
8.8k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
PRO
188
55k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Reflections from 52 weeks, 52 projects
jeffersonlam
352
21k
Transcript
Monitoring Containerized Elixir
.。oO(さっちゃんですよヾ(〃l _ l)ノ゙☆) @ Mackerel.io / Hatena.ne.jp
Imagine, you have a great Elixir app.
Imagine, you have a great Elixir app. And will be
working it on Docker
Imagine, you have a great Elixir app. And will be
working it on Docker on Kubernetes.
Why container, Docker? Container is an abstract layer over infrastructure.
It runs everywhere. It easy to create, share, transfer & run.
Why Kubernetes? K8s is a common language. K8s can run
every types of applications.
Elixir on Containers https://speakerdeck.com/ne_sachirou/elixir-on- containers s/:distillery/Mix.Release/
When you've deployed the app on K8s. How to monitor
it?
Why Monitoring? Is your app works well? Know how your
app works. What caused when your app is broken. Keep your app observable.
Liveness check Is your app is living or not?
K8s has liveness probe. --- apiVersion: apps/v1 kind: Deployment spec:
template: spec: containers: - name: app livenessProbe: httpGet: path: /ops/heartbeat port: 4000
How to check your app is live or not?
KomachiHeartbeat https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.
KomachiHeartbeat The name is originate from a great Rubygems :
https://rubygems.org/gems/komachi_heartbeat
Add KomachiHeartbeat. defmodule Example.Router do use Plug.Router plug(:match) plug(:dispatch) forward("/ops",
to: KomachiHeartbeat) end defmodule ExampleWeb.Router do use ExampleWeb, :router forward("/ops", KomachiHeartbeat) end
Check it. % curl http://localhost/ops/heartbeat heartbeat:ok Also available from Zabbix,
ALB health check or Mackerel's "External Http monitor (外形監視)".
Error collection Listening canary song.
Sentry https://sentry.io/ Collect & sort errors. Show the error environments:
stacktrace, HTTP params, app host, release ver… Assign & resolve issue. Integrate with Sass: Slack, GitHub…
Add https://hex.pm/packages/sentry in your app. defmodule Example.Application do use Application
def start(_type, _args) do Logger.add_backend(Sentry.LoggerBackend) … end defmodule Example.Router do use Sentry.Plug … end & add many metadata, by yourself (・﹏・)
Metrics monitoring Know the app behaviour & detect anomary.
Prometheus + Grafana + deadtrickster/beam-dashboards Beautiful. But to keep Prometheus
servers have many toils.
Mackerel https://mackerel.io
Add mackerel-container-agent as a sidecar. --- apiVersion: apps/v1 kind: Deployment
spec: template: spec: containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:latest imagePullPolicy: Always resources: limits: {memory: 128Mi} env: - {name: MACKEREL_CONTAINER_PLATFORM, value: kubernetes} - name: MACKEREL_KUBERNETES_KUBELET_HOST valueFrom: fieldRef: fieldPath: status.hostIP - name: MACKEREL_KUBERNETES_NAMESPACE valueFrom: fieldRef: {fieldPath: metadata.namespace} - name: MACKEREL_KUBERNETES_POD_NAME valueFrom: fieldRef: {fieldPath: metadata.name} - {name: MACKEREL_ROLES, value: example:app} envFrom: - secretRef: {name: mackerel}
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization … secretGenerator: - name: mackerel type:
Opaque files: [MACKEREL_APIKEY=secret/MACKEREL_APIKEY]
None
OK. I see container's metrics. How about BEAM metrics?
KomachiHeartbeat + mackerel- plugin-json https://hex.pm/packages/komachi_heartbeat Vital monitoring Elixir Web application.
https://github.com/mackerelio/mackerel-plugin-json Json custom metrics plugin for mackerel.io agent.
KomachiHeartbeat has /stats endpoint that returns app's stats in JSON.
mackerel-plugin-json posts any data gotten from JSON API as Mackerel custome metrics.
We have KomachiHeartbeat.BeamVital . https://github.com/ne- sachirou/ex_komachi_heartbeat/pull/29 (Sorry for under development.)
Add KomachiHeartbeat.BeamVital . defmodule TeijiBot.Router do use Plug.Router plug(:match) plug(:dispatch)
forward( "/ops", to: KomachiHeartbeat, init_opts: [vitals: [KomachiHeartbeat.BeamVital]] ) end
Configure mackerel-plugin-json. --- apiVersion: apps/v1 kind: Deployment spec: template: spec:
volumes: - name: mackerel-agent-config configMap: name: mackerel-agent-config containers: - name: app … - name: mackerel-container-agent image: mackerel/mackerel-container-agent:plugins … env: - {name: MACKEREL_AGENT_CONFIG, value: /etc/mackerel/mackerel.yaml} … volumeMounts: - name: mackerel-agent-config mountPath: /etc/mackerel readOnly: true
--- apiVersion: v1 kind: ConfigMap metadata: name: mackerel-agent-config data: mackerel.yaml:
| plugin: metrics: json: command: | mackerel-plugin-json \ -url="http://localhost:4000/ops/stats" \ -prefix='beam'
None
Current metrics : atom_count memory_ets_tables memory_system_bytes_total atom, binary, code, ets,
other port_count process_count (More, under development.)