Share
Embed
Practical Phoenix Monitoring
* what to measure
* how to mesure it
* how to use exometer and elixometer
* how we implemented
junsumida
June 30, 2016
Transcript
37 slides
-
2 Introduction @junsumida Jun Sumida self() |> at( XFLAG™ STUDIO,
mixi, inc.) |> develops(game backend servers) |> works(elixir, ruby, javascript, devops) |> github(junsumida) -
3 Introduction • What to measure • How to measure
• How to use Exo/Elixometer • How we Implemented Table of Contents -
• Application information • api request/response count, api response time
• query count, query exec time • Stats of the Erlang VM • memory usage • process count • garbage collection • reductions • io 5 What to measure What to measure -
7 Metric libraries • Erlang • exometer • folsom •
vmstats • Elixir • Elixometer • Beaker • metrix see also https://github.com/h4cc/awesome-elixir#debugging How to measure -
8 Metric libraries • Erlang • exometer • folsom •
vmstats • Elixir • Elixometer • Beaker • metrix see also https://github.com/h4cc/awesome-elixir#debugging How to measure -
9 Exometer https://github.com/Feuerlabs/exometer • a great measurement tool written in
Erlang • developed by Basho (Riak) • easy to use (probably, maybe, possibly…) • multipule backends • InfluxDB, loggers, Graphite, etc… How to measure -
10 Elixometer https://github.com/pinterest/elixometer • a light wrapper around exometer •
developed by Pinterest • easy to use (i swear!) • no complex configurations How to measure -
Why Exo/Elixometer? 11 PROS • well-documented • both of them
“were” well-maintained • still better than most of other measuring libraries • comparatively easy to find examples How to measure -
Why Exo/Elixometer? 12 CONS • dependency hell • lager vs
Logger • configuring exometer is a little bit confusing • struggling with Erlang due to the reporters anyway How to measure -
14 ELIXOMETER • counter • histogram • timer • gauge
• spiral How to use Elixometer -
15 How to use Elixometer • ordinary histograms • stores
all values during a configured time span • provides some analysis • min, max • mean, median • percentile histogram (1) -
16 histogram (2) defmodule Myapp.Plug.Metrics do import Plug.Conn use Elixometer
def call(conn, _default) do before_time = :os.timestamp diff = :timer.now_diff(:os.timestamp, before_time) conn |> register_before_send(fn conn -> update_histogram("response_time", diff) end) end end microseconds How to use Elixometer -
• a sort of histogram • useful when measuring the
performance of a function 18 timer use Elixometer @timed(key: :"timed.super_heavy") def super_heavy do HeavyModule.call_something end def triple_super_heavy do timed("timed.triple.super_heavy") do HeavyModule.call_something HeavyModule.call_something HeavyModule.call_something end end annotation do block How to use Elixometer -
19 • holds one numeric value • always returns the
most recently provided one • good for metrics like memory usage or disk space gauge photo credit: Pressure Cooker - All American Model 910 - Pressure gauge How to use Elixometer -
20 • increases value over time • when reporting the
value, the metric is reset • good for metrics like queries per second or requests per second spiral (1) photo credit: University of Washington How to use Elixometer -
spiral (2) 21 # config/config.exs config :elixometer, reporter: :exometer_report_lager, env:
"webapp", metric_prefix: "myapp", update_frequency: 10_000 config :myapp, Myapp.Repo, adapter: Ecto.Adapters.MySQL, username: "root", password: "", loggers: [{Myapp.Repo.Metrics, :record_metric, []}], hostname: "localhost", database: "myapp", # lib/myapp/repo/metrics.ex defmodule Myapp.Repo.Metrics do use Elixometer def record_metric(entry) do update_spiral("query_count", 1) end end How to use Elixometer -
22 EXOMETER • function • counter • gauge • duration
• histogram How to use Exometer -
function (1) 23 How to use Exometer allows to use
Erlang function as an exometer -
function (2) 24 histogram_stats = ~w(min max mean 95 90)a
memory_stats = ~w(atom binary ets processes total)a polling_interval = 60_000 config :exometer, predefined: [ { ~w(erlang memory)a, {:function, :erlang, :memory, [], :proplist, memory_stats}, [] }, { ~w(erlang system_info)a, {:function, :erlang, :system_info, [:'$dp'], :value, [:process_count, :port_count, :thread_pool_size]}, [] } ], report: [ reporters: [ {:exometer_report_lager, []}, ], subscribers: [ {:exometer_report_lager, [:erlang, :memory], memory_stats, polling_interval, true}, {:exometer_report_lager, [:erlang, :system_info], [:process_count, :port_count, :thread_pool_size], polling_interval, true}, ] ] How to use Exometer -
• Application information • api response count, api response time
• query count, query exec time • Stats of the Erlang VM • memory usage • process count • garbage collection • reductions • io 25 works well with Exometer (pre-defined components) What to measure (again) What to measure (again) works well with Elixometer -
27 Load Balancer Database sensu client sensu client sensu client
Cache Apps Architecture Sensu / influxDB Grafana How we implemented -
28 Our requirements • Metrics of the whole system •
outputs logs in JSON format, and transfers them to ElasticSearch via fluentd • -> Transforming metrics into JSON • Metrics of each server • uses sensu for monitoring • -> HTTP interface for sensu client How we implemented -
29 Load Balancer Database sensu client sensu client sensu client
Cache Apps Sensu / influxDB Grafana How we implemented How we implemented -Transforming metrics into JSON- need some magic metrics -
30 • a Logger backend which outputs log files in
JSON format • transforms metadata into JSON https://github.com/xflagstudio/elixir_logger_json_file_backend {"time":"2016-06-27 12:10:32.023","metric_value": 1443953,"metric_name":"erlang_memory_atom","message":"exomet er_report_lager: erlang_memory_atom: 1443953","level":"info","from":"exometer_report_larger"} {"time":"2016-06-27 12:10:32.023","metric_value": 114784072,"metric_name":"erlang_memory_binary","message":"ex ometer_report_lager: erlang_memory_binary: 114784072","level":"info","from":"exometer_report_larger"} LoggerJSONFileBackend How we implemented -Transforming metrics into JSON- -
elixometer phoenix app exometer lager Logger lager_logger Logging metrics depends
on lager conflicts LoggerJsonFileBackend forked exometer_core -
32 Load Balancer Database sensu client sensu client sensu client
Cache Apps Sensu / influxDB Grafana How we implemented How we implemented -HTTP interface for sensu client- -
elixometer phoenix app exometer HTTP Interface for sensu client 3
ets Grafana Sensu / InfluxDB easy to get a subscription list since dynamically defined, a bit tricky to get a subscription list directly from ets! (*exometer’s spiral doesn’t work as expected.) 33 How we implemented How we implemented -HTTP interface for sensu client- -
34 Other References • Monitoring Phoenix • Monitoring with Exometer:
An Ongoing Love Story • ࣌Ӎಊ BOT αʔόʔ (Վب࠲.tech#5) • A guide to measuring your Elixir app • Feuerlabs/exometer_core References