Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Observability: Mackerel による観測と Mackerel の観測 / ...

astj
September 17, 2019

Observability: Mackerel による観測と Mackerel の観測 / NoOps Meetup Tokyo #8

astj

September 17, 2019
Tweet

More Decks by astj

Other Decks in Technology

Transcript

  1. ?

  2. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
  3. Metrics Observability Log Observability Trace Observability Metrics Observation Log Observation

    Trace Observation ʮϝτϦοΫ؂ࢹʯ͸͜ͷลʁ • Server Metrics • Middleware Metrics • Application Metrics • Business KPI • Etc… • Alerting • Dashboard Visualization • Analysis • Etc…
  4. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ٕज़ͱͯ͠ͷ Observability ͕
 ͜ͷลΓΛࢦ͢͜ͱ΋͋Ε͹
  5. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ιϦϡʔγϣϯͷ࿩୊Ͱ Observability ͕ݴٴ͞ΕΔ࣌͸
 ͜͜·Ͱؚ·ΕΔ͜ͱ΋ଟ͍
  6. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ϝτϦοΫ؂ࢹ͸: • ϝτϦοΫΛऩू͢Δ • ऩूͨ͠ϝτϦοΫΛूੵ͢Δ • άϥϑ΍μογϡϘʔυͰՄࢹԽ͢Δ • ΞϥʔτΛൃใ͢Δ
  7. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. ϝτϦοΫऩू ϝτϦοΫूੵ • άϥϑ • Ξϥʔτ • μογϡϘʔυ ˞Πϝʔδ • ਓؒ • ࣗಈԽରԠ
  8. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͞ΕΔʣ
  9. Service:Blog Role:App Role:DB Host A Host B Host C Host

    D Service:Bookmark Role:App Host E Host F … αʔϏεɾϩʔϧ
  10. • αʔόʔ্Ͱಈ͘ΤʔδΣϯτ • ΤʔδΣϯτ͔Β Mackerel ʹ POST ͢Δ • γεςϜϝτϦοΫ͸ࣗಈͰऩू

    • CPU, memory, interface… • ϓϥάΠϯͰऔಘ৘ใΛ֦ுͰ͖Δ αʔόʔ͔Βͷऩू
  11. • ECS 1 Task ΍ k8s 1 pod Λ1ϗετ •

    αΠυΧʔίϯςφͱͯ͠ΤʔδΣϯτΛಋೖ • γεςϜϝτϦοΫΛࣗಈͰऩू • ϓϥάΠϯͰ֦ுՄೳ ίϯςφ͔Βͷऩू
  12. Task Task Task Mackerel Agent Container Fluentd container Golang app

    container monitor-agent Plugins HTTP gostats POST Orchestration Metadata ྫ
  13. Observability Observability Observability Observation Observation Observation Observe impl. Observe impl.

    Observe Impl. • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ • Մ؍ଌੑ͸؂ࢹͷલఏͱͳΔ • ʢͷͰɺ؂ࢹͷจ຺ͰՄ؍ଌੑʹݴٴ͢Δʣ ࠶ܝ • ؍ଌ͢ΔͨΊʹ͸؍ଌՄೳʹ͢Δඞཁ͕͋Δ
  14. HTTP ܦ༝Ͱͷऩू $ mackerel-plugin-json \ > -url=http://localhost/stats active3 1568666460 waiting

    2 1568666460 $ curl -fsSL http://localhost/stats {“active”:3,”waiting”:2}
  15. Mackerel ͷٕज़ελοΫ • AWS ্ʹߏங • EC2 ӡ༻ͱ ECS Fargate

    ͕ࠞࡏ • ϝΠϯͷ JVM ͳͲ͕ EC2 • গͣͭ͠ίϯςφϫʔΫϩʔυʹҠߦத
  16. JVM ΞϓϦέʔγϣϯͷ؍ଌ • ϝτϦΫεɾ؂ࢹ: mackerel • jstat ͱ jolokia ʹΑΔJVM಺෦ߏ଄ͷ؍ଌ

    • αʔόʔ͝ͱͷϨΠςϯγ؂ࢹ • ίωΫγϣϯϓʔϧ؂ࢹ • etc…