Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beyond top: Command-Line Monitoring on the JVM ...

Avatar for Colin Jones Colin Jones
October 29, 2015
Programming
710
1

Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)

A session from JavaOne 2015

Avatar for Colin Jones

Colin Jones

October 29, 2015

More Decks by Colin Jones

See All by Colin Jones
A Bug's Life: What if `select` is Broken After All?
Avatar for Colin Jones trptcolin
0
200
Underestimated costs of microservice architectures
Avatar for Colin Jones trptcolin
3
1.6k
FP vs. OOP: Beyond the Bikeshed
Avatar for Colin Jones trptcolin
0
470
Diving into the Details with DTrace! (RubyConf 2016 edition)
Avatar for Colin Jones trptcolin
2
550
Diving into the Details with DTrace
Avatar for Colin Jones trptcolin
3
540
Adopting FP: the good, the familiar, and the unknown
Avatar for Colin Jones trptcolin
0
290
Finding out what's *really* going on, with DTrace!
Avatar for Colin Jones trptcolin
1
420
Beyond top: Command-Line Monitoring on the JVM (ClojureRemote)
Avatar for Colin Jones trptcolin
0
180
ZooKeeper: Wait-free coordination for Internet-scale systems
Avatar for Colin Jones trptcolin
2
240

Other Decks in Programming

See All in Programming
実装をデザインガイドラインに追従させるための取り組み / 260731-dip-mosh-design-system
Avatar for dachi dachi023
0
7.6k
そこに3びきプロダクトがいるじゃろう——生成AI時代における“価値が届かない理由”の構造
Avatar for kosuke T kosuket
0
570
MySQLとPostgreSQLって何が違うの？
Avatar for 赤神青空 akagami
0
130
まだ間に合う！今年の夏こそSchemeのマクロ展開器を完全理解！
Avatar for Masanori Ogino omasanori
0
400
今さら聞けない .NET CLI
Avatar for tkym htkym
0
210
Flow は今どうなっているか
Avatar for mizdra mizdra
PRO
0
680
夏だ！祭りだ！祭りとはドメインモデリングでは？
Avatar for ryugen04 ryugen04
0
360
Detecting Compromised CI with eBPF and Cilium Tetragon
Avatar for Liz Rice lizrice
0
250
S3 を使うアプリケーションをローカル完結で動かすことに全力を注いでみた / Running S3 Apps Offline
Avatar for Daisuke Garaike contour_gara
0
580
変わらないものが、変わるものを決める — 意図駆動開発 × イベントソーシング × イミュータブル | What Doesn't Change Decides What Can — IDD × Event Sourcing × Immutability
Avatar for Tomohisa Takaoka tomohisa
0
1.8k
ソフトウェアエンジニアにとっての生成AI - 特性を知って使い倒す / generative ai for software enginner
Avatar for Naoki Kishida kishida
7
2.1k
AWS DevOps AgentのAzure接続機能を検証して見えた活用法／Use Cases Verified for the AWS DevOps Agent's Azure Connectivity Feature
Avatar for モブエンジニア（Masaki Okuda） masakiokuda
1
260

Featured

See All Featured
Everyday Curiosity
Avatar for Cassini Nazir cassininazir
0
290
Skip the Path - Find Your Career Trail
Avatar for Mark Kilby mkilby
1
190
The State of eCommerce SEO: How to Win in Today's Products SERPs - #SEOweek
Avatar for Aleyda Solis aleyda
2
11k
Groundhog Day: Seeking Process in Gaming for Health
Avatar for Sebastian Deterding codingconduct
0
290
Producing Creativity
Avatar for Steve Smith orderedlist
PRO
348
40k
jQuery: Nuts, Bolts and Bling
Avatar for Doug Neiner dougneiner
66
8.5k
Why Your Marketing Sucks and What You Can Do About It - Sophie Logan
Avatar for Sophie Logan marketingsoph
0
390
Visualization
Avatar for Eitan Lees eitanlees
152
17k
Building Applications with DynamoDB
Avatar for Matt Wood mza
96
7.2k
Fight the Zombie Pattern Library - RWD Summit 2016
Avatar for Marcelo Somers marcelosomers
234
17k
Between Models and Reality
Avatar for mayunak mayunak
4
400
How Software Deployment tools have changed in the past 20 years
Avatar for Geshan Manandhar geshan
1
34k

Transcript

  1. Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin

    8th Light
  2. None

  3. What to expect

  4. command-line tooling

  5. on the JVM

  6. introspection & serviceability

  7. --all-flags=false

  8. war stories

  9. real-life usage (well, re-enacted anyway)

  10. A long time ago in a software shop far, far

    away…

  11. Things are going pretty well

  12. What does this thing look like? app-architecture Postgres Web /

    API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers

  13. But strange things are afoot

  14. the server sometimes gets really slow

  15. the team has to manually restart the application server

  16. incident response time is ~5 minutes

  17. Yes, strange things are afoot

  18. Pain, frustration, anger

  19. Just the facts

  20. sometimes, things get slow

  21. all requests seem to be affected

  22. the JVM stays up

  23. restart the JVM and everything is fine

  24. What could it be?

  25. Demo

  26. More facts, please!

  27. constant full GCs

  28. what’s in the heap

  29. what application code was running

  30. The right tools for the job

  31. vmstat system-level: CPU, memory, disk, context switching

  32. top per-process: CPU & memory

  33. jps what’s our PID?

  34. jstack status of all threads (right now-ish!)

  35. jcmd what can’t it do?! jcmd [PID] help (sorry, JVM

    6 users: see jinfo/jmap/jstack)

  36. jstat GC classloader compiler

  37. Mystery solved!

  38. Now “just” fix it

  39. idea 1: eliminate the leak

  40. idea 2: eliminate the cache altogether?

  41. idea 3: delete the feature

  42. idea 4: full-text search engine

  43. So we’re good now… until the next incident

  44. Lessons

  45. “it’s slow” could mean lots of things

  46. “high CPU” could mean lots of things

  47. collecting data is crucial in a crisis

  48. reproducing the issue helps me sleep at night

  49. Other “right tools for the job”

  50. Heap analyzers

  51. Profilers

  52. Constant monitoring & alerting

  53. Dynamic tracing

  54. Learning more

  55. Books

  56. operators are standing by! man jstat man jstack jcmd [PID]

    help [COMMAND] etc.

  57. Thank you! Colin Jones @trptcolin 8th Light