Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Beyond top: Command-Line Monitoring on the JVM ...
Search
Colin Jones
October 29, 2015
Programming
1
660
Beyond top: Command-Line Monitoring on the JVM (JavaOne 2015)
A session from JavaOne 2015
Colin Jones
October 29, 2015
Tweet
Share
More Decks by Colin Jones
See All by Colin Jones
A Bug's Life: What if `select` is Broken After All?
trptcolin
0
140
Underestimated costs of microservice architectures
trptcolin
3
1.5k
FP vs. OOP: Beyond the Bikeshed
trptcolin
0
400
Diving into the Details with DTrace! (RubyConf 2016 edition)
trptcolin
2
430
Diving into the Details with DTrace
trptcolin
3
500
Adopting FP: the good, the familiar, and the unknown
trptcolin
0
200
Finding out what's *really* going on, with DTrace!
trptcolin
1
350
Beyond top: Command-Line Monitoring on the JVM (ClojureRemote)
trptcolin
0
130
ZooKeeper: Wait-free coordination for Internet-scale systems
trptcolin
2
170
Other Decks in Programming
See All in Programming
Doma で目指す ORM 最適解
nakamura_to
1
160
Babylon.js 8.0のアプデ情報を 軽率にキャッチアップ / catch-up-babylonjs-8
drumath2237
0
110
Proxmoxをまとめて管理できるコンソール作ってみました
karugamo
1
410
MLOps Japan 勉強会 #52 - 特徴量を言語を越えて一貫して管理する, 『特徴量ドリブン』な MLOps の実現への試み
taniiicom
2
570
TVer iOSチームの共通認識の作り方 - Findy Job LT iOSアプリ開発の裏側 開発組織が向き合う課題とこれから
techtver
PRO
0
710
#QiitaBash TDDでAIに設計イメージを伝える
ryosukedtomita
2
1.6k
Devinで実践する!AIエージェントと協働する開発組織の作り方
masahiro_nishimi
6
2.6k
【TSkaigi 2025】これは型破り?型安全? 真実はいつもひとつ!(じゃないかもしれない)TypeScript クイズ〜〜〜〜!!!!!
kimitashoichi
1
300
tsconfigのオプションで変わる型世界
keisukeikeda
1
130
型安全なDrag and Dropの設計を考える
yudppp
5
660
TSConfigからTypeScriptの世界を覗く
planck16
2
1.3k
TypeScript エンジニアが Android 開発の世界に飛び込んだ話
yuisakamoto
6
960
Featured
See All Featured
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
14
1.5k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
123
52k
How to Ace a Technical Interview
jacobian
276
23k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.5k
Designing for humans not robots
tammielis
253
25k
GraphQLとの向き合い方2022年版
quramy
46
14k
VelocityConf: Rendering Performance Case Studies
addyosmani
329
24k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
15
890
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
850
Documentation Writing (for coders)
carmenintech
71
4.8k
Transcript
Beyond top(1) Command-Line Monitoring on the JVM Colin Jones @trptcolin
8th Light
None
What to expect
command-line tooling
on the JVM
introspection & serviceability
--all-flags=false
war stories
real-life usage (well, re-enacted anyway)
A long time ago in a software shop far, far
away…
Things are going pretty well
What does this thing look like? app-architecture Postgres Web /
API Application Server Load Balancer Periodic Job Application Server 3rd-party Service A 3rd-party Service B Monitored email account End users: native mobile app Admin users: desktop browsers
But strange things are afoot
the server sometimes gets really slow
the team has to manually restart the application server
incident response time is ~5 minutes
Yes, strange things are afoot
Pain, frustration, anger
Just the facts
sometimes, things get slow
all requests seem to be affected
the JVM stays up
restart the JVM and everything is fine
What could it be?
Demo
More facts, please!
constant full GCs
what’s in the heap
what application code was running
The right tools for the job
vmstat system-level: CPU, memory, disk, context switching
top per-process: CPU & memory
jps what’s our PID?
jstack status of all threads (right now-ish!)
jcmd what can’t it do?! jcmd [PID] help (sorry, JVM
6 users: see jinfo/jmap/jstack)
jstat GC classloader compiler
Mystery solved!
Now “just” fix it
idea 1: eliminate the leak
idea 2: eliminate the cache altogether?
idea 3: delete the feature
idea 4: full-text search engine
So we’re good now… until the next incident
Lessons
“it’s slow” could mean lots of things
“high CPU” could mean lots of things
collecting data is crucial in a crisis
reproducing the issue helps me sleep at night
Other “right tools for the job”
Heap analyzers
Profilers
Constant monitoring & alerting
Dynamic tracing
Learning more
Books
operators are standing by! man jstat man jstack jcmd [PID]
help [COMMAND] etc.
Thank you! Colin Jones @trptcolin 8th Light