Threads @ Devoxx 2018

When was the last time you run any production code on single-core CPU? And even if, was it a single-threaded code? We shield ourselves from threads with layers of abstractions that promise to take care of everything. However it is often impossible to talk about the application without understanding the underlying thread model.

During this talk i would like to get back to basics. Where does the JVM take threads from? How much a thread cost? How do the threads interact and is my CPU any different from distributed system? Why do we create thread pools and how to control them? How many (and what) threads are there in typical app? You will find answers to those questions on my presentation.

Transcript

1. Threads Adam Dubiel @dubieladam
2. Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise. Edsger W. Dijkstra
3. 12 mln users 700+ microservices 500+ engineers
8. J s a t
9. V t e d co li (~20 t e d )
10. s in n t bo p (+~15 t e d )
11. t a f t s f o n (+~80 t e d )
12. The process is born
13. systemd(1) |-agetty(1338) |-sshd(1133)---sshd(1790)---bash(1791)
14. java -version
15. systemd(1) |-agetty(1338) |-sshd(1133)-...-bash(1791)-java(1876)
16. systemd(1) |-agetty(1338) |-java(1876)
17. Process Memory T1 T2 TN
18. Threads in JVM
19. pstree -Ap
20. systemd(1) |-agetty(1338) |-java(1876)-+-{java}(1893) | |-{java}(1892) | |-{java}(1896) | |-{java}(1890) | |-{java}(1882) | |-{java}(1881) | |-{java}(1885) | |-{java}(1886) | |-{java}(1887) | |-{java}(1879) | |-{java}(1880) | |-{java}(1897) | |-{java}(1898) | |-{java}(1889)
21. JVM threads == system threads
22. pthread_t tid; int ret = pthread_create(&tid, &attr, (void* (*)(void*)) thread_native_entry, thread); http://hg.openjdk.java.net/jdk10/jdk10/hotspot/file/5ab7a67bc155/src/os/linux/vm/os_linux.cpp#l731
23. If JVM threads are system threads.. observable using system tools scheduled using system scheduler
24. ls /proc/1876/task 1876 1893 1892 1896 1890 1882 1881 1885 1886 1887 1879 1880 1897 1898 1889
25. ps H -p <pid> -o pid,tid,pmem,pcpu,time,comm
26. top -H -p <pid>
27. ps H -p <pid> -o pid,tid,pmem,pcpu,time,comm p o s t e d
29. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
30. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
31. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
32. How to match sys & JVM thread?
33. jstack <pid>
34. jcmd <pid> Thread.print
35. "pool-1-thread-10" #19 prio=5 os_prio=31 tid=0x00007fcd05868000 nid=0x6103 runnable [0x000070000ffc5000]
36. "pool-1-thread-10" #19 prio=5 os_prio=31 tid=0x00007fcd05868000 nid=0x6103 runnable [0x000070000ffc5000]
37. pool-1-thread-10 tid=0x00007fcd05868000 nid=0x6103 he (p re d) he (ti )
38. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
39. hex(8521) = 2149 jstack <pid> | grep 2149
40. "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd7ec01f800 nid=0x2149 runnable
41. Java 8 http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/os/linux/vm/os_linux.cpp#l5038 void os::set_native_thread_name(const char *name) { // Not yet implemented. return; }
42. Java 9+ http://hg.openjdk.java.net/jdk10/jdk10/hotspot/file/5ab7a67bc155/src/os/linux/vm/os_linux.cpp#l5038 void os::set_native_thread_name(const char *name) { if (Linux::_pthread_setname_np) { char buf [16]; // according to glibc manpage, 16 chars incl. '/0' snprintf(buf, sizeof(buf), "%s", name); buf[sizeof(buf) - 1] = '\0'; const int rc = Linux::_pthread_setname_np(pthread_self(), buf); // ERANGE should not happen; all other errors should just be ignored. assert(rc != ERANGE, "pthread_setname_np failed"); } }
43. Thread name propagation Java 9+ Thread.setThreadName() is propagated to pthreads 15 chars limit Java thread names visible in system tools
44. systemd(1) |-agetty(1338) |-java(1876)-+-{C1 CompilerThre}(1893) | |-{C2 CompilerThre}(1892) | |-{Common-Cleaner}(1896) | |-{Finalizer}(1890) | |-{G1 Conc#0}(1882) | |-{G1 Main Marker}(1881) | |-{G1 Refine#0}(1885) | |-{G1 Refine#1}(1886) | |-{G1 Young RemSet}(1887) | |-{GC Thread#0}(1879) | |-{GC Thread#1}(1880) | |-{Service Thread}(1895) | |-{XNIO-1 Accept}(1904) | |-{XNIO-1 I/O-1}(1903)
45. PID TID %MEM %CPU TIME COMMAND 3799 3799 15.9 0.0 00:00:00 java 3799 3800 15.9 2.5 00:00:08 java 3799 3804 15.9 10.0 00:00:00 GC Thread#0 3799 3805 15.9 10.0 00:00:00 GC Thread#1 3799 3806 15.9 0.0 00:00:00 G1 Main Marker 3799 3807 15.9 0.0 00:00:00 G1 Conc#0 3799 3808 15.9 0.0 00:00:00 G1 Refine#0 3799 3809 15.9 0.0 00:00:00 G1 Refine#1 3799 3810 15.9 0.0 00:00:00 G1 Young RemSet 3799 3811 15.9 0.0 00:00:00 VM Thread
47. Cost of a thread
48. Cost of thread stack memory - 1MB by default: explicit context switches: implicit safepointing: implicit gc roots: implicit
49. Thread stack
50. Heap Offheap t e d c li re
51. java -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -version
52. java -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -version
53. Thread (reserved=18548KB, committed=18548KB) (thread #18) (stack: reserved=18468KB, committed=18468KB) (malloc=59KB #101) (arena=21KB #34)
54. Thread (reserved=18548KB, committed=18548KB) (thread #18) (stack: reserved=18468KB, committed=18468KB) (malloc=59KB #101) (arena=21KB #34) nu r t e d me y en f o t y m 18468K / 18 = 1024K
55. total app memory = heap size + thread count * stack size + ...
56. How many threads can i spawn? Heap Offheap RAM
57. How many threads can i spawn? Heap Offheap RAM 2G 500M
58. How many threads can i spawn? Heap Offheap RAM 2G 512M 300M fo t .. 1.2G fo h s
59. How many threads can i spawn? 1.2G / 1M = 1200
62. VSZ vs RES
63. VSZ vs RES vi l e r ze ad s p e p o s n a s do 't e t e is u h y i l me y
64. VSZ vs RES re n et me y u l ed b he c t e l t i s o n al r us
66. core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7873 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7873 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
67. core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7873 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7873 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
68. Why less is more?
69. CPU CPU CPU CPU wa g..
70. Cost of thread stack memory - 1MB by default: explicit context switches: implicit safepointing: implicit gc roots: implicit
71. Context switch
72. core L3 cache L1 cache L2 cache s a d ex s e
73. core L3 cache L1 cache L2 cache <1 ns ~4 ns ~20-40 ns ~100 ns RAM
74. core RAM da l L2 -> L1
75. core RAM an r re t i d be g o ot t ad - p o b us s o n on
76. Context switches up to 30 micro seconds each observable via perf and vmstat
77. CPU CPU CPU CPU
78. Back to the app...
80. V t e d co li (~20 t e d )
81. G1 G t e d V h e C1 & C2 Per i T s r a
82. t a f t s f o n (+~80 t e d ) s in n t bo p (+~15 t e d )
83. Managing threads
84. new Thread() Executors.fixedThreadPool()
85. Spring default: SimpleAsyncTaskExecutor TaskExecutor implementation that fires up a new Thread for each task, executing it asynchronously. By default, the number of concurrent threads is unlimited. This implementation does not reuse threads!
86. Thread pool
87. Thread pool w a h n he it's u l?
88. Thread pool ho g i ?
89. Thread pool
90. new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, keepAliveTimeUnit, taskQueue, threadFactory, rejectionPolicy )
91. new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, keepAliveTimeUnit, taskQueue, threadFactory, rejectionPolicy ) ta q u im m at
92. new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, keepAliveTimeUnit, taskQueue, threadFactory, rejectionPolicy ) w a h n w e q u s fu
93. Rejection: Happy path Thread Pool Caller Thread
94. Rejection: Default unhappy path Thread Pool Caller Thread
95. Rejection: CallerRunsPolicy Thread Pool Caller Thread
96. Rejection: AbortPolicy Thread Pool Caller Thread Rejected Execution Exception
97. ThreadPoolExecutor.DiscardPolicy Thread Pool Caller Thread /dev/null
98. ThreadPoolExecutor.DiscardOldestPolicy Thread Pool Caller Thread /dev/null
99. t a f t s f o n (+~80 t e d )
102. on ed r ar t a
103. ThreadPoolExecutor.prestartAllCoreThreads()
104. t a f t s f o n (+~80 t e d ) s in n t bo p (+~15 t e d )
105. -Xlog:thread+os:file=/tmp/threads.log
106. Key takeaways JVM threads are system threads can be observed using system tools WebFlux, because less is more tune your thread pools
107. github.com/adamdubiel @dubieladam allegro.pl/praca