Back to basics: Threads @ Geecon 2018

Transcript

1. Threads Adam Dubiel
2. Being abstract is something profoundly different from being vague … The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise. Edsger W. Dijkstra
3. 12 mln users 700+ microservices 500+ engineers
8. J s a t
9. V t e d co li (~20 t e d )
10. s in n t bo p (+~15 t e d )
11. t a f t s f o n (+~80 t e d )
12. The process is born
13. systemd(1) |-agetty(1338) |-sshd(1133)---sshd(1790)---bash(1791)
14. java -version
15. systemd(1) |-agetty(1338) |-sshd(1133)-...-bash(1791)-java(1876)
16. systemd(1) |-agetty(1338) |-java(1876)
17. Process Memory T1 T2 TN
18. Threads in JVM
19. pstree -Ap
20. systemd(1) |-agetty(1338) |-java(1876)-+-{java}(1893) | |-{java}(1892) | |-{java}(1896) | |-{java}(1890) | |-{java}(1882) | |-{java}(1881) | |-{java}(1885) | |-{java}(1886) | |-{java}(1887) | |-{java}(1879) | |-{java}(1880) | |-{java}(1897) | |-{java}(1898) | |-{java}(1889)
21. JVM threads == system threads
22. pthread_t tid; int ret = pthread_create(&tid, &attr, (void* (*)(void*)) thread_native_entry, thread); http://hg.openjdk.java.net/jdk10/jdk10/hotspot/file/5ab7a67bc155/src/os/linux/vm/os_linux.cpp#l731
23. If JVM threads are system threads.. observable using system tools scheduled using system scheduler
24. ls /proc/1876/task 1876 1893 1892 1896 1890 1882 1881 1885 1886 1887 1879 1880 1897 1898 1889
25. ps H -p <pid> -o pid,tid,pmem,pcpu,time,comm
26. top -H -p <pid>
27. ps H -p <pid> -o pid,tid,pmem,pcpu,time,comm p o s t e d
29. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
30. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
31. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
32. How to match sys & JVM thread?
33. jstack <pid>
34. "pool-1-thread-10" #19 prio=5 os_prio=31 tid=0x00007fcd05868000 nid=0x6103 runnable [0x000070000ffc5000]
35. "pool-1-thread-10" #19 prio=5 os_prio=31 tid=0x00007fcd05868000 nid=0x6103 runnable [0x000070000ffc5000]
36. pool-1-thread-10 tid=0x00007fcd05868000 nid=0x6103 he (p re d) he (ti )
37. PID TID %MEM %CPU TIME COMMAND 8511 8511 28.4 0.0 00:00:00 java 8511 8520 28.4 0.5 00:00:00 java 8511 8521 28.4 30.2 00:00:09 java 8511 8522 28.4 30.0 00:00:09 java 8511 8523 28.4 30.2 00:00:09 java 8511 8524 28.4 30.3 00:00:10 java 8511 8525 28.4 1.9 00:00:00 java 8511 8526 28.4 0.0 00:00:00 java 8511 8527 28.4 0.0 00:00:00 java 8511 8528 28.4 0.0 00:00:00 java
38. hex(8521) = 2149 jstack <pid> | grep 2149
39. "GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007fd7ec01f800 nid=0x2149 runnable
40. Java 8 http://hg.openjdk.java.net/jdk8/jdk8/hotspot/file/87ee5ee27509/src/os/linux/vm/os_linux.cpp#l5038 void os::set_native_thread_name(const char *name) { // Not yet implemented. return; }
41. Java 9+ http://hg.openjdk.java.net/jdk10/jdk10/hotspot/file/5ab7a67bc155/src/os/linux/vm/os_linux.cpp#l5038 void os::set_native_thread_name(const char *name) { if (Linux::_pthread_setname_np) { char buf [16]; // according to glibc manpage, 16 chars incl. '/0' snprintf(buf, sizeof(buf), "%s", name); buf[sizeof(buf) - 1] = '\0'; const int rc = Linux::_pthread_setname_np(pthread_self(), buf); // ERANGE should not happen; all other errors should just be ignored. assert(rc != ERANGE, "pthread_setname_np failed"); } }
42. Thread name propagation Java 9+ Thread.setThreadName() is propagated to pthreads 15 chars limit Java thread names visible in system tools
43. systemd(1) |-agetty(1338) |-java(1876)-+-{C1 CompilerThre}(1893) | |-{C2 CompilerThre}(1892) | |-{Common-Cleaner}(1896) | |-{Finalizer}(1890) | |-{G1 Conc#0}(1882) | |-{G1 Main Marker}(1881) | |-{G1 Refine#0}(1885) | |-{G1 Refine#1}(1886) | |-{G1 Young RemSet}(1887) | |-{GC Thread#0}(1879) | |-{GC Thread#1}(1880) | |-{Service Thread}(1895) | |-{XNIO-1 Accept}(1904) | |-{XNIO-1 I/O-1}(1903)
44. PID TID %MEM %CPU TIME COMMAND 3799 3799 15.9 0.0 00:00:00 java 3799 3800 15.9 2.5 00:00:08 java 3799 3804 15.9 10.0 00:00:00 GC Thread#0 3799 3805 15.9 10.0 00:00:00 GC Thread#1 3799 3806 15.9 0.0 00:00:00 G1 Main Marker 3799 3807 15.9 0.0 00:00:00 G1 Conc#0 3799 3808 15.9 0.0 00:00:00 G1 Refine#0 3799 3809 15.9 0.0 00:00:00 G1 Refine#1 3799 3810 15.9 0.0 00:00:00 G1 Young RemSet 3799 3811 15.9 0.0 00:00:00 VM Thread
46. Cost of a thread
47. Cost of thread stack memory - 1MB by default: explicit context switches: implicit gc roots: implicit safepointing: implicit
48. Thread stack
49. Heap Offheap t e d c li re
50. java -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -version
51. java -XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics -version
52. Thread (reserved=18548KB, committed=18548KB) (thread #18) (stack: reserved=18468KB, committed=18468KB) (malloc=59KB #101) (arena=21KB #34)
53. Thread (reserved=18548KB, committed=18548KB) (thread #18) (stack: reserved=18468KB, committed=18468KB) (malloc=59KB #101) (arena=21KB #34) nu r t e d me y en f o t y m 18468K / 18 = 1024K
54. total app memory = heap size + thread count * stack size + ...
55. How many threads can i spawn? Heap Offheap RAM
56. How many threads can i spawn? Heap Offheap RAM 2G 500M
57. How many threads can i spawn? Heap Offheap RAM 2G 512M 300M fo t .. 1.2G fo h s
58. How many threads can i spawn? 1.2G / 1M = 1200
61. VSZ vs RES
62. VSZ vs RES vi l e r ze ad s p e p o s n a s do 't e t e is u h y i l me y
63. VSZ vs RES re n et me y u l ed b he c t e l t i s o n al r us
65. core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7873 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7873 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
66. core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 7873 max locked memory (kbytes, -l) 16384 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 7873 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
67. Why less is more?
68. CPU CPU CPU CPU wa g..
69. Context switch
70. core L3 cache L1 cache L2 cache s a d ex s e
71. core L3 cache L1 cache L2 cache <1 ns ~4 ns ~20-40 ns ~100 ns RAM
72. core RAM da l L2 -> L1
73. core RAM an r re un s a y!
74. Context switches up to 30 micro seconds each observable via perf and vmstat
75. CPU CPU CPU CPU
76. Back to the app...
78. V t e d co li (~20 t e d )
79. G1 G t e d V h e C1 & C2 Per i T s r a
80. t a f t s f o n (+~80 t e d ) s in n t bo p (+~15 t e d )
81. Managing threads
82. new Thread() Executors.fixedThreadPool()
83. Spring default: SimpleAsyncTaskExecutor TaskExecutor implementation that fires up a new Thread for each task, executing it asynchronously. By default, the number of concurrent threads is unlimited. This implementation does not reuse threads!
84. Thread pool
85. Thread pool w a h n he it's u l?
86. Thread pool ho g i ?
87. Thread pool
88. new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, keepAliveTimeUnit, taskQueue, threadFactory, rejectionPolicy )
89. new ThreadPoolExecutor( corePoolSize, maxPoolSize, keepAliveTime, keepAliveTimeUnit, taskQueue, threadFactory, rejectionPolicy ) ta q u im m at
90. t a f t s f o n (+~80 t e d )
93. on ed r ar t a
94. ThreadPoolExecutor.prestartAllCoreThreads()
95. t a f t s f o n (+~80 t e d ) s in n t bo p (+~15 t e d )
96. -Xlog:thread+os:file=/tmp/threads.log
97. Key takeaways JVM threads are system threads can be observed using system tools WebFlux, because less is more tune your thread pools
98. github.com/adamdubiel @dubieladam allegro.pl/praca