Game of Loom 2: life and dead(lock) of a virtual thread

Virtual threads finally exited their development and preview phases and with JVM 21 are available as a stable and supported Java feature. During the latest Devoxx edition I started exploring the characteristics of virtual threads and their performance implications putting them at work, with a funny but practical example, using a Conway's Game of Life implementation based on Project Loom. Starting from the same playground this time we will explore more in depth the internal implementation details of virtual threads, trying to answer to some interesting questions. What does it mean in practice that virtual threads aren't preemptive? Are local variables enough to replace ThreadLocals in all possible scenarios? What does it happen if you try to replace the fork/join pool, used as default carrier thread pool, with something different? At the end we will conclude this exploration trying to experience the multithreaded programming equivalent of the sound of one hand clapping or how virtual threads make it possible to cause a deadlock using one single lock.

Transcript

1. Game of Loom 2: life and dead(lock) of a virtual thread by Mario Fusco @mariofusco
2. Where did we leave? A recap on virtual threads … ❖ Virtual threads make a better use of OS threads getting rid of the 1:1 relationship with operating system threads ➢ Many virtual threads can be multiplexed on the same platform thread (carrier)
3. Where did we leave? A recap on virtual threads … ❖ Virtual threads make a better use of OS threads getting rid of the 1:1 relationship with operating system threads ➢ Many virtual threads can be multiplexed on the same platform thread (carrier) ❖ When a virtual thread calls a blocking operation the JDK performs a nonblocking OS call and automatically suspends the virtual thread until the operation finishes ➢ Virtual threads perform blocking calls without consuming resources, thus allowing to write asynchronous code in a way that looks synchronous ➢ What virtual threads are really good for is … waiting
4. Where did we leave? A recap on virtual threads … ❖ Virtual threads make a better use of OS threads getting rid of the 1:1 relationship with operating system threads ➢ Many virtual threads can be multiplexed on the same platform thread (carrier) ❖ When a virtual thread calls a blocking operation the JDK performs a nonblocking OS call and automatically suspends the virtual thread until the operation finishes ➢ Virtual threads perform blocking calls without consuming resources, thus allowing to write asynchronous code in a way that looks synchronous ➢ What virtual threads are really good for is … waiting ❖ Virtual threads are lightweight ➢ 200-300B metadata ➢ Pay-as-you-go stack (allocated on heap) ➢ Some ns (or below 1µs) for context switch (in user space)
6. Should we really use virtual threads *EVERYWHERE*?
7. Should we really use virtual threads *EVERYWHERE*? ❖ Not ideal for CPU-bound task ➢ Lack of fairness due to the fact that virtual threads are not pre-emptive ➢ Overall performance concerns caused by overhead of virtual threads scheduler
8. Should we really use virtual threads *EVERYWHERE*? ❖ Not ideal for CPU-bound task ➢ Lack of fairness due to the fact that virtual threads are not pre-emptive ➢ Overall performance concerns caused by overhead of virtual threads scheduler ❖ Not virtual-threads friendly frameworks & tools ➢ Use of synchronize can lead to pinning ➢ Use of ThreadLocal
9. Should we really use virtual threads *EVERYWHERE*? ❖ Not ideal for CPU-bound task ➢ Lack of fairness due to the fact that virtual threads are not pre-emptive ➢ Overall performance concerns caused by overhead of virtual threads scheduler ❖ Not virtual-threads friendly frameworks & tools ➢ Use of synchronize can lead to pinning ➢ Use of ThreadLocal ❖ Poorly customizable ➢ Carrier native thread pool is not pluggable (but for a very good reason)
10. Lack of fairness in CPU bound tasks Virtual Threads are not pre-emptive, they cannot be descheduled while running heavy calculations without ever calling a JDK’s blocking methods, so they are not a good fit for CPU-bound tasks when fairness is key Interrupts the processing of a task and transfers the CPU to another process Non-preemptive operation usually proceeds towards completion uninterrupted
11. Lack of fairness in CPU bound tasks Virtual Threads are not pre-emptive, they cannot be descheduled while running heavy calculations without ever calling a JDK’s blocking methods, so they are not a good fit for CPU-bound tasks when fairness is key
12. The cost of a continuation In Project Loom, the word continuation means delimited continuation, also sometimes called a coroutine. It can be thought of as sequential code that may suspend or yield execution at some point by itself and can be resumed by a caller. When a virtual thread hits a blocking call its continuation is suspended and the JVM unmounts the thread from its carrier …
13. The cost of a continuation In Project Loom, the word continuation means delimited continuation, also sometimes called a coroutine. It can be thought of as sequential code that may suspend or yield execution at some point by itself and can be resumed by a caller. When a virtual thread hits a blocking call its continuation is suspended and the JVM unmounts the thread from its carrier … … but this also has a cost!
14. Native Threads Execution
15. Virtual Threads Execution - The cost of a continuation
16. Virtual Threads Execution - The cost of a continuation
17. ThreadLocals as thread scoped variables enum RightsLevel { ADMIN, GUEST } record Principal(RightsLevel rights) { public boolean canOpen() { return rights == RightsLevel.ADMIN; } } record Request(boolean authorized) { } class Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } }
18. ThreadLocals as thread scoped variables static class DBConnection { static DBConnection open() { var principal = Server.PRINCIPAL.get(); if (!principal.canOpen()) throw new IllegalArgumentException(); return new DBConnection(); } String doQuery() { return "Result"; } } static class Application { public static String handle(Request request) { return DBConnection.open().doQuery(); } } enum RightsLevel { ADMIN, GUEST } record Principal(RightsLevel rights) { public boolean canOpen() { return rights == RightsLevel.ADMIN; } } record Request(boolean authorized) { } class Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } }
19. ThreadLocals as thread scoped variables A ThreadLocal is a construct that allows us to store data accessible only by a specific thread. static class DBConnection { static DBConnection open() { var principal = Server.PRINCIPAL.get(); if (!principal.canOpen()) throw new IllegalArgumentException(); return new DBConnection(); } String doQuery() { return "Result"; } } static class Application { public static String handle(Request request) { return DBConnection.open().doQuery(); } } enum RightsLevel { ADMIN, GUEST } record Principal(RightsLevel rights) { public boolean canOpen() { return rights == RightsLevel.ADMIN; } } record Request(boolean authorized) { } class Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } }
20. ThreadLocals as (virtual) thread scoped variables public static void main(String[] args) { var server = new Server(); var authorized = Thread.ofVirtual().name("Authorized").unstarted(() -> callServer(server, true)); var notAuthorized = Thread.ofVirtual().name("NOT Authorized").unstarted(() -> callServer(server, false)); authorized.start(); notAuthorized.start(); try { Thread.sleep(1000L); } catch (InterruptedException e) { throw new RuntimeException(e); } } private static void callServer(Server server, boolean auth) { var result = server.serve(new Request(auth)); System.out.println( "thread " + Thread.currentThread().getName() + " got result " + result ); }
21. ThreadLocals as (virtual) thread scoped variables public static void main(String[] args) { var server = new Server(); var authorized = Thread.ofVirtual().name("Authorized").unstarted(() -> callServer(server, true)); var notAuthorized = Thread.ofVirtual().name("NOT Authorized").unstarted(() -> callServer(server, false)); authorized.start(); notAuthorized.start(); try { Thread.sleep(1000L); } catch (InterruptedException e) { throw new RuntimeException(e); } } private static void callServer(Server server, boolean auth) { var result = server.serve(new Request(auth)); System.out.println( "thread " + Thread.currentThread().getName() + " got result " + result ); } Exception in thread "NOT Authorized" java.lang.IllegalArgumentException at org.mfusco.loom.experiments.threadlocal.ThreadLocalMain$DBConnection.open(ThreadLocalMain.java:48) at org.mfusco.loom.experiments.threadlocal.ThreadLocalMain$Application.handle(ThreadLocalMain.java:41) thread Authorized got result Result
22. ThreadLocals as (virtual) thread scoped variables So? Everything is fine, right? What is the problem to use ThreadLocal with virtual threads?
23. ThreadLocals as (virtual) thread scoped variables So? Everything is fine, right? What is the problem to use ThreadLocal with virtual threads? ❖ That we can have a huge number of virtual threads, and each virtual thread will have its own ThreadLocal ➢ The memory footprint of the application may quickly become very high ➢ ThreadLocal will be useless in a one-thread-per-request scenario since data won’t be shared between different requests
24. ScopedValues to the rescue class Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } }
25. ScopedValues to the rescue (still preview in JDK 21) class Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } } class Server { final static ScopedValue<Principal> PRINCIPAL = ScopedValue.newInstance(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; try { return ScopedValue.where(PRINCIPAL, new Principal(level)) .call(() -> Application.handle(request)); } catch (Exception e) { throw new RuntimeException(e); } } }
26. ThreadLocals as Cache / Objects Pool ??? Do ScopedValues cover all the possible typical usage patterns of ThreadLocals?
27. ThreadLocals as Cache / Objects Pool ??? Do ScopedValues cover all the possible typical usage patterns of ThreadLocals? ❖ It’s very common to use a ThreadLocal as a thread-safe object pool ➢ Easy to implement and use ➢ Does not require an explicit lifecycle: it is not necessary to put the pooled resource back into the pool when finished
28. ThreadLocals as Cache / Objects Pool ??? Do ScopedValues cover all the possible typical usage patterns of ThreadLocals? ❖ It’s very common to use a ThreadLocal as a thread-safe object pool ➢ Easy to implement and use ➢ Does not require an explicit lifecycle: it is not necessary to put the pooled resource back into the pool when finished ➢ ScopedValues don’t help in this scenario ➢ This pattern plays particularly bad with virtual threads
29. ThreadLocals as Cache / Objects Pool ??? Do ScopedValues cover all the possible typical usage patterns of ThreadLocals? ❖ It’s very common to use a ThreadLocal as a thread-safe object pool ➢ Easy to implement and use ➢ Doesn’t require an explicit lifecycle: it is not necessary to put the pooled resource back into the pool when finished ➢ ScopedValues don’t help in this scenario ➢ This pattern plays particularly bad with virtual threads
30. ThreadLocals as Cache / Objects Pool into the wild …
31. … and the fix 🎉
32. Plugging a different thread pool as carriers Can we outperform the default Fork/Join pool based carrier thread pool?
33. Plugging a different thread pool as carriers Can we outperform the default Fork/Join pool based carrier thread pool? Let’s put at work Game of Life again!
34. Plugging a different thread pool as carriers Can we outperform the default Fork/Join pool based carrier thread pool? Let’s put at work Game of Life again!
35. Plugging a different thread pool as carriers Can we outperform the default Fork/Join pool based carrier thread pool? Benchmark (executionStrategy) Mode Cnt Score Error Units GameOfLifeBenchmark.benchmark Native thrpt 40 40.317 ± 0.353 ops/s GameOfLifeBenchmark.benchmark ForkJoinVirtual thrpt 40 244.639 ± 1.419 ops/s GameOfLifeBenchmark.benchmark FixedCarrierPoolVirtual thrpt 40 129.525 ± 5.346 ops/s GameOfLifeBenchmark.benchmark PinnedCarrierVirtual thrpt 40 59.071 ± 1.532 ops/s Let’s put at work Game of Life again!
36. So why we may want to plug a different carrier? The Quarkus use case - Reactive ≊ Virtual Threads
37. So why we may want to plug a different carrier? The Quarkus use case - Reactive ≊ Virtual Threads ≊
38. So why we may want to plug a different carrier? The Quarkus use case - The Event Loop as carrier @RunOnVirtualThread
39. So why we may want to plug a different carrier? The Quarkus use case - The Event Loop as carrier → @RunOnVirtualThread
40. Why not using the event-loop as carrier thread??? ❖ No (more) API to do that in Loom - API removed in fall 2021
41. Why not using the event-loop as carrier thread??? ❖ No (more) API to do that in Loom - API removed in fall 2021 ❖ It lead to…. deadlocks!
42. What’s going on ??? Main EventLoop/Carrier VirtualThread newThread execute lockAcquired.join() lockAcquired.join()
43. What’s going on ??? Main EventLoop/Carrier VirtualThread newThread execute lockAcquired.join() lockAcquired.join() lock() lockAcquired.complete()
44. What’s going on ??? Main EventLoop/Carrier VirtualThread newThread execute lockAcquired.join() lockAcquired.join() lock() lockAcquired.complete() lock() lock() busy wait for both carrier and vThread to ask the lock
45. What’s going on ??? Main EventLoop/Carrier VirtualThread newThread execute lockAcquired.join() lockAcquired.join() lock() lockAcquired.complete() lock() lock() busy wait for both carrier and vThread to ask the lock unlock() wait for vThread to acquire the lock cannot make any progress and acquire the lock since its own carrier is blocked
46. Conclusions ❖ Virtual threads are not faster threads ⇒ they don’t magically execute more instructions per second than native ones do. ❖ Virtual threads are not drop in replacement for native threads ⇒ they are a different (new) tool and you have to know their features and characteristics in order to use them appropriately. ❖ Virtual threads are generally not a good fit for CPU bound tasks ⇒ lack of fairness and the performance cost of coroutines can be problematic. ❖ What virtual threads are really good for is waiting ⇒ their goal is maximizing the utilization of external resources and then improving throughput, without affecting readability.
47. Quarkus Virtual Threads - REST / HTTP - Hibernate, Bean Validation, Transactions - Kafka, AMQP, JMS - gRPC - Scheduled tasks - Event Bus - … Learn More