Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Game of Loom 2: life and dead(lock) of a virtua...

Game of Loom 2: life and dead(lock) of a virtual thread

Virtual threads finally exited their development and preview phases and with JVM 21 are available as a stable and supported Java feature. During the latest Devoxx edition I started exploring the characteristics of virtual threads and their performance implications putting them at work, with a funny but practical example, using a Conway's Game of Life implementation based on Project Loom. Starting from the same playground this time we will explore more in depth the internal implementation details of virtual threads, trying to answer to some interesting questions. What does it mean in practice that virtual threads aren't preemptive? Are local variables enough to replace ThreadLocals in all possible scenarios? What does it happen if you try to replace the fork/join pool, used as default carrier thread pool, with something different? At the end we will conclude this exploration trying to experience the multithreaded programming equivalent of the sound of one hand clapping or how virtual threads make it possible to cause a deadlock using one single lock.

Mario Fusco

October 04, 2023
Tweet

More Decks by Mario Fusco

Other Decks in Programming

Transcript

  1. Game of Loom 2: life and dead(lock) of a virtual

    thread by Mario Fusco @mariofusco
  2. Where did we leave? A recap on virtual threads …

    ❖ Virtual threads make a better use of OS threads getting rid of the 1:1 relationship with operating system threads ➢ Many virtual threads can be multiplexed on the same platform thread (carrier)
  3. Where did we leave? A recap on virtual threads …

    ❖ Virtual threads make a better use of OS threads getting rid of the 1:1 relationship with operating system threads ➢ Many virtual threads can be multiplexed on the same platform thread (carrier) ❖ When a virtual thread calls a blocking operation the JDK performs a nonblocking OS call and automatically suspends the virtual thread until the operation finishes ➢ Virtual threads perform blocking calls without consuming resources, thus allowing to write asynchronous code in a way that looks synchronous ➢ What virtual threads are really good for is … waiting
  4. Where did we leave? A recap on virtual threads …

    ❖ Virtual threads make a better use of OS threads getting rid of the 1:1 relationship with operating system threads ➢ Many virtual threads can be multiplexed on the same platform thread (carrier) ❖ When a virtual thread calls a blocking operation the JDK performs a nonblocking OS call and automatically suspends the virtual thread until the operation finishes ➢ Virtual threads perform blocking calls without consuming resources, thus allowing to write asynchronous code in a way that looks synchronous ➢ What virtual threads are really good for is … waiting ❖ Virtual threads are lightweight ➢ 200-300B metadata ➢ Pay-as-you-go stack (allocated on heap) ➢ Some ns (or below 1µs) for context switch (in user space)
  5. Should we really use virtual threads *EVERYWHERE*? ❖ Not ideal

    for CPU-bound task ➢ Lack of fairness due to the fact that virtual threads are not pre-emptive ➢ Overall performance concerns caused by overhead of virtual threads scheduler
  6. Should we really use virtual threads *EVERYWHERE*? ❖ Not ideal

    for CPU-bound task ➢ Lack of fairness due to the fact that virtual threads are not pre-emptive ➢ Overall performance concerns caused by overhead of virtual threads scheduler ❖ Not virtual-threads friendly frameworks & tools ➢ Use of synchronize can lead to pinning ➢ Use of ThreadLocal
  7. Should we really use virtual threads *EVERYWHERE*? ❖ Not ideal

    for CPU-bound task ➢ Lack of fairness due to the fact that virtual threads are not pre-emptive ➢ Overall performance concerns caused by overhead of virtual threads scheduler ❖ Not virtual-threads friendly frameworks & tools ➢ Use of synchronize can lead to pinning ➢ Use of ThreadLocal ❖ Poorly customizable ➢ Carrier native thread pool is not pluggable (but for a very good reason)
  8. Lack of fairness in CPU bound tasks Virtual Threads are

    not pre-emptive, they cannot be descheduled while running heavy calculations without ever calling a JDK’s blocking methods, so they are not a good fit for CPU-bound tasks when fairness is key Interrupts the processing of a task and transfers the CPU to another process Non-preemptive operation usually proceeds towards completion uninterrupted
  9. Lack of fairness in CPU bound tasks Virtual Threads are

    not pre-emptive, they cannot be descheduled while running heavy calculations without ever calling a JDK’s blocking methods, so they are not a good fit for CPU-bound tasks when fairness is key
  10. The cost of a continuation In Project Loom, the word

    continuation means delimited continuation, also sometimes called a coroutine. It can be thought of as sequential code that may suspend or yield execution at some point by itself and can be resumed by a caller. When a virtual thread hits a blocking call its continuation is suspended and the JVM unmounts the thread from its carrier …
  11. The cost of a continuation In Project Loom, the word

    continuation means delimited continuation, also sometimes called a coroutine. It can be thought of as sequential code that may suspend or yield execution at some point by itself and can be resumed by a caller. When a virtual thread hits a blocking call its continuation is suspended and the JVM unmounts the thread from its carrier … … but this also has a cost!
  12. ThreadLocals as thread scoped variables enum RightsLevel { ADMIN, GUEST

    } record Principal(RightsLevel rights) { public boolean canOpen() { return rights == RightsLevel.ADMIN; } } record Request(boolean authorized) { } class Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } }
  13. ThreadLocals as thread scoped variables static class DBConnection { static

    DBConnection open() { var principal = Server.PRINCIPAL.get(); if (!principal.canOpen()) throw new IllegalArgumentException(); return new DBConnection(); } String doQuery() { return "Result"; } } static class Application { public static String handle(Request request) { return DBConnection.open().doQuery(); } } enum RightsLevel { ADMIN, GUEST } record Principal(RightsLevel rights) { public boolean canOpen() { return rights == RightsLevel.ADMIN; } } record Request(boolean authorized) { } class Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } }
  14. ThreadLocals as thread scoped variables A ThreadLocal is a construct

    that allows us to store data accessible only by a specific thread. static class DBConnection { static DBConnection open() { var principal = Server.PRINCIPAL.get(); if (!principal.canOpen()) throw new IllegalArgumentException(); return new DBConnection(); } String doQuery() { return "Result"; } } static class Application { public static String handle(Request request) { return DBConnection.open().doQuery(); } } enum RightsLevel { ADMIN, GUEST } record Principal(RightsLevel rights) { public boolean canOpen() { return rights == RightsLevel.ADMIN; } } record Request(boolean authorized) { } class Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } }
  15. ThreadLocals as (virtual) thread scoped variables public static void main(String[]

    args) { var server = new Server(); var authorized = Thread.ofVirtual().name("Authorized").unstarted(() -> callServer(server, true)); var notAuthorized = Thread.ofVirtual().name("NOT Authorized").unstarted(() -> callServer(server, false)); authorized.start(); notAuthorized.start(); try { Thread.sleep(1000L); } catch (InterruptedException e) { throw new RuntimeException(e); } } private static void callServer(Server server, boolean auth) { var result = server.serve(new Request(auth)); System.out.println( "thread " + Thread.currentThread().getName() + " got result " + result ); }
  16. ThreadLocals as (virtual) thread scoped variables public static void main(String[]

    args) { var server = new Server(); var authorized = Thread.ofVirtual().name("Authorized").unstarted(() -> callServer(server, true)); var notAuthorized = Thread.ofVirtual().name("NOT Authorized").unstarted(() -> callServer(server, false)); authorized.start(); notAuthorized.start(); try { Thread.sleep(1000L); } catch (InterruptedException e) { throw new RuntimeException(e); } } private static void callServer(Server server, boolean auth) { var result = server.serve(new Request(auth)); System.out.println( "thread " + Thread.currentThread().getName() + " got result " + result ); } Exception in thread "NOT Authorized" java.lang.IllegalArgumentException at org.mfusco.loom.experiments.threadlocal.ThreadLocalMain$DBConnection.open(ThreadLocalMain.java:48) at org.mfusco.loom.experiments.threadlocal.ThreadLocalMain$Application.handle(ThreadLocalMain.java:41) thread Authorized got result Result
  17. ThreadLocals as (virtual) thread scoped variables So? Everything is fine,

    right? What is the problem to use ThreadLocal with virtual threads?
  18. ThreadLocals as (virtual) thread scoped variables So? Everything is fine,

    right? What is the problem to use ThreadLocal with virtual threads? ❖ That we can have a huge number of virtual threads, and each virtual thread will have its own ThreadLocal ➢ The memory footprint of the application may quickly become very high ➢ ThreadLocal will be useless in a one-thread-per-request scenario since data won’t be shared between different requests
  19. ScopedValues to the rescue class Server { final static ThreadLocal<Principal>

    PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } }
  20. ScopedValues to the rescue (still preview in JDK 21) class

    Server { final static ThreadLocal<Principal> PRINCIPAL = new ThreadLocal<>(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; PRINCIPAL.set( new Principal(level) ); return Application.handle( request ); } } class Server { final static ScopedValue<Principal> PRINCIPAL = ScopedValue.newInstance(); String serve(Request request) { var level = request.authorized() ? RightsLevel.ADMIN : RightsLevel.GUEST; try { return ScopedValue.where(PRINCIPAL, new Principal(level)) .call(() -> Application.handle(request)); } catch (Exception e) { throw new RuntimeException(e); } } }
  21. ThreadLocals as Cache / Objects Pool ??? Do ScopedValues cover

    all the possible typical usage patterns of ThreadLocals?
  22. ThreadLocals as Cache / Objects Pool ??? Do ScopedValues cover

    all the possible typical usage patterns of ThreadLocals? ❖ It’s very common to use a ThreadLocal as a thread-safe object pool ➢ Easy to implement and use ➢ Does not require an explicit lifecycle: it is not necessary to put the pooled resource back into the pool when finished
  23. ThreadLocals as Cache / Objects Pool ??? Do ScopedValues cover

    all the possible typical usage patterns of ThreadLocals? ❖ It’s very common to use a ThreadLocal as a thread-safe object pool ➢ Easy to implement and use ➢ Does not require an explicit lifecycle: it is not necessary to put the pooled resource back into the pool when finished ➢ ScopedValues don’t help in this scenario ➢ This pattern plays particularly bad with virtual threads
  24. ThreadLocals as Cache / Objects Pool ??? Do ScopedValues cover

    all the possible typical usage patterns of ThreadLocals? ❖ It’s very common to use a ThreadLocal as a thread-safe object pool ➢ Easy to implement and use ➢ Doesn’t require an explicit lifecycle: it is not necessary to put the pooled resource back into the pool when finished ➢ ScopedValues don’t help in this scenario ➢ This pattern plays particularly bad with virtual threads
  25. Plugging a different thread pool as carriers Can we outperform

    the default Fork/Join pool based carrier thread pool?
  26. Plugging a different thread pool as carriers Can we outperform

    the default Fork/Join pool based carrier thread pool? Let’s put at work Game of Life again!
  27. Plugging a different thread pool as carriers Can we outperform

    the default Fork/Join pool based carrier thread pool? Let’s put at work Game of Life again!
  28. Plugging a different thread pool as carriers Can we outperform

    the default Fork/Join pool based carrier thread pool? Benchmark (executionStrategy) Mode Cnt Score Error Units GameOfLifeBenchmark.benchmark Native thrpt 40 40.317 ± 0.353 ops/s GameOfLifeBenchmark.benchmark ForkJoinVirtual thrpt 40 244.639 ± 1.419 ops/s GameOfLifeBenchmark.benchmark FixedCarrierPoolVirtual thrpt 40 129.525 ± 5.346 ops/s GameOfLifeBenchmark.benchmark PinnedCarrierVirtual thrpt 40 59.071 ± 1.532 ops/s Let’s put at work Game of Life again!
  29. So why we may want to plug a different carrier?

    The Quarkus use case - Reactive ≊ Virtual Threads
  30. So why we may want to plug a different carrier?

    The Quarkus use case - Reactive ≊ Virtual Threads ≊
  31. So why we may want to plug a different carrier?

    The Quarkus use case - The Event Loop as carrier @RunOnVirtualThread
  32. So why we may want to plug a different carrier?

    The Quarkus use case - The Event Loop as carrier → @RunOnVirtualThread
  33. Why not using the event-loop as carrier thread??? ❖ No

    (more) API to do that in Loom - API removed in fall 2021
  34. Why not using the event-loop as carrier thread??? ❖ No

    (more) API to do that in Loom - API removed in fall 2021 ❖ It lead to…. deadlocks!
  35. What’s going on ??? Main EventLoop/Carrier VirtualThread newThread execute lockAcquired.join()

    lockAcquired.join() lock() lockAcquired.complete() lock() lock() busy wait for both carrier and vThread to ask the lock
  36. What’s going on ??? Main EventLoop/Carrier VirtualThread newThread execute lockAcquired.join()

    lockAcquired.join() lock() lockAcquired.complete() lock() lock() busy wait for both carrier and vThread to ask the lock unlock() wait for vThread to acquire the lock cannot make any progress and acquire the lock since its own carrier is blocked
  37. Conclusions ❖ Virtual threads are not faster threads ⇒ they

    don’t magically execute more instructions per second than native ones do. ❖ Virtual threads are not drop in replacement for native threads ⇒ they are a different (new) tool and you have to know their features and characteristics in order to use them appropriately. ❖ Virtual threads are generally not a good fit for CPU bound tasks ⇒ lack of fairness and the performance cost of coroutines can be problematic. ❖ What virtual threads are really good for is waiting ⇒ their goal is maximizing the utilization of external resources and then improving throughput, without affecting readability.
  38. Quarkus Virtual Threads - REST / HTTP - Hibernate, Bean

    Validation, Transactions - Kafka, AMQP, JMS - gRPC - Scheduled tasks - Event Bus - … Learn More