Gerrit Grunwald on What the CRaC... SUPERFAST JVM STARTUP

What the CRaC... SUPERFAST JVM STARTUP

Gerrit Grunwald | Developer Advocate | Azul | @hansolo_ ABOUTME.

JAVA IS GREAT...

VIBRANT COMMUNITY...

HUNDREDS OF JUGs...

THOUSANDS OF FOSS PROJECTS...

JAVA VIRTUAL MACHINE JAVA VIRTUAL MACHINE

HOW DOES IT WORK...

MyClass.java MyClass.class SOURCE CODE COMPILER BYTE CODE

MyClass.class BYTE CODE CLASS LOADER JVM MEMORY

JVM MEMORY 􀊫 􀈒 EXECUTION ENGINE

EXECUTION ENGINE Interpreter C1 JIT Compiler (client) C2 JIT Compiler
(server) 􀊫 Pro fi ler 􀈒 Garbage Collector

(server) 􀊫 Pro fi ler 􀈒 Garbage Collector Tiered compilia ti on DEFAULT SINCE JDK 8

(server) ti on DEFAULT SINCE JDK 8

INTERPRETER Converts ByteCode into instruction set of CPU Detects hot
spots by counting method calls and loop back edges JVM THRESHOLD REACHED

Pass the "hot" code to C1 JIT Compiler JVM C1
JIT COMPILER Compiles code as quickly as possible with low optimisation

C1 JIT COMPILER Compiles code as quickly as possible with
low optimisation Pro fi les the running code (detecting hot code) JVM THRESHOLD REACHED

Pass the "hot" code to C2 JIT Compiler JVM C2
JIT COMPILER Compiles code with best optimisation possible (slower)

EXECUTION CYCLE

INTERPRETA TIO N EXECUTION CYCLE Slow (Execution Level 0)

PROFILING INTERPRETA TIO N EXECUTION CYCLE Finding "hot spots" Slow
(Execution Level 0)

CO MPILING C1 PROFILING INTERPRETA TIO N EXECUTION CYCLE Fast
compile, low optimisation (Execution Level 3) Finding "hot spots" Slow (Execution Level 0)

PROFILIN G CO MPILING C1 PROFILING INTERPRETA TIO N EXECUTION
CYCLE Finding "hot code" Fast compile, low optimisation (Execution Level 3) Finding "hot spots" Slow (Execution Level 0)

COMPILING C2 PROFILIN G CO MPILING C1 PROFILING INTERPRETA TIO
N EXECUTION CYCLE Finding "hot code" Fast compile, low optimisation (Execution Level 3) Finding "hot spots" Slow (Execution Level 0) Slower compile, high optimisation (Execution Level 4)

D EO PTIMISATION COMPILING C2 PROFILIN G CO MPILING C1
PROFILING INTERPRETA TIO N EXECUTION CYCLE Can happen (performance hit) Slower compile, high optimisation (Execution Level 4) Finding "hot code" Fast compile, low optimisation (Execution Level 3) Finding "hot spots" Slow (Execution Level 0)

DEOPTIMISATION

DEOPTIMISATION e.g. BRANCH ANALYSIS value > 9 bias = compute(value)
bias = 1 Math.log10(bias + 99) TRUE FALSE int computeMagnitude(int value) { int bias; if (value > 9) { bias = compute(value); } else { bias = 1: } return Math.log10(bias + 99); }

DEOPTIMISATION e.g. BRANCH ANALYSIS value > 9 bias = compute(value)
bias = 1 Math.log10(bias + 99) TRUE FALSE int computeMagnitude(int value) { int bias; if (value > 9) { bias = compute(value); } else { bias = 1: } return Math.log10(bias + 99); } Value was never greater than 9

DEOPTIMISATION e.g. BRANCH ANALYSIS int computeMagnitude(int value) { if (value
> 9) { uncommonTrap(); } int bias = 1; return Math.log10(bias + 99); } value > 9 deoptimise Math.log10(1 + 99) TRUE FALSE

> 9) { uncommonTrap(); } return Math.log10(100); } value > 9 deoptimise Math.log10(100) TRUE FALSE

> 9) { uncommonTrap(); } return 2; } value > 9 deoptimise return 2 TRUE FALSE

DEOPTIMISATION e.g. BRANCH ANALYSIS int computeMagnitude(int value) { int bias;
if (value > 9) { bias = compute(value); } else { bias = 1: } return Math.log10(bias + 99); } INTERPRETER C1 C2 value > 9 bias = compute(value) bias = 1 Math.log10(bias + 99) TRUE FALSE

JVM PERFORMANCE GRAPH GarbageCollector pauses Deoptimisations Performance Interpreter C1 Compiler
C2 Compiler

JVM STARTUP

JVM STARTUP JVM Load & Initialize Optimization FAST JVM START

JVM STARTUP JVM Load & Initialize Optimization JVM Load application
classes Initialize all resources Kick off application speci fi c logic Optimization FAST TAKES A BIT JVM START APPLICATION START

classes Initialize all resources Kick off application speci fi c logic Optimization FAST TAKES A BIT Generally referred to as JVM Startup (Time to fi rst response) JVM START APPLICATION START

classes Initialize all resources Kick off application speci fi c logic Optimization JVM Optimizing (Compile/Decompile) FAST TAKES A BIT TAKES SOME TIME Generally referred to as JVM Startup (Time to fi rst response) App Apply application speci fi c workloads JVM START APPLICATION START APPLICATION WARMUP

classes Initialize all resources Kick off application speci fi c logic Optimization JVM Optimizing (Compile/Decompile) FAST TAKES A BIT TAKES SOME TIME Generally referred to as JVM Startup (Time to fi rst response) Generally referred to as JVM Warmup (Time to n operations) App Apply application speci fi c workloads JVM START APPLICATION START APPLICATION WARMUP

THAT'S GREAT...

...BUT...

...IT TAKES TIME !

MICROSERVICE ENVIRONMENT

MICROSERVICE ENVIRONMENT FIRST RUN JVM STARTUP Performance SECOND RUN JVM
STARTUP Performance THIRD RUN JVM STARTUP Performance

WOULDN'T IT BE GREAT...? FIRST RUN JVM STARTUP Performance SECOND
RUN NO STARTUP OVERHEAD Performance THIRD RUN NO STARTUP OVERHEAD Performance

SOLUTIONS...?

CLASS DATA SHARING

WHAT ABOUT CDS? Dump internal class representations into fi le

Shared on each JVM start (CDS)

Shared on each JVM start (CDS) No optimization or hotspot detection

Shared on each JVM start (CDS) No optimization or hotspot detection Only reduces class loading time

Shared on each JVM start (CDS) No optimization or hotspot detection Only reduces class loading time Startup up to 2 seconds faster

Shared on each JVM start (CDS) No optimization or hotspot detection Only reduces class loading time Startup up to 2 seconds faster Good info from Ionut Balosin

MyClass.class BYTE CODE CLASS LOADER JVM MEMORY CDS

AHEAD OF TIME COMPILATION

WHY NOT USE AOT? No interpreting bytecodes

WHY NOT USE AOT? No interpreting bytecodes No analysis of
hotspots

hotspots No runtime compilation of code

hotspots No runtime compilation of code Start at 'full speed', straight away

hotspots No runtime compilation of code Start at 'full speed', straight away GraalVM native image does that PROBLEM SOLVED...?

NOT SO FAST... AOT is, by de fi nition, static

Code is compiled before it is run

Code is compiled before it is run Compiler has no knowledge of how the code will actually run

Code is compiled before it is run Compiler has no knowledge of how the code will actually run Pro fi le Guided Optimisation (PGO) can partially help

JVM PERFORMANCE GRAPH AOT Compiled Code AOT Compiled Code with
Profile Guided Optimisation (not in GraalVM Community) Needs to run once for pro fi ling Performance

JVM PERFORMANCE https://www.baeldung.com/spring-boot-vs-quarkus

JVM PERFORMANCE https://www.baeldung.com/spring-boot-vs-quarkus JVM NATIVE IMAGE

JVM PERFORMANCE https://www.baeldung.com/spring-boot-vs-quarkus 74% 100% 53% 100% JVM NATIVE IMAGE

AOT VS JIT Limited use of method inlining No runtime
bytecode generation Re fl ection is possible but complicated Unable to use speculative optimisations Must be compiled for least common denominator Overall performance will typically be lower Deployed env != Development env. 'Full speed' from the start No overhead to compile code at runtime Small memory footprint Can use aggressive method inlining at runtime Can use runtime bytecode generation Re fl ection is simple Can use speculative optimisations Can even optimise for Haswell, Skylake, Ice Lake etc. Overall performance will typically be higher Deployed env. == Development env. Requires more time to start up (but will be faster) Overhead to compile code at runtime Larger memory footprint AOT JIT

JIT DISADVANTAGES Requires more time to start up (requires many
slow operations to happen before optimisation and faster execution can happen)

slow operations to happen before optimisation and faster execution can happen) CPU overhead to compile code at runtime

slow operations to happen before optimisation and faster execution can happen) CPU overhead to compile code at runtime Larger memory footprint

A DIFFERENT APPROACH

CRIU CHECKPOINT RESTORE IN USERSPACE

Linux project CRIU

Linux project Part of kernel >= 3.11 (2013) CRIU

Linux project Part of kernel >= 3.11 (2013) Freeze a
running container/application CRIU

running container/application Checkpoint its state to disk CRIU

running container/application Checkpoint its state to disk Restore the container/application from the saved data. CRIU

running container/application Checkpoint its state to disk Restore the container/application from the saved data. Used by/integrated in OpenVZ, LXC/LXD, Docker, Podman and others CRIU

Heavily relies on /proc fi le system CRIU

Heavily relies on /proc fi le system It can checkpoint:
Processes and threads Application memory, memory mapped fi les and shared memory Open fi les, pipes and FIFOs Sockets Interprocess communication channels Timers and signals CRIU

Heavily relies on /proc fi le system It can checkpoint:
Processes and threads Application memory, memory mapped fi les and shared memory Open fi les, pipes and FIFOs Sockets Interprocess communication channels Timers and signals Can rebuild TCP connection from one side only CRIU

CRIU CHALLENGES

Restart from saved state on another machine (open fi les,
shared memory etc.) CRIU CHALLENGES

shared memory etc.) Start multiple instances of same state on same machine (PID will be restored which will lead to problems) CRIU CHALLENGES

shared memory etc.) Start multiple instances of same state on same machine (PID will be restored which will lead to problems) A Java Virtual Machine would assume it was continuing its tasks (very dif fi cult to use effectively, e.g. running applications might have open fi les etc.) CRIU CHALLENGES

CRaC Coordinated Restore at Checkpoint

RUNNING APPLICATION Aware of checkpoint being created RUNNING APPLICATION Aware
of restore happening CRaC A way to solve the problems when checkpointing a JVM (e.g. no open fi les, sockets etc.)

CRaC Comes with a simple API

CRaC Comes with a simple API Creates checkpoints using code
or jcmd

or jcmd Throws CheckpointException (in case of open fi les/sockets)

or jcmd Throws CheckpointException (in case of open fi les/sockets) Heap is cleaned, compacted (using JVM safepoint mechanism -> JVM is in a safe state)

CRaC START >java -XX:CRaCCheckpointTo=PATH -jar app.jar RESTORE >java -XX:CRaCRestoreFrom=PATH Additional
command line parameters

openjdk.org/projects/crac Lead by Anton Kozlov (Azul)

CRaC API

<<interface>> Resource beforeCheckpoint() afterRestore() Resource interface (can be noti fi
ed about a Checkpoint and Restore) CRaC API

ed about a Checkpoint and Restore) Classes in application code implement the Resource interface CRaC API

ed about a Checkpoint and Restore) Classes in application code implement the Resource interface Application receives callbacks during checkpointing and restoring CRaC API

ed about a Checkpoint and Restore) Classes in application code implement the Resource interface Application receives callbacks during checkpointing and restoring Makes it possible to close/restore resources (e.g. open fi les, sockets) CRaC API

Resource objects need to be registered with a Context so
that they can receive noti fi cations CRaC API

Resource objects need to be registered with a Context so
that they can receive noti fi cations There is a global Context accessible via the static method Core.getGlobalContext() CRaC API

<<interface>> Resource beforeCheckpoint() afterRestore() Core getGlobalContext() <<abstract>> Context register(Resource) CRaC
API

CREATING A CHECKPOINT

CREATING A CHECKPOINT FROM THE COMMAND LINE: >jcmd YOUR_AWESOME.jar JDK.checkpoint
>jcmd PID JDK.checkpoint

Core.checkpointRestore(); FROM THE CODE: CREATING A CHECKPOINT

WHEN ?

Start your app with -XX:+PrintCompilation WHEN TO CHECKPOINT ?

Start your app with -XX:+PrintCompilation Apply typical workload to your
app WHEN TO CHECKPOINT ?

app Observe the moment the compilations are ramped down WHEN TO CHECKPOINT ?

app Observe the moment the compilations are ramped down Create the checkpoint WHEN TO CHECKPOINT ?

CRAC OVERVIEW

CRaC OVERVIEW JVM

CRaC OVERVIEW JVM APPLICATION JVM startup...

CRaC OVERVIEW JVM APPLICATION RESOURCE 1 RESOURCE 2 beforeCheckpoint() afterRestore()
beforeCheckpoint() afterRestore()

beforeCheckpoint() afterRestore() Register resources in global context

beforeCheckpoint() afterRestore() Warmup the application 􀣔

beforeCheckpoint() afterRestore() jcmd JDK.checkpoint

beforeCheckpoint() afterRestore() JVM noti fi es the resources

beforeCheckpoint() afterRestore() Application closes open resources

beforeCheckpoint() afterRestore() JVM stores checkpoint to disc 􀤄

beforeCheckpoint() afterRestore() Time... 􀉉

beforeCheckpoint() afterRestore() Restore from checkpoint java -XX:CRaCRestoreFrom

beforeCheckpoint() afterRestore() JVM noti fi es the resources

beforeCheckpoint() afterRestore() Application re-open resources

beforeCheckpoint() afterRestore() No JVM startup and no application warmup !!!

TYPICAL USAGE

TYPICAL USAGE... Run app in a docker container

Run app in a docker container Create checkpoint (store in
container or external volume) TYPICAL USAGE...

container or external volume) Commit the state of container (only if checkpoint in container) TYPICAL USAGE...

container or external volume) Commit the state of container (only if checkpoint in container) Start the container (point jvm to container or external volume) TYPICAL USAGE...

LINUX ONLY X64 / AARCH64

WINDOWS MACOS ?

ORG.CRAC

Designed to provide smooth CRaC adoption ORG.CRAC

Designed to provide smooth CRaC adoption Total mirror of jdk.crac
api at compile-time ORG.CRAC

api at compile-time Can be used with any OpenJDK implementation ORG.CRAC

api at compile-time Can be used with any OpenJDK implementation Detects CRaC implementation at runtime ORG.CRAC

api at compile-time Can be used with any OpenJDK implementation Detects CRaC implementation at runtime No CRaC support -> won't call CRaC speci fi c code ORG.CRAC

api at compile-time Can be used with any OpenJDK implementation Detects CRaC implementation at runtime No CRaC support -> won't call CRaC speci fi c code CRaC support -> will forward all CRaC speci fi c calls to jdk.crac ORG.CRAC

<dependency> <groupId>org.crac</groupId> <artifactId>crac</artifactId> <version>0.1.3</version> </dependency> implementation 'org.crac:crac:0.1.3' ORG.CRAC

github.com/CRaC/org.crac ORG.CRAC

COMPATIBILITY

COMPATIBILITY... Upgrade (Haswell -> restore: Ice Lake, no problem)

Upgrade (Haswell -> restore: Ice Lake, no problem) Downgrade (Ice
Lake -> restore: Haswell, problematic) COMPATIBILITY...

Lake -> restore: Haswell, problematic) Solved in CRaC by speci fi c fl ag (little drop in performance) COMPATIBILITY...

Lake -> restore: Haswell, problematic) Solved in CRaC by speci fi c fl ag (little drop in performance) Node groups stick to same cpu architecture COMPATIBILITY...

Lake -> restore: Haswell, problematic) Solved in CRaC by speci fi c fl ag (little drop in performance) Node groups stick to same cpu architecture Virtualized Linux environments work on all OS's (as long as cpu architecture is x64/aarch64) COMPATIBILITY...

DEMO...

DEMO Name service

DEMO Name service Load 258 000 names from json fi
le at startup

le at startup Return 5 random names for girls

le at startup Return 5 random names for girls Return 5 random names for boys

public class Main implements Resource { public Main() { System.out.println("Start
without CRaC"); Core.getGlobalContext().register(Main.this); init(); printRandomGirlNames(5); printRandomBoyNames(5); System.out.println("Time to first response: " + ((System.nanoTime() - startTime) / MILLISECOND_IN_NS) + "ms"); } private void init() { allNames = loadNames(); } @Override public void beforeCheckpoint(Context<? extends Resource> context) throws Exception {} @Override public void afterRestore(Context<? extends Resource> context) throws Exception {} DEMO CALLED AT FIRST STARTUP

public class Main implements Resource { public Main() {} private
void init() {} @Override public void beforeCheckpoint(Context<? extends Resource> context) throws Exception {} @Override public void afterRestore(Context<? extends Resource> context) throws Exception { System.out.println("Start using CRaC"); startTime = System.nanoTime(); printRandomGirlNames(5); printRandomBoyNames(5); System.out.println("Time to first response: " + ((System.nanoTime() - startTime) / MILLISECOND_IN_NS) + "ms"); } DEMO CALLED AFTER RESTORE

>docker run -it --rm --name crac8 hansolo/crac8 java -jar /opt/
app/crac8-17.0.0.jar >docker run -it --privileged --rm --name crac8 hansolo/ crac8:checkpoint java -XX:CRaCRestoreFrom=/opt/crac-files DEMO SHELL 1 SHELL 2 NORMAL START AFTER RESTORE

>docker run -it --rm --name crac8 hansolo/crac8 java -jar /opt/
app/crac8-17.0.0.jar DEMO SHELL 1 SHELL 2 >docker run -it --privileged --rm --name crac8 hansolo/ crac8:checkpoint java -XX:CRaCRestoreFrom=/opt/crac-files Folder where the checkpoint will be stored NORMAL START AFTER RESTORE

DEMO SHELL 1 SHELL 2 >docker run -it --rm --name
crac6 hansolo/crac6 java -jar /opt/ app/crac6-17.0.0.jar JVM startup time -> 20ms Start without CRaC Loading 258000 names took 1292ms 5 random names for girls: Colleen Dedra Elisabeth Frankie Samantha 5 random names for boys: Clayton Cliff Hollis Johnnie Winfield Time to frist response: 1360ms >docker run -it --privileged --rm --name hansolo/crac6:checkpoint java -XX:CRaCRestoreFrom=/opt/crac-files Start using CRaC 5 random names for girls: Adelaide Angelina Christa Kathleen Rebecca 5 random names for boys: Antione Burl Jerel Trenton Wyatt Time to frist response: 48ms NORMAL START AFTER RESTORE

github.com/HanSolo/crac8

FRAMWORK SUPPORT ?

FRAMEWORK SUPPORT ? Micronaut (good support)

FRAMEWORK SUPPORT ? Micronaut (good support) Quarkus (rudimentary support)

FRAMEWORK SUPPORT ? Micronaut (good support) Quarkus (rudimentary support) Spring
(will get support with Spring 6.1)

OK...BUT HOW GOOD IS IT...?

Time to fi rst opera ti on Spring-Boot Micronaut Quarkus
xml-transform [ms] 0 1250 2500 3750 5000 4,352 980 1,001 3,898 OpenJDK

Time to fi rst opera ti on Spring-Boot Micronaut Quarkus
xml-transform [ms] 0 1250 2500 3750 5000 53 33 46 38 4,352 980 1,001 3,898 OpenJDK OpenJDK on CRaC

SUMMARY...

SUMMARY... CRaC is a way to pause and restore a
JVM based application

CRaC is a way to pause and restore a JVM
based application It doesn't require a closed world as with a native image SUMMARY...

based application It doesn't require a closed world as with a native image Extremely fast time to full performance level SUMMARY...

based application It doesn't require a closed world as with a native image Extremely fast time to full performance level No need for hotspot identi fi cation, method compiles, recompiles and deoptimisations SUMMARY...

based application It doesn't require a closed world as with a native image Extremely fast time to full performance level No need for hotspot identi fi cation, method compiles, recompiles and deoptimisations Improved throughput from start SUMMARY...

based application It doesn't require a closed world as with a native image Extremely fast time to full performance level No need for hotspot identi fi cation, method compiles, recompiles and deoptimisations Improved throughput from start CRaC is an OpenJDK project SUMMARY...

based application It doesn't require a closed world as with a native image Extremely fast time to full performance level No need for hotspot identi fi cation, method compiles, recompiles and deoptimisations Improved throughput from start CRaC is an OpenJDK project CRaC can save infrastructure cost SUMMARY...

CPU Utilization 0 % 25 % 50 % 75 %
100 % Time INFRASTRUCTURE COST Checkpoint JVM startup time Interpretation + Compilation Overhead Start after restore Eliminates startup time Eliminates cpu overhead

WANNA KNOW MORE ?

github.com/CRaC INFORMATION...

azul.com JDK 17.0.8 LINUX X64 / AARCH64 DOWNLOAD...

SUPPORTED VERSION ?

SUPPORTED VERSION... No pre-build/early access

No pre-build/early access Getting updates (17.0.9, 17.0.10, 17.0.11...) SUPPORTED VERSION...

No pre-build/early access Getting updates (17.0.9, 17.0.10, 17.0.11...) Commercial support
(Patches, Hotline, etc.) SUPPORTED VERSION...

(Patches, Hotline, etc.) Support for x64 and aarch64 SUPPORTED VERSION...

(Patches, Hotline, etc.) Support for x64 and aarch64 Possibility to get JDK8 or JDK11 with CRaC (If high demand) SUPPORTED VERSION...

THANK YOU

Gerrit Grunwald on What the CRaC... SUPERFAST J...

Gerrit Grunwald on What the CRaC... SUPERFAST JVM STARTUP

Transcript