Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Four Approaches to Reducing Java Startup Time

Four Approaches to Reducing Java Startup Time

Avatar for Paul

Paul

May 24, 2025
Tweet

More Decks by Paul

Other Decks in Technology

Transcript

  1. Catherine Edelveis 🥑 Developer Advocate at BellSoft 😍Love Java, Spring,

    JavaFX 👩‍💻Tech writer cat_edelveis cat-edelveis.bsky.social
  2. About BellSoft Member of: JCP Executive Committee OpenJDK Vulnerability Group

    GraalVM Advisory Board Linux Foundation Cloud Native Computing Foundation Founded in 2017 by Java and Linux experts with 15+ years of experience working at Sun/Oracle.
  3. About BellSoft Products: Liberica JDK Liberica Native Image Kit Alpaquita

    Linux Liberica is the JDK officially recommended by
  4. Our startup is doing well! Java 21 LTS Spring Boot

    3.4 Spring AI MongoDB, Redis JTE Deploying to the cloud @GetMapping(value = "/assistant", produces = TEXT_HTML_VALUE) public JteModel assistant() { return templates.assistantView("Bot Assistant - Chat"); }
  5. Starting point: Dockerfile FROM bellsoft/liberica-runtime-container:jdk-21-musl as builder ARG project ENV

    project=${project} WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ RUN cd ${project} && ./mvnw package FROM bellsoft/liberica-runtime-container:jre-21-musl ARG project ENV project=${project} RUN apk add curl WORKDIR /app COPY --from=builder /app/${project}/target/*.jar /app/app.jar ["j " " j " "/ / j "]
  6. Starting point: Dockerfile FROM bellsoft/liberica-runtime-container:jdk-21-musl as builder RUN cd ${project}

    && ./mvnw package ARG project ENV project=${project} WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ FROM bellsoft/liberica-runtime-container:jre-21-musl ARG project ENV project=${project} RUN apk add curl WORKDIR /app COPY --from=builder /app/${project}/target/*.jar /app/app.jar ["j " " j " "/ / j "]
  7. Starting point: Dockerfile FROM bellsoft/liberica-runtime-container:jre-21-musl COPY --from=builder /app/${project}/target/*.jar /app/app.jar /

    j ARG project ENV project=${project} WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ RUN cd ${project} && ./mvnw package ARG project ENV project=${project} RUN apk add curl WORKDIR /app ENTRYPOINT ["java", "-jar", "/app/app.jar"]
  8. Starting point: Dockerfile ENTRYPOINT ["java", "-jar", "/app/app.jar"] / j ARG

    project ENV project=${project} WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ RUN cd ${project} && ./mvnw package FROM bellsoft/liberica-runtime-container:jre-21-musl ARG project ENV project=${project} RUN apk add curl WORKDIR /app COPY --from=builder /app/${project}/target/*.jar /app/app.jar
  9. Starting point: Startup Total startup time of both services: 6.5

    s docker compose logs chat-api bot-assistant | grep Started Started BotAssistantApplication in 2.108 seconds (process running for 3.181) Started ChatApiApplication in 2.83 seconds (process running for 3.469)
  10. We are growing! So many new users! Scale the services

    to meet the traffic But wait... Why the rollout is so slow?
  11. We are an agile team, we need a fast feedback

    loop! Need faster rollout for faster feedback
  12. We are an agile team, we need a fast feedback

    loop! Need faster rollout for faster feedback The longer the start, the slower the rollout
  13. We are an agile team, we need a fast feedback

    loop! Need faster rollout for faster feedback The longer the start, the slower the rollout Maybe Java is not cut out for the cloud?
  14. We are an agile team, we need a fast feedback

    loop! Need faster rollout for faster feedback The longer the start, the slower the rollout Maybe Java is not cut out for the cloud? Maybe it’s not too late to migrate to Go?
  15. They are very different, but have something in common: They

    care about their software product and can’t sit idle in the troubled time.
  16. AppCDS Save class metadata to the disc and read it

    from the archive Even better with Spring AOT (load even more classes!) Considerations: Use the same JVM Use the same classpath Bigger Docker image due to the archive Legal JVM Dopes Fo Legal JVM Dopes Fo… …
  17. AppCDS Save class metadata to the disc and read it

    from the archive Even better with Spring AOT (load even more classes!) Considerations: Use the same JVM Use the same classpath Bigger Docker image due to the archive Let's take it for a spin! Legal JVM Dopes Fo Legal JVM Dopes Fo… …
  18. FROM bellsoft/liberica-runtime-container:jdk-21.0.7_9-musl as builder RUN cd ${project} && ./mvnw package

    ARG project ENV project=${project} WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ FROM bellsoft/liberica-runtime-container:jre-21.0.7_9-cds-musl as optimizer ARG project ENV project=${project} WORKDIR /app COPY --from=builder /app/${project}/target/*.jar app.jar RUN java -Djarmode=tools -jar app.jar extract --layers --destination extracted
  19. FROM bellsoft/liberica-runtime-container:jre-21.0.7_9-cds-musl as optimizer COPY --from=builder /app/${project}/target/*.jar app.jar RUN java

    -Djarmode=tools -jar app.jar extract --layers --destination extracted ADD ../pom.xml ./ RUN cd ${project} && ./mvnw package ARG project ENV project=${project} WORKDIR /app FROM bellsoft/liberica-runtime-container:jre-21.0.7_9-cds-musl RUN apk add curl WORKDIR /app COPY --from=optimizer /app/extracted/dependencies/ ./
  20. FROM bellsoft/liberica-runtime-container:jre-21.0.7_9-cds-musl COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./

    COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ RUN java -Djarmode=tools -jar app.jar extract --layers --destination extracted RUN apk add curl WORKDIR /app RUN java -Dspring.aot.enabled=true \ -XX:ArchiveClassesAtExit=./application.jsa \ -Dspring.context.exit=onRefresh -jar /app/app.jar ENTRYPOINT ["java", "-Dspring.aot.enabled=true", \ "-XX:SharedArchiveFile=application.jsa", \ "-jar", "/app/app.jar"]
  21. RUN java -Dspring.aot.enabled=true \ -XX:ArchiveClassesAtExit=./application.jsa \ -Dspring.context.exit=onRefresh -jar /app/app.jar j

    j j pp j y FROM bellsoft/liberica-runtime-container:jre-21.0.7_9-cds-musl RUN apk add curl WORKDIR /app COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ ENTRYPOINT ["java", "-Dspring.aot.enabled=true", \ "-XX:SharedArchiveFile=application.jsa", \ "-jar", "/app/app.jar"]
  22. ENTRYPOINT ["java", "-Dspring.aot.enabled=true", \ "-XX:SharedArchiveFile=application.jsa", \ "-jar", "/app/app.jar"] j j

    j pp j y FROM bellsoft/liberica-runtime-container:jre-21.0.7_9-cds-musl RUN apk add curl WORKDIR /app COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ RUN java -Dspring.aot.enabled=true \ -XX:ArchiveClassesAtExit=./application.jsa \ -Dspring.context.exit=onRefresh -jar /app/app.jar
  23. Project Leyden Beyond AppCDS: AOT Cache with pre-compiled code Condensers

    -> optimizations Considerations: Still in the makings The more condensers applied, the bigger the cache
  24. Project Leyden Beyond AppCDS: AOT Cache with pre-compiled code Condensers

    -> optimizations Considerations: Still in the makings The more condensers applied, the bigger the cache Meanwhile, we can experiment!
  25. Let’s start with what is already implemented JEP 483: Ahead-of-Time

    Class Loading & Linking (JDK 24) Improve startup time by making the classes of an application instantly available, in a loaded and linked state, when the HotSpot Java Virtual Machine starts. Achieve this by monitoring the application during one run and storing the loaded and linked forms of all classes in a cache for use in subsequent runs. Lay a foundation for future improvements to both startup and warmup time.
  26. FROM bellsoft/liberica-runtime-container:jdk-24-musl AS builder RUN cd ${project} && ./mvnw package

    ARG project ENV project=${project} WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ FROM bellsoft/liberica-runtime-container:jre-24-cds-musl AS optimizer ARG project ENV project=${project} WORKDIR /app COPY --from=builder /app/${project}/target/*.jar app.jar RUN java -Djarmode=tools -jar app.jar extract --layers --destination extracted FROM bellsoft/liberica-runtime-container:jre-24-cds-musl AS runner
  27. FROM bellsoft/liberica-runtime-container:jre-24-cds-musl AS optimizer COPY --from=builder /app/${project}/target/*.jar app.jar RUN java

    -Djarmode=tools -jar app.jar extract --layers --destination extracted / pp ADD ${project} /app/${project} ADD ../pom.xml ./ RUN cd ${project} && ./mvnw package ARG project ENV project=${project} WORKDIR /app FROM bellsoft/liberica-runtime-container:jre-24-cds-musl AS runner RUN apk add curl WORKDIR /app COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ f i i / / d/ li i / /
  28. FROM bellsoft/liberica-runtime-container:jre-24-cds-musl AS runner COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer

    /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ WORKDIR /app COPY --from=builder /app/${project}/target/*.jar app.jar RUN java -Djarmode=tools -jar app.jar extract --layers --destination extracted RUN apk add curl WORKDIR /app RUN java -Dspring.aot.enabled=true -XX:AOTMode=record \ -XX:AOTConfiguration=app.aotconf -Dspring.context.exit=onRefresh -jar /app/app.jar RUN java -Dspring.aot.enabled=true -XX:AOTMode=create \ -XX:AOTConfiguration=app.aotconf -XX:AOTCache=app.aot -jar /app/app.jar ENTRYPOINT ["java", "-Dspring.aot.enabled=true", "-XX:AOTCache=app.aot", " j " "/ / j "]
  29. RUN java -Dspring.aot.enabled=true -XX:AOTMode=record \ -XX:AOTConfiguration=app.aotconf -Dspring.context.exit=onRefresh -jar /app/app.jar RUN

    java -Dspring.aot.enabled=true -XX:AOTMode=create \ -XX:AOTConfiguration=app.aotconf -XX:AOTCache=app.aot -jar /app/app.jar COPY --from=builder /app/${project}/target/*.jar app.jar RUN java -Djarmode=tools -jar app.jar extract --layers --destination extracted FROM bellsoft/liberica-runtime-container:jre-24-cds-musl AS runner RUN apk add curl WORKDIR /app COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ ENTRYPOINT ["java", "-Dspring.aot.enabled=true", "-XX:AOTCache=app.aot", "-jar", "/app/app.jar"]
  30. ENTRYPOINT ["java", "-Dspring.aot.enabled=true", "-XX:AOTCache=app.aot", "-jar", "/app/app.jar"] COPY --from=builder /app/${project}/target/*.jar app.jar

    RUN java -Djarmode=tools -jar app.jar extract --layers --destination extracted FROM bellsoft/liberica-runtime-container:jre-24-cds-musl AS runner RUN apk add curl WORKDIR /app COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ RUN java -Dspring.aot.enabled=true -XX:AOTMode=record \ -XX:AOTConfiguration=app.aotconf -Dspring.context.exit=onRefresh -jar /app/app.jar RUN java -Dspring.aot.enabled=true -XX:AOTMode=create \ -XX:AOTConfiguration=app.aotconf -XX:AOTCache=app.aot -jar /app/app.jar
  31. Not enough? Let’s push it even further and use the

    builds of premain in Leyden! We will be happy to get your feedback on what’s broken!
  32. FROM bellsoft/alpaquita-linux-base:glibc AS downloader ADD https://is.gd/tZhyPF /java.tar.gz # just a

    placeholder RUN tar -zxvf java.tar.gz && mv /jdk-24 /java RUN apk add tar FROM bellsoft/alpaquita-linux-base:glibc AS builder ARG project WORKDIR /app COPY --from=downloader /java /java ADD ../pom.xml ./ ADD ${project} /app/${project} ENV JAVA_HOME=/java \ project=${project} RUN cd ${project} && ./mvnw package FROM bellsoft/alpaquita-linux-base:glibc AS optimizer ARG project
  33. FROM bellsoft/alpaquita-linux-base:glibc AS builder COPY --from=downloader /java /java ADD ../pom.xml

    ./ ADD ${project} /app/${project} ENV JAVA_HOME=/java \ RUN cd ${project} && ./mvnw package p / g / y /j g j p RUN apk add tar RUN tar -zxvf java.tar.gz && mv /jdk-24 /java ARG project WORKDIR /app project=${project} FROM bellsoft/alpaquita-linux-base:glibc AS optimizer ARG project WORKDIR /app f b ild / /${ j }/ * j j
  34. FROM bellsoft/alpaquita-linux-base:glibc AS optimizer COPY --from=builder /app/${project}/target/*.jar app.jar COPY --from=downloader

    /java /java RUN /java/bin/java -Djarmode=tools -jar app.jar extract --layers --destination extracted _ /j \ project=${project} RUN cd ${project} && ./mvnw package ARG project WORKDIR /app ENV project=${project} FROM bellsoft/alpaquita-linux-base:glibc AS runner RUN apk add curl WORKDIR /app f d l d /j /j
  35. FROM bellsoft/alpaquita-linux-base:glibc AS runner COPY --from=downloader /java /java COPY --from=optimizer

    /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ ENV project=${project} RUN /java/bin/java -Djarmode=tools -jar app.jar extract --layers --destination extracted RUN apk add curl WORKDIR /app RUN /java/bin/java -XX:CacheDataStore=./application.cds \ -Dspring.context.exit=onRefresh -jar /app/app.jar ENTRYPOINT ["/java/bin/java", "-XX:CacheDataStore=./application.cds", "-jar", "/app/app.jar"]
  36. RUN /java/bin/java -XX:CacheDataStore=./application.cds \ -Dspring.context.exit=onRefresh -jar /app/app.jar ENV project=${project} RUN

    /java/bin/java -Djarmode=tools -jar app.jar extract --layers --destination extracted FROM bellsoft/alpaquita-linux-base:glibc AS runner RUN apk add curl WORKDIR /app COPY --from=downloader /java /java COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ ENTRYPOINT ["/java/bin/java", "-XX:CacheDataStore=./application.cds", "-jar", "/app/app.jar"]
  37. ENTRYPOINT ["/java/bin/java", "-XX:CacheDataStore=./application.cds", "-jar", "/app/app.jar"] ENV project=${project} RUN /java/bin/java -Djarmode=tools

    -jar app.jar extract --layers --destination extracted FROM bellsoft/alpaquita-linux-base:glibc AS runner RUN apk add curl WORKDIR /app COPY --from=downloader /java /java COPY --from=optimizer /app/extracted/dependencies/ ./ COPY --from=optimizer /app/extracted/spring-boot-loader/ ./ COPY --from=optimizer /app/extracted/snapshot-dependencies/ ./ COPY --from=optimizer /app/extracted/application/ ./ RUN /java/bin/java -XX:CacheDataStore=./application.cds \ -Dspring.context.exit=onRefresh -jar /app/app.jar
  38. Native Image: Any Considerations? Resource-demanding build process Several minutes to

    build the image Static compilation instead of dynamic one
  39. Native Image: Any Considerations? Resource-demanding build process Several minutes to

    build the image Static compilation instead of dynamic one Let the journey begin!
  40. First, let’s try it locally Build a fat JAR Build

    the image: The image was built, let’s run it! $JAVA_HOME/bin/native-image \ -jar bot-assistant/target/bot-assistant-1.0-SNAPSHOT-spring-boot.jar
  41. A dragon! Exception in thread "main" java.lang.IllegalStateException: java.util.zip.ZipException: zip END

    header not found at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLaun at org.springframework.boot.loader.JarLauncher.<init>(JarLauncher.java:42) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) Caused by: java.util.zip.ZipException: zip END header not found
  42. A dragon! java.util.zip.ZipException: zip END header not found Exception in

    thread "main" java.lang.IllegalStateException: at org.springframework.boot.loader.ExecutableArchiveLauncher.<init>(ExecutableArchiveLaun at org.springframework.boot.loader.JarLauncher.<init>(JarLauncher.java:42) at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65) Caused by: java.util.zip.ZipException: zip END header not found
  43. Apparently, a known issue ➡️ Use mvn -Pnative native:compile to

    build the image. Maven plugin calls native-image java -jar under the hood but uses lots of flags that solve problems like the one above.
  44. Apparently, a known issue ➡️ Use mvn -Pnative native:compile to

    build the image. Maven plugin calls native-image java -jar under the hood but uses lots of flags that solve problems like the one above. Alright, let's try with a plugin!
  45. If you use Spring Boot Parent POM, the Native Image

    profile is already configured! ➡️ Enable it by simply running mvn -Pnative native:compile
  46. Another dragon! Error: Classes that should be initialized at run

    time got initialized during image building: com.ctc.wstx.api.CommonConfig was unintentionally initialized at build time. com.ctc.wstx.stax.WstxInputFactory was unintentionally initialized at build time. com.ctc.wstx.api.ReaderConfig was unintentionally initialized at build time. com.ctc.wstx.util.DefaultXmlSymbolTable was unintentionally initialized at build time. To see how the classes got initialized, use --trace-class-initialization=com.ctc.wstx.api.CommonConfig,com.ctc.wstx.stax.WstxInputFactory,com
  47. Another dragon! Error: Classes that should be initialized at run

    time got initialized during image building: To see how the classes got initialized, use --trace-class-initialization=com.ctc.wstx.api.CommonConfig,com.ctc.wstx.stax.WstxInputFactory,com com.ctc.wstx.api.CommonConfig was unintentionally initialized at build time. com.ctc.wstx.stax.WstxInputFactory was unintentionally initialized at build time. com.ctc.wstx.api.ReaderConfig was unintentionally initialized at build time. com.ctc.wstx.util.DefaultXmlSymbolTable was unintentionally initialized at build time.
  48. Add native profile: <profiles> <profile> <id>native</id> <build> <plugins> <plugin> <groupId>org.graalvm.buildtools</groupId>

    <artifactId>native-maven-plugin</artifactId> <executions> <execution> <id>build-native</id>
  49. Add the --trace-class-initialization argument: Run <profile> <id>native</id> <build> <plugins> <plugin>

    <groupId>org.graalvm.buildtools</groupId> <artifactId>native-maven-plugin</artifactId> <executions> <execution> <id>build-native</id> <goals> mvn -Pnative native:compile
  50. More detailed output: Error: Classes that should be initialized at

    run time got initialized during image building: com.ctc.wstx.api.CommonConfig was unintentionally initialized at build time. org.springframework.http.codec.xml.XmlEventDecoder caused initialization of this class with the following trace: at com.ctc.wstx.api.CommonConfig.<clinit>(CommonConfig.java:59) at com.ctc.wstx.stax.WstxInputFactory.<init>(WstxInputFactory.java:149) at java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(DirectMethodHandle$Holder) at java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) at jdk.internal.reflect.DirectConstructorHandleAccessor.invokeImpl(DirectConstructorHandle at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandl
  51. More detailed output: We found the culprit! org.springframework.http.codec.xml.XmlEventDecoder caused initialization

    of this class Error: Classes that should be initialized at run time got initialized during image building: com.ctc.wstx.api.CommonConfig was unintentionally initialized at build time. with the following trace: at com.ctc.wstx.api.CommonConfig.<clinit>(CommonConfig.java:59) at com.ctc.wstx.stax.WstxInputFactory.<init>(WstxInputFactory.java:149) at java.lang.invoke.DirectMethodHandle$Holder.newInvokeSpecial(DirectMethodHandle$Holder) at java.lang.invoke.Invokers$Holder.invokeExact_MT(Invokers$Holder) at jdk.internal.reflect.DirectConstructorHandleAccessor.invokeImpl(DirectConstructorHandle at jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandl
  52. Alright, let’s add the option to initialize org.springframework.http.codec.xml.XmlEventDecoder at runtime:

    Run <profile> <id>native</id> <build> <plugins> <plugin> <groupId>org.graalvm.buildtools</groupId> <artifactId>native-maven-plugin</artifactId> <executions> <execution> <id>build-native</id> <goals> mvn -Pnative native:compile
  53. Alright, let’s add the option to initialize org.springframework.http.codec.xml.XmlEventDecoder at runtime:

    Run <buildArgs>--initialize-at-run-time=org.springframework.http.codec.xml.XmlE <configuration> <imageName>bot-assistant</imageName> <outputDirectory>target/native</outputDirectory> <mainClass>com.github.asm0dey.botassistant.BotAssistantApplication</mainClass> <buildArgs> </buildArgs> </configuration> </plugin> </plugins> </build> mvn -Pnative native:compile
  54. The same dragon! This is one evasive dragon… Error: Classes

    that should be initialized at run time got initialized during image building: com.ctc.wstx.api.CommonConfig was unintentionally initialized at build time. To see why com.ctc.w com.ctc.wstx.stax.WstxInputFactory was unintentionally initialized at build time. To see why com. com.ctc.wstx.api.ReaderConfig was unintentionally initialized at build time. To see why com.ctc.ws com.ctc.wstx.util.DefaultXmlSymbolTable was unintentionally initialized at build time. To see why To see how the classes got initialized, use --trace-class-initialization=com.ctc.wstx.api.CommonCo
  55. Apparently, another known issue ➡️ Need to add the --strict-image-heap

    option. This mode requires only the classes that are stored in the image heap to be marked with –initialize-at- build-time. This effectively reduces the number of configuration entries necessary to achieve build-time initialization. Note that --strict-image-heap is enabled by default in Native Image starting from GraalVM for JDK 22.
  56. Let’s add the --strict-image-heap option: Run <buildArgs>--strict-image-heap</buildArgs> <configuration> <imageName>bot-assistant</imageName> <outputDirectory>target/native</outputDirectory>

    <mainClass>com.github.asm0dey.botassistant.BotAssistantApplication</mainClass <buildArgs> </buildArgs> </configuration> </plugin> </plugins> </build> mvn -Pnative native:compile
  57. Another dragon! 2025-04-11T12:25:33.511+03:00 ERROR 64619 --- [bot-assistant] [nio-8081-exec-2] o.a.c.c.C.[.[.[/] java.lang.UnsatisfiedLinkError:

    jdk.jfr.internal.JVM.isExcluded(Ljava/lang/Class;)Z [symbol: Java_jdk_jfr_internal_JVM_isExcluded or Java_jdk_jfr_internal_JVM_isExcluded__Ljava_lang_Class_2] at org.graalvm.nativeimage.builder/com.oracle.svm.core.jni.access.JNINativeLinkage.getOrF at org.graalvm.nativeimage.builder/com.oracle.svm.core.jni.JNIGeneratedMethodSupport.nativ at [email protected]/jdk.jfr.internal.JVM.isExcluded(Native Method) ~[na:na] at [email protected]/jdk.jfr.internal.MetadataRepository.register(MetadataRepository
  58. Another dragon! Apparently, when we create a connection to Redis,

    Spring creates a custom JFR event java.lang.UnsatisfiedLinkError: jdk.jfr.internal.JVM.isExcluded(Ljava/lang/Class;)Z [symbol: Java_jdk_jfr_internal_JVM_isExcluded or Java_jdk_jfr_internal_JVM_isExcluded__Ljava_lang_Class_2] 2025-04-11T12:25:33.511+03:00 ERROR 64619 --- [bot-assistant] [nio-8081-exec-2] o.a.c.c.C.[.[.[/] at org.graalvm.nativeimage.builder/com.oracle.svm.core.jni.access.JNINativeLinkage.getOrF at org.graalvm.nativeimage.builder/com.oracle.svm.core.jni.JNIGeneratedMethodSupport.nativ at [email protected]/jdk.jfr.internal.JVM.isExcluded(Native Method) ~[na:na] at [email protected]/jdk.jfr.internal.MetadataRepository.register(MetadataRepository
  59. Even the Tracing Agent doesn’t detect that! So, let’s enable

    JFR explicitly: Run <buildArgs>--enable-monitoring=jfr</buildArgs> <imageName>bot-assistant</imageName> <outputDirectory>target/native</outputDirectory> <mainClass>com.github.asm0dey.botassistant.BotAssistantApplication</mainClass <buildArgs> <buildArgs>--strict-image-heap</buildArgs> </buildArgs> </configuration> </plugin> </plugins> </build> mvn -Pnative native:compile
  60. FROM bellsoft/liberica-native-image-kit-container:jdk-21-nik-23.1.6-musl as builder ARG project ENV project=${project} WORKDIR /app

    ADD ${project} /app/${project} ADD ../pom.xml ./ RUN cd ${project} && ./mvnw -Pnative native:compile FROM bellsoft/alpaquita-linux-base:stream-musl ARG project ENV project=${project} RUN apk add curl WORKDIR /app ENTRYPOINT ["/app/app"] COPY --from=builder /app/${project}/target/native/${project} /app/app
  61. FROM bellsoft/liberica-native-image-kit-container:jdk-21-nik-23.1.6-musl as builder RUN cd ${project} && ./mvnw -Pnative

    native:compile ARG project ENV project=${project} WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ FROM bellsoft/alpaquita-linux-base:stream-musl ARG project ENV project=${project} RUN apk add curl WORKDIR /app ENTRYPOINT ["/app/app"] COPY --from=builder /app/${project}/target/native/${project} /app/app
  62. FROM bellsoft/alpaquita-linux-base:stream-musl ENTRYPOINT ["/app/app"] COPY --from=builder /app/${project}/target/native/${project} /app/app FROM bellsoft/liberica-native-image-kit-container:jdk-21-nik-23.1.6-musl

    as builder ARG project ENV project=${project} WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ RUN cd ${project} && ./mvnw -Pnative native:compile ARG project ENV project=${project} RUN apk add curl WORKDIR /app
  63. Another dragon! 2189.3 [6/8] Compiling methods... [*************] 2189.3 2189.3 Fatal

    error: org.graalvm.compiler.debug.GraalError: org.graalvm.compiler.core.common.PermanentBailoutException: Compilation exceeded 300.000000 seconds during CFG traversal 2189.3 at method: Future
  64. Another dragon! 2189.3 Fatal error: org.graalvm.compiler.debug.GraalError: org.graalvm.compiler.core.common.PermanentBailoutException: Compilation exceeded 300.000000

    seconds during CFG traversal 2189.3 [6/8] Compiling methods... [*************] 2189.3 2189.3 at method: Future
  65. The build was taking too long and never finished Solution:

    Increase timeout time Build the image on a powerful PC or in the CI Plug a laptop into a power socket
  66. The build was taking too long and never finished Solution:

    Increase timeout time Build the image on a powerful PC or in the CI Plug a laptop into a power socket
  67. The build was taking too long and never finished Solution:

    Increase timeout time Build the image on a powerful PC or in the CI Plug a laptop into a power socket Let's give it another try!
  68. Another dragon! 520.7 [8/8] Creating image... [*****] 520.7 --------------------------------------------------------------------------------------------- 520.7

    53.2s (13.5% of total time) in 2385 GCs | Peak RSS: 6.77GB | CPU load: 520.7 --------------------------------------------------------------------------------------------- 520.7 Produced artifacts: 520.7 /app/bot-assistant/target/native/svm_err_b_20250415T153317.420_pid287.md (build_info) 520.7 ============================================================================================= 520.7 Failed generating 'bot' after 6m 32s. 520.7 520.7 The build process encountered an unexpected error: 520.7
  69. Another dragon! 520.7 The build process encountered an unexpected error:

    520.7 > java.lang.RuntimeException: There was an error linking the native image: Linker command exited with 1 520.7 ============================================================================================ 520.7 Failed generating 'bot' after 6m 32s. 520.7 520.7 520.7 520.7 Linker command executed: 520.7 /usr/bin/gcc -z noexecstack -Wl,--gc-sections -Wl,--version-script,/tmp/SVM-1092355479500053 520.7
  70. Linking issue: musl libc lacks required tools Option 1: add

    required packages (libstc++, etc.) with apk add Option 2: switch from musl-based Alpaquita to glibc-based FROM bellsoft/liberica-native-image-kit-container:jdk-21-nik-23.1.6-stream-musl as builder WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ RUN cd ${project} && ./mvnw -Pnative native:compile FROM bellsoft/alpaquita-linux-base:stream-musl ARG project ENV project=${project} RUN apk add curl WORKDIR /app
  71. Linking issue: musl libc lacks required tools Option 1: add

    required packages (libstc++, etc.) with apk add Option 2: switch from musl-based Alpaquita to glibc-based FROM bellsoft/liberica-native-image-kit-container:jdk-21-nik-23.1.6-stream-glibc as builder WORKDIR /app ADD ${project} /app/${project} ADD ../pom.xml ./ RUN cd ${project} && ./mvnw -Pnative native:compile FROM bellsoft/alpaquita-linux-base:stream-glibc ARG project ENV project=${project} RUN apk add curl WORKDIR /app
  72. Coordinated Restore at Checkpoint (CRaC) The CRaC (Coordinated Restore at

    Checkpoint) Project researches coordination of Java programs with mechanisms to checkpoint (make an image of, snapshot) a Java instance while it is executing. Restoring from the image could be a solution to some of the problems with the start-up and warm-up times. The primary aim of the Project is to develop a new standard mechanism-agnostic API to notify Java programs about the checkpoint and restore events. Other research activities will include, but will not be limited to, integration with existing checkpoint/restore mechanisms and development of new ones, changes to JVM and JDK to make images smaller and ensure they are correct. https://openjdk.org/projects/crac/
  73. TL;DR Pause and restart a Java application A snapshot of

    the current JVM state JVM is still there: dynamic performance optimization is possible after restore
  74. CRaC is not a canonical part of JDK Only some

    vendors provide it: BellSoft, Azul Azul was the first to implement, BellSoft supports a fork
  75. Project CRaC: Any Considerations? A snapshot may contain sensitive data

    May need to augment the code for reliable checkpoint and restore
  76. Trivial example public static void main(String args[]) throws InterruptedException {

    // This is a part of the saved state long startTime = System.currentTimeMillis(); for(int counter: IntStream.range(1, 10000).toArray()) { Thread.sleep(1000); long currentTime = System.currentTimeMillis(); System.out.println("Counter: " + counter + "(passed " + (currentTime-startTime) + " ms)"); startTime = currentTime; } }
  77. Trivial example public static void main(String args[]) throws InterruptedException {

    // This is a part of the saved state long startTime = System.currentTimeMillis(); for(int counter: IntStream.range(1, 10000).toArray()) { Thread.sleep(1000); long currentTime = System.currentTimeMillis(); System.out.println("Counter: " + counter + "(passed " + (currentTime-startTime) + " ms)"); startTime = currentTime; } }
  78. Trivial example // This is a part of the saved

    state public static void main(String args[]) throws InterruptedException { long startTime = System.currentTimeMillis(); for(int counter: IntStream.range(1, 10000).toArray()) { Thread.sleep(1000); long currentTime = System.currentTimeMillis(); System.out.println("Counter: " + counter + "(passed " + (currentTime-startTime) + " ms)"); startTime = currentTime; } }
  79. Trivial example long startTime = System.currentTimeMillis(); public static void main(String

    args[]) throws InterruptedException { // This is a part of the saved state for(int counter: IntStream.range(1, 10000).toArray()) { Thread.sleep(1000); long currentTime = System.currentTimeMillis(); System.out.println("Counter: " + counter + "(passed " + (currentTime-startTime) + " ms)"); startTime = currentTime; } }
  80. Trivial example for(int counter: IntStream.range(1, 10000).toArray()) { public static void

    main(String args[]) throws InterruptedException { // This is a part of the saved state long startTime = System.currentTimeMillis(); Thread.sleep(1000); long currentTime = System.currentTimeMillis(); System.out.println("Counter: " + counter + "(passed " + (currentTime-startTime) + " ms)"); startTime = currentTime; } }
  81. Trivial example Thread.sleep(1000); long currentTime = System.currentTimeMillis(); System.out.println("Counter: " +

    counter + "(passed " + (currentTime-startTime) + " ms)"); public static void main(String args[]) throws InterruptedException { // This is a part of the saved state long startTime = System.currentTimeMillis(); for(int counter: IntStream.range(1, 10000).toArray()) { startTime = currentTime; } }
  82. Trivial example startTime = currentTime; public static void main(String args[])

    throws InterruptedException { // This is a part of the saved state long startTime = System.currentTimeMillis(); for(int counter: IntStream.range(1, 10000).toArray()) { Thread.sleep(1000); long currentTime = System.currentTimeMillis(); System.out.println("Counter: " + counter + "(passed " + (currentTime-startTime) + " ms)"); } }
  83. Docker… ...is not that simple FROM bellsoft/liberica-runtime-container:jdk-crac-slim ADD Example.java /app/Example.java

    WORKDIR /app RUN javac Example.java ENTRYPOINT java -XX:CRaCCheckpointTo=/app/checkpoint Example
  84. Docker… ...is not that simple FROM bellsoft/liberica-runtime-container:jdk-crac-slim ADD Example.java /app/Example.java

    WORKDIR /app RUN javac Example.java ENTRYPOINT java -XX:CRaCCheckpointTo=/app/checkpoint Example
  85. Docker… ...is not that simple ADD Example.java /app/Example.java FROM bellsoft/liberica-runtime-container:jdk-crac-slim

    WORKDIR /app RUN javac Example.java ENTRYPOINT java -XX:CRaCCheckpointTo=/app/checkpoint Example
  86. Docker… ...is not that simple RUN javac Example.java FROM bellsoft/liberica-runtime-container:jdk-crac-slim

    ADD Example.java /app/Example.java WORKDIR /app ENTRYPOINT java -XX:CRaCCheckpointTo=/app/checkpoint Example
  87. Docker… ...is not that simple ENTRYPOINT java -XX:CRaCCheckpointTo=/app/checkpoint Example FROM

    bellsoft/liberica-runtime-container:jdk-crac-slim ADD Example.java /app/Example.java WORKDIR /app RUN javac Example.java
  88. Docker… ...is not that simple This does not create the

    checkpoint yet! ENTRYPOINT java -XX:CRaCCheckpointTo=/app/checkpoint Example FROM bellsoft/liberica-runtime-container:jdk-crac-slim ADD Example.java /app/Example.java WORKDIR /app RUN javac Example.java
  89. And then Build it Run it docker build -t pre_crack

    -f crac2/Dockerfile crac2 docker run -d pre_crack
  90. And then Build it Run it docker build -t pre_crack

    -f crac2/Dockerfile crac2 docker run -d pre_crack # will fail
  91. And then Build it Run it docker build -t pre_crack

    -f crac2/Dockerfile crac2 docker run --privileged -d pre_crack
  92. And then Build it Run it docker build -t pre_crack

    -f crac2/Dockerfile crac2 docker run --privileged -d pre_crack # will work, but security folks will hate us
  93. And then Build it Run it docker build -t pre_crack

    -f crac2/Dockerfile crac2 docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -d pre_crack
  94. And then Build it Run it docker build -t pre_crack

    -f crac2/Dockerfile crac2 docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -d pre_crack # Not so frightening if you think about it
  95. And then Build it Run it CAP_SYS_PTRACE : we need

    to access the whole process tree transfer data to or from the memory of arbitrary processes using process_vm_readv(2) and process_vm_writev(2) docker build -t pre_crack -f crac2/Dockerfile crac2 docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -d pre_crack # Not so frightening if you think about it
  96. And then Build it Run it CAP_SYS_PTRACE : we need

    to access the whole process tree transfer data to or from the memory of arbitrary processes using process_vm_readv(2) and process_vm_writev(2) CAP_CHECKPOINT_RESTORE : somehow there is a special cap for this Update /proc/sys/kernel/ns_last_pid ; Read the contents of the symbolic links in /proc/pid/map_files for other processes docker build -t pre_crack -f crac2/Dockerfile crac2 docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -d pre_crack # Not so frightening if you think about it
  97. And then Checkpoint it ID=$(docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE

    -p8080:8080 -d pre_crac # wait some time docker exec -it $ID jcmd 129 JDK.checkpoint docker commit $ID cracked docker run --rm -d \ --entrypoint java \ --network host cracked:latest \ -XX:CRaCRestoreFrom=/app/checkpoint
  98. And then Checkpoint it ID=$(docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE

    -p8080:8080 -d pre_crac # wait some time docker exec -it $ID jcmd 129 JDK.checkpoint docker commit $ID cracked docker run --rm -d \ --entrypoint java \ --network host cracked:latest \ -XX:CRaCRestoreFrom=/app/checkpoint
  99. And then Checkpoint it # wait some time ID=$(docker run

    --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -p8080:8080 -d pre_crac docker exec -it $ID jcmd 129 JDK.checkpoint docker commit $ID cracked docker run --rm -d \ --entrypoint java \ --network host cracked:latest \ -XX:CRaCRestoreFrom=/app/checkpoint
  100. And then Checkpoint it docker exec -it $ID jcmd 129

    JDK.checkpoint ID=$(docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -p8080:8080 -d pre_crac # wait some time docker commit $ID cracked docker run --rm -d \ --entrypoint java \ --network host cracked:latest \ -XX:CRaCRestoreFrom=/app/checkpoint
  101. And then Checkpoint it docker commit $ID cracked ID=$(docker run

    --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -p8080:8080 -d pre_crac # wait some time docker exec -it $ID jcmd 129 JDK.checkpoint docker run --rm -d \ --entrypoint java \ --network host cracked:latest \ -XX:CRaCRestoreFrom=/app/checkpoint
  102. And then Now we’re ready! Checkpoint it docker commit $ID

    cracked ID=$(docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -p8080:8080 -d pre_crac # wait some time docker exec -it $ID jcmd 129 JDK.checkpoint docker run --rm -d \ --entrypoint java \ --network host cracked:latest \ -XX:CRaCRestoreFrom=/app/checkpoint
  103. And then Now we’re ready! Checkpoint it docker commit $ID

    cracked ID=$(docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -p8080:8080 -d pre_crac # wait some time docker exec -it $ID jcmd 129 JDK.checkpoint docker run --rm -d \ --entrypoint java \ --network host cracked:latest \ -XX:CRaCRestoreFrom=/app/checkpoint
  104. And then Now we’re ready! Checkpoint it docker commit $ID

    cracked ID=$(docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -p8080:8080 -d pre_crac # wait some time docker exec -it $ID jcmd 129 JDK.checkpoint --entrypoint java \ docker run --rm -d \ --network host cracked:latest \ -XX:CRaCRestoreFrom=/app/checkpoint
  105. And then Now we’re ready! Checkpoint it docker commit $ID

    cracked ID=$(docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -p8080:8080 -d pre_crac # wait some time docker exec -it $ID jcmd 129 JDK.checkpoint --network host cracked:latest \ docker run --rm -d \ --entrypoint java \ -XX:CRaCRestoreFrom=/app/checkpoint
  106. And then Now we’re ready! Checkpoint it docker commit $ID

    cracked ID=$(docker run --cap-add CAP_SYS_PTRACE --cap-add CAP_CHECKPOINT_RESTORE -p8080:8080 -d pre_crac # wait some time docker exec -it $ID jcmd 129 JDK.checkpoint -XX:CRaCRestoreFrom=/app/checkpoint docker run --rm -d \ --entrypoint java \ --network host cracked:latest \
  107. In our startup bot-assistant module just works chat-api doesn’t work!

    Only some projects are guaranteed to work. Others… Request support from the team: is not supported for now :(
  108. What are the options? Fall back to another solution Request

    support Implement support on our own!
  109. Implementing support on our own. Custom MongoClient public class MongoClientProxy

    implements MongoClient { volatile MongoClient delegate; public MongoClientProxy(MongoClient initialClient) { this.delegate = initialClient; } public void close() { delegate.close(); } // Delegate everything else the same way
  110. Implementing support on our own. Custom MongoClient volatile MongoClient delegate;

    public MongoClientProxy(MongoClient initialClient) { this.delegate = initialClient; } public class MongoClientProxy implements MongoClient { public void close() { delegate.close(); } // Delegate everything else the same way
  111. Implementing support on our own. Custom MongoClient public void close()

    { delegate.close(); } public class MongoClientProxy implements MongoClient { volatile MongoClient delegate; public MongoClientProxy(MongoClient initialClient) { this.delegate = initialClient; } // Delegate everything else the same way
  112. Implementing support on our own. Custom MongoClient // Delegate everything

    else the same way public class MongoClientProxy implements MongoClient { volatile MongoClient delegate; public MongoClientProxy(MongoClient initialClient) { this.delegate = initialClient; } public void close() { delegate.close(); }
  113. Implementing support on our own Custom CRaC Resource @Component static

    public class MongoClientResource implements Resource { private final MongoClientProxy mongoClientProxy; private final MongoConnectionDetails details; public MongoClientResource(MongoClient mongoClientProxy, MongoConnectionDetails details) { this.mongoClientProxy = (MongoClientProxy) mongoClientProxy; this.details = details; Core.getGlobalContext().register(this); } @Override public void beforeCheckpoint(Context<? extends Resource> context) {
  114. Implementing support on our own Custom CRaC Resource public MongoClientResource(MongoClient

    mongoClientProxy, MongoConnectionDetails details) { this.mongoClientProxy = (MongoClientProxy) mongoClientProxy; this.details = details; Core.getGlobalContext().register(this); } static public class MongoClientResource implements Resource { private final MongoClientProxy mongoClientProxy; private final MongoConnectionDetails details; @Override public void beforeCheckpoint(Context<? extends Resource> context) { mongoClientProxy.delegate.close(); }
  115. Implementing support on our own Custom CRaC Resource private final

    MongoClientProxy mongoClientProxy; this.mongoClientProxy = (MongoClientProxy) mongoClientProxy; @Component static public class MongoClientResource implements Resource { private final MongoConnectionDetails details; public MongoClientResource(MongoClient mongoClientProxy, MongoConnectionDetails details) { this.details = details; Core.getGlobalContext().register(this); } @Override public void beforeCheckpoint(Context<? extends Resource> context) { mongoClientProxy.delegate.close();
  116. Implementing support on our own Custom CRaC Resource private final

    MongoConnectionDetails details; this.details = details; static public class MongoClientResource implements Resource { private final MongoClientProxy mongoClientProxy; public MongoClientResource(MongoClient mongoClientProxy, MongoConnectionDetails details) { this.mongoClientProxy = (MongoClientProxy) mongoClientProxy; Core.getGlobalContext().register(this); } @Override public void beforeCheckpoint(Context<? extends Resource> context) { mongoClientProxy.delegate.close(); }
  117. Implementing support on our own Custom CRaC Resource Core.getGlobalContext().register(this); private

    final MongoClientProxy mongoClientProxy; private final MongoConnectionDetails details; public MongoClientResource(MongoClient mongoClientProxy, MongoConnectionDetails details) { this.mongoClientProxy = (MongoClientProxy) mongoClientProxy; this.details = details; } @Override public void beforeCheckpoint(Context<? extends Resource> context) { mongoClientProxy.delegate.close(); }
  118. Implementing support on our own Custom CRaC Resource @Override public

    void beforeCheckpoint(Context<? extends Resource> context) { mongoClientProxy.delegate.close(); } this.mongoClientProxy = (MongoClientProxy) mongoClientProxy; this.details = details; Core.getGlobalContext().register(this); } @Override public void afterRestore(Context<? extends Resource> context) { mongoClientProxy.delegate = MongoClients.create(details.getConnectionString()); } }
  119. Implementing support on our own Custom CRaC Resource @Override public

    void afterRestore(Context<? extends Resource> context) { mongoClientProxy.delegate = MongoClients.create(details.getConnectionString()); } this.details = details; Core.getGlobalContext().register(this); } @Override public void beforeCheckpoint(Context<? extends Resource> context) { mongoClientProxy.delegate.close(); } }
  120. Implementing support on our own Custom CRaC Resource mongoClientProxy.delegate =

    MongoClients.create(details.getConnectionString()); this.details = details; Core.getGlobalContext().register(this); } @Override public void beforeCheckpoint(Context<? extends Resource> context) { mongoClientProxy.delegate.close(); } @Override public void afterRestore(Context<? extends Resource> context) { } }
  121. Implementing support on our own Replacing MongoClient in the context

    @Bean public MongoClient mongoClient(MongoConnectionDetails details) { @Primary MongoClient initialClient = MongoClients.create(details.getConnectionString()); return new MongoClientProxy(initialClient); }
  122. Implementing support on our own Replacing MongoClient in the context

    MongoClient initialClient = MongoClients.create(details.getConnectionString()); @Bean @Primary public MongoClient mongoClient(MongoConnectionDetails details) { return new MongoClientProxy(initialClient); }
  123. Implementing support on our own Replacing MongoClient in the context

    @Primary return new MongoClientProxy(initialClient); @Bean public MongoClient mongoClient(MongoConnectionDetails details) { MongoClient initialClient = MongoClients.create(details.getConnectionString()); }
  124. Quick Recap 🛡 → AppCDS for the smoothest integration 🧙‍♂️

    → Project Leyden EA builds to prepare to use it in the future 🎒 → Native Image for fast start ✊🏼 → CRaC for almost instant start Testing locally? Use a power cord! Which hero are you?