Optimizing Java Applications for Serverless

This deck captures the different knobs to turn for a JVM application and the trade offs to make.

Full documentation here: https://cloud.google.com/run/docs/tips/java

1. Optimizing Java apps for Serverless...
2. 2 @saturnism @gcpcloud Mostly startup speed & memory usage
3. 3 @saturnism @gcpcloud It's a trade off though… Faster startup != more efficient
4. 4 @saturnism @gcpcloud JVM traditionally optimize for... Long Running, High throughput, High concurrency
5. 5 @saturnism @gcpcloud So we… Cache, Connection Pool, JIT
6. 6 @saturnism @gcpcloud Minimize Memory Usages Minimize JVM startup time Minimize Framework init time Minimize Application init time
7. Container Image
8. 8 @saturnism @gcpcloud Use Jib Jib produces optimized container image for fast reads Trade off - almost none Some frameworks require additional configuration Works out of the box with Spring Boot though
9. JVM
10. 10 @saturnism @gcpcloud Java -XX:+TieredCompilation -XX:TieredStopAtLevel=1 Turn off optimization compiler Trade off - no optimization for hot paths Turn it off means no speed up...
11. 11 @saturnism @gcpcloud java -noverify Don't verify classes Trade off - undetected malicious edits to bytecode JVM might crash Worse, you might get hacked
12. 12 @saturnism @gcpcloud java -Xss256k Reduce thread stack size, and thus non-heap memory Trade off - StackOverflowError Need to find optimal value
13. 13 @saturnism @gcpcloud Use OpenJDK 8u142 or above Container awareness Trade off - none, just do it! If you don't... See wrong # of CPUs, use more than you should See wrong amount of Memory, get OOMKilled
14. 14 @saturnism @gcpcloud Dealing with OOMKilled Need to understand Memory usage Heap + Non-Heap = Heap + (Metaspace + Code Cache + Thread stack + ...)
15. 15 @saturnism @gcpcloud Native Memory Tracking -XX:NativeMemoryTracking=summary -XX:+PrintNMTStatistics (Doesn't work when set via JAVA_TOOL_OPTIONS - must be part of the argument)
16. 16 @saturnism @gcpcloud Use Memory Calculator Non-heap usage estimated O(classes) https://github.com/cloudfoundry/java-buildpack-memory-calculator
17. 17 @saturnism @gcpcloud Understand Memory Usage and Accounting JVM Heap Non-Heap tmpfs Write filesystem Read Memory Accounting - JVM - Heap - Non-Heap - Code Cache - Thread - GC - … - tmpfs (/tmp/) - Filesystem buffers/cache - What system reads - JVM, JDK, .so lib, ...
18. Threads
19. 19 @saturnism @gcpcloud Reduce # of Threads Tomcat defaults to 10 min spare threads, 200 max threads More threads = more memory too (stack size, -Xss) Trade off - lower throughput/concurrency Cloud Run max at 80 concurrent reqs per instance anyways...
20. 20 @saturnism @gcpcloud Pending Requests Most thread-per-connection apps queues up requests/connections that it cannot immediately serve. Understand what a single app instance is capable of Reduce this number
21. 21 @saturnism @gcpcloud Use Reactive / Webflux Asynchronous, non-blocking, high concurrency Defaults to CPU cores * 2 threads Use less memory Trade off - if you block... Then your exhaust the few threads Worse, if new worker threads are created in unbounded pool...
22. 22 @saturnism @gcpcloud Avoid Background Tasks Cloud Run throttles CPU if instances has no active requests Something to be aware of... What's running in the background? Scheduled tasks, @Timer, @Async, message receivers Metrics senders, trace senders, ...
23. 23 @saturnism @gcpcloud Connection Pools Set Max Instances Cloud Run Max Instances x Connections per Instance < Max Connections Allowed Something to be aware of... Connection eviction may not happen if CPU throttled
24. Application / Framework
25. 25 @saturnism @gcpcloud Use Spring Boot 2.2.x or up spring.main.lazy-initialization=true Trade off No tradeoffs. Spring Boot 2.2.x implemented a number of optimizations For < 2.2, consider disabling JMX, actuator, ...
26. 26 @saturnism @gcpcloud Use 2-CPU gcloud run deploy … --cpu=2 Trade off - Cost Spring Boot utilizes multiple threads for initialization Single CPU allocation may cause contention
27. 27 @saturnism @gcpcloud Spring Boot Lazy Initialization spring.main.lazy-initialization=true Trade off - initialization on first request First request may be slower Use if there is a readiness check (GKE, App Engine)
28. 28 @saturnism @gcpcloud Use Context Indexer <dependency> <groupId>org.springframework</groupId> <artifactId>spring-context-indexer</artifactId> <optional>true</optional> </dependency> Trade off - slower if few components May help in the case of deep component/bean graphs
29. 29 @saturnism @gcpcloud Make sure Dev Tool is not in Production Trade off - no trade offs Spring Boot Dev Tool helps w/ local development, but is not needed in Production If using Jib, make sure this is not included
30. 30 @saturnism @gcpcloud Application Lifecycle Trade off - no trade offs In App Engine, implement /_ah/health and /_ah/warmup Prevents app receiving requests prematurely
31. Native Image
32. 32 @saturnism @gcpcloud To Graal or not to Graal? More accurately, to native image or not?
33. 33 @saturnism @gcpcloud Trade Offs! Compile time GC JIT Throughput
34. 34 @saturnism @gcpcloud Frameworks support for Native Image Quarkus Micronaut Spring Boot (Experimental) More than framework - libraries need to support them too
35. 35 @saturnism @gcpcloud Additional Resources Cloud Run Java Optimization Guide https://cloud.google.com/run/docs/tips/java Spring Boot in a Container (Dave Syer) https://spring.io/blog/2018/11/08/spring-boot-in-a-container How Fast is Spring? (Dave Syer) https://spring.io/blog/2018/12/12/how-fast-is-spring OpenJDK and Containers (Christine Flood) https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/ Running Spring Boot App in GraalVM (Sebastian Deluze) https://www.youtube.com/watch?v=3eoAxphAUIg Spring to Kubernetes Faster and Easier (Ray Tsang) https://saturnism.me/talk/kubernetes-spring-java-best-practices/
36. Thanks! @saturnism | saturnism.me