Share
Leap from Java 8 to Java 11 in Jenkins. Our war story
Jenkins, one of the leading open-source automation servers, has not been supporting Java 9 and above since its official release. Over the last year Jenkins contributors were working towards the support of Java 11 (Jenkins JEP-211). Jenkins has a huge and sometimes legacy Java/Groovy/native codebase, includes DSL engines, and depends on many upstream libraries and modules (including ones deleted from Java 11). It was a thorny way, but Java 11 support is almost ready to be released.
In this talk I would like to share our experiences and talk about potential obstacles others may hit while implementing Java 11 support in their projects. What issues have we hit while implementing support of Java 11 in the core? Are modules and deprecations that tricky for a web application (spoiler: no)? How have we enabled building and testing with OpenJDK 11 in our Maven-based pipelines? And how have we retained Java 8 compatibility after all these changes?
Video: https://www.youtube.com/watch?v=18wVyYaZvTw&feature=youtu.be
Meetup link: https://www.meetup.com/Swiss-Jenkins-Area-Meetup/events/255914568/
Demo: https://github.com/oleg-nenashev/demo-jenkins-config-as-code/commits/java11-support
Oleg Nenashev PRO
Transcript
98 slides
-
© 2018 CloudBees, Inc. All Rights Reserved. JUG Lausanne: Leap
from Java 8 to Java 11 in Jenkins. Our war story Oleg Nenashev, CloudBees Nov 14, 2018 -
© 2018 CloudBees, Inc. All Rights Reserved. > whoami 2
@oleg_nenashev oleg-nenashev • Based in Neuchâtel • Principal SW Engineer, CloudBees • CloudBees Core • CloudBees Jenkins Support -
© 2018 CloudBees, Inc. All Rights Reserved. > whoami -jenkins
3 @oleg_nenashev oleg-nenashev • Contributor since 2012 • Jenkins Core Maintainer • Jenkins Ambassador • Platform SIG Leader -
© 2018 CloudBees, Inc. All Rights Reserved. > whoami -java
2006 - J# and Java ME 2010+ - patches to Hudson/Jenkins 2013+ - Jenkins & Java non-stop @oleg_nenashev oleg-nenashev 4 -
© 2018 CloudBees, Inc. All Rights Reserved. Swiss Jenkins Area
Meetup • Building local community • Sharing experiences • Looking for contributors https://www.meetup.com/Swiss-Jenkins-Area-Meetup https://twitter.com/JenkinsMeetupCH 5 -
© 2018 CloudBees, Inc. All Rights Reserved. Java 11 Released
on Sep 25, 2018 6 -
© 2018 CloudBees, Inc. All Rights Reserved. 7 Image source:
https://medium.com/criciumadev/its-time-migrating-to-java-11-5eb3868354f9 -
© 2018 CloudBees, Inc. All Rights Reserved. Agenda. Jenkins and
Java 11 • Why Java 11? • Jenkins architecture overview • Java 11. Our war story • Current status 8 These slides: http://bit.ly/jugl-jenkins-java11 -
© 2018 CloudBees, Inc. All Rights Reserved. Target Audience •
Java developers • Hardcore Jenkins Admins 9 -
© 2018 CloudBees, Inc. All Rights Reserved. 1. Most popular
CI/CD tool in the world 2. 200k+ installations 3. Open-source 4. Big community 5. Plugin ecosystem (>1500 plugins) 6. … Who is Mr. Jenkins? https://jenkins.io 11 -
© 2018 CloudBees, Inc. All Rights Reserved. Under the hood
12 JENKINS Image: https://www.youtube.com/watch?v=Qae_3RDa6sM -
© 2018 CloudBees, Inc. All Rights Reserved. Under the hood
13 JENKINS Image: https://www.youtube.com/watch?v=Qae_3RDa6sM Under the hood: • Java • Groovy • Jetty • ... -
© 2018 CloudBees, Inc. All Rights Reserved. Jenkins for Java
developers Plugins for all common use-cases: • Maven/Gradle/SBT/etc. • Test plugins • Coverage plugins (Cobertura, JaCoCo, ..) • 14 -
© 2018 CloudBees, Inc. All Rights Reserved. Goals 1. Support
running Jenkins with Java 11 2. Packages seamlessly run with both Java 8 and 11 3. Get incremental improvements from Java -
© 2018 CloudBees, Inc. All Rights Reserved. Non-goals… for now
• Go all-in Java 11 • Updating codebase to Java 11 • Adopting modules (JEP-261) • Hardcore optimization for Java 11 17 -
© 2018 CloudBees, Inc. All Rights Reserved. Performance: • JEP-307:
Parallel Full GC for G1 - All instances, OOTB • JEP-318: Epsilon: A No-Op Garbage Collector - Single-shot Jenkins Masters Security: • JEP-332: Transport Layer Security (TLS) 1.3 - HTTPs, Jenkins Remoting & Co • JEP-324/JEP-329: New Key agreement and cryptographic algorithms Experimental features: • JEP-333: ZGC: A Scalable Low-Latency Garbage Collector • JEP-317: Experimental Java-Based JIT Compiler (Graal as JIT) Profiling: • JEP-331: Low-Overhead Heap Profiling - better profiling in production • JEP-328: Flight Recorder - better profiling in production Java 9...11. Why? 18 -
© 2018 CloudBees, Inc. All Rights Reserved. Java 9...11. Risks
• Modules (JEP-261)… • Feature deprecation/removal • Incompatibilities in upstream libraries • Development Tools 19 -
© 2018 CloudBees, Inc. All Rights Reserved. Timeline 09.2017 –
First experiments with Java 9 • Jenkins World 2017 hackfest • Baptiste Mathus, Mark Waite 20 -
© 2018 CloudBees, Inc. All Rights Reserved. Initial state. May
2018 Jenkins can build Java 9+ projects • Using different Java version for build steps • Supported in Shell steps, Maven Pipeline Plugin, etc. • Not supported in Maven Project type 21 -
© 2018 CloudBees, Inc. All Rights Reserved. Initial state. May
2018 Jenkins can build Java 9+ projects • Using different Java version for build steps • Supported in Shell steps, Maven Pipeline Plugin, etc. • Not supported in Maven Project type Jenkins cannot run with Java 9+ • Jenkins Pipeline does not start • Glitches in Freestyle projects • Broken process management 22 -
© 2018 CloudBees, Inc. All Rights Reserved. Timeline 09.2017 –
First experiments with Java 9 06.2018 – Jenkins and Java 10+ hackathon • Organized by Kohsuke Kawaguchi (Jenkins creator) • Blogposts: https://jenkins.io/node/tags/java11/ • Summary slides 23 -
© 2018 CloudBees, Inc. All Rights Reserved. June 2018. Jenkins
& Java 10+ hackathon • 26 contributors • Pipeline is working! • Patches to the core and plugins • Development tools • Exploratory testing 24 https://jenkins.io/blog/2018/06/19/jenkins-java10-hackathon-day2/ -
© 2018 CloudBees, Inc. All Rights Reserved. June 2018. Meeting
with OpenJDK Jigsaw Team • Open Q&A with Mandy Chung, Paul Sandoz • Got feedback for current and incoming obstacles • Meeting notes: http://bit.ly/jenkins-java10-questions 25 -
© 2018 CloudBees, Inc. All Rights Reserved. Timeline 1. 09.2017
– First experiments with Java 9 2. 06.2018 – Jenkins and Java 10+ hackathon 3. 06.2018 – Java 10 and 11ea preview availability 4. 06.2018 – Jenkins JEP-211 5. 09.2018 – Jenkins runs well on Java 11 RC (22 Sep) 6. 09.2018 – All known fixes to the core are upstreamed 26 -
© 2018 CloudBees, Inc. All Rights Reserved. Java 11 progress
JEP-211 - JEP for Java support Preview in weekly Preview, feature branch GA in weekly GA in LTS We are here 27 -
© 2018 CloudBees, Inc. All Rights Reserved. How to run
Jenkins with Java 11? Vanilla: > docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins-experimental:latest-jdk11 With Pipeline and BlueOcean: > docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins-experimental:blueocean-jdk11 More options: https://jenkins.io/blog/2018/06/17/running-jenkins-with-java10-11/ 28 -
© 2018 CloudBees, Inc. All Rights Reserved. Jenkins - one
of the biggest JVM-based projects in the world 31 -
© 2018 All Rights Reserved. 33 Jenkins • Automation Framework,
[too] flexible • Architecture created before 2008 • Distributed Remote Execution Service • 1500+ plugins for any use-case • 10+ years of backward compatibility -
© 2018 CloudBees, Inc. All Rights Reserved. © 2018 CloudBees,
Inc. All Rights Reserved. Backend 34 • In-house libraries • Executable WAR files • No class isolation -
© 2018 CloudBees, Inc. All Rights Reserved. © 2018 CloudBees,
Inc. All Rights Reserved. Legacy frontend 35 In Jenkins… it’s also Java • Jetty • External web containers • Jelly/Groovy • Stapler Work in progress: JavaScript and REST API -
© 2018 CloudBees, Inc. All Rights Reserved. Off-topic: Shifting gears.
Cloud Native Jenkins & Architecture Aug 31, 2018 – Kohsuke Kawaguchi, “Jenkins: Shifting Gears” https://jenkins.io/blog/2018/08/31/shifting-gears/ 36 -
© 2018 CloudBees, Inc. All Rights Reserved. 37 Lib 1
Lib 2 Lib 3 Plugin 1 Plugin 2 Plugin 3 Lib 4 Lib 5 Plugin 4 -
© 2018 CloudBees, Inc. All Rights Reserved. 39 Dependency updates
• A LOT of dependencies • Low-level logic in plugins ◦ JNA, JavaMelody, ... -
© 2018 CloudBees, Inc. All Rights Reserved. 40 Dependency updates
• A LOT of dependencies • Low-level logic in plugins ◦ JNA, JavaMelody, ... • No class isolation - risk of binary conflicts • Major upgrades / breaking changes in recent versions • JAR Hell -
© 2018 CloudBees, Inc. All Rights Reserved. 41 Dependency updates
Long tail of plugins • Limited maintenance • Already had fun with Jenkins JEP-200 (class deserialization whitelist) -
© 2018 CloudBees, Inc. All Rights Reserved. Dependency updates •
ASM 6 - bytecode Manipulations • Jetty - compatibility patches, TLS 1.3 • Groovy - Pipeline support • JBoss Marshalling - Pipeline support • JNR, JNA - illegal reflective access 42 -
© 2018 CloudBees, Inc. All Rights Reserved. ASM Upgrade ByteCode
Transformer (Binary compatibility engine) • jenkinsci/bytecode-compatibility-transformer FileLeak Detector • kohsuke/file-leak-detector/ Pipeline: CPS Plugin 43 -
© 2018 CloudBees, Inc. All Rights Reserved. Dependencies. Groovy Updated
from 2.4.8 to 2.4.9 • Required for Declarative Pipeline • Updates are tough in Jenkins • Groovy scripting, Pipeline, Script Security, ... Groovy 3.x update??? 44 -
© 2018 CloudBees, Inc. All Rights Reserved. JBoss Marshalling •
Context Persistency in Jenkins Pipeline • Major upgrade required • Data formats are not compatible. Upgradeability issues • First multi-release JAR, many tooling updates 45 -
© 2018 CloudBees, Inc. All Rights Reserved. Modules Expectations Image:
https://www.kdnuggets.com/2016/10/big -data-science-expectation-reality.html -
© 2018 CloudBees, Inc. All Rights Reserved. Modules Expectations Image:
https://www.kdnuggets.com/2016/10/big -data-science-expectation-reality.html Reality -
© 2018 CloudBees, Inc. All Rights Reserved. Modules. Classloading changes
in Java 9+ • http://java9.wtf/class-loading/ • Restricted Bootstrap Class Loader (JEP-261) • Impact: references to default classloaders • Jenkins has a complex classloading logic • Issue: Pipeline Utility Steps (javax.xml.soap.*) 48 new URLClassloader(null, ...) -
© 2018 CloudBees, Inc. All Rights Reserved. Java 9+. Not
longer visible in Default Classloader • java.sql.* • javax.activation.* • javax.annotation.* • javax.jws.* • javax.lang.model.* • javax.rmi.* • javax.script.* • javax.smartcardio.* 49 • javax.sql.* • javax.tools.* • javax.transaction.xa.* • javax.xml.bind.* • javax.xml.crypto.* • javax.xml.soap.* • javax.xml.ws.* -
© 2018 CloudBees, Inc. All Rights Reserved. JAXB • javax.xml.bind
& Co • Widely used in plugins (query) • Detached from Java 9 to a module • Removed from Java 11 (JEP-320) 50 -
© 2018 CloudBees, Inc. All Rights Reserved. JAXB. Workarounds 51
Download the following libraries to the same directory as jenkins.war • jaxb-api-2.3.0.jar (save as jaxb-api.jar) • jaxb-core-2.3.0.1.jar (save as jaxb-core.jar) • jaxb-impl-2.3.0.1.jar (save as jaxb-impl.jar) • javax.activation v.1.2.0 (save as javax.activation.jar) Run: ${JAVA11_HOME}/bin/java \ -p jaxb-api.jar:javax.activation.jar --add-modules java.xml.bind,java.activation \ -cp jaxb-core.jar:jaxb-impl.jar \ -jar jenkins.war --enable-future-java --httpPort=8080 --prefix=/jenkins https://jenkins.io/blog/2018/06/17/running-jenkins-with-java10-11/ -
© 2018 CloudBees, Inc. All Rights Reserved. JAXB. Multi-release WAR?
OpenJDK Jigsaw Team: No solution Option 1: Support conditional JAR extraction • Jenkins Extras Executable WAR: • https://github.com/jenkinsci/extras-executable-war/ Option 2: Bundle JAXB JAR files Option 3: Detaching to a plugin 52 -
© 2018 CloudBees, Inc. All Rights Reserved. CORBA Also removed
from Java 11 (JEP-320) We do not use CORBA We have class deserialization tests depending on it 53 -
© 2018 CloudBees, Inc. All Rights Reserved. CORBA. No Multi-release
Test JARs ☹ • Class Deserialization Attack Tests (SECURITY-218) • CORBA, JAXB • … • Solution: Detach tests to a separate Maven Module + profiles to skip them 54 -
© 2018 CloudBees, Inc. All Rights Reserved. Modules. REAL Adoption
(JEP-261) Plugin => Module mapping Plugin isolation Classloading isolation -
© 2018 CloudBees, Inc. All Rights Reserved. Illegal Reflective Access
(JEP-260) TL;DR: setAccessible(true) in the code Was enforced in Java 9 EA, reverted (JEP-260) Query in GitHub: https://github.com/search?q=org%3Ajenkinsci+setAcce ssible%28true%29&type=Code 56 -
© 2018 CloudBees, Inc. All Rights Reserved. 57 • Internal
Libraries • Upstream Dependencies • Object serialization frameworks Escape hatch: “--illegal-access=permit” Illegal Reflective Access -
© 2018 CloudBees, Inc. All Rights Reserved. 58 Java Web
Start deprecated - Java 9 removed - Java 11 https://www.oracle.com/technetwork/j ava/eol-135779.html https://www.oracle.com/technetwork/j ava/javase/javaclientroadmapupdate 2018mar-4414431.pdf -
© 2018 CloudBees, Inc. All Rights Reserved. Java Web Start
in Jenkins UX: Used for Agent startup from Web UI 59 https://support.cloudbees.com/hc/en-us/articles/217423827-Ho w-to-Install-Several-Windows-Slaves-as-a-Service- -
© 2018 CloudBees, Inc. All Rights Reserved. Java Web Start.
Our Plan No industry-standard replacement so far Our plan: • Update Documentation / Web UI • Detach Java Web Start to a plugin 60 -
© 2018 CloudBees, Inc. All Rights Reserved. Jenkins & Docker.
Packaging https://hub.docker.com/r/jenkins/ AND https://hub.docker.com/r/jenkinsci 63 -
© 2018 CloudBees, Inc. All Rights Reserved. Docker Packaging Our
base image: https://hub.docker.com/_/openjdk/ 64 -
© 2018 CloudBees, Inc. All Rights Reserved. Docker Packaging Our
base image: https://hub.docker.com/_/openjdk/ 65 • Maintained by Docker, not OpenJDK • Lags behind the releases, esp. for EA • No multi-platform support (Arm & Co) • Unclear future post Jan 2019 • Not recommended by the OpenJDK Jigsaw Team -
© 2018 CloudBees, Inc. All Rights Reserved. Docker Packaging OpenJDK
Jigsaw Team: • Build OpenJDK on your own We consider: • Build OpenJDK on your own • AdoptOpenJDK packages 66 -
© 2018 CloudBees, Inc. All Rights Reserved. DockerHub flow •
Build on DockerHub for some components • Dockerfile in the root • Branches, branches, branches • Separating build and packaging? 67 https://github.com/jenkinsci/docker-jnlp-slave -
© 2018 CloudBees, Inc. All Rights Reserved. Goal: Test with
Java 11, not build with JDK 11 -
© 2018 CloudBees, Inc. All Rights Reserved. Goal: Test with
Java 11, not build with JDK 11 • Debugging with Java 11 • No separation in development tools -
© 2018 CloudBees, Inc. All Rights Reserved. Updates we needed.
Upstream projects • Maven Compiler Plugin 3.8.0 • Animal Sniffer • Maven Enforcer & Rule plugins • Annotation Indexer / SezPoz • JaCoCo, Cobertura • Access Modifier • … 72 -
© 2018 CloudBees, Inc. All Rights Reserved. Updates we needed.
Internal • Plugin and Core Parent POMs • Maven HPI Plugin • Maven Jenkins Dev Plugin (jetty-maven-plugin fork) • All test tools ◦ Jenkins Test Harness (functional tests) ◦ Acceptance Test Harness (UI, etc.) ◦ Plugin Compatibility Tester • ... 73 -
© 2018 CloudBees, Inc. All Rights Reserved. FindBugs • We
still use FindBugs • Random build failures with Java 11 ◦ JENKINS-53692, JENKINS-53720 • FindBugs is R.I.P., right? Mitigation: • Disabling failing rules for now 74 -
© 2018 CloudBees, Inc. All Rights Reserved. FindBugs workaround 75
https://github.com/jenkinsci/jenkins/blob/java11-support/src /findbugs/findbugs-excludes.xml#L13-L15 . . . -
© 2018 CloudBees, Inc. All Rights Reserved. FindBugs => SpotBugs
SpotBugs 3.1.8+ is JDK11-compatible • spotbugs/spotbugs/issues/711 76 -
© 2018 CloudBees, Inc. All Rights Reserved. Javadoc issues •
Randomly fails in Docker builds (JENKINS-53788) • OpenJDK Bug? Under triage 77 javadoc: error - An internal exception has occurred. (java.lang.AssertionError) Please file a bug against the javadoc tool via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include error messages and the following diagnostic in your report. Thank you. java.lang.AssertionError at jdk.compiler/com.sun.tools.javac.util.Assert.error(Assert.java:155) at jdk.compiler/com.sun.tools.javac.util.Assert.checkNonNull(Assert.java:62) at jdk.compiler/com.sun.tools.javac.comp.Annotate.fromAnnotations(Annotate.java:167) at jdk.compiler/com.sun…..lambda$annotateTypeSecondStage$5(Annotate.java:1025) at jdk.compiler/com.sun.tools.javac.comp.Annotate.flush(Annotate.java:194) at ... -
© 2018 CloudBees, Inc. All Rights Reserved. Jenkins-on-Jenkins 78 •
Yes, we run CI/CD with Jenkins • https://ci.jenkins.io/ • Jenkins can build projects with JDK 11 while running with Java 8 -
© 2018 CloudBees, Inc. All Rights Reserved. Jenkins-on-Jenkins • Testing
with Java 8 and 11 • Testing on Linux and Windows • Doubling the tests cost • We got used to the Java 8-only world... 79 -
© 2018 CloudBees, Inc. All Rights Reserved. Jenkins-on-Jenkins. Changes •
OpenJDK11 on tool infrastructure • OpenJDK11 Docker images for agents • OpenJDK11 Docker images for tools • Pipeline Library: Java version as parameter • Pipeline Library: Version matrix 80 -
© 2018 CloudBees, Inc. All Rights Reserved. Jenkins-on-Jenkins. Demo 81
https://ci.jenkins.io/blue/organizatio ns/jenkins/Core%2Fjenkins/detail/j ava11-support/18/pipeline -
© 2018 CloudBees, Inc. All Rights Reserved. Jenkins & Java
11 Current Status -
© 2018 CloudBees, Inc. All Rights Reserved. Current state JEP-211
- JEP for Java support All functional patches are in Jenkins Weekly 83 Preview in weekly Preview, feature branch GA in weekly GA in LTS We are here -
© 2018 CloudBees, Inc. All Rights Reserved. Preview in Weekly.
Blockers “Pipeline: Support” release to enable Pipeline JBoss Marshalling update caused regressions Needs investigation/fix 84 -
© 2018 CloudBees, Inc. All Rights Reserved. Preview in Weekly.
Blockers “Pipeline: Support” release to enable Pipeline JBoss Marshalling update caused regressions Needs investigation/fix Official Docker packaging for masters/agents 85 -
© 2018 CloudBees, Inc. All Rights Reserved. Preview in Weekly.
Blockers “Pipeline: Support” release to enable Pipeline JBoss Marshalling update caused regressions Needs investigation/fix Official Docker packaging for masters/agents CI flow for Java 11 JTH - almost done ATH - work-in-progress PCT - not started 86 -
© 2018 CloudBees, Inc. All Rights Reserved. Post-GA. Java 11
• Java 11 maintenance (special team) • Cleanup of Illegal Reflective Access and technical debt • Jenkins X (serverless) • Jenkins Evergreen • … 87 -
© 2018 CloudBees, Inc. All Rights Reserved. Post-GA. Java 12+
Jenkins is a part of Quality Outreach • https://wiki.openjdk.java.net/display/quality/Quality+Outreach Enabling testing early is critical for projects More breaking changes are coming • sun.misc.Signal deprecation • Illegal Reflective Access enforcement? 88 -
© 2018 CloudBees, Inc. All Rights Reserved. Java 11. Where
can you help? 1. Feedback is needed! https://jenkins.io/blog/2018/06/17/running-jenkins-with-java10-11/ 2. Test your projects/plugins with Java 11 3. Lots of “minor” fixes/improvements (see JENKINS-40689) 89 -
© 2018 CloudBees, Inc. All Rights Reserved. Java Expert? Join
the Jenkins Platform SIG! https://jenkins.io/sigs/platform/ 90 Jenkins needs you! -
© 2018 CloudBees, Inc. All Rights Reserved. What is platform
SIG? 20+ members, regular meetings • Java support. Java != OpenJDK • Operating Systems & Architectures • Jenkins packaging 91 -
© 2018 CloudBees, Inc. All Rights Reserved. Java 11 adoption
• Was a lot easier than we expected • Java 8 => 10. PoC in 2 days • Java 10 => 11. PoC in 1 day • Multiply by 10+ for upstreaming changes ◦ Productization overheads ◦ Ensuring compatibility ◦ Build/test automation 93 -
© 2018 CloudBees, Inc. All Rights Reserved. What’s missing? •
Java 11 compatibility scanners for Maven Dependencies • Maven Plugins are still behind, it’s getting better • Tools for Java 11 and JavaEE (multi-release WARs, JAXB, etc.) 94 -
© 2018 CloudBees, Inc. All Rights Reserved. Takeaways • Java
11 is here to stay • Java release cycles speedup • Start working towards Java 11 upgrades now • Migrate from Oracle JDK if you use it as Freemium ◦ OpenJDK if you target latest versions ◦ AdoptOpenJDK ◦ Other packages 95 -
© 2018 CloudBees, Inc. All Rights Reserved. Useful links. Jenkins
Platform SIG • https://jenkins.io/sigs/platform/ JEP-211: Java 11 support in Jenkins • https://github.com/jenkinsci/jep/tree/master/jep/211 Running with Java 11: • https://jenkins.io/blog/2018/06/17/running-jenkins-with-java10-11/ My demo: • https://github.com/oleg-nenashev/demo-jenkins-config-as-code/tree /java11-support 96 -
© 2018 CloudBees, Inc. All Rights Reserved. Useful links. Java
9, 10, 11, … JEPs and Oracle articles in the slides Release notes for Java 9..11 (deprecations, etc.) Leonardo Zanivan, “It’s time! Migrating to Java 11” • https://medium.com/criciumadev/its-time-migrating-to-java-11-5eb3868354f9 Stephen Colebourne, “Oracle's Java 11 trap - Use OpenJDK instead!” • https://blog.joda.org/2018/09/do-not-fall-into-oracles-java-11-trap.html Nicolai Parlog • http://java9.wtf/ 97 -
© 2018 CloudBees, Inc. All Rights Reserved. Contacts: E-mail: [email protected]
GitHub: oleg-nenashev Twitter: @oleg_nenashev QUESTIONS? go.cloudbees.com 98