Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1BRC – Nerd Sniping the Java Community (JavaZone)

1BRC – Nerd Sniping the Java Community (JavaZone)

Your mission, should you decide to accept it, is the following: aggregate temperature values from a CSV file and group them by weather station name. There’s only one caveat: the file has one 1,000,000,000 rows!

This is the task of the "One Billion Row Challenge" which went viral within the Java community earlier this year. Come and join me for this talk where I’ll dive into some of the tricks employed by the fastest solutions for processing the challenge’s 13 GB input file within less than two seconds. Parallelization and efficient memory access, optimized parsing routines using SIMD and SWAR, as well as custom map implementations are just some of the topics which we are going to discuss.

I will also share some of the personal experiences and learnings which I made while running this challenge for and with the community.

Gunnar Morling

September 05, 2024
Tweet

More Decks by Gunnar Morling

Other Decks in Programming

Transcript

  1. #1BRC | @gunnarmorling The Goals Learn something new Have some

    fun along the way Inspire others to do the same
  2. #1BRC | @gunnarmorling The Rules • Java-only; Version of your

    choice • No dependencies • No caching • 10K stations, -99.9°C - +99.9°C • Copying allowed
  3. #1BRC | @gunnarmorling Evaluation Environment • 32 core AMD EPYC™

    7502P (Zen2), 8 Cores used • 128 GB RAM • File on RAM disk • Five runs, slowest and fastest discarded Image © Joe Haupt https://flic.kr/p/2mG3vWA (CC BY-SA 2.0)
  4. #1BRC | @gunnarmorling JEP 454 Foreign Function & Memory API

    https://openjdk.org/jeps/454 Introduce an API by which Java programs can interoperate with code and data outside of the Java runtime. By efficiently invoking foreign functions (i.e., code outside the JVM), and by safely accessing foreign memory (i.e., memory not managed by the JVM), the API enables Java programs to call native libraries and process native data without the brittleness and danger of JNI.
  5. #1BRC | @gunnarmorling Parsing – SIMD Single Instruction, Multiple Data

    https://speakerdeck.com/gunnarmorling/to-the-moon-and-beyond-with-java-17-apis
  6. #1BRC | @gunnarmorling Further Tricks & Techniques • The “spawn

    trick” • Unsafe • Super-scalar execution • Perfect value hashing
  7. #1BRC | @gunnarmorling Getting the Basics Right Gets You Very

    Far CalculateAverage_MeanderingProgrammer.java
  8. #1BRC | @gunnarmorling Lessons Learned • Precise rules, covering corner

    cases • Automate, automate, automate • Build a community • Use dedicated hardware