Leitner Inauguration Lecture Chalmers University of Technology

Transcript

1. Performance Optimization at the Intersection of Software and Systems Philipp Leitner Chalmers University of Technology & University of Gothenburg · 2026
4. “Healthcare.gov was officially launched on 1 October 2013 (…) High website demand (…) caused the website to go down within 2 hours of launch.” “A total of 6 users completed and submitted their applications and selected a health insurance plan on the first day.” Source: rctom.hbs.org/submission/the-failed-launch-of-www-healthcare-gov/
5. “performance tests were executed at most once per release” “35 performance requirements were never tested in the last 6 releases, i.e., since 2 years” L. Traini. “Exploring performance assurance practices and challenges in agile software development: an ethnographic study.” EMSE 2022. 5/37
6. Performance engineering is hard
7. Non-Deterministic Results 7/37
8. Software Systems code infrastructure & platforms Performance Usage workloads 8/37
9. Looking back
10. SW bloat · SIGMETRICS’24 ju2jmh · TSE’23 TEP-GNN · PROFES’22 PerformanceHat · ICSE’19 BLAFS · SoCC’25 microbenchmarking · FSE’20 JMH pitfalls · TSE’21 JMH parameters · ICPE’21 continuous experiments · IST’18 10/37 tracing · FGCS’25
11. SW bloat · SIGMETRICS’24 ju2jmh · TSE’23 TEP-GNN · PROFES’22 PerformanceHat · ICSE’19 BLAFS · SoCC’25 microbenchmarking · FSE’20 JMH pitfalls · TSE’21 JMH parameters · ICPE’21 continuous experiments · IST’18 11/37 tracing · FGCS’25
12. How do cloud application developers actually learn about (performance) faults? 12/37
13. How do cloud application developers actually learn about (performance) faults? 12/37
14. Development and production don’t talk Development · IDE Production · monitoring context switching J. Cito, P. Leitner, M. Rinard, H. C. Gall. “Interactive Production Performance Feedback in the IDE.” ICSE 2019. 13/37
15. Augmenting Source Code with Runtime Data Development ≠ Production Many performance problems surface only under real production workloads. Bring it into the code Production monitoring data is mapped onto the AST and shown in-situ — right on the method or loop, no dashboard switch. J. Cito, P. Leitner, M. Rinard, H. C. Gall. “Interactive Production Performance Feedback in the IDE.” ICSE 2019. 14/37
16. PerformanceHat: Runtime Traces in the IDE Predicts the cost of new code before it ships. Faster than Kibana — no slowdown on other tasks. J. Cito, P. Leitner, M. Rinard, H. C. Gall. “Interactive Production Performance Feedback in the IDE.” ICSE 2019. 15/37
17. Impact? Coverage in Adrian Colyer’s morning paper (and various keynotes) At least 1 US startup company explicitly referencing our work PerformanceSpotter (SAP HANA) J. Cito, P. Leitner, H. C. Gall, A. Dadashi, A. Keller, A. Roth. “Runtime Metric Meets Developer – Building Better Cloud Applications Using Feedback” Onward! 2015. 16/37
18. SW bloat · SIGMETRICS’24 ju2jmh · TSE’23 TEP-GNN · PROFES’22 PerformanceHat · ICSE’19 BLAFS · SoCC’25 microbenchmarking · FSE’20 JMH pitfalls · TSE’21 JMH parameters · ICPE’21 continuous experiments · IST’18 17/37 tracing · FGCS’25
19. Software Microbenchmarking The unit tests of performance — powerful, but surprisingly hard to trust. 18/37
20. Software Microbenchmarking The unit tests of performance — powerful, but surprisingly hard to trust. Rarely, and ad hoc Measurement pitfalls Environment adds noise Few Java projects test performance systematically. Common JMH bad practices quietly skew results. Variability ranges from 0.03% to over 100% (CV). state of practice · ICPE’17 JMH bad practices · TSE’21 μbench in cloud · EMSE’19 with C. Laaber, C.-P. Bezemer, J. Scheuner. 18/37
21. Slow Execution times of modest microbenchmark suites are measured in hours C. Laaber, P. Leitner. “An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment” MSR 2018. 19/37
22. C. Laaber, S. Würsten, H.C. Gall, P. Leitner. “Dynamically Reconfiguring Software Microbenchmarks” FSE 2020. 20/37
23. C. Laaber, S. Würsten, H.C. Gall, P. Leitner. “Dynamically Reconfiguring Software Microbenchmarks” FSE 2020. 21/37
24. ~66% - ~80% time savings Without meaningfully changing results C. Laaber, S. Würsten, H.C. Gall, P. Leitner. “Dynamically Reconfiguring Software Microbenchmarks” FSE 2020. 22/37
25. Looking forward Generative AI for performance engineering?
26. Initial LLMs-for-Performance Work Used standard LLM benchmarks (Python, toy examples) Algorithmic (toy) problems Performance assessment by timing unit test execution duration Real performance issues are a lot more complex than that 24/37 Coignion et al. “A performance study of LLMgenerated code on LeetCode.” EASE 2024.
27. PerfOpt A more realistic benchmark dataset 65 cases, mined real-life performance improvements From Kafka, Netty, Presto, RoaringBitmap Each change is associated with 1+ benchmark and a humanwritten ground truth L. Yi, G. Gay, P. Leitner. “An Experimental Study of Real-Life LLMProposed Performance Improvements.” arXiv 2025 · arXiv:2510.15494. 25/37
28. Step 1: Mining Real-World Performance Issues Identify Performance Hotspots (PRs) OSS Projects (Kafka, Netty, Presto, RB) Verify via Microbenchmark PerfOpt Dataset (65 Tasks) Step 4: Evaluation Pipeline Step 2: Generating LLM Optimizations Prompts (Source Code, Test, Problem, Benchmark) Prompts Combination RQ1: Performance Improvement Call LLMs (Open AI o4-mini, Gemini 2.5 Pro, DeepSeek V3, DeepSeek R1) Generated Patches Automated Repair Loop (Compile/Build & Test) Error Feedback Run JMH Benchmarks (Warmup+Meas.) RQ2: Human Competitiveness RQ3: Developer-LLM Similarity Pass? Plausible Patches Patch Labeling Step 3: Constructing the Benchmark L. Yi, G. Gay, P. Leitner. “An Experimental Study of Real-Life LLM-Proposed Performance Improvements.” arXiv 2025 · arXiv:2510.15494. 26/37
29. L. Yi, G. Gay, P. Leitner. “An Experimental Study of Real-Life LLM-Proposed Performance Improvements.” arXiv 2025 · arXiv:2510.15494. 27/37
30. Two-fold issue: LLMs struggle with root cause analysis and repair L. Yi, G. Gay, P. Leitner. “An Experimental Study of Real-Life LLM-Proposed Performance Improvements.” arXiv 2025 · arXiv:2510.15494. 28/37
31. Software code 29/37
32. Closing the Profiling Loop L. Yi, L. Salerno, G. Gay, P. Leitner. “Closing the Profiling Loop: An Agentic Framework for Performance Optimization.” Under submission, 2026. 30/37
33. Agentic Architectures: Design & Evaluation What makes it work Evidence-grounded refinement Re-profile after every patch — the biggest single gain. Experiential memory Patterns & anti-patterns reused across tasks → faster convergence (+16% first-iteration). Generalizes: on Python (SWE-Perf) 95% plausible, 1.42×. Beats a strong single-pass LLM on both speed and correctness. L. Yi, L. Salerno, G. Gay, P. Leitner. “Closing the Profiling Loop: An Agentic Framework for Performance Optimization.” Under submission, 2026. 31/37
34. Conclusions
35. 33/37
36. 33/37
37. 33/37
38. https://icpe2027.spec.org 34/37
39. Thank you to my team! (and sponsors) 2018 Funded by 35/37 2024 2026
40. References [1] L. Yi, L. Salerno, G. Gay, P. Leitner. “Closing the Profiling Loop: An Agentic Framework for Performance Optimization.” Under submission, 2026. [2] L. Yi, G. Gay, P. Leitner. “An Experimental Study of Real-Life LLM-Proposed Performance Improvements.” arXiv 2025. [3] S. Talluri, S. Eismann, J. Scheuner, A. Iosup, P. Leitner, C. Abad, E. van Eyk. “Let’s Trace It: Fine-Grained Serverless Benchmarking.” FGCS 2025. [4] H. Zhang, M. Alhanahnah, P. Leitner, A. Ali-Eldin. “BLAFS: A Bloat-Aware Container File System.” SoCC 2025. [5] H. Zhang, M. Alhanahnah, F. A. Ahmed, D. Fatih, P. Leitner, A. Ali-Eldin. “Machine Learning Systems are Bloated and Vulnerable.” SIGMETRICS 2024. [6] Coignion et al. “A Performance Study of LLM-Generated Code on LeetCode.” EASE 2024. [7] M. Jangali, Y. Tang, N. Alexandersson, P. Leitner, J. Yang, W. Shang. “Automated Generation and Evaluation of JMH Microbenchmark Suites from Unit Tests.” TSE 2023. [8] H. P. Samoaa, A. Longa, M. Mohamad, M. H. Chehreghani, P. Leitner. “TEPGNN: Accurate Execution Time Prediction of Functional Tests Using GNNs.” PROFES 2022. [9] L. Traini. “Exploring Performance Assurance Practices and Challenges in Agile Software Development.” EMSE 2022. 37/37 [10] D. Costa, C.-P. Bezemer, P. Leitner, A. Andrzejak. “What’s Wrong with My Benchmark Results? Studying Bad Practices in JMH Benchmarks.” TSE 2021. [11] H. Samoaa, P. Leitner. “An Exploratory Study of the Impact of Parameterization on JMH Measurement Results.” ICPE 2021. [12] C. Laaber, S. Würsten, H. C. Gall, P. Leitner. “Dynamically Reconfiguring Software Microbenchmarks.” FSE 2020. [13] C. Laaber, J. Scheuner, P. Leitner. “Software Microbenchmarking in the Cloud. How Bad is it Really?” EMSE 2019. [14] J. Cito, P. Leitner, M. Rinard, H. C. Gall. “Interactive Production Performance Feedback in the IDE.” ICSE 2019. [15] G. Schermann, J. Cito, P. Leitner, U. Zdun, H. C. Gall. “We’re Doing It Live: A Multi-Method Empirical Study on Continuous Experimentation.” IST 2018. [16] C. Laaber, P. Leitner. “An Evaluation of Open-Source Software Microbenchmark Suites for Continuous Performance Assessment.” MSR 2018. [17] P. Leitner, C.-P. Bezemer. “An Exploratory Study of the State of Practice of Performance Testing in Java-Based Open Source Projects.” ICPE 2017. [18] J. Cito, P. Leitner, H. C. Gall, A. Dadashi, A. Keller, A. Roth. “Runtime Metric Meets Developer — Building Better Cloud Applications Using Feedback.” Onward! 2015.