Performance Optimization With LLMs

Transcript

1. Performance Optimization With LLMs Philipp Leitner Chalmers University of Technology & University of Gothenburg · 2026
2. Define software performance engineering Software Performance Engineering (SPE) is a systematic, quantitative approach to ensuring that software systems meet their performance requirements (such as response time, throughput, scalability, and resource utilization) throughout the entire software development lifecycle. Response time Throughput Scalability Resource utilization How long a single request takes How much work per unit of time How it behaves as load grows CPU, memory, I/O, cost 2 / 16
3. IN D U S TRY R E A LITY Tested Not tested 50 41 No. of Performance Requirements 40 30 20 16 10 18 4 0 Priority 1 Priority 2 Priority 3 Priority 4 L. Traini. “Exploring Performance Assurance Practices and Challenges in Agile Software Development: An Ethnographic Study.” EMSE 2022 3 / 16
4. Can LLMs (also) do performance optimization? 4 / 16
5. LLM S FO R P E R FO R M A N C E Generate optimized code Optimize existing code Write new code that is efficient from the start. Make an existing, real code base faster. Without breaking it. Problem spec LLM New code Code + benchmark LLM Faster patch 5 / 16
6. EXISTIN G B EN C H M A R K S test_3sum.py # input prompt """ Given an integer array nums, return all the triplets [nums[i], nums[j], nums[k]] such that i != j, i != k, and j != k, and nums[i] + nums[j] + nums[k] == 0. Notice that the solution set must not contain duplicate triplets. """ class Solution: def threeSum(self, nums: List[int]) -> List[List[int]]: … <generate> … # benchmarking code def check(): Solution().threeSum([82597, -9243, 83030, ...]) Solution().threeSum([0, 0, 0, 0, ...]) Solution().threeSum([0, 0, -1, -1, ...]) import pytest @pytest.mark.benchmark(group="3sum") def test_3sum_generated_1(benchmark): benchmark(check) T. Coignion, C. Quinton, R. Rouvoy. “A Performance Study of LLM-Generated Code on LeetCode.” EASE 2024 6 / 16
7. EXISTIN G B EN C H M A R K S test_3sum.py # input prompt """ Given an integer array nums, return all the triplets [nums[i], nums[j], nums[k]] such that i != j, i != k, and j != k, and nums[i] + nums[j] + nums[k] == 0. Notice that the solution set must not contain duplicate triplets. """ class Solution: def threeSum(self, nums: List[int]) -> List[List[int]]: # generated code nums.sort() res = [] for i in range(len(nums) - 2): ... Real performance issues are a lot more complex than that. # benchmarking code def check(): Solution().threeSum([82597, -9243, 83030, ...]) Solution().threeSum([0, 0, 0, 0, ...]) Solution().threeSum([0, 0, -1, -1, ...]) import pytest @pytest.mark.benchmark(group="3sum") def test_3sum_generated_1(benchmark): benchmark(check) T. Coignion, C. Quinton, R. Rouvoy. “A Performance Study of LLM-Generated Code on LeetCode.” EASE 2024 7 / 16
8. B EN C H M A R K Lirong Yi PerfOpt: a realistic benchmark dataset lead author Root-cause categories (number of tasks) 65 4 1+ real-life performance improvements, mined from open source Java projects: Kafka · Netty · Presto · RoaringBitmap JMH benchmark per task, plus the developer's fix as ground truth Excessive Objec t Allocation 13 Inefficient API Usage 9 Inefficient Algorithm 9 Unnec essary Generality 9 Redundant Computation 6 Synchroniz ation Overhead 6 Inefficient Loop 3 Inappropriate Data Structure 3 Inefficient Arithmetic 3 Inefficient String / Encoding Excessive I/O 2 1 L. Yi, G. Gay, P. Leitner. “An Experimental Study of Real-Life LLM-Proposed Performance Improvements.” arXiv 2025 · arXiv:2510.15494 8 / 16
9. R ESU LTS · PER FO R M A N C E LLM patches improve performance (barely) HO W TO RE AD IT pss = speedup score; 1.0 = no change, > 1 = faster Red dashed: original code (baseline) Blue dotted: developer solution (median) Context helps With no hints, LLM patches barely change performance. Not human-competitive Developer solutions remain superior to every LLM configuration. L. Yi, G. Gay, P. Leitner. “An Experimental Study of Real-Life LLM-Proposed Performance Improvements.” arXiv 2025 · arXiv:2510.15494 9 / 16
10. TH E M IS S IN G C O N TE X T Why do LLMs struggle? They only see the code What an LLM sees The source code — and nothing else Software Systems code infrastructure & platforms What it doesn’t see Performance Usage The system it runs on, and how it is actually used. Performance emerges from all three. Fixing it from the code alone is guesswork. workloads 10 / 16
11. A N A G EN TIC FR A M EW O R K Closing the profiling loop ↻ Evidence-grounded refinement · re-profile after every patch 1 2 3 Performance Analyzer Patch Generator Diagnoses the bottleneck explore code · retrieve patterns Diagnosis report 4 Performance Evaluator Reflector Proposes a code change Measures the change Learns from the evidence explore code · edit code build & test · bench · profiler reasoning trace · hotspots test errors Candidate patch Plausible patch + speedup Lessons learned retrieve Experiential memory long-term, reused across tasks distill ✓ Patterns ✗ Anti-patterns ✗ Pitfalls L. Yi, L. Salerno, G. Gay, P. Leitner. “Closing the Profiling Loop: An Agentic Framework for Performance Optimization.” Under submission, 2026 11 / 16
12. A N A G EN TIC FR A M EW O R K Faster code Better-performing code than a single-pass LLM 12 / 16
13. A N A G EN TIC FR A M EW O R K Faster code … at a price Better-performing code than a single-pass LLM BUT 45+ min Tokens < Human Long run-time Higher token consumption Not human-competitive for a single optimization task — benchmarking and profiling dominate every loop iteration means more LLM calls — cost and sustainability still nowhere near what experienced performance engineers achieve 13 / 16
14. W H ER E D O ES TH IS LEAVE U S? 1 2 3 Give agents systems & usage context Keep developers in the loop AI can’t fully capture the human secret sauce Production profiles, workload traces and infrastructure data — not just source code. The agent does the legwork — profiling, benchmarking, candidate patches. Engineers decide. Experienced engineers still outperform the agent — intuition and domain knowledge are hard to automate. 14 / 16
15. SAVE TH E D ATE Join us at ICPE 2027 in Gothenburg We are looking for industrial sponsors Meet the performance engineering community from academia and industry. May 24–28, 2027 18th ACM/SPEC ICPE Gothenburg, Sweden Talk to me after the session → icpe2027.spec.org Program, calls and registration 15 / 16
16. FU R TH E R R E A D IN G References 1 Closing the Profiling Loop: An Agentic Framework for Performance Optimization. L. Yi, L. Salerno, G. Gay, P. Leitner. Under submission, 2026. 2 An Experimental Study of Real-Life LLM-Proposed Performance Improvements. L. Yi, G. Gay, P. Leitner. arXiv 2025 (arXiv:2510.15494). 3 A Performance Study of LLM-Generated Code on LeetCode. T. Coignion, C. Quinton, R. Rouvoy. EASE 2024. 4 Exploring Performance Assurance Practices and Challenges in Agile Software Development: An Ethnographic Study. L. Traini. EMSE 2022. 16 / 16