On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Support

Paper: https://arxiv.org/abs/2203.10356
Talk: https://youtu.be/VoegPhYnh84
Code: https://github.com/miguelvelezmj25/icse22-sm/

Transcript

1. On Debugging the Performance of Configurable Software Systems Miguel Velez (CMU -> Google) Norbert Siegmund (Leipzig) Pooyan Jamshidi (UofSC & Google) Sven Apel (Saarland) Christian Kästner (CMU) Developer Needs and Tailored Tool Support
2. Most software is con fi gurable
3. Indexing Encryption Compression Logging Core functionality Configurable System Need: Need: Need:
4. Indexing Encryption Compression Logging Core functionality Configurable System
6. 6 Performance Execution time Energy consumption Operational costs
7. Developer • Design, implement, and maintain efficient software • Debug surprising performance behaviors
8. 8 59% - Con fi guration-related performance issues 61% - Average fi x time of 5 weeks 50% - Con fi guration-related patches
10. Traditional Off-the-Shelf Profilers
11. Searching Inefficient Coding Patterns
12. Program Debugging Techniques 12
14. 14 Limited empirical evidence of the usefulness to help debug the performance of con fi gurable systems
17. Developer • Understand how options affect performance • Design, implement, and maintain efficient software Help developers debug the performance of configurable software systems
18. Help developers debug the performance of configurable software systems
19. Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems
20. Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems Tailor Ingredients CPU Profiling Program Slicing Performance Modeling Global Local
22. 22 User Study to Identify Information Needs
23. 23 User Study to Identify Information Needs Overview of Ingredients
24. 24 User Study to Identify Information Needs Overview of Ingredients Tailor and Evaluate Ingredients
25. 25 Overview of Ingredients Tailor and Evaluate Ingredients User Study to Identify Information Needs
26. Information Needs RQ1: What information do developers look for when debugging the performance of con fi gurable systems? RQ2: What is the process that developers follow and the activities that they perform to obtain this information?
27. Information Needs RQ1: What information do developers look for when debugging the performance of con fi gurable systems? RQ2: What is the process that developers follow and the activities that they perform to obtain this information? Exploratory User Study
28. Exploratory User Study
29. Exploratory User Study Influencing Options The option or interaction causing the unexpected performance behavior
30. Exploratory User Study Influencing Options Option Hotspots The methods where options affect the performance of the system
31. Exploratory User Study Influencing Options Option Hotspots Cause-Effect Chain How in fl uencing options are used in the implementation to directly and indirectly affect the performance of option hotspots
32. Exploratory User Study Influencing Options Option Hotspots Cause-Effect Chain Compared problematic con fi guration to a non-problematic con fi guration
34. User Study to Identify Information Needs Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems
35. 35 Tailor and Evaluate Ingredients User Study to Identify Information Needs Overview of Ingredients
36. The option or interaction causing the unexpected performance behavior Influencing Options
37. The option or interaction causing the unexpected performance behavior Influencing Options Global Performance-In fl uence Models
38. … 23 seconds 28 seconds 20 seconds Con fi gurations Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models
39. … 23 seconds 28 seconds 20 seconds Con fi gurations Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models
40. … 23 seconds 28 seconds 20 seconds Con fi gurations Performance T = 25 + 3· - 5· + 9· · Performance-Influence Models
41. Modeling Global T = 25 + 3· - 5· + 9· · Influencing Options
42. The methods where options affect the performance of the system Option Hotspots
43. The methods where options affect the performance of the system Option Hotspots Local Performance-In fl uence Models
44. Modeling Local Tfoo = 5 + 3· + 9· · Option Hotspots
45. How in fl uencing options are used in the implementation to directly and indirectly affect the performance of option hotspots Cause-Effect Chain
46. How in fl uencing options are used in the implementation to directly and indirectly affect the performance of option hotspots Cause-Effect Chain CPU Pro fi ling & Program Slicing
47. CPU Profiling Cause-Effect Chain
48. Program Slicing Cause-Effect Chain
49. Overview of Ingredients Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems CPU Profiling Program Slicing Modeling Global Local
50. 50 User Study to Identify Information Needs Overview of Ingredients Tailor and Evaluate Ingredients
51. Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing Performance Modeling Global Local
52. Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing Performance Modeling Global Local Information Providers
53. Influencing Options Performance Modeling Global
54. Influencing Options Performance Modeling Global
55. Influencing Options Performance Modeling Global
56. Influencing Options Performance Modeling Global
57. Influencing Options Performance Modeling Global
58. Option Hotspots Performance Modeling Local
59. Option Hotspots Performance Modeling Local
60. Option Hotspots Performance Modeling Local
61. Option Hotspots Performance Modeling Local
62. Cause-Effect Chain CPU Profiling Program Slicing
63. Cause-Effect Chain CPU Profiling
64. Cause-Effect Chain CPU Profiling
65. Cause-Effect Chain CPU Profiling
66. Cause-Effect Chain CPU Profiling
67. Cause-Effect Chain CPU Profiling
68. Cause-Effect Chain Program Slicing
69. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO, "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
70. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO, "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
71. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO, "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
72. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO, "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
73. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO, "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
74. Main.main(…) Cursor.put(…) Main.init(…) Database.init(…) def init(Database db) Database.checkForNullParam(db.name, "dbName"); Log.msg(Level.INFO, "db " + db.name + " open"); ... if(db.replicated) configReplicated(...); ... this.cacheMode = db.cacheMode; this.commit = db.trans ? true : false; this.sync = db.dups ? true : false; if(db.evict) Evictor.init(db.evictorThreads); ... Cause-Effect Chain Program Slicing
75. Influencing Options Option Hotspots Cause-Effect Chain Performance Modeling Global
76. Influencing Options Option Hotspots Cause-Effect Chain Performance Modeling Global Local Performance Modeling
77. Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing Performance Modeling Global Local Performance Modeling
78. Influencing Options Option Hotspots Cause-Effect Chain CPU Profiling Program Slicing Performance Modeling Global Local Performance Modeling GLIMPS
79. Evaluation
80. Evaluation RQ1: To what extent do the designed information providers help developers debug the performance of con fi gurable software systems?
81. Evaluation RQ1: To what extent do the designed information providers help developers debug the performance of con fi gurable software systems? Validation User Study Confirmatory User Study
82. “Within-subject” study Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS System: Density Converter
83. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
84. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
85. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS “Within-subject” study System: Density Converter
86. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS SCALE +86.3 AConverter.compress(…) +67.4 “Within-subject” study System: Density Converter
87. Validation User Study 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Control - Did not use GLIMPS Treatment - Used GLIMPS SCALE +86.3 AConverter.compress(…) +67.4 “Within-subject” study System: Density Converter Information providers support information needs
88. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
89. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
90. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
91. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
92. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study
93. 30’ 35’ 45’ 05’ 10’ 20’ 25’ 15’ 40’ 50’ Influencing Options Option Hotspots Cause-Effect Chain Confirmatory User Study 55’ 60’ Treatment - Used GLIMPS Control - Did not use GLIMPS System: Berkeley DB Between-subject study Information providers help developers debug the performance of complex con fi gurable systems
94. Influencing Options Option Hotspots Cause-Effect Chain Help developers debug the performance of configurable software systems CPU Profiling Program Slicing Performance Modeling Global Local Tailor and Evaluate Ingredients
95. On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Support
96. 96 Overview of Ingredients Tailor and Evaluate Ingredients CONTRIBUTION User Study to Identify Information Needs CONTRIBUTION White-box Performance-In fl uence Modeling CONTRIBUTION
97. Compositionality Compression Insights!
98. Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code System Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose 98
99. Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 99 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
100. Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 100 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
101. Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 101 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
102. Performance-In fl uence Model Local Performance-In fl uence Model Region Local Performance-In fl uence Model Local Performance-In fl uence Model Local Performance-In fl uence Model Region Region Region Measure System Decompose Measure Regions Compose System 102 Compositionality Performance-in fl uence models can be built by composing models built independently for smaller regions of code
103. def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 103
104. Time TRUE FALSE Time TRUE TRUE TRUE FALSE def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 104
105. Time TRUE 5 FALSE 0 def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 105 Time TRUE TRUE TRUE FALSE
106. Time TRUE 5 FALSE 0 Tprocess = 5· def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 106 Time TRUE TRUE TRUE FALSE
107. Time TRUE 5 FALSE 0 Tprocess = 5· Time TRUE TRUE 3 TRUE FALSE 2 Tconvert = 2· + 1·. · def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 107
108. Tprocess = 5· Tconvert = 2· + 1·. · Compositionality + def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } 108
109. Tprocess = 5· Tconvert = 2· + 1·. · Compositionality + def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } T = 7· + 1· · 109
110. Compositionality System System 27 = 128 con fi gurations 22 + 24 + 22 + 24 + 22 = 38 con fi gurations 110
111. Compositionality System System 27 = 128 con fi gurations 22 + 24 + 22 + 24 + 22 = 38 con fi gurations 111
112. Compositionality System System 27 = 128 con fi gurations 22 + 24 + 22 + 24 + 22 = 38 con fi gurations 112
113. Compositionality Insights! Compression
114. Compression Compression allow us to simultaneously explore paths in multiple independent regions with a few con fi gurations 114
115. Compression if(a) ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds 115
116. Compression if(a) ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 116
117. FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE Compression if(a) ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 117
118. FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a) ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE Time TRUE FALSE Time TRUE FALSE 118 Compression
119. FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a) ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE FALSE 0 Time TRUE FALSE 0 Time TRUE FALSE 0 119 Compression
120. FALSE FALSE FALSE TRUE FALSE FALSE TRUE TRUE FALSE if(a) ... // execution time: 1 second if(b) ... // execution time: 2 seconds if(c) ... // execution time: 3 seconds Time TRUE 1 FALSE 0 Time TRUE 3 FALSE 0 Time TRUE 2 FALSE 0 120 Compression
121. System 22 + 24 + 22 + 24 + 22 = 38 con fi gurations System 4 con fi gurations 121 Compression
122. System 22 + 24 + 22 + 24 + 22 = 38 con fi gurations System 4 con fi gurations 122 Compression
123. System 22 + 24 + 22 + 24 + 22 = 38 con fi gurations System 4 con fi gurations 123 Compression
124. System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 124 Compression
125. System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 125 Compression
126. System 4 con fi gurations FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE TRUE TRUE FALSE TRUE TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE TRUE FALSE FALSE TRUE FALSE 126 Compression
127. Compression Measure Performance Build Model Compositionality System 127
128. Taint Analysis
129. def main() { boolean a = ... process(a, b); } def process(boolean x, boolean y) { if(x) convert(y); ... // execution time: 5 seconds } def convert(boolean x) { if(x) ... // execution time: 3 seconds else ... // execution time: 2 seconds } In fl uencing options 129
130. Analyze Regions Measure Performance Build Model Taint Analysis Compression Compositionality System 130
131. ConfigCrusher Comprex Static Taint Analysis Regions: Control-flow statements Performance measurement: Instrumentation Dynamic Taint Analysis Regions: Methods Performance measurement: Off-the-shelf profiler 131
132. ConfigCrusher Comprex Static Taint Analysis Regions: Control-flow statements Dynamic Taint Analysis Regions: Methods 132 Both prototypes ef fi ciently build accurate and interpretable models Dynamic taint analysis scales to larger systems Method-level granularity does not sacri fi ces compression potential
133. Evaluation
134. 134 ConfigCrusher Comprex Evaluate cost, accuracy, interpretability 13 open-source Java systems Evaluation 50 combinations of Sampling and Machine Learning
135. Results: Density Converter 0 Cost (minutes) Error (MAPE) 0 40 20 Comprex x 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 x x 60 Con fi gCrusher x 135
136. x + + + Results: Density Converter 0 Cost (minutes) Error (MAPE) 0 40 20 Comprex 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 60 Con fi gCrusher + Interpretable x Not interpretable 136
137. x + + + Results: Density Converter 0 Cost (minutes) Error (MAPE) 0 40 20 Comprex 200 random con fi gurations & Random Forest 5 10 200 random con fi gurations & Stepwise linear regression 200 400 100 300 60 Con fi gCrusher + Interpretable x Not interpretable 137 Ef fi ciently build accurate performance- in fl uence models Models are interpretable for debugging purposes
138. Influencing Options Option Hotspots White-box Modeling Global Local 138 Help developers debug the performance of configurable software systems White-box Performance-In fl uence Modeling CONTRIBUTION
139. Thesis Statement Tailoring speci fi c white-box analyses to track how con fi guration options in fl uence the performance of code-level structures in con fi gurable software systems helps developers to (1) ef fi ciently build accurate and interpretable global and local performance-in fl uence models (2) more easily inspect, trace, understand, and debug con fi guration-related performance issues. 139