The story of RefactoringMiner. Slow research, long-term impact

LATECE Seminar @UQAM
In this seminar, I cover over 10 years of research on the RefactoringMiner project by presenting how the state-of-the-art in Refactoring detection evolved over time and the important lessons learned in my career both research-wise and engineering-wise.
I conclude the seminar with the next big challenging problems that need to be addressed by the Refactoring detection research community.

Transcript

1. Refactoring Miner Slow research… long-term impact Nikolaos Tsantalis Concordia University
2. “Grok” was first used in this novel a Martian term meaning to understand something by becoming one with it.
3. Refactoring Miner
4. Alexander Chatzigeorgiou Eleni Stroulia Zhenchang Xing Fabio Rocha
5. Conception CASCON'13 version 0.0
6. • Ideas inspired from UMLDiff • Structure of method bodies is ignored • Method body includes only method calls and field accesses • Most refactorings detected based on signature matching • Only precision is provided • Evaluation included only 3 systems
7. Refactoring motivations MIP
9. Why did I stop working on this research? Ref-Finder (ICSM 2010): "The precision and recall on open source projects were 0.74 and 0.96 respectively." "Since these programs did not document refactorings, we created a set of correct refactorings by running REF-FINDER with a similarity threshold (σ=0.65) and manually verified them. We then measured a recall by comparing this set with the results found using a higher threshold (σ=0.85)"
10. Lesson #1 Don't be intimidated by super results Always question the results
11. Until one day... Danilo Silva Marco Tulio Valente
12. Where was the code?
13. Lesson #2 Always make your code & data available in a repository You never know what it can enable in the future
14. Open Science initiatives M. T. Baldassarre, N. Ernst, B. Hermann, T. Menzies, R. Yedida Mandatory Data Availability field
15. Let's do an empirical study about refactoring We must study Why We Refactor
16. Fetus ICSE'16 version 0.1
17. • Danilo developed the API of RefactoringMiner • Tooling for checking out and parsing Git commits • Infrastructure for monitoring GitHub projects • Automatic generation of emails to contact developers • A web app for thematic analysis
18. Firehouse interview • Monitored 124 GitHub projects between June 8th and August 7th, 2015 • Sent 465 emails and received 195 responses (42%) • +27 commits with a description explaining the reasons • Compiled a catalogue of 44 distinct motivations for 12 well-known refactoring types
19. Motivation Catalogue Artifact https://github.com/aserg- ufmg/why-we-refactor
20. ICSE'16 rejection Reviewer #1: "A major threat to the research is not discussed or considered, that RefFinder has poor recall (0.24 [31]). The authors did a good job of combating the low-precision by manually inspecting results, the low recall is not discussed or dealt with."
21. Lesson #3 Even ICSE reviewers make mistakes Don't get disappointed. Improve, advance, re-submit
22. FSE'16 re-submission • Renamed the tool from RefDetector to RefactoringMiner • Addressed most of reviewers' comments
24. RefDiff MSR'17 RefactoringMiner ...
25. The path of Virtue or Vice Lesson #4
26. Rebirth ICSE’18 version 1.0
27. Limitations of previous approaches 1. Dependence on similarity thresholds • thresholds need calibration for projects with different characteristics 2. Dependence on built versions • only 38% of the change history can be successfully compiled [Tufano et al., 2017] 3. Unreliable oracles for evaluating precision/recall • Incomplete (refactorings found in release notes or commit messages) • Biased (applying a single tool with two different similarity thresholds) • Artificial (seeded refactorings)
28. Thresholds are banned
29. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } After Before
30. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(int count) { List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } After Before
31. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } After Before private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", ports.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; }
32. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } After Before private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", ports.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; }
33. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { } return addresses; } try { addresses[i] = new Address("127.0.0.1", ports.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } After Before
34. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } protected static Address createAddress(String host, int port) { try { return new Address(host, port); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } After Before
35. protected static Address createAddress(String host, int port) { try { return new Address(host, port); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } After Before textual similarity » 30%
36. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } protected static Address createAddress(String host, int port) { try { return new Address(host, port); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } After Before (1) Abstraction
37. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } protected static Address createAddress(String host, int port) { try { return new Address(host, port); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } After Before (1) Abstraction
38. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } protected static Address createAddress(String host, int port) { try { return new Address(host, port); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } After Before (2) Argumentization
39. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } protected static Address createAddress(String host, int port) { try { return new Address("127.0.0.1", ports.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } After Before (2) Argumentization
40. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } protected static Address createAddress(String host, int port) { try { return new Address("127.0.0.1", ports.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } After Before (3) AST Node Replacements
41. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } protected static Address createAddress(String host, int port) { try { return new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } After Before (3) AST Node Replacements
42. private static Address[] createAddresses(int count) { Address[] addresses = new Address[count]; for (int i = 0; i < count; i++) { try { addresses[i] = new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } } return addresses; } private static List<Address> createAddresses(AtomicInteger ports, int count){ List<Address> addresses = new ArrayList<Address>(count); for (int i = 0; i < count; i++) { addresses.add(createAddress("127.0.0.1", ports.incrementAndGet())); } return addresses; } protected static Address createAddress(String host, int port) { try { return new Address("127.0.0.1", PORTS.incrementAndGet()); } catch (UnknownHostException e) { e.printStackTrace(); } return null; } After Before textual similarity = 100%
43. Reliable oracle • We used the “Why We Refactor” dataset (538 commits from 185 open source projects) • We executed both available tools (RefactoringMiner and RefDiff) • Converted the output of the tools to the same format • We manually validated all refactoring instances (4,108 instances) • The validation process was labor-intensive and involved 3 validators for a period of 3 months (i.e., 9 person-months) • To compute recall, we considered the union of the true positives reported by both tools as the ground truth. Matin Mansouri Laleh Eshkevari Davood Mazinanian
46. ICSE’18 retrospective • Our solution was still far from perfect • Time pressure • It was urgent to establish RefactoringMiner with a publication • ICSE deadline: August 25, 2017 • Sophia’s birth: August 15, 2017 • Despite working over 1 year on this paper, there was still space for improvement • The entire team graduated after this work
47. Wisdom TSE’20 version 2.0
48. • Added more replacement types • 20 new sub-method level refactoring types detected based on AST node replacements • Nested refactoring detection • Refactoring inference • Improved the matching of method calls to method declarations with argument type inference
49. Even more reliable oracle • We executed all available tools • RefactoringMiner 1.0 and 2.0 • RefDiff 0.1.1, 1.0, 2.0 • GumTreeDiff 2.1.2 • Converted the output of the tools to the same format • We validated 5,830 new unique refactoring instances, out of which 4,038 were true positives and 1,792 were false positives. • 7,226 true positives in total for 40 different refactoring types (72% of true instances are detected by two or more tools) Ameya Ketkar
53. Emperor ToSEM’24 version 3.0
54. I want to study source code diff algorithms Let’s make RefactoringMiner an AST diff tool Pouria Alikhanifard
56. Eugene Myers Language independent Super-fast and scalable Line-level granularity Does not handle well moves within a file Does not handle formatting changes
57. Encyclopedia 3000: The picture shows humans, known as “software engineers” using source code diff tools to perform a “code review” in the year 2026.
59. But even AST diff is NOT perfect
60. Limitations of previous approaches 1. No support for multi-mappings 2. Semantic ignorance (matching AST nodes based on AST type, regardless of their semantic role in the program) 3. Refactoring un-awareness 4. Tree matching with the goal to minimize size of edit-script 5. No support for moved code between files 6. Poor evaluation standards based on edit-script size (no benchmarks)
61. The first AST Diff benchmark • Process (6 months): 1. Run all ASTDiff tools (GumTree 3.0, GumTree 2.1, IJM, MTDiff, RefMiner) 2. Manually validate the diffs 3. Construct the “perfect” diff • Datasets: • 800 bug fixing commits from Defects4J • 187 refactoring commits from Refactoring Oracle Pouria Alikhanifard
62. RefactoringMiner Statement mappings Program declaration mappings Import declaration mappings Refactoring mappings based on mechanics Tree Matcher Tree Matcher Overwrite conflicting mappings AST mappings AST mappings Final AST mappings Edit script version1 version2 AST diff generation
63. Tool Refactoring Miner iASTMapper IJM GumTree Spoon GumTree 4.0 Simple GumTree 3.0 Greedy GumTree 2.1 MTDiff Precision 99.6 93.2 89.2 84.2 85.7 82.9 81.6 61.7 Recall 99.3 85.9 78.5 81.2 77.0 74.7 74.1 81.0 F-score 99.4 89.4 83.5 82.7 81.1 78.6 77.7 70.0 Accuracy Tree Matching Statement Mapping
64. Current status • Multi-language support • 100+ supported refactoring types • 19K validated true positives in oracles • Precision: 99% • Recall: 97% • Most tested and reliable version (200K commits without exception) • Independent studies confirm it has the best precision
65. Lesson #5 What does it take to make a reliable & usable tool? üMinimum 5 years of research and development üExtensive testing (2K tests) üDetailed documentation (README with API code snippets) üSupporting users (700+ issues resolved) üStable project leader üGreat team
67. Cross-language refactoring detection
69. Inline Java Variable to Kotlin
70. Multi-language development is common • Machine learning: C, C++, Python, Java • Web apps: frontend vs. backend language • Mobile apps: Android with Java and Kotlin, React with ts and tsx • Unity: C++ for engine internals + C# for game scripting
71. Intelligent Code Review Assistants
72. Oliver Kopp JabRef
73. Variable url is extracted to construct the pagination url based on the value of the pageNumber parameter This line has been added to the extracted method to return whether the coursesContainer includes more pages with courses. This newly added while loop calls the extracted method by incrementing the pageNumber argument by 1 in each iteration and terminates when the extracted method returns false (i.e., there are no more pages left).
74. Impact • RefactoringMiner 0.0 (CASCON 2013) • RefactoringMiner 0.1 (FSE 2016) • RefactoringMiner 1.0 (ICSE 2018) • RefactoringMiner 2.0 (TSE 2020) • RefactoringMiner 3.0 (TOSEM 2025) citations 110 403 397 302 36 1248