Systematic Literature Tools: Are we there yet? - ZEUS 2021

Presentation for paper, abstract:

The number of publications is steadily growing. systematic literature reviews (SLRs) are one answer to this issue. A variety of tools exists designed to support the review process. This paper summarizes requirements for adequate tooling support and shows that existing tools do not meet all of them. We further investigate whether reference management tools can be used in conjunction with existing SLR tools to address the current gaps in supporting SLRs. For that we evaluate three reference management tools, JabRef, BibSonomy, and Zotero, against currently unaddressed requirements and outline the next steps.

Full paper available via https://dblp.org/rec/conf/zeus/VoigtKW21.html

Transcript

1. Systematic Literature Tools: Are we there yet? Dominik Voigt, Oliver Kopp, Karoline Wild JabRef e.V. and University of Stuttgart 1 Presented at ZEUS 2021 – full paper at https://dblp.org/rec/conf/zeus/VoigtKW21.html
2. Agenda • Why systematic literature reviews? • What are systematic literature reviews? • Criteria for tooling • Evaluation of tooling • Next steps 2
3. Why Systematic Literature Reviews? • Increased quality of your contribution by a better literature evaluation • Quality of the set of known and presented related work • Easier identifying relevant related work • Comprehensible and transparent collection of related work 3
4. How does a Systematic Literature Review work? Plan Review Conduct Review Document Review Develop Review Protocol Search Studies Select relevant Studies Write the SLR Report Synthesise the Results Extract relevant Data Kitchenham, B.A.: Guidelines for performing systematic literature reviews in software engineering. Tech. rep., Keele University (2007) 4
5. SLR Search Process by Kitchenham & Brereton (2013) 5 A systematic review of systematic review process research in software engineering Information and Software Technology 55(12), 2049-2075, 2013
6. Search and Select Steps (Variant) 6 Based on: “A Systematic Mapping Study on Engineering Function-as-a-Service Platforms and Tools”
7. What are the main requirements on SLR tools? • Search multiple databases with a standardized query (R1, 48 times) • Removing duplicate studies (R2, 13 times) • Provide filtering for studies (R3, 7 times) • Merging new results into the existing database (R4, 6 times) • Synonym recommendation for search strings (R5, 6 times) • Repository for studies (R6, 6 times) • Standardized export formats (R7, 6 times) • Automatic download of full-text papers (R8, 6 times) 7 • Multiple providers: ACM, arXiv, dblp, IEEE, Springer, … • Mass access • Different search syntax Al-Zubidy, A., Carver, J.C.: Identification and prioritization of SLR search tool requirements: an SLR and a survey. Empirical Software Engineering 24(1), 139–169 (2018) Carver, J.C., Hassler, E., Hernandes, E., Kraft, N.A.: Identifying barriers to the systematic literature review process. In: International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE (2013)
8. What do existing tools do? • List of all SLR Tools at http://systematicreviewtools.com/ • All tools have lacking support of search and selection steps for Computer Science 8
9. What’s the state of reference managers? 9 • Claim: Researchers use reference managers, not SLR tools • Claim: Open source tools are easier to extend • • Candidates: (hosted on bitbucket)
10. What’s the state of reference managers? 10 Search: Multiple sources + standardized Removing duplicates Filtering Merging Synonym recommend- dation Repository Standardized export Full text download BibSonomy no yes no yes no yes yes yes JabRef partially yes yes yes no yes yes yes Zotero no yes yes yes no yes yes yes
11. Next Steps • JabRef has integrated search • JabRef’s search is available via an API • Enhance JabRef to a generic query syntax • Map the syntax to the syntax to each paper provider (arXiv, IEEE, …) • Integrate JabRef with existing SLR tools 11
12. Conclusion and Outlook 12