DevGPT: Studying Developer-ChatGPT Conversations

The slides for the 2024 MSR challenge "DevGPT: Studying Developer-ChatGPT Conversations"

Transcript

1. 2024 MSR Challenge (AI) DevGPT: Studying Developer-ChatGPT Conversations Tao Xiao Nara Institute of Science and Technology Christoph Treude University of Melbourne Hideaki Hata Shinshu University Kenichi Matsumoto Nara Institute of Science and Technology
2. 01 What is DevGPT? Table of contents 02 Potential research questions 03 Dataset structure 04 Submission instructions 05 Q&A
3. What is DevGPT? 01 (AI)
4. (DevGPT) = Developer-ChatGPT Conversations
5. DevGPT A curated dataset which encompasses 17,913 prompts and ChatGPT’s responses including 11,751 code snippets, coupled with the corresponding software development artifacts—ranging from source code, commits, issues, pull requests, to discussions and Hacker News threads—to enable the analysis of the context and implications of these developer interactions with ChatGPT.
6. Goal of the Challenge Enable a comprehensive analysis of the context and implications of developer interactions with ChatGPT.
7. Example of ChatGPT Sharing
8. Why DevGPT? Understand developer usage of ChatGPT in software development. Analyze developer questions and interaction dynamics. Evaluate the impact on software development artifacts. Gain insights into AI model integration in development. Inform future AI model strategies for dev tools.
9. Overview of DevGPT
10. Potential research questions 02 (AI)
11. (a) What types of issues (bugs, feature requests, theoretical questions, etc.) do developers most commonly present to ChatGPT? (b) Can we identify patterns in the prompts developers use when interacting with ChatGPT, and do these patterns correlate with the success of issue resolution? (c) What is the typical structure of conversations between developers and ChatGPT? How many turns does it take on average to reach a conclusion? Questions to be answered
12. (d) In instances where developers have incorporated the code provided by ChatGPT into their projects, to what extent do they modify this code prior to use, and what are the common types of modifications made? (e) How does the code generated by ChatGPT for a given query compare to code that could be found for the same query on the internet (e.g., on Stack Overflow)? (f) What types of quality issues (for example, as identified by linters) are common in the code generated by Chat- GPT? Questions to be answered (Cont.)
13. (g) How accurately can we predict the length of a conver- sation with ChatGPT based on the initial prompt and context provided? (h) Can we reliably predict whether a developer’s issue will be resolved based on the initial conversation with ChatGPT? (i) If developers were to rerun their prompts with ChatGPT now and/or with different settings, would they obtain the same results? Questions to be answered (Cont.)
14. Dataset structure 03 (AI)
15. (a) Snapshot (b) File format (c) CSV File organizations {obtained_time}_{source}_sharings.json All shared ChatGPT links 9
16. Structure (a) JSON (b) Type attribute (c) General attributes (d) ChatgptSharing [pull request, commit, hacker news, issue, discussion, code file] [URL, Author, RepoName, RepoLanguage, …] [{URL, Status, Conversations, Mention, HTMLContent … } …]
17. Structure (Cont.) (e) Conversations (f) Mention [MentionedURL, MentionedProperty, MentionedAuthor, MentionedText, …] [{Prompt, Answer, ListOfCode}, …]
18. JSON files (a) Type attributes (b) General attributes (c) ChatgptSharing (e) Mention (d) Conversations
19. CSV file
20. Dataset snapshot_20231012 GitHub repository
21. Submission instructions 04 (AI)
22. Submission instructions (+) Specify which snapshot/version of the DevGPT dataset was utilized (+) ACM Primary Article Template (+) Latex code \documentclass[sigconf,review,anonymous]{acmart} \acmConference[MSR 2024]{MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories}{April 15–16, 2024}{Lisbon, Portugal}
23. Submission instructions (Cont.) (+) 4 pages plus 1 additional page of references) (+) Double-anonymous review (+) https://msr2024-challenge.hotcrp.com/ (+) Cite @inproceedings{ title={DevGPT: Studying Developer-ChatGPT Conversations}, author={Xiao, Tao and Treude, Christoph and Hata, Hideaki and Matsumoto, Kenichi}, year={2024}, booktitle={Proceedings of the International Conference on Mining Software Repositories (MSR 2024)}, }
24. Q&A 05 (AI)
25. CREDITS: This presentation template was created by Slidesgo and includes icons by Flaticon, infographics & images by Freepik and content by Eliana Delacour Thanks! Any questions? Create new issues or discussions: https://github.com/NAIST-SE/DevGPT Please, keep this slide as attribution