unique? Easy access to industrial problems and data Easy access to engineers Near term impact Collaborations with Microsoft researchers Collaborations with external researchers
submitted a bug report. Will it be fixed? What metrics are the best predictors of failures? Should I be writing unit tests in my software project? Is strong code ownership good or bad for software quality? Does Distributed/Global software development affect quality? How can I tell if a piece of software will have vulnerabilities? Does Test Driven Development (TDD) produce better code in shorter time? If I increase test coverage, will that actually increase software quality? What is the data quality level used in empirical studies and how much does it actually matter? Are there any metrics that are indicators of failures in both Open Source and Commercial domains?
Budi, Bora Caglayan, Gul Calikli, Joshua Charles Campbell, Jacek Czerwonka, Kostadin Damevski, Madeline Diep, Robert Dyer, Linda Esker, Davide Falessi, Xavier Franch, Thomas Fritz, Nikolas Galanis, Marco Aurélio Gerosa, Ruediger Glott, Michael W. Godfrey, Alessandra Gorla, Georgios Gousios, Florian Groß, Randy Hackbarth, Abram Hindle, Reid Holmes, Lingxiao Jiang, Ron S. Kenett, Ekrem Kocaguneli, Oleksii Kononenko, Kostas Kontogiannis, Konstantin Kuznetsov, Lucas Layman, Christian Lindig, David Lo, Fabio Mancinelli, Serge Mankovskii, Shahar Maoz, Daniel Méndez Fernández, Andrew Meneely, Audris Mockus, Murtuza Mukadam, Brendan Murphy, Emerson Murphy-Hill, John Mylopoulos, Anil R. Nair, Maleknaz Nayebi, Hoan Nguyen, Tien Nguyen, Gustavo Ansaldi Oliva, John Palframan, Hridesh Rajan, Peter C. Rigby, Guenther Ruhe, Michele Shaw, David Shepherd, Forrest Shull, Will Snipes, Diomidis Spinellis, Eleni Stroulia, Angelo Susi, Lin Tan, Ilaria Tavecchia, Ayse Tosun Misirli, Mohsen Vakilian, Stefan Wagner, Shaowei Wang, David Weiss, Laurie Williams, Hamzeh Zawawy, and Andreas Zeller
T. Barik, E.T. Barr, O. Baysal, A. Bener, G. Bergersen, C. Bird, D. Budgen, B. Caglayan, T. Carnahan, J. Czerwonka, P. Devanbu, M. Di Penta, S. Diehl, T. Dyba, T. Fritz, M.W. Godfrey, G. Gousios, P. Guo, K. Herzig, A. Hindle, R. Holmes, Zhitao Hou, J. Huang, Andrew J. Ko, N. Juristo, S. Just, M. Kim, E. Kocaguneli, K. Kuutti, Qingwei Lin, Jian- Guang Lou, N. Medvidovic, A. Meneely, T. Menzies, L.L. Minku, A. Mockus, J. Munch, G.C. Murphy, B. Murphy, E. Murphy-Hill, M. Nagappan, M. Nayebi, M. Oivo, A. Orso, T. Ostrand, F. Peters, D. Posnett, L. Prechelt, Venkatesh-Prasad Ranganath, B. Ray, R. Robbes, P. Rotella, G. Ruhe, P. Runeson, B. Russo, M. Shepperd, E. Shihab, D.I.K. Sjøberg, D. Spinellis, M.-A. Storey, C. Theisen, A. Tosun, B. Turhan, H. Valdivia- Garcia, S. Vegas, S. Wagner, E. Weyuker, J. Whitehead, L. Williams, Tao Xie, A. Zeller, Dongmei Zhang, Hongyu Zhang, Haidong Zhang, T. Zimmermann http://tiny.cc/superdog
I'm looking to the engineering teams to build the experiences our customers love. […] In order to deliver the experiences our customers need for the mobile-first and cloud- first world, we will modernize our engineering processes to be customer-obsessed, data- driven, speed-oriented and quality-focused. http://news.microsoft.com/ceo/bold-ambition/index.html
Applied Science resources that will focus on measurable outcomes for our products and predictive analysis of market trends, which will allow us to innovate more effectively. http://news.microsoft.com/ceo/bold-ambition/index.html
Begel: The Emerging Role of Data Scientists on Software Development Teams. ICSE 2016. Data Scientists in Software Teams: State of the Art and Challenges. In preparation. Miryung Kim Robert DeLine Andrew Begel
women and 11 men from eight different Microsoft organizations Snowball sampling – data-driven engineering meet- ups and technical community meetings – word of mouth Coding with Atlas.TI Clustering of participants Survey 793 responses – full-time data scientists – employees with interest in data science Questions about – demographics – skills – self-perception – working styles – time spent – challenges and best practices
interdisciplinary backgrounds Many have higher education degrees Survey: 41% have master’s degrees, and 22% have PhDs Strong passion for data I’ve always been a data kind of guy. I love playing with data. I’m very focused on how you can organize and make sense of data and being able to find patterns. I love patterns. [P14] “Machine learning hackers”. Need to know stats My people have to know statistics. They need to be able to answer sample size questions, design experiment questions, know standard deviations, p- value, confidence intervals, etc.
to working style It has never been, in my four years, that somebody came and said, “Can you answer this question?” I mostly sit around thinking, “How can I be helpful?” Probably that part of your PhD is you are figuring out what is the most important questions. [P13] I have a PhD in experimental physics, so pretty much, I am used to designing experiments. [P6] Doing data science is kind of like doing research. It looks like a good problem and looks like a good idea. You think you may have an approach, but then maybe you end up with a dead end. [P5]
platform; Telemetry injection; Experimentation platform Analysis Data merging and cleaning; Sampling; Data shaping including selecting and creating features; Defining sensible metrics; Building predictive models; Defining ground truths; Hypothesis testing Dissemination Operationalizing predictive models; Defining actions and triggers; Translating insights and models to business values
2013 Generalists Polymath “describes data scientists who ‘do it all’ ” Data Creatives “data scientists [who] can often tackle the entire soup-to-nuts analytics process on their own” Specialists Data Preparer Data Shaper Data Analyzer / Insight Provider “main task is to generate insights and to support and guide their managers in decision making” Platform Builder “build shared data platforms used across several product teams” Data Developer “people focused on the technical problem of managing data” Modelling Specialist “data scientists who act as expert consultants and build predictive models” Data Researcher people with “deep academic training in the use of data to understand complex processes” Manager Data Evangelist / Team Leader “senior data scientists who run their own data science teams act as data science ‘evangelists’ ” Data Businesspeople people who “are most focused on the organization and how data projects yield profit” Insight Actor Moonlighter 50% Moonlighter 20% Moonlighter
Questions for Researchers Posted Aug 22, 2012 by Greg Wilson I gave the opening talk at MSR Vision 2020 in Kingston on Monday (slides), and in the wake of that, an experienced developers at Mozilla sent me a list of ten questions he'd really like empirical software engineering researchers to answer. They're interesting in their own right, but I think they also reveal a lot about what practitioners want from researchers in general; comments would be very welcome. 1. Vi vs. Emacs vs. graphical editors/IDEs: which makes me more productive? 2. Should language developers spend their time on tools, syntax, library, or something else (like speed)? What makes the most difference to their users? 3. Do unit tests save more time in debugging than they take to write/run/keep updated?
Asia, Europe, North America, Other Number of Full-Time Employees Current Role: Manager, Individual Contributor Years as Manager Has Management Experience: yes, no. Years at Microsoft
Worthwhile How do users typically use my application? 80.0% 99.2% What parts of a software product are most used and/or loved by customers? 72.0% 98.5% How effective are the quality gates we run at checkin? 62.4% 96.6% How can we improve collaboration and sharing between teams? 54.5% 96.4% What are the best key performance indicators (KPIs) for monitoring services? 53.2% 93.6% What is the impact of a code change or requirements change to the project and its tests? 52.1% 94.0% What is the impact of tools on productivity? 50.5% 97.2% How do I avoid reinventing the wheel by sharing and/or searching for code? 50.0% 90.9% What are the common patterns of execution in my application? 48.7% 96.6% How well does test coverage correspond to actual code usage by our customers? 48.7% 92.0%
individual measures correlate with employee productivity (e.g. employee age, tenure, engineering skills, education, promotion velocity, IQ)? 25.5% Which coding measures correlate with employee productivity (e.g. lines of code, time it takes to build software, particular tool set, pair programming, number of hours of coding per day, programming language)? 22.0% What metrics can use used to compare employees? 21.3% How can we measure the productivity of a Microsoft employee? 20.9% Is the number of bugs a good measure of developer effectiveness? 17.2% Can I generate 100% test coverage? 14.4% Who should be in charge of creating and maintaining a consistent company-wide software process and tool chain? 12.3% What are the benefits of a consistent, company-wide software process and tool chain? 10.4% When are code comments worth the effort to write them? 9.6% How much time and money does it cost to add customer input into your design? 8.3%
You have communication going back and forth where you will find what you’re actually looking for, what is anomalous and what is not anomalous in the set of data that they looked at.
[the model] and they understood all the results and they were very excited about it. Then, there’s a phase that comes in where the actual model has to go into production. … You really need to have somebody who is confident enough to take this from a dev side of things.
of convincing, if you just present all these numbers like precision and recall factors… that is important from the knowledge sharing model transfer perspective. But if you are out there to sell your model or ideas, this will not work because the people who will be in the decision- making seat will not be the ones doing the model transfer. So, for those people, what we did is cost benefit analysis where we showed how our model was adding the new revenue on top of what they already had.
team (a) Is it a priority for the organization (b) is it actionable, if I get an answer to this, is this something someone can do something with? and, (c), are you as the feature team — if you're coming to me or if I'm going to you, telling you this is a good opportunity — are you committing resources to deliver a change? If those things are not true, then it's not worth us talking anymore.
You begin to find out, you begin to ask questions, you being to see things. And so you need that interaction with the people that own the code, if you will, or the feature, to be able to learn together as you go and refine your questions and refine your answers to get to the ultimate insights that you need.
super smart data scientist, their understanding and presentation of their findings is usually way over the head of the managers…so my guidance to [data scientists], is dumb everything down to seventh- grade level, right? And whether you're writing or you're presenting charts, you know, keep it simple.
• number of modification requests and added lines of code per year • number of tasks per month • number of function points per month • number of source lines of code per hour • number of lines of code per person month of coding effort • amount of work completed per reported hour of effort for each technology • ratio of produced logical code lines and spent effort • average number of logical source statements output per month over the product development cycle • total equivalent lines of code per person-month • resolution time defined as the time, in days, it took to resolve a particular modification request • number of editing events to number of selection and navigation events needed to find where to edit code
quality, constraints Process: maturity, completeness of design Development environment: tools, modern development practices, programming language, documentation Wagner, S. and Ruhe, M. (2008) A Systematic Review of Productivity Factors in Software Development. In Proc. 2nd International Workshop on Software Productivity Analysis and Cost Estimation (SPACE 2008).
fairness, respect, credibility Team culture: team cohesion, turnover Capabilities and experiences: programmer capability, experience with application domain, platform, language, and tool, Work environment: fragmentation, separation Project: average team size Wagner, S. and Ruhe, M. (2008) A Systematic Review of Productivity Factors in Software Development. In Proc. 2nd International Workshop on Software Productivity Analysis and Cost Estimation (SPACE 2008).
Murphy, Thomas Zimmermann, Thomas Fritz. The Work Life of Developers: Activities, Switches and Perceived Productivity. IEEE TSE André N. Meyer, Thomas Fritz, Gail C. Murphy, Thomas Zimmermann: Software developers' perceptions of productivity. SIGSOFT FSE 2014 Andre Meyer Thomas Fritz Gail Murphy
background, perceptions, assessing, measuring and improving productivity Recruitment through emails and posts in online forums 379 participants 194 within Microsoft, 185 public survey 9.2 years of professional experience on average
get through my list of tasks for the day. I have at most one meeting. I have no code reviews to complete. I have a productive workday when I have autonomy on what I do and don't have to wait for response from other side of earth. I am focused on one work item and have all the inputs I need to close on that work item. I write more than 10 lines of code I wrote at least 1 line of code I complete the goals I set for myself that day. I am in a good mood and slept well :) Get what I have planned done When the weather is dry and I can work at a silent and well ventilated place. I get enough rest and can take nap when I feel tired. I am not randomized too much by meetings and interruptions from others
feel productive when they make progress on tasks with few context switches. Complete tasks or goals 53.2% Have no/few interruptions and distractions 50.4% Have no meetings 21.9% Have clear goals 19.9% Plan my workday 17.2%
19% Unplanned work 18% Productive activities Coding 72% Meetings 17% Planning 7% Based on a survey of 379 professional developers from many different companies
helpful to assess your productivity? Number of API methods I learned Code elements I changed Code elements I changed for the first time Code reviews I’ve contributed to Code reviews I’ve signed off Commits I made Emails I wrote Lines of code I changed per day Meetings I attended Test cases I wrote Test cases I wrote that subsequently failed Work items I closed Work items I created Work items I created that were fixed Time spent Writing code Reviewing code Meetings Browsing for work Browsing for personal matters Per code project or package Per work item Sign off on code reviews (average) Respond to email (average)
helpful to assess your productivity? “The number of commits I made.” Mean: 2.84, Standard Deviation: ±1.38 “The time I spent in meetings.” Mean: 3.31, Standard Deviation: ±1.34
20 users of the retrospection tool Developers only spend about half their time active on their computer. For every work hour, developers have an average of 2.5 short breaks, totaling 10.5 minutes of unplanned time away from their computer. Developers spend about a fourth of their time on coding related activities and another fourth of their time on collaborative activities. The range and time spent on activities varies greatly depending on the individual and company. Developers’ work is highly fragmented, spending very short amounts of time (0.3 to 2 minutes) in one activity before switching to another one.
the Number of Mouse Clicks (7× each) have more often positive influence. No single factor provides explanatory power across all participants. The percent of activities Email (5×), Planned Meeting (6×), Work Unrelated Browsing (5×), and Idle (6×) have more often negative influence.
in the morning than in the afternoon. I feel more productive in the afternoon than in the morning. I feel productive on a day with little to no meetings. I feel productive when I do code reviews. I feel productive when I write code. I feel productive when I listen to music. I feel productive when I take a quick break, e.g., on Facebook. I feel productive when I help my co-workers. I feel productive when I read fewer emails than usual. … (nine additional statements on productivity)
of productivity in a survey of 413 developers at Microsoft. Social developer • Feels productive when helping coworkers, collaborating, and doing code review Lone developer • Avoids disruptions such as noise, meetings, and code reviews • Feels productive when they are not interrupted Focused developer • Feels most productive when working on a single task Balanced developer • Less affected by disruptions • Feels unproductive when unfamiliar with tasks or when the tasks are unclear Leading developer • More comfortable with meetings • Places less importance on coding activities than other developers Goal-oriented developer •Feels productive when completing or making progress on tasks •Feels less productive when multi-tasking, without goals, or stuck •More positive about meetings
I feel productive when I write code (5 of the 6 types) I feel productive on a day with little to no meetings (4 of the 6 types) I feel productive when I am happy (4 of the 6 types) I feel productive when I have fewer interruptions (4 of the 6 types) The time I spent coding (5 of the 6) The longest period focused on a task without an interruption LOWER SCORES I feel productive when I send more emails than usual (5 of the 6 types) I feel I had a productive work day when my email inbox is emptier in the evening than in the morning (4 of the 6 types) I feel productive when I visit social networks or news websites to do a quick break (4 of the 6 types) If I have many program windows open on my screen, it decreases my perceived productivity I feel productive on a particular day of the week, e.g., on Wednesdays (5 of the 6) I feel more productive in the morning than in the afternoon (5 of the 6 types) I feel less productive after lunch compared to the rest of the day (4 of the 6 types)
Autonomist Acumen Lilo, the Continuous Learner Isabelle, the Investigator Cameron, the Communicator Iman, the Interactive Ava, the Advisor Ciara, the Team Coder Denae Ford, Thomas Zimmermann, Christian Bird and Nachiappan Nagappan Characterizing Software Engineering Work with Personas Based on Knowledge Worker Actions In International Symposium on Empirical Software Engineering and Measurement (ESEM 2017)
tomzimmermann
hargon24
agiats
hponka
fta98
a1da4
sansan_randd
eumesy
tam1110
hf149
kampersanda
chemical_tree
ryotoitoi
dougneiner
sonatard
hatefulcrawdad
zakiyama
holman
kastner
lauravandoore
chriscoyier
jeffersonlam
jakevdp
tenderlove
