Software Analytics for Digital Games

Keynote presented at the Fourth International Workshop on Games and Software Engineering (GAS 2015).

http://2015.gasworkshop.org/

Transcript

1. © Microsoft Corporation Software Analytics for Digital Games Thomas Zimmermann, Microsoft Research, USA Joint work with Nachi Nagappan and many others. http://aka.ms/esegroup
2. © Microsoft Corporation
3. © Microsoft Corporation Use of data, analysis, and systematic reasoning to [inform and] make decisions 3
4. © Microsoft Corporation history of software analytics Tim Menzies, Thomas Zimmermann: Software Analytics: So What? IEEE Software 30(4): 31-37 (2013)
5. © Microsoft Corporation
6. © Microsoft Corporation trinity of software analytics Dongmei Zhang, Shi Han, Yingnong Dang, Jian-Guang Lou, Haidong Zhang, Tao Xie: Software Analytics in Practice. IEEE Software 30(5): 30-37, September/October 2013. MSR Asia Software Analytics group: http://research.microsoft.com/en-us/groups/sa/
7. © Microsoft Corporation Tom’s three Cupcakes of Software Analytics diversity people sharing
8. © Microsoft Corporation diversity
9. © Microsoft Corporation The Stakeholders The Tools The Questions
10. © Microsoft Corporation http://aka.ms/145Questions Andrew Begel, Thomas Zimmermann. Analyze This! 145 Questions for Data Scientists in Software Engineering. ICSE 2014
11. © Microsoft Corporation Microsoft’s Top 10 Questions Essential Essential + Worthwhile How do users typically use my application? 80.0% 99.2% What parts of a software product are most used and/or loved by customers? 72.0% 98.5% How effective are the quality gates we run at checkin? 62.4% 96.6% How can we improve collaboration and sharing between teams? 54.5% 96.4% What are the best key performance indicators (KPIs) for monitoring services? 53.2% 93.6% What is the impact of a code change or requirements change to the project and its tests? 52.1% 94.0% What is the impact of tools on productivity? 50.5% 97.2% How do I avoid reinventing the wheel by sharing and/or searching for code? 50.0% 90.9% What are the common patterns of execution in my application? 48.7% 96.6% How well does test coverage correspond to actual code usage by our customers? 48.7% 92.0%
12. © Microsoft Corporation people
13. © Microsoft Corporation The Decider The Brain The Innovator Photo of MSA 2010 by Daniel M German (dmg@turingmachine.org) The Researcher
14. © Microsoft Corporation sharing
15. © Microsoft Corporation Sharing Insights Sharing Methods Sharing Models Sharing Data
16. © Microsoft Corporation Games Research in the ESE Group
17. © Microsoft Corporation Xbox Live Influence of games and achievements on (paid) Xbox live memberships Influence of friends on titles played Characterizing players with Xbox Live data Gameplay Game play dynamics (Project Gotham Racing 4) => ESEM 2012 Impact of social behavior on retention (Beta of a AAA title) Influence of gameplay on skill (Halo Reach) => CHI 2013 Assists in a car racing game (Forza 4) => FDG 2014 How to create a successful initial session in games => CHI Play 2014 Engineering Differences between game and traditional software development => ICSE 2014 Lessons learned from game development (ongoing) Mining software repositories from games (ongoing) Exploratory Personalization with Avatars in Xbox Geographic influence, temporal influence, and structural influence Thanks to our collaborators in Xbox and Microsoft Games Studios. Thanks to interns Ken Hullett, Sauvik Das, Jeff Huang, Gifford Cheung, Thomas Debeauvais, Erik Harpstead and visiting researchers Tim Menzies and Emerson Murphy-Hill.
18. © Microsoft Corporation Skill in Halo Reach Jeff Huang, Thomas Zimmermann, Nachiappan Nagappan, Charles Harrison, Bruce C. Phillips: Mastering the art of war: how patterns of gameplay influence skill in Halo. CHI 2013: 695-704
19. © Microsoft Corporation
20. How do patterns of play affect players’ skill in Halo Reach? 5 Skill and Other Titles 6 Skill Changes and Retention 7 Mastery and Demographics 8 Predicting Skill 2 Play Intensity 3 Skill after Breaks 4 Skill before Breaks 1 General Statistics
21. The Cohort of Players The mean skill value µ for each player after each Team Slayer match µ ranges between 0 and 10, although 50% fall between 2.5 and 3.5 Initially µ = 3 for each player, stabilizing after a couple dozen matches TrueSkill in Team Slayer We looked at the cohort of players who started in the release week with complete set of gameplay for those players up to 7 months later (over 3 million players) 70 Person Survey about Player Experience
22. 2 Play Intensity Telegraph operators gradually increase typing speed over time
23. 2.1 2.3 2.5 2.7 2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 2 Play Intensity Median skill typically increases slowly over time Skill
24. 2 Play Intensity (Games per Week) 2.1 2.3 2.5 2.7 2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 0 - 2 games / week [N=59164] 2 - 4 games / week [N=101448] 4 - 8 games / week [N=226161] 8 - 16 games / week [N=363832] 16 - 32 games / week [N=319579] 32 - 64 games / week [N=420258] 64 - 128 games / week [N=415793] 128 - 256 games / week [N=245725] Median skill typically increases slowly over time Skill
25. 2 Play Intensity (Games per Week) 2.1 2.3 2.5 2.7 2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 0 - 2 games / week [N=59164] 2 - 4 games / week [N=101448] 4 - 8 games / week [N=226161] 8 - 16 games / week [N=363832] 16 - 32 games / week [N=319579] 32 - 64 games / week [N=420258] 64 - 128 games / week [N=415793] 128 - 256 games / week [N=245725] Median skill typically increases slowly over time Skill
26. 2 Play Intensity (Games per Week) 2.1 2.3 2.5 2.7 2.9 3.1 0 10 20 30 40 50 60 70 80 90 100 mu Games Played So Far 0 - 2 games / week [N=59164] 2 - 4 games / week [N=101448] 4 - 8 games / week [N=226161] 8 - 16 games / week [N=363832] 16 - 32 games / week [N=319579] 32 - 64 games / week [N=420258] 64 - 128 games / week [N=415793] 128 - 256 games / week [N=245725] But players who play more overall eventually surpass those who play 4–8 games per week (not shown in chart) Players who play 4–8 games per week do best Median skill typically increases slowly over time Skill
27. 3 Change in Skill Following a Break “In the most drastic scenario, you can lose up to 80 percent of your fitness level in as few as two weeks [of taking a break]…”
28. -0.03 -0.02 -0.01 0 0.01 0.02 0.03 0 5 10 15 20 25 30 35 40 45 50 Δmu Days of Break Next Game 2 Games Later 3 Games Later 4 Games Later 5 games later 10 games later 3 Change in Skill Following a Break Median skill slightly increases after each game played without breaks Longer breaks correlate with larger skill drops, but not linearly On average, it takes 8–10 games to regain skill lost after 30 day breaks Breaks of 1–2 days correlate in tiny drops in skill Change in Skill
29. © Microsoft Corporation Driving skill in Forza Motorsports 4 Thomas Debeauvais, Thomas Zimmermann, Nachiappan Nagappan, Kevin Carter, Ryan Cooper, Dan Greenawalt, Tyson Solberg: An Empirical Study of Driving Skill in Forza Motorsports 4. FDG 2014
30. © Microsoft Corporation Driving skill in Forza Motorsports 4 5% of player base, sampled randomly 200k players who played 25M races Assist usage Assist transitions
31. © Microsoft Corporation
32. © Microsoft Corporation Approaching a turn in Forza 4 – in EASY mode –
33. © Microsoft Corporation Approaching a turn in Forza 4 – in EXPERT mode –
34. © Microsoft Corporation The assist bundles in Forza 4 Easy Medium Hard Advanced Expert Stability prevents the car from spinning when cornering too fast ON OFF Traction prevents the car from spinning when accelerating ON OFF Braking supports the player when he/she brakes or should brake Assisted w/ ABS ABS OFF Shifting helps the player in passing gears Automatic w/o clutch Manual w/o clutch Manual w/ clutch Line overlays the optimal trajectory to follow on the track Full Brake OFF Damage determines how much the performance of the car can change during the race Cosmetic Limited Simulation
35. © Microsoft Corporation number of races Assist usage over number of races career mode online multiplayer number of races
36. © Microsoft Corporation Assist transitions time The player disables the assist
37. © Microsoft Corporation Assist transitions time The player disables the assist
38. © Microsoft Corporation 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% success failure yoyo never disabled Assist transitions
39. © Microsoft Corporation Factors that contribute to the success of disabling an assist Factor More likely to keep the assist disabled … Significant for … Number of races Players who disable an assist early All assists Races per day Players who race fewer games a day All assists Rear-wheel drive (race before) Players who drove a car with rear-wheel drive All assists Car Performance Index (race before) Players who drove a car with lower PI All assists Position (race before) Players who finished first All assists but Traction and Clutch Career mode (race before) Players who did not play career mode Autobrake, ABS, Autoshift, Full line, Brake line
40. © Microsoft Corporation The First Hour Experience Gifford Cheung, Thomas Zimmermann, Nachiappan Nagappan: The First Hour Experience: How the Initial Play can Engage (or Lose) New Players. CHI Play 2014
41. © Microsoft Corporation the “initial experience” the “first time user experience” “tutorial time” the “5-5-5” (five seconds, minutes, hours) “the first 10 seconds, the first minute, the first ten minutes, the first hour” “setting the hook” also known as
42. © Microsoft Corporation “If your game is abandoned in the first hour that’s a pretty low moment. It’s a complete rejection of the game play in all ways, you know, game play, narrative, style. … It’s a pretty catastrophic failure.” Designer C a Critical Hour
43. © Microsoft Corporation The Data 212 Amazon reviews of Xbox 360 games Searching for keywords such as “first”, “hour”, “minute”, “start”
44. © Microsoft Corporation The Data 212 Amazon reviews of Xbox 360 games Searching for keywords such as “first”, “hour”, “minute”, “start” 35 First hour reviews
45. © Microsoft Corporation The Data 212 Amazon reviews of Xbox 360 games Searching for keywords such as “first”, “hour”, “minute”, “start” 35 First hour reviews 6 Interviews with Game Industry Professionals (designer, tester, user researcher)
46. © Microsoft Corporation Methodology Grounded Theory Open, Axial, Selective Analysis (Amazon + First hour reviews) Invited an auditor to review the process and emergent categories for comprehensibility and closeness to data. Used the interviews in part to check the findings (Selective).
47. © Microsoft Corporation Findings: The First Experience Arc Expectations Specific Anticipations Experiences Momentary & Engaging Annoying, Frustrating, or Tedious Outcome Abandonment Continuing Exceptions Deal-breakers Holdouts
48. © Microsoft Corporation Deal-breakers: Frustrations that supersede all positives. “An UNSKIPPABLE cut scene. Annoying. So, we continue playing and now there are UNSKIPPABLE cut scenes every few plays of the players taunting each other before the ball is hiked! Really? I needed to see his defenseman saying something foul to my running back RIGHT NOW! NO, I don't! We made it through the rest of the game some how and decided that it was one of the worst playing experiences of any game we've ever played. Reselling this ASAP! BTW, the football game play was actually pretty good. Running, throwing and fielding all seemed to work well. In fact, I would say that it worked better than Madden. Its such a shame that they made the rest of the game so unplayable.” (Blitz: The League, Amazon Review)
49. © Microsoft Corporation Holdouts: Playing a game you hate Anticipated Elements “I can't stop before meeting some aliens, right?” (Crysis 2) A good game mechanic “Would I keep playing? I guess so. I really do like the combat system, which captures Captain America's acrobatic grace and power quite well, but the limited enemy variety is allowing that repetitious feeling to sink in. … I'll soldier on for a bit, but I don't expect to finish this fight.” (First hour review of Captain America: Super Soldier) Narrative Completionists (external factor) Social reasons (external factor)
50. © Microsoft Corporation Game development Emerson R. Murphy-Hill, Thomas Zimmermann, Nachiappan Nagappan: Cowboys, ankle sprains, and keepers of quality: how is video game development different from software development? ICSE 2014
51. © Microsoft Corporation
52. © Microsoft Corporation Methodology: Interviews • Drew 14 participants from LinkedIn 2+ years dev experience in games AND 2+ years dev experience outside of games • Opened with a general question about “what’s the difference between games and other types of software?” • Then focused on SWEBOK and Work topics
53. © Microsoft Corporation Methodology: Surveys • Created 84 candidate statements based on interviews, narrowed down to 28 • Randomly sampled three populations, within Microsoft: – 300 engineers who work on games – 300 engineers who work on Microsoft Office – 300 engineers who work elsewhere
54. © Microsoft Corporation
55. © Microsoft Corporation
56. © Microsoft Corporation Significant Differences Game developers have less clear requirements than non-game developers. Game developers tend to use what they perceive as an Agile process more than non-game developers. Creativity is valued more in game development teams. The ability to communicate with non-engineers is valued more on game development teams. Game development requires a more diverse team. People are more impressed by game developers’ work.
57. © Microsoft Corporation Software Analytics for Digital Games Thomas Zimmermann, Microsoft Research, USA Joint work with Nachi Nagappan and many others. http://aka.ms/esegroup
58. © Microsoft Corporation Thank you!