About CB Hi, I’m Chrissie Brodigan ✴ Live in Sausalito ✴ Trained as a historian ✴ Focus on gender & labor ✴ Competitive figure skater ✴ Built GitHub’s early UXR practice (2013 - 2016) Twitter: @tenaciouscb
more complete story ✴ Experienced both highly marginalizing & empowering work conditions. ✴ Skilled, professional, & organized workers in their own labor union. ✴ Were part of a process that changed constitutional law.
Tools & Workflows survey START FINISH 2 3 2. New Users Tools & Workflows survey 3. Inactive Users “365” survey The path to a more complete user story may be completely surprising! 16
changed our approach Our team realized we were asking new users to tell us about skills they didn’t have. We changed strategy to ask about skills outside of git and GitHub that new users did have. (Kind of like a census.)
designed a new instrument 1. Tools in a developer toolkit 2. Channels for tool discovery 3. Biggest personal challenge 4. Ways to solve that challenge 5. Demographics
design provided us with a cross-sectional view of data Analyze a snapshot of data vs. studying multiple data points (time-series data). 17 escalator accidents in 2014.
“hows” and “whys” were still a mystery. To better understand outcomes, we needed to study tools, workflows, & skills-development over time. Establishing a baseline
& analyze a single cohort’s data with two types of studies: ✴ Prospective – identify outcomes as they happen in real time. ✴ Retrospective – look back at variables over time and identify how they contributed to known outcomes. We “accidentally on-purpose” designed a longitudinal study
an exit survey: 1. What were you looking for …? 2. Why did you stop using . . . . . ? 3. What product are you using? 4. What’s one thing we could have done better?
Q. Which VCS are you using? Strong pattern in the yellows & greens, which represent “Nothing” & “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub.
Q. What’s one thing we could have done better? Free private repos are NOT universally the most valuable GitHub good. Only among the most experienced programmers are FPR a plurality of requests.
Golden Ticket ✴ Classic controlled experiment, but with a nice twist. ✴ 39,800 eligible candidates between the treatment & control. ✴ Coupons for free private repositories (FPR) to individuals with 1+ year of tenure.
Ticket email ✴ Sent a total of 39,800 emails ✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate
different types of data Human behaviors with activity data: 1. Coupon redemption 2. Repository creation Perception of value with survey data: 3. Attitudinal data
exit survey provides us with insight into why people did or did NOT engage in one or both of the first two activities (redemption & creation). Understanding attitudes helps inform what levers we can design and pull with experiences to effect change in behaviors. Attitudinal Data
Which would you value the most? Good # % Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else. 24% wanted practical training in Git or GitHub. 34% reported that publicly consumable goods ( t-shirt) would be more valuable.
The Collaboration Study ✴ Customers told us they needed a complex feature: branch permissions. ✴ Competitor products offered branch permissions. ✴ Designing for a large audience, means we need to be thoughtful and deliberate. Solve for human motivations & goals behind feature requests.
✴ Include items that don’t exist, but sound like they might. ✴ Listen to people define what they think the “feature” is. ✴ Ask how, where, when, & why they would use the feature. Sneak Attack
Finding the story ✴ Products have new users, tenured users, and inactive users, understanding each experience provides a more complete view. ✴ Researching hard-to-reach places– reading open text responses & listening to humans share their motivations and goals is how you find the story. ✴ Research is your flashlight.
Wrapping Up 1. What’s obvious vs. interesting in your data? 2. How can you gather and use attitudinal data to study perception of value? 3. Where does a sneak attack make sense? 4. How will you uncover a more complete story?