Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Measuring Hard-to-Measure Things (Inactives, Pr...

Measuring Hard-to-Measure Things (Inactives, Pricing, Collaboration)

This talk covers some of the hard-to-measure things we're studying at GitHub. Stories include: cross-sectional survey project, pricing experiment with attitudinal data, and a Think Aloud with a research sneak attack.

Chrissie Brodigan

January 16, 2016
Tweet

More Decks by Chrissie Brodigan

Other Decks in Research

Transcript

  1. Hi, It’s Nice To Meet You ✴ Live in Sausalito

    ✴ Trained as a historian ✴ Focus on gender & labor ✴ Competitive figure skater ✴ Synchronized swimming 
 (keep it weird mom) ✴ GitHub’s first UXR 3 GitHub
  2. “Single, Unmarried”5 GitHub ✴Age 21 – 27 ✴Unmarried ✴Weight –

    not over 135 lbs ✴Registered nurse ✴No eyeglasses
  3. Larry 6 GitHub ✴You’ve written a clear, but incomplete story.

    ✴You need to go talk to these women. ✴You need to listen to their stories.
  4. “ I tal o 17 core, mo w ne l

    o un.” 9 GitHub
  5. Listening to people changed everything • Experienced both highly marginalizing

    & empowering 
 work conditions. • Skilled, professional, & organized workers in their own labor union. • Were part of a process that changed constitutional law. 10 GitHub
  6. There’s nothing like connecting with people. Listening to stories can

    flip what you think you know, what the data says, on its head. GitHub
  7. 12 I believe that research studies begin with possibility &

    conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
  8. 13 I believe that research studies begin with possibility &

    conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
  9. 14 I believe that research studies begin with possibility &

    conclude with discovery.
 
 There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
  10. In this talk, we’ll cover: ✴ GitHub product & background

    ✴ 3 research techniques & stories ✴ Q&A / Discussion 15 “Tools & Workflows survey – Cross-sectional study “The Golden Ticket” – Controlled pricing experiment “Collaboration” study – Exploratory “think aloud” GitHub Three Stories
  11. GitHub
 ✴ Hosted git version control
 ✴ Business: - Free

    plans for open source - Paid private plans - On-premise Enterprise
 ✴ Powered-by Pull Requests
 (Code review workflows) 16 GitHub
  12. GitHub 4 years ago … 19 @kneath persisted & hired

    me in 2013 
 Justin is a research superfan! GitHub
  13. Hard to Measure ✴ Emotions ✴ Intentions ✴ Motivations +

    Goals ✴ Workflow workarounds ✴ Prior knowledge ✴ Perception 22 GitHub
  14. Researchers are
 human instruments ✴ Researchers guide customers through interviews,

    encouraging them to share experiences that depict the why to the what of data.
 ✴ Qualitative insights often inform how we shape questions for our quantitative studies (surveys, large data set analysis). 23 GitHub
  15. Story #1. Tools & Workflows Survey ✴ Cross-sectional study ✴

    Run annually; repeatable ✴ Able to analyze by user attributes ✴ Informed by prior projects 24 GitHub Push the limits of what we knew with a census-like survey.
  16. Upon sign-up we learned: Q. How familiar are you with

    Git for version control? 76% of people arriving from the U.S. were 
 brand new to git. 3-point scale. GitHub
  17. We realized that we were asking about skills people didn’t

    have. " We shifted strategy to ask people about what they do know. GitHub
  18. Tools & Workflows Instrument Design
 (35 questions) 1. Tools in

    your developer toolkit 2. Channels used for tool discovery 3. Biggest personal challenge 4. Ways to solve that challenge 5. Demographics (human age, etc.) 27 GitHub
  19. Cross-sectional study View & interpret a single photo vs. time-series

    data, which looks at many moments to understand change over time. 17 escalator accidents in 2014. 30 GitHub
  20. Who? We always begin analysis by identifying the “Who.” And,

    we realized that we had a blind spot 
 –new users.
 31 GitHub
  21. Studying 
 new users # # # 32 1. The

    beginning (inception) – When newcomers sign up, poke around, & experiment. It’s harder to find them after they leave (rely upon email outreach). 2. The messy-but-sticky middle – When newcomers are regularly active; in GitHub where the workflows & workarounds happen (they imprint onto & are imprinted by the product experience). 3. The end – Where newcomers have abandoned the product; GitHub “inactives,” of which a large number are “omg duplicates!” & project-sensitive dormant accounts. GitHub
  22. We divided up the 35-question survey into several smaller surveys,

    which we rolled out in waves. We used the opportunity to design a 12-month longitudinal study. 34 GitHub
  23. 
 Longitudinal study 
 (our flashlight into the hows &

    whys) 
 Observe a single cohort over time, gathering data about points of interest at repeated intervals. 
 We analyze the data with both prospective and retrospective studies. GitHub
  24. The Harvard Grant Study 
 Followed 268 men for 75

    years as they both died & aged on into their 90s. 37 GitHub
  25. New Account Creators Study Researchers and their methodology will 


    naturally age alongside their subjects. 38 GitHub
  26. We took a cohort of 90,000 new accounts created in

    September 2015 & divided them into two groups. GitHub
  27. We’re 5 months into the NAC, so we’ll look at

    a cross-sectional view of the current data. Think: 17 escalator accidents last year GitHub
  28. Are new users different because people change over time? (evolution)


    
 Or, is GitHub attracting a new type of user?
 (replacement) GitHub
  29. 43 1. First, when reading graphs identify the strongest pattern.


    2. Next, cover up what’s obvious & look for what’s interesting. Obvious vs. Interesting GitHub
  30. Q. What’s in your toolkit? Obvious: Tenured accounts are more

    likely to use a text editor than an IDE. GitHub
  31. Experience with tools Obvious Interesting! Newcomers are as likely to

    say they use neither an IDE or a Text Editor, as to say they use one. GitHub
  32. Q. Primary text editor? New accounts are more likely to

    be using Notepad++. 29% of the sample GitHub
  33. Q. Primary text editor? Interesting! Atom’s use is much smaller

    among new users than we thought it would be. Obvious Obvious GitHub
  34. Q. Where do you go for advice? One area where

    both newcomers & tenured users act similarly – tool discovery. GitHub
  35. When we put all 3 insights together & looked more

    closely at the world, 
 we noticed a big blind spot. GitHub
  36. Google + % of people who don’t use a text

    editor = . . .audience opportunity GitHub
  37. Will you share why @name? Tips: ✴Include a question from

    the survey to set expectations and encourage click-through. ✴Keep the responsibility on the app’s failure to engage vs. the user for not engaging. ✴Be human. GitHub
  38. Q. Which VCS are you using? Insight: Strong pattern in

    the yellows & greens, which represent “Nothing” and “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub. GitHub
  39. Q. If we could have done one thing … Insight:


    
 Free private repos are NOT universally the most valuable GitHub good.
 
 Only among the 
 most experienced programmers are FPR a plurality of requests. GitHub
  40. With an exit survey ask … 59 1. What were

    you looking for …? 2. Why did you stop using . . . . . ? 3. What’s one thing we could have done better? GitHub
  41. We’re talking about free private repositories, so let’s discuss how

    to measure something like pricing your product. GitHub
  42. 61 Q. How much would you pay for GitHub? $

    $ $ GitHub ? Photo credit: William Warby (Flickr)
  43. 62 Q. How much would you pay for GitHub? ?

    ? ? GitHub ? Photo credit: William Warby (Flickr)
  44. Ask about value –GitHub goods ✴ Mug ✴ T-shirt ✴

    Hoodie ✴ Feature(s) ✴ Experiences 63 GitHub
  45. Story #2. The Golden Ticket ✴Classic controlled experiment, but with

    a nice twist. ✴39,800 eligible candidates between the treatment & control. ✴Coupons for free private repositories (FPR) to individuals with 1+ year of tenure. 64 GitHub
  46. Experiment Design
 39,800 Humans 65 Treatment (19,949) 3 arms of

    6,600 Exit Survey (2,039) 
 
 Shared Feedback Control: 19,851 Screener
 (4,418) Redeemed their (FPR) coupon GitHub
  47. 67 … from the exit survey tells us why people

    did or didn’t engage in one or both of the first two activities. … provides greater insight into what levers to pull with experiences to effect change in behaviors. Attitudinal Data GitHub
  48. Golden Ticket Email ✴ Sent a total of 39,800 emails

    ✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate GitHub
  49. Golden Ticket Email ✴ Your original draw to GitHub ✴

    Experience with other VCS ✴ If you used a competitor product ✴ Technical & social challenges GitHub
  50. We were interested in what people valued most: 
 free

    private repositories or some other good? 75 GitHub
  51. Placing a value on GitHub Goods 77 Good # %

    Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else.
 24% wanted practical training in Git or GitHub. 
 34% reported that publicly consumable goods (e.g. t-shirt, merged PR) would be more valuable. GitHub
  52. Open Text Responses 78 No amount of machine learning or

    text analysis 
 can surface the insights reading open text does. GitHub
  53. Q. If we could grant you one wish to make

    GitHub even better, what would you wish for? Tip: “Wishes” help surface emerging trends GitHub
  54. “at t fie re rite, or pit es a p

    fie pe” 82 GitHub ⋆
  55. Unlimited Collaborators 83 Private appears to be understood as private

    only to me vs. 
 working with other people privately. GitHub
  56. Faster Horses Speaking of listening to customers and anyone who

    spends their time listening to customer requests …. GitHub
  57. Story #3. The Collaboration Study ✴ Customers told us they

    needed a feature: branch permissions. ✴ More permissions = more complexity. ✴ Competitor products offered them. ✴ Pressure was on! We wanted to be thoughtful with how we solved the motivation & goals behind the request. GitHub
  58. Feature Prioritization 89 ✴ Branch Permissions ✴ Automatically syncing forks

    ✴ Sign-off ✴ Only merge with passing tests ✴ Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line GitHub
  59. Feature Prioritization 90 ✴ Branch Permissions ✴ Automatically syncing forks

    ✴ Sign-off ✴ Only merge with passing tests ✴Undo button ✴ Disable force push ✴ Private forks ✴ Prevent merging from the command line GitHub
  60. “Tel bo im n a 
 un to w ha

    le y.” 92 GitHub
  61. 93 Include items in your list that maybe don’t exist,

    but sound like they might. Listen to people define what they think the “feature” is. Ask how, where, when, & why they would use it. Think Aloud GitHub Sneak Attack
  62. Wrapping Up 1. What’s obvious vs. interesting in your data?


    2. How can you use attitudinal data to study perception of value?
 3. Where does a sneak attack make sense? GitHub
  63. Medium (medium.com/@tenaciouscb) 96 ✴ Product Pricing, Attitudinal Data, & GitHub

    Underpants ✴ What’s Obvious vs. What’s Interesting ✴ GitHub Transformers: Tools & Workflows ✴ New Year, New User Journeys ✴ 365 Project: Listening to inactive users ✴ Measuring Hard-to-Measure Things GitHub