Measuring Hard-to-Measure Things (Inactives, Pricing, Collaboration)
This talk covers some of the hard-to-measure things we're studying at GitHub. Stories include: cross-sectional survey project, pricing experiment with attitudinal data, and a Think Aloud with a research sneak attack.
& empowering work conditions. • Skilled, professional, & organized workers in their own labor union. • Were part of a process that changed constitutional law. 10 GitHub
conclude with discovery. There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
conclude with discovery. There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
conclude with discovery. There’s a moment in every study where we get to learn something new about humans, something new about the world together. GitHub
encouraging them to share experiences that depict the why to the what of data. ✴ Qualitative insights often inform how we shape questions for our quantitative studies (surveys, large data set analysis). 23 GitHub
Run annually; repeatable ✴ Able to analyze by user attributes ✴ Informed by prior projects 24 GitHub Push the limits of what we knew with a census-like survey.
your developer toolkit 2. Channels used for tool discovery 3. Biggest personal challenge 4. Ways to solve that challenge 5. Demographics (human age, etc.) 27 GitHub
beginning (inception) – When newcomers sign up, poke around, & experiment. It’s harder to find them after they leave (rely upon email outreach). 2. The messy-but-sticky middle – When newcomers are regularly active; in GitHub where the workflows & workarounds happen (they imprint onto & are imprinted by the product experience). 3. The end – Where newcomers have abandoned the product; GitHub “inactives,” of which a large number are “omg duplicates!” & project-sensitive dormant accounts. GitHub
whys) Observe a single cohort over time, gathering data about points of interest at repeated intervals. We analyze the data with both prospective and retrospective studies. GitHub
the survey to set expectations and encourage click-through. ✴Keep the responsibility on the app’s failure to engage vs. the user for not engaging. ✴Be human. GitHub
the yellows & greens, which represent “Nothing” and “SVN.” As programming experience increases people are much more likely to be using another VCS vs. GitHub. GitHub
Free private repos are NOT universally the most valuable GitHub good. Only among the most experienced programmers are FPR a plurality of requests. GitHub
a nice twist. ✴39,800 eligible candidates between the treatment & control. ✴Coupons for free private repositories (FPR) to individuals with 1+ year of tenure. 64 GitHub
did or didn’t engage in one or both of the first two activities. … provides greater insight into what levers to pull with experiences to effect change in behaviors. Attitudinal Data GitHub
✴ “Free private repositories for @name” ✴ “Free for life” ✴ Misunderstandings about the offer ✴ Good email deliverability, but . . . ✴ Overall low redemption rate GitHub
Private repositories 663 36% GitHub T-shirt 324 17% Merged Pull Request 311 17% Git Training 265 14% GitHub Training 189 10% “Other” 103 6% 64% indicated they would get more value out of something else. 24% wanted practical training in Git or GitHub. 34% reported that publicly consumable goods (e.g. t-shirt, merged PR) would be more valuable. GitHub
needed a feature: branch permissions. ✴ More permissions = more complexity. ✴ Competitor products offered them. ✴ Pressure was on! We wanted to be thoughtful with how we solved the motivation & goals behind the request. GitHub
but sound like they might. Listen to people define what they think the “feature” is. Ask how, where, when, & why they would use it. Think Aloud GitHub Sneak Attack