Solution Validation and Metrics - UBC PM Club

March 2021 presentation for the UBC PM Club on solution validation and metrics. Had a ton of fun and turned this into an interactive workshop as well.

1. © 2021 Productboard, Inc. proprietary & confidential Solution Validation (and Metrics) UBC Product Management Club
2. © 2021 Productboard, Inc. proprietary & confidential i Product Excellence Consultant scott@productboard.com Scott Baldwin
3. © 2021 Productboard, Inc. proprietary & confidential Former customer and long-time product leader About Scott 🇨🇦 Based in Vancouver, BC, Canada 😀 20+ years experience as a PM and previously in UX research & IA 👨🏻‍💻 Led product teams in a variety of companies — Thinkific, Finning Digital, Central 1, Inovera Solutions, Blue Zone Entertainment 🤝 Active participant in the product community – passionate about coaching product teams and mentoring those looking to enter PM and grow in their careers 📚 Featured in the O’Reilly book Design Leadership 🥁 Music, ultramarathons 🏃🏻‍♂, and family
4. © 2021 Productboard, Inc. proprietary & confidential 1. Why we need to validate 2. Tenets of good validation practices 3. Methods/techniques and how to choose the right one 4. Workshop 5. Q&A 6. Toolkit Does anyone even pay attention to these agenda slides? Agenda 📒
5. © 2021 Productboard, Inc. proprietary & confidential 🙋🏽‍♂ Why validate? Go to www.slido.com and enter code #90055 to share your thoughts
6. © 2021 Productboard, Inc. proprietary & confidential ⓘ Start presenting to display the poll results on this slide. Why should we do solution validation?
7. © 2021 Productboard, Inc. proprietary & confidential People = who Problem = why Product = how Whoa, he’s already making assumptions? My assumptions 🧐
8. © 2021 Productboard, Inc. proprietary & confidential Most ideas fail because... High failure rate
9. © 2021 Productboard, Inc. proprietary & confidential Valuable = will customers buy this or choose to use it? Usable = can the user figure out how to use this? Feasible = can we build this given the time, skills, and technology we have? Viable = does this work for the various aspects of our business Valuable, usable, feasible, and viable Four risks
10. © 2021 Productboard, Inc. proprietary & confidential Build Measure Learn prototype data ideas
11. © 2021 Productboard, Inc. proprietary & confidential Guidelines
12. © 2021 Productboard, Inc. proprietary & confidential Think in experiments, not projects
13. © 2021 Productboard, Inc. proprietary & confidential DE&I
14. © 2021 Productboard, Inc. proprietary & confidential Fast, easy, and effective
15. © 2021 Productboard, Inc. proprietary & confidential Start together, not alone
16. © 2021 Productboard, Inc. proprietary & confidential Get out of the building
17. © 2021 Productboard, Inc. proprietary & confidential 🤨 Choosing the right method
18. © 2021 Productboard, Inc. proprietary & confidential Which of the four risks are you validating for? What type of hypothesis are you testing? How much evidence do you already have for the specific hypothesis? How much time do you have until the next major decision point or until you run out of money? How will you know if you’re successful or not? Before you start Questions to ask
19. © 2021 Productboard, Inc. proprietary & confidential Reduce uncertainty as much as possible before you build anything Cheap and fast early in journey Use multiple touchpoints to increase the strength and likelihood of gathering evidence Pick your approach wisely - strongest evidence given constraints that your team can support Keep in mind
20. © 2021 Productboard, Inc. proprietary & confidential Methods/techniques 🤠
21. © 2021 Productboard, Inc. proprietary & confidential Time Strength of evidence Clickable prototype Single feature MVP Mash up Concierge Landing page Crowdfunding Split test Presale Survey Wizard of Oz Mock sale Letter of intent Pop up store Spike Usability testing Guerilla testing
22. © 2021 Productboard, Inc. proprietary & confidential Multi-question survey used to learn Surveys
23. © 2021 Productboard, Inc. proprietary & confidential Digital UI with clickable zones to represent the software’s reactions to the customer’s interactions Clickable prototype
24. © 2021 Productboard, Inc. proprietary & confidential Time-bound exploration of technical or design solutions to address feasibility and help answer build, borrow, or buy decisions Spike
25. © 2021 Productboard, Inc. proprietary & confidential Workshop
26. © 2021 Productboard, Inc. proprietary & confidential We're a three-person team creating a B2C food ordering service for family members that are taking care of elderly parents. These family members may be located in a different location than their family members. They are interested in finding ways to ensure folks like their parents/grandparents have enough food and are eating healthy. We know they have lots of in-market options — grocery stores, UBER Eats, meal services that they can use. To address this problem we’ve built a mobile app where people can order on behalf of their family members -- managing everything from ordering to payments. How would we validate that our solution is valuable? https://miro.com/app/board/o9 J_lPm1BEM=/ Scenario
27. © 2021 Productboard, Inc. proprietary & confidential Align on what you want to validate Step 1: Priorities
28. © 2021 Productboard, Inc. proprietary & confidential Make a list of the most important things you want to validate Rate each 1-5 for importance (5 = critical, 1 = not very important) Rate each 1-5 for doubt (5 = most confident, 1 = least confident) Multiple each and write the result -- do those with the highest score Write out a short summary of what you want to know
29. © 2021 Productboard, Inc. proprietary & confidential We believe {this capability} Will result in {this outcome} We will have confidence to process when we see {this measurable signal} State your hypothesis. Write it down. Step 2: Hypothesis
30. © 2021 Productboard, Inc. proprietary & confidential Think: average, ratio, rates, percentage X% increase in customers that {take the desired action} Average # of customers were able to {complete the primary action} Ratio of customers were willing to buy vs. not What’s a measurable signal? Metrics
31. © 2021 Productboard, Inc. proprietary & confidential Pick your approach(es), design your experiment(s) Step 3: How
32. © 2021 Productboard, Inc. proprietary & confidential Determine who will be involved and what roles they will play Step 4: Who
33. © 2021 Productboard, Inc. proprietary & confidential Get clear on when you need to do this by Step 5: When
34. © 2021 Productboard, Inc. proprietary & confidential Do a test run Step 6: Practice
35. © 2021 Productboard, Inc. proprietary & confidential Let’s do it Step 7: Execute
36. © 2021 Productboard, Inc. proprietary & confidential Evaluate and summarize Step 8: Analyze
37. © 2021 Productboard, Inc. proprietary & confidential Share, determine next steps, rinse and repeat Step 9: Share
38. © 2021 Productboard, Inc. proprietary & confidential Q&A
39. © 2021 Productboard, Inc. proprietary & confidential Toolkit
40. © 2021 Productboard, Inc. proprietary & confidential • Surveys • Microsurveys • Clickable prototype • Single feature MVP • Mash up • Spike • Landing page • Physical prototype • Wizard of Oz • Split test • Presale • Letter of intent • Pop up • Crowdfunding Choose from any of these Methods for validation
41. © 2021 Productboard, Inc. proprietary & confidential Asking better questions Question type Purpose Examples Sample Open Learn What, how, who, where, why What areas of your product process do you think needs the most improvement? Why hasn’t that change moved forward? Closed Confirm understanding Do, So, Is, Are Do you think what you’re doing today works? Are you finding that approach is helpful? Closed Seek commitment If, Can, Will, Would, Should, Could Can we do this by next week? Will you be able to meet on Tuesday to over our next steps?
42. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Brainstorm questions • Write the questions • Edit and order • Write instructions • Test it • Field it & invite • Analyze Pros/Cons: • Cheap and easy to execute • Need to get respondents Multi-question survey used to learn Surveys
43. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Grab a tool to do this for you (e.g. Examples: https://userpilot.com or https://userleap.com) or roll your own Pros/cons: • Collect a lot • High completion/submission rate • Can be targeted (e.g. based on product usage) Short survey shown to users of your product Microsurveys
44. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Define your goals and script • Build your prototype • Test it internally • Walk customers through the script and take notes • Summarize key learnings Pros/cons: • Easy to build and validate key jobs • Can rely on a lot of tools to help Digital UI with clickable zones to represent the software’s reactions to the customer’s interactions Clickable prototype
45. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Design the smallest version of a feature the solves a high-impact need/job • Test it internally • Get customers to use it • Gather feedback Pros/cons: • Focussed feedback on single need/job Functioning MVP with one working feature you need to validate Single feature MVP
46. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Map out your flow • Find existing tools to integrate that complete the process/workflow • Integrate • Test • Have customers use it and gather feedback Pros/cons: • Increasingly lots of options here Functioning MVP that combines existing services to illustrate the value Mash up
47. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Define acceptance criteria • Define time box • Undertake work • Analyze Pros/Cons: • Time bound go/no-go decisions • Takes team off other work Time-bound exploration of technical or design solutions to address feasibility and help answer build, borrow, or buy decisions Spike
48. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Domain name • Build page - value proposition, CTA (email signup), customer pains, your solution • Integrate analytics • Drive traffic to your page • Analyze (viewed, signed up, spent time, engaged with content) Pros/Cons: • Relatively inexpensive • Not good for all products Simple web page with value proposition and CTA Landing page
49. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Determine key job you want to test • Create physical example that substitutes • Try it • Validate learnings Pros/cons: • Both cassette tapes and iPhone used this approach (block of wood) • May not address key job you have for your product Something that substitutes for the actual physical product and helps you validate a particular use case Physical prototype
50. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Plan the steps • Test the steps • Conduct the experiment • Analyze learnings Pros/Cons • Great way to test experiences • Can be labour intensive and require more setup time Deliver a customer experience manually with people rather than technology Wizard of Oz
51. © 2021 Productboard, Inc. proprietary & confidential How to do it • Identify behaviour you want to test • Create control and variant • Identify sample size and confidence level • Run test (50% of traffic to each Pros/Cons: • Inexpensive • Traffic requirements for statistical significance not always possible Comparison of two versions, control A against variant B, to see which performs better Split test
52. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Create landing page • Insert options • Make page available and drive traffic • Encourage people to buy before it’s available • Analyze (which options resonated?) Pros/Cons: • Sale of an item before it’s available for purchase Presale
53. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Create your letter of intent • Show letter of intent to target audience • See if they’ll signup • Take notes (quotes, pains, needs, body language) • Review notes Pros/cons: • A good way to find customers who may be willing to continue and help you push ideas forward Short not legally binding contract Letter of intent
54. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Find location • Design experience • Promote • Run store • Gather feedback Pros/Cons: • Can be a great way to validate physical products resonate with customers • Costs involved (permits, leasing, etc.) Temporary retail store where you sell your goods Pop up
55. © 2021 Productboard, Inc. proprietary & confidential How to do it: • Define needs • Choose platform • Create pitch (video, text) of value proposition, CTA, customer pains and your solution, and pledge amounts • Make public • Drive traffic Pros/Cons: • Instant validation • Need to be ready to deliver Fund a project by raising many small amounts of money from a large group of people Crowdfunding