DR2023 Laura Ryan & Brendan Reed - Design Research and the Evaluation Frontier

Transcript

1. Design Research and the Evaluation Frontier Laura Ryan — Partner, Strategy Brendan Reed — Product Strategist futurefriendly.team
2. | 2 | | Future | Friendly | Three key questions | 01. Why should measurement matter to me? 02. How do I decide what to measure? 03. How do I put it into practice? Three key questions
3. | 3 | | Future | Friendly | Chapter 1 Why does measurement matter?
4. | 4 | Every day, digital services have the power to change people's lives and build a better future. | Future | Friendly | Testing a prototype of the Cost of Living Savings Finder for ServiceNSW
5. | 5 | A cadence of user testing is critical for de-risking during concept development. | Future | Friendly | Usability testing helps us build confidence with new feature releases.
6. | 6 | | Future | Friendly | How certain are you that the digital product you researched, is delivering positive outcomes for people in the real world?
7. | 7 | “If you cannot measure it, you cannot improve it.” — Lord Kelvin | Future | Friendly |
8. | 8 | | Future | Friendly | Chapter 2 Measurement, Evaluation & Learning Framework
9. | 9 | | Future | Friendly | Align stakeholders on what is important to measure. Be highly actionable. Be easily consumed. We designed this measurement and evaluation framework to:
10. | 10 | Share an example of it in practice. Introduce the MEL framework. | Future | Friendly | The Measurement, Evaluation & Learning (MEL) Framework
11. | 11 | Mission Our purpose; the change we are seeking to make. The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
12. | 12 | Mission Our purpose; the change we are seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
13. | 13 | Mission Our purpose; the change we are seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission Evaluation Questions The questions we need to ask to know if we’re on track to achieve this impact. The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
14. | 14 | Mission Our purpose; the change we are seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission Evaluation Questions The questions we need to ask to know if we’re on track to achieve this impact. Metrics The metrics we’ll use to answer these questions. Collection Methods The methods we’ll use to collect the data. The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
15. | 15 | Mission Our purpose; the change we are seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission Evaluation Questions The questions we need to ask to know if we’re on track to achieve this impact. Metrics The metrics we’ll use to answer these questions. Collection Methods The methods we’ll use to collect the data. The Measurement, Evaluation & Learning (MEL) Framework | Future | Friendly |
16. | 16 | | Future | Friendly | NSW state government was looking to create a career development tool supporting citizens to build a digital career in government. MEL Framework case study — Digital Careers Pathway Questionnaire Career advice Two key features:
17. | 17 | | Future | Friendly | Having conducted over 50 hours of concept and usability testing, it was time to pilot the product in the real world.
18. Mission | 18 | The NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway | Future | Friendly |
19. Metric Collection method Mission Evaluation Questions | 19 | The NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway Impact areas | Future | Friendly | Clarity and confidence when exploring digital career pathways. On a scale of 1-5, how easy was it for you to use the tool? Was the tool easy to use and complete? User Interview and Exit Survey
20. Metric Collection method Mission Evaluation Questions | 20 | The NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway Impact areas | Future | Friendly | Clarity and confidence when exploring digital career pathways. On a scale of 1-5, how easy was it for you to use the tool? Was the tool easy to use and complete? Digital career mobility, progression and capability development. Proportion of managers who feel better equipped in career progression discussion Did it lead to more strategic career conversations between managers and users of the tool? User Interview and Exit Survey Manager Interview and Survey
21. Metric Collection method Mission Evaluation Questions | 21 | The NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway Impact areas | Future | Friendly | Clarity and confidence when exploring digital career pathways. On a scale of 1-5, how easy was it for you to use the tool? Was the tool easy to use and complete? Digital career mobility, progression and capability development. Proportion of managers who feel better equipped in career progression discussion Did it lead to more strategic career conversations between managers and users of the tool? NSW public sector is seen as an employer of choice for digital specialists User Interview and Exit Survey Manager Interview and Survey Analytics Proportion of users who were not working for the public service Did people use the tool, inside and outside the public service?
22. Metric Collection method Mission Evaluation Questions | 22 | The NSW public sector attracts and retains the highest quality digital talent MEL in action — Digital Careers Pathway Impact areas | Future | Friendly | Clarity and confidence when exploring digital career pathways. On a scale of 1-5, how easy was it for you to use the tool? Was the tool easy to use and complete? Digital career mobility, progression and capability development. Proportion of managers who feel better equipped in career progression discussion Did it lead to more strategic career conversations between managers and users of the tool? NSW public sector is seen as an employer of choice for digital specialists User Interview and Exit Survey Manager Interview and Survey Analytics Proportion of users who were not working for the public service Did people use the tool, inside and outside the public service?
23. | 23 | | Future | Friendly | Evaluation principles Chapter 3
24. | 24 | | Future | Friendly | Start with what matters — not what's easy. Take a top-down, outcomes-focused approach This allows us to; — Be led by what’s important, rather than what exists — Evaluate product performance against purpose — Measure and report on what matters How can we realise our mission through our products and services? How can we measure this? ↓ ↓
25. | 25 | | Future | Friendly | Tell your story simply. The language of evaluation is complex and confusing. This makes frameworks difficult to interpret. Having evaluation questions allows us to; — Improve organisational comprehension and ownership — Create actionable reports with believable recommendations Outcome? Success metric? Indicator? Output? Did it lead to more strategic career conversations between managers and users of the tool? ↓
26. | 26 | | Future | Friendly | Minimum Viable Metrics. There are myriad ways to measure a key evaluation question. The more you have, the more complex it becomes to determine performance and ultimately to report. Aim for no more than two metrics per evaluation question - one qualitative and quantitative. This allows us to; — Mitigate the risk of relying on one point of data, or too much data — Have enough data to gain insight into patterns and performance
27. | 27 | | Future | Friendly | Create space for unintended consequences. Unintended consequences are outcomes generated by the product that weren’t predicted and therefore, aren’t included for evaluation in the MEL framework. They can be positive or negative; either way they are an important learning opportunity. Consider having an impact area dedicated to capturing successes and missteps that don’t fit neatly within the MEL. Creating space to identify and evaluate them allows us to; — Celebrate unprecedented wins — Address and mitigate negative outcomes in the future — Iterate on the product to recreate and capture those positive outcomes more intentionally in the future … … … ↓ ↓ …?
28. | 28 | | Future | Friendly | Don’t create in isolation. In order to believe that something is right, key stakeholders need to be involved in the creation and co-design of the framework. Involve them early and often, to: — Increase understanding and alignment of the goals this framework is trying to achieve — Get their input on what questions they want answered — Increase buy-in that the evaluation questions and metrics, and thus the results we have are correct.
29. Evaluation principles — MEL Framework | 29 | | Future | Friendly | Start with what matters — not what's easy. Tell your story simply. Minimum Viable Metrics. Create space for unintended consequences. Don’t create in isolation. Mission Measures ↓ ↓ Outcome? Success metric? Indicator? Output? …? Take a top down approach to evaluate product performance against purpose. Take a mixed methods approach to minimise the risk of too much, or too little data. Include an impact area that captures successes and missteps that don’t fit neatly within the MEL. Co-create framework with key stakeholders to create believability and buy-in. Translate impact areas into evaluation questions.
30. | 30 | Mission Our purpose; the change we are seeking to make. Impact Areas The impact we need to see to be confident we’ve delivered on the mission Evaluation Questions The questions we need to ask to know if we’re on track to achieve this impact. Metrics The metrics we’ll use to answer these questions. Collection Methods The methods we’ll use to collect the data. The Measurement, Evaluation & Learning (MEL) Framework
31. | 31 | | Future | Friendly | Thank you. → futurefriendly.team