Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DR2023 Laura Ryan & Brendan Reed - Design Research and the Evaluation Frontier

DR2023 Laura Ryan & Brendan Reed - Design Research and the Evaluation Frontier

uxaustralia
PRO

March 15, 2023
Tweet

More Decks by uxaustralia

Other Decks in Design

Transcript

  1. Design Research
    and the Evaluation
    Frontier
    Laura Ryan — Partner, Strategy
    Brendan Reed — Product Strategist futurefriendly.team

    View Slide

  2. | 2 |
    | Future | Friendly | Three key questions |
    01. Why should measurement matter to me?
    02. How do I decide what to measure?
    03. How do I put it into practice?
    Three
    key
    questions

    View Slide

  3. | 3 |
    | Future | Friendly |
    Chapter 1
    Why does
    measurement
    matter?

    View Slide

  4. | 4 |
    Every day,
    digital services
    have the power
    to change
    people's lives and
    build a better
    future.
    | Future | Friendly |
    Testing a prototype of the Cost of Living Savings Finder for ServiceNSW

    View Slide

  5. | 5 |
    A cadence of user testing
    is critical for de-risking
    during concept
    development.
    | Future | Friendly |
    Usability testing helps us
    build confidence with
    new feature releases.

    View Slide

  6. | 6 |
    | Future | Friendly |
    How certain are you that
    the digital product you
    researched, is delivering
    positive outcomes for
    people in the real world?

    View Slide

  7. | 7 |
    “If you cannot measure it,
    you cannot improve it.”
    — Lord Kelvin
    | Future | Friendly |

    View Slide

  8. | 8 |
    | Future | Friendly |
    Chapter 2
    Measurement,
    Evaluation &
    Learning
    Framework

    View Slide

  9. | 9 |
    | Future | Friendly |
    Align stakeholders on what
    is important to measure.
    Be highly actionable. Be easily consumed.
    We designed this measurement and evaluation framework to:

    View Slide

  10. | 10 |
    Share an example of it in practice.
    Introduce the MEL framework.
    | Future | Friendly |
    The Measurement, Evaluation &
    Learning (MEL) Framework

    View Slide

  11. | 11 |
    Mission
    Our purpose; the change we are seeking to make.
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |

    View Slide

  12. | 12 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |

    View Slide

  13. | 13 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    Evaluation Questions
    The questions we need to ask to know
    if we’re on track to achieve this impact.
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |

    View Slide

  14. | 14 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    Evaluation Questions
    The questions we need to ask to know
    if we’re on track to achieve this impact.
    Metrics
    The metrics we’ll use to answer
    these questions.
    Collection Methods
    The methods we’ll use to collect
    the data.
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |

    View Slide

  15. | 15 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    Evaluation Questions
    The questions we need to ask to know
    if we’re on track to achieve this impact.
    Metrics
    The metrics we’ll use to answer
    these questions.
    Collection Methods
    The methods we’ll use to collect
    the data.
    The Measurement, Evaluation & Learning (MEL) Framework
    | Future | Friendly |

    View Slide

  16. | 16 |
    | Future | Friendly |
    NSW state government was
    looking to create a career
    development tool supporting
    citizens to build a digital
    career in government.
    MEL Framework case study — Digital Careers Pathway
    Questionnaire Career advice
    Two key features:

    View Slide

  17. | 17 |
    | Future | Friendly |
    Having conducted over
    50 hours of concept and
    usability testing, it was
    time to pilot the product
    in the real world.

    View Slide

  18. Mission
    | 18 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    | Future | Friendly |

    View Slide

  19. Metric
    Collection method
    Mission
    Evaluation Questions
    | 19 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    Impact areas
    | Future | Friendly |
    Clarity and confidence when exploring
    digital career pathways.
    On a scale of 1-5, how easy was it for you to
    use the tool?
    Was the tool easy
    to use and complete?
    User Interview and Exit Survey

    View Slide

  20. Metric
    Collection method
    Mission
    Evaluation Questions
    | 20 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    Impact areas
    | Future | Friendly |
    Clarity and confidence when exploring
    digital career pathways.
    On a scale of 1-5, how easy was it for you to
    use the tool?
    Was the tool easy
    to use and complete?
    Digital career mobility, progression and
    capability development.
    Proportion of managers who feel better
    equipped in career progression discussion
    Did it lead to more strategic career
    conversations between managers and
    users of the tool?
    User Interview and Exit Survey Manager Interview and Survey

    View Slide

  21. Metric
    Collection method
    Mission
    Evaluation Questions
    | 21 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    Impact areas
    | Future | Friendly |
    Clarity and confidence when exploring
    digital career pathways.
    On a scale of 1-5, how easy was it for you to
    use the tool?
    Was the tool easy
    to use and complete?
    Digital career mobility, progression and
    capability development.
    Proportion of managers who feel better
    equipped in career progression discussion
    Did it lead to more strategic career
    conversations between managers and
    users of the tool?
    NSW public sector is seen as an employer
    of choice for digital specialists
    User Interview and Exit Survey Manager Interview and Survey Analytics
    Proportion of users who were not working
    for the public service
    Did people use the tool, inside
    and outside the public service?

    View Slide

  22. Metric
    Collection method
    Mission
    Evaluation Questions
    | 22 |
    The NSW public sector attracts and retains the highest quality digital talent
    MEL in action — Digital Careers Pathway
    Impact areas
    | Future | Friendly |
    Clarity and confidence when exploring
    digital career pathways.
    On a scale of 1-5, how easy was it for you to
    use the tool?
    Was the tool easy
    to use and complete?
    Digital career mobility, progression and
    capability development.
    Proportion of managers who feel better
    equipped in career progression discussion
    Did it lead to more strategic career
    conversations between managers and
    users of the tool?
    NSW public sector is seen as an employer
    of choice for digital specialists
    User Interview and Exit Survey Manager Interview and Survey Analytics
    Proportion of users who were not working
    for the public service
    Did people use the tool, inside
    and outside the public service?

    View Slide

  23. | 23 |
    | Future | Friendly |
    Evaluation
    principles
    Chapter 3

    View Slide

  24. | 24 |
    | Future | Friendly |
    Start with what
    matters — not
    what's easy.
    Take a top-down, outcomes-focused approach
    This allows us to;
    — Be led by what’s important, rather than what exists
    — Evaluate product performance against purpose
    — Measure and report on what matters
    How can we realise our mission
    through our products and
    services?
    How can we
    measure this?
    ↓ ↓

    View Slide

  25. | 25 |
    | Future | Friendly |
    Tell your story
    simply.
    The language of evaluation is complex and confusing.
    This makes frameworks difficult to interpret.
    Having evaluation questions allows us to;
    — Improve organisational comprehension and
    ownership
    — Create actionable reports with believable
    recommendations
    Outcome?
    Success metric?
    Indicator?
    Output?
    Did it lead to more strategic career conversations
    between managers and users of the tool?

    View Slide

  26. | 26 |
    | Future | Friendly |
    Minimum Viable
    Metrics.
    There are myriad ways to measure a key evaluation
    question. The more you have, the more complex it
    becomes to determine performance and ultimately to
    report.
    Aim for no more than two metrics per evaluation question
    - one qualitative and quantitative.
    This allows us to;
    — Mitigate the risk of relying on one point of data, or
    too much data
    — Have enough data to gain insight into patterns and
    performance

    View Slide

  27. | 27 |
    | Future | Friendly |
    Create space for
    unintended
    consequences.
    Unintended consequences are outcomes generated by
    the product that weren’t predicted and therefore, aren’t
    included for evaluation in the MEL framework. They can
    be positive or negative; either way they are an important
    learning opportunity.
    Consider having an impact area dedicated to capturing
    successes and missteps that don’t fit neatly within the
    MEL.
    Creating space to identify and evaluate them allows us to;
    — Celebrate unprecedented wins
    — Address and mitigate negative outcomes in the
    future
    — Iterate on the product to recreate and capture those
    positive outcomes more intentionally in the future





    …?

    View Slide

  28. | 28 |
    | Future | Friendly |
    Don’t create in
    isolation.
    In order to believe that something is right, key
    stakeholders need to be involved in the creation and
    co-design of the framework.
    Involve them early and often, to:
    — Increase understanding and alignment of the goals
    this framework is trying to achieve
    — Get their input on what questions they want
    answered
    — Increase buy-in that the evaluation questions and
    metrics, and thus the results we have are correct.

    View Slide

  29. Evaluation principles — MEL Framework
    | 29 |
    | Future | Friendly |
    Start with what
    matters — not
    what's easy.
    Tell your story
    simply.
    Minimum
    Viable Metrics.
    Create space
    for unintended
    consequences.
    Don’t create in
    isolation.
    Mission
    Measures
    ↓ ↓
    Outcome?
    Success
    metric? Indicator?
    Output? …?
    Take a top down
    approach to evaluate
    product performance
    against purpose.
    Take a mixed methods
    approach to minimise
    the risk of too much,
    or too little data.
    Include an impact area
    that captures
    successes and
    missteps that don’t fit
    neatly within the MEL.
    Co-create framework
    with key stakeholders
    to create believability
    and buy-in.
    Translate impact areas
    into evaluation
    questions.

    View Slide

  30. | 30 |
    Mission
    Our purpose; the change we are seeking to make.
    Impact Areas
    The impact we need to see to be confident we’ve delivered on
    the mission
    Evaluation Questions
    The questions we need to ask to know
    if we’re on track to achieve this impact.
    Metrics
    The metrics we’ll use to answer
    these questions.
    Collection Methods
    The methods we’ll use to collect
    the data.
    The Measurement, Evaluation & Learning (MEL) Framework

    View Slide

  31. | 31 |
    | Future | Friendly |
    Thank you.
    → futurefriendly.team

    View Slide