Evaluation Methods - Lecture 6 - Human-Computer Interaction (1023841ANR)

This lecture forms part of the course Human-Computer Interaction given at the Vrije Universiteit Brussel.

Transcript

1. Human-Computer Interaction Evaluation Methods Prof. Beat Signer Department Vrije beatsigner.com Universiteit of Computer Brussel Science Department of Computer Science Vrije Universiteit Brussel beatsigner.com 2 December 2005
2. Evaluation ▪ Evaluation is an integral part of the design process usability of the system ▪ user experience ▪ ▪ Observe participants and measure their performance usability testing ▪ experiments ▪ field studies ▪ October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 2
3. Why, What, Where and When to Evaluate ▪ Why evaluate do we fulfill user requirements? ▪ ensure that users can use the product and they like it ▪ “User experience encompasses all aspects of the end-user’s interaction […] the first requirement for an exemplary user experience is to meet the exact needs of the customer, without fuss or bother. Next come simplicity and elegance, which produces products that are a joy to own, a joy to use.” ▪ What to evaluate Nielsen Norman Group conceptual models ▪ early low-fidelity prototypes or high-fidelity prototypes ▪ individual function, complete workflow, aesthetic design, safety, … ▪ October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 3
4. Why, What, Where and When to Evaluate … ▪ Where to evaluate laboratory ▪ natural setting (in-the-wild studies) ▪ - better for user experience ▪ living labs ▪ When to evaluate ▪ Aware Home, Georgia Tech formative evaluations - throughout the design process - what and how to redesign? ▪ summative evaluations - assess the final product - how well did we do? October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 4
5. Three Types of Evaluation ▪ Controlled settings involving users laboratories or living labs ▪ methods: usability testing and experiments ▪ test hypotheses and measure or observe certain behaviour under controlled conditions ▪ - reduce outside influences and distractions - same instructions for all participants and results can be generalised ▪ Natural settings involving users public places and online communities ▪ methods: direct observation (field study), interviews and logging ▪ - identify opportunities for new technology - establish requirements for a new design - decide how to best introduce new technology October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 5
6. Three Types of Evaluation … ▪ Natural settings involving users … ▪ investigate how product is used in the real world with little or no control of users’ activities - due to lack of control, it might be difficult to anticipate what is going to happen - might get unexpected data and new insights ▪ should be unobtrusive but some methods might influence how people behave ▪ Any setting not involving users consultants and researchers critique, predict and model parts of the interfaces to identify obvious usability problems ▪ methods: heuristics, walkthroughs, analytics and models ▪ ▪ Often a combination of methods is used across these three categories in a single study October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 6
7. DECIDE Evaluation Framework ▪ DECIDE framework provides a checklist (guide) to plan an evaluation study and remind about important issues Determine the goals ▪ Explore the questions ▪ Choose the evaluation methods ▪ Identify the practical issues ▪ Decide how to deal with the ethical issues ▪ Evaluate, analyse, interpret and present the data ▪ October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 7
8. Determine the Goals ▪ What are the high-level goals of the evaluation? ▪ Who wants the evaluation and why? ▪ Goals influence the methods used for the study ▪ Possible goals check that user requirements are met ▪ improve the usability of the product ▪ identify the best metaphor for the design ▪ check for consistency ▪ investigate how a product affects working practices ▪ … ▪ October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 8
9. Explore the Questions ▪ Questions help to guide the evaluation ▪ The goal of finding out why some customers prefer to buy paper airline tickets (rather than e-tickets) can for example be broken down into specific sub-questions what are customers’ attitudes to e-tickets? ▪ are customers concerned about security? ▪ is the interface to obtain the e-tickets poor? ▪ - is the system difficult to navigate? - is the response time too slow? - is the terminology confusing (inconsistent)? October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 9
10. Choose the Evaluation Methods ▪ Evaluation method influences how data is collected, analysed and presented ▪ For example, field studies involve observations and interviews ▪ observe users in natural settings ▪ do not involve controlled tests ▪ produce mainly qualitative data ▪ … ▪ October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 10
11. Identify the Practical Issues ▪ Selection of users people with particular level of expertise ▪ gender distribution ▪ age ▪ ▪ Find evaluators ▪ Selection of equipment ▪ will participants be disturbed by cameras? ▪ Stay within the budget ▪ Respect the schedule ▪ Should a pilot study be organised? October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 11
12. Decide How to Deal with the Ethical Issues ▪ Develop an informed consent form ▪ Information for participants goals of the study ▪ what happens with the findings ▪ - anonymity when quoting them confidentiality of personal information (coding) ▪ offer draft of final report ▪ ▪ Participants are free to stop at any time October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 12
13. Evaluate, Interpret and Present the Data ▪ Evaluation method influences how data is collected, analysed and presented ▪ The following needs to be considered ▪ reliability - can the study be replicated by another evaluator or researcher? ▪ validity - does the method measure what we expect? ▪ ecological validity - does the environment influence the findings - are participants aware of being studied (Hawthorne effect)? ▪ biases - is the process creating biases (e.g. preferences of evaluators)? ▪ scope - can the findings be generalised October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 13
14. Usability Testing ▪ Record the performance (quantitative data) of typical users doing typical tasks in a controlled setting ▪ Participants are observed and timed ▪ Data is recorded on video and interactions (e.g. key presses) are logged ▪ users might be asked to think aloud while carrying out tasks ▪ Data is used to calculate the time to complete a task and to identify the number and type of errors ▪ User satisfaction and opinion is evaluated based on questionnaires and interviews ▪ Field observations may provide contextual understanding October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 14
15. Usability Lab with User and Assistant October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 15
16. Testing Conditions ▪ Usability lab or other controlled space ▪ usability-in-a-box and remote usability testing as more affordable and mobile alternatives to a usability lab ▪ Emphasis on selecting representative users ▪ defining representative tasks ▪ ▪ 5–12 participants and tasks no longer than 30 minutes number of participants depends on schedule, availability and cost of running tests ▪ some experts argue that testing should continue until no new insights are gained (saturation) ▪ ▪ Same test conditions for every participant October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 16
17. System Usability Scale (SUS) ▪ Tool to evaluate the overall usability of interactive systems and user interfaces ▪ 10 standard questions with a 5-point Likert scale ▪ SUS 𝑠𝑐𝑜𝑟𝑒 = 2.5 ൫ 𝑄1 − 1 + 5 − 𝑄2 + 𝑄3 − 1 + 5 − 𝑄4 + 𝑄5 − 1 + 5 − 𝑄6 + 𝑄7 − 1 + 5 − 𝑄8 + 𝑄9 − 1 + 5 − 𝑄10 ൯ = 2.5ሺ20 + 𝑄1 + 𝑄3 + 𝑄5 + 𝑄7 + 𝑄9 − 𝑄2 − 𝑄4 − 𝑄6 − 𝑄8 − 𝑄10 ሻ ▪ excellent (>80.3), good (>68), ok (68), poor (<68), awful (<51) J. Brooke, SUS: A Quick and Dirty Usability Scale, 1986 October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 17
18. Experiments ▪ Test hypothesis to discover new knowledge by investigating the relationship between two or more variables ▪ Independent variable is manipulated by the investigator ▪ e.g. 'cascaded menus' vs. 'context menus' ▪ Dependent variable depends on the independent variable ▪ e.g. time to select an option from the menu ▪ We further define a null hypothesis (e.g. "there is no difference in selection time") and an alternative hypothesis (e.g. "there is a difference between the two menus on selection time") October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 18
19. Experiments … ▪ Statistical analysis of the data can be used to contradict the null hypothesis ▪ Experimenter has to set up the conditions and find ways to keep other variables constant (experimental design) October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 19
20. Experimental Design ▪ We have to decide which participants to use for which conditions in an experiment ▪ different participants (between-subjects design) - single group of participants is allocated randomly to the experimental conditions - no order or training effects - large number of participants is needed (to minimise individual differences) ▪ same participants (within-subjects design) - all participants appear in both conditions - less participants needed - need counter-balancing to avoid order effect ▪ matched participants (pair-wise design) - participants are matched in pairs (e.g. based on expertise, gender etc.) - same as different participants, but individual differences are reduced October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 20
21. Usability Testing vs. Research Usability Testing ▪ improve products ▪ a few participants ▪ results inform design ▪ usually not completely replicable ▪ conditions controlled as much as possible ▪ procedure planned ▪ results reported to developers October 26, 2026 Experiments for Research ▪ discover knowledge ▪ many participants ▪ results validated statistically ▪ must be completely replicable ▪ strongly controlled conditions ▪ experimental design ▪ scientific report to scientific community Beat Signer - Department of Computer Science - bsigner@vub.be 21
22. Field Studies ▪ Field studies are done in natural settings ▪ delivers mainly qualitative data ▪ “in-the-wild studies” is a term for prototypes being used freely in natural settings ▪ Aim to understand what users do naturally and how technology impacts them ▪ Field studies are used in product design to identify opportunities for new technology ▪ establish requirements for a new design ▪ decide how to best introduce new technology ▪ evaluate technology in use ▪ ▪ Findings of field studies can sometimes be unexpected October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 22
23. Analysis of Qualitative Data ▪ Qualitative methods for coding data (e.g. via tools such as MAXQDA) MAXQDA October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 23
24. UEQ+ User Experience Questionnaire ▪ Tool to build customised UX questionnaire modular extension of UEQ ▪ customised selection of UX scales ▪ Word templates with questions ▪ - more than 20 languages - answers on a 7-point Likert scale ▪ UEQ+ Data Analysis Tool ▪ October 26, 2026 Excel document creating graphics etc. Beat Signer - Department of Computer Science - bsigner@vub.be 24
25. UEQ+ Available Scales October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 25
26. Online Survey Tools Qualtrics October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 26
27. Inspections, Analytics and Models ▪ Understand users through knowledge codified in heuristics, remotely collected data or models that predict users’ performance ▪ user does not have to be present during the evaluation ▪ Inspection heuristic evaluation and walkthroughs ▪ expert plays role of a user and analyses aspect of the interface ▪ ▪ Analytics ▪ based on user interaction logging (often done remotely) ▪ Predictive models ▪ October 26, 2026 analysing and quantifying physical and mental operations needed for a task Beat Signer - Department of Computer Science - bsigner@vub.be 27
28. Inspections ▪ Experts use their knowledge of users and technology to review the usability of a product ▪ Expert critiques can be formal or informal reports ▪ Heuristic evaluation is a review guided by a set of heuristics ▪ Walkthroughs involve stepping through a pre-planned scenario noting down potential problems October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 28
29. Heuristic Evaluation ▪ Developed by Jacob Nielsen and his colleagues in the early 1990s ▪ Based on heuristics distilled from an empirical analysis of 249 usability problems ▪ Over time the original heuristics have been revised for current technology Jacob Nielsen ▪ Heuristics being developed for mobile devices, wearables, virtual worlds, … ▪ Design guidelines form a basis for developing heuristics October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 29
30. Nielsen’s Original Heuristics ▪ Visibility of system status ▪ Match between system and the real world ▪ User control and freedom ▪ Consistency and standards ▪ Error prevention ▪ Recognition rather than recall ▪ Flexibility and efficiency of use ▪ Aesthetic and minimalistic design ▪ Help users recognise, diagnose and recover from errors ▪ Help and documentation October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 30
31. Discount Evaluation ▪ Heuristic evaluation is referred to as discount evaluation when 3–5 evaluators are used ▪ Empirical evidence suggests that on average 5 evaluators identify 75-80% of the usability problems October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 31
32. Three Stages of Heuristic Evaluation ▪ Briefing session to tell experts what to do ▪ Evaluation period of 1–2 hours in which each expert works separately ▪ each expert takes one pass to get a feel for the product ▪ each expert takes a second pass to focus on specific features ▪ ▪ Debriefing session in which experts work together in order to prioritise the problems October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 32
33. Advantages and Problems ▪ Few ethical and practical issues to consider because no users are involved ▪ Can be difficult (and expensive) to find experts ▪ Only best experts have knowledge of the application domain and the users ▪ Important problems might get missed ▪ Many trivial problems and often also problems that are no problems (false alarms) are identified ▪ Experts have biases October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 33
34. Cognitive Walkthroughs ▪ Focus on ease of learning ▪ Designer presents an aspect of the design together with usage scenarios (focused evaluation of small parts) ▪ Expert is told the assumptions about the user population, the context of use and the task details ▪ One or more experts walk through the design prototype with the scenario and guided by the following 3 questions will the correct action be sufficiently evident to the user? ▪ will the user notice that the correct action is available? ▪ will the user associate and interpret the response from the action correctly? ▪ ▪ As experts work through the scenario they note problems October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 34
35. Analytics ▪ Method for evaluating user traffic through a system or parts of a system ▪ analysing logged parameters of user interactions ▪ Google Analytics is an example for the analytics of web-based solutions ▪ times of day, visitor IP address, exit pages, … ▪ A/B testing ▪ October 26, 2026 large-scale testing of two slightly different designs Beat Signer - Department of Computer Science - bsigner@vub.be 35
36. Predictive Models ▪ Predictive models provide a way of evaluating products or designs without direct user involvement ▪ Less expensive than user testing ▪ Usefulness is limited to solutions with predictable tasks ▪ e.g. telephone answering system, mobile phones, … ▪ Based on expert error-free behaviour October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 36
37. GOMS Model ▪ Goals ▪ what does the user want to achieve - e.g. find a website ▪ Operators ▪ cognitive processes and physical actions needed to attain goals - e.g. decide which search engine to use ▪ Methods ▪ procedure to accomplish the goals - e.g. drag mouse over input field, type in keywords and press the ‘go’ button ▪ Selection rules ▪ decide which method to select when there is more than one - e.g. press the ‘go’ button or the ‘Enter’ key on the keyboard October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 37
38. Keystroke Level Model ▪ GOMS model has been further developed into the quantitative keystroke level model ▪ Predicts how long it takes an expert user to perform a task by summing up the necessary operations Operator K P P1 H M R(t) October 26, 2026 Description Pressing a single key or button Average skilled typist (55 wpm) Average non-skilled typist (40 wpm) Pressing shift or control key Typist unfamiliar with the keyboard Pointing with a mouse or other device on a display to select an object. This value is derived from Fitts’ Law which is discussed below. Clicking the mouse or similar device Bring ‘home’ hands on the keyboard or other device Mentally prepare/respond The response time is counted only if it causes the user to wait. Time (sec) 0.22 0.28 0.08 1.20 0.40 0.20 0.40 1.35 t Beat Signer - Department of Computer Science - bsigner@vub.be 38
39. Fitts’s Law (1954) ▪ Fitts’s Law predicts that the time to point at an object using a device (e.g. mouse) is a function of the distance from the target object and the target object’s size 𝑇 = 𝑘 log 2 𝐷Τ𝑆 + 1.0 T = time to move the pointer to the target D = distance between the pointer and the target S = size of the target k = constant Paul Fitts ▪ The further away and the smaller the object, the longer the time to locate it and point to it October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 39
40. Further Reading ▪ Parts of this lecture are based on the Interaction Design: Beyond Human-Computer Interaction book ▪ chapter 14 - Introducing Evaluation ▪ chapter 15 - Evaluation Studies: From Controlled to Natural Settings ▪ chapter 16 - Evaluation: Inspections, Analytics and Models October 26, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 40
41. References ▪ Interaction Design: Beyond Human-Computer Interaction, Yvonne Rogers, Helen Sharp and Jenny Preece, Wiley (6th edition), April 2023 ISBN-13: 978-1119901099 ▪ M. Schrepp and J. Thomaschewski, Design and Validation of a Framework for the Creation of User Experience Questionnaires, International Journal of Interactive Multimedia and Artificial Intelligence 5(7), 2019 ▪ https://dx.doi.org/10.9781/ijimai.2019.06.006 ▪ UEQ+: A Modular Extension of the User Experience Questionnaire ▪ October 26, 2026 https://ueqplus.ueq-research.org Beat Signer - Department of Computer Science - bsigner@vub.be 41
42. References ... ▪ J. Brooke, SUS: A Quick and Dirty Usability Scale, Usability Evaluation in Industry, 189(194), 1986 ▪ https://doi.org/10.1201/9781498710411-35 ▪ MAXQDA ▪ https://www.maxqda.com ▪ Qualtrics XM ▪ October 26, 2026 https://www.qualtrics.com Beat Signer - Department of Computer Science - bsigner@vub.be 42
43. Next Lecture HCI Research Methods 2 December 2005