HCI Research Methods - Lecture 7 - Human-Computer Interaction (1023841ANR)

This lecture forms part of the course Human-Computer Interaction given at the Vrije Universiteit Brussel.

Transcript

1. Human-Computer Interaction HCI Research Methods Prof. Beat Signer Department Vrije beatsigner.com Universiteit of Computer Brussel Science Department of Computer Science Vrije Universiteit Brussel beatsigner.com 2 December 2005
2. Human-Computer Interaction Human-computer interaction is a discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and with the study of major phenomena surrounding them. ACM SIGCHI Curricula for Human-Computer Interaction ▪ Human-Computer Interaction is a multidisciplinary field Computer Science ▪ Design ▪ Cognitive Science ▪ Psychology ▪ … ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 2
3. HCI Research Contributions ▪ Empirical contributions ▪ collected quantitative or qualitative data ▪ Artefact contributions (systems research) development and evaluation of new artefacts ▪ interfaces, toolkits, mock-ups, … ▪ ▪ Methodological contributions ▪ new method or modification of existing method, new metrics ▪ Theoretical contributions concepts and models as vehicles for thought ▪ frameworks, design spaces, conceptual models ▪ ▪ Dataset contributions ▪ corpus for the benefit of the research community November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 3
4. HCI Research Contributions … ▪ Literature survey contributions ▪ review and synthesis of work done in a specific area ▪ Opinion contributions ▪ trying to persuade the readers to change their minds ▪ HCI research as well as HCI research methods have changed over time web interfaces, user-generated content, touch screens, collaboration, … ▪ usability engineering, eye tracking, crowdsourcing, … ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 4
5. HCI Research ▪ Most HCI researchers must collect their own data ▪ in other domains (e.g. sociology or economics) data often collected by large entities or government agencies ▪ Studies with many participants (big data) can help us determine correlations ▪ Smaller studies might provide us with a deeper understanding of the meaning of data ▪ Longitudinal studies in HCI are rare technology and tools change rapidly → often does not make sense to compare over multiple years or decades ▪ often seen as a shortcoming ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 5
6. Measurement ▪ What to measure? ▪ In early days of HCI (early 1980s) mainly human performance for individual tasks (micro-HCI) measured in labs task correctness ▪ time performance ▪ error rate ▪ time to learn ▪ user satisfaction ▪ ▪ Broader level (macro-HCI) such as motivation, collaboration or trust not easy to measure with existing metrics ▪ use of multimethod approaches - case studies, observations, interviews, data logging, … November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 6
7. Different Types of Research Type of Research Focus General Claims Typical Methods Descriptive Describe a situation or a set of events X is happening Observations, field studies, focus groups, interviews Relational (Correlational) Identify relations between multiple variables (correlation) X is related to Y Observations, field studies surveys Experimental (Causal) Identify causes of a situation or a set of events (causality) X is responsible for Y Controlled experiments November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 7
8. Experimental Research Hypothesis Study Data ▪ Null hypothesis and alternative hypothesis (see lecture 6) ▪ Independent variables technology, e.g. typing vs. voice input, mouse vs. joystick, … ▪ different types of design, e.g. pull-down vs. pop-up menu, font size, colours, … ▪ user related, e.g. age, gender, culture, computer experience, … ▪ context, e.g. noise, lighting, temperature, … ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 8
9. Experimental Research … ▪ Dependent variables ▪ efficiency - time to complete task, speed (e.g. words per minute) ▪ accuracy - error rate ▪ subjective satisfaction - normally collected via Likert scale ratings (e.g. via questionnaires) ▪ ease of learning or memorability and retention rate - important for adoption of information technology ▪ physical or cognitive demand - how long can we interact without significant fatigue November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 9
10. Experimental Procedure 1. Identify a research hypothesis how many independent variables? ▪ how many values for each independent variable? ▪ 2. Specify experimental design (see lecture 6) ▪ between-subjects design, within-subjects design and pair-wise design 3. Run a pilot study ▪ test design and study instruments 4. Recruit participants 5. Run the data collection sessions 6. Analyse the data (statistical analysis) 7. Report the results November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 10
11. Statistical Analysis ▪ Preprocessing of data ▪ cleaning up data and fixing errors if possible - in anonymous studies we might not be able to contact participants for clarification - if error cannot be fixed, we have to remove the data item and treat it as missing value in the analysis ▪ coding data - e.g. gender: “male” → 1 and “female” → 0 - e.g. degree: “high school” → 1, “university college” → 2 and “university” → 3 ▪ organising data - bring data in layout / format for specific data processing software (e.g. SPSS (Statistical Package for the Social Sciences, by IBM), available via the VUB software shop) November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 11
12. Descriptive Statistical Tests ▪ Distribution of data points, ▪ means, medians, variances, standard deviations and ranges ▪ Comparing means ▪ cannot just compare the means but also have to compute some statistical significance tests (e.g. t tests or ANOVA) ▪ Example Null hypothesis: “There is no significant difference in the task completion time between individuals who use the word-prediction software and those who do not use the software” ▪ Collect some data in an experiment where users use two different versions of the software (with and without word-prediction) ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 12
13. Descriptive Statistical Tests … ▪ Example … ▪ independent-samples t test for data on the next slide results in a t value of 2.169 - higher than the t value (2.131) for the specific degree of freedom (df =15) at the 95% confidence interval - “An independent-samples t test suggests that there is significant difference in the task completion time between the group who used the standard wordprocessing software and the group who used word-processing software with word prediction functions (t(15) = 2.169, p < 0.05).” November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 13
14. Data for Independent-Samples t Test November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 14
15. Data for Paired-Samples t Test November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 15
16. Identifying Relationships ▪ Identify whether there is a relationship (correlation) between various variables ▪ compute Pearson’s product moment correlation coefficient - SPSS will compute a correlation matrix between all variables ▪ Pearson’s r value ranges from -1 to 1 - r =-1.0 means that there is a perfect negative linear relationship between two variables → specific increase in one variable perfectly predicts decrease in the other variable - r =1.0 means that there is a perfect positive linear relationship between two variable → specific increase in one variable perfectly predicts increase in the other variable - r =0 means that there is no linear relationship between the two variables ▪ Note that correlation does not imply a causal relationship ▪ might also be based on hidden (“intervening”) variable November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 16
17. Limitations of Experimental Research ▪ Requires well-defined testable hypotheses with a limited number of independent and dependant variables ▪ many problems not clearly defined or involve a large number of potential variables ▪ Need strict control of factors that might influence the dependent variables ▪ not always possible to control these factors ▪ Lab experiments might not be a good representation of a user’s typical interaction behaviour ▪ study participants might behave differently in lab-based experiments - stress of being observed, different environment, … November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 17
18. Questionnaires/Surveys ▪ Closed as well as open questions (see lecture 3) ▪ Pilot test to ensure validity and reliability ▪ ensure that questions are clear, unambiguous and unbiased ▪ Need higher response rates than surveys in interaction design to get statistically relevant results ▪ Data analysis clean the data ▪ statistical analysis of closed questions ▪ - often just descriptive statistics (percentages etc.) - inferential statistics by understanding the relationships between variables ▪ Might be combined with other research methods ▪ e.g. interviews, focus groups or diaries November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 18
19. Diaries ▪ Individual maintaining regular recordings ▪ fills the gap between observations in naturalistic settings and lab studies ▪ Advantages good for understanding the “why” of user interaction with technology ▪ useful for technology that is used on the go ▪ - difficult to do in lab setting or via observation ▪ Disadvantages participants might not record a sufficient number of entries ▪ data analysis (mix of qualitative and quantitative) might take a long time ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 19
20. Interviews ▪ Open-ended (unstructured), semi-structured or structured interviews (see lecture 3) ▪ Content analysis ▪ usually relies on qualitative methods for coding data - try to find common structures and themes from qualitative data - frequency of terms etc. (e.g. use MAXQDA) ▪ if validity is a particular concern, then multiple researchers should independently analyse the interviews - individual analysts might have some bias ▪ Might best be conducted as complements to other data collection approaches November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 20
21. Grounded Theory Study Data Theory ▪ For new topics with limited literature to build on ▪ no established theories to develop coding categories ▪ Use emergent coding approach based on the notion of grounded theory qualitative research method that seeks to develop theory that is “grounded in data systematically gathered and analysed” ▪ might have multiple rounds of data collection and analysis to allow the underlying theory to fully emerge ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 21
22. Ethnography ▪ Combination of observation, interviews and participation ▪ has its roots in anthropological studies ▪ Examples ▪ home settings - country, culture and religion have a great impact on how technology is used in homes ▪ work settings - e.g. London underground control centre (video ethnography) - non-office-based settings (e.g. vineyards and use of sensors) ▪ educational settings - use of technologies in learning activities ▪ Virtual ethnography ▪ use of web cams or videos November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 22
23. Ethnography … ▪ Very useful in understanding the context of technology usage ▪ Often used as a first step to understand a group of users, their problems, challenges, norms and processes ▪ eventual goal of building some type of technology for them or with them ▪ More recently also ethnographic investigations of ubiquitous computing environments ▪ e.g. navigation needs of firefighters to find their way out of hazardous, smoke-filled environments November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 23
24. Automated Data Collection Methods ▪ Website access log analysers ▪ Activity logging software ▪ Custom software with reporting features November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 24
25. Measuring the Human ▪ Eye tracking ▪ Muscular and skeletal position sensing Wii remote ▪ smartwatches ▪ Microsoft Kinect ▪ ▪ Motion tracking ▪ e.g. for interaction with large wall-sized displays ▪ Physiological data heart rate and blood volume/pressure ▪ galvanic skin response ▪ respiration ▪ brain activity (EEG) ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 25
26. Online HCI Research ▪ Observational online studies usability and think-aloud studies ▪ remote screen sharing ▪ web cam live feed (audio/video) ▪ remote keyboard/mouse control ▪ potential recording of different streams ▪ ▪ A/B testing for Internet business November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 26
27. Human Computation ▪ Humans are often better in tasks requiring detailed interpretation of complex input ▪ Computers can be used to augment humans (see NLS project in lecture 1), but humans can also be used to augment computers → human computation ▪ Task that is hard for a computer but easy for humans ask multiple humans to complete small pieces of the task ▪ e.g. reCAPTCHA ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 27
28. Conducting Human Computation Studies ▪ Use crowdsourcing services for large pool of inexpensive study participants e.g. Amazon’s Mechanical Turk with specific APIs ▪ potentially less bias and increased validity ▪ - participants do not directly interact with researchers November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 28
29. Main HCI Conferences ▪ CHI: ACM Conference on Human Factors in Computing Systems ▪ https://dl.acm.org/conference/chi/ ▪ UIST: ACM Symposium on User Interface Software and Technology ▪ https://dl.acm.org/conference/uist/ ▪ CSCW: ACM Conference on Computer-supported Cooperative Work and Social Computing ▪ https://dl.acm.org/conference/cscw/ ▪ IUI: Annual Conference on Intelligent User Interfaces ▪ https://dl.acm.org/conference/iui/ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 29
30. Main HCI Conferences … ▪ TEI: International Conference on Tangible, Embedded, and Embodied Interaction ▪ https://dl.acm.org/conference/tei/ ▪ EICS: International Conference on Engineering Interactive Computing Systems ▪ https://dl.acm.org/conference/eics/ ▪ DIS: ACM Conference on Designing Interactive Systems ▪ https://dl.acm.org/conference/dis/ ▪ ICMI: International Conference on Multimodal Interaction ▪ https://dl.acm.org/conference/icmi/ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 30
31. Exercise 6 ▪ Heuristic Evaluation November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 31
32. Further Reading ▪ Major parts of this lecture are based on the book Research Methods in Human-Computer Interaction ▪ chapter 1–14 November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 32
33. References ▪ Research Methods in Human-Computer Interaction, Jonathan Lazar, Jinjuan Heidi Feng and Harry Hochheiser, Morgan Kaufmann (2nd edition), May 2019, ISBN-13: 978-0128053904 ▪ Technology in Action (Learning in Doing: Social, Cognitive and Computational Perspectives), Christian Heath and Paul Luff, Cambridge University Press, June 2000, ISBN-13: 978-0521568692 ▪▪ SPSS (Statistical Package for the Social Sciences) https://www.ibm.com/spss ▪ available via VUB Software Webshop ▪ November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 33
34. References … Statistics for HCI: Making Sense of Quantitative Data, A. Dix, Morgan & Claypool, April 2020, ISB-13: 978-1681737430 ▪ J. O. Wobbrock and J. A. Kientz, Research Contributions in Human-Computer Interaction, Interactions 23(3), April 2016 ▪ https://doi.org/10.1145/2907069 November 5, 2026 Beat Signer - Department of Computer Science - bsigner@vub.be 34
35. Next Lecture Use Cases and Course Review 2 December 2005