Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥

[第62回NLPコロキウム]「なりきり」を促すHCI設計:対話型接客ロボットの遠隔操作者へのリ...

 [第62回NLPコロキウム]「なりきり」を促すHCI設計:対話型接客ロボットの遠隔操作者へのリアルタイム変換音声フィードバックの適用

Nami Ogawa

July 30, 2024
Tweet

Other Decks in Research

Transcript

  1. ର࿩ܕ઀٬ϩϘοτͷԕִૢ࡞ऀ΁ͷ ϦΞϧλΠϜม׵Ի੠ϑΟʔυόοΫͷద༻ Investigating Effect of Altered Auditory Feedback on Self-Representation,

    Subjective Operator Experience, and Task Performance in Teleoperation of a Social Robot (CHI ’24 Full Paper) ࿦จɿhttps://dl.acm.org/doi/full/10.1145/3613904.3642561 ϓϨεϦϦʔεɿhttps://www.cyberagent.co.jp/news/detail/id=29980 ֓ཁಈըɿhttps://www.youtube.com/watch?v=IEAr3WpYNIU ൃදಈըɿhttps://www.youtube.com/watch?v=5so3PTDnWsk Nami Ogawa, Jun Baba, Junya Nakanishi
  2. લஔ͖ Disclaimerʢ͓͜ͱΘΓʣ • ͍ΘΏΔNLPݚڀͰ͸ͳ͍Ͱ͢ɿ • ݚڀͷத਎΍݁࿦ͦͷ΋ͷΑΓ΋ɺʮHCIݚڀͷงғؾʯ͕఻͑ΒΕͨΒͱࢥ͍ͬͯ·͢ • Technical Contribution͸΄΅θϩ •

    ݚڀߩݙɿ • ࣮ূੑ • Ϣʔβʔ࣮ݧΛ௨ͯ͡ΞΠσΞΛ͖ͪΜͱ࣮ূ͠ɺྑ͍఺/ѱ͍఺Λఆྔతɾఆੑతʹચ͍ग़͠ɺ design implicationsΛٞ࿦ • Ξϓϩʔνͷֶࡍੑɾಠ૑ੑ • ʮԕִϩϘοτ઀٬ʯͱ͍͏࣮༻తγφϦΦʹɺ৺ཧֶత஌ݟΛԠ༻ 8
  3. લஔ͖ HCI (Human-Computer Interaction) ෼໺ͱ͸ • ਓͱܭࢉػΛؚΉܥΛ໌ࣔతʹऔΓѻ͏෼໺ • ଟ͘ͷٕज़ྖҬ͸HCIͱ͸ແؔ܎Ͱ͸͍ΒΕͳ͘ͳ͍ͬͯ͘ʁ •

    ٕज़ྖҬ͕੒ख़͍ͯ͘͠ʹͭΕʮͲ͏࢖(ͬͯ΋Β)͏͔ʯ͕՝୊ʹ • ୅දతͳHCIݚڀͷύλʔϯ • γεςϜఏҊܥɿʮͲ͏࢖ΘΕΔͱخ͍͔͠ʯΛఏҊ • Human FactorsܥɿʮͲ͏࢖ΘΕ͍ͯΔ/࢖ΘΕ͏Δ͔ʯ ɹɹɹɹɹɹɹɹɹʮԿ͕Өڹ͢Δ͔ʯΛඥղ͘ • ࣮ੈքʹࠜͨ͟͠՝୊ղܾࢤ޲ͷྖҬ • ʮ໾ʹཱͭʯͱ͸Կ͔͸ɺඇࣗ໌ͱ͍͏લఏ • ର৅΋Ξϓϩʔν΋ଟ༷ͳ૯߹֨ಆٕ 9 https://pgl.jp/papers?conference=chi2024&search=large%20language%20model CHI 2024Ͱ͸LLMؔ࿈Ͱ100݅ۙ͘ͷൃද https://chi2024.acm.org/subcommittees/selecting-a-subcommittee/ ର৅΋Ξϓϩʔν΋ଟ༷
  4. ໨࣍ - Overview - Background - Approach - Related Work

    - Experiment - Results - Discussion - Conclusion ՝୊ͷಛఆ ΞΠσΞͷཧ࿦తͳཪ෇͚ ΞΠσΞͷ࣮ূతͳཪ෇͚
  5. ໨࣍ - Overview - Background - Approach - Related Work

    - Experiment - Results - Discussion - Conclusion ՝୊ͷಛఆ ΞΠσΞͷཧ࿦తͳཪ෇͚ ΞΠσΞͷ࣮ূతͳཪ෇͚
  6. Overview | Background | Approach | Related Work | Experiment

    | Results | Discussion | Conclusion Improving Teleoperation Experience of Social Robots • Aim: to support teleoperators to ‘ speak as the robot ’ 16 Hello. How can I help you? Teleoperator Social Robot Hello. How can I help you?
  7. Teleoperator Social Robot Improving Teleoperation Experience of Social Robots with

    AAF • Aim: to support teleoperators to ‘ speak as the robot ’ • Idea: to use Altered Auditory Feedback (AAF) 17 to transform acoustic traits of speech and feed it back to the speaker transform Hello. How can I help you? transform & feedback Hello. How can I help you? Hello. How can I help you? Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  8. Teleoperator Social Robot Improving Teleoperation Experience of Social Robots with

    AAF • Aim: to support teleoperators to ‘ speak as the robot ’ • Idea: to use Altered Auditory Feedback (AAF) 18 to transform acoustic traits of speech and feed it back to the speaker transform Hello. How can I help you? transform & feedback Hello. How can I help you? Hello. How can I help you? • Hypothesis: AAF can transform self-representation towards ‘ becoming the robot. ’ Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  9. VRΞόλݚڀ • ʢ਎ମ্ʣଞऀʹͳΔମݧ • ʮࣗ෼Λม͑Δʯπʔϧͱͯ͠ͷVR • ࣗݾද৅͕มԽ͠ɺৼΔ෣͍΋มԽ͢Δ • VRͷεʔύʔώʔϩʔମݧͰ޲ࣾձతߦಈ͕૿͑Δ[1] •

    എ͕ߴ͍ΞόλʔΛ࢖͏ͱɺڧؾͳަবΛ͢Δ[2] [1]: Rosenberg, R. S., Baughman, S. L., & Bailenson, J. N. (2013). Virtual superheroes: Using superpowers in virtual reality to encourage prosocial behavior. PloS one, 8(1), e55003. [2]: Yee, N., & Bailenson, J. (2007). The Proteus effect: The effect of transformed self-representation on behavior. Human communication research, 33(3), 271-290. ԕִϩϘοτ઀٬ • ΩϟϥΫλͱͯ͠ৼΔ෣͏ۀ຿ • VTuberɺςʔϚύʔΫɺ… • →ࣗݾද৅ΛมԽͤ͞ΒΕͳ͍͔ʁ Ξϓϩʔνͷֶࡍੑɾಠ૑ੑ 19 ʮԕִϩϘοτ઀٬ʯͱ͍͏࣮༻తγφϦΦʹɺ৺ཧֶత஌ݟΛԠ༻
  10. ໨࣍ - Overview - Background - Approach - Related Work

    - Experiment - Results - Discussion - Conclusion ՝୊ͷಛఆ ΞΠσΞͷཧ࿦తͳཪ෇͚ ΞΠσΞͷ࣮ূతͳཪ෇͚
  11. The Need to “Speak as the Robot” 21 [1] Seaborn

    et al. “Voice in Human–Agent Interaction: A Survey,” ACM Computing Surveys (2021) incl. acoustic features, style of speech, linguistic content Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion A mismatch between the robot ’ s voice and appearance reduces user acceptance [1].
  12. Limitation in Voice Transformation (VT) 22 No-VT VT-only Operator Robot

    Customers Voice Transformer Mic input • requires skill and effort • simple voice transformation often used in practice • fully automatic, natural, and real-time speech conversion not yet perfect Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  13. Use AAF to Transform Operators’ Self-representation 23 No-VT VT-only Real-time

    Altered Auditory Feedback (AAF) Operator Robot Customers Voice Transformer VT-AAF Mic input • feeding ‘the robot-like’ transformed voice back to the operator • to elicit the ability to speak as if they were the robot’s character • requires skill and effort • simple voice transformation often used in practice • fully automatic, natural, and real-time speech conversion not yet perfect Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  14. Manipulating Voice Self-perception to Affect Emotion • AAF that modifies

    the emotional tone (e.g., happiness, sadness, or fear) elicits the congruent emotional state during speech [3] and conversation [4] • Manipulated change in voice attributed as one ’ s own 24 [3] Aucouturier et al. "Covert digital manipulation of vocal emotion alter speakers’ emotional states in a congruent direction,” PNAS (2016) [4] Costa et al. “Regulating Feelings During Interpersonal Conflicts by Changing Voice Self-perception,” (CHI ’18) adapted from [3] Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  15. Transforming Self-representation with AAF • AAF that converts the age

    of the voice affects the speaker ’ s self-representation [5]. • Virtual reality studies: • Visual (adult or child avatar) and auditory (real or child-like transformed voice) congruence is important [6]. • Change in self-representation affects one ’ s behavior, abilities, and thinking [7]. 25 A “ [5] Arakawa et al. “Digital Speech Makeup: Voice Conversion Based Altered Auditory Feedback for Transforming Self-Representation,” (ICMI ’21) [6] Tajadura-Jiménez et al. "Embodiment in a Child-Like Talking Virtual Body Influences Object Size Perception, Self-Identification, and Subsequent Real Speaking,” Sci. Rep. (2017) Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion [7] Yee et al. "The Proteus Effect: The Effect of Transformed Self-Representation on Behavior,” Hum. Commun. Res.,. (2007) [5] [6]
  16. ໨࣍ - Overview - Background - Approach - Related Work

    - Experiment - Results - Discussion - Conclusion ՝୊ͷಛఆ ΞΠσΞͷཧ࿦తͳཪ෇͚ ΞΠσΞͷ࣮ূతͳཪ෇͚
  17. Research Questions • [RQ1: Change in Self-Representation] • Does AAF

    transform the operator ’ s self-representation toward ‘ becoming the robot ’ ? • [RQ2: Subjective Task Evaluation] • Does AAF make it subjectively easier for the operator to perform the service? • [RQ3: Objective Task Performance] • Does AAF improve service performance objectively? 27 • in social robot teleoperation in a service context • with AAF that transforms the operator’s voice to match the robot’s representation Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  18. Overview • Task: verbally teleoperating a robot installed outside the

    entrance of a bakery • two aspects: to speak a lot (Service) and to speak as the robot (Roleplay) • Participants: N=30 • Gender: 15 Female, 15 Male • Age: 38.00 ± 13.19 (SD), from 21 to 58 years old 28 Operator ’ s equipment Teleoperating interface on a web browser Robot placed at a bakery Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  19. Voice conditions (within-subject factor) *: a hardware audio effect processor

    (ROLAND VT-4) that can shift pitch and formants in real-time (end-to-end latency ~5ms) **: perceived gender and age from appearance are based on the ABOT Database [7] 29 No-VT VT-only Real-time Altered Auditory Feedback (AAF) Operator Robot Customers Voice Transformer* VT-AAF Mic input • transform acoustic traits (i.e., pitch and formants) of the voice to match the appearance (i.e., gender and age) of the robot** • transformed voice fed back to the participant in real-time • participants’ speech is output from the robot as is Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion [7] Phillips et al. “What is Human-like?: Decomposing Robots' Human-like Appearance Using the Anthropomorphic roBOT (ABOT) Database,” (HRI ’18)
  20. Voice conditions (within-subject factor) *: a hardware audio effect processor

    (ROLAND VT-4) that can shift pitch and formants in real-time (end-to-end latency ~5ms) **: perceived gender and age from appearance are based on the ABOT Database [7] 30 No-VT VT-only Real-time Altered Auditory Feedback (AAF) Operator Robot Customers Voice Transformer* VT-AAF Mic input • transform acoustic traits (i.e., pitch and formants) of the voice to match the appearance (i.e., gender and age) of the robot** • transformed voice fed back to the participant in real-time • participants’ speech is output from the robot as is Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion [7] Phillips et al. “What is Human-like?: Decomposing Robots' Human-like Appearance Using the Anthropomorphic roBOT (ABOT) Database,” (HRI ’18)
  21. Voice conditions (within-subject factor) *: a hardware audio effect processor

    (ROLAND VT-4) that can shift pitch and formants in real-time (end-to-end latency ~5ms) **: perceived gender and age from appearance are based on the ABOT Database [7] 31 No-VT VT-only Real-time Altered Auditory Feedback (AAF) Operator Robot Customers Voice Transformer* VT-AAF Mic input • transform acoustic traits (i.e., pitch and formants) of the voice to match the appearance (i.e., gender and age) of the robot** • transformed voice fed back to the participant in real-time • participants’ speech is output from the robot as is Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion [7] Phillips et al. “What is Human-like?: Decomposing Robots' Human-like Appearance Using the Anthropomorphic roBOT (ABOT) Database,” (HRI ’18)
  22. Measures • Subjective • Questionnaire • Robot Embodiment: RQ1 •

    Change in Self-Representation: RQ1 • Task Evaluation: RQ2 • Voice Ownership & Agency: RQ1 • NASA-TLX (mental workload): RQ2 • General Preference: RQ2 • Objective • Implicit Association Test (IAT) : RQ1 • Audio & Video Recordings (Speech Analysis) • Vocal Production: RQ1 • Amount of Conversation and Speech: RQ3 32 Category Scale Item Robot Embodiment Ownership I felt as if the robot's body was my body. Agency The movements of the robot's body were caused by my speaking. Change in Self- Representation FeltChild I felt like a child. FeltRobot I felt like a robot. FeltExtraverted I felt more extroverted than usual. RobotExtraversion1 I see the robot I played as extraverted, enthusiastic. RobotExtraversion2 I see the robot I played as reserved, quiet. Task Evaluation Enjoyment I enjoyed serving the customers. Motivation I was motivated to serve customers. RoleplayEase To speak as Sota was easy. RoleplayConfidence I could speak as Sota with confidence. RoleplaySatisfaction I could speak as Sota satisfactorily. ServiceEase To speak a lot of variety and quantity was easy. ServiceConfidence I could speak a lot of variety and quantity with confidence. ServiceSatisfaction I could speak a lot of variety and quantity satisfactorily. Voice Ownership and Agency OwnVoice I felt as if the voice I heard when I spoke was mine. VoiceFeatures I felt as if the voice I heard when I spoke resembled my (real) voice in terms of tone, pitch, or other acoustical features. VoiceAgency I felt as if I caused the voice I heard. Questionnaire Items Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  23. Measures Used in Self-representation Transformation Studies 33 A “ [5]

    Arakawa et al. “Digital Speech Makeup: Voice Conversion Based Altered Auditory Feedback for Transforming Self-Representation,” (ICMI ’21) [6] Tajadura-Jiménez et al. "Embodiment in a Child-Like Talking Virtual Body Influences Object Size Perception, Self-Identification, and Subsequent Real Speaking,” Sci. Rep. (2017) • Implicit association test (IAT) • Questionnaire • voice ownership Measures used in [5]: Measures used in [6]: • IAT • Vocal Production • F0 (≒ pitch) of participants' speech • Questionnaire • body ownership and agency • experience of being a child • voice ownership and agency Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  24. [RQ1: Change in Self-Representation] Inconclusive: evidenced by questionnaires & vocal

    analysis but not IAT 34 *: asked only in the VT-AAF condition Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion Measurements Hypotheses (in VT-AAF compared to the others) Results Subj. Questionnaire Robot Embodiment strong embodiment over a robot ✅ (partiy) Ownership: No-VT, VT-only < VT-AAF Agency: n.s. Change in Self- representation change in self-representation towards a robot ✅ No-VT, VT-only < VT-AAF Voice Ownership and Agency* ownership and agency over AAF ✅ (partly) Ownership: Low Agency: Moderate Obj. Implicit Association Test (IAT) strong self-association with a childlike robot n.s. F0 (≒ pitch) of the participants' speech shift toward F0 of the feedback ✅ VT-only < No-VT, VT-AAF
  25. [RQ2: Subjective Task Evaluation] • vs No-VT: VT-AAF better in

    all indices • vs VT-only: VT-AAF either better or comparable —- effective in enjoyment, motivation, roleplay • 87% of the participants preferred VT-AAF 35 Measurements Hypotheses (in VT-AAF compared to the others) Results Questionnaire Task Evaluation improved subjective operator experience ✅ (partiy) general experience (enjoyment and motivation): No-VT, VT-only < VT-AAF Roleplay: No-VT < VT-only < VT-AAF Service: No-VT < VT-only, VT-AAF NASA-TLX reduced overall mental workload ✅ (partiy) No-VT > VT-only, VT-AAF General Preference most preferred ✅ 87% (26/30) preferred AAF Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  26. [RQ3: Objective Task Performance] Task performance not signi fi cantly

    in fl uenced by voice conditions 36 Measurements Hypotheses (in VT-AAF compared to the others) Results Amount of Conversation and Speech Conversational Duration duration increases n.s. Amount of Speech amount increases n.s. Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion •1. Total duration of conversation (w/ local users) •how much operator's speech attracted local users •2. Total word count of operator ’ s speech •how much an operator was motivated to speak
  27. 37 • No-VT < VT (VT-only & VT-AAF): • reducing

    mental workload • easing customer service aspect • Without-AAF (No-VT & VT-only) < VT-AAF: • increasing motivation and enjoyment • easing role-play aspect • 87% preferred AAF Inconclusive: evidenced subjectively but not objectively No in fl uence observed in “speaking a lot” & “attracting the users ’ attention” [RQ3: Objective Task Performance] [RQ2: Subjective Task Evaluation] [RQ1: Change in Self-Representation] Apply VT at least, add AAF for operators’ subjective experience Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  28. • 1) Application of AAF with practical scenario: teleoperation of

    a social robot • 2) Demonstration of aspects of AAF that benefit the operator through a field experiment • 3) Design implications for teleoperation interface 38 Contributions: Teleoperator Social Robot Hello. How can I help you? transform & feedback Hello. How can I help you? Hello. How can I help you? Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  29. AAF Can Enhance Teleoperation Experience of Social Robots 40 Overview

    | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion • 1) Application of AAF with practical scenario: teleoperation of a social robot • 2) Demonstration of aspects of AAF that benefit the operator • subjective operator experience … ✅ • task performance … n.s. • 3) Design implications for teleoperation interface Teleoperator Social Robot Hello. How can I help you? transform & feedback Hello. How can I help you? Hello. How can I help you?
  30. ]

  31. Teleoperation: A Promising Solution 44 Social Robots: robots capable of

    human-like social communication Teleoperation: How can technology support teleoperators? remote manipulation of a robot by a human operator • can offer natural and compelling communication • support for operators necessary for this to spread • promising in service fields • airport, cafe, hotel, shopping mall, etc. • fully-autonomous type yet to be perfected Overview | Background | Approach | Related Work | Experiment | Results | Discussion | Conclusion
  32. ۤ࿑࿩ 50 2݄ɿΞΠσΞݕূ 3݄ɿ࣮ݧܭըɾ४උ 4݄ɿ༧උ࣮ݧɾϒϥογϡΞοϓ 5݄ɿ༧උ࣮ݧ 6݄ɿຊ࣮ݧʢɾ࿦จࣥචʣ 7݄ɿղੳɾٞ࿦ʢɾ࿦จࣥචʣ 8݄ɿ࿦จࣥච 9݄ɿCHI౤ߘʢ9/15ʒʣ

    . . . 5݄ɿ༧උ࣮ݧ 6݄ɿຊ࣮ݧ։࢝ → தࢭɾ࢓੾Γ௚͠ 7݄ɿਅɾຊ࣮ݧ։࢝ˠલ൒τϥϒϧͰ੒ཱͤͣ 8݄ɿ8/30ʹຊ࣮ݧऴྃʢฒߦͯ͠ղੳɾٞ࿦ʣ 9݄ɿ࿦จࣥචɾCHI౤ߘʢ9/15ʒʣ 1ϲ݄Ͱऴ͑Δ༧ఆ࣮ͩͬͨݧ͕
 ݁ہؙ3ϲ݄͔͔Δ͜ͱʹɾɾɾ
  33. ࣮ݧʹ͍ͭͯ • Ϣʔβ࣮ݧϝΠϯͷݚڀ • ʮAAFΛ૊ΈࠐΜͩԕִૢ࡞γεςϜͷఏҊݚڀʯͰ͸ͳ͘ɺʮAAF͸ԕִૢ࡞઀٬ʹ༗ޮ͔ʁʯͷΞΠσΞݕূ • ࿦จ΋ɺ΄΅࣮ݧͷใࠂ • ຊจ16pதɿ࣮ݧઃܭʴৄࡉ…5pɺ࣮ݧ݁Ռͷهड़…4pɺ࣮ݧ݁Ռͷߟ࡯…3p •

    Technical Implementationʹ͍ͭͯ • AAFͷ࣮ݱɿཁ݅Λ໌֬ʹ্ͨ͠Ͱෳ਺ͷطଘγεςϜΛൺֱݕ౼͠ɺࢢൢͷϋʔυ΢ΣΞΛબఆ • ROLAND VT-4: ϐονʢ㲈੠ͷߴ͞ʣͱϑΥϧϚϯτʢ࿩ऀͷੑผ΍೥ྸʹ൐ͬͯมԽ͢Δಛ௃ྔʣΛ௒௿஗ Ԇʢ~5msʣͰม׵Մೳ • ςϨΦϖϨʔγϣϯγεςϜ΋طଘٕज़ʢνʔϜͰ։ൃӡ༻͍ͯ͠Δ΋ͷʣ 51
  34. • 1) Application of AAF with practical scenario: teleoperation of

    a social robot • 2) Demonstration of aspects of AAF that benefit the operator through a field experiment • 3) Design implications for teleoperation interface 53 Contributions: Design Implications/Guidelinesͱ͍͏จԽ ↑HCIจԽʁ
  35. ]

  36. ]

  37. HCIͰͷcontributionͱ͸ 57 Why HCI? How to Write a Good CHI

    Paper https://speakerdeck.com/codingconduct/how-to-write-a-good-chi-paper-that-might-just-get-accepted?slide=18
  38. HCIͰͷcontributionͱ͸ 58 Why HCI? How to Write a Good CHI

    Paper https://speakerdeck.com/codingconduct/how-to-write-a-good-chi-paper-that-might-just-get-accepted?slide=18
  39. Ͳ͏ධՁ͢Δ͔͸Ή͔͍ͣ͠ • UISTͷϨϏϡϫʔ޲͚ΨΠυϥΠϯͰڍ͛ΒΕ͍ͯͨɺHCIݚڀʢಛʹUIܥʣͷߩݙͱධՁʹͭ ͍ͯ࿦͍ͯ͡Δࢀߟจݙू • - Daniel R. Olsen Jr.

    (UIST 2007): Evaluating User Interface Systems Research (https://doi.org/ 10.1145/1294211.1294256). • - Saul Greenberg and Bill Buxton (2008): Usability evaluation considered harmful (some of the time) (https://doi.org/10.1145/1357054.1357074) • - David Ledo, Steven Houben, Jo Vermeulen, Nicolai Marquard, Lora Oehlberg & Saul Greenberg (2018): Evaluation Strategies for HCI Toolkit Research (https://doi.org/10.1145/ 3173574.3173610) • - James Fogarty (2017): Code and Contribution in Interactive Systems Research (https:// homes.cs.washington.edu/~jfogarty/publications/workshop-chi2017-codeandcontribution.pdf) 64