Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People #ASSETS2022 (Oral presentation by Ryo Iijima)

This slide was presented in "Session 4: Composition in Music, Programming and Design" at the 24th International ACM SIGACCESS Conference on Computers and Accessibility（ASSETS 2022）.
https://assets22.sigaccess.org/index.html

【Publication】
Ryo Iijima, Akihisa Shitara, and Yoichi Ochiai. 2022. Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People. In the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS ’22). October 23–26, 2022, Athens, Greece, 8 pages. https://doi.org/10.1145/3517428.3544828

【Project page】
https://digitalnature.slis.tsukuba.ac.jp/2022/10/gestures-for-digital-musical-instruments/

【Presenter】
Ryo Iijima （飯嶋稜）
M.S. Student (Doctoral Program in Informatics)
Graduate School of Comprehensive Human Sciences
University of Tsukuba (JP)
Digital Nature Group (Yoichi Ochiai)

【Abstract】
When playing musical instruments, deaf and hard-of-hearing (DHH) people typically sense their music from the vibrations transmitted by the instruments or the movements of their bodies while performing. Sensory substitution devices now exist that convert sounds into light and vibrations to support DHH people’s musical activities. However, these devices require specialized hardware, and the marketing profiles assume that standard musical instruments are available. Hence, a significant gap remains between DHH people and their musical performance enjoyment. To address this issue, this study identifies end users’ preferred gestures when using smartphones to emulate the musical experience based on the instrument selected. This gesture elicitation study applies 10 instrument types. Herein, we present the results and a new taxonomy of musical instrument gestures. The findings will support the design of gesture-based instrument interfaces to enable DHH people to more directly enjoy their musical performances.

Transcript

1. ©︎ R&D Center for Digital Nature / xDiversity 1 Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65) recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40) pellet drum (AR=0.29) jaw harp (AR=0.05) tam-tam (AR=0.05) castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65) recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40) pellet drum (AR=0.29) jaw harp (AR=0.05) tam-tam (AR=0.05)
2. ©︎ R&D Center for Digital Nature / xDiversity 2 Introduction
3. ©︎ R&D Center for Digital Nature / xDiversity Introduction / Motivation 3 Smartphone apps allow us to play musical instruments
4. ©︎ R&D Center for Digital Nature / xDiversity Introduction / Motivation 4 Di ff icult for DHH users to enjoy applications
5. ©︎ R&D Center for Digital Nature / xDiversity Introduction / Previous Work 5 Muss-Bits Ontenna SOUND HUG https://pixiedusttech.com/technologies/sound-hug/ https://ahlab.org/project/muss-bits/ https://ontenna.jp/
6. ©︎ R&D Center for Digital Nature / xDiversity Introduction / Our Previous Work 6
7. ©︎ R&D Center for Digital Nature / xDiversity Introduction / Research Question 7 How can we design other musical instruments?
8. ©︎ R&D Center for Digital Nature / xDiversity 8 Experiment / Result
9. ©︎ R&D Center for Digital Nature / xDiversity N=11 (Deaf or hard-of-hearing people) 25.4 years old on average (SD=5.39, range=19-38) Experiment / Purpose & Participants 9 Participants Identify gestures for playing musical instruments that is pleasing and easier to use for DHH people. Purpose
10. ©︎ R&D Center for Digital Nature / xDiversity Experiment / Study App 10 Idiophones concussion vessel clappers Idiophones shaken idiophones Idiophones scraped idiophones Membranophones Chordophones Idiophones Idiophones Idiophones Aerophones percussion vessels plucked idiophones percussion sticks Idiophones concussion vessel clappers with manufactured rim
11. ©︎ R&D Center for Digital Nature / xDiversity Experiment / How to Play the Study App 11 tap a button vibrational feedback
12. ©︎ R&D Center for Digital Nature / xDiversity Experiment / Procedure 12
13. ©︎ R&D Center for Digital Nature / xDiversity Experiment / Five Point Likert scale questions 13 This gesture is a good match for its purpose. Goodness of fit This gesture is a easy to perform. Ease of use
14. ©︎ R&D Center for Digital Nature / xDiversity castanet (AR=0.33) triangle (AR=0.22) guiro (AR=0.40) maracas (AR=0.65) recorder (AR=0.13) guitar (AR=0.31) cymbal (AR=0.40) pellet drum (AR=0.29) jaw harp (AR=0.05) tam-tam (AR=0.05) Result / Gesture Set 14
15. ©︎ R&D Center for Digital Nature / xDiversity Result / Taxonomy 15
16. ©︎ R&D Center for Digital Nature / xDiversity Result / Correlation Analysis 16 Goodness of fit & Agreement Rate Ease of use & Agreement Rate Positive correlation Ease of use & Thinking time Negative correlation
17. ©︎ R&D Center for Digital Nature / xDiversity 17 Design Implications
18. ©︎ R&D Center for Digital Nature / xDiversity Design Implications / 1~3 18 The hand not holding a smartphone be able to move as well. Diverse hand poses should be assumed. Avoid designing gestures based on vibrations. 1. 2. 3.
19. ©︎ R&D Center for Digital Nature / xDiversity Design Implications / 4~5 19 Do not disrupt the comfortable holding of the smartphone in normal situations. Encourage users to install a ring-type accessory or remove a notebook-type case. 5. 4.
20. ©︎ R&D Center for Digital Nature / xDiversity 20 Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba Q&A
21. ©︎ R&D Center for Digital Nature / xDiversity Q: How was the vibration pattern designed? 21 Iteratively designed, tested, and improved in a laboratory environment with multiple researchers.
22. ©︎ R&D Center for Digital Nature / xDiversity Q: Were they influenced by the sensing & recognition technologies? 22 They were instructed to treat their smartphone as a “ Magic Brick ” that could recognize any gesture. They were less influenced by the sensing and recognition technologies in creating the gestures.
23. ©︎ R&D Center for Digital Nature / xDiversity Q: Participants did not play to the rhythm, right? 23 The goal is to use gestures to play music and not to select which instrument to play. To this end, this study aimed to define the gestures as preliminary to playing to a rhythm.
24. ©︎ R&D Center for Digital Nature / xDiversity Q: Any cultural differences? 24 Yes, there may be some di ff erences. One participant mimicked a gesture that meant “sutra” in sign language, associating the vibration of a wooden fish that is used in a Buddhist temple, with that of the cymbal. GESs for playing musical instruments with other use groups may provide deeper insights into inclusion.
25. ©︎ R&D Center for Digital Nature / xDiversity Q: Why is the AR of jaw harp so low? 25 No participants were familiar with the jaw harp. With other instruments, participants often created gestures that imitated how they imagined they would play the real one. However, this approach did not work with the jaw harp.
26. ©︎ R&D Center for Digital Nature / xDiversity Q: Why is the AR of tam-tam so low? 26 The real tam-tam does not require the performer to hold the sonorous object, and it can be played with one hand. All other idiophones are two-handed instruments, so the roles of the hand holding the device and the other hand are easily fixed.
27. ©︎ R&D Center for Digital Nature / xDiversity 27 Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People Ryo Iijima, Akihisa Shitara, Yoichi Ochiai / University of Tsukuba Figures
28. ©︎ R&D Center for Digital Nature / xDiversity Correlation Analysis 28
29. ©︎ R&D Center for Digital Nature / xDiversity Randomizing process 29
30. ©︎ R&D Center for Digital Nature / xDiversity Agreement Rate 30
31. ©︎ R&D Center for Digital Nature / xDiversity Gesture Grouping Criteria 31
32. ©︎ R&D Center for Digital Nature / xDiversity Other instruments to be explored 32
33. ©︎ R&D Center for Digital Nature / xDiversity 10 instruments 33
34. ©︎ R&D Center for Digital Nature / xDiversity Classification 34