Interactive Tabletops and Surfaces - Lecture 5 - Next Generation User Interfaces (4018166FNR)

This lecture forms part of a course on Next Generation User Interfaces given at the Vrije Universiteit Brussel.

Transcript

1. 2 December 2005 Next Generation User Interfaces Interactive Tabletops and Surfaces Prof. Beat Signer Department of Computer Science Vrije Universiteit Brussel beatsigner.com Department of Computer Science Vrije Universiteit Brussel beatsigner.com
2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2 March 10, 2025 Tabletops ▪ The term tabletop was introduced in 2001 ▪ Tabletop interfaces are suitable for a number of specific activities ▪ multi-user (group) collaborative work ▪ shared horizontal workspace ▪ interaction with the table as well as objects placed on the table (affordances of the table) - relies on a user's mental model of traditional tables ▪ Tabletops support new forms of ▪ hand and gesture interaction ▪ tangible interaction ▪ interactive visualisations (WIMP is not good enough)
3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3 March 10, 2025 Tabletops … ▪ Typical tabletop applications ▪ viewing photos or videos ▪ brainstorming ▪ advanced visualisation and data analysis ▪ what about productivity tasks? - word processing - email - … ▪ Current tabletop solutions seem to be used for dedicated tasks but not as all-purpose computing devices ▪ missing keyboard input, precise pointing and standard applications ▪ what about combining multi-touch with keyboard and mouse input?
4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4 March 10, 2025 Tabletops … ▪ Ergonomic issues ▪ neck muscle strain or back problems when using a horizontal tabletop as primary input device ▪ what about tabletops with an adjustable angle? ▪ visibility and reachability of elements when working in larger groups
5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5 March 10, 2025 Tabletop Research Domains ▪ Computer Supported Cooperative Work (CSCW) ▪ computer-mediated group collaboration and social interactions ▪ Human Computer Interaction (HCI) ▪ beyond the desktop metaphor ▪ individual user actions and performance ▪ Ubiquitous Computing (UbiComp) ▪ beyond traditional desktop computer displays ▪ Tangible User Interfaces (TEI) ▪ interaction with physical objects on a tabletop CSCW HCI UbiComp TEI
6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6 March 10, 2025 Multi-User Tabletop Interfaces ▪ One of the core features of a tabletop interface is that there is enough room for multiple people to sit or stand around the table ▪ Users share the interface ▪ Studies have shown that users adopt territories on a tabletop for better coordination ▪ personal territories - reserved for personal use (generally in front of the user) ▪ group territories - space to perform main task activities (reachable by all users) ▪ storage territories - used to store task resources and non-task items (in the periphery)
7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7 March 10, 2025 Multi-User Tabletop Interfaces … ▪ Some multi-user tabletop interfaces guidelines ▪ provide visibility and transparency of actions - users should see that actions are going on and which actions are performed - personal territory on laptop vs. personal territory on shared tabletop ▪ provide appropriate table space - users should not run out of space when not participating in the group activity Scott et al., 2004
8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8 March 10, 2025 Multi-User Tabletop Interfaces … ▪ Some multi-user tabletop interface guidelines … ▪ provide functionality in the appropriate locality - e.g. reading/writing support in personal territory ▪ support casual grouping of items and tools in the workspace - e.g. group/organise resources in piles as a reminder Scott et al., 2004
9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9 March 10, 2025 Enabling Technologies ▪ Basic components of a tabletop system ▪ touch sensor technology - size that can be covered - real-time and multi-touch tracking - integration with display - planar or support for curved surfaces? ▪ display - projectors - LCD displays - OLED displays ▪ software - potentially events from multiple users and multiple fingers to be tracked - beyond traditional WIMP interfaces - rotation of display areas
10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10 March 10, 2025 Resistive Touch Panels ▪ Typically two clear layers coated with transparent conductive substances ▪ insulating layer between the two conductive sheets ▪ Controller alternates between the layers ▪ driving electric current on one and measuring the current on the other - detection of horizontal and vertical position Schöning et al., 2008
11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11 March 10, 2025 Resistive Touch Panels … ▪ Low-cost manufacturing ▪ Low power consumption compared to other approaches ▪ Input requires pressure on the outer layer ▪ can also be used with a stylus or when wearing gloves ▪ Reduced display quality due to the additional layers ▪ Typically used in older mobile phones, PDAs, digital cameras, …
12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12 March 10, 2025 Surface Capacitive Touch Panels ▪ Uniform transparent conduc- tive coating on glass panel ▪ Electrodes in each corner ▪ uniform electric field across the conductive layer ▪ Touch with finger (or other conductive object) results in transport of charge ▪ Better display quality than resistive panels ▪ High positional accuracy ▪ Difficult to detect multi-touch Schöning et al., 2008
13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13 March 10, 2025 Projected Capacitive Touch Panels ▪ Sensor grid (electrodes) covered by front layer ▪ Enables accurate detection of multi-touch ▪ High positional accuracy ▪ Surface/projected ca- pacitive solutions only work with finger/special stylus ▪ Not suited for large panels ▪ slower transmission of electrical current ▪ Typically used in more recent mobile phones, … Schöning et al., 2008
14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14 March 10, 2025 Surface Acoustic Wave (SAW) ▪ Glass panel with trans- mitting and receiving transducers as well as reflectors ▪ ultrasonic waves ▪ Soft material (e.g. finger) absorbs ultrasonic waves ▪ detected by transducers ▪ Excellent display quality since no layer on top ▪ Can support dual-touch ▪ Wide frame area necessary for transducers [http://www.dmccoltd.com/english/museum/touchscreens/technologies/AcousticWave.asp]
15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15 March 10, 2025 Frustrated Total Internal Reflection (FTIR) ▪ Optical total internal reflection ▪ inner material must have a higher refractive index than outer material ▪ angle at the boundary must be sufficiently small ▪ Infrared light is injected and reflected (leaving material) if a finger touches the surface ▪ computer vision algorithms used to compute the location ▪ Back projection can be used in combination with FTIR [http://www.teksol.in/2015/11/how-screen-touch-works.html]
16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16 March 10, 2025 Diffused Illumination (DI) ▪ Infrared light placed be- hind the projection surface ▪ Depending on the diffuser, DI might also detect fingers and objects above the surface ▪ Easier tracking of physical objects which might be identified by their shape or fiducial markers
17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17 March 10, 2025 DigitalDesk ▪ DigitalDesk developed by Pierre Wellner at Xerox EuroPARC ▪ camera-based tracking and projection ▪ "Instead of making the workstation more like a desk, make the desk more like a workstation" ▪ Many follow-up augmented desk projects Wellner, DigitalDesk, 1991
18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18 March 10, 2025 DiamondTouch Table ▪ Developed at Mitsubishi Electric Research Labora- tories (MERL) in 2001 ▪ Front-projected interactive display ▪ Can detect who is touching the surface ▪ capacitive coupling between the touch surface and receivers located in the chair of each user [http://www.guillaumeriviere.name/collection/tabletop.html]
19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19 March 10, 2025 Video: DiamondTouch Table
20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20 March 10, 2025 Jeff Han’s Multi-Touch Table ▪ Uses refined version of Frustrated Total Internal Reflection (FTIR) ▪ Simple and cheap implementation of a multi-touch surface presented in 2006
21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21 March 10, 2025 Video: Jeff Han’s Multi-Touch Table
22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22 March 10, 2025 BendDesk ▪ Seamlessly combines a vertical and a horizontal multi-touch surface ▪ Uses 2 projectors and 3 cameras ▪ FTIR-based tracking Weiss et al., 2010
23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23 March 10, 2025 Video: BendDesk
24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24 March 10, 2025 Microsoft PixelSense (Samsung SUR40) ▪ Samsung produces the hardware and Microsoft the software ▪ 40 inch LED backlit LCD display (1920×1080) ▪ integrated PC and PixelSense technology ▪ Backlight with IR LEDs ▪ reflected by fingers and other objects - multi-touch and real objects ▪ pixels act as IR sensors [http://www.embeddedinsights.com/channels/wp-content/uploads/2011/03/110329-surface.png]
25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25 March 10, 2025 Video: Microsoft PixelSense
26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26 March 10, 2025 ReacTIVision ▪ Open source toolkit for tangible multi-touch surfaces ▪ Fiducial markers and multi-touch finger tracking Kaltenbrunner and Bencina, 2007 Reactable
27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27 March 10, 2025 Video: Reactable
28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28 March 10, 2025 iTable Interactive Tabletop ▪ Interactive tabletop based on digital pen a paper technology ▪ developed by the GlobIS group at ETH Zurich ▪ table surface covered with Anoto pattern - e.g. pattern printed on paper and covered with glass ▪ top projection ▪ high precision pen tracking test2 Pen-based Google Earth Browser itable Pen-based Drawing Tool
29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29 March 10, 2025 Pen and Touch Interaction on Tabletops ▪ Pen and touch interaction for authoring and editing documents on tabletops ▪ simultaneous use of two modalities ▪ alternative to on-screen widgets (for experts) ▪ non-dominant hand postures define the pen mode Matulic and Norrie, 2013
30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30 March 10, 2025 we-inspire Room ▪ Ideation technology ▪ based on Anoto's digital pen and paper technology ▪ Large collaborative interactive wall surface ▪ Various applications ▪ sketch, write, brainstorm and capture
31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31 March 10, 2025 Video: we-inspire Room
32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32 March 10, 2025 HP Sprout Pro ▪ Immersive computing platform by HP ▪ All-in-one PC ▪ capacitive tactile touchpad with top projection ▪ 3D scanning cameras ▪ creative work without mouse and keyboard … ▪ “blended reality”
33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33 March 10, 2025 Video: HP Sprout Pro
34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34 March 10, 2025 Microsoft Surface Studio 2 ▪ All-in-one PC ▪ 28-inch 4.5K PixelSense display ▪ screen can be tilted to flat position (hinge design) ▪ New tools for creative process ▪ e.g. surface dial - control for different applications when placed on the screen - haptic feedback (menu options)
35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35 March 10, 2025 Video: Microsoft Surface Studio 2
36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36 March 10, 2025 OLED Technology ▪ Displays based on Organic Light Emitting Diodes (OLEDs) ▪ flexible/thinner than LCDs ▪ no background light ▪ 1000 times faster than LEDs ▪ can be printed and produced at less costs ▪ Combination of OLED displays with thin multi- touch devices ▪ ultimate tabletop system
37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37 March 10, 2025 Windowless Plane ▪ Futuristic windowless plane design concept by Technicon Design ▪ Video stream from wing- mounted cameras ▪ Screen estate could also be used for ▪ showing movies ▪ video conferences ▪ … ▪ Plane needs less fuel
38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38 March 10, 2025 Video: Windowless Plane
39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39 March 10, 2025 AquaTop Display ▪ Interactive water surface that can for example be used in a bathroom ▪ gesture-based interaction ▪ Kinect in combination with top projection ▪ New possibilities to interact with a water surface ▪ e.g.poking fingers from beneath the water Takahashi et al., 2012
40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40 March 10, 2025 Video: AquaTop Display
41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41 March 10, 2025 Video: A Day Made of Glass
42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42 March 10, 2025 Video: A Day Made of Glass 2
43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43 March 10, 2025 Exercise 3 ▪ Evaluation
44. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44 March 10, 2025 References ▪ S.D. Scott, M. Sheelagh, T. Carpendale and K.M. Inkpen, Territoriality in Collaborative Tabletop Workspaces, Proceedings of CSCW 2004, International Conference on Computer Supported Cooperative Work, Chicago, USA, November 2004 ▪ https://dx.doi.org/10.1145/1031607.1031655 ▪ J. Schöning et al., Multi-Touch Surfaces: A Technical Guide, Technical Report TUM-I0833, 2008 ▪ http://campar.in.tum.de/pub/schoening2008multitouch/schoening2008multit ouch.pdf ▪ Jeff Han’s Multi-Touch Table, 2006 ▪ https://www.ted.com/talks/jeff_han_demos_his_breakthrough_touchscreen
45. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45 March 10, 2025 References … ▪ P. Wellner, The DigitalDesk Calculator: Tangi- ble Manipulation on a Desk Top Display, Proceedings of ACM UIST 1991, Symposium on User Interface Software and Technology, Hilton Head, USA, November 1991 ▪ https://dx.doi.org/10.1145/120782.120785 ▪ M. Weiss, S. Voelker, C. Sutter and J. Borchers, BendDesk: Dragging Across the Curve, Proceedings of ITS 2010, International Conference on Interactive Tabletops and Surfaces, Saarbrücken, Germany, November 2010 ▪ https://dx.doi.org/10.1145/1936652.1936654
46. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46 March 10, 2025 References … ▪ M. Kaltenbrunner and R. Bencina, reacTIVision: A Computer-Vision Framework for Table- based Tangible Interaction, Proceedings of TEI 2007, International Conference on Tangible and Embedded Interaction, Baton Rouge, USA, February 2007 ▪ https://dx.doi.org/10.1145/1226969.1226983 ▪ F. Matulic and M.C. Norrie, Pen and Touch Gestural Environ- ment for Document Editing on Interactive Tabletops, Proceed- ings of ITS 2013, International Conference on Interactive Tabletops and Surfaces, St Andrews, UK, October 2013 ▪ https://dx.doi.org/10.1145/2512349.2512802
47. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 47 March 10, 2025 References … ▪ Y. Takahashi, Y. Matoba and H. Koike, Fluid Surface: Interactive Water Surface Display for Viewing Information in a Bathroom, Proceedings of ITS 2012, International Conference on Interactive Tabletops and Surfaces, Cambridge, USA, November 2012 ▪ https://dx.doi.org/10.1145/2396636.2396687 ▪ C. Müller-Tomfelde (Ed.), Tabletops: Horizontal Interactive Displays, Springer 2010 ▪ https://dx.doi.org/10.1007/978-1-84996-113-4 ▪ BendDesk Video ▪ https://www.youtube.com/watch?v=5VNTPwVvLzE
48. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 48 March 10, 2025 References … ▪ Microsoft Pixelsense Video ▪ https://www.youtube.com/watch?v=58dsqozft3k ▪ HP Sprout Pro Video ▪ https://www.youtube.com/watch?v=axKbn0-WZrU ▪ Microsoft Surface Studio 2 Video ▪ https://www.youtube.com/watch?v=fsQSWu8gBUs ▪ we-inspire Collaboration Technology Video ▪ https://www.youtube.com/watch?v=eLfpMlyt4BA ▪ AquaTop Display Video ▪ https://www.youtube.com/watch?v=fYJneaa2O8I ▪ Reactable Video ▪ https://www.youtube.com/watch?v=Mgy1S8qymx0
49. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 49 March 10, 2025 References … ▪ A Day Made of Glass Video ▪ https://www.youtube.com/watch?v=6Cf7IL_eZ38 ▪ A Day Made of Glass 2 Video ▪ https://www.youtube.com/watch?v=jZkHpNnXLB0 ▪ Technicon Design Windowless Plane Video ▪ https://www.youtube.com/watch?v=Ad5R5m7O7dM
50. 2 December 2005 Next Lecture Gesture-based Interaction