Cross-Media Information Spaces and Architectures (CISA)

Research on cross-media technologies, information science and human-information interaction.

More information can be found at https://beatsigner.com

Transcript

1. 2 December 2005 Cross-Media Technologies, Information Science and Human-Information Interaction Prof. Dr. Beat Signer beatsigner.com Web & Information Systems Engineering Lab Department of Computer Science Vrije Universiteit Brussel logo.wmf WEB & INFORMATION SYSTEMS ENGINEERING
2. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 2 May 2025 edfestComponents 01001000100 0101001100010010100010010 0100100010010001000100001001000 100010001100001001000110010101010110 1100010010100010010010010100010101011001 101001100001100 101010100 0010100 100 110101010101 00100 01010101000 1001 1111111111 111 0000000010 0 11111111 0101001011 010101010 111010101 001010010110 splash.gif Fluid Cross-Media Information Spaces
3. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 3 May 2025 Information Systems & Management Information Visualisation & Navigation Information Visualisation nd Navigation Human-Machine & Human-Information Interaction CISA RSL [4] MindXpres [8] OC2 [5] associative file system [2] associative file system [2] data-driven storytelling [11] XIMA [13] cross-device interaction [9] iServer/iPaper [3] ViDaX [16] data physicalisation [32] OpenHPS [18] non-linear storytelling [27] PaperProof [29] iGesture [23] PaperSketch [30] TangHo [31] Context Modelling Toolkit [10] Midas [21] SpeeG2 [24] PaperPoint [26] Print-n-Link [17] digital libraries [12] Mudra [22] mixed reality [28] ArtVis [15] open cross-media linking [1] PimVis [14] DocTr [6] EdFest [25] source code visualisation [19] INFEX [7] technology-enhanced learning [20] Cross-Media Information Spaces and Architectures
4. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 4 May 2025 Models and Frameworks edfestComponents dynamic data physicalisation framework RSL hypermedia metamodel iPaper framework iGesture framework OpenHPS hybrid positioning system MindXpres presentation platform A picture containing text, indoor, table, computer Description automatically generated
5. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 5 May 2025 Prof. Dr. Beat Signer Cross-Media Technology, Interac- tive Paper, Data Physicalisation Dr. Audrey Sanctorum User-defined XDI and IoT Inter- action, Human-AI Interaction CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation logo.wmf WEB INFORMATION SYSTEMS ENGINEERING WEB & INFORMATION SYSTEMS ENGINEERING Maxim Van de Wynckel Interoperable Positioning Systems, Implicit HCI Ingela Rossing Dynamic Data Physicalisation Framework and Guidelines CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) A close-up of a person smiling Description automatically generated
6. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 6 May 2025 logo.wmf WEB INFORMATION SYSTEMS ENGINEERING WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Bingyan Lee Next Generation User Interfaces for Human-Information Interaction Isaac Valadez Knowledge Physicalisation and Augmentation, Tangible UIs Yoshi Malaise Technology-enhanced Learning, Content-driven Presentations Qi Xu Next Generation Scholarly Workflows, Cross-Media PIM A person with glasses and a white background Description automatically generated A person with long hair wearing glasses Description automatically generated A person with long hair Description automatically generated with low confidence A person with a beard and glasses AI-generated content may be incorrect.
7. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 7 May 2025 logo.wmf WEB INFORMATION SYSTEMS ENGINEERING WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Piet Van Der Paelt Julia-based Framework for Simulation and Optimisation Ekene Attoh IoT Middleware, Context-aware Computing, Implicit HCI Xuyao Zhang Extensible Platform for Dynamic Data Physicalisation Patrick Sello Digital Twins for Healthcare, Federated Learning
8. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 8 May 2025 logo.wmf WEB INFORMATION SYSTEMS ENGINEERING WEB & INFORMATION SYSTEMS ENGINEERING CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) CROSS-MEDIA INFORMATION SPACES AND ARCHITECTURES (CISA) Arun Sojan Physicalisation for Digital Twins Evan Cole Technology-enhanced Learning, Study Lenses A person wearing glasses Description automatically generated with medium confidence
9. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 9 May 2025 A screenshot of a computer AI-generated content may be incorrect.
10. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 10 May 2025 What is Wrong with Digital Documents? ▪ Existing document formats are based on the simulation of paper affordances on desktop computers ▪ How to manage mixed-media "documents" in open and fluid cross-media information spaces? ▪ on the data level - context-sensitive adaptation - cross-media transclusion ▪ on the visualisation and navigation level - zoomable user interfaces ▪ on the cross-media interaction level - fluid multimodal cross-media interfaces ▪ Remediation of the "paper simulation" approach ▪ WYSIWYG is only one out of many options! CISA Human-Machine & Human-Information Interaction Information Systems & Management Information Visualisation & Navigation
11. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 11 May 2025 Digital Documents as a Paper Simulator? Vannevar Bush Ted Nelson Most people don't understand the logic of the concept: "What You See Is What You Get" is based on printing the document out ("get" means "get WHEN YOU PRINT IT OUT"). And that means a metaphysical shift: a document can only consist of what can be printed! [...] No overlays [...] – PAPER UNDER GLASS. When data of any sort are placed in storage, they are filed alphabetically or numerically, and information is found (when it is) by tracing it down from subclass to subclass. It can be in only one place, unless duplicates are used [...] The human mind does not work that way. It operates by association. As We May Think, Atlantic Monthly, July 1945 Geeks Bearing Gifts: How the Computer World Got This Way, Mindful Press 2009
12. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 12 May 2025 Research Methodology picture1 iserver.png Conceptual Modelling General Frameworks Applications User Studies ipaperArchitecture.jpg
13. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 13 May 2025 RSL Hypermedia Metamodel iwebfirefox movieplugin ipaper Global Information Systems Group, ETH Zurich
14. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 14 May 2025 iserverNew_blue Links Users Layers Plug-ins Structure Active Content Global Information Systems Group, ETH Zurich
15. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 15 May 2025 Personal Information Management (PIM) ▪ Keeping, organising and re-finding information ▪ digital and physical ▪ Study of human- information interaction ▪ files, piles, mixtures, … ▪ Cross-Media PIM system ▪ explicit as well as implicit associations between entities ▪ Next generation scholarly workflows
16. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 16 May 2025 PimVis Setup
17. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 17 May 2025 PimVis Document View Audrey Sanctorum
18. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 18 May 2025 PimVis Focus View Audrey Sanctorum
19. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 19 May 2025 RSL-based Associative File System signer My Pictures Vancouver ER 2010 Granville Island Vancouver Harbour Public Market Keynote J. Thorp Olga and Lamia Dinner ER 2010 multiple classification structural link navigational link My Videos Vancouver Aquarium ER 2010 Keynote cross-media association My Presentations ER 2010 WISE 2009 cross-media transclusion single slide transclusion (via OOXML RSL selector) Gregory Cardone
20. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 20 May 2025 RSL-based Associative File System ... Brent Willems Graphical user interface Description automatically generated
21. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 21 May 2025 RSL-based Hypermedia Engine RSL-based Hypermedia Engine Legacy Applications File System Driver Hypermedia-enabled Applications HT2021.docx HT2021.docx …
22. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 22 May 2025 Bridging the Paper-Digital Divide picture1.png PIC00008.jpg map1.jpg IMG_0738.JPG
23. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 23 May 2025 Interactive Paper Applications ▪ Enhanced Reading ▪ paper-digital maps ▪ interactive festival brochures ▪ educational materials ▪ auction brochures ▪ scientific publications ▪ … ▪ Enhanced Writing ▪ photo album ▪ researcher's notebook ▪ mammography annotation ▪ query by sketching ▪ … ▪ Paper-based Interfaces ▪ PaperPoint ▪ document proof-editing ▪ digital libraries ▪ video analysis tool ▪ educational games ▪ Art Installations ▪ Lost Cosmonaut ▪ Generosa Enterprise ▪ Interactive Tabletops ▪ photo browser ▪ collaborative sketch tool
24. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 24 May 2025 Interactive Festival Guide (EdFest) edfestComponents Global Information Systems Group ETH Zurich
25. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 25 May 2025 Visual and Interaction Design ▪ Visualisation of interactive areas and functionality ▪ design patterns ▪ Interaction design ▪ online versus offline input processing ▪ lack of modal dialogues ▪ multimodal interaction ▪ Findings might be appli- cable to traditional GUIs ▪ e.g. cross-application widget interaction booklet 2004 booklet 2005 Global Information Systems Group, ETH Zurich
26. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 26 May 2025 Next Generation Scholarly Workflows ▪ Investigation of current scholarly workflows ▪ RSL-based PIM model for cross-media scholarly workflows ▪ Researcher’s One-Stop Shop (ROSS) platform ▪ plug-ins for existing scholarly tools ▪ cross-media interfaces A diagram of a computer software AI-generated content may be incorrect.
27. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 27 May 2025 EUD of Cross-Device and IoT Applications ▪ Rapid prototyping platform for cross-device and IoT applications (eSPACE) ▪ End-user authoring ▪ customised distribution of user interface components ▪ mashup tool for digital and physical (IoT) components
28. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 28 May 2025 Mixed Reality in IoT Environments ▪ eSPACE IoT platform ▪ end-user authoring ▪ mashup tool for digital and physical (IoT) components ▪ Context Modelling Toolkit (CMT) ▪ OpenHPS hybrid positioning system ▪ stream-based processing ▪ interoperability ▪ discoverability Elena Zambon
29. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 29 May 2025 “Linking” in IoT Environments Yishen Yu
30. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 30 May 2025 Hybrid Positioning System ▪ OpenHPS hybrid positioning system ▪ stream-based processing ▪ interoperability ▪ discoverability ▪ Combine existing and new positioning techniques ▪ Data Structure ▪ data objects ▪ data frames ▪ absolute or relative positions Shape, circle Description automatically generated A hand holding a phone AI-generated content may be incorrect.
31. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 31 May 2025 Positioning System Ontology A diagram of a system AI-generated content may be incorrect. A diagram of a computer system AI-generated content may be incorrect. ▪ Ontology for interoperable positioning systems ▪ describe different parts of positioning system in RDF ▪ POSO extends the SSN and SOSA ontologies ▪ Use of Solid Pods for storing user-controlled positioning data
32. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 32 May 2025 Semantic Beacons ▪ Discovery using Bluetooth indoor positioning ▪ SemBeacon specification ▪ extension of POSO combined with Bluetooth specification ▪ combination of AltBeacon and Eddystone-URL ▪ encoding URI of online RDF description ▪ hardware beacons Screens screenshots of a phone AI-generated content may be incorrect. A close-up of a red circuit board AI-generated content may be incorrect.
33. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 33 May 2025 Integrated Positioning Solution A screenshot of a computer AI-generated content may be incorrect. Maxim Van de Wynckel
34. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 34 May 2025 Cross-Platform IoT Solution ▪ Many different IoT plat- forms ▪ rules normally not compatible ▪ NLP-based solution to translate proprietary rules to high-level EUPont model ▪ high-level rules managed in personal Solid Pods ▪ rules can be executed on different IoT platforms via an EUPont runtime A diagram of a computer system Description automatically generated Attoh and Signer, 2024
35. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 35 May 2025 Context Modelling Toolkit (CMT) ▪ Multi-layered context modelling approach ▪ programmer, expert user and end user ▪ Beyond simple "if this then that" rules ▪ Client-server architecture ▪ server: context reasoning based on Drools ▪ client: sensor input as well as applications End User Expert User Functions Actions Template Filled in template Situation Situations Facts Rule Programmer Rule (4) (5) (6) (7) (8) ▪ Increase trust via intelligibility
36. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 36 May 2025 Context Modelling Toolkit (CMT) …
37. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 37 May 2025 Personalised Learning Environments ▪ Diagnostic assessments to detect knowledge gaps ▪ Suggestion of learning content ▪ domain knowledge graphs ▪ user knowledge graph ▪ learning paths ▪ Automatic content adap- tation ▪ based on user preferences and constraints Enhanced Curriculum Assessments Tagged Learning Content Knowledge Graphs Adaptive Learning Content Learning Paths Knowledge Representation Framework Assessment Module Assessment Analyser Knowledge Gap Analyser Adaptive Learning Environment Suggestion Module User Knowledge Graph Personalised Presentation Module Learner Constraints Groups Module
38. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 38 May 2025 Personalised Learning Environments … A diagram of a company Description automatically generated
39. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 39 May 2025 Cross-Space Collaboration ▪ Cross-Space Collaboration model ▪ meeting aspects influencing in-person collaboration, such as cognitive impact, meeting variables and participant characteristics ▪ Meeting scheduler to decide on the best form of a (cross-space) meeting A screenshot of a computer AI-generated content may be incorrect.
40. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 40 May 2025 Framework for Julia-based Simulation A diagram of a software flow AI-generated content may be incorrect.
41. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 41 May 2025 Tangible Hologram (TangHo) Setup Timothy J. Curtin
42. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 42 May 2025 TangHo Prototype ▪ Lego Mindstorms-based 6DOF arm prototype ▪ motors can be locked or set to float mode - useful for bidirectional I/O ▪ Replaceable end effector for different types of non-visual feedback ▪ Challenges ▪ inverse kinematics (final system is body mounted) - use robotics toolbox (MATLAB) ▪ limited HoloLens hand tracking ▪ 3D printing of final arm Timothy J. Curtin
43. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 43 May 2025 TangHo Interactions
44. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 44 May 2025 Real-Time Cross-Media Data Exploration
45. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 45 May 2025 Dynamic Data Physicalisation ▪ Physical objects used for input as well as output ▪ How can we achieve dynamic data physicali- sation with dynamic affordances ▪ use physical variables such as temperature or texture ▪ exploration of big data sets ▪ experimental tangible holograms (TangHo) platform
46. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 46 May 2025 Data Physicalisation Framework ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Understanding the perceptual effectiveness ▪ e.g.what are the just-noticeable differences that a physical variable can convey? ▪ Framework for dynamic data physicalisation ▪ data processing with device-independent output - TextDIO: text for static or dynamic reports - PhysDIO: data points, data variables and physical variables ▪ text and visualisation drivers for representation on specific devices (e.g. TangHo) ▪ new data physicalisation design guidelines (as in InfoVis)
47. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 47 May 2025 Data Physicalisation Framework ...
48. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 48 May 2025 Dynamic Data Physicalisation ▪ Understanding the design space ▪ how do users map data to physicalisations ▪ Data physicalisation design guidelines ▪ what are the just-perceptual differences that a physical variable can convey? ▪ Dynamic data physicalisation framework ▪ data physicalisation grammar ▪ data processing with device-independent output ▪ software drivers for different physicalisations (e.g. TangHo) ▪ New application domains Jacques Bertin, 1967
49. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 49 May 2025 Modular DataPhys Hardware Platform ▪ Modular and extensible platform for dynamic data physicalisation ▪ communication architecture for plug-and-play of modules ▪ modules representing different physical variables - self-describing language for physicalisation of modules ▪ Rapid prototyping of dy- namic data physicalisation A drawing of different sizes of objects AI-generated content may be incorrect. A white rectangular object with black and white objects AI-generated content may be incorrect.
50. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 50 May 2025 What is Wrong with Slideware? ▪ Simulation of physical slides ▪ limited space due to the slide concept ▪ linear navigation from slide to slide ▪ difficult to reuse content and embed rich media types ▪ MindXpres addresses these issues ▪ unlimited canvas with zoomable user interface ▪ non-linear navigation and associative linking ▪ content-based approach with automatic visualisation
51. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 51 May 2025 Presentation Solutions ▪ PaperPoint ▪ interactive PowerPoint paper handouts ▪ non-linear navigation and annotations ▪ in use since 2005 ▪ MindXpres ▪ extensible presentation platform ▪ non-linear navigation via zoomable user interface ▪ rich media plug-ins ▪ developed since 2011
52. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 52 May 2025 PaperPoint Features ▪ Enhanced mobility during presentations ▪ Improved presentation overview for the presenter ▪ see content of current and nearby slides (orientation) ▪ Annotation of static slide content ▪ highlighting of content ▪ dynamic creation of new content at presentation time ▪ Improved control of embedded media (e.g. video) ▪ Digital whiteboard ▪ Multi-pen support ▪ collaborative presentations ▪ brainstorming
53. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 53 May 2025 MindXpres Presentation Platform ▪ Flexible representation of presentations ▪ use of structural RSL links ▪ separation of content and structure ▪ Extensible platform ▪ content-based approach ▪ cross-media content reuse ▪ non-linear navigation and zoomable user interface ▪ associative linking ▪ rich media types Reinout Roels
54. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 54 May 2025 MindXpres Platform ▪ MindXpres presentations are currently represented in the XML-based MindXpres document format ▪ Compiler (node.js application) translates XML to HTML ▪ Presentation engine based on HTML5 and related APIs ▪ e.g. WebSockets for connectivity
55. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 55 May 2025 MindXpres Document Representation R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI 3(Issue EICS), 2019 R. Roels and B. Signer, A Conceptual Framework and Content Model for Next Generation Presentation Solutions, PACMHCI 3(Issue EICS), 2019
56. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 56 May 2025 Navigation Mirroring Plug-in
57. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 57 May 2025 Interactive Source Code Plug-in Paul Mestereaga
58. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 58 May 2025 MindXpres Video
59. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 59 May 2025 Interactive Data Visualisation
60. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 60 May 2025 Open Cross-Media Linking ▪ Integration of a new resource type includes ▪ development of a data plug-in ▪ implementation of a visual plug-in ▪ registration with the resource plug-in repository ▪ Link browser loads plug-ins on demand ▪ internal or external (third party application) visualisation ▪ Third-party applications use the link browser for visualisation ▪ similar to the integration of third-party applications with web browsers ▪ Non-monolithic annotation model and authoring tool ▪ resource-specific plug-ins
61. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 61 May 2025 Visual Plug-ins annotationBrowserNew.emf Annotation/Link Browser & Editor Client Application showEntity(), getSelectedEntity() showEntity()
62. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 62 May 2025 Cross-Document Linking local visual plug-ins external visual plug-ins communicate Link Browser
63. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 63 May 2025 Document Retrieval and Discovery ▪ Use explicit as well as implicit document relationships
64. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 64 May 2025 NRX Headset.jpg Multimodal and Multi-Touch Interaction ew-1.jpg kinect.jpg wr.jpg SamsungGalaxy.jpg EPOC, emotiv Echo Smartpen, Livescribe Kinect, Microsoft
65. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 65 May 2025 ▪ Midas/Mudra Framework ▪ declarative definition of multimodal and multi-touch interactions ▪ rule-based language approach ▪ rapid prototyping and application development ▪ iGesture Workbench ▪ create and test gesture sets and algorithms ▪ different modalities: digital pen, Wii remote, … ▪ open source (http://igesture.org) ▪ HephaisTK Toolkit ▪ domain-specific human-computer dialogue language (SMUIML) for multimodal I/O ▪ speech recognisers, RFID readers, … gestureSet.png Screen shot 2011-04-22 at 20.04.29.jpg Multimodal and Multi-Touch Interaction
66. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 66 May 2025 Mudra ▪ Fusion across different levels of abstraction ▪ unified fusion framework based on shared fact base ▪ Interactions defined via declarative rule-based language (Midas) ▪ Rapid prototyping ▪ simple integration of new input devices and external recognisers Hoste et al., 2011
67. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 67 May 2025 Video: Kinect Presenter
68. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 68 May 2025 Video: Augmented Reality (Mental Nomad) Toon Duwee 25.04.jpg
69. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 69 May 2025 Document Tracking (DocTr) Framework ▪ Unified document (object) tracking solution ▪ plug-ins for QR codes, RFID tags, SIFT algorithm, … ▪ tracking of documents across organisational structures ▪ avoid fragmentation when using multiple tracking setups Document Management Pending Pool Data Model Identification Mechanism OC2 GUI REST Input Interface REST Output Interface Third-party Applications DocTr Delegator Tracking Setups
70. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 70 May 2025 SpeeG2: Multimodal Text Input architecture.wmf User Speech recognition (Microsoft SAPI 5.4) Skeletal tracking (Microsoft Kinect) 5 4 2 3 SpeeG2 GUI 6 1 Sven De Kock
71. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 71 May 2025 Video: SpeeG2 Performance (21 WPM) 25.03.jpg Sven De Kock
72. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 72 May 2025 INFEX: Cross-Device Information Exchange
73. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 73 May 2025 INFEX Architecture Smartphone HR Reader Camera Laptop ... Tangible Remote Device RFID QR Codes PixelSense Interface Lookup Service Bluetooth TCP/IP WebSockets FTP HTTP REST NFC PBAB List Inspect Transfer Input Handler Renderer Images PDF Contacts Tabletop Projector + Kinect Laptop HoloLens ...
74. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 74 May 2025 HCI and HCII in Smart Environments
75. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 75 May 2025 TCER Model
76. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 76 May 2025 Knowledge Physicalisation ▪ Innovative educational and office tools ▪ augmenting the scribbling process ▪ knowledge physicalisation via tangible objects ▪ Design guidelines for knowledge physicalisation and augmentation Innoty's Snnap Board solution Diagram Description automatically generated with low confidence
77. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 77 May 2025 Interactive Tables test2 itable test3 test4 Pen-Based Google Earth Browser Pen-Based Drawing Tool Photo Browser Photo Browser (Slider) Global Information Systems Group, ETH Zurich
78. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 78 May 2025 Non-Linear Cross-Media Narrative ▪ Writing/drawing as a colla- borative act of storytelling ▪ Captured handwriting is linked to existing digital and physical content ▪ Link server driving the augmentation process ▪ active components - ambient sound and light cosmo A red star with black text Description automatically generated In collaboration with Axel Vogelsang, Artists in Labs Programme
79. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 79 May 2025 ArtVis: TUI-based Information Exploration ▪ Advanced visualisation techniques in combination with a TUI ▪ explore Web Gallery of Art ▪ faceted browsing ▪ phidgets-based TUI ▪ RFID-tagged physical objects ▪ Three main components to explore, analyse and browse the information palette.jpg Bram Moerman
80. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 80 May 2025 ArtVis Setup artvis.JPG Bram Moerman
81. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 81 May 2025 Conclusions ▪ Innovative cross-media document formats that go beyond simulating paper on desktop computers ▪ RSL-based representation of open and fluid cross-media information spaces ▪ interactive paper and data physicalisation solutions ▪ personal information management and associative file systems ▪ MindXpres presentation tool ▪ Mixed reality and dynamic data physicalisation ▪ ArtVis, Mental Nomad and Tangible Holograms (TangHo) ▪ Multimodal and multi-touch interaction ▪ Midas/Mudra, HephaisTK and iGesture ▪ SpeeG2, Kinect Presenter
82. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 82 May 2025 Interested in a Thesis/PostDoc? ▪ Possibilities for MA and PhD theses ▪ Dynamic Data Physicalisation ▪ Personal Information Management (PIM) - next generation scholarly workflows ▪ Technology-enhanced Learning ▪ Innovative Mixed Reality Interfaces - paper-digital interaction - AR-based human-information interaction, … ▪ Next Generation Presentation Solutions (e.g. MindXpres) ▪ Human-AI Interaction and Digital Twin Solutions ▪ Hybrid Positioning and Implicit Human-Computer Interaction ▪ Smart Environments and Cross-Domain Internet of Things (IoT) ▪ ... ▪ Do you have your own ideas? Come along to discuss them ...
83. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 83 May 2025 References ▪ B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Second Edition, ISBN 978-3-8370-2713-6, August 2017 cover2.jpg B. Signer, Pen-based Interaction, Handbook of Human Computer Interaction, Major Reference Work, Springer Nature, 2025  https://beatsigner.com/publications/pen-based-interaction.pdf ▪ ArtVis Project ▪ https://www.beatsigner.com/flyers/ArtVis.pdf
84. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 84 May 2025 References ... ▪ MindXpres Project ▪ https://mindxpres.com ▪ OpenHPS ▪ https://openhps.org ▪ SemBeacon ▪ https://sembeacon.org ▪ Dynamic Data Physicalisation and Tangible Holograms ▪ https://beatsigner.com/dataPhysicalisation.html ▪ https://beatsigner.com/tangibleHolograms.html ▪ iGesture ▪ http://igesture.org
85. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 85 May 2025 References ... [1] B. Signer and M.C. Norrie, A Model and Architecture for Open Cross-Media Annotation and Link Services, Information Systems 36(3), Elsevier, May 2011 ▪ https://beatsigner.com/publications/signer_IS2011.pdf [2] B. Signer, What is Wrong with Digital Documents? A Conceptual Model for Structural Cross-Media Content Composition and Reuse, Proceedings of the 29th International Conference on Conceptual Modeling (ER 2010), Vancouver, Canada, November 2010 ▪ https://beatsigner.com/publications/signer_ER2010.pdf
86. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 86 May 2025 References ... [3] B. Signer, Fundamental Concepts for Interactive Paper and Cross-Media Information Spaces, Dissertation ETH No. 16218, Zurich, Switzerland, 2005 ▪ https://beatsigner.com/publications/signer2006.pdf [4] B. Signer and M.C. Norrie, As We May Link: A General Metamodel for Hypermedia Systems, Proceedings of ER 2007, 26th International Conference on Conceptual Modeling, Auckland, New Zealand, November 2007 ▪ https://beatsigner.com/publications/signer_ER2007.pdf
87. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 87 May 2025 References ... [5] S. Trullemans and B. Signer, Towards a Con- ceptual Framework and Metamodel for Context-Aware Personal Cross-Media Information Management Systems, Proceedings of ER 2014, 33rd International Conference on Conceptual Modelling, Atlanta, USA, October, 2014 ▪ https://beatsigner.com/publications/trullemans_ER2014.pdf [6] S. Trullemans, A. Vercruysse and B. Signer, DocTr: A Unifying Framework for Tracking Physical Docu- ments and Organisational Structures, Proceedings of EICS 2016, Brussels, Belgium, June 2016 ▪ https://beatsigner.com/publications/trullemans_EICS2016.pdf
88. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 88 May 2025 References ... [7] R. Roels, A. De Witte and B. Signer, INFEX: A Unifying Framework for Cross-Device Information Exploration and Exchange, Proceedings of the ACM on Human-Computer Interaction (PACMHCI), 1(3), 2017 ▪ https://beatsigner.com/publications/roels_PACMHCI2017.pdf [8] R. Roels and B. Signer, MindXpres: An Exten- sible Content-driven Cross-Media Presentation Platform, Proceedings of WISE 2014, 15th Interna- tional Conference on Web Information System Engineering, Thessaloniki, Greece, October, 2014 ▪ https://beatsigner.com/publications/roels_WISE2014.pdf
89. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 89 May 2025 References ... [9] A. Sanctorum and B. Signer, Towards User- defined Cross-Device Interaction, Proceedings of DUI 2016, 5th Workshop on Distributed User Interfaces, Lugano, Switzerland, June 2016 ▪ https://beatsigner.com/publications/sanctorum_DUI2016.pdf [10] S. Trullemans and B. Signer, A Multi-layered Context Modelling Approach for End Users, Expert Users and Programmers, Proceedings of SERVE 2016, International Workshop on Smart Ecosystems cReation by Visual dEsign, Bari, Italy, June 2016 ▪ https://beatsigner.com/publications/trullemans_SERVE2016.pdf
90. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 90 May 2025 References ... [11] R. Roels, Y. Baeten and B. Signer, An Inter- active Data Visualisation Approach for Next Generation Presentation Tools: Towards Rich Presentation-based Data Exploration and Storytelling, Proceedings of CSEDU 2016, Rome, Italy, April, 2016 ▪ https://beatsigner.com/publications/roels_CSEDU2016.pdf [12] M.C. Norrie, B. Signer and N. Weibel, Interactive Paper as a Reading Medium in Digital Libraries, Pro- ceedings of ECDL 2008, 12th European Conference on Research and Advanced Technology for Digital Libraries, Aarhus, Denmark, September 2008 ▪ https://beatsigner.com/publications/norrie_ECDL2008.pdf
91. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 91 May 2025 References ... [13] B. Signer, M.C. Norrie, P. Geissbuehler and D. Heiniger, Telephone Interface for Avalanche War- nings based on Information Server for Adaptable Con- tent Delivery, Proceedings of Pervasive 2002, Zurich, Switzerland, August 2002 ▪ https://beatsigner.com/publications/signer_Pervasive2002.pdf [14] S. Trullemans, A. Sanctorum and B. Signer, PimVis: Exploring and Re-finding Documents in Cross- Media Information Spaces, Proceedings of AVI 2016, International Working Conference on Advanced Visual Interfaces, Bari, Italy, June 2016 ▪ https://beatsigner.com/publications/trullemans_AVI2016.pdf
92. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 92 May 2025 References ... [15] B. Dumas, B. Moerman, S. Trullemans and B. Signer, ArtVis: Combining Advanced Visualisation and Tangible Interaction for the Exploration, Analysis and Browsing of Digital Artwork Collections, Proceed- ings of AVI 2014, Como, Italy, May 2014 ▪ https://beatsigner.com/publications/dumas_AVI2014.pdf [16] B. Dumas, T. Broché, L. Hoste and B. Signer, ViDaX: An Interactive Semantic Data Visualisation and Exploration Tool, Proceedings of AVI 2012, Internatio- nal Working Conference on Advanced Visual Inter- faces, Capri Island, Italy, May 2012 ▪ https://beatsigner.com/publications/dumas_AVI2012.pdf
93. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 93 May 2025 References ... [17] Moira C. Norrie, Beat Signer and Nadir Weibel, Print-n-Link: Weaving the Paper Web, Proceedings of DocEng 2006, ACM Symposium on Document Engineering, Amsterdam, The Netherlands, October 2006 ▪ https://beatsigner.com/publications/norrie_DocEng2006.pdf [18] M. Van de Wynckel and B. Signer, Indoor Positioning Using the OpenHPS Framework, Proceedings of IPIN 2021, International Conference on Indoor Positioning and Indoor Navigation, Lloret de Mar, Spain, November 2021 ▪ https://beatsigner.com/publications/vanDeWynckel_IPIN2021.pdf
94. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 94 May 2025 References ... [19] R. Roels, P. Mestereaga and B. Signer, An Interactive Source Code Visualisation Plug-in for the MindXpres Presentation Platform, Communications in Computer and Information Science (CCIS), 583, 2016 ▪ https://beatsigner.com/publications/roels_CCIS2016.pdf [20] Y. Malaise and B. Signer, Personalised Learning Environments Based on Knowledge Graphs and the Zone of Proximal Development, Proceedings of CSEDU 2022, International Conference on Computer Supported Education, Online Event, April 2022 ▪ https://beatsigner.com/publications/malaise_CSEDU2022.pdf
95. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 95 May 2025 References ... [21] C. Scholiers, L. Hoste, B. Signer and W. De Meuter, Midas: A Declarative Multi-Touch Interaction Framework, Proceedings of TEI 2011, 5th Interna- tional Conference on Tangible, Embedded and Em- bodied Interaction, Funchal, Portugal, January 2006 ▪ https://beatsigner.com/publications/scholliers_TEI2011.pdf [22] L. Hoste, B. Dumas and B. Signer, Mudra: A Unified Multimodal Interaction Framework, Proceedings of ICMI 2011, 13th International Conference on Multi- modal Interaction, Alicante, Spain, November 2011 ▪ https://beatsigner.com/publications/hoste_ICMI2011.pdf
96. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 96 May 2025 References ... [23] B. Signer, U. Kurmann and M.C. Norrie, iGesture: A General Gesture Recognition Framework, Proceedings of ICDAR 2007, 9th International Conference on Document Analysis and Recognition, Curitiba, Brazil, September 2007 ▪ https://beatsigner.com/publications/signer_ICDAR2007.pdf [24] L. Hoste and B. Signer, SpeeG2: A Speech- and Gesture-based Interface for Efficient Controller-free Text Entry, Proceedings of ICMI 2013, 15th Inter- national Conference on Multimodal Interaction, Sydney, Australia, December 2013 ▪ https://beatsigner.com/publications/hoste_ICMI2013.pdf
97. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 97 May 2025 References ... [25] B. Signer, M. Grossniklaus and M.C. Norrie, Interactive Paper as a Mobile Client for a Multi- Channel Web Information System, World Wide Web Journal (WWW), Vol. 10, No. 4, Springer, December 2007 ▪ https://beatsigner.com/publications/signer_WWWJ2007.pdf [26] B. Signer and M.C. Norrie, PaperPoint: A Paper- based Presentation and Interactive Paper Prototyping Tool, Proceedings of TEI 2007, First International Conference on Tangible and Embedded Interaction, Baton Rouge, USA, February 2007 ▪ https://beatsigner.com/publications/signer_TEI2007.pdf
98. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 98 May 2025 References ... [27] A. Vogelsang and B. Signer, The Lost Cosmo- naut: An Interactive Narrative Environment on Basis of Digitally Enhanced Paper, Proceedings of the Inter- national Conference on Virtual Storytelling 2005, Strasbourg, France, December 2005 ▪ https://beatsigner.com/publications/vogelsang_ICVS2005.pdf [28] L. Hoste and B. Signer, Expressive Control of Indirect Augmented Reality During Live Music Performances, Proceedings of NIME 2013, 13th International Confe- rence on New Interfaces for Musical Expression, Daejeon, Korea Republic, May 2013 ▪ https://beatsigner.com/publications/hoste_NIME2013.pdf
99. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 99 May 2025 References ... [29] N. Weibel, A. Ispas, B. Signer and M.C. Norrie, PaperProof: A Paper-Digital Proof-Editing System, Proceedings of CHI 2008, 26th International Confe- rence on Human Factors in Computing Systems (Interactivity Track), Florence, Italy, April 2008 ▪ https://beatsigner.com/publications/weibel_CHI2008.pdf [30] N. Weibel, B. Signer, M.C. Norrie, H. Hofstetter, H.-C. Jetter and H. Reiterer, PaperSketch: A Paper- Digital Collaborative Remote Sketching Tool, Proceed- ings of IUI 2011, International Conference on Intelli- gent User Interfaces, Palo Alto, USA, February 2011 ▪ https://beatsigner.com/publications/weibel_IUI2011.pdf
100. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 100 May 2025 References ... [31] B. Signer and T.J. Curtin, Tangible Holograms: Towards Mobile Physical Augmentation of Virtual Objects, Technical Report WISE Lab, WISE-2017-01, March 2017 ▪ https://beatsigner.com/publications/signer_arXiv2017.pdf [32] B. Signer, P. Ebrahimi, T.J. Curtin and Ahmed K.A. Abdullah, Towards a Framework for Dynamic Data Physicalisation, International Workshop Toward a Design Language for Data Physicalisation, Berlin, Germany, October 2018 ▪ https://beatsigner.com/publications/signer_DataPhys2018.pdf
101. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 101 May 2025 References ... [33] M. Van de Wynckel and B. Signer, A Solid-based Architecture for Decentralised Interoperable Location Data, Proceedings of IPIN 2022, International Conference on Indoor Positioning and Indoor Navigation, Beijing, China, September 2022 ▪ https://beatsigner.com/publications/vanDeWynckel_IPIN2022.pdf [34] M. Van de Wynckel and B. Signer, POSO: A Generic Positioning System Ontology, Proceedings of ISWC 2022, International Semantic Web Conference, Hangzhou, China, October 2022 ▪ https://beatsigner.com/publications/vanDeWynckel_ISWC2022.pdf
102. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 102 May 2025 References ... [35] M. Van de Wynckel and B. Signer, SemBeacon: A Semantic Proximity Beacon Solution for Discovering and Detecting the Position of Physical Things, Proceedings of IoT 2023, International Conference on the Internet of Things, Nagoya, Japan, November 2023 ▪ https://beatsigner.com/publications/vanDeWynckel_IOT2023.pdf [36] M. Van de Wynckel, I. Valadez and B. Signer, FidMark: A Fiducial Marker Ontology for Semantically Describing Visual Markers, Proceedings ESWC 2024, Interna- tional Conference on the Semantic Web Conference, Hersonissos, Greece, May 2024 ▪ https://beatsigner.com/publications/vanDeWynckel_ESWC2024.pdf
103. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 103 May 2025 References ... [37] B. Signer and M.C. Norrie, Interactive Paper: Past, Present and Future, Proceedings of PaperComp 2010, 1st International Workshop on Paper Com- puting, Copenhagen Denmark, September 2010 ▪ https://beatsigner.com/publications/signer_PaperComp2010.pdf [38] A.A.O Tayeh and B. Signer, A Dynamically Extensible Open Cross-Document Link Service, Proceedings of WISE 2015, 16th International Conference on Web Information Systems Engineering, Miami, USA, November, 2015 ▪ https://beatsigner.com/publications/tayeh_WISE2015.pdf
104. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 104 May 2025 References ... [39] S. Trullemans and B. Signer, From User Needs to Opportunities in Personal Information Management: A Case Study on Organisational Strategies in Cross- Media Information Spaces, Proceedings of DL 2014, London, UK, September, 2014 ▪ https://beatsigner.com/publications/trullemans_DL2014.pdf [40] R. Roels, C. Vermeylen and B. Signer, A Unified Communication Platform for Enriching and Enhancing Presentations with Active Learning Components, Pro- ceedings of ICALT 2014, Athens, Greece, July 2014 ▪ https://beatsigner.com/publications/roels_ICALT2014.pdf
105. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 105 May 2025 References ... [41] A. Ispas, H. Schuldt, M.C. Norrie and B. Signer, Towards Query by Sketch, Michael Springmann, Second DELOS Conference on Digital Libraries, Pisa, Italy, December 2007 ▪ https://beatsigner.com/publications/springmann_DELOS2007.pdf [42] B. Signer, Towards Cross-Media Information Spaces and Architectures, Proceedings of RCIS 2019, Brussels, Belgium, May 2019 ▪ https://beatsigner.com/publications/signer_RCIS2019.pdf
106. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 106 May 2025 References ... [43] B. Signer, R. Roels, R. van Barlingen and B. Willems, Back to the Future: Bringing Original Hypermedia and Cross-Media Concepts to Modern Desktop Environments, Proceedings of Hypertext 2021, Virtual Event, August 2021 ▪ https://beatsigner.com/publications/signer_Hypertext2021.pdf [44] E. Cole, Y. Malaise, and B. Signer, Computing Edu- cation Research as a Translational Transdiscipline, Proceedings of SIGCSE 2023, Toronto, Canada, March 2023 ▪ https://beatsigner.com/publications/cole_SIGCSE2023.pdf
107. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 107 May 2025 References ... [45] X. Zhang, M. Ilić and B. Signer, A Modular and Extensible Hardware Platform Prototype for Dynamic Data Physicalisation, WISE-2025-01, arXiv preprint, March 2025 ▪ https://beatsigner.com/publications/xuyao_CoRR2025.pdf [46] I. Valadez, S. Trullemans and B. Signer, Where Do We Meet? Key Factors Influencing Collaboration Across Meeting Spaces, WISE-2023-02, arXiv preprint, November 2023 ▪ https://beatsigner.com/publications/valadez_WISE-2023-02.pdf
108. Beat Signer - Department of Computer Science - bsigner@vub.ac.be 108 May 2025 References ... [47] P. Van Der Paelt, B. Lauwens and B. Signer, Seamless Data Persistence in Simulation Models: A Metaprogramming Approach in Julia, Proceedings of SIMUtools 2024, International Conference on Simulation Tools and Techniques, Bratislava, Slovakia, December 2024 ▪ https://beatsigner.com/publications/vanDerPaelt_SIMUtools2024.pdf [48] B. Signer, As We May Interact: Challenges and Opportunities for Next-Generation Human-Information Interaction, Proceedings of HUMAN 2024, Poznań, Poland, September 2024 ▪ https://beatsigner.com/publications/signer_HUMAN2024.pdf