Share
Embed
Meta-Augmented Human: Augmenting the Capability of Augmenting the Capability / 2022-12-07
My talk at the Dagstuhl Seminar Cognitive Augmentation on 07.12.2022.
Shoya Ishimaru
December 07, 2022
Transcript
39 slides
-
.FUB"VHNFOUFE)VNBO Meta-skill augmentation Skill augmentation • .PTUBVHNFOUJOHIVNBOUFDIOPMPHJFTIFMQBVTFSXIJMFUIFZBSFXPSO • *GXFBVHNFOUUIFTLJMMPGTLJMMBDRVJTJUJPO JF
NFUBTLJMM UIFBDRVJSFETLJMMIFMQTUIF VTFSJOBMPOHUFSN 4LJMMBDRVJTJUJPOMFBSOJOH TQPSUTUSBJOJOH NVTJDQSBDUJDF FUD • 5IJTUBMLJOUSPEVDFTTPNFTUVEJFTBQQMJDBUJPOTUIBUSFDPHOJ[FJOUFSOBMTUBUFTXIJMF MFBSOJOHCZNBDIJOFMFBOSJOHBOEHJWFBQQSPQSJBUFGFFECBDLCBTFEPOUIFDPOUFYU -
Physical Activity walking, standing, cycling, sleeping, ... Iceberg Model of
Activity Recognition -
Physical Activity walking, standing, cycling, sleeping, ... Cognitive Activity reading,
writing, memorizing, talking, ... Iceberg Model of Activity Recognition -
Physical Activity walking, standing, cycling, sleeping, ... Cognitive Activity reading,
writing, memorizing, talking, ... Iceberg Model of Activity Recognition Cognitive and A ff ective State interest, workload, con fi dence, fatigue, ... -
"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF -FDUVSFBOEUFYUCPPL 2VFTUJPOT UFTU %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF
$BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ -
"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF -FDUVSFBOEUFYUCPPL 2VFTUJPOT UFTU 8IJDIRVFTUJPOTIPVMECFSFWJFXFE DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT
BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ %BJMZIBCJUTBOEDVSJPTJUZ -
Sleeping Walking Attending a lecture Reading a book Watching a
movie Eating 6OEFSTUBOEJOH%BJMZ"DUJWJUJFT GSPN1IZTJDBMUP$PHOJUJWF 8FBSBCMFEFWJDF<#VMMJOH FUBM> 4NBSUFZFXFBS $PNNFSDJBMMZBWBJMBCMFBDUJWJUZUSBDLFST 4UVEFOUTEBJMZBDUJWJUJFT A. Bulling, et al. Eye movement analysis for activity recognition using electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence, 33.4 (2010): 741-753. J. Steil & A. Bulling. Discovery of everyday human activities from long-term visual behaviour using topic models. Proc. UbiComp 2015, pp. 75–85, 2015. S. Ishimaru, et al. Smart Eyewear for Interaction and Activity Recognition. CHI 2015 Extended Abstracts, pp. 307–310, 2015. ɾɾɾ -
+*/4.&.&&MFDUSPPDVMPHSBQIZ &0( (MBTTFT Electrooculography # 3 - Electrodes Horizontal axis:
L - R [mV] Vertical axis: B - (L + R)/2 [mV] -
8PSEPNFUFS&TUJNBUJOHUIFOVNCFSPGSFBEXPSET 2VBOUJGZJOHSFBEJOHTBNFBTXBMLJOH *OUFSWFOUJPO read! Nice condition. Read one paragraph more!
There are interesting articles for you. S. Ishimaru, et al. The Wordometer 2.0: Estimating the Number of Words You Read in Real Life using Commercial EOG Glasses. Proc. UbiComp 2016 Adjunct, pp. 293–296, 2016. S. Ishimaru, et al. Reading Interventions: Tracking Reading State and Designing Interventions. Proc. UbiComp 2016 Adjunct, pp. 1759–1764, 2016. -
&0( )FBENPUJPOCBTFE"DUJWJUZ$MBTTJ fi DBUJPO 69% 23% 3% 10% 18%
66% 1% 0% 0% 0% 77% 14% 12% 9% 18% 74% Predicted class reading typing eating talking Actual class talking eating typing reading 78% 36% 0% 12% 8% 19% 6% 37% 0% 1% 93% 7% 12% 42% 0% 41% Predicted class reading typing eating talking Actual class talking eating typing reading EOG (Acc.: 54%) majority vote -> 72 % EOG and head (Acc.: 72%) majority vote -> 100 % • 5IFSFXFSFGSFRVFOUDIBSBDUFSJTUJD QBUUFSOTPO&0(XIJMFSFBEJOHPS FBUJOH&0(CBTFEBQQSPBDIDPVME DMBTTJGZUIFN 4UVEZ&0(#BTFE"DUJWJUZ$MBTTJ fi DBUJPO S. Ishimaru, et al. “Smarter Eyewear: Using Commercial EOG Glasses for Activity Recognition”. Proc. UbiComp '14 Adjunct, pp. 239–242, 2014. -
&0( )FBENPUJPOCBTFE"DUJWJUZ$MBTTJ fi DBUJPO 69% 23% 3% 10% 18%
66% 1% 0% 0% 0% 77% 14% 12% 9% 18% 74% Predicted class reading typing eating talking Actual class talking eating typing reading 78% 36% 0% 12% 8% 19% 6% 37% 0% 1% 93% 7% 12% 42% 0% 41% Predicted class reading typing eating talking Actual class talking eating typing reading EOG (Acc.: 54%) majority vote -> 72 % EOG and head (Acc.: 72%) majority vote -> 100 % • 5IFSFXFSFGSFRVFOUDIBSBDUFSJTUJD QBUUFSOTPO&0(XIJMFSFBEJOHPS FBUJOH&0(CBTFEBQQSPBDIDPVME DMBTTJGZUIFN • 5ZQJOHBOEUBMLJOHXFSFPGUFONJT DMBTTJ fi FEJOUIF&0(CBTFEBQQSPBDI TJNJMBSFZFNPWFQBUUFSOT #VUUIF DPNCJOBUJPOXJUIIFBENPUJPOTPMWFE UIJTQSPCMFN 4UVEZ&0(#BTFE"DUJWJUZ$MBTTJ fi DBUJPO S. Ishimaru, et al. “Smarter Eyewear: Using Commercial EOG Glasses for Activity Recognition”. Proc. UbiComp '14 Adjunct, pp. 239–242, 2014. -
• 'PVSGFBUVSFT UIFOVNCFSTPGGPSXBSECBDLXBSETBDDBEFT NFBO&0(XIJMFUIFTBDDBEFT DPVMEFTUJNBUFUIFOVNCFSPGSFBEXPSET • 8PSEDPVOUFTUJNBUJPOFSSPSJOVTFSEFQFOEFOU JOVTFSJOEFQFOEFOUBQQSPBDI FWBMVBUFEJOBOFYQFSJNFOUXJUIQBSUJDJQBOUTSFBEEPDVNFOUT
8PSEPNFUFS4ZTUFN0WFSWJFX Recording EOG signals Detecting forward/backward-saccades Support Vector Regression 4UVEZ&0(#BTFE8PSE$PVOU&TUJNBUJPO S. Ishimaru, et al. "The wordometer 2.0: estimating the number of words you read in real life using commercial EOG glasses." UbiComp '16 Adjunct, pp. 293–296, 2016. -
"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF 2VFTUJPOT UFTU %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT
BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ -FDUVSFBOEUFYUCPPL -
4FOTPS#BTFE$PHOJUJWF4UBUF3FDPHOJUJPO *OUFSFTU $PNQSFIFOTJPO $PHOJUJWFMPBE &ZFUSBDLFS 5IFSNBMDBNFSB 4BDDBEFT 'BDFUFNQFSBUVSF 'JYBUJPOT S.
Ishimaru, et al. Cognitive State Measurement on Learning Materials by Utilizing Eye Tracker and Thermal Camera. Proc. ICDAR HDI 2017, pp. 32–36, 2017. S. Ishimaru, et al. Augmented Learning on Anticipating Textbooks with Eye Tracking. Positive Learning in the Age of Information (PLATO), pp. 387–398, 2018. -
"UUFOUJPO"OBMZTJT intro. def. appl. intro. def. appl. intro. def.
appl. intro. def. appl. intro. def. appl. intro. def. appl. Expert Intermediate Novice reading solving reading solving reading solving B SFBEJOH C TPMWJOH F SFBEJOH G TPMWJOH /PWJDF D SFBEJOH E TPMWJOH *OUFSNFEJBUF &YQFSU Introduction De nitions Applications (a) text $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT -
$PNQSFIFOTJPO3FDPHOJUJPO 'FBUVSFSFQSFTFOUBUJPOPGBMMQBSUJDJQBOUTEBUB JOTVCTFRVFODFT NJO CBTFEBQQSPBDI • BDDVSBDZGPSDMBTTJGZJOH
FYQFSUJTFTJOUPUISFFDMBTTFT VTFSJOEFQFOEFOUUSBJOJOH $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398. -
$PNQSFIFOTJPO3FDPHOJUJPO 'FBUVSFSFQSFTFOUBUJPOPGBMMQBSUJDJQBOUTEBUB JOTVCTFRVFODFT NJO CBTFEBQQSPBDI • BDDVSBDZGPSDMBTTJGZJOH
FYQFSUJTFTJOUPUISFFDMBTTFT VTFSJOEFQFOEFOUUSBJOJOH • /PWJDFTUVEFOUTSFBEB UFYUCPPLTMPXMZXJUITNBMMTUFQT $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398. -
$PNQSFIFOTJPO3FDPHOJUJPO 'FBUVSFSFQSFTFOUBUJPOPGBMMQBSUJDJQBOUTEBUB JOTVCTFRVFODFT NJO CBTFEBQQSPBDI • BDDVSBDZGPSDMBTTJGZJOH
FYQFSUJTFTJOUPUISFFDMBTTFT VTFSJOEFQFOEFOUUSBJOJOH • /PWJDFTUVEFOUTSFBEB UFYUCPPLTMPXMZXJUITNBMMTUFQT • &YQFSUTUVEFOUTTPNFUJNFTTLJQ UIFJSFZFTPOUIFUFYUBOEGPDVTPO UIFDPOUFOUUIFZBSFJOUFSFTUFEJO $PHOJUJWFBOE" ff FDUJWF4UBUF3FDPHOJUJPO 4UVEZ(B[F#BTFE-FBSOJOH"OBMZUJDT S. Ishimaru, et al. "Augmented learning on anticipating textbooks with eye tracking." Positive Learning in the Age of Information. Springer VS, Wiesbaden, 2018. 387-398. -
&YQFSJNFOUBM%FTJHO • VOJWFSTJUZTUVEFOUTSFBEOFXTQBQFSBSUJDMFT ʶXPSET • %FWJDF4.*3&%OTDJFOUJ fi D)[FZFUSBDLFS
• $BMJCSBUFUIFEFWJDFˠ3FTU TFD ˠ3FBEBEPDVNFOUˠ"OTXFSTVSWFZT 4UVEZ(B[F#BTFE*OUFSFTU%FUFDUJPO S. Jacob, et al. "Gaze-based interest detection on newspaper articles." Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. 2018. -
*OUFSFTU3FDPHOJUJPO0WFSWJFXBOE3FTVMU 'PSXBSETBDDBEF 3FHSFTTJPO 'FBUVSFDBMDVMBUJPO 47. 4UVEZ(B[F#BTFE*OUFSFTU%FUFDUJPO + ongoing work
S. Jacob, et al. "Gaze-based interest detection on newspaper articles." Proceedings of the 7th Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. 2018. NFBO NFBO 62% 16% 2% 1% 25% 66% 7% 0% 6% 15% 83% 6% 7% 3% 8% 93% Predicted class 1 2 3 4 Actual class 4 3 2 1 *OVTFSEFQFOEFOUFWBMVBUJPO MFBWFPOFEPDVNFOUPVUDSPTTWBMJEBUJPOJOBQBSUJDJQBOU 47.DMBTTJ fi FEJOUFSFTUTJOUPMFWFMTXJUIBDDVSBDZ $//-45.DMBTTJ fi FEXJUIBDD $//-45. -
"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF -FDUVSFBOEUFYUCPPL %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT
BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ 2VFTUJPOT UFTU -
$P"-"$PO fi EFODF"XBSF-FBSOJOH"TTJTUBOU $PSSFDU *ODPSSFDU $PO fi EFOU 6ODPO fi
EFOU .VMUJQMF$IPJDF2VFTUJPO -
(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO %FDJTJPOXJUIDPO fi EFODF %FDJTJPOXJUIPVUDPO fi EFODF S.
Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts, p. 8, 2020. S. Ishimaru, et al. Confidence-Aware Learning Assistant. In arXiv preprint arXiv:2102.07312, 2021 -
4ZTUFN0WFSWJFX 1SFQSPDFTTJOH fi YBUJPOTBDDBEFEFUFDUJPO 'FBUVSFDBMDVMBUJPO $MBTTJ fi DBUJPO
8FVUJMJ[FUIFGPMMPXJOH fi WFF ff FDUJWFGFBUVSFTTFMFDUFECZ'PSXBSETUFQXJTFGFBUVSFTFMFDUJPO GTVNPG fi YBUJPOEVSBUJPOTPO$IPJDFT GWBSJBODFPGYDPPSEJOBUFPG fi YBUJPOT GUIFOVNCFSPGTBDDBEFTCFUXFFODIPJDFBSFBT GTVNPGTBDDBEFEVSBUJPOT GSFBEJOHUJNF • #JOBSZDMBTTJ fi DBUJPO • 4VQQPSU7FDUPS .BDIJOF 47. • 3#'LFSOFM • $ Ѝ 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020. -
3FTFBSDI)ZQPUIFTFT 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO • 3)$PSSFDUBOTXFSTXJUIPVUDPO fi EFODF WBHVFLOPXMFEHF
UFOEUP CFJODPSSFDUMZBOTXFSFEXIFOBTLFEBHBJODPNQBSFEUPDPSSFDU BOTXFSXJUIDPO fi EFODF • 3)*ODPSSFDUBOTXFSTXJUIDPO fi EFODF NJTVOEFSTUBOEJOH UFOEUP CFNJTUBLFOBHBJODPNQBSFEUPJODPSSFDUBOTXFSTXJUIPVUDPO fi EFODF • 3)&TUJNBUJOHTFMGDPO fi EFODFGSPNMFBSOJOHCFIBWJPSTBOEHJWJOH GFFECBDL FH BEEJOHRVFTUJPOTUPBSFWJFXMJTU IJHIMJHIUJOHUIFN XIJMFSFWJFXJOH BWPJETUIFTDFOBSJPTPVUMJOFEJO3)BOE3) S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020. -
&YQFSJNFOUBM%FTJHO 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO • BOE B )PXPGUFOBOTXFSTXJUIPVUDPO
fi EFODFBSFGPSHPUUFO • B BOE C )PX$P"-"JTF ff FDUJWFJOBTVDIDPOEJUJPO • BOE B )PXPGUFOBOTXFSTXJUIDPO fi EFODFBSFJODPSSFDUMZBOTXFSFEBHBJO • B BOE C )PX$P"-"JTF ff FDUJWFJOBTVDIDPOEJUJPO $PNQBSJTPOPG -
&YQFSJNFOUBM3FTVMUT 4UVEZ(B[F#BTFE4FMG$PO fi EFODF&TUJNBUJPO • 3)$PSSFDUBOTXFSTXJUIPVUDPO fi EFODFUFOEUPCFJODPSSFDUMZ BOTXFSFEXIFOBTLFEBHBJODPNQBSFEUPBOTXFSXJUIDPO
fi EFODF • 4VQQPSUFE B ESPQQFEDPNQBSFEUP Q • 3)*ODPSSFDUBOTXFSTXJUIDPO fi EFODFUFOEUPCFNJTUBLFOBHBJO DPNQBSFEUPJODPSSFDUBOTXFSTXJUIPVUDPO fi EFODF • /PUTVQQPSUFE/PTJHOJ fi DBOUEJ ff FSFODFCFUXFFO BOE B • 3)&TUJNBUJOHTFMGDPO fi EFODFGSPNMFBSOJOHCFIBWJPSTBOEHJWJOH GFFECBDL FH BEEJOHRVFTUJPOTUPBSFWJFXMJTU IJHIMJHIUJOHUIFN XIJMFSFWJFXJOH BWPJETTVDITDFOBSJPT • 4VQQPSUFE$PNQBSJTPOTCFUXFFO B WT C B WT C B C DPSSXDPOG B DPSSXPDPOG C DPSSXPDPOGGFFECBDL B C JODPSSXPDPOG B JODPSXDPOG C JODPSSXDPOGGFFECBDL S. Ishimaru, et al. Gaze-Based Self-Confidence Estimation on Multiple-Choice Questions and Its Feedback. CHI 2015 Extended Abstracts (Asian CHI Symposium), p. 8, 2020. -
"VHNFOUFE1PTJUJWF-FBSOJOH$ZDMF 8PSEPNFUFS )ZQFS.JOE $P"-" -FDUVSFBOEUFYUCPPL 2VFTUJPOT UFTU %BJMZIBCJUTBOEDVSJPTJUZ 8IJDIRVFTUJPOTIPVMECFSFWJFXFE
DBSFGVMMZ DPOTJEFSJOHQSJPSJUZMFBLBHF $BO"*VOEFSTUBOEJOUFSFTUPGTUVEFOUT BOEOVEHFUIFNUPMFBSOWPMVOUBSJMZ