Addressing the music information needs of musicologists

The music information needs of musicologists are not being met by the current generation of MIR tools and techniques. While evaluation has always been central to the practice of the music information retrieval community, the tasks tackled most often address the music information needs of recreational users, such as playlist recommendation systems; or are specified at a level which is not very relevant to the needs of music researchers, such as beat or key finding; or have focused on--and possibly even become over-fitted to--a narrow range of musical repertoire which doesn't cover musicological interests. In this tutorial we will present those music information needs through topics including at least the following: the metadata requirements of historical musicology; working with symbolic corpora; studying musical networks; passage-level audio search; and musical understandings of audio features. As as well as these scheduled presentations and discussions, we will ask the attendees to submit suggestions of musicologically motivated research questions suitable for MIR during the course of the tutorial. These will then be reviewed and discussed during the conclusion of the tutorial. Finally, we have invited Meinard Müller to conclude the tutorial by outlining his view on the current state of MIR for musicology. We are aiming to enable attendees, as experts in their own areas of MIR, to find new applications of their tools and techniques that can also serve the needs of musicologists. Given the selection of MIR topics we intend to cover, this tutorial will be of particular interest to those working in: musical metadata; symbolic MIR; audio search; and graph analytics. We believe contemporary musicology to be a rich source of new and exciting challenges for MIR and we are confident the community can rise to those challenges. In the long term, we hope this tutorial will give rise to a selection of new MIREX tasks that focus on musicological challenges.

Presented/written with Richard J. Lewis, Tim Crawford, Kevin Page, David Weigl, Meinard Mueller, David Lewis, Christophe Rhodes, Justin Gagen

Transcript

1. "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 5SBOTGPSNJOH .VTJDPMPHZ .POEBZ  0DUPCFS  .¡MBHB 5SBOTGPSNJOH .VTJDPMPHZ .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT   
2. 0VUMJOF 5JNF %FTDSJQUJPO  *OUSPEVDUJPO "MBO .BSTEFO  .FUBEBUB 3JDIBSE -FXJT %BWJE 8FJHM  /FUXPSLT #FO 'JFMET  1FSGPSNBODFSFBMUJNF ,FWJO 1BHF %BWJE 8FJHM  #SFBL  4ZNCPMJD NVTJD 5JN $SBXGPSE 3JDIBSE -FXJT  "VEJP TFBSDI #FO 'JFMET 5JN $SBXGPSE  .FJOBSE .¼MMFS  4USVDUVSFE EJTDVTTJPO  'JOJTI 5SBOTGPSNJOH .VTJDPMPHZ .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT   
3. TR A N S FO R M I N G M U S I C O LO G Y An AHRC Digital Transformations project Addressing the Music Information Needs of Musicologists Introduction ISMIR 2015
4. TR A N S FO R M I N G M U S I C O LO G Y An AHRC Digital Transformations project • Musicology is the study of music – vast range of what is meant by ‘music’ – vast range of what is meant by ‘study’ • Classic divisions – Historical Musicology (Am. ‘Musicology’) – Systematic Musicology (Am. ‘Music Theory’) – Ethnomusicology now perhaps of diminishing significance • See the tutorial ‘Musicology’ by Frans Wiering and Anya Volk at ISMIR 2011 – http://ismir2011.ismir.net/tutorials/ISMIR2011-Tutorial-Musicology.pdf 2 ISMIR, Malaga 26 October 2015 Musicology
5. TR A N S FO R M I N G M U S I C O LO G Y An AHRC Digital Transformations project • The sound of music (or at least things directly connected with the sound) – old music – current art music, current popular music – possible music • Musical activities – composing – performing – perception • Music’s role and function – music in our society – music in other societies • Data – scores – recordings – instruments – documents (letters, etc.) – interviews 3 ISMIR, Malaga 26 October 2015 What musicologists study
6. TR A N S FO R M I N G M U S I C O LO G Y An AHRC Digital Transformations project • Collation – relation of items to one another – categorisation, etc. • Close examination – verification of details – reconstruction – interpretation • Theory formation – ... but not much testing • Communication/disputation among themselves 4 ISMIR, Malaga 26 October 2015 How musicologists study
7. TR A N S FO R M I N G M U S I C O LO G Y An AHRC Digital Transformations project • To identify value • To establish authoritative versions – but the very idea of authoritative versions is subject to challenge! • To establish principles and trends • To influence thinking about music 5 ISMIR, Malaga 26 October 2015 Why musicologists study
8. TR A N S FO R M I N G M U S I C O LO G Y An AHRC Digital Transformations project MIR and Musicology can inform and interrogate each other, e.g. • Genre: – From MIR for Musicology: Why are some genres closely definable by audio features and some are not? – From Musicology for MIR: Are genres not just a marketing device with little musical benefit? • Audio transcription: – From MIR for Musicology: If the notes are so poorly represented in the audio signal, do they really matter so much? – From Musicology for MIR: Concentrate on the important notes! 6 ISMIR, Malaga 26 October 2015 MIR and musicology – cross-fertilisation
9. TR A N S FO R M I N G M U S I C O LO G Y An AHRC Digital Transformations project • Different concerns about accuracy – MIR: is processing and analysis valid? – Musicology: are the sources correct? • Different criteria of validity – MIR: test against ground truth – Musicology: test against agreement • Different notions of basic material: – MIR: definitive recordings, the ‘song’, feature sets – Musicology: fluid notion of ‘a piece’, the ‘passage’ (i.e., part of a piece), complex metadata 7 ISMIR, Malaga 26 October 2015 MIR and musicology – dislocations
10. .*3 GPS NVTJDPMPHZ ٔ 'JWF UPQJDT ٔ &BDI QSFTFOUT TPNF .*3 UFDIOJRVFT ٔ "OE TIPXT IPX UIPTF UFDIOJRVFT NBZ CF BQQMJFE UP NVTJDPMPHJDBM SFTFBSDI RVFTUJPOT ٔ 8F BJN UP HJWF TPNF JEFBT PG OFX BQQMJDBUJPOT ZPV DBO FYQMPSF XJUI ZPVS FYJTUJOH UFDIOJRVFT 5SBOTGPSNJOH .VTJDPMPHZ .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT   
11. 4USVDUVSFE EJTDVTTJPO ٔ -BTU IBMGIPVS JT B TUSVDUVSFE EJTDVTTJPO TFTTJPO ٔ %JWJEF JOUP QBJST BOE EJTDVTT ZPVS .*3 UFDIOJRVFT BOE DVSSFOU BQQMJDBUJPOT ٔ 5IFO TVHHFTU QPTTJCMF NVTJDPMPHJDBM BQQMJDBUJPOT PG FBDI PUIFSًT UFDIOJRVFT ٔ +PJO JOUP MBSHFS HSPVQT  PS  QSFTFOU FBDI PUIFST UFDIOJRVFT BOE QSPQPTFE NVTJDPMPHJDBM BQQMJDBUJPOT ٔ %FDJEF PO UXP PS UISFF NVTJDPMPHJDBM BQQMJDBUJPOT UP EJTDVTT ٔ &BDI HSPVQ UIFO QSFTFOUT UIPTF BQQMJDBUJPOT BU UIF FOE 5SBOTGPSNJOH .VTJDPMPHZ .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT   
12. 0VUMJOF 5JNF %FTDSJQUJPO  *OUSPEVDUJPO "MBO .BSTEFO  .FUBEBUB 3JDIBSE -FXJT %BWJE 8FJHM  /FUXPSLT #FO 'JFMET  1FSGPSNBODFSFBMUJNF ,FWJO 1BHF %BWJE 8FJHM  #SFBL  4ZNCPMJD NVTJD 5JN $SBXGPSE 3JDIBSE -FXJT  "VEJP TFBSDI #FO 'JFMET 5JN $SBXGPSE  .FJOBSE .¼MMFS  4USVDUVSFE EJTDVTTJPO  'JOJTI ?iiT,ffrrrXi@KmbXQ`;f2p2MibfBbKB`kyR8@imiQ`BHf 5SBOTGPSNJOH .VTJDPMPHZ .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT   
13. "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
14. 8IBU JT NFUBEBUB ٔ .VTJDPMPHZ JT NPTUMZ BCPVU UIF DPOUFYU PG NVTJDBM QSBDUJDF ٔ .FUBEBUB JT UIFSFGPSF KVTU BT JNQPSUBOU BT EBUB ٔ FH IJTUPSJDBM NVTJDPMPHZ TUVEZJOH IPX QJFDFT NPWF UISPVHI TPDJBM OFUXPSLT ٔ FH DVMUVSBM NVTJDPMPHJTUT TUVEZJOH IPX DVMUVSBM HSPVQT JEFOUJGZ XJUI NVTJD ٔ 5IFTF RVFTUJPOT EPOًU SFMZ PO NVTJDBM DPOUFOU ٔ #VU UIFZ EP SFMZ PO ٔ JEFOUJUZ PG NVTJDBM XPSLT ٔ XIFO BOE XIFSF NVTJDBM XPSLT BSF QFSGPSNFE ٔ XIFO BOE XIFSF NVTJDBM XPSLT BSF QVCMJTIFE DPQJFE PS NFOUJPOFE ٔ 5IFSF JT NPSF NVTJDBM LOPXMFEHF JO NFUBEBUB UIBO JO DPOUFOU 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
15. 3FTFBSDI QSBDUJDFT ٔ 4DIPMBST NBLJOH MJTUT BOE DBUBMPHVFT ٔ 3FTFBSDI VOEFSUBLFO JO BSDIJWFT BOE MJCSBSJFT ٔ #SPXTJOH TPVSDFT JODMVEJOH ٔ NVTJD OPUBUJPO EPDVNFOUT ٔ OFXTQBQFS BSDIJWFT ٔ MFࡻFST ٔ QIPUPHSBQIT BOE QBJOUJOHT ٔ JOTUJUVUJPOBM SFDPSET BOE BDDPVOUT ٔ (BUIFS FWJEFODF BCPVU UIF NVTJDBM QBTU FH ٔ BDDPVOUJOH SFDPSET TIPX B NVTJDJBO BT BO FNQMPZFF PG TPNF JOTUJUVUJPO ٔ MFBSO GBDUT BCPVU B NVTJDJBOًT MJGF GSPN MFࡻFST CZ BOE BCPVU IJNIFS ٔ 3FTFBSDI FNQMPZT َNBOVBMُ TBNQMJOH NFUIPET ٔ $PODMVTJPOT BSF UIF SFTVMU PG UIF TDIPMBS EJHFTUJOH BMM UIF FWJEFODF 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
16. %JHJUJTBUJPO ٔ 1SFEJHJUJTBUJPO TDIPMBST SFMJFE PO FYQFSUJTF PG MJCSBSJBOT ٔ 3FTFBSDI XBT PࡺFO B DBTF PG CSPXTJOH ٔ *O UIJT FSB OFX EJTDPWFSJFT XFSF PࡺFO XPSUI B QVCMJDBUJPO ٔ %JHJUJTBUJPO NBLFT SFUSJFWJOH BOE WJFXJOH PG QSJNBSZ TPVSDFT USJWJBM ٔ 'VMMUFYU JOEFYFT HJWF B CBTJD XFMMVOEFSTUPPE NPEF PG EJTDPWFSBCJMJUZ ٔ #VU SFTVMUT BSF OPU BMXBZT SFMFWBOU JG DPOUFYU JT BNCJHVPVT 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
17. 4UBUJTUJDBM BQQSPBDIFT ٔ "SDIJWBM SFTFBSDI QSFEJHJUBM BOE EJHJUBM QSPEVDFT EFUBJMFE OBSSPX SFTFBSDI GJOEJOHT ٔ 4DIPMBST UFOE UP QSPCF EFFQMZ JOUP B SFTFBSDI RVFTUJPO ٔ $PNQVUBUJPOBM BOE TUBUJTUJDBM UFDIOJRVFT DBO CF BQQMJFE UP UBLF CSPBE WJFX ٔ &H " #JH %BUB )JTUPSZ PG .VTJD 4UFQIFO 3PTF BOE 4BOESB 5VQQFO ٔ 5BLFO BOE DMFBOFE WBSJPVT DPMMFDUJPOT PG CJCMJPHSBQIJD EBUB PO QVCMJTIFE NVTJD TJODF $UI ٔ "QQMJFE TVNNBSJTBUJPO BOE WJTVBMJTBUJPO UFDIOJRVFT UP TIPX USFOET JO NVTJD QVCMJTIJOH 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
18. 4UBUJTUJDBM BQQSPBDIFT T T GSPN ?iiT,ff2KXQt7Q`/DQm`MHbXQ`;f+QMi2Mif2`HvfkyR8fyNfykf 2KX+pydRX7mHH ٔ 4IPXJOH IPX RVBOUJUJFT PG NVTJD QVCMJTIFE JO EJख़FSFOU DJUJFT DIBOHFE PWFS UJNF 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
19. 4UBUJTUJDBM BQQSPBDIFT ٔ 1SPWJEFT GJSN FWJEFODF GPS IZQPUIFTFT BCPVU IPX NVTJD XBT CFJOH QSPEVDFE BOE VTFE ٔ 'SBODP .PSFࡻJ EFTDSJCFT UIF EJख़FSFODF CFUXFFO َDMPTF SFBEJOHُ BOE َEJTUBOU SFBEJOHُ 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
20. #VJMEJOH .*3 TZTUFNT GPS NVTJDPMPHZ ٔ .FUBEBUB NBZ CF TFFO BT َFBTZُ PS َTPMWFEُ ٔ 'PS TPNF .*3 BQQMJDBUJPOT UIJT NBZ CF ٔ &H GPS SFDSFBUJPOBM NVTJD JOGPSNBUJPO OFFET XF NBZ IBWF BVUIPSJUBUJWF BSUJTU BOE USBDL UJUMF EBUB ٔ 5IJT NBZ CF BVHNFOUFE XJUI EJTDPWFSBCJMJUZ MBCFMT HFOSF NPPE ٔ 'PS BQQMJDBUJPOT TVDI BT SFDPNNFOEFS TZTUFNT TVDI NFUBEBUB JT TVख़JDJFOU ٔ 4VDI TZTUFNT PࡺFO QSJPSJUJTF FBTFPGVTF TJOHMF JOUFSBDUJPO ٔ +VTU GJOE B QJFDF UIBU GJUT UIF SFRVJSFNFOU 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
21. #VJMEJOH .*3 TZTUFNT GPS NVTJDPMPHZ ٔ 5IJT JT JO DPOUSBTU UP B QPUFOUJBM .*3 TZTUFN GPS BTTJTUJOH IJTUPSJDBM NVTJDPMPHZ ٔ 4VDI B TZTUFN NBZ JOWPMWF MPOH SFTFBSDI TFTTJPOT ٔ .BOZ JOUFSBDUJPOT XJUI UIF TZTUFN ٔ 6TJOH KVEHFNFOU UP EFUFSNJOF XIBU UIF OFYU SFUSJFWBM SFRVJSFNFOU XJMM CF ٔ 'SFRVFOU DPOUFYU TXJUDIJOH ٔ NVTJDBM XPSL JOGPSNBUJPO ٔ HFPHSBQIJDBMMPDBUJPO JOGPSNBUJPO ٔ QSPTPQPHSBQIZ JOGPSNBUJPO ٔ /PU BVUPNBUJD DPNQVUBUJPOBM NVTJDPMPHZ ٔ #VU SBUIFS DPNQVUFSBTTJTUFE NVTJDPMPHZ 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
22. $IBMMFOHFT PG NFUBEBUB 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
23. $IBMMFOHFT PG NFUBEBUB ٔ %BUB JT PࡺFO NBEF BWBJMBCMF EJHJUBMMZ BT QSPTF ٔ &H &.0 DPMMFDUJPO IBT DBUBMPHVF SFDPSET GPS FBDI CPPL ٔ #VU UIF UBCMFT PG DPOUFOUT PG UIF CPPLT BSF VOTUSVDUVSFE UFYU ٔ &H NBOZ VTFGVM TPVSDFT BSF QSPTF BOZXBZ MFࡻFST OFXTQBQFST 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
24. $IBMMFOHFT PG NFUBEBUB ٔ %BUB NBZ CF EJTUSJCVUFE CFUXFFO WBSJPVT DBUBMPHVFT ٔ &H XIFO TFQBSBUF MJCSBSJFT IBWF IPMEJOHT PO UIF TBNF NVTJD ٔ &H XIFO B TJOHMF MJCSBSZ IBT TFQBSBUF DBUBMPHVFT GPS QSJOUFE BOE NBOVTDSJQU NVTJD 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
25. $IBMMFOHFT PG NFUBEBUB ٔ %BUB OFFET UP CF JOUFSQSFUFE DPSSFDUMZ ٔ )VNBO SFBEFST FTQFDJBMMZ EPNBJO FYQFSUT BQQMZ QSJPS LOPXMFEHF XIFO SFBEJOH TPVSDFT ٔ #VU BDDFTTJOH B EJHJUJTBUJPO WJB GVMMUFYU JOEFY NBZ MPTF DPOUFYU 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
26. $IBMMFOHFT PG NFUBEBUB ٔ &H 4JY %FHSFFT PG 'SBODJT #BDPO TJUF TIPXT DPOOFDUJPO CFUXFFO OBNFT َ)FOSZ 1VSDFMMُ BOE َ8JMMJBN #ZSEُ ٔ #PUI SFDPHOJTBCMF &OHMJTI DPNQPTFST ٔ #VU UIJT َ8JMMJBN #ZSEُ JT B $UI MBOEPXOFS 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
27. $IBMMFOHFT PG NFUBEBUB ٔ %BUB NBZ PࡺFO CF JODPNQMFUF ٔ &H NBOZ TVSWJWJOH CPPLT PG NVTJD IBWF BCCSFWJBUFE PS BCTFOU DPNQPTFS OBNFT BOE QJFDF UJUMFT ٔ "OE EBUB NBZ FWFO CF JODPSSFDU ٔ &H QSJNBSZ TPVSDFT IBWF BࡻSJCVUJPOT XIJDI BSF KVTU XSPOH 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
28. $IBMMFOHFT PG NFUBEBUB ٔ %BUB NBZ CF BNCJHVPVT PS BQQSPYJNBUF ٔ &H EBUFT JO DBUBMPHVFT HJWFO BT DJSDB PS SBOHFT ٔ &H BCCSFWJBUFE OBNFT UIBU CFDPNF BNJHVPVT 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
29. 5FDIOJRVFT 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
30. 5FDIOJRVFT ٔ %BUB FOSJDIJOH BOE DMFBOJOH ٔ -JCSBSZ DBUBMPHVFT BSF EFWFMPQFE PWFS NBOZ ZFBST ٔ $IBOHJOH TUBOEBSET PG EBUB FOUSZ ٔ &OPSNPVT WBSJFUZ PG OBUVSF PG EBUB ٔ &H #JH %BUB )JTUPSZ PG .VTJD TQFOU Fख़PSU DMFBOJOH #SJUJTI -JCSBSZ NVTJD DBUBMPHVFT ٔ .BLJOH VTF PG (PPHMF 3FGJOF ٔ &H PVS 4FNBOUJD .FEJB /FUXPSL NJOJQSPKFDU XPSLFE XJUI &BSMZ .VTJD 0OMJOF DBUBMPHVF EBUB ٔ "O FYBNQMF PG B DBUBMPHVF XJUI UBCMFT PG DPOUFOUT FODPEFE JO POF ."3$ GJFME 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
31. 5FDIOJRVFT ٔ -JOLJOH BOE BMJHOJOH ٔ -JOLFE %BUB BT B QPXFSGVM UFDIOJRVF GPS PQFOJOH VQ SFTFBSDI EBUB ٔ &WFSZUIJOH UIBU OFFET UP CF BEESFTTFE JT B SFTPMWBCMF 63* ٔ &WFSZUIJOH JT EFTDSJCFE VTJOH 3%' USJQMFT ٔ 5IPTF USJQMFT TIPVME VTF QVCMJDMZ BWBJMBCMF QSFEJDBUFT ٔ 'PMMPXJOH UIFTF QSJODJQMFT EBUB CFDPNFT FBTJFS UP JOUFHSBUF ٔ 8F Fख़FDUJWFMZ CVJME VQ B CJH 8FC PG EBUB 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
32. 5FDIOJRVFT ٔ 0VS 4FNBOUJD .FEJB /FUXPSL NJOJQSPKFDUT IBWF EPOF UIJT XJUI &.0 5JNًT MVUF NVTJD DBUBMPHVF BOE ##$ &BSMZ .VTJD 4IPX SFDPSEJOHT ٔ 1SPEVDJOH HPPE RVBMJUZ -JOLFE %BUB JOWPMWFT FYQFSUT BTTFSUJOH MJOLT ٔ 5PPM XIJDI IFMQT GJOE QPUFOUJBM MJOLBHFT JO EBUB TFUT ٔ 1SPWJEFT DPOUFYU UP IFMQ DBUBMPHVFST EFDJEF PO UIF QMBVTJCJMJUZ PG UIF MJOLT 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
33. $PODMVTJPOT ٔ .FUBEBUB JT DPSF NVTJDPMPHJDBM LOPXMFEHF ٔ .VDI PG UIF QSBDUJDF PG NVTJDPMPHJDBM JT DPODFSOFE XJUI NFUBEBUB ٔ .FUBEBUB JO UIF XJME JT NFTTZ ٔ 8FًSF SFBMMZ MPPLJOH GPS DPNQVUFSBTTJTUFE NVTJDPMPHZ OPU BVUPNBUJPO 3JDIBSE -FXJT BOE %BWJE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT .FUBEBUB   
34. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Social-Network Analytics and Music Ben Fields #ISMIRMusicologists !@alsothings
35. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists social-network analytics and music Social Network Analytics: methods to describe and analyse complex networks that concern social relationships
36. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists social-network analytics and music Music and the Web
37. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists social-network analytics and music Music curation and the Web
38. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists social-network analytics and music Music curation and the Web
39. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists social-network analytics and music Music curation and the Web
40. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Tools I use •networkX •iGraph •graphViz •Gephi •postgresql
41. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Common Techniques •diameter •average degree •shortest path problem (common solution: Dijkstra’s algorithm) •edge betweenness •community segmentation
42. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Diameter: the longest shortest-path between two nodes in a network Common Techniques diameter
43. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Average Degree: the mean number of edges across a network Common Techniques average degree
44. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Shortest Path Problem: broadly, how does one most efficiently find the shortest distance in edges between two nodes, e.g. Dijkstra’s algorithm Common Techniques shortest path problem
45. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists edge betweenness: for any edge, the count of shortest paths, that must run through that edge. A measurement of an edge’s importance in the community structure of a network Common Techniques edge betweenness
46. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists community segmentation: the process of separating a network into subgraphs, where the node connectivity within the subgraph is a high as possible and the node connectivity across subgraphs is a low as possible Common Techniques community segmentation
47. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists social-network analytics and music Let’s take a look at genius.com
48. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Lyric for Hypnotize, by The Notorious B.I.G. with annotation
49. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Lyric for Hypnotize, by The Notorious B.I.G. with annotation score
50. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists The recent annotations of the user OldJeezy
51. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists basic contributor graph •60,472 users (nodes) •1,156,940 edges (one+ common songs)
52. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists data acquisition •Scraping data from genius december 2014 •Collected the contributions from 704,438 •Associated metadata from 146,186 works •Scraper tool released under MIT license: https://github.com/gearmonkey/pyrg http://dx.doi.org/10.5281/zenodo.17515
53. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists data acquisition - basic stats description count total users 704,438 total annotations 1,256,912 total works 146,186 contributing users 71,129 annotation edits 2,196,522 annotations w/2+ editors 194,795
54. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists data acquisition - genre genre works percentage rap 107,270 73.3% rock 16,393 11.2% lit 9,386 6.2% news 3,720 2.5% pop 3,715 2.5% sports 1,140 0.7% x 1,140 0.6% country 744 0.5% screen 697 0.4% r-b 655 0.4% history 502 0.3% unbranded 370 0.2% law 250 0.1% tech 159 0.1% meta 151 0.1%
55. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists data acquisition - top tags tag works Rap Genius France 9,135 Genius France 6,009 Deutscher Rap 5,725 Polski Rap 3,298 West Coast 1,384 Brasil 841 Bay Area 839 Indie Rock 716 Chicago 710 Genius Britain 540
56. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists graph modelling - bipart •modelled first as a bipartite graph •nodes are either works or users •edges represent an instance where a user has contributed to an annotation of a work •216,943 nodes (71,129 users, 145,814 works) •439,835 edges (4.24x10-5 of all possible)
57. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists graph modelling - projected •project graph to works with weighted edges •keep only the largest connected component •resulting graph has 125,044 nodes and 38,259,818 edges
58. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists graph modelling - communities method modularity communities fast-greedy 0.529 498 leading eigenvector 0.003 11488 multilevel 0.582 169
59. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists arguments •full history of each annotation •find ‘edit wars’
60. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists most contentious annotations 1. 2040 | 3599357 | http://genius.com/Eminem-rap-god-lyrics 2. 1239 | 4086888 | http://genius.com/Kendrick-lamar-i-lyrics 3. 1224 | 2092508 | http://genius.com/Big-sean-control-lyrics 4. 1026 | 3680563 | http://genius.com/Big-sean-control-lyrics 5. 976 | 2371035 | http://genius.com/Eminem-bad-guy-lyrics score | anno id | song url
61. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists ecifying the origins song-by-song basis. e networks are cre- ation of social net- and characteristics. is proposed, unify- artist and genre net- . Such metrics are eresting patterns of ic suitable for mu- and visualizations led-based influence tribution; heavy in- on modern hip-hop, usicological results. N cant tool for under- ms. Social network garnered the atten- mputer science, and n retrieval commu- collaboration, rec- networks. d in Cano and Kop- arity networks from fluence using web scraping, web services, and audio simi- larity to construct influence graphs of a collection of synth pop music [7]. The work outlines the difficulty of construct- ing influence networks and motivates further investigation. Figure 1. Visualization of Genre Flow. The size and opacity of a directed edge indicates the relative flow of samples from one genre to another. Other Examples Musical Influence Network Analysis and Rank of Sample-Based Music Nicholas J. Bryan and Ge Wang ! Musical Influence Network Analysis and Rank of Sample-Based Music, Proc. ISMIR, 2011
62. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Other Examples Finding Community Structure in Music Genre Networks Debora C. Corrêa, Luciano da F. Costa, and Alexandre L. M. Levada ! Finding Community Structure in Music Genre Networks, Proc. ISMIR, 2011 ety for Music Information Retrieval Conference (ISMIR 2011) pb 5 0 7 Figure 6. od review on the aph theory. titioning method of the Laplacian incidence matri- et of vertices and Q, is given by: (1) s of V . Laplacian matrix Figure 6. The network of genres by the Fiedler vector.
63. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists In Summary •SNA provides a set of tools to observe socio- cultural phenomena •when applied to networks with musical intent and meaning: we get musicology(?)
64. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Questions Questions or comments to: •! @alsothings / b.fields@gold.ac.uk •! @TMusicology •#ISMIRMusicologists
65. 1 Information capture, consolidation, and analysis to support the musicological investigation of opera performance Experiences with Wagner's Ring Cycle Kevin Page, David Weigl Oxford e-Research Centre, University of Oxford Addressing the music information needs of musicologists ISMIR tutorial, October 2015
66. 2 Capturing an annotated copy of a complete Ring performance Outline • Musicology motivation • Hardware and software for collecting the data • Pragmatics of data collection • Preliminary data investigation • Applied data analysis (demo) • Bonus(?!): video during the break
67. 3 Musicology motivation: performance studies “In opera and music theatre, the individual realisation of a work in performance differs significantly from the abstract concept that is captured in the score. The scenic interpretation, with its own characteristics and its specific perspective on the work, is created afresh in every new staging - thus a performance and its experience cannot be measured on the basis of the score alone. Especially for studies that want to consider the reception and perception of an operatic work, taking into account the characteristics of individual stagings and keeping a record of particular performances is an important requirement. In recent decades, methodological shifts such as a ‘performative turn’ and reception theory accorded an increasing value to the performance as a source for musicological research. This poses the question of how best to document the ephemeral phenomenon of an operatic performance.” Carolin Rindfleisch, Oxford, October 2015
68. 4 Hearing Wagner? Seeing Wagner? Reading Wagner? Georg Unger as Siegfried, Der Ring des Nibelungen, Bayreuth 1876, Universitätsbibliothek Frankfurt am Main, urn:nbn:de:hebis:30:2-151468 Siegfried; Valery Gergiev and Mariinsky Opera: Der Ring des Nibelungen, Birmingham 2014 http://www.birminghamhippodr ome.com/WhatsOn_focus.asp ?showId=1823 Siegfried and Mime; Pierre Boulez and Patrice Chereau: Der Ring des Nibelungen, Bayreuth 1976, http://mostlyopera.blogspot.co.uk/ 2008/11/chreau-and-boulez- nibelungen-ring-on.html
69. 5 Hearing Wagner? Seeing Wagner? Reading Wagner? How can we capture performance? • Audio-visual recording – Neither objective nor exhaustive • Live annotation of performance – Di-erent kinds of information – Integration of document and analysis – Adaptable to research questions • If we can distinguish between the annotation technology and the meaning of the annotations
70. 6 Opportunity: the Marinsky Ring in Birmingham • Mariinsky Opera Company under Valery Gergiev • Birmingham Hippodrome • 4 nights over 5 days in November 2014 • Genesis of the project to capture bio-sensed physiological data – Create a musicological ground truth to form hypotheses – Builds on work in generating semantic hyperstructures for detailed and dense information spaces – A worthy investigation of performance in its own right • Hearing Wagner public engagement event two weeks(!) later presenting initial results
71. 7 The Musical Score Annotation Kit – ‘MuSAK’ ...an exercise in pragmatism! Tablet interface and annotation server • Considered a detailed interface that would enable the musicologist to annotated from a full coded pallette of annotations • But needed an interface that was fast enough to use quickly, and familiar enough to learn quickly • Developed a system using the 'Union' platform server and a tablet interface displaying a scanned score • Each tablet stroke is an annotation
72. 8 The Musical Score Annotation Kit – ‘MuSAK’ Digital Pen and Audio Notes • Livescribe 'Echo' pen • Uses a small camera and special 'Anoto' paper • Can also capture audio notes • Data downloaded over USB
73. 9 The Musical Score Annotation Kit – ‘MuSAK’ Score following and replay tool • Also web based • Captured page turns in sync with the performance – As opposed to turns when annotating • Software reused to create a replay of data for the Hearing Wagner event
74. 10 The Musical Score Annotation Kit – ‘MuSAK’ Audio and video • Video of the musicologist at work • No rights to capture and publish the actual performance – But need a notion of its implicit existence to synchronise timelines – Substitute (commercial) audio recordings for playback • Also background noise for synchronisation from Livescribe Echo pen
75. 11 Preparation and deployment IMSLP short piano score 250-365 pages per opera Trimmed and cleaned for tablet Marked up for musical annotation before performance ~3 days per opera Annotated on tablet during performance Each annotation layer can be replayed or processed separately Combined with other data sources: Echo pen, video, substitute audio
76. 12 Data collection • Fieldwork can be trying! • Some problems: – Annotator location – System problems (Die Walküre, Act III, Scene I) – Clock drift • Collected data – 15 hours, consisting four nights’ performance over Cve days – Over 100,000 tablet strokes → 8,216 annotations – 1,300 performance based page turns – 1,316 digital images – 104 pages of writing → 13 hours of digital pen replay – 15 hours of video footage
77. 13 Data publication – Linked Data and metaMuSAK
78. 14 Data publication – even “simple” things are complex! Representation of Ring annotation data
79. 15 Data publication – Linked Data and metaMuSAK http://www.transforming-musicology.org/tools/metaMuSAK
80. 16 Preliminary Data investigation – demo in a minute!
81. 17 Preliminary Data investigation – demo in a minute!
82. 18 Preliminary Data investigation – demo in a minute!
83. 19 ReJections • What started as a supporting data capture quickly became worthy of investigation in its own right – This was an iterative collaboration as we demonstrated to the musicologist what was possible technologically, then exploring the research data this provided musicologically • Also an exercise in compromise – It is far from a perfect system from a data capture perspective but the technologist's perfect tool would have been dysfunctional for the musicologist in the Celd • Relatively simplistic (for now) analysis, but still insightful for the musicologist... – Annotation rate has implications on suitability and analytical granularity of annotation key – Periods of peak annotation density can be identiCed and revisited by the musicologist for review
84. 20 Demo time... empirical reception studies using R 1) Inspect the raw and processed musicological annotation data 2) Compare musicologist's page viewing durations with page performance-event durations to quantify the overhead of the annotation task 3) Generate summaries of annotation activity in terms of annotations per page and per minute 4) Determine the variability of annotation rate (with implications on analytical granularity). 5) Isolate moments of peak annotation density for review by the musicologist
85. 21 Thank you – questions? kevin.page@oerc.ox.ac.uk david.weigl@oerc.ox.ac.ukk To find out more about the (Meta)MuSAK system and data... … see our paper in the ISMIR proceedings and poster (and demo) tomorrow (Tuesday). Also see: http://www.transforming-musicology.org/tools/metaMuSAK Thanks to all our co-authors and collaborators, but especially Terhi Nurmikko- Fuller (University of Oxford e-Research Centre), Carolin Rindfleisch (University of Oxford Faculty of Music) and Richard Lewis (Goldsmiths University of London Department of Computing). …and it ain't over 'til...
86. "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJD 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
87. 3FDBQ ٔ $POUFYUVBM BOE CJCMJPHSBQIJD JOGPSNBUJPO ٔ /FUXPSL BOBMZTJT UP VOEFSTUBOE DPOUFYU ٔ 3JDI NVMUJSFTPVSDFT BCPVU XPSLT BOE QFSGPSNBODFT ٔ /PX XF UVSO UP BO BQQBSFOU TUBQMF PG NVTJDPMPHJDBM TUVEZ UIF TDPSF 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
88. 4ZNCPMJD &ODPEJOH ٔ %JTDSFUF FODPEJOH BT PQQPTFE UP DPOUJOVPVT ٔ 3FRVJSFT B MFWFM PG BUPNJDJUZ HSBOVMBSJUZ ٔ /PSNBMMZ B QJUDIFE FWFOU XJUI B EVSBUJPO ٔ 0ࡺFO DBMMFE B َOPUFُ ٔ 0ࡺFO PSUIPHPOBM TUSFBNT PG FWFOUT MJLF َTUBWFTُ ٔ &YBNQMF FODPEJOHT ٔ .*%* GJMFT ٔ 1MBJOF  &BTJF $PEF ٔ %"3.4 ٔ .VTJD9.- ٔ .&* ٔ ,FSO ٔ .BOZ NBOZ PUIFST 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
89. .VTJD /PUBUJPOT ٔ 4ZNCPMJD FODPEJOHT PࡺFO NVTJDOPUBUJPO MJLF ٔ 8SJUJOH TZTUFNT GPS QSFTDSJCJOH PS EFTDSJCJOH NVTJDBM QFSGPSNBODF ٔ &ODPEJOHT PࡺFO CBTFE PO 8FTUFSO $PNNPO 1SBDUJDF QFSJPE OPUBUJPO ٔ &ODPEJOHT PࡺFO UBLF QJUDI BOE EVSBUJPO BT NPTU JNQPSUBOU JOGPSNBUJPO ٔ "MUIPVHI TDPSFT DBSSZ PUIFS JOGPSNBUJPO UPP ٔ &ODPEJOHT PG EJTDSFUF QJUDI FWFOUT HJWF B LJOE PG NBUSJY ٔ 'PS QPMZQIPOJD NVTJD SPXT PG QBSUT ٔ "OE FBDI SPX DPNQSJTFT B TFSJFT PG QJUDIFT 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
90. 4ZNCPMJD 5FDIOJRVFT ٔ 4USJOH BMHPSJUINT ٔ 3FQSFTFOUJOH NVTJD BT B TFSJFT PG TZNCPMT ٔ "QQMZJOH UFDIOJRVFT TVDI BT FEJU EJTUBODF GPS TJNJMBSJUZ ٔ .BSLPW NPEFMT ٔ 0OMJOF QBࡻFSO EJTDPWFSZ JO TZNCPM TUSFBNT ٔ &WFOU GSFRVFODZ BOBMZTFT ٔ &H QJUDI IJTUPHSBNT ٔ (FPNFUSJD BMHPSJUINT ٔ 3FQSFTFOUJOH NVTJD BT NVMUJEJNFOTJPOBM TZNCPMT ٔ "QQMZJOH USBOTGPSNBUJPOT GPS TJNJMBSJUZ BOE QBࡻFSO EJTDPWFSZ ٔ "O FYBNQMF JT UIF 4*" BMHPSJUIN GPS QBࡻFSO JEFOUJGJDBUJPO ٔ "OE 4*"5&$ GPS QBࡻFSO TFBSDI 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
91. 4ZNCPMJD 5BTLT ٔ ,FZ GJOEJOHNPEF GJOEJOH ٔ &H VTJOH QJUDI GSFRVFODZ BOBMZTJT ٔ #BTJDT PG BVUPNBUJD NVTJD BOBMZTJT ٔ 1JUDI TQFMMJOH ٔ 1BࣄFSO JEFOUJGJDBUJPOSFUSJFWBM ٔ &YUBDU NBUDIJOH WT BQQSPYJNBUF NBUDIJOH ٔ ,OPXO QBࡻFSOT WT VOLOPXO QBࡻFSOT ٔ 4FNBOUJD BVEJP ٔ &YUSBDUJOH OPUFMJLF JOGPSNBUJPO BOE PUIFS TFNBOUJD MBCFMT GSPN BVEJP ٔ ࣉFSZCZFYBNQMF ٔ 8IFSF UIF RVFSZ JT TZNCPMJD CVU UIF DPSQVT JT BVEJP 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
92. 4ZNCPMJD 5BTLT GPS .VTJDPMPHZ 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
93. 3FTPVSDF 1VCMJDBUJPO ٔ .BOZ FYBNQMFT PG َEJHJUJTBUJPOُ JO UIF TZNCPMJD FODPEJOH XPSME ٔ $$"3) BU 4UBOGPSE IBWF CFFO FEJUJOH BOE QVCMJTIJOH )VNESVN BOE .VTF%BUB DPOUFOU GPS NBOZ ZFBST ٔ .D(JMM XPSL PO FBSMZ NVTJD TPVSDFT BOE 0.3 ٔ 3*4. 01"$ QSPWJEFT JODJQJU TFBSDI 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
94. %PDVNFOUBSZ &WJEFODF ٔ 8IZ EP NVTJDPMPHJTUT TUVEZ OPUBUJPO TPVSDFT ٔ 5P EJTDPWFS EPDVNFOUBSZ FWJEFODF PG NVTJDBM QSBDUJDF ٔ #PUI DVSSFOU BOE IJTUPSJDBM ٔ 'SPN NVTJDBM TPVSDFT XF DBO MFBSO BCPVU XIBU NVTJD XBT LOPXO XIFSF BOE XIFO ٔ "MUIPVHI SFNFNCFS UIF NFUBEBUB QSPCMFNT JO IJTUPSJDBM TPVSDFT ٔ -FBSO GSPN EPDVNFOUBSZ FWJEFODF BCPVU IPX NVTJD XBT USBOTNJࡻFE ٔ .BOVTDSJQUT CPPLT PࡺFO DPOUBJO DPQJFT PG QJFDFT CVU PࡺFO XJUI FSSPST ٔ -POH USBEJUJPO PG QSPEVDJOH FEJUJPOT PG NBOVTDSJQU NVTJD CPPLT 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
95. %PDVNFOUBSZ &WJEFODF ٔ 4DIPMBST CVJME VQ EPNBJO LOPXMFEHF EVSJOH UIFJS DBSFFST ٔ 0O NVTJDBM XPSLT NVTJDJBOT TDSJCFT MPDBUJPOT BOE EBUFT PG CPPLT ٔ 6TJOH UIJT UIFZ BSF BCMF UP BQQSPBDI SFTFBSDI RVFTUJPOT TVDI BT ٔ XIP DPQJFE UIJT OFX VOBࡻSJCVUFE TPVSDF ٔ HJWFO UIJT TJOHMF QBSUCPPL XIJDI QJFDFT BSF DPQJFE JOUP JU ٔ PG XIJDI PUIFS QJFDF JT UIJT OFX TPVSDF BO BSSBOHFNFOU ٔ 5P FODPEF TPVSDFT XJUI UIF LJOE PG EFUBJM SFRVJSFE GPS EPDVNFOUBSZ SFTFBSDI XF OFFE SJDI BOE GMFYJCMF FODPEJOHT ٔ 'PS UIJT UIF .VTJD &ODPEJOH *OUJUBUJWF JT QSPNJTJOH 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
96. $POUFOUCBTFE $PSQVT .VTJDPMPHZ ٔ "SHVFE FBSMJFS UIBU NVTJDPMPHZ JT MBSHFMZ OPU BCPVU NVTJDBM DPOUFOU ٔ #VU DPNQVUBUJPOBM UFDIOJRVFT QSPWJEF BDDFTT UP DPOUFOU ٔ &TQFDJBMMZ PO B MBSHFTDBMF UIBU IBT OFWFS CFFO USBDUBCMF GPS NVTJDPMPHJTUT ٔ 4P DPSQVT NVTJDPMPHZ IBT KVTU OFWFS GFBUVSFE BT QBSU PG UIF EJTDJQMJOF ٔ "MTP WBSJPVT IJTUPSJDBM SFBTPOT XIZ NVTJDPMPHJTUT UFOE UP BWPJE XIBU UIFZ TFF BT َDPNQBSBUJWFُ XPSL JO TUVEZJOH NVTJD ٔ "SFBT PG SFTFBSDI UIBU NBZ CF PQFOFE VQ CZ MBSHFTDBMF UFDIOJRVFT JODMVEF ٔ 4UVEZJOH UIF َNVTJDBM CBDLHSPVOEُ 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
97. .VTJDBM 4FBSDI ٔ ,OPXMFEHF PG SFQFSUPJSF TPVSDFT BOE USBEJUJPOT NPTU SFMJBCMF TPVSDF PG OFX EJTDPWFSJFT ٔ (JWFO TVख़JDJFOU DPSQPSB DPOUFOUCBTFE TFBSDI NBZ FOBCMF SFTFBSDI RVFTUJPOT BSPVOE NVTJDBM BMMVTJPO ٔ RVPUBUJPO CPSSPXJOH BSSBOHFNFOU DPQZJOH FSSPST FUD ٔ 1PUFOUJBM UFDIOJRVFT JODMVEF ٔ 1ISBTFCBTFE SFUSJFWBM HJWFO B TZNCPMJD NVTJD QISBTF SFUVSO BMM UIF QJFDFT UIBU DPOUBJOT NBUDIFT XJUIJO B UISFTIPME BOE UIFJS MPDBUJPOT ٔ &H 5IFNF'JOEFS 1FBDIOPUF 3*4. 01"$ 5JN $SBXGPSE BOE 3JDIBSE -FXJT .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT 4ZNCPMJD .VTJ   
98. EMO & ECOLM! Printed 16th-century vocal and lute music
99. Elements for a Workflow! Will need to be roughly similar for any medium-large scale automatic music-encoding project
100. Secular vocal Sacred vocal Free compositions Dances ('suites' count as one) Total Totals 486 89 232 275 1082 45% 8% 22% 25% Secular vocal Sacred vocal Free composi6ons Dances ('suites' count as one) 300 books of music printed before 1600 All are ‘anthologies’ (i.e. by >1 composer) About 10% (27 books) are in tablature About 50% of tablature is arrangements of vocal music Need to identify unknown source-models for the arrangements Early Music Online (EMO)!
101. Early Music Online (EMO)! Plenty of musicological literature about the arrangements Essential resources is: Howard Mayer Brown, Instrumental Music Published before 1600 (Harvard, 1965)
102. H.M. Brown, Instrumental Music!
103. H.M. Brown, Instrumental Music!
104. H.M. Brown, Instrumental Music!
105. H.M. Brown, Instrumental Music!
106. Early Music Online (EMO)! But also, scholars (and, more recently, players) have identified quotations of vocal music in fantasias, etc. As well as direct quotations, lute music oen contains less-clear allusions or even paraphrases of well-known music of the time Perhaps we can discover more of these?
107. Early Music Online (EMO)! In manuscript sources, most lute music is anonymous Titles and/or composer names (where given) are oen incorrect, misspelt or otherwise garbled in manuscripts - and even in prints
108. Early Music Online (EMO)! We are looking at ways to use musical content to ‘amplify’ metadata searches, since approximate matching is required in both domains Different approaches needed for lute and vocal music, because of the differences between tablature and mensural notation
109. EMO/ECOLM cross-searching! Need a common/compatible data structure for searching We treat a score as a discrete pitch/onset- time matrix (Something like MIDI) Use geometric search methods such as SIAMESE We can also generate chroma and use audio methods (e.g. to match with recordings)
110. EMO! 16-c vocal music was mostly published in part-books
111. EMO – part-books! Superius! Altus! Tenor! Bassus!
112. EMO – part-books! 16-c vocal music was mostly published in part-books OMR - Correction interface within Aruspix
113. EMO – Aruspix - Optical Music Recognition! Vocal music in part-books OMR - Correction interface within Aruspix
114. EMO – part-books! 16-c vocal music was mostly published in part-books OMR - Correction interface within Aruspix -> MEI encodings (part-books)
115. EMO – part-books – MEI encodings! Vocal music in part-books OMR - Correction interface within Aruspix -> MEI encodings (part-books)
116. EMO – part-books -> scores! Given corrected MEI part-book encodings … … making scores is easy, but correction takes time Making scores from error-ridden parts is an open problem we are addressing Converting score to pitch/time matrix is trivial
117. EMO – metadata for pieces?! … but, where do pieces begin/end?
118. EMO – metadata for pieces?! Image 014! Image 030! Image 046! Image 062!
119. EMO – metadata for pieces?! … but, where do pieces begin/end? EMO collation interface, based on library metadata
120. EMO – ‘Inventory Maker’!
121. ECOLM – Tablature recogntion is simpler ! Each piece is complete on a page-sequence in a single book Optical Tablature Recognition based on Gamera Errors are inevitable with optical recognition Crowd-source expert correction via web interface
122. ECOLM – Tablature correction! Crowd-source expert correction via web interface Double-entry: correct twice for each line of music … but, where do pieces begin/end? Easier than for vocal music: use Brown metadata to collate corrected lines
123. ECOLM – from tablature to search format! With corrected tablature for complete pieces … conversion to pitch/time matrix is easy We can search within ECOLM using SIAMESE We can generate chroma and use audio methods (e.g. to match with recordings)
124. ECOLM! ECOLM editing/playback/search interface demo
125. Near-neighbour search in acoustic feature space A case study in contrafactum and parody Presented/adopted by Ben Fields Material originally by Christophe Rhodes
126. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Introduction The ‘remixes’ and ‘cover songs’ of the Renaissance Parody and contrafacta Lugebat David Absalon Je prens congies J'ay mis mon cueur Credo a 8 mass
127. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Introduction Parody and contrafacta Nirvana Weird Al
128. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Introduction Josquin des Prez (c.1450–1521?) • contemporary reputation: greatest composer of the age Josquin des Prez *OUSPEVDUJPO 4JNJMBSJUZ 3FTVMUT $PODMVTJPOT *OUSPEVDUJPO +PTRVJO EFT 1SF[ +PTRVJO EFT 1SF[ Dم ٔ DPOUFNQPSBSZ SFQVUBUJPO HSFBUFTU DPNQPTFS PG UIF BHF
129. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Introduction Nicolas Gombert (c.1495–c.1560) • transition figure between Josquin and Palestrina Nicolas Gombert
130. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Introduction The Je prens congies complex •Je prens congies chanson •Tu sola es and Tulerunt Dominum motets •Lugebat David Absalon motet •Credo a 8 mass fragment •J’ay mis mon cueur chanson
131. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Introduction Attribution Attribution shift over almost a century •19th century (Otto Kade) •Tulerunt authentic Josquin •Lugebat David Absalon doubtful •Josquin Edition (Albert Smijers, 1921–) •neither Tulerunt nor Lugebat •DasChorwerk (Blume,1933) •Tulerunt attributed to Josquin •Early Venetian Motets (Norbert Böker-Heil, 1969) •Identification of Je prens congas as source material
132. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Introduction Attribution *OUSPEVDUJPO "ࡻSJCVUJPO
133. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Similarity Audio Features: Non-Negative Least Squares Chroma •log-frequency short-time Fourier Transform Yk,m •note dictionary Ek,n •find note activations xn,. to minimize ||Yk,. − Ek,nxn,.||2 •subject to xn,m ≥ 0 Matthias Mauch and Simon Dixon, Approximate Note Transcription for the Improved Identification of Difficult Chords, Proc. ISMIR, 2010!
134. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Similarity Audio similarity: sequences •In this investigation: •frames: 1s granularity; •feature dimensionality d: 12; •sequences: searchlength 10-30 frames. *OUSPEVDUJPO 4JNJMBSJUZ 3FTVMUT $PODMVTJPOT 4JNJMBSJUZ "VEJP TJNJMBSJUZ TFRVFODFT E E × TFBSDIMFOHUI O − TFBSDIMFOHUI + 1 *O UIJT JOWFTUJHBUJPO ٔ GSBNFT T HSBOVMBSJUZ ٔ GFBUVSF EJNFOTJPOBMJUZ E  ٔ TFRVFODFT TFBSDIMFOHUI  GSBNFT $ISJTUPQIF 3IPEFT 'SJEBZ UI 4FQUFNCFS  /FBSOFJHICPVS TFBSDI JO BDPVTUJD GFBUVSF TQBDFT   
135. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Similarity Audio similarity Fragment to fragment: ! •metric distance (maybe?) •non-negative(maybe?) •bounded above (maybe?) NJMBSJUZ P TJNJMBSJUZ HNFOU UP GSBHNFOU E-(GJ N, GK O) NFUSJD EJTUBODF NBZCF OPOOFHBUJWF NBZCF CPVOEFE BCPWF NBZCF
136. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Similarity Audio similarity Fragment to fragment: ! •metric distance (maybe?) •non-negative(maybe?) •bounded above (maybe?) NJMBSJUZ P TJNJMBSJUZ HNFOU UP GSBHNFOU E-(GJ N, GK O) NFUSJD EJTUBODF NBZCF OPOOFHBUJWF NBZCF CPVOEFE BCPWF NBZCF Fragment to track: HNFOU UP GSBHNFOU E-(GJ N, GK O) ٔ NFUSJD EJTUBODF NBZCF ٔ OPOOFHBUJWF NBZCF ٔ CPVOEFE BCPWF NBZCF HNFOU UP USBDL E-(GJ N, UK) = NJO O E-(GJ N, GK O)
137. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Similarity Audio similarity Fragment to fragment: ! •metric distance (maybe?) •non-negative(maybe?) •bounded above (maybe?) NJMBSJUZ P TJNJMBSJUZ HNFOU UP GSBHNFOU E-(GJ N, GK O) NFUSJD EJTUBODF NBZCF OPOOFHBUJWF NBZCF CPVOEFE BCPWF NBZCF Fragment to track: HNFOU UP GSBHNFOU E-(GJ N, GK O) ٔ NFUSJD EJTUBODF NBZCF ٔ OPOOFHBUJWF NBZCF ٔ CPVOEFE BCPWF NBZCF HNFOU UP USBDL E-(GJ N, UK) = NJO O E-(GJ N, GK O) Track to track: NFOU UP GSBHNFOU E-(GJ N, GK O) NFUSJD EJTUBODF NBZCF OPOOFHBUJWF NBZCF CPVOEFE BCPWF NBZCF NFOU UP USBDL E-(GJ N, UK) = NJO O E-(GJ N, GK O) L UP USBDL E-(UJ, UK) = NJO N E-(GJ N, UK)
138. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Similarity Outlier analysis *OUSPEVDUJPO 4JNJMBSJUZ 3FTVMUT $PODMVTJPOT 4JNJMBSJUZ 0VUMJFS BOBMZTJT $ISJTUPQIF 3IPEFT 'SJEBZ UI 4FQUFNCFS  /FBSOFJHICPVS TFBSDI JO BDPVTUJD GFBUVSF TQBDFT   
139. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists *OUSPEVDUJPO 4JNJMBSJUZ 3FTVMUT $PODMVTJPOT 4JNJMBSJUZ 0VUMJFS BOBMZTJT $ISJTUPQIF 3IPEFT 'SJEBZ UI 4FQUFNCFS  /FBSOFJHICPVS TFBSDI JO BDPVTUJD GFBUVSF TQBDFT    Similarity Outlier analysis
140. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists *OUSPEVDUJPO 4JNJMBSJUZ 3FTVMUT $PODMVTJPOT 4JNJMBSJUZ 0VUMJFS BOBMZTJT $ISJTUPQIF 3IPEFT 'SJEBZ UI 4FQUFNCFS  /FBSOFJHICPVS TFBSDI JO BDPVTUJD GFBUVSF TQBDFT    Similarity Outlier analysis
141. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Results Josquin attribution With some care: can find all of the recordings in the Je prens congies complex as low-distance retrievals: •aggregate over many fragments of source tracks; •chroma rotation.
142. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Results Reproducibility Gave (broadly) this investigation to students at Oxford Digital Humanities Summer School: •no prior programming assumed; •audio features precomputed. Student feedback for five-day “Digital Musicology”: 92% “met” or “exceeded” expectations
143. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Conclusions •audio similarity can be used to guide attribution... •…if recordings are available •(mechanically-generated recordings would be better for this task) •similarity outliers can help increase understanding
144. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Conclusions •query language (Hendrik Blokeel “Declarative Data Analysis”) •hierarchical indexing (Fionn Murtagh “High Dimensional Data Scaling”) •shrink-wrap software •new interfaces for exploring collections of music Future Work
145. 26 October 2015 ISMIR, Malaga, Spain Ben Fields, @alsothings #ISMIRMusicologists Questions Investigation materials: •audioDB software: https://github.com/TransformingMusicology/ audioDB •linked from http://www.doc.gold.ac.uk/~mas01cr/papers/: •iPython notebook •numeric feature data Questions or comments to: •! @ascii19 / c.rhodes@gold.ac.uk (Christophe Rhodes) •! @alsothings / b.fields@gold.ac.uk (Me!) •! @TMusicology •#ISMIRMusicologists
146. Wagner leitmotifs! Proof-of-concept work mostly done under OMRAS2 project, c2008-11
147. Richard Wagner! Richard Wagner (1813-1883) is well-known for developing a method of composition which employed ‘leitmotifs’ (leading motifs) to help the audience follow the dramatic narrative and identify with the emotional states of the characters on-stage. Technique still used today in advertising and film music.
148. Research question! Since the leitmotif technique depended on the audience perceiving the leitmotifs clearly, can we use audio MIR to ‘recognise’ them as humans do? In the psychological part of Transforming Musicology, we are working to establish how well audiences hear and respond to Wagner’s leitmotifs.
149. Research challenge! Early work (from the OMRAS2 project) used NNLS chroma features as a proof-of-concept Some leitmotifs are easy to recognise, but others much harder It depends on musical context, and the extent to which they differ from the query example Also, tempo, timbre (instrument, voice) and orchestral texture (scoring) play a part, too
150. Wagner’s ‘Ring’ Cycle! Sir Georg Solti’s famous recording of the ‘Ring’ Cycle of four operas (Das Rheingold, Die Walküre, Siegfried & Göerdämmerung) Recorded in Vienna, 1958-66 Re-mastered and re-released on CD oen 148 tracks, ~12 hours of music, avge 5 min
151. Method (i)! Extract NNLS-chroma features (4 & 10 frames/sec) from Solti Ring recordings and make a pair of audioDB databases Extract NNLS-chroma features (4 & 10 f/s) from a set of leitmotifs for which we have names (from books by Donington and Mann) and score-extracts (graphics) Build simple web-interface
152. Method (ii)! Perform searches a) at same pitch b) transposed (queries ‘rotated’) to all 12 keys c) at 4 and 10 f/s (user choice) Select 10 top best matches
153. 4USVDUVSFE EJTDVTTJPO ٔ -BTU IBMGIPVS JT B TUSVDUVSFE EJTDVTTJPO TFTTJPO ٔ %JWJEF JOUP QBJST BOE EJTDVTT ZPVS .*3 UFDIOJRVFT BOE DVSSFOU BQQMJDBUJPOT ٔ 5IFO TVHHFTU QPTTJCMF NVTJDPMPHJDBM BQQMJDBUJPOT PG FBDI PUIFSًT UFDIOJRVFT ٔ +PJO JOUP MBSHFS HSPVQT  PS  QSFTFOU FBDI PUIFST UFDIOJRVFT BOE QSPQPTFE NVTJDPMPHJDBM BQQMJDBUJPOT ٔ %FDJEF PO UXP PS UISFF NVTJDPMPHJDBM BQQMJDBUJPOT UP EJTDVTT ٔ &BDI HSPVQ UIFO QSFTFOUT UIPTF BQQMJDBUJPOT BU UIF FOE 5SBOTGPSNJOH .VTJDPMPHZ .POEBZ  0DUPCFS  .¡MBHB "EESFTTJOH UIF .VTJD *OGPSNBUJPO /FFET PG .VTJDPMPHJTUT