Share
Embed
ヒューマンコンピュテーションとクラウドソーシング / Human Computation and Crowdsourcing
2018年度人工知能学会全国大会チュートリアル
Yukino Baba PRO
June 08, 2018
Transcript
58 slides
-
ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽SF$"15$)" 俑㶵钠陎ءأذيח➂穈鴥 ˑNPSPJEH˒ ˑNPSSJOH˒ ˑNPSOJOH˒ 4UFQ剅硂⚥ך俑㶵אך0$3ءأذيח钠陎ׇׁ 4UFQ穠卓ָ♶♧荜ךהֹ➂ח㉏ְさׇ L. von Ahn
et al.: reCAPTCHA: Human-based character recognition via web security measures, In Science, 2008. 4/58 -
-tap to take a photo. -tap to begin
recording your question and again to stop. side, User ? Database - al Client mote Services and Worker Interface ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽7J[8J[ 鋔鋙ꥺְָ罏佄䴂ءأذيח➂穈鴥 Ύءأذيⰻ鿇 ך➂ָ㔐瘶 J. Bigham et al.: VizWiz: Nearly real-time answers to visual questions, In UIST, 2010. ِ٦ؠָ颵㉏䫎珲 ⢽չ؝٦ָٝⰅ 綸כוպ 5/58 -
ؼُ٦وٝ؝ٝؾُذ٦ءّٝך⢽'PME*U S. Cooper et al.: Predicting protein structures with a
multiplayer online game, In Nature, 2010. فٖ؎َ٦כ 넝أ؝،杆ְ 圓鸡㢌⻉ׇׁ ةٝػؙ颵ך圓鸡✮庠 ،ىظꃐꂁַ 甧⡤圓鸡✮庠ׅ㉏겗 ةٝػؙ颵圓鸡✮庠٦ي⻉׃➂ח鍑ַׇ 6/58 -
ؙٓؐسا٦ءؚٝ ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝדכ㢳ֻך⸇罏ָ䗳銲 ! ؙٓؐسا٦ءؚٝ o ؎ٝة٦طحز鸐ׄג➂ח⡲噟⣛걾ׅ➬穈 o ⢽"NB[PO.FDIBOJDBM5VSL ٓٝ؟٦ؤ
o ♧鿇ך؟٦ؽأדכ"1*ח״⡲噟⣛걾ָ〳腉 㢳侧ך➂ח،ؙإأׅך➬穈 ؙٓؐس ا٦ءؚٝ ⣛걾罏 ⡲噟罏 ةأؙ涪遤 Ύةأؙⶴ䔲 Ώ穠卓䲿⳿ ΐ㜠ꂹ锜実 7/58 -
SF$"15$)"ך،فٗ٦ث ! ⹛堣בֽ俑㶵钠陎⡲噟钠鏾ءأذيח穈鴥 o 㔐瘶罏כչ荈ⴓָ➂պה爙ׅ溪ח䮋 ! Ύ 姻鍑傀濼ך㉏겗ךⵃ欽אך俑㶵䲿爙ׅ o
姻鍑劢濼ך俑㶵钠陎׃ְ俑㶵 o 姻鍑傀濼ך俑㶵➂ַنحزַךⴻ㹀חⵃ欽 ! Ώ ⚛⻉醱侧ךչ➂պָずׄ瘶ִ鵤׃䱰欽 姻鍑劢濼 姻鍑傀濼 姻瘶ז➂ 铎瘶זنحز 钠鏾ءأذيח穈鴥姻׃ְ瘶ִ䒷ֹ⳿ׅ 12/58 -
劤ثُ٦زٔ،ٕךذ٦و꧊㔚崞ַ׃،فٗ٦ث ➂䊨濼腉ח״㉏겗鍑寸 ➂ָ佄䴂 ➂ח״㉏겗鍑寸 ➂䊨濼腉ָ佄䴂 ꧊㔚ך⸂ד姻׃ְ瘶ִ䒷ֹ⳿ׅ 㼔㹺涪鋅 ⼿锃佄䴂 ⚛㉏さׇ 湫㉏さׇ
知⽃٥⽃秪ז㉏겗 㔭ꨇ٥醱꧟ז㉏겗 ! ꧊㔚ד㉏겗鍑ַׇ٥鍑ֽ➂鋅אֽֿהד ➂ַ姻׃ְ瘶ִ䒷ֹ⳿ׅ 13/58 -
⚛㉏さׇ ! ㉏ְさׇ湱䩛ָ♧➂ה ך➂ָىأׅה姻׃ְ瘶ִָ䖤זְ ! 醱侧➂ח㉏ְさׇג㔐瘶窟さֿׅהד 姻׃ְ瘶ִ䒷ֹ⳿ׅ o 知⽃ז䩛岀㢳侧寸 ⴽղח鍑ַׇג瘶ִתה
ⱖ溪ח둷ָⱖגְתַׅ :&4 /0 NO YES YES YES 瘶ִה׃ג 䱰欽 15/58 -
窟鎘涸ז㔐瘶窟さ ! 㢳侧寸״峤箺ׁ倯岀窟鎘涸ז㔐瘶窟さ ! 醱侧➂ך㔐瘶ַ姻鍑✮庠ׅ㉏겗ה׃ג㹀䒭⻉ o 欰䖝ך㔐瘶ַֽ鑐꿀ך姻鍑✮庠ׅ״ֲזך YES YES YES
NO NO YES YES YES NO YES NO YES ? ? ? ㉏겗 醱侧➂ך㔐瘶ַ窟鎘涸ח姻鍑✮庠ׅ 姻鍑 16/58 -
㔐瘶罏ך腉⸂罋䣁 ! 㔐瘶罏ָ醱侧㉏ח㔐瘶ֿׅהⵃ欽׃ג 㔐瘶罏ך腉⸂䱿㹀׃姻鍑✮庠ח欽ְ YES YES YES NO NO YES
YES YES NO YES NO YES 㔐瘶罏 ? ? ? ㉏겗 腉⸂䱿㹀 A. P. Dawid and A. M. Skene: Maximum likelihood estimation of observer error-rates using the EM algorithm, Journal of the Royal Statistical Society. Series C (Applied Statistics), 1979. 㔐瘶罏ך腉⸂䱿㹀׃姻鍑✮庠ח欽ְ 姻鍑 17/58 -
㔐瘶罏ך腉⸂罋䣁 ! 㔐瘶罏ך腉⸂珏겲ך然桦ד邌植ׅ o 姻鍑ָ:&4ךהֹח姻׃ֻ:&4ה瘶ִ然桦 o 姻鍑ָ/0ךהֹח姻׃ֻ/0ה瘶ִ然桦 ! ֿך然桦欽ְה如ך״ֲז㔐瘶欰䧭ٌرָٕ䖤 腉⸂然桦ד邌植׃㔐瘶欰䧭麓玎ٌرٕ⻉
✓j j 姻鍑錁庠דֹזְ 㔐瘶 /0:&4 Pr [yij | ti = 1] = ✓yij j (1 ✓j)(1 yij ) Pr [yij | ti = 0] = (1 yij ) j (1 j)yij 18/58 -
㔐瘶罏ך腉⸂罋䣁 ! 姻鍑悵㖈㢌侧ה׃&.،ٕ؞ٔؤيד 姻鍑ה腉⸂❛✼ח䱿㹀ׅ o 4UFQ腉⸂㔿㹀׃ג姻鍑䱿㹀 o 4UFQ姻鍑㔿㹀׃ג腉⸂䱿㹀 ㉏겗ך姻鍑ה㔐瘶罏ך腉⸂❛✼ח䱿㹀 qi
= Pr [ti = 1 | {yij }] / p Y j ✓yij j (1 ✓j)(1 yij ) 姻鍑ָ:&4ך㉏겗דך姻瘶桦ך״ֲזך ✓j = P i qiyij P i qi , j = P i (1 qi) yij P i (1 qi) p = Pr [ti = 1] 19/58 姻鍑 -
㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝罋䣁 ! չꨇ׃ְ㉏겗ד姻瘶ׅ㔐瘶罏ך倯ָ腉⸂ָ넝ֲպ ̔㉏겗ךꨇ僒䏝罋䣁ׅ YES YES YES NO NO YES
YES YES NO YES NO YES ? ? ? ㉏겗 J. Whitehill et al.: Whose vote should count more: optimal integration of labels from labelers of unknown expertise, In NIPS, 2009. 㔐瘶罏 㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝䱿㹀׃姻鍑✮庠 ꨇ僒䏝䱿㹀 姻鍑 20/58 -
㔐瘶罏ך腉⸂ה㉏겗ךꨇ僒䏝罋䣁 ! 㔐瘶罏ה㉏겗ךػًٓ٦ة㼪Ⰵ o 㔐瘶罏ך腉⸂ o ㉏겗ך知⽃ׁ ! 㔐瘶罏٥㉏겗ךػًٓ٦ة欽ְג姻瘶然桦邌植׃ 姻鍑הぐػًٓ٦ة䱿㹀
腉⸂ָPS知⽃ָׁה姻瘶然桦 腉⸂٥知⽃ָׁ㣐ְֹקו姻瘶然桦ָח鵚בֻ 姻鍑 㔐瘶 腉⸂ 知⽃ׁ Pr [ yij = ti] = 1 1 + exp ( ✓j⌫i) ✓j ⌫i 腉⸂٥ꨇ僒䏝ח㛇בֻ姻瘶然桦邌植 21/58 -
㔐瘶罏ך然⥋䏝罋䣁 ! 㔐瘶罏ח然⥋䏝耀ֻ ! ׃然⥋䏝姻׃ֻ瘶ִ⥂鏾כזְ o 铎瘶זךחչ荈⥋ָ֮պה瘶ִ➂ְָ 荈⥋麓ⶱ o 姻瘶זךחչ荈⥋ָזְպה瘶ִ➂ְָ
荈⥋麓㼭 ⱖ溪ח둷ָⱖגְתַׅ 㔐瘶ח荈⥋ָ֮תַׅ כְ כְ ְְִ ְְִ 40ZBNB :#BCB :4BLVSBJ BOE) ,BTIJNB"DDVSBUFJOUFHSBUJPOPGDSPXETPVSDFEMBCFMT VTJOHXPSLFSTTFMGSFQPSUFEDPOGJEFODFTDPSFT *O *+$"* 㔐瘶罏ח然⥋䏝耀ְג姻鍑✮庠ח欽ְ 22/58 -
㔐瘶罏ך然⥋䏝罋䣁 ! 然⥋䏝ך⫘ぢ然桦ד邌植 ! 然⥋䏝ך欰䧭ٌرٕ㼪Ⰵ׃ 㔐瘶欰䧭ٌرٕה穈さׇג姻鍑✮庠ׅ ⇡(00) j ⇡(01) j
⇡(10) j ⇡(11) j 姻鍑ָ/0 姻鍑ָ:&4 㔐瘶ָ/0 㔐瘶ָ:&4 Pr [cij | ti = 0, yij = 1] = ⇣ ⇡(01) j ⌘cij ⇣ 1 ⇡(01) j ⌘(1 cij ) 㔐瘶 姻鍑 然⥋䏝 չ荈⥋ָ֮պה 瘶ִ然桦 然⥋䏝ך欰䧭麓玎ٌرٕ⻉ 23/58 -
♧㼎嫰鯰ך㔐瘶窟さ ! 醱侧➂ח♧㼎嫰鯰遤ׇ姻鍑ؚٓٝؗٝ✮庠ׅ o ⢽ֶװ־➂孡갫חؚٓٝؗٝ׃ְ > > > ♧㼎嫰鯰 㔐瘶窟さ穠卓ؚٓٝؗٝ
9$IFOFUBM1BJSXJTFSBOLJOHBHHSFHBUJPOJOBDSPXETPVSDFETFUUJOH *O84%. > > > ♧㼎嫰鯰ך穠卓ַ姻鍑ؚٓٝؗٝ✮庠ׅ 24/58 -
♧㼎嫰鯰ך㔐瘶窟さ ! ،؎ذي׀החչ䓼ׁպָ֮הׅ o ،؎ذيך䓼ׁ ! 㔐瘶罏ך腉⸂然桦ד邌植 o 䓼ְ،؎ذيח姻׃ֻ䫎牰ׅ然桦 !
如䒭ך䫎牰ٌرٕח㛇בֹծ،؎ذيך䓼ׁ✮庠ׅ ti ָׁ،؎ذيח䫎牰ׅ然桦 ،؎ذيָ״䓼ְ然桦 姻׃ֻח䫎牰ׅ然桦 铎גח䫎牰ׅ然桦 Pr [i j k] = ✓j eti eti + etk + (1 ✓j) etk eti + etk ✓j ،؎ذيך䓼ׁ٥㔐瘶罏腉⸂ח㛇בֻ䫎牰ٌرٕ 25/58 -
侧⦼㔐瘶ך窟さ ! 醱侧ך侧⦼㔐瘶ַ姻鍑✮庠ׅ ! 䎂㖱《ה姻鍑הך铎䊴ָ㣐ֹֻז㜥さ֮ ! ✳⦼㔐瘶װ♧㼎嫰鯰ה殯ז㔐瘶⦪酡ָ搀ꣲח㶷㖈 43 215 215
860 860 860 860 860 215 215 姻鍑כ ㉏겗⢽չך秈侧ךֲ剑㣐ך㣼侧כְֻאַպ $-JOFUBM$SPXETPVSDJOHDPOUSPMNPWJOHCFZPOENVMUJQMFDIPJDF *O6"* 㔐瘶⦪酡ָ搀ꣲח֮侧⦼㉏겗ך姻鍑✮庠ׅ 26/58 -
侧⦼㔐瘶ך窟さ ! 麩ִ倯ך⨉⚥螟俱椚䏄麓玎ד邌植 ! 珏겲ך㔐瘶遤⹛ٌرٕ⻉ 姻瘶ׅ Ύ 傀⳿ך铎瘶ׅ Ώ
劢⳿ך铎瘶ׅ 860 43 … ⚥螟俱椚䏄麓玎 չ➂ָ㢳ְذ٦ـٕח➂ָ꧊תװְׅ⫘ぢպ邌植 ⚥螟俱椚䏄麓玎ח㛇בֻ㔐瘶遤⹛ٌرٕ㼪Ⰵ 27/58 -
湫㉏さׇ ! ➭➂ך瘶ִ⢪ג鍑ַׇֿהד姻׃ְ瘶ִ䒷ֹ⳿ׅ o ⢽ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״俑剅吤姻 4PZMFOUכծ'JOE'JY7FSJGZך媮ꥡד吤姻㹋倵 ⎼ 'JOE㉏겗ָ֮皘䨽ך嗚⳿ ⎼ 'JY吤姻
⎼ 7FSJGZ 吤姻铎ך嗚⳿ ➭➂ך瘶ִ⢪ג鍑ַׇ Figure 2. Crowdproof is a human-augmented proofreader. J. Bigham et al.: Soylent: A word processor with a crowd inside, In UIST, 2010. 穠卓 穠卓 剑穄穠卓 29/58 -
荈䊹鎍姻 ! 4UFQ⡦鋅ׇ׆ח㔐瘶ׇׁ ! 4UFQ➭ך㔐瘶罏ך窟さ鍑䲿爙׃ծⱄ䏝㔐瘶ׇׁ N. B. Shah and D.
Zhou: No oops, you won’t do it again: Mechanisms for self-correction in crowdsourcing, In ICML, 2016. ➭➂ך瘶ִ䲿爙׃荈䊹鎍姻⤛ׅ չ؟ٝؿٓٝءأ؝ךⱖ溪鼅דְֻׁպ չ֮זך㔐瘶ָ麩גְה䙼ֲ㜥さכ⥜姻׃גְֻׁպ ֮זך㔐瘶 ➭ך➂ך㔐瘶 30/58 -
⥜姻ה鐰⣣ך鷵如㹋遤 ! 5VS,POUSPM䧭卓暟ך⥜姻ה鐰⣣粸鵤ׅ،ٕ؞ٔؤي o ⥜姻٥鐰⣣ך銲♶銲 ㅷ颵朐䡾הׅ鿇ⴓ錁庠وٕ؝ؿ寸㹀麓玎ד寸㹀 o ⼧ⴓח⥜姻דֹהֿד穄✪ 䧭卓暟 ⥜姻ָ
䗳銲 ⥜姻 㹋倵 鐰⣣ָ 䗳銲 ⥜姻䖓ך 嫰鯰 穄✪ YES NO NO YES P. Dai et al.: Decision-theoretic control of crowd-sourced workflows, In AAAI, 2010. ،ٕ؞ٔؤيⵖ䖴ח״⥜姻ה鐰⣣粸鵤ׅ 31/58 ⥜姻䖓ךוַ䱰欽 -
鐰⣣罏腉⸂罋䣁窫㼎鐰⣣ ! 鐰⣣ָ姻׃ְהכꣲזְ ! 鐰⣣罏٥⡲䧭罏ך腉⸂䱿㹀׃♳ד䧭卓暟ךㅷ颵✮庠 ? 䧭卓暟 鐰⣣罏 ㅷ颵 鐰⣣罏ػًٓ٦ة
˖غ؎،أ % ˖⥋걾䚍 % ⡲䧭罏ػًٓ٦ة ˖腉⸂ ( ˖⥋걾䚍 ( ⡲䧭罏 鐰⣣罏腉⸂罋䣁׃䧭卓暟ךㅷ颵✮庠ׅ Y. Baba and H. Kashima: Statistical quality estimation for general crowdsourcing tasks, In KDD, 2013. 32/58 -
鐰⣣罏腉⸂罋䣁窫㼎鐰⣣ ! 䧭卓暟ה鐰⣣ך欰䧭ٌرٕ㼪Ⰵ׃ 鐰⣣穠卓ַ䧭卓暟ךㅷ颵✮庠ׅ o 4UFQ⡲䧭罏ָㅷ颵,,( א䧭卓暟欰䧭 o 4UFQ鐰⣣罏ָ鐰⣣,,(,% 欰䧭ׅ
䧭卓暟ה鐰⣣ך欰䧭麓玎ٌرٕ⻉ qt,a ⇠ N qt,a | µa, 1 a yt,a,r ⇠ N yt,a,r | qt,a + ⌘r, 1 r ⡲䧭罏ך腉⸂ 鐰⣣罏ךغ؎،أ ㅷ颵 ㅷ颵 33/58 -
鐰⣣罏腉⸂罋䣁♧㼎嫰鯰 ! 鐰⣣ָ♧㼎嫰鯰ד遤㜥さח 鐰⣣罏ך腉⸂䱿㹀׃䧭卓暟ךㅷ颵✮庠 ♧㼎嫰鯰ך穠卓ַ䧭卓暟ךㅷ颵✮庠ׅ ? "ךㅷ颵 ⡲䧭罏 T. Sunahase,
Y. Baba and H. Kashima, Pairwise HITS: Quality estimation from pairwise comparisons in creator-evaluator crowdsourcing process, In AAAI, 2017 䧭卓暟 " 䧭卓暟 # " " # # # " ? #ךㅷ颵 34/58 -
鐰⣣罏腉⸂罋䣁♧㼎嫰鯰 ! 1BJSXJTF)*54 )*54،ٕ؞ٔؤي䘔欽׃䧭卓暟ךㅷ颵✮庠 ! ⟃♴ך䩛竲ֹ粸鵤ׅ o 4UFQ鐰⣣罏腉⸂0 㔿㹀׃ㅷ颵1 刿倜
o 4UFQㅷ颵1 㔿㹀׃鐰⣣罏腉⸂0 刿倜 qj qj0 = X i2Vj j0 ri X i2Vj0 j ri ח䫎牰׃➂ ri = |{(j j0) 2 Vi | qj > qj0 }| |Vi | 姻׃ְ鐰⣣ ךⶴさ 鐰⣣罏腉⸂ה䧭卓暟ㅷ颵❛✼ח刿倜 35/58 -
㼔㹺涪鋅 ! 㼔䚍ָ䗳銲ז㉏겗דכ 鍑ֽ➂鋅אֽזְה姻׃ְ瘶ִָ䖤זְ o ⢽⼔洽חꟼׅ㉏겗 ! 知⽃ז倯岀姻鍑傀濼㉏겗דך姻瘶桦ד㼔㹺鋅אֽ ! ذأز欽䠐׃Ⰻ㆞ח「ׇֽׁךכ撕꧟זךד
➭ך䩛ַָⵃ欽׃ְ 鍑ֽ➂꧊㔚ך⚥ַ鋅אֽ 2過⚥慬䏝然钠ׅ䗳銲䚍ָ剑넝ְ⼔讒ㅷכוַ " ،أؾٔٝ # ؿٗإىس $ ذؔؿ؍ٔٝ % ؎ٝسًةءٝ 㼔䚍ָ䗳銲ז㉏겗⢽ 37/58 -
㾩䚍ח״㼔㹺涪鋅 ! 㼔䚍ָ㔐瘶罏ך㾩䚍㷕娖ծ㷕鿇瘝ח ⣛㶷ֿׅהכ״ֻ֮ ! 㔐瘶罏ך㾩䚍ⵃ欽׃ג㼔㹺涪鋅 o 㔐瘶罏ך㾩䚍كؙزָٕ♷ִ o 㔐瘶罏腉⸂㾩䚍كؙزٕך简䕎ㄤד邌植׃
ぐ㾩䚍ח㼎ׅꅾ䱿㹀 㼔㹺חⰟ鸐ׅ㾩䚍䩛ַָהׅ aj = (aj1, aj2, . . . , ajn) ⼔㷕鿇⳿魦 穗幥㷕鿇⳿魦 ✓j ⇠ N w>aj, 2 )-JFUBM5IF8JTEPNPGNJOPSJUZEJTDPWFSJOHBOEUBSHFUJOHUIFSJHIUHSPVQPGXPSLFSTGPS DSPXETPVSDJOH *O888 腉⸂ 㾩䚍ךꅾ 38/58 -
嗚稊ؙؒٔח״㼔㹺涪鋅 ! 嗚稊鸬⹛㘗䎢デַ㉏겗פ钷㼪׃ 㼔㹺ָ״ֻ⢪ֲ嗚稊ؙؒٔ涪鋅 Figure 3: Example ad to attract
users time a user clicks on the a we record a conversion ev the advertising system. T the system to optimize th mizing the number of con contribution yield, instead the number of clicks. Although optimizing fo Figure 1: Screenshot of the Quizz system. healthline 1*QFJSPUJT BOE&(BCSJMPWJDI2VJ[[UBSHFUFEDSPXETPVSDJOHXJUIBCJMMJPO QPUFOUJBM VTFST *O888 㼔㹺ָ״ֻ⢪ֲ嗚稊ؙؒٔ䩛ַָהׅ 姻瘶桦؝ٝغ٦آّٝ桦 ה׃ג⢪欽 39/58 -
姻鍑劢濼ך㉏겗⢪㼔㹺涪鋅 ! 姻鍑䌢ח欽䠐דֹהכꣲזְ ! 㢳侧寸鍑ד➿欽׃ְָ 㼔㹺ָ㼰侧崢הז㉏겗דכ㢳侧寸כ㣟侁ׅ C A A E
D E A D A C A B B C A A D B E A D E A C E D A A E E D E D C D E A E D E A D D B C D A A A A A A A A A A A A A A A A A A B B B B B B B B B B B B B B B B B B B B B B B C C C C C C C C C C C C C C C C C C C C C C D D D D D D D D D D D D D D E E E E E E E E E E E E E E E E E E Question Worker { Experts Question Question A D A B C C D MV ꬊ㼔㹺 㢳 侧 寸 ㉏겗 㼔㹺 䌢ח姻瘶 ٓٝتيח㔐瘶 ˟"։&ך䫛 Ⰻגך㉏겗ד"ָ姻瘶 1 2 3 4 铎瘶ָ涪欰 J. Li, Y. Baba and H. Kashima: Hyper Questions: Unsupervised Targeting of a Few Experts in Crowdsourcing, In CIKM, 2017. 姻鍑劢濼ך㉏겗ח㼎ׅ㔐瘶ַ㼔㹺鋅אֽ 㢳侧寸ָ㣟侁ׅ噰畭ז⢽ 40/58 -
姻鍑劢濼ך㉏겗⢪㼔㹺涪鋅 ! 㼔㹺ず㡦ך㔐瘶כ醱侧㉏ד♧荜׃װְָׅ ꬊ㼔㹺ず㡦ך㔐瘶כ♧荜׃חְֻ ! 醱侧㉏תהג♧אך㉏겗הז׃ծ )ZQFSRVFTUJPOח㼎ׅ㔐瘶ד㢳侧寸《 ㉏겗 1 2
3 4 (1, 2, 3) (1, 2, 4) (1, 3, 4) (2, 3, 4) B B B )ZQFS RVFTUJPO ח㢌䳔 A A A A A A A A A A A B B B B C D D E E ਪᕚ 㙢㖽㘞 {1, 2, 3} AAA AAA {1, 2, 4} AAA AAA {1, 3, 4} {2, 3, 4} AAA AAA AAA AAA ABD ABA ADA BDA EBC EBA ECA BCA EBB EBD EBD BBD ൴ਪᕚ 㙢㖽㘞 ൴ਪᕚ 㗸㗮Ϋഈʹ ม౬ 1 εӔ 2 A A A A A A A A A A A B B B B C D D E E ਪᕚ 㙢㖽㘞 {1, 2, 3} AAA AAA {1, 2, 4} AAA AAA {1, 3, 4} {2, 3, 4} AAA AAA AAA AAA ABD ABA ADA BDA EBC EBA ECA BCA EBB EBD EBD BBD ൴ਪᕚ 㙢㖽㘞 ൴ਪᕚ 㗸㗮Ϋഈʹ ม౬ 1 εӔ 2 㼔㹺ず㡦ך㔐瘶ָ♧荜ֿׅהⵃ欽׃㢳侧寸 㼔㹺ך㔐瘶כ♧荜׃ 㢳侧崢הז ꬊ㼔㹺ך㔐瘶 כלאֻ ˘ ˘ 41/58 -
䱿讂ח״㼔㹺涪鋅 ! %"31"/FUXPSL$IBMMFOHF o Ⰻ碛皘䨽ח饔ְ괏菹鏣縧 o 剑ⴱחⰋגך괏菹鋅אֽث٦يח颣ꆃ o ⮚ث٦ي չ괏菹鋅אֲֽז➂պ
〡؝ىד涪鋅 +5BOHFUBM3FGMFDUJOHPOUIF%"31"SFECBMMPPODIBMMFOHF *O $PNNVOJDBUJPOTPGUIF"$. contributed articles pla use pu nin ins et ing dis on Ma use cro wa cia on ba the tea in pro firs Figure 1. Locations in the DARPA Red Balloon Challenge. Figure 2. Example recursive incentive-structure process for the MIT team. ➭➂ַך䱿讂ח㛇בְג㼔㹺鋅אֽ 42/58 -
䱿讂ח״㼔㹺涪鋅 ! 鄃䱿讂罏ָ괏菹涪鋅׃㜥さծ 䱿讂罏ח㜠ꂹָⴓꂁׁ״ֲח鏣鎘 䱿讂 䱿讂 䱿讂 괏菹ך 㜠ꂹ 䱿讂ך
㜠ꂹ 䱿讂ך 㜠ꂹ 䱿讂ך 㜠ꂹ 괏菹ך 㜠ꂹ 剣渣ז䱿讂ָ䖤״ֲח㜠ꂹ鏣鎘 43/58 괏菹涪鋅 괏菹涪鋅 -
䱿讂ח״㼔㹺涪鋅 ! ⟰噟ⰻדךةأؙⶴ䔲穗騟ⴓ匿 o ةأؙⶴ䔲荈ⴓד鍑ֽזְةأؙ➭罏ח⣛걾 ! אךⶴ䔲ػة٦ٝ穈さׇגٌرٕ⻉ o 荈魦הך㼔䚍ך䊴ח㛇בֻⶴ䔲 ⎼
㼔䚍ָ鵚ַ׆黅ַ׆ך➂ח ⣛걾ׅ⫘ぢָ֮ o 植㖈ךةأؙꆀח㛇בֻⶴ䔲 o ٓٝتيⶴ䔲 ! 㼔䚍כةأؙ⚥ך⽃铂ה麓ך㸣✪㾶娖ַ䱿㹀 ➂ָ㼔㹺䱱ׅ麓玎ٌرٕ⻉ׅ H. Sun et al.: Analyzing expert behaviors in collaborative networks, In KDD, 2014. customers. A task is posted e network from an expert to When an expert cannot solve ., where to transfer a task) is y affect the completion time empt to deduce the cognitive model the decision making of s where a routing decision is patterns. interesting phenomenon that ask to someone whose knowl- or too different from his own. xpertise difference based rout- e formalize multiple routing nt both rational and random a generative model to com- of tasks, our model not only ences very well, but also accu- n time. Under three different significantly outperforms al- han 75% accuracy gain. In In service businesses, a service provider of expert network where service agents coll problems reported by customers. Bugzilla ing system where software developers joint ed bugs in projects. In a classic collaborati receiving a task, an expert first tries to so the expert will route the task to another is completed until it reaches an expert w solution. ( $ % & ' ) * + ƚϭ ƚϮ Figure 1: A Sample Collaborativ Figure 1 shows a sample collaborative n 44/58 -
⼿锃㉏겗ⴓⶴ ! 5VSLPNBUJD㉏겗ⴓⶴ٥鍑寸꧊㔚ד遤ֲ➬穈 o 4UFQ㉏겗ⴓⶴׅ ⎼ ⼧ⴓ⽃秪ז鿇ⴓ㉏겗חזתדⴓⶴ粸鵤ׅ o 4UFQぐ鿇ⴓ㉏겗➂ָ鍑ֻ o
4UFQ鿇ⴓ㉏겗ך鍑꧊秈׃剑穄涸ז鍑⳿⸂ A. Kulkarni et al.: Collaboratively crowdsourcing workflows with Turkomatic, In CSCW, 2012. ㉏겗ⴓⶴ꧊㔚ד遤ְ㣐鋉垷ז㉏겗鍑寸ׅ ㉏겗⢽չջ㎳倯⤑ռכ姻׃ְַպָذ٦وך㼭锷俑㛁瘗 鿇ⴓ㉏겗չ㎳כ״ֻזְպהְֲ媮衅㛁瘗 鿇ⴓ㉏겗չ䗳銲ז㎳ָ֮պהְֲ媮衅㛁瘗 ˘ 47/58 -
⼿锃剑黝⻉ ! 鍑ָ侧⦼ד邌ׁ㉏겗חֶֽ⼿锃剑黝⻉ o ٓٝتيח㔐瘶罏鼅ע o 㔐瘶罏כ♧㹀眔㔲ⰻך㥨ך㜥䨽ח鍑⹛ַׅ o 眔㔲杞זָ鍑ך刿倜粸鵤ׅ !
圫ղז؛٦أדך 勲䚍ָ 椚锷涸٥㹋꿀涸ח爙ׁגְ N. Garg et al.: Collaborative optimization for collective decision-making in continuous spaces, In WWW, 2017. 然桦涸⺟ꂁ꣬♴岀ח⦺ג꧊㔚ד鍑䱱稊ׅ ㉏겗⢽✮皾ך寸㹀 猰㷕䮶莆顤 爡⠓⥂ꥺ顤 植㖈ך鍑 鍑⹛ַׅ 48/58 -
،؎ر،ך ꧊ ! *OOPDFOUJWF 暴㹀ך㉏겗ך鍑،؎ر،꧊ׅفٓحزؿؓ٦ي o ㉏겗⢽ չؿح稆ػؐت٦瑞孡⚥ח䭁侔ׇׁ׆ח娎熊ֹ ثُ٦ـחⰅ倯岀כպ o ،؎ر،ָ䱰欽ׁה颣ꆃָ佄䩪
꧊㔚ַ،؎ر،꧊ׅفٓحزؿؓ٦ي *OOPDFOUJWF 49/58 -
،؎ر،ך䱱稊 ! ꧊㔚ד،؎ر،ך鼅䫙٥❛⿷粸鵤ׅ 鼋⠗涸،ٕ؞ٔؤيח⦺ג،؎ر،䱱稊ׅ ⢽㶨⣘ぢֽך喱㶨ךرؠ؎ٝ L. Yu and J. V.
Nickerson: Cooks or cobblers? crowd creativity through combination, In CHI, 2011. 4UFQ痥⚅➿ך⦐⡤ぐ荈ָ欰䧭 4UFQ䫎牰ד⮚葺⦐⡤ 鼅䫙׃如⚅➿ך⦐⡤ ❛⿷ח״欰䧭 4UFQ鼅䫙٥❛⿷粸鵤ׅ 50/58 -
،؎ر،겲⡂䏝ך〳鋔⻉ ! ،؎ر،겲⡂䏝ך〳鋔⻉ ⡂،؎ر،鵚ֻחꂁ縧ֿׅהדⰋ⡤⫷䪾䳢佄䴂 ! ،؎ر،겲⡂䏝꧊㔚ָⴻ倖 o 湱㼎涸ז겲⡂䏝㼥י չ"ה#ךוך倯ָ $ח⡂גְպ
P. Siangliulue et al.: Toward collaborative ideation at scale: Leveraging ideas from others to generate more creative and diverse ideas, In CSCW, 2015. ،؎ر،鋅ׇさֲֿהד✼ְך涪䟝⤛ׅ " # $ 51/58 -
،؎ر،겲⡂䏝٥⮚⸋ך〳鋔⻉ ! 겲⡂䚍ֽדכזֻ⮚⸋䲿爙ֿׅהד ،؎ر،ך鼅䫙佄䴂 ! 珏겲ך㉏さׇח״겲⡂䏝٥⮚⸋꧊㔚דⴻ倖 o ،؎ر،ل،ך겲⡂䏝鐰⣣ o ،؎ر،ل،ך⮚⸋鐰⣣
J. Li, Y. Baba, and H. Kashima: Simultaneous clustering and ranking from pairwise comparisons, In IJCAI, 2018 (to appear). ،؎ر،겲⡂䏝٥⮚⸋꧊㔚דⴻ倖׃〳鋔⻉חⵃ欽 ㉏겗⢽չ㹀劍ذأزדךؕٝصּؚٝחכպ "չؕٝصؚٝכ⽯鷌㷕պ #չ㉏겗ךꂁ㢌ִպ "չ娄ֹ㔐ג湊鋔պ #չ欰䖝׀הח㉏겗㢌ִպ "ה#כ⡂גְ "ה#כ⡂גְזְ "ך倯ָ葺ְ #ך倯ָ葺ְ 53/58 -
،؎ر،겲⡂䏝٥⮚⸋ך〳鋔⻉ ! 겲⡂䏝٥⮚⸋ך鐰⣣穠卓ꟼ鸬בֽ倯ך䱿㹀⸬桦⻉ o չ⡂גְ،؎ر،ず㡦כずְֻׄ葺ְպ o չ葺ְ،؎ر،ה䝤ְ،؎ر،כ⡂גְזְպ ! ،؎ر،ך悵㖈暴䗙 ∈
ℝ5➜׃겲⡂䏝ה葺ׁꟼ鸬בֽ ! 겲⡂䏝٥⮚⸋ך鐰⣣穠卓ַ 0 0 ה䱿㹀׃ 〳鋔⻉חⵃ欽 悵㖈暴䗙ך㼪Ⰵד겲⡂䏝٥⮚⸋ך䱿㹀⸬桦⻉ ⌧i = w > xi ،؎ر،Jך葺ׁ ،؎ر،JהKך 겲⡂䏝 54/58 ij = exp k xi xj k2 <latexit sha1_base64="BJwBkKHX7X3gTfshn+WrZVYnYIQ=">AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/</latexit> <latexit sha1_base64="BJwBkKHX7X3gTfshn+WrZVYnYIQ=">AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/</latexit> <latexit sha1_base64="BJwBkKHX7X3gTfshn+WrZVYnYIQ=">AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/</latexit> <latexit sha1_base64="BJwBkKHX7X3gTfshn+WrZVYnYIQ=">AAACbnicbVDLbhMxFHWGVwmvFCQ2CNUiRQqLRjMVUtkgVbBhWSTSVoqHke25k7j1Y2R7SiJ3PoWvYQsfwF/wCTjpLGjKkSyde869ur6H1VI4n6a/e8mt23fu3tu633/w8NHjJ4Ptp8fONJbDhBtp7CmjDqTQMPHCSzitLVDFJJyw848r/+QCrBNGf/HLGnJFZ1pUglMfpWJwQOq5KII4a/F7TGBREwmVH+2RS8JUWLSFwHu4o2fk8us+sWI292+KwTAdp2vgmyTryBB1OCq2ezukNLxRoD2X1LlpltY+D9R6wSW0fdI4qCk/pzOYRqqpApeH9YUtfh2VElfGxqc9Xqv/TgSqnFsqFjsV9XO36a3E/3nTxlfv8iB03XjQ/GpR1UjsDV7FhUthgXu5jIRyK+JfMZ9TS7mPofaJhm/cKEV1GQi7AN5OszwQxcwi1kaWq4V4d5jtttcODExt1FGIkWabAd4kx/vjLB1nn98ODz904W6hF+gVGqEMHaBD9AkdoQni6Dv6gX6iX70/yfPkZbJz1Zr0upln6BqS0V8ATr2/</latexit> -
תה ➂䊨濼腉ח״㉏겗鍑寸 ➂ָ佄䴂 ➂ח״㉏겗鍑寸 ➂䊨濼腉ָ佄䴂 㼔㹺涪鋅 ⼿锃佄䴂 ⚛㉏さׇ 湫㉏さׇ 知⽃٥⽃秪ז㉏겗
㔭ꨇ٥醱꧟ז㉏겗 ➂ך꧊㔚ה➂䊨濼腉ח״㉏겗鍑寸ך䩛岀稱➜ 56/58 -
➙䖓ך铬겗 ! ؼُ٦وٝ؝ٝؾُذ٦ءّٝךؚٓٝسثٍٖٝآ o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״灇瑔涪 ⎼ 僥歗չٖٗٝخؓךؔ؎ٕպ 稆➂㣗㮑ָ俑柃锃叨ה㼔㹺הך陽锷ח״ 䜁㶨ך氺孡ך屚洽岀涪鋅 o
ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״鄩ⴻ㆞鄩ⴻ o ؼُ٦وٝ؝ٝؾُذ٦ءّٝח״佟瘻寸㹀 넝䏝ד醱꧟ז㹋爡⠓㉏겗פך黝欽 57/58