Human+AI Crowd Task Assignment Considering Result Quality Requirements (HCOMP2021)

Masaki Kobayashi, Kei Wakabayashi, and Atsuyuki Morishima, "Human+AI Crowd Task Assignment Considering Result Quality Requirements", Proc. of the 9th AAAI Conference on Human Computation and Crowdsourcing (HCOMP2021)., Online conference, November 2021, pp. 97-107.

Transcript

1. )VNBO "*
   $SPXE
   5BTL
   "TTJHONFOU
   $POTJEFSJOH
   3FTVMU
   2VBMJUZ
   3FRVJSFNFOUT
   .BTBLJ
   ,PCBZBTIJ
   ,FJ
   8BLBCBZBTIJ
   BOE
   "UTVZVLJ
   .PSJTIJNB JST CREST CyborgCrowd Immediate, United, Optimized
2. 1PJOUT
   PG
   5IJT
   5BML  2VBMJUZ
   DPOUSPM
   PG
   UBTL
   SFTVMUT
   JO
   UIF
   )VNBO
   
   "*
   $SPXE
   XJMM
   CF
   BO
   JNQPSUBOU
   JTTVF
   JO
   UIF
   OFBS
   GVUVSF  &YJTUJOH
   UFDIOJRVFT
   BSF
   TVCPQUJNBM
   JO
   UFSNT
   PG
   FGGJDJFODZ
   BOE
   TDBMBCJMJUZ  8F
   QSPQPTF
   UBTL
   BTTJHONFOU
   BMHPSJUINT
   XIJDI
   UBLF
   BEWBOUBHF
   PG
   UIF
   TXFFU
   TQPUT
   PG
   FBDI
   CMBDLCPY
   "*
   XPSLFS  0VS
   NFUIPE
   DBO
   SFEVDF
   IVNBO
   XPSLFS
   BTTJHONFOUT
   XIJMF
   TBUJTGZJOH
   UIF
   RVBMJUZ
   SFRVJSFNFOU
   JO
   UIF
   QSFTFODF
   PG
   NBOZ
   CMBDL CPY
   "*
   XPSLFST 
3. 4PNF
   "QQMJDBUJPOT
   /FFE
   UP
   1SPDFTT
   #JH
   %BUB
   "DDVSBUFMZ
   CVU
   *NNFEJBUFMZ FH
   /BUVSBM
   EJTBTUFS
   SFTQPOTF 
   $SPXETPVSDJOH
   JT
   FGGFDUJWF
   GPS
   TVDI
   BQQT  "FSJBM
   1IPUP -BCFMJOH -BCFMFE
   "FSJBM
   1IPUP (PWFSONFOU 3FRVFTUFS /FFE
   FWJEFODF
   JNNFEJBUFMZ
    /FYU
   "DUJPO
4. 0VUTPVSDJOH
   "*
   QSPHSBN
   %FWFMPQNFOU
   JT
   $PNNPO
   1SBDUJDF  4NBSUFS
   "*
   .PEFM %BUBTFU $SPXETPVSDFE $SPXETPVSDJOH $SPXE
   8PSLFST .BDIJOF
   -FBSOJOH
   .PEFMT FH
   ,BHHMF
   "*$SPXE
5. $PNCJOJOH
   )VNBOT
   BOE
   "*
   JT
   OPU
   "MXBZT
   &BTZ 8PSLGMPX
   UP
   MBCFM
   JNBHFT
   VTJOH
   DSPXETPVSDJOH
   BOE
   "*
   XJUI
   
   RVBMJUZ
   BDDVSBDZ   #FHJO $SPXETPVSDF
   GPS GFX
   JNBHFT
   UP
   NBLF
   B
   EBUBTFU 4UBSU
   B
   DPNQFUJUJPO
   GPS
   "*
   EFWFMPQNFOU *T
   UIF
   "*
   BDDFQUBCMF &OE :FT "TTJHO
   UIF
   "*
   GPS
   SFNBJOJOH
   JNBHFT 6TF
   IVNBOT
   GPS
   "--
   JNBHFT /P 3FEFTJHO
   UIF
   XPSLGMPX *NQSPWF
   UIF
   EBUBTFU • "DUJWF
   MFBSOJOH • &OTFNCMF
   MFBSOJOH • "VUP.- • .-
   XJUI
   SFKFDU
   PQUJPOT %FDJTJPO
   NBLJOH $PNCJOJOH
   UIFN
   XFMM
   UP
   BDIJFWF
   UIF
   SFRVJSFE
   RVBMJUZ
   JT
   IBSE
   OPU
   TDBMBCMF )PX
   DBO
   XF
   BDIJFWF
   UIF
   SFRVJSFE
   RVBMJUZ
6. )VNBO "*
   $SPXE 8F
   QSPQPTF
   UP
   EFBM
   XJUI
   "*
   NPEFMT
   BT
   CMBDLCPY
   "*
   XPSLFST  0VS
   "QQSPBDI 
   )PX
   DBO
   XF
   DPOUSPM
   UIF
   UBTL
   SFTVMU
   RVBMJUZ &WBMVBUF
   UIF
   RVBMJUZ
   PG
   XPSLFST 5BTL
   BTTJHONFOU #MBDL#PY "*
   8PSLFS EFWFMPQ 2VBMJUZ
   SFRVJSFNFOU
    5BTLT 3FRVFTUFS
7. 0VS
   $POUSJCVUJPOT  8F
   EFGJOF
   B
   OPWFM
   )VNBO "*
   $SPXE
   5BTL
   "TTJHONFOU
   1SPCMFN
   )"$5"1 UIBU
   BTTJHOT
   B
   GJYFE
   TFU
   PG
   UBTLT
   UP
   IVNBO
   BOE
   "*
   XPSLFST  8F
   QSPQPTF
   UXP
   BMHPSJUINT UIBU
   QSPWJEF
   FGGJDJFOU
   TPMVUJPOT
   UP
   UIF
   )"$5"1
   XJUI
   TPNF
   UIFPSFUJDBM
   HVBSBOUFFT  8F
   DPOEVDUFE
   BO
   FYUFOTJWF
   TFU
   PG
   FYQFSJNFOUT
   VTJOH
   PQFO
   CFODINBSL
   BOE
   SFBMXPSME
   EBUBTFUT
   0VS
   DPEF
   BOE
   EBUB
   BSF
   BWBJMBCMF
   BU
   IUUQTHJUIVCDPNDSPXEV)"$5"1'SBNFXPSL 
8. 1PJOUT
   PG
   5IJT
   5BML  2VBMJUZ
   DPOUSPM
   PG
   UBTL
   SFTVMUT
   JO
   UIF
   )VNBO
   
   "*
   $SPXE
   XJMM
   CF
   BO
   JNQPSUBOU
   JTTVF
   JO
   UIF
   OFBS
   GVUVSF  &YJTUJOH
   UFDIOJRVFT
   BSF
   TVCPQUJNBM
   JO
   UFSNT
   PG
   FGGJDJFODZ
   BOE
   TDBMBCJMJUZ  8F
   QSPQPTF
   UBTL
   BTTJHONFOU
   BMHPSJUINT
   XIJDI
   UBLF
   BEWBOUBHF
   PG
   UIF
   TXFFU
   TQPUT
   PG
   FBDI
   CMBDLCPY
   "*
   XPSLFS  0VS
   NFUIPE
   DBO
   SFEVDF
   IVNBO
   XPSLFS
   BTTJHONFOUT
   XIJMF
   TBUJTGZJOH
   UIF
   RVBMJUZ
   SFRVJSFNFOU
   JO
   UIF
   QSFTFODF
   PG
   NBOZ
   CMBDL CPY
   "*
   XPSLFST 
9. )VNBO
   
   "*
   $SPXE
   5BTL
   "TTJHONFOU
   1SPCMFN *OQVU 
   
   5BTLT
   "*
   8PSLFST
   BOE
   UIF
   2VBMJUZ
   SFRVJSFNFOU
   NJOJNVN
   BDDVSBDZ 0VUQVU 
   
   "
   TFU
   PG
   UBTL
   BTTJHONFOUT
   UBTL
   XPSLFS
   QBJST $POTUSBJOUT
   
   
   
   
   
   
   4BUJTGZ
   UIF
   RVBMJUZ
   SFRVJSFNFOU
   
   .BYJNJ[F
   BTTJHONFOU
   UP
   "*
   XPSLFST  0VS
   "QQSPBDI )"$5"1 " # " % " # $ # AI Workers Human Workers Tasks while keeping the accuracy requirement Task Cluster: subset of tasks Task (Label)
10. 4PMWF
    UIF
    )"$5"1
    XJUI "DUJWF
    -FBSOJOH<>
    CBTFE
    "TTJHONFOU
    • "TTJHO
    UBTLT
    UP
    IVNBOT
    VOUJM
    UIF
    "*
    XPSLFS
    BDIJFWFT
    UIF
    UISFTIPME
    BMMPSOPUIJOH  • .-
    NPEFM
    DBO
    RVFSZ
    TPNF
    VOMBCFMFE
    EBUB
    QPJOUT
    UP
    JNQSPWF
    UIFJS
    QFSGPSNBODF  <>
    :BO
    :BO
    3PNFS
    3PTBMFT
    (MFOO
    'VOH
    BOE
    +FOOJGFS
    (
    %Z
    "DUJWF
    MFBSOJOH
    GSPN
    DSPXET
    *$.-` DPNQMFUFE
    UBTLT IVNBO
    MBCFMT
    "MMPS/PUIJOH 0OMZ
    )VNBOT #FHJO "MM
    UBTLT
    XFSF
    MBCFMFE &OE :FT /P "TTJHO
    DIPTF
    UBTLT
    UP
    IVNBOT 5SBJO
    "*
    XPSLFST CZ
    IVNBO
    MBCFMT "*
    8PSLFS
    DIPPTF
    /
    VOMBCFMFE
    UBTLT "TTJHO
    UBTLT
    UP
    UIF
    "*
    XPSLFST &WBMVBUF
    "*
    8PSLFS
    BDDVSBDZ "DDFQU 3FKFDU
11. 4IPSUDPNJOHT
    PG
    &YJTUJOH
    .FUIPET  5IF
    NFUIPE
    BTTJHOT
    UBTLT
    UP
    "*T
    CZ
    UIF
    lBMMPSOPUIJOHz TUSBUFHZ
    DPOTFRVFOUMZ
    XF
    DBOOPU
    FYQMPJU
    BOZ
    PG
    UIF
    CFOFGJUT
    PG
    UIF
    "*
    CFGPSF
    UIF
    "*
    JT
    BDDFQUFE
     5IF
    NFUIPE
    BTTVNFT
    B
    TJOHMF
    .-
    NPEFM
    DPOTFRVFOUMZ
    XF
    DBOOPU
    MFWFSBHF
    NBOZ
    PUIFS
    "*T JO
    UIF
    DSPXE  5P
    HVBSBOUFF
    UIF
    RVBMJUZ
    PG
    UBTL
    SFTVMUT
    JU
    OFFET
    UP
    DPNCJOF
    PUIFS
    NFUIPET
    FH
    TUBUJTUJDBM
    UFTUJOH
    XJUI
    SBOEPN
    TBNQMFT  
    8F
    QSPQPTF
    UIFPSFUJDBMMZ
    HVBSBOUFFE
    BMHPSJUINT
    CBTFE
    PO
    UIF
    EJWJEFBOEDPORVFS
    TUSBUFHZ
    UP
    VUJMJ[F
    EJWFSTF
    "*
    XPSLFST 
12. 1PJOUT
    PG
    5IJT
    5BML  2VBMJUZ
    DPOUSPM
    PG
    UBTL
    SFTVMUT
    JO
    UIF
    )VNBO
    
    "*
    $SPXE
    XJMM
    CF
    BO
    JNQPSUBOU
    JTTVF
    JO
    UIF
    OFBS
    GVUVSF  &YJTUJOH
    UFDIOJRVFT
    BSF
    TVCPQUJNBM
    JO
    UFSNT
    PG
    FGGJDJFODZ
    BOE
    TDBMBCJMJUZ  8F
    QSPQPTF
    UBTL
    BTTJHONFOU
    BMHPSJUINT
    XIJDI
    UBLF
    BEWBOUBHF
    PG
    UIF
    TXFFU
    TQPUT
    PG
    FBDI
    CMBDLCPY
    "*
    XPSLFS  0VS
    NFUIPE
    DBO
    SFEVDF
    IVNBO
    XPSLFS
    BTTJHONFOUT
    XIJMF
    TBUJTGZJOH
    UIF
    RVBMJUZ
    SFRVJSFNFOU
    JO
    UIF
    QSFTFODF
    PG
    NBOZ
    CMBDL CPY
    "*
    XPSLFST 
13. "*
    8PSLFSXJTF
    &WBMVBUJPO CBTFMJOF  5BTL
    $MVTUFSXJTF
    
    &WBMVBUJPO
    QSPQPTFE &WBMVBUJPO
    UBSHFU 5BTL "OPUIFS
    UBTL
    DMVTUFS 5BTL
    DMVTUFS 2VBMJUZ
    3FRVJSFNFOU
     "DDVSBDZ
     
    3FKFDU "DDVSBDZ
     
    3FKFDU "DDVSBDZ
     
    "DDFQU 
    5BTL
    DMVTUFSXJTF
    DBO
    BTTJHO
    NPSF
    UBTLT
    UP
    NVMUJQMF
    "*T 0VS
    *EFB
14. $MVTUFS
    XJTF
    5FTUCBTFE
    "TTJHONFOU
    $5" $5"
    FWBMVBUFT
    UBTL
    DMVTUFST
    GSPN
    "*
    XPSLFST
    VTJOH
    IVNBO
    XPSLFS
    MBCFMT  #FHJO "TTJHO
    VOMBCFMFE
    UBTLT
    UP
    IVNBOT 5SBJO
    "*
    XPSLFST CZ
    IVNBO
    MBCFMT "MM
    UBTLT
    XFSF
    MBCFMFE "TTJHO
    UBTLT
    UP
    UIF
    "*
    XPSLFST &OE :FT /P 'PS
    FBDI
    UBTL
    DMVTUFS 4UBUJTUJDBM
    UFTU "DDFQU 3FKFDU DPNQMFUFE
    UBTLT IVNBO
    MBCFMT
    %JWJEFBOE$PORVFS "MMPS/PUIJOH 0OMZ
    )VNBOT 1SPQPTFE
    .FUIPE
    
15. $MVTUFS
    XJTF
    5FTUCBTFE
    "TTJHONFOU
    $5" 5IF
    RVBMJUZ
    PG
    UIF
    UBTL
    SFTVMUT
    GSPN
    "*
    XPSLFST TBUJTGJFT
    UIF
    RVBMJUZ
    SFRVJSFNFOU  (MPCBM
    5FTUCBTFE
    "TTJHONFOU
    (5" 5IF
    RVBMJUZ
    PG
    UIF
    PWFSBMM
    UBTL
    SFTVMUT TBUJTGJFT
    UIF
    RVBMJUZ
    SFRVJSFNFOU 
    (5"
    BTTJHO
    NPSF
    UBTLT
    UP
    "*
    XPSLFST &WBMVBUJPO
    UBSHFU 5BTL 5BTL
    DMVTUFS "DDFQUFE
    UBTL
    DMVTUFS T "TTJHOFE
    UP
    IVNBO
    XPSLFST 1SPQPTFE
    .FUIPE
     1SPQPTFE
    .FUIPE
     .PSF
    PQUJNBM
16. 1PJOUT
    PG
    5IJT
    5BML  2VBMJUZ
    DPOUSPM
    PG
    UBTL
    SFTVMUT
    JO
    UIF
    )VNBO
    
    "*
    $SPXE
    XJMM
    CF
    BO
    JNQPSUBOU
    JTTVF
    JO
    UIF
    OFBS
    GVUVSF  &YJTUJOH
    UFDIOJRVFT
    BSF
    TVCPQUJNBM
    JO
    UFSNT
    PG
    FGGJDJFODZ
    BOE
    TDBMBCJMJUZ  8F
    QSPQPTF
    UBTL
    BTTJHONFOU
    BMHPSJUINT
    XIJDI
    UBLF
    BEWBOUBHF
    PG
    UIF
    TXFFU
    TQPUT
    PG
    FBDI
    CMBDLCPY
    "*
    XPSLFS  0VS
    NFUIPE
    DBO
    SFEVDF
    IVNBO
    XPSLFS
    BTTJHONFOUT
    XIJMF
    TBUJTGZJOH
    UIF
    RVBMJUZ
    SFRVJSFNFOU
    JO
    UIF
    QSFTFODF
    PG
    NBOZ
    CMBDLCPY
    "*
    XPSLFST 
17. &YQFSJNFOU
    VTJOH
    0QFO
    #FODINBSL
    ,./*45  5BTLT DMBTT
    DMBTTJGJDBUJPO
    GPS
    L
    JNBHFT
    ,./*45 )VNBO
    8PSLFST "OTXFS
    DPSSFDU
    MBCFM "*
    8PSLFST
    
    EJGGFSFOU .-
    NPEFMT
    GSPN TDJLJUMFBSO
    5IFZ
    XPSL
    JO
    QBSBMMFM 2VBMJUZ
    3FRVJSFNFOU 
    
    
     5BTL
    "TTJHONFOU
    "MHPSJUINT  8PSLFSXJTF
    3BOEPN
    4BNQMJOH
    5FTU
    85"  "DUJWF
    -FBSOJOHCBTFE
    "-"
     $5"
    BOE
    (5" 1FSGPSNBODF
    .FUSJDT UBTL
    BTTJHONFOUT
    UP
    "*
    XPSLFST      KXPDQODEHOV      FRPSOHWHGWDVNV :7$      KXPDQODEHOV      $/$      KXPDQODEHOV      &7$      KXPDQODEHOV      *7$ T  T  T  T  .-1$MBTTJGJFS
    &YUSB5SFF$MBTTJGJFS
    -PHJTUJD3FHSFTTJPO ,NFBOT
    %FDJTJPO5SFF$MBTTJGJFS
    47$
    ,/FJHICPST$MBTTJGJFS
    (BVTTJBO1SPDFTT$MBTTJGJFS
    .VMUJOPNJBM/#
    "EB#PPTU$MBTTJGJFS 1BTTJWF"HHSFTTJWF$MBTTJGJFS
    3JEHF$MBTTJGJFS 3JEHF$MBTTJGJFS$7
    $PNQMFNFOU/#
    /FBSFTU$FOUSPJE
18. 3FTVMU
    PG
    0QFO
    #FODINBSL
    ,./*45       KXPDQODEHOV      FRPSOHWHGWDVNV :7$      KXPDQODEHOV      $/$      KXPDQODEHOV      &7$      KXPDQODEHOV      *7$ T  T  T  T  • 'PS
    FBDI
    R
    BMM
    NFUIPET
    XPSLFE
    UP
    NFFU
    UIF
    RVBMJUZ
    SFRVJSFNFOU
    BU
    MFBTU
    • $5"
    BOE
    (5"
    BTTJHOFE
    UBTLT
    UP
    "*
    XPSLFST
    CZ
    EJWJEFBOEDPORVFS
    TUSBUFHZ
    XIJMF
    TBUJTGZJOH
    UIF
    SFRVJSFNFOU
    • (5"
    PVUQFSGPSNFE
    PUIFST
    BU
    UIF
    OVNCFS
    PG
    BTTJHOFE
    UBTLT
    UP
    "*
    XPSLFST %JWJEFBOE$PORVFS "MMPS/PUIJOH #FTU
19. &YQFSJNFOU
    VTJOH
    3FBMXPSME
    %BUBTFUT  5BTLT %JTBTUFS
    EBNBHF
    BTTFTTNFOU
    GPS
    GMPPEFE
    BSFB
    
    UBTLT )VNBO
    8PSLFST "NB[PO
    .FDIBOJDBM
    5VSL "*
    8PSLFST
    
    "*
    8PSLFST 5IF
    POF
    JT
    EFWFMPQFE
    CZ
    BOPOZNPVT 5BTL
    "TTJHONFOU
    "MHPSJUINT  "DUJWF
    -FBSOJOHCBTFE
    "-"
     (5" 1FSGPSNBODF
    .FUSJDT -BCFMFE
    SFTVMUT
20.  ˙ OPOGMPPEFE
    ˙ GMPPEFE
    ˙ DPWFSFE
    CZ
    DMPVET (5"
    GPVOE
    B
    HPPE
    UBTL
    DMVTUFS XIJDI
    JT
    GPS
    OPO GMPPEFE
    QBSUT
    BOE
    JNNFEJBUFMZ
    BTTJHOFE
    UIF
    UBTLT %BSL
    DPMPS
    NFBOT
    UIF
    MBCFM
    GSPN
    IVNBO
    XPSLFST -JHIU
    DPMPS
    NFBOT
    UIF
    MBCFM
    GSPN
    "*
    XPSLFST )VNBO
    XPSLFST
    XFSF
    BTTJHOFE
    UBTLT
    UIBU
    GPDVTFE
    PO
    NPSF
    DPNQMFY
    BSFBT
    
21. 4VNNBSZ • 8F
    JOUSPEVDFE
    B
    OPWFM
    )VNBO "*
    DSPXE
    UBTL
    BTTJHONFOU
    QSPCMFN
    )"$5"1
    BOE
    QSPQPTFE
    UXP
    BMHPSJUINT
    $5"
    (5"  • 0VS
    SFTVMUT
    TVHHFTU
    UIF
    QPTTJCJMJUZ
    PG
    FGGJDJFOU
    UBTL
    QSPDFTTJOH
    CZ
    BQQSPQSJBUFMZ
    TIBSJOH
    UBTLT
    CFUXFFO
    "*
    XPSLFST
    BOE
    IVNBO
    XPSLFST
     'VUVSF
    8PSL *NQSPWF
    BMHPSJUINT
    CZ
    
    QBSBMMFM
    UBTL
    DMVTUFST
    FWBMVBUJPO
    
    QSVOJOH
    UBTL
    DMVTUFST
    UIBU
    BSF
    VOMJLFMZ
    UP
    CF
    BDDFQUFE