$30 off During Our Annual Pro Sale. View Details »

会社訪問アプリ「Wantedly Visit」のデータで見る相互推薦システム / dei...

会社訪問アプリ「Wantedly Visit」のデータで見る相互推薦システム / deim2021-rrs-wantedly-visit

2021年3月2日 DEIM2021 (https://db-event.jpn.org/deim2021/) における技術報告の資料です。

[F21] 情報検索・情報推薦④ 3月2日 10:00 ~ 11:40
https://cms.deim-forum.org/deim2021/program/?oral#/F21

会社訪問アプリ「Wantedly Visit」の実データを用いて、相互推薦システムの既存手法の評価実験を行った上で、出てきた課題に対する改善手法を提案して評価実験を行いその有用性を検証しました。

Yuya Matsumura

March 02, 2021
Tweet

More Decks by Yuya Matsumura

Other Decks in Science

Transcript

  1. ©2021 Wantedly, Inc. 1. ͸͡Ίʹ • ࣗݾ঺հɺձࣾͱϓϩμΫτͷ঺հ • ϓϩμΫτʹ͓͚ΔσʔλαΠΤϯεͷऔΓ૊Έࣄྫ •

    ΞΧσϛΞʹ͓͚Δ׆ಈ 2. ૬ޓਪનγεςϜͱ͸ • ૬ޓਪનγεςϜͷ֓ཁɾಛ௃ • طଘख๏ͷ঺հ 3. ձࣾ๚໰ΞϓϦʮWantedly VisitʯͷσʔλͰݟΔ૬ޓਪનγεςϜ • Wantedly Visit ͷσʔλʹରͯ͠طଘख๏Λద༻࣮ͨ͠ݧɾߟ࡯ • ࣮ݧ݁ՌΛड͚ͨվྑख๏ͷఏҊɾ࣮ݧɾߟ࡯ ձࣾ๚໰ΞϓϦʮWantedly VisitʯͷσʔλͰݟΔ૬ޓਪનγεςϜ
  2. ©2021 Wantedly, Inc. 1. ͸͡Ίʹ • ࣗݾ঺հɺձࣾͱϓϩμΫτͷ঺հ • ϓϩμΫτʹ͓͚ΔσʔλαΠΤϯεͷऔΓ૊Έࣄྫ •

    ΞΧσϛΞʹ͓͚Δ׆ಈ 2. ૬ޓਪનγεςϜͱ͸ • ૬ޓਪનγεςϜͷ֓ཁɾಛ௃ • طଘख๏ͷ঺հ 3. ձࣾ๚໰ΞϓϦʮWantedly VisitʯͷσʔλͰݟΔ૬ޓਪનγεςϜ • Wantedly Visit ͷσʔλʹରͯ͠طଘख๏Λద༻࣮ͨ͠ݧɾߟ࡯ • ࣮ݧ݁ՌΛड͚ͨվྑख๏ͷఏҊɾ࣮ݧɾߟ࡯ ձࣾ๚໰ΞϓϦʮWantedly VisitʯͷσʔλͰݟΔ૬ޓਪનγεςϜ
  3. ©2021 Wantedly, Inc. ✓ দଜ ༏໵ʢYuya Matsumuraʣ ✓ 2018೥3݄ ژ౎େֶେֶӃ৘ใֶݚڀՊ

    म࢜՝ఔमྃ ✓ ΢ΥϯςουϦʔגࣜձࣾ Recommendation νʔϜϦʔυ ✓ σʔλαΠΤϯεɺϓϩμΫτϚωδϝϯτ ✓ Wantedly Visit ʹ͓͚ΔਪનγεςϜͷ։ൃͳͲΛ୲౰ @yu-ya4 @yu__ya4 ࣗݾ঺հ
  4. ©2021 Wantedly, Inc. 1. ͸͡Ίʹ • ࣗݾ঺հɺձࣾͱϓϩμΫτͷ঺հ • ϓϩμΫτʹ͓͚ΔσʔλαΠΤϯεͷऔΓ૊Έࣄྫ •

    ΞΧσϛΞʹ͓͚Δ׆ಈ 2. ૬ޓਪનγεςϜͱ͸ • ૬ޓਪનγεςϜͷ֓ཁɾಛ௃ • طଘख๏ͷ঺հ 3. ձࣾ๚໰ΞϓϦʮWantedly VisitʯͷσʔλͰݟΔ૬ޓਪનγεςϜ • Wantedly Visit ͷσʔλʹରͯ͠طଘख๏Λద༻࣮ͨ͠ݧɾߟ࡯ • ࣮ݧ݁ՌΛड͚ͨվྑख๏ͷఏҊɾ࣮ݧɾߟ࡯ ձࣾ๚໰ΞϓϦʮWantedly VisitʯͷσʔλͰݟΔ૬ޓਪનγεςϜ
  5. ©2021 Wantedly, Inc. User Item ैདྷͷҰൠతͳਪનγεςϜ ૬ޓਪનγεςϜ User(Female) User(Male) User(Job

    Seeker) User(Recruiter/Company) ex. Amazon, Netflix ex. Tinder, Pairs ex. Wantedly, LinkedIn ૬ޓਪનγεςϜʢReciprocal Recommender Systemsʣͱ͸ʁ ʮαʔϏε಺ͷϢʔβΛޓ͍ʹਪન͢ΔγεςϜʯ ਪન ਪન ਪન
  6. ©2021 Wantedly, Inc. User A to User B Preference Score

    User B to User A Preference Score Reciprocal Preference Score Aggregation 1. γεςϜ಺ͷϢʔβ͸ A ͱ B ͷ̎ͭͷάϧʔϓʹ෼͔Ε͓ͯΓɺҟͳΔάϧʔϓͷϢʔβ͕ޓ͍ਪન͞ΕΔ ΋ͷͱ͢Δɻʢe.g. σʔςΟϯάαʔϏεʹ͓͚ΔஉঁɺٻਓαʔϏεʹ͓͚Δٻ৬ऀͱاۀʣ 2. ୯ํ޲ͷᅂ޷ͷେ͖͞Λද͢ Preference Score ΛɺA ͔Β B ΁ͷϢʔβٴͼ B ͔Β A ͷϢʔβͷͦΕͧΕ ʹ͍ͭͯܭࢉ 3. Aggregation Function Λར༻ͯ͠ɺ̎ͭͷ Preference Score Λ૊Έ߹Θͤͯ૒ํ޲ͷᅂ޷ͷେ͖͞Λද͢ Reciprocal Preference Score Λܭࢉ ૬ޓਪનγεςϜʹ͓͚Δᅂ޷ͷ༧ଌ
  7. ©2021 Wantedly, Inc. ✓ [RECON] (Pizzato 2010) ‣ ϢʔβͷϓϩϑΟʔϧ৘ใΛར༻ͨ͠ίϯςϯπϕʔεϑΟϧλϦϯάͰ Preference

    Score Λࢉग़ ‣ Aggregation Function ʹ͸ௐ࿨ฏۉΛར༻ ✓ [RCF] (Xia 2015) ‣ ߦಈཤྺʹجͮ͘ϝϞϦϕʔεͷϢʔβϕʔεڠௐϑΟϧλϦϯάʢk-ۙ๣ʣͰ Preference Score Λࢉग़ ✓ [LFRR](Neve 2019) ‣ ߦಈཤྺʹج͖ͮ࡞੒ͨ͠ User-User ߦྻʹ Matrix Factorization Λద༻ͯ͠ Preference Score Λࢉग़ ‣ Aggregation Function ʹ͍ͭͯɼௐ࿨ฏۉҎ֎ͷؔ਺ʹ͍ͭͯ΋ൺֱ࣮ݧ ✓ [ImRec](Neve 2020) ‣ ϢʔβͷϓϩϑΟʔϧը૾Λར༻ͨ͠ίϯςϯπϕʔεϑΟϧλϦϯάͰ Preference Score Λࢉग़ ૬ޓਪનγεςϜʹ͓͚Δᅂ޷ͷ༧ଌͷطଘख๏
  8. ©2021 Wantedly, Inc. 1. ͸͡Ίʹ • ࣗݾ঺հɺձࣾͱϓϩμΫτͷ঺հ • ϓϩμΫτʹ͓͚ΔσʔλαΠΤϯεͷऔΓ૊Έࣄྫ •

    ΞΧσϛΞʹ͓͚Δ׆ಈ 2. ૬ޓਪનγεςϜͱ͸ • ૬ޓਪનγεςϜͷ֓ཁɾಛ௃ • طଘख๏ͷ঺հ 3. ձࣾ๚໰ΞϓϦʮWantedly VisitʯͷσʔλͰݟΔ૬ޓਪનγεςϜ • Wantedly Visit ͷσʔλʹରͯ͠طଘख๏Λద༻࣮ͨ͠ݧɾߟ࡯ • ࣮ݧ݁ՌΛड͚ͨվྑख๏ͷఏҊɾ࣮ݧɾߟ࡯ ձࣾ๚໰ΞϓϦʮWantedly VisitʯͷσʔλͰݟΔ૬ޓਪનγεςϜ
  9. ©2021 Wantedly, Inc. • طଘख๏ʹ͓͚ΔධՁ࣮ݧ͸σʔςΟϯάαʔϏεʹ͓͚Δ΋ͷ͕ଟ͍ɻ ➡ ٻਓαʔϏεͷ࿮૊ΈͰ͋Δ Wantedly Visit ͷσʔλͰͷ࣮ݧʹҰఆͷՁ஋͕͋ΔͷͰ͸ʁ

    • Wantedly Visit ͸ҰൠతͳٻਓαʔϏεΑΓ΋ΧδϡΞϧʹϢʔβ͕ߦಈ͢Δ αʔϏεͰ͋ΔͨΊɺൺֱతϢʔβͱاۀؒͷߦಈϩά͕ଟ͍ɻ ➡ ڠௐϑΟϧλϦϯάϕʔεͷطଘख๏Λͦͷ··ద༻ͯ͠΋Ұఆͷੑೳ͕ग़ΔͷͰ͸ʁ • ݱࡏͷطଘख๏͸ൺֱతφΠʔϒͳ΋ͷͰ͋ΔͨΊɺվྑͷ༨஍͕͋Δɻ ➡ طଘख๏Ͱͷ࣮ݧΛ௨ͯ͠վྑख๏ΛఏҊͰ͖ΔͷͰ͸ʁ Ϟνϕʔγϣϯ
  10. ©2021 Wantedly, Inc. • Preference Score ༧ଌܭࢉͷͨΊͷΞϧΰϦζϜ • ߦಈཤྺʹجͮ͘ϝϞϦϕʔεͷϢʔβϕʔεڠௐϑΟϧλϦϯά [RCF]

    (Xia 2015) • ߦಈཤྺʹج͖ͮ࡞੒ͨ͠ User-User ߦྻʹ Matrix Factorization Λద༻ [LFRR](Neve 2019) • Aggregation Function • (Neve 2019)Ͱ࣮ݧ͞Ε͍ͯͨ4छྨ • Arithmetic Mean (AM) • Geometric Mean (GM) • Harmonic Mean (HM) • Cross-Ratio Uninorm (CRU) ࣮ݧ͢Δطଘख๏ User A to User B Preference Score User B to User A Preference Score Reciprocal Preference Score Aggregation Function CRU:
  11. ©2021 Wantedly, Inc. ࣮ݧ֓ཁ Wantedly Visit ʹ͓͚ΔϢʔβͱاۀͷ Matching Λ༧ଌ Company

    User Ԡื ϝοηʔδฦ৴ Company User εΧ΢τૹ৴ ϝοηʔδฦ৴ Company User Ԡื ӾཡͷΈ Company User εΧ΢τૹ৴ ӾཡͷΈ • Ϣʔβ͕Ԡื or اۀ͕εΧ΢τૹ৴ͨ͠ࡍʹ૬ख͕ϝοηʔδΛฦ৴͢Ε͹ Match (positive) • Ԡื or εΧ΢τૹ৴Λ૬ख͕Ӿཡ্ͨ͠ͰϝοηʔδΛฦ৴͠ͳ͚Ε͹ Not Match (negative) Match Not Match
  12. ©2021 Wantedly, Inc. ᅂ޷σʔλ/ධՁ஋ • Ϣʔβͷᅂ޷σʔλ • اۀ΁ͷԠื • اۀ͔ΒͷεΧ΢τૹ৴ʹର͢Δϝοηʔδฦ৴

    • اۀͷᅂ޷σʔλ • Ϣʔβ΁ͷεΧ΢τૹ৴ • Ϣʔβ͔ΒͷԠืʹର͢Δϝοηʔδฦ৴ • ධՁ஋: Ϣʔβͱاۀͷ૊ʹରͯ͠༩͑ΒΕΔ (boolean) • Ϣʔβ͕Ԡื or اۀ͕εΧ΢τૹ৴ͨ͠ࡍʹ૬ख͕ϝοηʔδΛฦ৴͢Ε͹ Match (positive) • Ԡื or εΧ΢τૹ৴Λ૬ख͕Ӿཡ্ͨ͠ͰϝοηʔδΛฦ৴͠ͳ͚Ε͹ Not Match (negative) • Ԡื or εΧ΢τૹ৴͕ൃੜ͕ͨ͠૬ख͕Ӿཡ͍ͯ͠ͳ͍΋ͷͷධՁ஋͸ෆ໌ʢະධՁʣ
  13. ©2021 Wantedly, Inc. ࣮ݧσʔλ • Wantedly Visit ʹ͓͚Δ 2019/11 -

    2020/10 ͷ1೥෼ͷߦಈϩά • ৬छΛʮΤϯδχΞʯʹઃఆ͍ͯ͠ΔϢʔβ • ืू৬छΛʮΤϯδχΞʯʹઃఆ͍ͯ͠Δاۀʢืूʣ • ֘౰ظؒதʹ5݅Ҏ্ͷᅂ޷σʔλΛ༗͢ΔϢʔβͱاۀ • ֘౰ظؒதʹ100݅Ҏ্ͷᅂ޷σʔλΛ༗͢ΔϢʔβΛআ֎
  14. ©2021 Wantedly, Inc. ࣮ݧ݁Ռ AUC AM GM HM CRU RCF

    0.558 0.618 0.639 0.616 LFRR 0.475 0.578 0.622 0.552 RCF LFRR • طଘݚڀͱಉ༷ɺHM(ௐ࿨ฏۉ)͕΋ͬͱ΋ߴ͍ੑೳʹ • طଘݚڀͱҟͳΓɺLFRR ΑΓ΋ RCF ͷํ͕ߴ͍ੑೳʹ • طଘख๏ͷ࿦จͷ࣮ݧͱൺ΂ͯ΋ܦݧతʹ΋ AUC ͷ஋͕ খ͍͞ Aggregation Function Algorithm
  15. ©2021 Wantedly, Inc. ߟ࡯ɾԾઆ طଘख๏ͷ࿦จͷ࣮ݧͱൺ΂ͯ΋ܦݧతʹ΋ AUC ͷ஋͕খ͍͞ • Ϣʔβ͔Βاۀɺاۀ͔ΒϢʔβͱ͍͏ҟͳΔ Preference

    Score Λ୯७ͳ Aggregation Function ʹ͔͚͍ͯΔͷ͕ྑ͘ͳ͍ʁ • ୯ํ޲ͷ Preference Score ͷ༧ଌͷ࣌఺Ͱਫ਼౓͕ྑ͘ͳ͍ʁ
  16. ©2021 Wantedly, Inc. ߟ࡯ɾԾઆ طଘख๏ͷ࿦จͷ࣮ݧͱൺ΂ͯ΋ܦݧతʹ΋ AUC ͷ஋͕খ͍͞ • Ϣʔβ͔Βاۀɺاۀ͔ΒϢʔβͱ͍͏ҟͳΔ Preference

    Score Λ୯७ͳ Aggregation Function ʹ͔͚͍ͯΔͷ͕ྑ͘ͳ͍ʁ • ͦΕͧΕͷ෼෍΍εέʔϧ͕ҟͳΔ • ಉ͡஋Λऔ͍ͬͯΔ͔Βͱ͍ͬͯɺಉ͘͡Β͍ͷᅂ޷ͷେ͖͞Λද͢ͷ͔ʁ • ୯ํ޲ͷ Preference Score ͷ༧ଌͷ࣌఺Ͱਫ਼౓͕ྑ͘ͳ͍ʁ
  17. ©2021 Wantedly, Inc. ߟ࡯ɾԾઆ طଘख๏ͷ࿦จͷ࣮ݧͱൺ΂ͯ΋ܦݧతʹ΋ AUC ͷ஋͕খ͍͞ • Ϣʔβ͔Βاۀɺاۀ͔ΒϢʔβͱ͍͏ҟͳΔ Preference

    Score Λ୯७ͳ Aggregation Function ʹ͔͚͍ͯΔͷ͕ྑ͘ͳ͍ʁ • ୯ํ޲ͷ Preference Score ͷ༧ଌͷ࣌఺Ͱਫ਼౓͕ྑ͘ͳ͍ʁ
  18. ©2021 Wantedly, Inc. ߟ࡯ɾԾઆ ୯ํ޲ͷ Preference Score ͷ༧ଌͷ࣌఺Ͱਫ਼౓͕ྑ͘ͳ͍ʁ • Ϣʔβͷ༧ଌ

    Preferenc Score ٴͼ اۀͷ༧ଌ Preference Score ΛͦΕͧΕධՁ AUC Ϣʔβ اۀ RCF 0.850 0.737 LFRR 0.833 0.727 Subject Algorithm → اۀͷ Preference Score ͷ༧ଌਫ਼౓͕௿͍
  19. ©2021 Wantedly, Inc. ߟ࡯ɾԾઆ ͳͥاۀͷ Preference Score ͷ༧ଌਫ਼౓͕௿͍ͷ͔ • اۀͷᅂ޷σʔλ͸Ϣʔβʹൺ΂ͯগͳ͍

    • اۀͷᅂ޷Λڧ͘ද͢ೳಈతͳᅂ޷σʔλʢεΧ΢τૹ৴ʣͷର৅ͱͳΔϢʔβ਺͕গͳ͍ • डಈతͳᅂ޷σʔλΑΓ΋ೳಈతͳᅂ޷σʔλͷํ͕ڧ͘Ϣʔβͷᅂ޷Λද͢ͷͰ͸ͳ͍͔ʁ • اۀͷडಈతͳᅂ޷σʔλʢϢʔβͷԠืʹର͢Δϝοηʔδฦ৴ʣͷର৅ͱͳΔϢʔβ਺ ͸ଟ͍͕ɺ͋·Γᅂ޷Λද͍ͤͯͳ͍ • Ϣʔβʹൺ΂ͯɺاۀ͸͋·Γᅂ޷ʹ߹͍ͬͯͳ͍Ϣʔβʹରͯ͠΋ϝοηʔδฦ৴Λߦ͏ʁ
  20. ©2021 Wantedly, Inc. ఏҊख๏ᶃ • εέʔϧΛ͋ΘͤΔ͜ͱͰ̎ͭͷ Preference Score ΛൺֱՄೳͳঢ়ଶʹ্ͨ͠Ͱ Aggregation

    Function ʹ͔͚Δɻ • اۀɾϢʔβ͝ͱʹ Preference Score Λ MinMaxScaler ʹ͔͚Δɻ Ϣʔβͱاۀͷ Preference Score ͷεέʔϧΛ߹ΘͤΔʢScalerʣ
  21. ©2021 Wantedly, Inc. ఏҊख๏ᶄ ෛͷᅂ޷σʔλΛ׆༻͢ΔʢNegʣ • Ϣʔβ/اۀͷᅂ޷ΛΑΓਖ਼֬ʹ༧ଌ͢ΔͨΊɺԠื/εΧ΢τૹ৴ ͞ΕͯӾཡ͕ͨ͠ϝο ηʔδฦ৴͠ͳ͔ͬͨͱ͍͏ෛͷᅂ޷σʔλΛར༻ •

    ͜Ε·Ͱͷਖ਼ͷᅂ޷σʔλΛར༻ͨ͠ਖ਼ͷ Preference Score ͷ༧ଌʹՃ͑ɺෛͷ Preference Score Λܭࢉ͢Δɻ • ਖ਼ͱෛͷ༧ଌ Preference Score Λ଍͠߹Θͤͯ༧ଌ Preference Score Λܭࢉ
  22. ©2021 Wantedly, Inc. ࣮ݧ݁Ռɾߟ࡯ʢ୯ํ޲ͷᅂ޷: Preference Scoreʣ • اۀͷ Preference Score

    ͷ༧ଌਫ਼౓͕େ͖͘վળ • Ϣʔβͷ Preference Score ͷ༧ଌਫ਼౓͸اۀʹൺ΂Δͱ͋·Γվળͤͣ • Ϣʔβ͸ᅂ޷ʹؔ܎ͳ͘ϝοηʔδΛฦ৴͠ͳ͍͜ͱ͕ଟ͍ͨΊɺෛͷ Preference Score ͷޮՌ ͕খ͍͞ʁʢe.g. ΊΜͲ͍͔͘͞Βϝοηʔδฦ৴͠ͳ͍ʣ AUC Ϣʔβ اۀ RCF 0.850 0.737 LFRR 0.833 0.727 LFRR + Scaler + Neg 0.850 0.817 Subject Algorithm
  23. ©2021 Wantedly, Inc. ࣮ݧ݁Ռɾߟ࡯ʢMatching: Reciprocal Preference Scoreʣ AUC AM GM

    HM CRU RCF 0.558 0.618 0.639 0.616 LFRR 0.475 0.578 0.622 0.552 LFRR + Scaler 0.536 0.619 0.639 0.603 LFRR + Scaler + Neg 0.543 0.686 0.712 0.674 LFRR + Scaler + Neg Aggregation Function Algorithm • ఏҊख๏Ͱ͋Δ LFRR + Scaler + Neg ͱ HM ͷ૊Έ߹Θ͕ͤ࠷΋ߴ͍ੑೳʹ • Scaler ͷΈͷద༻Ͱ΋༧ଌੑೳͷ޲্͕֬ೝͰ͖Δ • ґવɺAUC ͸͞΄Ͳେ͖͘ͳ͍ɻ • Preference Score ͷ༧ଌܭࢉ΍ Aggregation Function ʹ΋ͬͱߴ౓ͳΞϧΰϦζϜΛར༻͢Δɻ • ೳಈతͳᅂ޷σʔλͱडಈతͳᅂ޷σʔλͷॏΈΛมֶ͑ͯश͢Δɻ
  24. ©2021 Wantedly, Inc. ·ͱΊ A. Wantedly Visit ͷσʔλʹରͯ͠طଘख๏Λద༻࣮ͨ͠ݧɾߟ࡯ • ઌߦݚڀͱྨࣅ͢ΔΑ͏ͳ࣮ݧ݁Ռ

    • શମతʹ༧ଌਫ਼౓͕௿͍ • Ϣʔβ͔Βاۀɺاۀ͔ΒϢʔβͱ͍͏ҟͳΔ Preference Score Λ୯७ͳ Aggregation Function ʹ ͔͚͍ͯΔ͜ͱ͕ྑ͘ͳ͍ʁ • اۀͷ Preference Score ͷ༧ଌਫ਼౓͕௿͍͜ͱ͕ྑ͘ͳ͍ʁ B. ࣮ݧ݁ՌΛड͚ͨվྑख๏ͷఏҊɾ࣮ݧɾߟ࡯ • ख๏ͷఏҊ • Ϣʔβͱاۀͷ Preference Score ͷεέʔϧΛ߹ΘͤΔʢScalerʣ • ෛͷᅂ޷σʔλΛ׆༻͢ΔʢNegʣ • ఏҊख๏͕طଘख๏ͷੑೳΛ্ճͬͨ
  25. ©2021 Wantedly, Inc. 3FGT • (Pizzato 2010) Luiz Pizzato, Tomek

    Rej, Thomas Chung, Irena Koprinska, and Judy Kay. 2010. RECON: a reciprocal recommender for online dating. Proceedings of the fourth ACM conference on Recommender systems P. 207-214. • (Pizzato 2012) Luiz Pizzato, Tomasz Rej, Joshua Akehurst, Irena Koprinska, Kalina Yacef, and Judy Kay. 2012. Recommending people to people: the nature of reciprocal recommenders with a case study in online dating. User Model User-Adap Inter (2013) 23: 447. • (Xia 2015) Peng Xia, Benyuan Liu, Yizhou Sun, and Cindy Chen. 2015. Reciprocal Reciprocal recommendation System for Online Dating. Proceedings of the 2015 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining P. 234-241. • (Neve 2019) J Neve, I Palomares.2019. Latent Factor Models and Aggregation Operators for Collaborative Filtering in Reciprocal Recommender Systems Proceedings of the 13th ACM Conference on Recommender Systems, 219-227. • (Neve 2020) J Neve, R McConville.2020. ImRec: Learning Reciprocal Preferences Using Images. Proceedings of the 14th ACM Conference on Recommender Systems, 170-179.