Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ジョブマッチングサービスにおける相互推薦システムの応用事例と課題
Search
Shuhei Goda
November 07, 2024
Technology
3
980
ジョブマッチングサービスにおける相互推薦システムの応用事例と課題
第27回情報論的学習理論ワークショップ (IBIS2024)
企画セッション3:ビジネスと機械学習
https://ibisml.org/ibis2024/os/
Shuhei Goda
November 07, 2024
Tweet
Share
More Decks by Shuhei Goda
See All by Shuhei Goda
Turing × atmaCup #18 - 1st Place Solution
hakubishin3
0
930
とある事業会社にとっての Kaggler の魅力
hakubishin3
8
2.8k
課題の解像度が荒かったことで意図した改善ができなかった話
hakubishin3
3
1k
Wantedly におけるマッチング体験を最大化させるための推薦システム
hakubishin3
4
1.2k
Recommendation Industry Talks #1 Opening
hakubishin3
1
410
会社訪問アプリ「Wantedly Visit」での シゴトに関する興味選択機能と推薦改善
hakubishin3
0
660
論文紹介: Improving Implicit Feedback-Based Recommendation through Multi-Behavior Alignment(Xin Xin et al., 2023)
hakubishin3
0
660
Feedback Prize - English Language Learning における擬似ラベルの品質向上の取り組み
hakubishin3
0
1.1k
ウォンテッドリーにおける推薦システムのオフライン評価の仕組み
hakubishin3
7
7.1k
Other Decks in Technology
See All in Technology
Kiroと学ぶコンテキストエンジニアリング
oikon48
5
5k
AI時代にPdMとPMMはどう連携すべきか / PdM–PMM-collaboration-in-AI-era
rakus_dev
0
240
コスト削減の基本の「キ」~ コスト消費3大リソースへの対策 ~
smt7174
2
320
【5分でわかる】セーフィー エンジニア向け会社紹介
safie_recruit
0
30k
【初心者向け】ローカルLLMの色々な動かし方まとめ
aratako
3
1.9k
スプリントレトロスペクティブはチーム観察の宝庫? 〜チームの衝突レベルに合わせたアプローチ仮説!〜
electricsatie
1
140
RSCの時代にReactとフレームワークの境界を探る
uhyo
6
880
実践アプリケーション設計 ②トランザクションスクリプトへの対応
recruitengineers
PRO
4
1.2k
Snowflakeの生成AI機能を活用したデータ分析アプリの作成 〜Cortex AnalystとCortex Searchの活用とStreamlitアプリでの利用〜
nayuts
0
140
Nstockの一人目エンジニアが 3年間かけて向き合ってきた セキュリティのこととこれから〜あれから半年〜
yo41sawada
0
170
「魔法少女まどか☆マギカ Magia Exedra」の必殺技演出を徹底解剖! -キャラクターの魅力を最大限にファンに届けるためのこだわり-
gree_tech
PRO
0
430
衝突して強くなる! BLUE GIANTと アジャイルチームの共通点とは ― いきいきと活気に満ちたグルーヴあるチームを作るコツ ― / BLUE GIANT and Agile Teams
naitosatoshi
0
290
Featured
See All Featured
Practical Orchestrator
shlominoach
190
11k
Writing Fast Ruby
sferik
628
62k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
GitHub's CSS Performance
jonrohan
1032
460k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
3.1k
Faster Mobile Websites
deanohume
309
31k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
Unsuck your backbone
ammeep
671
58k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Speed Design
sergeychernyshev
32
1.1k
The Power of CSS Pseudo Elements
geoffreycrofte
77
5.9k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
910
Transcript
© 2024 Wantedly, Inc. δϣϒϚονϯάαʔϏεʹ͓͚Δ ૬ޓਪનγεςϜͷԠ༻ࣄྫͱ՝ Nov. 7 2024
- Shuhei Goda ୈ27ճใతֶशཧϫʔΫγϣοϓ (IBIS2024) اըηογϣϯ3ɿϏδωεͱػցֶश
© 2024 Wantedly, Inc. ໊લɿ ߹ా पฏ Shuhei Goda
ॴଐͱׂɿ ΥϯςουϦʔגࣜձࣾ ɾData Team Manager ɾMachine Learning Tech Lead ɾProduct Manager Kaggle Tierɿ Kaggle Competitions Grandmaster @jy_msc ࣗݾհ https://www.kaggle.com/shuheigoda
©2024 Wantedly, Inc. ڀۃͷదࡐదॴʹΑΓɺ γΰτͰίίϩΦυϧͻͱΛ;͢ ࢲͨͪͷϛογϣϯ ©2024 Wantedly, Inc.
© 2024 Wantedly, Inc. iOS, Android and Web ؾܰʹձࣾ๚ ϛογϣϯՁ؍ͷڞײͰϚονϯά
• څ༩རްੜͳͲͷ݅Ͱͳ͘ɺ͍͕͋Εձࣾͷ نʹͱΒΘΕͳ͍ ·ͣʮΛฉ͖ʹߦ͘ʯͱ͍͏৽͍͠ମݧ • ݸਓͱاۀ͕ϑϥοτͳઢͰग़ձ͑Δ͜ͱͰɺΑΓັྗ తͳॴΛݟ͚ͭΔ͜ͱ͕Մೳʹ ձࣾ๚ΞϓϦʮWantedly Visitʯ
© 2024 Wantedly, Inc. తΛୡ͢ΔͨΊʹɺ֤ొਓҎԼͷΑ͏ʹߦಈ͢Δ αʔϏεΛར༻͢Δਓͷతͱߦಈ ొਓ αʔϏεΛར༻͢Δత Ϣʔβʔ
ઓతͰΓ͕͍ͷ͋Δࣄʹͭ͘ اۀ ࣗࣾͰ׆༂Ͱ͖ΔਓࡐΛ࠾༻͢Δ ࣗͷίϯςϯπ Λ࡞͢Δ ૬खͷίϯςϯπ ΛӾཡ͢Δ ૬खʹ໘ஊͷػձ Λਃ͠ࠐΉ
© 2024 Wantedly, Inc. ϓϥοτϑΥʔϜߏʢίϯςϯπʣ Ϣʔβʔ ձࣾһ ෭ۀ ϑϦʔϥϯε ֶੜ
اۀ ܦӦ ࣾһ ਓࣄ Wantedly Visit ϓϩϑΟʔϧ ࡞ Ӿཡ ձࣾϖʔδɾืू ࡞ Ӿཡ
© 2024 Wantedly, Inc. ϓϥοτϑΥʔϜߏʢϚονϯάʣ Ϣʔβʔ ձࣾһ ෭ۀ ϑϦʔϥϯε ֶੜ
اۀ ܦӦ ࣾһ ਓࣄ Wantedly Visit Ԡื Λฉ͍ͯΈ͍ͨͰ͢ ͥͻ͓͠·͠ΐ͏ʂ εΧτ ͓͠·ͤΜ͔ʁ ͓Λฉ͔͍ͤͯͩ͘͞
© 2024 Wantedly, Inc. ϓϥοτϑΥʔϜͷϞσϧʢུ֓ʣ ܧଓɾ෮ؼ ܧଓɾ෮ؼ Ԡื εΧτ ྲྀೖ
৽ن • 2-sided marketplaceͰ͋ΓɺاۀͱϢʔβʔͷ྆ํ͕υϥΠόʔ • ྑ࣭ͳίϯςϯπͷੵɺΓͳ͍Ϛονϯάʢޭମݧʣ͕ॏཁ
© 2024 Wantedly, Inc. ϓϥοτϑΥʔϜͷ՝ - ͷϘτϧωοΫԿ͔ 20248݄ظͷొͱਪҠ 403ສਓ 4.1ສࣾ
ෳࡶͷ૿ՃͱͦΕʹ͏ϚονϯάޮͷԼ Ϣʔβʔ૿Ճʹ͏՝ • ෳࡶͷ૿Ճ→ϚονϯάޮͷԼ →Ϛονϯά૿ՃͷಷԽ→ͷఀϦεΫ Ϛονϯάޮ ͷԼ Ϣʔβʔ Ϛονϯάޮ
© 2024 Wantedly, Inc. ϓϥοτϑΥʔϜͷʹඞཁͳऔΓΈ Ϣʔβʔ Ϛονϯάޮ Ϣʔβʔ Ϛονϯάޮ Ϛονϯάޮ
Ϛονϯάޮ Ϛονϯάޮ͕ߴ͘ҡ࣋͞ΕΔঢ়ଶΛ࡞Γ αʔϏεશମͷ࣋ଓతͳΛ࣮ݱ͢Δ Before After
© 2024 Wantedly, Inc. Ϛονϯάޮ͕ߴ͘ҡ࣋͞ΕΔঢ়ଶΛͲ͏࣮ݱ͢Δ͔ p(match = 1|c, j) =
p(exam = 1|c, j) × p(scout = 1|c, j, exam = 1) × p(reply = 1|c, j, scout = 1) اۀͲͷϢʔβʔΛ ݟΔ͔ اۀʹͱͬͯͦͷϢʔβʔ ັྗతʹײ͡ΒΕΔ͔ Ϣʔβʔʹͱͬͯͦͷืू ັྗతʹײ͡ΒΕΔ͔ ਪનγεςϜͰϢʔβʔɾاۀํͷίϯςϯπͷදࣔΛ੍ޚ͢Δ • To اۀɿاۀʹͱͬͯັྗతͰɺͦͷاۀͱϚονϯά͍͢͠ϢʔβʔΛදࣔ • To ϢʔβʔɿϢʔβʔʹͱͬͯັྗతͰɺͦͷϢʔβʔͱϚονϯά͍͢͠ืूΛදࣔ j … Job seeker c … Company ͜͜ʹհೖ͢Δ p(match = 1| j, c) = p(exam = 1| j, c) × p(apply = 1| j, c, exam = 1) × p(reply = 1| j, c, apply = 1) ϢʔβʔͲͷاۀΛ ݟΔ͔ Ϣʔβʔʹͱͬͯͦͷืू ັྗతʹײ͡ΒΕΔ͔ اۀʹͱͬͯͦͷϢʔβʔ ັྗతʹײ͡ΒΕΔ͔ εΧτ༝དྷ ͷϚονϯά֬ Ԡื༝དྷ ͷϚονϯά֬
© 2024 Wantedly, Inc. ղ͖͘ػցֶशλεΫ Ϛονϯά͕࠷େͱͳΔΑ͏ͳਪનϦετΛ֤ඃਪનऀʹରͯ͠࡞͢ΔλεΫ σ* c := argsortj∈
𝒥 m(c, j) σ* j := argsortc∈ 𝒞 m(c, j) →֤اۀ c ʹͱͬͯཧతͳϥϯΩϯάɺ m(c, j)ਅͷϚονϯά֬ →֤Ϣʔβʔ j ʹͱͬͯཧతͳϥϯΩϯά • ཧతͳϥϯΩϯάਅͷϚονϯά֬ʹґଘ͍ͯ͠Δ͕ɺͦͷ֬ͷ͔Βͳ͍ • ֶशσʔλʹج͍ͮͯਪఆͨ͠Ϛονϯά֬ΛͬͯϥϯΩϯάΛ࡞͠ɺ࠷దԽΛਤΔ
© 2024 Wantedly, Inc. ػցֶश͕αʔϏεͷίΞՁͰ͋ΔϚονϯάΛΓཱͨͤΔ ػցֶशϞσϧͷ༧ଌਫ਼্͕͢Δ΄ͲɺͦΕʹͬͯϓϩμΫτͷॏཁࢦ ඪ্͕͍ͯ͘͠ߏ → ػցֶश͕ϓϩμΫτͷՁΛఏڙ͢ΔͨΊͷ
“must have” ͳঢ়ଶ ػցֶशϞσϧͷੑೳ αʔϏεͷ ఏڙՁ
© 2024 Wantedly, Inc. Ϛονϯά༧ଌͷయܕతΞϓϩʔν ҎԼͷೋछྨʹେผͨ͠߹ɺجຊతʹΘΕΔͷ Predict-then-Aggregate ͷܗࣜ • Direct
Match Prediction (DMP) : ؍ଌ͞ΕͨϚονϯάʹج͍ͮͯϚον֬Λ༧ଌ͢Δ • Predict-then-Aggregate (PtA) : ํͷબΛಠཱʹϞσϧԽ͠ɺͦΕΒͷ༧ଌΛू͢Δ Direct Match Prediction Predict-then-Aggregate 𝒟 = {(ci , ji , yc→j i , yj→c i )}n i=1 ֶशσʔληοτɿ 𝒟 = {(ci , ji , mi )}n i=1 ؍ଌ͞ΕͨϚονϯά ̂ m = argmin ̂ m′  n ∑ i=1 ℓ( ̂ m′  (ci , ji ), mi ) Ϛονϯά֬ͷ ֶशɾ༧ଌɿ ֶशσʔληοτɿ اۀ͕ϢʔβʔʹΞΫγϣϯ͔ͨ͠ Ϣʔβʔ͔ΒΞΫγϣϯ͔͋ͬͨ ̂ pc→j = argmin ̂ p′  n ∑ i=1 ℓ( ̂ p′  (ci , ji ), yc→j i ) اۀ→Ϣʔβʔͷ બ֬ɿ Ϣʔβʔ→اۀͷ બ֬ɿ ̂ pj→c = argmin ̂ p′  n ∑ i=1 ℓ( ̂ p′  (ci , ji ), yj→c i ) ̂ m = M( ̂ pc→j, ̂ pj→c) Ϛονϯά֬ɿ ଛࣦؔ ूؔ • DMPతͳϚονϯάͷ༧ଌ͕ՄೳͰ͋Δ • ҰํͰϚονͷϥϕϧඇৗʹεύʔεͰ͋Γɺֶश͕ࠔ • PtAൺֱతີͳೋछྨͷϥϕϧΛͦΕͧΕ༧ଌ͢Δ λεΫʹׂ͢Δ͜ͱͰɺεύʔεੑͷʹରॲ ※ ؆୯ԽͷͨΊɺҎ߱ اۀଆͷέʔεͷΈΛߟ͑Δ
© 2024 Wantedly, Inc. ૬ޓਪનγεςϜ(Reciprocal Recommender Systems) ૬ޓਪનγεςϜͱʮαʔϏεͷϢʔβʔΛޓ͍ʹਪન͠߹͏γεςϜʯ • ਪનΛड͚औΔϢʔβʔͱਪન͞ΕͨϢʔβʔͷ྆ํ͕ຬͯ͠ਪનޭͱ͢Δ
• ૬ޓਪનγεςϜ Predict-then-Aggregate(PtA) ΞϓϩʔνΛ࠾༻͢Δ ઌߦݚڀͷΞϓϩʔν • ίϯςϯπϕʔε [Pizzato+, 2010] • ڠௐϑΟϧλϦϯάϕʔε [Xia+, 2015] [Neve+, 2019] • ϋΠϒϦοτϕʔε [Neve+, 2020] • DLϕʔε [Yıldırım+, 2021] [Luo+, 2020] [Liu+, 2024] ूؔ • 2ͭͷผʑͷ༧ଌΛΈ߹ΘͤΔׂΛ࣋ͭ • جຊతʹώϡʔϦεςΟοΫͳͷɻ୯७ੵɺௐ ฏۉɺزԿฏۉͳͲ [Pizzato+, 2010] [Neve+, 2019] جຊతͳߏ ̂ pa→b Preference Score from a to b M( ̂ pa→b, ̂ pb→a) Aggregation ̂ pb→a Preference Score from b to a ߦಈϩά ଐੑσʔλ ͳͲ
© 2024 Wantedly, Inc. ࣮ݧ݁Ռͷ֓ཁ ϓϩμΫτͷ࣮ࡍͷσʔλΛ࣮ͬͨݧͷ࣮ࢪ • ΦϑϥΠϯɿDMP ͱෳͷूؔͷύλʔϯͷ PtA
Λൺֱɻํͷᅂͷूͷ༗ޮੑΛ֬ೝ • ΦϯϥΠϯɿPtA(Scout-Only)ͱൺֱͨ͠ PtA(Harmonic Mean) ͷੑೳΛݕূɺେ෯ͳKPIͷ্Λ֬ೝ ϕʔεϥΠϯ M( ̂ pc→j, ̂ pj→c) = ̂ pc→j • PtA (Scout-Only)ɿ M( ̂ pc→j, ̂ pj→c) = ̂ pj→c • PtA (Reply-Only)ɿ ݕ౼ख๏ M( ̂ pc→j, ̂ pj→c) = ̂ pc→j ⋅ ̂ pj→c • PtA (Multiplication)ɿ M( ̂ pc→j, ̂ pj→c) = 2 ̂ pc→j ⋅ ̂ pj→c ̂ pc→j + ̂ pj→c • PtA (Harmonic Mean)ɿ ΦϑϥΠϯධՁͷҰࣄྫ
© 2024 Wantedly, Inc. ٕज़త՝ - ਪનػձͷภΓʹΑΔҰ෦Ϣʔβʔͷूத ਪનػձͷภΓ͕ੜ͡Δ͜ͱͰɺϓϥοτϑΥʔϜશମͷརӹ(Ϛον૯)͕େ͖͘ͳΒͳ͍ • ֤ϢʔβʔʹΩϟύγςΟ(Ϛονͷ্ݶ)͕ଘࡏɺͦΕΛ͑ΔҙΛΒͬͯରԠͰ͖ͳ͍
• ඃਪનػձͷগͳ͍ϢʔβʔɺޭମݧͱͳΔϚονϯάΛ࣮ݱ͢Δػձ͕ݶΒΕͯ͠·͏ • طଘͷ૬ޓਪનγεςϜݸผͷϚονΛ࠷దԽ͠ϥϯΩϯά͝ͱʹಠཱͯ͠ܭࢉ͍ͯ͠ΔͨΊɺਪ નػձͷภΓΛੜͤͯ͡͞͠·͏ શ෦ରԠ Ͱ͖ͳ͍… εΧτ͕ དྷͳ͍… ՝ʹର͢ΔΞϓϩʔν • ٻ৬ऀ͕اۀ͔ΒͷεΧτʹԠ͢Δ͕֬ɺ ٻ৬ऀ͕ΑΓଟ͘ͷεΧτΛड͚ΔʹͭΕͯ ݮগ͢ΔՄೳੑΛߟྀ͠ɺϚον૯͕࠷େԽ ͞ΕΔΑ͏ϥϯΩϯάΛ࠷దԽ [Su+, 2022] • Ϛονϯάཧʹج͖ͮɺํͷϢʔβͷᅂ ͚ͩͰͳ͘ΩϟύγςΟΛߟྀͨ͠ूΛߦ͏ [Tomita+, 2022]
© 2024 Wantedly, Inc. ٕज़త՝ - ํͷᅂͷूํ๏ ᅂͷूํ๏αʔϏεͦΕΛར༻͢ΔϢʔβʔͷੑ࣭ʹԠͯ͡ઃܭ͢Δඞཁ͕͋Δ • ํͷᅂ༧ଌ݁ՌΛͲͷΑ͏ʹू͢Δ͔ࣗ໌Ͱͳ͍
• Ұൠతʹɺௐฏۉͱ͍ͬͨɺͲͪΒ͔ยํͷείΞ͕͍ͱूͨ͠είΞ͘ͳΔͱ͍͏ੑ࣭ Λ࣋ͭ͜ͱ͕·͍͠ͱ͞Ε͍ͯΔ [Palomares+, 2021] [Neve+, 2019] • ᘳʹ֬ΛਪఆͰ͖ͨͷͰ͋Εɺू ͍ؔΒͳ͍ͣ ɻ ֤ଆͷ༧ଌͷζϨΛमਖ਼͢ΔΛ ू͕ؔ୲͍ͬͯΔɺͱղऍͰ͖Δɻ ՝ʹର͢ΔΞϓϩʔν • ํͷᅂͷॏΈΛϢʔβʔ͝ͱʹ࠷దԽ͢Δ ख๏ΛఏҊ [Kleinermann+, 2018] ूํ๏ͷ·ͱΊ [Palomares+, 2021]
© 2024 Wantedly, Inc. ٕज़త՝ - Ϛονϯάͷεύʔεੑͷରॲ ϚονϯάϓϥοτϑΥʔϜͰϚονϯάͱ͍͏ใ͕ಘʹ͍͘ಛੑ͕͋Δ • δϣϒϚονϯάͷ߹ʮస৬ʯཱ͕͢Δͱɺ࣍ͷߦಈΛى͜͢·Ͱʹ͍͕͔͔࣌ؒΔ
• ਪનଆͱඃਪનଆͷํͷҙͱߦಈ͕߹கͯ͠ॳΊͯϚονϯάཱ͕͢Δ • Ϛονϯάͷ༧ଌਫ਼Λ্͛ΔͨΊʹɺͲͷΑ͏ͳใΛͲ͏ѻ͏͖͔͕՝ͱͳΔ ՝ʹର͢ΔΞϓϩʔν • ࣝάϥϑ͔ΒϝλύεΛநग़ͯ͠Ϛονϯάͷ ϞσϦϯάʹऔΓೖΕΔ͜ͱͰɺΠϯλϥΫγϣ ϯ͚ͩͰͳ͘ίϯςϯπใΛ༗ޮతʹ׆༻͢Δ [Lai+, 2024]
© 2024 Wantedly, Inc. ݚڀ։ൃ - Ϛον༧ଌਫ਼ͷ্ • ϚονϥϕϧΛֶश͢ΔతͳΞϓϩʔν͕ͩɺ
Ϛονϥϕϧͷۃͳεύʔεੑ͕ͱͳΓɺ ੑೳͷߴ͍ϞσϧΛ࡞Εͳ͍ →ΞΠσΞɿҟͳΔੑ࣭Λ࣋ͭ2छྨͷใΛޮՌతʹΈ߹Θͤͯɺີͱਫ਼ͷʮ͍͍ͱ ͜औΓʯΛ࣮ݱ͢Δ Predict-then-Aggregate(PtA)ͷ՝ Direct Match Prediction(DMP)ͷ՝ • Ϛονϯάͱ͍͏ϞσϧԽΛɺಠཱͨ͠2छྨͷϞσ ϧʹׂ͢Δ͜ͱʹΑ͕ͬͯੜ͡Δ • σʔλλεΫͷੑ࣭ʹ߹ΘͤͨूؔΛదʹઃ ܭ͠ͳ͍ͱύϑΥʔϚϯε͕ෆे • ֤Ϟσϧͷ༧ଌޡࠩͷ͕࠷ऴతͳϥϯΩϯάύ ϑΥʔϚϯεʹӨڹ͢Δ छྨ ਫ਼ ີ ਅͷϚονϥϕϧ ਖ਼֬ ↑ ૄ ↓ Ϛον༧ଌ ൺֱతෆਖ਼֬ ↓ ີ ↑ S. Goda, Y. Hayashi, Y. Saito, A Best-of-Both Approach to Improve Match Predictions and Reciprocal Recommendations for Job Search. arXiv preprint arXiv:2409.10992 (2024).
© 2024 Wantedly, Inc. ݚڀ։ൃ - Ϛον༧ଌਫ਼ͷ্ ఏҊख๏ spseudo (c,
j; αc,j ) = αc,j ⋅ m(c, j) + (1 − αc,j ) ⋅ ̂ pc→j ⋅ ̂ pj→c ਅͷϚονϥϕϧͱϚον༧ଌΛΈ߹Θͤͨ Pseudo Match Scores Λੜ͠ɺϝλϞσϧΛֶश ਅͷϚονϥϕϧ Ϛον༧ଌ ̂ f = argminf′  ∑ (c,j) ℓ( f′  (c, j), spseudo (c, j; α)) ΦϑϥΠϯධՁ݁Ռ • ϝλϞσϧΛ༻͢Δ͜ͱͰɺैདྷͷPtAͷूϑΣʔζͰ ൃੜ͢ΔΤϥʔͷӨڹΛݮ͍ͯ͠ΔՄೳੑ͕͋Δ • ҟͳΔείΞใΛΈ߹ΘͤΔ͜ͱʹΑͬͯɺΞϯαϯϒϧ తͳޮՌ͕ಘΒΕ͍ͯΔՄೳੑ͕͋Δ ղऍ S. Goda, Y. Hayashi, Y. Saito, A Best-of-Both Approach to Improve Match Predictions and Reciprocal Recommendations for Job Search. arXiv preprint arXiv:2409.10992 (2024).