「うまく言えない」検索を叶える ― OpenSearchと生成AIで作る 類似プロジェクト検索

Transcript

1. ʮ͏·͘ݴ͑ͳ͍ʯݕࡧΛ׎͑Δ ― OpenSearchͱੜ੒AIͰ࡞Δ ྨࣅϓϩδΣΫτݕࡧ 4FSWFSMFTT
   0QFSBUJPOT
   4POV
   ,JN
2. ࣗݾ঺հ ۚɹઋ༏
   
   4POV
   ,JN $00
   BU
   4FSWFSMFTT
   0QFSBUJPOT
   *OD
   
   'VMMTUBDL
   "84
   4FSWFSMFTT
   &OHJOFFS "84
   $PNNVOJUZ
   #VJMEFST
   
   
   4FSWFSMFTT
   +1
   ͓Αͼ
   άϩʔόϧ
   "84
   ίϛϡχςΟͰ׆ಈத 
   
   
3. Ϗδωε՝୊ ʮ͋ΔϓϩδΣΫτͱࣅ͍ͯΔผͷϓϩδΣΫτΛݟ͚ͭΔʯۀ຿ͷলྗԽ طଘۀ຿γεςϜͷݕࡧػೳ͸ෳࡶͳϑΟϧλʔΛࣄࡉ͔͘ઃఆ͠ͳ͚Ε͹ͳΒͣɺ
   γεςϜػೳ͚ͩͰ͸ർ࿑ײ͕૿͢͜ͱͰۀ຿͕஗ԆؾຯʹͳΓɺਓ΁ฉ͘աఔ͕ෆՄܽ طଘۀ຿γεςϜͷݕࡧػೳͰ͸ɺ
   ਺ଟ͘ͷ߲໨ʹରͯ͠਺஋΍ΩʔϫʔυΛ௚ ઀ೖྗඞཁ 
   
   
4. ͳͥɺࣅ͍ͯΔผͷϓϩδΣΫτΛݟ͚ͭΔඞཁ͕͋Δ͔ʁ "
   ϓϩδΣΫτ ɾ໘ੵɿ 
   ᶷ
   ɾඅ༻ɿԯԁ
   ɾظؒɿ೔
   ɾಛ௃ᶃ
   ݸ͕ࣨଟ͘ɺಛघͳઃඋͱஅ೤
   ɾಛ௃ᶄ
   ਖ਼໘ʹಓ࿏͕͋Γɺଞ͸܏ࣼ஍ #
   ϓϩδΣΫτ $
   ϓϩδΣΫτ %
   ϓϩδΣΫτ ઃඋͷઃஔཁྖ
   ɾಛੑ֬ೝ ೖࡳʹඞཁͳॻྨɾखଓ͖ɺ
   ৹ࠪཁྖɾཁ݅Λ֬ೝ ಛघͳ஍ܗɾಛੑʹΑΔ
   ࢿࡐௐୡํ๏Λݕ౼ 
   
   
5. ΞΠσΟΞ σʔλΛجװγεςϜ%#͔Βநग़ɺ0QFO4FBSDI
   ΁ΠϯσοΫε࡞੒ ࣗવݴޠΛؚΉ֤σʔλ߲໨ͷϕΫτϧԽɺྨࣅ౓ݕࡧʢϕΫτϧݕࡧʣΛ࣮૷ ֤߲໨ผʹείΞϦϯάΛద༻͠ɺ্Ґͷݕࡧ݁ՌΛྨࣅϓϩδΣΫτͯ͠
   Ϛονϯάɺ6*
   ग़ྗ ޻ࣄ%# &NCFEEJOH
   .PEFM 0QFO4FBSDI ݕࡧ 
   
   
6. ʮ͏·͘ݴ͑ͳ͍ʯ͍͋·͍ͳݕࡧ৚݅Λ൓ө͢Δํ๏ ਓʹΑͬͯɺঢ়گʹΑͬͯมΘͬͯ͘Δʮࣅ͍ͯΔʯΛݕࡧ݁Ռʹ൓ө͢Δ͜ͱ͕ཁ݅ ظؒΛͽͬͨΓ͋Δ࣌఺ͱܾͯ͠ΊΔͷͰ͸ͳ͘ɺʮͦͷลʯͱͯ͠ద༻͍ͨ͠ ࣗવݴޠʹهࡌ͞Ε͍ͯΔ͋ΔಛੑΛ࠷༏ઌͱ͠ɺ໘ੵɾඅ༻Λ࣍ͷ༏ઌॱҐͱͯ͠ద༻͍ͨ͠ ˞໘ੵɾඅ༻ͷ࠷େɾ࠷খ஋Λࢦఆͤͣᐆດͳঢ়ଶͰݕࡧͰ͖ΔΑ͏ʹ͍ͨ͠ ͦͷଞɺݟ͚͍ͭͨϓϩδΣΫτͷ৚͕݅ʮ͏·͘ݴ͑ͳ͍ʯ৔߹ طଘγεςϜͰͷϑΟϧλʔઃఆͰͷݕࡧମݧΛ
   ৽γεςϜͰΑΓศརͳܗͰ࣮ݱ͢Δํ๏ͱ͸ɾɾʁ 
   
   
7. ʮ͍͋·͍͞ʯʹରԠ͢ΔͨΊͷઓུ "
   ϓϩδΣΫτ ɾ໘ੵɿ 
   ᶷ
   ɾඅ༻ɿԯԁ
   ɾظؒɿ೔
   ɾ׬ྃ೔ɿ೥݄೔
   ɾಛ௃ᶃ
   ݸ͕ࣨଟ͘ɺಛघͳઃඋͱஅ೤
   ɾಛ௃ᶄ
   ਖ਼໘ʹಓ࿏͕͋Γɺଞ͸܏ࣼ஍ ᶃ
   ᐆດ͞Λߏ଄Խ͠ɺ ࣮૷ʹམͱ͠ࠐΊΔܗ ΁۩ମԽ ˞Ճॏׂ߹ͷ߹ܭ͕ ʹͳΔΑ͏ʹઃఆͯ͠ݕࡧ ᶄ
   ݕࡧ݁Ռ͸ෳ਺ͷϓϩδΣΫτ͕ Ұ໨Ͱݟ͑ΔΑ͏ʹఏڙ͢Δ #
   ϓϩδΣΫτ 50% 25% 25% 0% 0% ໘ੵ අ༻ ಛ௃ ׬ྃ೔ ֊਺ ɾ໘ੵɿ 
   ᶷ
   ɾඅ༻ɿԯԁ
   ɾ׬ྃ೔ɿ೥݄೔
   ɾಛ௃
   ओཁൖೖܦ࿏͕શͯ܏ࣼ஍ $
   ϓϩδΣΫτ ɾ໘ੵɿ 
   ᶷ
   ɾඅ༻ɿԯԁ
   ɾ׬ྃ೔ɿ೥݄೔
   ɾಛ௃
   ਖ਼໘͔ΒͷԼΓࡔʹ஫ҙ %
   ϓϩδΣΫτ ɾ໘ੵɿ 
   ᶷ
   ɾඅ༻ɿԯԁ
   ɾ׬ྃ೔ɿ೥݄೔
   ɾಛ௃
   ܏͖΍͍͢஍൫ߏ଄ʹ஫ҙ ྨࣅ౓
    ྨࣅ౓
    ྨࣅ౓
    
   
   
8. ʮ͍͋·͍ʯͳ஋ͱॏΈ͚ͮΛద༻͢Δ
   GVODUJPO@TDPSF public filterAreaWithScale( origin: number, offset = 10, scale = 1000 ) { this.queryTemplate.query.bool.must.push({ function_score: { functions: [ { exp: { 'document.໘ੵ': { origin: origin, offset: offset, scale: scale, }, }, weight: weight }, ], }, }) return this } %FDBZ
   'VODUJPOT (BVTTJBO
   ڑ཭͕͍ۙ͘͝΋ͷ͕΄΅ຬ఺ɺ཭ΕΔ΄Ͳٸݮ &YQPOFOUJBM
   ڑ཭ͷ૿ՃʹͭΕͯҰఆͷׂ߹Ͱܧଓతʹݮগ -JOFBS
   ڑ཭ʹൺྫͯ͠௚ઢతʹείΞ͕ݮগ 
   
   
9. ʮ͍͋·͍ʯͳ஋ͱॏΈ͚ͮΛద༻͢Δ
   GVODUJPO@TDPSF public filterAreaWithScale( origin: number, offset = 10, scale = 1000 ) { this.queryTemplate.query.bool.must.push({ function_score: { functions: [ { exp: { 'document.໘ੵ': { origin: origin, offset: offset, scale: scale, }, }, weight: weight }, ], }, }) return this } // ʮྨࣅ౓ʯΛ൓ө͢ΔͨΊͷϝιου // ޻ࣄબ୒ޙɺεϥΠμʔͷॳظ஋ϩδοΫʹج͍ͮͯ // offset ͱ scale Λ൓ө͠·͢ // origin : ൺֱͷج४ͱͳΔ஋ // offset : ج४஋ͱͷ͕ࠩ͜ͷ஋Ҏ಺ͷ৔߹ɺείΞ͕ຬ఺ʢ1ʣͱΈͳ͞ΕΔൣғ // scale : ج४஋ͱͷ͕ࠩ scale Λ௒͑Δ৔߹ɺείΞʹ͸൓ө͞Εͳ͍ const searchQuery = new SearchQueryHelper() .setPageNumber(1) .setPageSize(25) .filterAreaWithScale(origin, offset, scale) .filterCostWithScale(origin, offset, scale) .filterPeriodWithScale(origin, offset, scale) .filterDeliveryDateWithScale(origin, offset, scale) .setSearchTextVector('ಛ௃', inputTextEmbedding.vector) 
   
   
10. ࣗવݴޠͰϓϩδΣΫτͷಛ௃ΛݕࡧͰ͖ΔΑ͏ʹͳΔ·Ͱ ਖ਼نԽ͞Εͨ
    %#
    நग़σʔλҎ֎ͷݟੵॻɺܭըॻɺٞࣄ࿥ɺͦͷଞ֤छࢿྉͳͲɺ
    ςΩετؚ͕·Ε͍ͯͳ͍εΩϟϯ1%'Λݕࡧ݁Ռʹ൓ө͢Δํ๏ ສ݅Λ௒͑ΔϓϩδΣΫτͷಛ௃ΛࣗવݴޠͰ੔ཧ͠ɺ
    ςΩετ͓Αͼྨࣅ౓ݕࡧΛ࣮૷͢Δํ๏ɾɾʁ 
    
    
11. લॲཧ͕ՄೳͳશͯͷछྨͷσʔλΛΑΓࠐΈɺߏ଄Խ͢Δ | code | name | area | cost | period | deliveryDate | |------|-----------|------|------------|--------|--------------| | 0001 | A project | 2000 | 1000000000 | 365 | 2023-01-01 | | 0002 | B project | 2000 | 1000000000 | 365 | 2023-01-02 | | 0003 | C project | 2000 | 1000000000 | 365 | 2023-01-03 | root/ ɹᵋ 0001/ ɹ ├ ܖ໿ɾਪਐ४උ/ ├ ૯߹ܭըॻ.pdf └ ܖ໿ॻ.pdf ├ ٞࣄ࿥/ └ ͦͷଞ ... ϓϩδΣΫτ
    %# ϓϩδΣΫτ
    ؔ࿈ࢿྉ
    ɾॻྨ { "meta": { "code": "0001", "name": "A project" }, "document": { "area": 2000, "cost": 1000000000, "period": 1000000000, "startDate": "2022-01-01", "deliveryDate": 2023-01-01, "type": "GENERAL", "status": "DONE" }, "embeddings": [ { "vector": [ ... ], "text": { "key": "ಛ௃ᶃ", "value": "ݸ͕ࣨଟ͘ɺಛघͳઃඋͱஅ೤..." } }, { "vector": [ ... ], "text": { "key": "ಛ௃ᶄ", "value": "ਖ਼໘ʹಓ࿏͕͋Γɺଞ͸܏ࣼ஍..." } } ] } ਺஋σʔλͳͲઈର஋ͰදݱͰ͖Δ߲໨͸
    Ϛελʔ%#͔Β&5-ॲཧ ྨࣅ౓ݕࡧʹඞཁͳ
    ߏ଄Խσʔλੜ੒ 
    
12. ϓϩδΣΫτؒͷྨࣅ఺ɾಛ௃Λݟ͚ͭΔͨΊͷج४Λ੔ཧ root/ ɹᵋ 0001/ ɹ ├ ܖ໿ɾਪਐ४උ/ ├ ૯߹ܭըॻ.pdf └ ܖ໿ॻ.pdf ├ ٞࣄ࿥/ └ ͦͷଞ ... { "embeddings": [ { "vector": [ ... ], "text": { "key": "ಛ௃ᶃ", "value": "ݸ͕ࣨଟ͘ɺಛघͳઃඋͱஅ೤..." } }, { "vector": [ ... ], "text": { "key": "ಛ௃ᶄ", "value": "ਖ਼໘ʹಓ࿏͕͋Γɺଞ͸܏ࣼ஍..." } } ] } ϓϩδΣΫτ ؔ࿈
    ࢿྉɾॻྨ 0$3ରԠ
    ͷ
    7-.
    ͰಡΈऔΓ
    ֤߲໨ʹؔ͢ΔධՁͱαϚϦʔΛ࡞੒ ஍Ҭɾ஍ཧతཁ݅
    ɾ஍൫ɺߦ੓ɺؾީ ґཔओʹؔ͢Δࣄ߲
    ɾاۀಛੑɺ࢓༷ ֎؍ɾҙঊతಛ௃
    ɾ֎นɺృ૷ɺ಺૷ ن໛ʹؔ͢Δৄࡉ
    ɾ༧ࢉɺ໘ੵɺ֊਺ ઃඋʹؔ͢Δಛ௃
    ɾ֤छઃඋɺӄѹࣨ౳ ߏ଄తཁ݅
    ɾ޻๏ɺ໔਒ɺ଱਒ 
    
    
13. ࣗવݴޠσʔλͷϋΠϒϦουݕࡧ { embeddings: [ { text: { key: "஍Ҭɾ஍ཧతཁ݅", value: "..." }, vector: [...] }, { text: { key: "ઃඋʹؔ͢Δಛ௃", value: "..." }, vector: [...] }, ] } { query: { hybrid: { queries: [ { // ςΩετݕࡧΫΤϦʔ multi_match: { query: SEARCH_TEXT, fields: ["key", "value"] } }, { // ྨࣅݕࡧΫΤϦʔ knn: { embeddings: { vector: [...] } } } ] } }, search_pipeline: { phase_results_processors: [ { 'normalization-processor': { normalization: { technique: "min_max" }, combination: { technique: "arithmetic_mean", parameters: { // είΞϦϯάॏΈ͚ͮ // [text, vector] weights: [0.3, 0.7] } }, ignore_failure: false } } ] } 
    
    
14. σʔλߏ଄Խॲཧߏ੒ͷ֓ཁ ϓϩδΣΫτ %# ϓϩδΣΫτ ؔ࿈
    ࢿྉɾॻྨ .PEFMT 4UPSBHF &5- 4USVDUVSJOH 0QFO4FBSDI $47
    ࿈ܞ
    όονॲཧ ϑΝΠϧ
    Ξοϓϩʔυ
    όονॲཧ &5-
    ͓Αͼ
    +40/
    ߏ଄Խ 0$3
    4VNNBSJ[JOH
    &NCFEEJOH *OEFYJOH ར༻ͨ͠ੜ੒"*Ϟσϧ
    ɾamazon.nova-[lite/pro]-* ɾamazon.titan-embedv2-[text/image]-* ɾnanonets/Nanonets-OCR-s (OSS) 
    
    
15. -*7&
    %&.0 ྨࣅݕࡧΛద༻ͨ͠΢ΣϒΞϓϦέʔγϣϯͷϥΠϒσϞ 
    
    
16. serverless.co.jp