俺の全文検索エンジン(Go製)を作り始めた

1. 2020/11/11 @kotaroooo0 ԶͷશจݕࡧΤϯδϯ(Go੡) Λ࡞Γ࢝Ίͨ
2. ͜ͷLTΛฉ͘ͱ… 1. ͳΜͱͳ͘શจݕࡧΤϯδϯͷ࢓૊Έ͕෼͔ Δ 2. GoͰશจݕࡧΤϯδϯΛ࡞Γ࢝ΊΒΕΔ
3. -શจݕࡧΤϯδϯͷ࢓૊ΈΛ஌Δ -ElasticsearchͰ΋࢖ΘΕ͍ͯΔApache LuceneΆ͍࢓༷ͷ΋ͷΛ࡞Δ -GoΛֶΜͰ͍ΔͷͰԿ͔࡞Γ͍ͨ -Twitter BotΛ࡞͍ͬͯΔ͕ɺͪΐ͏Ͳ͍͍શจݕࡧΤϯδϯ͕ͳ͍ -ܰྔɺ͔ͭॏΈ෇͖ϨʔϕϯγϡλΠϯڑ཭ΛܭࢉͰ͖Δ΍ͭ શจݕࡧΤϯδϯΛ࡞Δཧ༝
4. 3෼Ͱ෼͔ΔશจݕࡧΤϯδϯ
5. શจݕࡧͷ࢓૊Έ INDEXING ୯ޠ จॻ have 1,2 pen 1 we 2 Desk 2 จॻ1 “I have a pen.” จॻ2 “We have desk.” CHAR FILTER TOKENIZER TOKEN FILTER Analyzer
6. શจݕࡧͷ࢓૊Έ SEARCH ୯ޠ จॻ have 1,2 pen 1 we 2 Desk 2 ݕࡧϫʔυ: “pen” จॻ1͕ώοτ CHAR FILTER TOKENIZER TOKEN FILTER Analyzer
7. ANALYZER͸ͳͥඞཁ? - τʔΫϯ෼ׂͯ͘͠ΕΔͨΊ - “I have a pen.” ͜ͷ··Ͱ͸సஔΠϯσοΫε Λ࡞੒Ͱ͖ͳ͍ͷͰɺI, have, a, penͱτʔΫ ϯ෼ׂ͍ͨ͠ - ΫΤϦͷදه༳ΕΛٵऩͨ͠Γ͢ΔͨΊ - “GOD”ͱ͍͏୯ޠΛؚΉυΩϡϝϯτ ͸ɺ”god”Ͱ΋ώοτ͢ΔΑ͏ʹখจࣈʹ౷Ұ ͍ͤͨ͞ - ແବͳτʔΫϯͷϑΟϧλϦϯά - theͳͲΠϯσΩγϯάͯ͠΋ແବ Analyzeલ Analyzeޙ “I have a BIG pen!” have, big, pen
8. ۩ମతͳANALYZERྫ - Char Filter(Tokenizerͷલʹॲཧ͢Δ) 0ݸҎ্ - Mapping: إจࣈΛ୯ޠʹม׵ͳͲ - HTMLstrip: HTMLΛύʔε - Tokenizer(τʔΫϯ෼ׂ͢Δ) 1ݸ - Standard: εϖʔεͳͲϧʔϧʹैͬͯ෼ׂ - Kuromoji: ܗଶૉղੳͰ෼ׂ - Ngram: Nจࣈ͝ͱʹ෼ׂ - Token Filter(Tokenizerͷޙʹॲཧ͢Δ) 0ݸҎ্ - Lowercase: খจࣈ - Stopword: ετοϓϫʔυআڈ - Stemming: දه༳Ε CHAR FILTER TOKENIZER TOKEN FILTER Analyzer
9. ࣮૷
10. ANALYZERͷ࣮ߦΠϝʔδ Analyzeલ MappingCharFilter StandardTokenizer LowercaseFilter StopWordFilter StemmerFilter I have a lot of TASKs. I am very sad :( I have a lot of TASKs. I am very sad _sad_ I, have, a, lot, of, TASKs, I, am, very, sad, sad I, have, a, lot, of, tasks, I, am, very, sad, sad lot, tasks, am, very, sad, sad lot, task, am, very, sad, sad ॲཧͷྲྀΕ
11. ANALYZERͷ࣮૷
12. CHAR FILTERͷ࣮૷
13. TOKENIZERͷ࣮૷
14. TOKEN FILTERͷ࣮૷
15. INDEX - సஔΠϯσοΫε͸ map[string][]int - సஔΠϯσοΫεΛϑΟʔϧυ ໊͝ͱʹ࣋ͭ - υΩϡϝϯτ͸ɺIDͱϑΟʔϧ υΛ࣋ͭ - Indexing͢Δͱ͖΋AnalyzerΛ ௨͢
16. SEARCH -ANDݕࡧͱORݕࡧ -ΫΤϦ: “pink orange blue” -ANDݕࡧ: 3 -OR: ݕࡧ1,2,3,4,5,6 pink Orange blue 6 3 4 5 2 1
17. ಈ͔ͯ͠ΈΔ
18. ݕࡧྫ - ※Analyzer͸͖ͬ͞ͱಉ༷ - Index΋Search΋AnalyzerΛ௨͢ - ”foxes”Ͱ”fox”ΛؚΉυΩϡϝϯτ͕Ϛον - ”happy”Ͱ”:)”ΛؚΉυΩϡϝϯτ͕Ϛον - ANDݕࡧ - fine,FaX,foxes,happy͕શؚͯ·Ε͍ͯΔυΩϡϝ ϯτ1,2͕ώοτ͍ͯ͠Δ
19. ࠓޙ -͍͋·͍ݕࡧΛ࣮૷͢Δ -Fuzzy QueryɺSuggesters -είΞܭࢉΛ࣮૷͢Δ -IFIDF, BM25
20. ࢀߟ -https://github.com/kotaroooo0/stalefish -https://artem.krylysov.com/blog/2020/07/28/lets-build-a-full-text-search-engine/