Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearchによる 全文検索の実装 in Rails
Search
FUJI Goro
March 24, 2018
Technology
5
9.4k
Elasticsearchによる 全文検索の実装 in Rails
Rails Developers Meetup 2018 (#railsdm) の発表資料です。
FUJI Goro
March 24, 2018
Tweet
Share
More Decks by FUJI Goro
See All by FUJI Goro
How to Boost Your Code with WebAssembly
gfx
2
2.8k
AssemblyScriptでライブラリコードの高速化をしてみる
gfx
5
2.9k
実践TypeScriptトークバトル
gfx
1
1.1k
歴史的経緯の説明 as code
gfx
7
2.7k
すばらしきGraphQLのSEKAIへようこそ
gfx
20
9.2k
マルチテナント・ウェブアプリケーションの実践
gfx
14
9.4k
How to choose the ORM on Android
gfx
1
4.1k
How Do We Get Along With Static Types
gfx
5
3.3k
"OSSにコントリビュート" なんてしてる場合じゃない!
gfx
21
15k
Other Decks in Technology
See All in Technology
SREが投資するAIOps ~ペアーズにおけるLLM for Developerへの取り組み~
takumiogawa
1
360
マルチプロダクトな開発組織で 「開発生産性」に向き合うために試みたこと / Improving Multi-Product Dev Productivity
sugamasao
1
310
Terraform未経験の御様に対してどの ように導⼊を進めていったか
tkikuchi
2
450
Amazon Personalizeのレコメンドシステム構築、実際何するの?〜大体10分で具体的なイメージをつかむ〜
kniino
1
100
B2B SaaSから見た最近のC#/.NETの進化
sansantech
PRO
0
860
AIチャットボット開発への生成AI活用
ryomrt
0
170
Security-JAWS【第35回】勉強会クラウドにおけるマルウェアやコンテンツ改ざんへの対策
4su_para
0
180
Why App Signing Matters for Your Android Apps - Android Bangkok Conference 2024
akexorcist
0
130
OCI Vault 概要
oracle4engineer
PRO
0
9.7k
Lexical Analysis
shigashiyama
1
150
Making your applications cross-environment - OSCG 2024 NA
salaboy
0
190
AWS Media Services 最新サービスアップデート 2024
eijikominami
0
200
Featured
See All Featured
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
27
4.3k
Reflections from 52 weeks, 52 projects
jeffersonlam
346
20k
Imperfection Machines: The Place of Print at Facebook
scottboms
265
13k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
47
5k
Designing for Performance
lara
604
68k
Product Roadmaps are Hard
iamctodd
PRO
49
11k
Building Your Own Lightsaber
phodgson
103
6.1k
Building an army of robots
kneath
302
43k
The Power of CSS Pseudo Elements
geoffreycrofte
73
5.3k
Testing 201, or: Great Expectations
jmmastey
38
7.1k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
44
2.2k
How to train your dragon (web standard)
notwaldorf
88
5.7k
Transcript
ElasticsearchʹΑΔ શจݕࡧͷ࣮ in Rails #railsdm by FUJI Goro (gfx)
ࠓ͢͜ͱ • ΣϒαʔϏεͰશจݕࡧΛఏڙ͢Δͱ͖ʹ ͱ͖ʹߟ͑Δ͜ͱ • ElasticsearchʹΑΔશจݕࡧʹ͓͚Δ࣭ͷ ੍ޚʹ͍ͭͯ
ࠓ͞ͳ͍͜ͱ • Elasticsearch vs ଞͷશจݕࡧΤϯδϯ • ϚωʔδυαʔϏε vs ࣗલ •
ಛఆͷ elasticsearch client library ͷ • e.g. elasticsearch-rails, searchkick, elastomer-client
ൃදऀʹ͍ͭͯ • ໊લ: gfx / ౻ޗ • ใڞ༗αʔϏεKibelaΛ։ൃ͍ͯ͠Δ • Kibela
Ruby on RailsαʔϏε • લճͷrailsdmͰʮϚϧνςφϯτɾΣϒΞϓϦέʔ γϣϯͷ࣮ફʯͱ͍͏ൃදΛͨ͠ • KibelaͰ Amazon Elasticsearch Service Λར༻
ΣϒαʔϏεͰશจݕࡧΛ ఏڙ͢Δͱ͖ʹߟ͑Δ͜ͱ
શจݕࡧ in ΣϒαʔϏε • ࣭Λ্͛Δʹఔίετֻ͕͔Δ • ≒ίετΛֻ͚Δ΄Ͳʹ࣭Λ্͛ΒΕΔ • Ͳͷ͘Β͍ίετΛֻ͚ΒΕΔ͔ͰΔ͜ͱ͕ ͔ͳΓมΘΔ
• ίετΛֻ͚ΒΕͳ͍ͳΒͦͦશจݕࡧ ෆཁ͔͠Εͳ͍
ݕࡧͷ࣭ • ݕࡧͷ࣭Ұൠʹʮద߹ʯͱʮ࠶ݱʯ ͰݟΔ • ద߹ʮਖ਼֬ੑʯͱ͍͏ • ࠶ݱʮཏੑʯͱ͍͏
ద߹ vs ࠶ݱ • ద߹:ʮݕࡧ͞Εͨద߹จॻͷ ÷ ݕࡧ݁Ռͷจॻͷ ʯ • ద߹͕ߴ͍ͱʮݕࡧ݁Ռͷଟ͕͘ద߹ੑ͕ߴ͍ʯ
• ࠶ݱ:ʮݕࡧ͞Εͨద߹จॻͷ ÷ શରจॻதͷਖ਼ ղจॻͷʯ • ࠶ݱ͕ߴ͍ͱʮద߹ੑͱ͔͘औΓ͜΅͠ͳ͍ʯ
ద߹ͱ࠶ݱͷࢉग़ • ԿΛ࣋ͬͯʮద߹ʯͱ͢Δ͔Ϣʔβʔͷ಄ͷதʹ ͷΈ͋Δ • ≒ ಉ͡ݕࡧΩʔϫʔυͰϢʔβʔ͝ͱʹٻΊΔ ͷҟͳΔՄೳੑ͕͋Δ • ࣮ࡍͷαʔϏεͰଌఆ͢Δ͜ͱෆՄೳ
• ⇢ϢʔβʔͷߦಈΛੳͯ͠ධՁ͢Δඞཁ͕͋Δ
ద߹ͱ࠶ݱͰॏࢹ͢Δͷʁ • Ұൠʹద߹ͱ࠶ݱτϨʔυΦϑͷ͕ؔ͋ Δͱ͍ΘΕΔ • ͲͪΒ͕ॏཁ͔αʔϏεͷੑ࣭ʹΑΔ • KibelaͷΑ͏ͳاۀͷจॻݕࡧͰʮऔΓ͜΅ ͠ ≒
όάʯͱೝࣝ͞ΕΔ • ͭ·Γ࠶ݱʮ࠷ݶकΔ͖࣭ʯͰ͋Δ
ElasticsearchʹΑΔશจݕࡧ ʹ͓͚ΔݕࡧείΞͷߟ͑ํ
Elasticsearchͱ • શจݕࡧΤϯδϯ • SQLͷLIKEݕࡧͱͷҧ͍: • ܗଶૉղੳޙͷ୯ޠͰݕࡧͰ͖Δ • ਖ਼نԽ͞Εͨ୯ޠͰݕࡧͰ͖Δ •
ෳࡶͳείΞʹجͮ͘ιʔτ͕Ͱ͖Δ
Elasicsearch༻ޠ RDBMS Table Record Column Elasticsearch Index Document Field ※
Elasticsearch v5·Ͱ table = type ͱઆ໌͞Ε͖͕ͯͨ v6 Ͱtypeඇྭɺ v7 Ͱআ͞ΕΔͷͰҙ
ܗଶૉղੳ
ܗଶૉղੳͯ͠ݕࡧ • શจݕࡧͷߴԽͷͨΊʹసஔΠϯσΫεͳͲͷ ΞϧΰϦζϜΛ͍͍ͨ • సஔΠϯσΫεͷͨΊʹೖྗ͞ΕΔจষΛదͳ ୯ҐͰׂ͢Δඞཁ͕͋Δ • ຊޠͷΑ͏ʹ୯ޠͷ۠Γ͕͖ͬΓ͠ͳ͍ݴ ޠͷ߹N
Gram and/or ܗଶૉղੳͰׂ͢Δ
N Gram • N௨ৗ2͔3 • 2 gram ͷ߹ɺʮ͜ΜʹͪʯΛ • [“͜Μ”,
“Μʹ”, “ʹͪ”, “ͪ”] ʹ͚Δ
ܗଶૉղੳ • ʮ౦ژʹҾͬӽͨ͠ʯΛ • [“౦ژ”, “ʹ”, “Ҿͬӽ͠”, “ͨ”] ͳͲʹׂ ͢Δ
• ࢺʢ໊ࢺ, ܗ༰ࢺʣใ͕ར༻Ͱ͖ΔͷͰɺ ޙͷਖ਼نԽͰͦͷใΛར༻
LIKEݕࡧͱͷࠩ: ܗଶૉղੳฤ • “౦ژʹҾͬӽͨ͠” LIKE “%ژ%” • Ϛον͢Δ • [“౦ژ”,
“ʹ”, “Ҿͬӽ͠”, “ͨ”].include?(“ژ ”) • Ϛον͠ͳ͍
ܗଶૉղੳͱN Gramͷࠩ • ͨͱ͑ʮژʯͰʮ౦ژʯΛؚΉจॻΛ… • ܗଶૉղੳϕʔεͩͱݕࡧͰ͖ͳ͍ • ͭ·Γద߹ͷ໘Ͱ༏Ε͍ͯΔ • N
GramϕʔεͩͱݕࡧͰ͖Δ • ͭ·Γ࠶ݱͷ໘Ͱ༏Ε͍ͯΔ
ਖ਼نԽ
ͳͥਖ਼نԽ͕ඞཁ͔ • ܗଶૉղੳʴసஔΠϯσΫεʹΑΔݕࡧͷࣜ Խ: [“Ҿͬӽ͠”, “ͨ”].include?(“Ҿͬӽ͠”) • “Ҿӽ”, “Ҿӽ͠” ಉ༷ʹݕࡧ͍ͨ͠
⇢ ਖ਼نԽ
ܗଶૉղੳͱਖ਼نԽ • ܗଶૉղੳث͕ࣙॻͱࢺใΛͬͯਖ਼نԽ͢Δ • Elasticsearchͷ߹kuromoji plugin͕ຊޠͷܗଶૉղੳ ͱਖ਼نԽΛαϙʔτ • kuromoji_baseform: “ҿΈ”
⇢ “ҿΉ” • kuromoji_stemmer: “αʔόʔ” ⇢ “αʔό” • kuromoji_number: “ඦԁ” ⇢ “100ԁ”
είΞϦϯά
ݕࡧॱͷॏཁੑ • ͨͱ͑ɿʰژʱͰʰ౦ژʱͱ͍͏จॻ ώοτͯ͠΄͍͕͠ɺʰژʱͰώοτ͢ Δจॻ͕͕͋ΔͳΒͦͪΒΛ্Ґʹ͍࣋ͬͯ ͖͍ͨ • ͭ·Γߴ͍࠶ݱΛҡ࣋ͭͭ͠ݕࡧ݁Ռͷ্ ҐʮΑΓద߹͢ΔͱࢥΘΕΔʯจॻʹͨ͠ ͍
୯ޠׂ๏ʹΑΔॏΈ͚ • ܗଶૉղੳͰׂ͢Δͱద߹ߴ͘ͳΓ͕ ͕ͪͩ࠶ݱ͘ͳΓ͕ͪ • N GramͰׂ͢Δͱ࠶ݱߴ͘ͳΓ͕ͪͩ ͕ద߹͘ͳΓ͕ͪ • ⇢είΞϦϯάͷࡍʹॏΈ͚Λ͢ΕΑ͍
ElasticsearchͰͷ࣮ํ๏ • elasticsearch-model Ͱͷσʔλιʔεఆٛ UJUMFLVSPNPKJͰ ׂ͢Δ UJUMF@OHSBN/ (SBNͰׂ͢Δ
ϑΟʔϧυ͝ͱͷॏΈ͚ • ElasticsearchͰϑΟʔϧυ͝ͱʹॏΈ͚ ΛઃఆͰ͖Δ
ͦͷଞͷείΞϦϯάख๏ • Phrase Match / ϑϨʔζϚον • Field Value Factor
/ ϑΟʔϧυ • Decay / είΞͷݮਰ
Phrase Match • TwitterݕࡧGoogleݕࡧͰμϒϧΫΥʔτͰғΜͩͱ͖ʹ ʮϑϨʔζͦͷͷʯ͕ݕࡧରʹͳΔͷ͕Phrase Match • Kibelaͷ߹ɺϑϨʔζʹϚονͨ͠ΒείΞΛϒʔετ ͍ͯ͠Δ •
ྫ:ʮঁਆసੜʯͰݕࡧ͢ΔͱʮਅɾঁਆసੜIII NOCTURNEʯͷ΄͏͕ʮҟੈքʹঁਆͱͯ͠సੜ͠ ͨʯΑΓείΞ͕ߴ͘ͳΔ
Field Value Factor • ಛఆͷϑΟʔϧυͷͰείΞΛϒʔετͤ͞ Δػೳ • Kibelaͷ߹ɺϥΠΫͱτϥοΫόοΫΛ Field Value
Factorͱͯ͠ө͍ͤͯ͞Δ • ͭ·ΓʮΑ͘ϥΠΫ͞ΕΔจॻʯʮΑ͘Ϧ ϯΫ͞ΕΔจॻʯͷείΞ͕ߴ͍
Decay • Field Value FactorͷٯͰɺಛఆϑΟʔϧυ ͰείΞΛݮਰͤ͞Δػೳ • KibelaͰɺจॻͷλΠϜελϯϓΛར༻͠ ͕ͯ࣌ؒܦͭ΄ͲʹείΞ͕ݮਰ͢ΔΑ͏ʹ ͍ͯ͠Δ
DecayͷΠϝʔδ • origin = now / timestampաڈͷҰͱ͢Δ
·ͱΊ • શจݕࡧͷ࣭ʮద߹ʯʮ࠶ݱʯʮε ίΞϦϯάʯͰߟ͑Δ • ʮద߹ʯͱʮ࠶ݱʯ୯ޠׂ๏ʹΑͬ ͯվળͰ͖Δ • ʮείΞϦϯάʯ༷ʑͳཁૉ͕͋ΔͷͰશ จݕࡧΤϯδϯͷυΩϡϝϯτΛख़ಡ͢Δ