Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Elasticsearchで多言語検索対応してみた話.pdf
Search
motsat
July 19, 2018
Programming
1.5k
2
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Elasticsearchで多言語検索対応してみた話.pdf
motsat
July 19, 2018
Transcript
34 slides
ElasticsearchͰଟݴޠݕࡧର Ԡͯ͠Έͨ
ࣗݾհ ɹɾ໊લ ɹɹࠤ౻ ݩل ɹɾϝυϐΞྺ ɹɹ 2017/9 ʙ WebͷιϑτΤΞΤϯδχΞɹ ɹ
͜Ε͔Β͓͢͠Δ͜ͱ ɾӳޠυΩϡϝϯτΛຊޠͰݕࡧ͢ΔΑ͏ͳɺҟͳΔݴޠ ؒͷݕࡧ ɾ༁Λߦ͏ࣄ͕Ͱ͖ͳ͍߹ͷରԠ ɾElasticsearchͰߦͬͨࡍͷํ๏ͷ1ͭͱɺ ɹϝϦοτ/σϝϦοτ
ҩࢣઐ༻αΠτʮMedPeerʯ
㲔 ৽αʔϏεͷ։ൃ
ʮPubmedจΛຊޠͰޮΑ͘ݕࡧʯ
PubMedจʁ ɹւ֎ҩֶจݙใͷσʔλϕʔε ɹɾӳޠ ʢ͘͝·Εʹผͷݴޠʣ ɾAPIɺFTPͰͷϑΝΠϧऔಘʹରԠ͍ͯ͠Δ
ɾӳจυΩϡϝϯτΛຊޠͰݕࡧ ͍ͨ͠ ɾݕࡧରλΠτϧɺຊจ ࣮ݱ͢Δ͜ͱ
ຊޠʹ༁͓͚ͯ͠ ͳ͍ͣ
༁Λߦ͏ͨΊͷAPI ɾGoogle Translation API 100 ສจࣈ - 20υϧ ɹˠ Pubmed༁ʹෆࣗવͳ͕গͳ͍
ɾMicrosoft Translator API 100 ສจࣈ - 10υϧʢ1120ԁ) ɹˠ ྉ͍͕ۚ҆ɺPubmed༁͢Δͱෆࣗવͳ͕ΘΓͱ͋Δ ɾAmazon Translate ɹຊޠະରԠ(2017/ळࠒ)
༁࣭Λ༏ઌͯ͠ɺPubmed༁ʹෆ ࣗવͳ͕গͳ͔ͬͨͷͰ Google Translation API ʹܾఆɻ
༁ྉۚ
ɾPubMedจ ɹ1700ສ݅ʢMedPeerͰͷऔΓࠐΈରɻʑ૿Ճʣ ɾฏۉจࣈ ɹ1300จࣈ (λΠτϧ100จࣈɺຊจ1200จࣈ) ɹɹɹɹɹˣɹɹɹɹ ɾ߹ܭจࣈ ɹ221ԯจࣈ (100 +
1200) * 1700ສ݅
Google Translation API100 ສจࣈ - 20υϧ ɾֹۚ (221ԯ / 100ສจࣈ)
* 20υϧ = 442000υϧ ɹ ຊԁ = 4889ສԁ ɹ (2017/07/07࣌)
4889ສԁ ߴ͍ʢฐࣾج४ʣ
4889ສԁߴ͍ ɾશͯ༁͢Δͱߴ͗͢Δ ɾͱ͍͑ɺ݅ݮΒͨ͘͠ͳ͍ → ʮӳจυΩϡϝϯτΛຊޠͰݕࡧʯΛͲ ͏͢Δ͔
1.ݩυΩϡϝϯτΛ༁͍ͯ͠ͳͯ͘ ݕࡧՄೳʹ → ຊޠݕࡧʹࣙॻΛ͏ɻ ɹElasticsearchͷʮSynonym Token Filterʯ
Elasticsearch Synonym Token Filter https://www.elastic.co/guide/en/elasticsearch/reference/current/ analysis-synonym-tokenfilter.html
ಉҙޠྨٛޠΛઃఆͰ͖Δػೳɻ ྫʣ ͱ͍͏ఆ͕ٛ͋Εɺ ʮi-podʯͰݕࡧ →ʮi podʯʯʮipodʯʹώοτ ʮi podʯͰݕࡧ →ʮi-podʯʯʮipodʯʹώοτ Synonym
Token Filter i-pod, i pod => ipod
ߴ݂ѹ => hypertension ΠϯϑϧΤϯβ => influenza ͜ΕΛ͍ɺ ຊޠ/ӳޠΛؔ࿈͚ͮΔ
pubmed: { properties: { title_en: { type: "text", analyzer: “english_analyzer"
}, title_ja: { type: "text", analyzer: "ja_analyzer" }, body_en: { type: “text”, analyzer: "english_analyzer" }, body_ja: { type: "text", analyzer: "ja_analyzer" }, }, }, Indexͷproperties(Ϛοϐϯά) ɹɾຊޠϑΟʔϧυ(title_ja/body_ja)ɺ ɹɹӳޠϑΟʔϧυ(title_en/body_en)ΛλΠτϧ/ຊจ ɹɹͦΕͧΕͰ༻ҙ ɾӳޠϑΟʔϧυɺຊޠϑΟʔϧυͰanalyzerΛ͚Δ Elasticsearch༻ͷઃఆ
Indexͷanalysisઃఆ ɹɾfilterʹtype:”synonym”ͰઃఆՃ ɹɾӳޠϑΟʔϧυ༻ͷʮenglish_analyzerʯͷfillterʹɺ ɹɹsynonym filterΛ͏Α͏ઃఆ(ଞެࣜͷEnglishઃఆΛϕʔεʹ) Elasticsearch༻ͷઃఆ { “index” : {
“analysis“: { “filter“ : { “synonym“ : { “type“ : "synonym", “synonyms“ : [‘ߴ݂ѹ => hypertension’, …]}, “analyzer“: { “english_analyzer“: { “tokenizer”: "standard", “filter”: [“synoncym”,”english_possessive_stemmer”,”lowercase “,…] }, …}, }, }, }
Analyze݁Ռ (kibana) ʮߴ݂ѹʯˠʮhypertensionʯͷtokenͱͳΔ (࣮ࡍʹɺઌఔͷαϯϓϧͷଞͷfilterʹΑΓՃ͞Εͨtokenʣ
Ωʔϫʔυʮߴ݂ѹʯͰຊޠ/ӳޠݕࡧ͕Մೳʹ ɾຊޠͷʮߴ݂ѹʯ ɾӳจͷʮhypertensionʯ
ࣙॻͲ͜ͰखʹೖΕΔ͔ ༗໊ͳࣙॻ ɾJMdict (Japanese-Multilingual Dictionary) http://www.edrdg.org/jmdict/j_jmdict.html ɹӳޠҎ֎ೖ͍ͬͯΔΑ͏ͳͷͰɺ ɹෳݴޠͷઃఆͰ͖Δ͔͠Ε·ͤΜɻ
ͲΜͳࣙॻͰྑ͍ͷ͔ ɾઐ༻ޠݫ͍͠ ɹઐ༻ޠ(ҩྍ)ઐͷࣙॻͰͳ͍ͱ୯ޠ͕ཏͰ͖ͳ͍ࣄ͕ ଟ͍ɻ ɹˠ ઐྖҬͷࣙॻΛ୳͢ɺݕࡧϩά͔ΒࣙॻΛΞοϓσʔτ͢Δ ͳͲ ϝυϐΞגࣜձࣾϩθολ༷ͱܖ͠ɺҩྍʹಛԽͨࣙ͠ॻΛఏ ڙ͖ͯ͠·ͨ͠ɻ
ࣙॻʹΑΔଟݴޠͷݕࡧ ϝϦοτ ɹɾશ༁͠ͳͯ͘ݕࡧͰ͖ΔΑ͏ʹͳΔ σϝϦοτ ɹɾࣙॻͷ༻ҙ͕ඞཁʹͳΔ ɹɾݕࡧਫ਼͕ࣙॻͷ࣭ʹࠨӈ͞Ε͍͢ ɹɹˠ จষͷ୯ޠΛཏͰ͖ͳ͍Մೳੑେ͖͍ ɾ୯ޠϕʔεͷݕࡧʹڧ͍͕ɺจͷݕࡧʹऑ͍ ɾݕࡧͰ͖Δ͕ɺ݁Ռදࣔӳޠ
Ϣʔβʔ ΠϯλʔϑΣʔε͍ͩ͠Ͱ͕͢ɺ ɾλΠτϧࣄલ ɹ༁APIͷ͕ͪ࣌ؒͭΒ͍ʢҰཡͰͬ͞ͱݟΔఔͳͷͰʣ ɹຊจΑΓจࣈগͳ͍ͷͰɺશ༁ֹͯۚ͠Ί 2. Ұ෦ࣄલ༁ɺΓදࣔ࣌ʹ
ɾຊจৄࡉදࣔ࣌ ɹදࣔϖʔδͰඇಉظͰॲཧ͞Εͨ༁݁ՌΛදࣔɹ 2. Ұ෦ࣄલ༁ɺΓදࣔ࣌ʹ
ྉۚ ɾ༁ྉ(ݻఆ) ɹ ࣄલʹλΠτϧ͚ͩ༁͢Δࣄʹͨ͠ ɹ 1700ສ݅ * λΠτϧͷΈ(100จࣈ) = 376ສԁɹ(Google
Translate API) ɾ༁ྉ(มಈ) ɹදࣔ࣌ʹ༁͢Δࣄʹͨ͠ = αສԁʢPVඇެදͷؔͰग़ ͤͳ͍Ͱ͕͢ɺݕࡧ͞Εදࣔ͞ΕΔϖʔδҰ෦ͳͷͰ͔ͳΓ҆͘ʣ ɾࣙॻ ɹ גࣜձࣾϩθολ༷ (ҩྍܥಛԽͷӳࣙॻ)ɹ= bສԁʢ͜Εެ ද☓ͳͷͰ͕͢ɺۃΊ͓ͯ҆͘ఏڙ͍͍ͯ·͢ʣ
࠷ऴతͳֹۚ 4889ສԁ → 376+α+b ສԁ େৎʢฐࣾج४ʣ
ิ 1700ສ݅Ҏ߱ʢPubMedͷίϯςϯπʑ૿Ճʣ ͷ༁ɺΑΓҩྍಛԽͷ༁Λߦ͏ͨΊGoogle༁͚ͩͰ ͳ͘ɺԼهͷύʔτφʔ༷ͱܖΛߦ͍APIΛར༻͍ͯ͠· ͢ɻ ɾגࣜձࣾγΣΞϝσΟΧϧ༷ ɹҩྍܥಛԽͷ༁API ɹhttps://www.ikotoba.jp/ ɾגࣜձࣾϩθολ༷ PubMedಛԽͷ༁API
ɹhttps://www.rozetta.jp/
·ͱΊ - ݴޠ͕ҟͳΔυΩϡϝϯτݕࡧʹɺ ɹ ElasticsearchͷSynonym Token FilterΛͬͨ - ༁ྉݮΒͤΔ͕ɺϝϦοτ/σϝϦοτ͋Γ
- ୯ޠͰͷݕࡧΛओͱ͢Δ߹ʹద༻͍͢͠
͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠
More Decks by motsat
See All by motsat
「SmartHR基本機能」の溜まっていく技術課題への取り組み
motsat
0
1.9k
メドピアの輪読会
motsat
2
1.4k
Other Decks in Programming
See All in Programming
AI時代のFlutter品質設計 — アプリ品質を守るQuality Layers
omatty198
0
390
学びを記事・スライド・Xへ高速展開する― AI-assisted Learning Loopで回すコンテンツ生成術
ochtum
0
210
疎結合とは何か?ソフトウェア設計における結合バランスについて考える #phpcon_niigata
kajitack
2
330
LLM 101
dalinaum
0
510
RaspberryPi 5で軽くはじめるk3sとtailscaleを用いたホームサーバー構築 (1)
nasum
0
410
DroidKnights 2026 Android A2UI Jetpack Compose 기반의 안전한 AI UI 렌더링 - 박제창
itsmedreamwalker
1
250
protojson における文字化けの扱い
kazuki1023
0
450
AI時代のIaC入門 〜なぜインフラをコードで管理したいのか?〜 / Introduction to IaC in the AI Era
akihisaikeda
6
1.6k
私なりのハーネスエンジニアリング ~独自の言語処理系の開発を通して~
seosoft
0
220
AI時代のTerminal環境を UNIX哲学で設計・運用する
yasunori0418
1
360
例外からRich Errorsまで Kotlinエラーハンドリングの進化を読み解く
shiita0903
0
220
Growing PicoRuby's Ecosystem on ESP32
yuuu
2
150
Featured
See All Featured
Keith and Marios Guide to Fast Websites
keithpitt
413
23k
Building a A Zero-Code AI SEO Workflow
portentint
PRO
0
840
Utilizing Notion as your number one productivity tool
mfonobong
4
650
AI Search: Where Are We & What Can We Do About It?
aleyda
0
8.2k
Deep Space Network (abreviated)
tonyrice
0
420
Ethics towards AI in product and experience design
skipperchong
2
470
Why You Should Never Use an ORM
jnunemaker
PRO
61
10k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
6.2k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
35
2.9k
The Power of CSS Pseudo Elements
geoffreycrofte
82
6.6k
Lightning talk: Run Django tests with GitHub Actions
sabderemane
0
320
How to Ace a Technical Interview
jacobian
280
24k