Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Elasticsearch at Wantedly
Search
Seigo Uchida
April 22, 2014
Technology
13
17k
Elasticsearch at Wantedly
2014/04/21 第 4 回 Elasticsearch 勉強会での発表資料。
当日の質問タイム及び懇親会で聞かれたことを追加しています。
Seigo Uchida
April 22, 2014
Tweet
Share
More Decks by Seigo Uchida
See All by Seigo Uchida
Kubernetes Cluster Monitoring
spesnova
4
5.9k
How we monitor microservices at Mercari microservices platform team
spesnova
13
10k
Kubernetes RBAC in microservices
spesnova
7
4.3k
Introduction to kustomize
spesnova
17
47k
Kubernetes 運用設計ガイド / A design guide for Kubernetes in production (Japanese)
spesnova
80
18k
Microservices Monitoring at mercari
spesnova
3
4.1k
Monitoring Kubernetes with Datadog
spesnova
12
4.5k
My Docker Best Practice (2016 Winter Short Version)
spesnova
33
7.6k
CoreOS 運用の所感
spesnova
17
9.4k
Other Decks in Technology
See All in Technology
FastAPIの魔法をgRPC/Connect RPCへ
monotaro
PRO
1
640
生成AIを活用したZennの取り組み事例
ryosukeigarashi
0
180
Why React!?? Next.jsそしてReactを改めてイチから選ぶ
ypresto
10
4k
インサイト情報からどこまで自動化できるか試してみた
takas0522
0
130
生成AIで「お客様の声」を ストーリーに変える 新潮流「Generative ETL」
ishikawa_satoru
1
260
Oracle Cloud Infrastructure:2025年9月度サービス・アップデート
oracle4engineer
PRO
0
340
AI×Data×SaaS×Operation
sansantech
PRO
0
110
Tomorrow graphlib, Let us use everybody
hayaosuzuki
0
160
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
Goに育てられ開発者向けセキュリティ事業を立ち上げた僕が今向き合う、AI × セキュリティの最前線 / Go Conference 2025
flatt_security
0
310
KAGのLT会 #8 - 東京リージョンでGAしたAmazon Q in QuickSightを使って、報告用の資料を作ってみた
0air
0
190
AIが書いたコードをAIが検証する!自律的なモバイルアプリ開発の実現
henteko
1
240
Featured
See All Featured
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Fireside Chat
paigeccino
40
3.7k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
34
6.1k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
840
jQuery: Nuts, Bolts and Bling
dougneiner
64
7.9k
Speed Design
sergeychernyshev
32
1.1k
Building Better People: How to give real-time feedback that sticks.
wjessup
368
20k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.5k
Balancing Empowerment & Direction
lara
4
670
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
114
20k
Transcript
Elasticsearch at
SEIGO UCHIDA
!TQFTOPWB w :BIPP+"1"/৽ଔೖࣾ w ݄8BOUFEMZʹδϣΠϯ w ٛͷΠϯϑϥΛ୲ w $IFGͱ)VCPUͱ͔͕͖
!TQFTOPWB
None
γΰτͰίίϩΦυϧਓΛ;͢
None
None
8BOUFEMZ *OD w ݄ઃཱ w ΞϧόΠτɺΠϯλʔϯؚΊ໊ w ໊ͷΤϯδχΞ͕׆༂ w γΰτॆ࣮ԽΞϓϦπʔϧͷఏڙ
w ࣾɺਓ͕͏
ੵۃ࠾༻த https://www.wantedly.com/projects/5106
WantedlyͰESΛ ͲͷΑ͏ʹ͍ͬͯΔͷ͔
ݕࡧର w γΰτ w ձࣾ w Ϣʔβ
γΰτ w ձ໊ࣾ w ۈ w ࣾһ
None
ืू w ืू໊ w ืूৄࡉ w ืू৬छ
None
Ϣʔβ w Ϣʔβ໊ w ࣗݾհจ w ͦͷଞ͍Ζ͍Ζʢޙड़ʣ
None
ΠϯσοΫε w ௨ৗͷݕࡧ༻ΠϯσοΫε w "VUPDPNQMFUF༻ΠϯσοΫε
None
λΠϓ w γΰτλΠϓ w ձࣾλΠϓ w ϢʔβʔλΠϓ
௨ৗͷݕࡧ༻ΠϯσοΫε w υΩϡϝϯτɿສ w ϓϥΠϚϦαΠζɿ(#
"VUPDPNQMFUF༻ΠϯσοΫε w υΩϡϝϯτɿສ w ϓϥΠϚϦαΠζɿ(#
ߏ w 3BJMTPO)FSPLV w &MBTUJDTFBSDIPO&$ w 1PTUHSFTRMPO3%4
FMBTUJDTFBSDISVCZ w UJSFϝϯς͕ऴྃ w TFBSDILJDLUJSFʹґଘ w ެࣜHFNΛ͏ͷ͕༨ܭͳ੍ݶ͕ͳ ͯ͘ྑ͍
Ϋϥελ w &$NTNBMMº w $IFGͰཧ w +7.ׂ.&.ɿࡌ.&.
ར༻ϓϥάΠϯ w )FBE w )2ʢ/PEF%JBHOPTUJDT4UBUT֬ೝʹར༻ʣ w .BSWFM TFOTFͰΫΤϦ࡞ʹར༻ʣ ։ൃڥ ͷΈʣ
w ,VSPNPKJ w *ORVJTJUPSʢ"OBMZ[FSͷڍಈ֬ೝʣ
Ͳ͏ֶͬͯΜ͔ͩ
None
ެࣜϦϑΝϨϯε w ඞཁͳ͜ͱॻ͍ͯ͋Δɺؾ͕͢Δ w ΫΤϦͷྫ͕෦తʹॻ͔Ε͍ͯͯࠔΔ w ݕࡧॳ৺ऀʹશମ૾͕͔ͭΈͮΒ͍ w ͰಡΜͩ
None
&MBTUJDTFBSDI8PSLTIPQ w શମ૾Λ௫Έͭͭɺ࣮ࡍʹ৮ΔͷͰΠ ϝʔδ͕௫Ίͨ w IUUQTTQFBLFSEFDLDPNEBEPPOFUFMBTUJDTFBSDIXPSLTIPQ
None
&MBTUJDTFBSDIνϡʔτϦΞϧ w ༻ޠɺશମ૾͕௫Έ͍͢ w ίϯςΩετ͕͋Γ࣮ફత w IUUQDPEFIBUFOBCMPHDPNFOUSZ
͜ͷຊ͕ͬͱૣ͘ग़ͯΕʜ
ۤ࿑ͨ͠ͱ͜Ζ
Ϣʔβʔݕࡧ w ϓϩϑΟʔϧ߲͕ଟ͍ w ֤߲ຖʹϓϥΠόγʔϨϕϧ ͕͋Δ w NBQQJOHRVFSZෳࡶʹ
Ͳ͏ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ྫ͑ɺϓϩϑΟʔϧͷʮ໊લʯʮॴଐʯʮॴ ࡏʯΛجຊใͷηοτͱͯ͠ϓϥΠόγʔ ϨϕϧʮҰൠެ։ʯͱ͍ͯͨ͠ͱ͢Δ w
࠷ॳ/FTUFEEPDVNFOUΛͬͯදݱ͠ ͍ͯͨ
"properties": {! "basic_privacy": {! "type": "nested"! },! "properties": {! "name":
{! "type": "string"! },! "working_for": {! "type": "string"! },! "location": {! "type": "string"! },! "privacy_level": {! "type": "string"! }! }! }! mapping
Ͳ͏ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ͔͠͠ɺ/FTUFEͩͱͰ͖ͳ͍͜ͱʢϋΠϥΠτͳͲʣ ͕ग़ͯ͘ΔͷͰϑϥοτͳߏͰɺϓϥΠόγʔϨϕ ϧΛදݱ w
ܕ@σʔληοτ@@ϓϩύςΟͷΑ͏ʹϚοϐ ϯά w ࠷ॳͷܕ͕ඞཁͳͷɺEBUFJOUFHFSͳͲ TUSJOHҎ֎ʹTJNQMF@RVFSZΛ͏ͱFSSPSͱͳΔͨ Ί
mapping query {! "query": {! "simple_query_string": {! "query": <QUERY>,! "fields":
["s_basic__*"],! "default_operator": "and"! }! }! },! "filter": {! "term": {! "basic_privacy": <PRIVACY_LEVEL>! }! }! }! "properties": {! "basic_privacy": {! "type": "string"! },! "s_basic__name": {! "type": "string"! },! "s_basic__working_for": {! "type": "string"! },! "s_basic__location": {! "type": "string"! }! }!
Ͳ͏ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ݁ՌతʹɺOFTUFEΛΘ͟ΔΛಘͳ͍໘ ͕ग़ͯདྷͯϑϥοτͳϓϩύςΟͱOFTUFE Λซ༻͍ͯ͠Δ͕ɺࠓճͦ͜লུ
࣭λΠϜ
4PMS$MPVETFBSDIධՁ্ͨ͠Ͱ &4ʹͨ͠ͷ͔ w ʮ༑ୡͱͷܨ͕ΓʯΛՃຯͯ͠ݕࡧͰ͖Δ͔ͱ͍͏͜ͱͰɺάϥϑ %#ͷ/FPKͱҰൠతͳݕࡧͱͯ͠&4Λൺֱͯ͠ɺ&4ͷํ͕அ વύϑΥʔϚϯε͕ྑ͔ͬͨ w 4PMSʹ͍ͭͯ4PMSϢʔβ͔Βࠓ͔Β͏ͳΒ&4ͱ͍͏Λฉ͍ ͍ͯͨ w
ར༻Ϣʔβʢ(JU)VC GPVSTRVBSF 4PVOE$MPVEͳͲʣࢀߟʹ͠ ͨ w ݁Ռతʹɺ4PMSʹͳ͍OFTUFE QBSFOUDIJMEEPDVNFOUΛͬͯ ͍ΔͷͰ&4Ͱྑ͔ͬͨ
։ൃڥͰ&4Λ͏ͱ NBQQJOHͳͲόϥόϥͰਏ͘ͳΒͳ͍͔ w #PPUTUSBQεΫϦϓτͱ͍͏ͷΛ༻ҙͯ͠HJUDMPOFޙ ͷηοτΞοϓʢMPDBMʹ&4ΛೖΕΔͳͲʣΛࣗಈԽ͠ ͍ͯΔ w .BQQJOHυΩϡϝϯτͷՃηοτΞοϓεΫϦϓ τΛ༻ҙͯ͠ɺϝϯόʔ͝ͱͷڥͷ͕ࠩग़ͳ͍Α͏ʹ ͍ͯ͠Δ
! #! # Check for Elasticsearch! #! if test !
$(which elasticsearch)! then! echo " x You need to install ElasticSearch:"! echo " + Installing Elasticsearch..."! brew install elasticsearch! else! echo " + ElasticSearch found."! fi! ! #! # Check for Elasticsearch version! #! ES_VERSION=$(elasticsearch -v | cut -d' ' -f2 | cut -d'.' -f1)! if test ! $ES_VERSION -eq 1! then! echo " x You need to upgrade elasticsearch."! brew upgrade elasticsearch! else! echo " + Your Elasticsearch version is correct."! fi! ! if test -d /usr/local/var/lib/elasticsearch/plugins/marvel! then! echo " + Elasticsearch Plugin Marvel found."! else! echo " + Installing Elasticsearch Plugins Marvel"! plugin --install elasticsearch/marvel/latest! fi! bootstrap script Ұ෦ൈਮ $ script/bootstrap! ! Wantedly! ! γΰτͰίίϩΦυϧਓΛ;͢! ! + Your Elasticsearch version is correct.! + ElasticSearch Plugin HQ found.! + ElasticSearch Plugin head found.! + ElasticSearch Plugin Kuromoji found.! + ElasticSearch Plugin Inquisitor found.! + ElasticSearch Plugin Marvel found.! ! Good work. We're ready.! ! Run 'elasticsearch' to start elasticsearch.! And you can see elasticsearch webui plugins 'locahost:9200/_plugin/head/'! ࣮ߦ݁Ռ γϯϓϧͳγΣϧεΫϦϓτɻelasticsearch ϓϥάΠϯ͕Πϯετʔϧ͞ΕͯΔ ͔ΛνΣοΫͯ͠͞Εͯͳ͍ͳΒΠϯετʔϧ͢Δɻ ʢ࣮ࡍͷεΫϦϓτͰ elasticsearch Ҏ֎ͷηοτΞοϓͬͯΔʣ
elasticsearch-ruby Λͬͨ setup script ྫ class ES! def self.client! @client
||= Elasticsearch::Client.new({! url: ES.url,! log: true! })! end! end! ! ES.client.indices.create({! index: "index_name",! body: {! settings: {! analysis: {! filter: {! ...! },! tokenizer: {! ...! },! analyzer: {! ...! }! }! }! }! })! ! ES.client.indices.put_mapping({! index: “index_name”,! type: 'user',! body: {! company: {! _id: { path: 'id' },! _timestamp: { enabled: true, path: 'updated_at', store: true },! _source: { enabled: true },! _all: { enabled: true, analyzer: 'kuromoji_analyzer' },! properties: {! name: { type: 'string', store: true, index: 'analyzed', analyzer: 'kuromoji_analyzer' },! . . .! }! }! }! })! ! ! User.find_in_batches do |users|! body = users.map do |user|! {! index: {! _index: "index_name",! _type: 'user',! _id: user.id,! data: user.elasticsearch_data,! }! }! end! ES.client.bulk body: body! end!
εΩʔϚมߋ࣌ʹαʔϏεʹӨڹΛ༩͑ͳ͍Α͏ɺ Ͳ͏ΫϥελΛߋ৽ͯ͠Δ͔ w #MVF(SFFO%FQMPZNFOUΈ͍ͨͳ͜ͱ ʢ࣮ࡍ৭ʣΛ͍ͬͯΔ
blue-es.wantedly.com blue-es-elb Nginx + ES Nginx + ES blue-es-cluster green-es.wantedly.com
green-es-elb Nginx + ES Nginx + ES green-es-cluster چεΩʔϚ ৽εΩʔϚ
&4ͷ#MVF(SFFO%FQMPZNFOU w CMVFFTFMC HSFFOFTFMCͷΑ͏ʹ&-# αϒυϝΠϯΛͭ༻ҙ͓ͯ͘͠ w $IFGͰlFMBTUJDTFBSDISPMF lHSFFOzFOWJSPONFOUͱͯ͠&4αʔόΛߏ ங͢ΔͱɺΫϥελϦϯάɺ&-# HSFFOFTFMC
ͷొ·ͰࣗಈͰηοτ Ξοϓ͞ΕΔΑ͏ʹͯ͠Δɻ͘Β͍ɻ w ࣾϝϯόʔͷΈΞΫηεՄೳͳڥʹ࠷৽ͷ3BJMTΞϓϦʢ&4ͷ .BQQJOHΛؚΉʣΛσϓϩΠޙɺ͔ͦ͜ΒHSFFOFTDMVTUFSʹ৽εΩʔϚ ͷΠϯσοΫε࡞ɺυΩϡϝϯτՃΛߦ͏ w Ϋϥελʹσʔλ͕ೖͬͨΒɺຊ൪ڥͷ3BJMT࠷৽ʹͨ͠ޙɺ3BJMT͔Β ར༻͢Δ&4ͷΤϯυϙΠϯτʢڥมͰઃఆ͍ͯ͠ΔʣΛΓସ͑ͯσϓ ϩΠྃ
ࠓޙͷ՝
ࠓޙͷ՝ w ύϑΥʔϚϯεͷ্ w ϢʔβʔݕࡧͷϋΠϥΠτ w ࣙॻ͏ w ݕࡧਫ਼
͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ