Elasticsearch at Wantedly

Elasticsearch at

SEIGO UCHIDA

!TQFTOPWB w ೥:BIPP+"1"/৽ଔೖࣾ w ೥݄8BOUFEMZʹδϣΠϯ w ޿ٛͷΠϯϑϥΛ୲౰ w $IFGͱ)VCPUͱ͔͕޷͖

!TQFTOPWB

γΰτͰίίϩΦυϧਓΛ;΍͢

8BOUFEMZ *OD w ೥݄ઃཱ w ΞϧόΠτɺΠϯλʔϯؚΊ໊ w ໊ͷΤϯδχΞ͕׆༂ w γΰτॆ࣮ԽΞϓϦ΍πʔϧͷఏڙ
w ࣾɺਓ͕࢖͏

ੵۃ࠾༻த https://www.wantedly.com/projects/5106

WantedlyͰESΛ ͲͷΑ͏ʹ࢖͍ͬͯΔͷ͔

ݕࡧର৅ w γΰτ w ձࣾ w Ϣʔβ

γΰτ w ձ໊ࣾ w ۈ຿஍ w ࣾһ਺

ืू w ืू໊ w ืूৄࡉ w ืू৬छ

Ϣʔβ w Ϣʔβ໊ w ࣗݾ঺հจ w ͦͷଞ͍Ζ͍Ζʢޙड़ʣ

ΠϯσοΫε w ௨ৗͷݕࡧ༻ΠϯσοΫε w "VUPDPNQMFUF༻ΠϯσοΫε

λΠϓ w γΰτλΠϓ w ձࣾλΠϓ w ϢʔβʔλΠϓ

௨ৗͷݕࡧ༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿ສ w ϓϥΠϚϦαΠζɿ໿(#

"VUPDPNQMFUF༻ΠϯσοΫε w υΩϡϝϯτ਺ɿ໿ສ w ϓϥΠϚϦαΠζɿ໿(#

ߏ੒ w 3BJMTPO)FSPLV w &MBTUJDTFBSDIPO&$ w 1PTUHSFTRMPO3%4

FMBTUJDTFBSDISVCZ w UJSF͸ϝϯς͕ऴྃ w TFBSDILJDL͸UJSFʹґଘ w ެࣜHFNΛ࢖͏ͷ͕༨ܭͳ੍ݶ͕ͳ ͯ͘ྑ͍

Ϋϥελ w &$NTNBMMº୆ w $IFGͰ؅ཧ w +7.ׂ౰.&.ɿ౥ࡌ.&.

ར༻ϓϥάΠϯ w )FBE w )2ʢ/PEF%JBHOPTUJDT΍4UBUT֬ೝʹར༻ʣ w .BSWFM TFOTFͰΫΤϦ࡞੒ʹར༻ʣ ։ൃ؀ڥ ͷΈʣ
w ,VSPNPKJ w *ORVJTJUPSʢ"OBMZ[FSͷڍಈ֬ೝʣ

Ͳ͏΍ֶͬͯΜ͔ͩ

ެࣜϦϑΝϨϯε w ඞཁͳ͜ͱ͸ॻ͍ͯ͋Δɺؾ͕͢Δ w ΫΤϦͷྫ͕෦෼తʹॻ͔Ε͍ͯͯࠔΔ w ݕࡧॳ৺ऀʹ͸શମ૾͕͔ͭΈͮΒ͍ w Ͱ΋ಡΜͩ

&MBTUJDTFBSDI8PSLTIPQ w શମ૾Λ௫Έͭͭɺ࣮ࡍʹ৮ΔͷͰΠ ϝʔδ͕௫Ίͨ w IUUQTTQFBLFSEFDLDPNEBEPPOFUFMBTUJDTFBSDIXPSLTIPQ

&MBTUJDTFBSDIνϡʔτϦΞϧ w ༻ޠɺશମ૾͕௫Έ΍͍͢ w ίϯςΩετ͕͋Γ࣮ફత w IUUQDPEFIBUFOBCMPHDPNFOUSZ

͜ͷຊ͕΋ͬͱૣ͘ग़ͯΕ͹ʜ

ۤ࿑ͨ͠ͱ͜Ζ

Ϣʔβʔݕࡧ w ϓϩϑΟʔϧ߲໨͕ଟ͍ w ֤߲໨ຖʹϓϥΠόγʔϨϕϧ ͕͋Δ w NBQQJOH΋RVFSZ΋ෳࡶʹ

Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ྫ͑͹ɺϓϩϑΟʔϧͷʮ໊લʯʮॴଐʯʮॴ ࡏ஍ʯΛجຊ৘ใͷηοτͱͯ͠ϓϥΠόγʔ ϨϕϧʮҰൠެ։ʯͱ͍ͯͨ͠ͱ͢Δ w
࠷ॳ͸/FTUFEEPDVNFOUΛ࢖ͬͯදݱ͠ ͍ͯͨ

"properties": {! "basic_privacy": {! "type": "nested"! },! "properties": {! "name":
{! "type": "string"! },! "working_for": {! "type": "string"! },! "location": {! "type": "string"! },! "privacy_level": {! "type": "string"! }! }! }! mapping

Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ͔͠͠ɺ/FTUFEͩͱͰ͖ͳ͍͜ͱʢϋΠϥΠτͳͲʣ ͕ग़ͯ͘ΔͷͰϑϥοτͳߏ଄ͰɺϓϥΠόγʔϨϕ ϧΛදݱ w
ܕ@σʔληοτ@@ϓϩύςΟͷΑ͏ʹϚοϐ ϯά w ࠷ॳͷܕ͕ඞཁͳͷ͸ɺEBUF΍JOUFHFSͳͲ TUSJOHҎ֎ʹTJNQMF@RVFSZΛ࢖͏ͱFSSPSͱͳΔͨ Ί

mapping query {! "query": {! "simple_query_string": {! "query": <QUERY>,! "fields":
["s_basic__*"],! "default_operator": "and"! }! }! },! "filter": {! "term": {! "basic_privacy": <PRIVACY_LEVEL>! }! }! }! "properties": {! "basic_privacy": {! "type": "string"! },! "s_basic__name": {! "type": "string"! },! "s_basic__working_for": {! "type": "string"! },! "s_basic__location": {! "type": "string"! }! }!

Ͳ͏΍ͬͯ ES Ͱ ACL Λ࣮ݱͯ͠Δ͔ w ݁ՌతʹɺOFTUFEΛ࢖Θ͟ΔΛಘͳ͍৔໘ ͕ग़ͯདྷͯϑϥοτͳϓϩύςΟͱOFTUFE Λซ༻͍ͯ͠Δ͕ɺࠓճ͸ͦ͜͸লུ

࣭໰λΠϜ

4PMS΍$MPVETFBSDI΋ධՁ্ͨ͠Ͱ &4ʹͨ͠ͷ͔ w ʮ༑ୡͱͷܨ͕ΓʯΛՃຯͯ͠ݕࡧͰ͖Δ͔ͱ͍͏͜ͱͰɺάϥϑ %#ͷ/FPKͱҰൠతͳݕࡧͱͯ͠&4Λൺֱͯ͠ɺ&4ͷํ͕அ વύϑΥʔϚϯε͕ྑ͔ͬͨ w 4PMSʹ͍ͭͯ͸4PMSϢʔβ͔Βࠓ͔Β࢖͏ͳΒ&4ͱ͍͏࿩Λฉ͍ ͍ͯͨ w
ར༻Ϣʔβʢ(JU)VC GPVSTRVBSF 4PVOE$MPVEͳͲʣ΋ࢀߟʹ͠ ͨ w ݁Ռతʹɺ4PMSʹͳ͍OFTUFE QBSFOUDIJMEEPDVNFOUΛ࢖ͬͯ ͍ΔͷͰ&4Ͱྑ͔ͬͨ

։ൃ؀ڥͰ&4Λ࢖͏ͱ NBQQJOHͳͲόϥόϥͰਏ͘ͳΒͳ͍͔ w #PPUTUSBQεΫϦϓτͱ͍͏΋ͷΛ༻ҙͯ͠HJUDMPOFޙ ͷηοτΞοϓʢMPDBMʹ&4ΛೖΕΔͳͲʣΛࣗಈԽ͠ ͍ͯΔ w .BQQJOH΍υΩϡϝϯτͷ௥Ճ΋ηοτΞοϓεΫϦϓ τΛ༻ҙͯ͠ɺϝϯόʔ͝ͱͷ؀ڥͷࠩ෼͕ग़ͳ͍Α͏ʹ ͍ͯ͠Δ

! #! # Check for Elasticsearch! #! if test !
$(which elasticsearch)! then! echo " x You need to install ElasticSearch:"! echo " + Installing Elasticsearch..."! brew install elasticsearch! else! echo " + ElasticSearch found."! fi! ! #! # Check for Elasticsearch version! #! ES_VERSION=$(elasticsearch -v | cut -d' ' -f2 | cut -d'.' -f1)! if test ! $ES_VERSION -eq 1! then! echo " x You need to upgrade elasticsearch."! brew upgrade elasticsearch! else! echo " + Your Elasticsearch version is correct."! fi! ! if test -d /usr/local/var/lib/elasticsearch/plugins/marvel! then! echo " + Elasticsearch Plugin Marvel found."! else! echo " + Installing Elasticsearch Plugins Marvel"! plugin --install elasticsearch/marvel/latest! fi! bootstrap script Ұ෦ൈਮ $ script/bootstrap! ! Wantedly! ! γΰτͰίίϩΦυϧਓΛ;΍͢! ! + Your Elasticsearch version is correct.! + ElasticSearch Plugin HQ found.! + ElasticSearch Plugin head found.! + ElasticSearch Plugin Kuromoji found.! + ElasticSearch Plugin Inquisitor found.! + ElasticSearch Plugin Marvel found.! ! Good work. We're ready.! ! Run 'elasticsearch' to start elasticsearch.! And you can see elasticsearch webui plugins 'locahost:9200/_plugin/head/'! ࣮ߦ݁Ռ γϯϓϧͳγΣϧεΫϦϓτɻelasticsearch ΍ϓϥάΠϯ͕Πϯετʔϧ͞ΕͯΔ ͔ΛνΣοΫͯ͠͞Εͯͳ͍ͳΒΠϯετʔϧ͢Δɻ ʢ࣮ࡍͷεΫϦϓτͰ͸ elasticsearch Ҏ֎ͷηοτΞοϓ΋΍ͬͯΔʣ

elasticsearch-ruby Λ࢖ͬͨ setup script ྫ class ES! def self.client! @client
||= Elasticsearch::Client.new({! url: ES.url,! log: true! })! end! end! ! ES.client.indices.create({! index: "index_name",! body: {! settings: {! analysis: {! filter: {! ...! },! tokenizer: {! ...! },! analyzer: {! ...! }! }! }! }! })! ! ES.client.indices.put_mapping({! index: “index_name”,! type: 'user',! body: {! company: {! _id: { path: 'id' },! _timestamp: { enabled: true, path: 'updated_at', store: true },! _source: { enabled: true },! _all: { enabled: true, analyzer: 'kuromoji_analyzer' },! properties: {! name: { type: 'string', store: true, index: 'analyzed', analyzer: 'kuromoji_analyzer' },! . . .! }! }! }! })! ! ! User.find_in_batches do |users|! body = users.map do |user|! {! index: {! _index: "index_name",! _type: 'user',! _id: user.id,! data: user.elasticsearch_data,! }! }! end! ES.client.bulk body: body! end!

εΩʔϚมߋ࣌ʹαʔϏεʹӨڹΛ༩͑ͳ͍Α͏ɺ Ͳ͏ΫϥελΛߋ৽ͯ͠Δ͔ w #MVF(SFFO%FQMPZNFOUΈ͍ͨͳ͜ͱ ʢ࣮ࡍ͸৭ʣΛ΍͍ͬͯΔ

blue-es.wantedly.com blue-es-elb Nginx + ES Nginx + ES blue-es-cluster green-es.wantedly.com
green-es-elb Nginx + ES Nginx + ES green-es-cluster چεΩʔϚ ৽εΩʔϚ

&4ͷ#MVF(SFFO%FQMPZNFOU w CMVFFTFMC HSFFOFTFMCͷΑ͏ʹ&-# αϒυϝΠϯΛͭ༻ҙ͓ͯ͘͠ w $IFGͰlFMBTUJDTFBSDISPMF lHSFFOzFOWJSPONFOUͱͯ͠&4αʔόΛߏ ங͢ΔͱɺΫϥελϦϯάɺ&-# HSFFOFTFMC
΁ͷొ࿥·ͰࣗಈͰηοτ Ξοϓ͞ΕΔΑ͏ʹͯ͠Δɻ෼͘Β͍ɻ w ࣾ಺ϝϯόʔͷΈΞΫηεՄೳͳ؀ڥʹ࠷৽ͷ3BJMTΞϓϦʢ&4ͷ .BQQJOHΛؚΉʣΛσϓϩΠޙɺ͔ͦ͜ΒHSFFOFTDMVTUFSʹ৽εΩʔϚ ͷΠϯσοΫε࡞੒ɺυΩϡϝϯτ௥ՃΛߦ͏ w Ϋϥελʹσʔλ͕ೖͬͨΒɺຊ൪؀ڥͷ3BJMT΋࠷৽ʹͨ͠ޙɺ3BJMT͔Β ར༻͢Δ&4ͷΤϯυϙΠϯτʢ؀ڥม਺Ͱઃఆ͍ͯ͠ΔʣΛ੾Γସ͑ͯσϓ ϩΠ׬ྃ

ࠓޙͷ՝୊

ࠓޙͷ՝୊ w ύϑΥʔϚϯεͷ޲্ w ϢʔβʔݕࡧͷϋΠϥΠτ w ࣙॻ࢖͏ w ݕࡧਫ਼౓

͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ

Elasticsearch at Wantedly

Elasticsearch at Wantedly

More Decks by Seigo Uchida

Other Decks in Technology

Featured

Transcript