niconicoの検索を支えるElasticsearch / niconico Elasticsearch

niconicoでのElasticsearch運用事例的なお話。
第7回elasticsearch勉強会 #elasticsearch #elasticsearchjp elasticsearch.doorkeeper.jp/events/16837

報告ブログ記事
http://ch.nicovideo.jp/dwango-engineer/blomaga/ar674946

運用事例や構成/設定などの情報を参考に伺いたいので @shoito / fb:shoito に是非お声かけを。

Transcript

1. OJDPOJDPͷݕࡧΛࢧ͑Δ
 &MBTUJDTFBSDI ҏ౻঵
   ג υϫϯΰ 2014/11/18
2. ҏ౻
   ঵
   
   !TIPJUP
   4PGUXBSF
   &OHJOFFS
   %XBOHP
3. "HFOEB ❖౷߹ݕࡧج൫Λ࡞ͬͨཧ༝
   ❖ͳͥ&MBTUJDTFBSDIͳͷ͔ʁ
   ❖Ͳ͏&MBTUJDTFBSDIΛ࢖͍ͬͯΔͷ͔ʁ
   ❖Ͳ͏&MBTUJDTFBSDIΛӡ༻͍ͯ͠Δͷ͔ʁ
   ❖'"2
4. ג υϫϯΰ
   ೥݄ظ
   ௨ظܾࢉࢿྉΑΓ
5. ౷߹ݕࡧج൫Λ࡞ͬͨཧ༝ ⿣ ݕࡧΛ௨ͨ͡OJDPOJDPશαʔϏεͷ17ɺ଺ࡏ࣌ؒͷ޲্
   ⿣ ϦΞϧλΠϜݕࡧ ߴ͍Ԡ౴ੑೳɾߋ৽൓өස౓ Λ࣮ݱ͠ɺ ͞Βʹকདྷతʹ৽͍͠ݕࡧϨίϝϯυٕज़ͳͲ΁ͷରԠ
   ⿣ αʔϏεຖʹߏங͞Ε͍ͯΔݕࡧΠϯϑϥΛҰຊԽ͠ɺ
 ޮ཰Խ ˞ݱ࣌఺Ͱ͸
 ɹ͍͔ͭ͘ͷαʔϏε͕͜ͷج൫Λར༻͍ͯ͠Δͱ͍͏
 ɹεςʔλεͰ͋ΓɺશαʔϏε͕࢖͍ͬͯΔΘ͚Ͱ͸ͳ͍
6. ͳͥ&MBTUJDTFBSDIͳͷ͔ʁ
7. ౷߹ݕࡧج൫ͷੑೳཁ݅ ⿣ χίಈͷϐʔΫλΠϜͷ214ͰϦΫΤετड͚ͭͭɺ
 ݕࡧ݁ՌΛඵҎ಺ʹฦͤΔ͜ͱ
   ⿣ ίϯςϯπ৘ใͷߋ৽͔Β෼Ҏ಺ʹݕࡧ݁Ռʹ
 ൓өͰ͖Δ͜ͱ
   ⿣ OJDPOJDPશαʔϏεͷݕࡧج൫ͱͳΓಘΔ͜ͱ 4DBMF
   0VU ;FSP EPXOUJNF
   NJHSBUJPO 1MVHHBCMF
8. ౷߹ݕࡧج൫ͷੑೳཁ݅ ⿣ χίಈͷϐʔΫλΠϜͷ214ͰϦΫΤετड͚ͭͭɺ
 ݕࡧ݁ՌΛඵҎ಺ʹฦͤΔ͜ͱ
   ⿣ ίϯςϯπ৘ใͷߋ৽͔Β෼Ҏ಺ʹݕࡧ݁Ռʹ
 ൓өͰ͖Δ͜ͱ
   ⿣ OJDPOJDPશαʔϏεͷݕࡧج൫ͱͳΓಘΔ͜ͱ 13+։࢝౰࣌ͷ4PMS
   Ͱ͸
 ཁ݅Λຬͨͤͦ͏ʹͳ͔ͬͨ
9. Ͳ͏&MBTUJDTFBSDIΛ࢖͍ͬͯΔͷ͔ʁ
10. Ͳ͏&MBTUJDTFBSDIΛ࢖͍ͬͯΔͷ͔ʁ ⿣෼ࢄܕશจݕࡧΤϯδϯͱͯ͠
    ⿣ϩάղੳͱՄࢹԽπʔϧͱͯ͠
    
    ,JCBOB
    ⿣αδΣετݕࡧͷࣙॻੜ੒༻όοΫΤϯυͱͯ͠
11. ݕࡧର৅͸Կ͔ʁ ⿣OJDPOJDPͷίϯςϯπ ˞ݱࡏ͸ݕࡧ݁ՌʹදࣔͰ͖Δίϯςϯπ͚ͩ
    ⿣χίχίಈը
    ໿  
    ⿣χίχίੜ์ૹ
    ໿ 
    ⿣χίχί੩ը
    ໿  
    ⿣χίχίνϟϯωϧ
    χίχίχϡʔε
    
    ⿣ ·ͩ·ͩର৅͸૿͑Δ
    ˞ΠϕϯτͷൃੜຖʹΠϯσοΫεߋ৽
12. ΞʔΩςΫνϟ ͍ͩͿϥϑʹ RabbitMQ Cluster MySQL Master/Slave Elasticsearch Cluster Indexer &
 Converter API front MessageReceiver niconico
 services niconico
 services ࢀরܥ ߋ৽ܥ Op module ids query contents on-premises
13. ϊʔυΠϯσοΫεઃܭ ⿣ ϊʔυͷϩʔϧ෼͚ͳ͠ OPEFNBTUFS
    OPEFEBUB
    USVF 
 
 
 
 ⿣ αʔϏε
    ΠϯσοΫε
    
    λΠϓ
    ϨϓϦΧ
    ˞Ϛοϐϯάมߋɺ3FJOEFYͳͲͷϝϯςΛαʔϏεຖʹͰ͖ΔΑ͏ʹ Ϋϥελ ϊʔυ ΠϯσοΫε λΠϓ
14. 
    
    NBQQJOHT
    \
    
    
    
    
    JUFN
    \
    
    
    
    
    
    
     ུ 
    
    
    
    
    
    
    QSPQFSUJFT
    \
    
    
    
    
    
    
    
    
    UJUMF
    \
    
    
    
    
    
    
    
    
    
    
    UZQF
    TUSJOH
    
    
    
    
    
    
    
    
    
    
    JOEFY@BOBMZ[FS
    JOEFYJOH"OBMZ[FS
    
    
    
    
    
    
    
    
    
    
     ུ 
    
    
    
    
    
    
    
    
    ^
    
    
    
    
    
    
    
    
    EFTDSJQUJPO
    \^
    
    
    
    
    
    
    
    
    WJFX@DPVOU
    \
    
    
    
    
    
    
    
    
    
    
    UZQF
    JOUFHFS
    
    
    
    
    
    
    
    
    ^
    
    
    
    
    
    
    
    
    NZMJTU@DPVOU
    \^
    
    
    
    
    
    
    
    
    VQMPBE@UJNF
    \^
    
    
    
    
    
    
    
    
    DPNNVOJUZ@JE
    \^ ϊʔυΠϯσοΫεઃܭ ⿣ Ϛοϐϯά͸ϑϥοτ ✤ χίχίಈը WJEFP
    ✤ UJUMF
    ✤ EFTDSJQUJPO
    ✤ WJFX@DPVOU
    ✤ NZMJTU@DPVOU
    ✤ VQMPBE@UJNF
    ✤ DPNNVOJUZ@JE
    ✤  ✤ χίχίੜ์ૹ MJWF
    ✤ UJUMF
    ✤ EFTDSJQUJPO
    ✤ WJFX@DPVOU
    ✤ DPNNFOU@DPVOU
    ✤ DPNNVOJUZ@MFWFM
    ✤ DPNNVOJUZ@JE
    ✤ 
15. ϊʔυΠϯσοΫεઃܭ ⿣ ΤΠϦΞεΛશΠϯσοΫεͰར༻ WJEFP@ WJEFP WJEFP@ WJEFP WJEFP@ ϝϯςલ ϝϯςޙ WJEFP@ WJEFP WJEFP@ ϝϯςத reindex
 (number_of_replicas͸0ʹɺ
 optimizeޙɺ0͔Β1ʹ͢Δ) Close
16. $6%ϝοηʔδ $SFBUF
    3FBE
    6QEBUF
    %FMFUF
17. Create { "timestamp": 1415850040, "cudtype": "create", "issuer": "publisher", "service_name": "video", "cmsid": "sm9", "values": { "user_id": 1234567, "title": "৽ɾ߽݂ࣉҰ଒ -൥೰ղ์ - Ϩοπΰʔʂӄཅࢣ", "description": “Ϩοπΰʔʂӄཅࢣʢϑϧίʔϥεόʔδϣϯʣ”, "comment_count": 0, “…”, // লུ - શϑΟʔϧυσʔλ͕ૹΒΕΔ "tags": “ӄཅࢣ ެࣜ …” } }
18. Update { "timestamp": 1415850040, "cudtype": "update", "issuer": "publisher", "service_name": "video", "cmsid": "sm9", "values": { “view_counter": 9876 } }
19. Delete { "timestamp": 1415850040, "cudtype": "delete", "issuer": "publisher", "service_name": "video", "cmsid": "sm9", }
20. CUD CUD CUD CUD CUD CUD CUD CUD CUD χίಈ χίੜ etc. MessageReceiver RabbitMQ CUD CUD CUD CUD CUD CUD CUD Elasticsearch MySQL Indexer & Converter αʔϏεڥք MessageReceiver MessageReceiver
21. ϓϥάΠϯ ⿣ ಺੡ϓϥάΠϯ
    ⿣ OJDPOJDP༻ͷಠࣗτʔΫφΠβˍϑΟϧλ
    ⿣ GVODUJPO
    TDPSFͷܭࢉ༻ϩδοΫ
    ɹ˞Πϯετʔϧ๨Ε๷ࢭ
    
    FMBTUJDTFBSDIZNM
 ɹɹ
    QMVHJONBOEBUPSZ
    OJDPBOBMZ[FS
    OJDPTDSJQU
    ⿣ ར༻ϓϥάΠϯ
    ⿣ IFBE
    LPQG
    CJHEFTL
    )2
    ⿣ .BSWFM ݕূ࣌ʹɻຊ൪؀ڥʹ΋ཉ͍͚͠Ͳ
22. Ͳ͏&MBTUJDTFBSDIΛӡ༻͍ͯ͠Δͷ͔ʁ
23. &MBTUJDTFBSDIΫϥελͷ੒௕ 2012 2013 2014 WΫϥελߏங WҠߦ
    2015 WҠߦ
    ϊʔυ਺✖️ WҠߦ
    ϊʔυ਺✖️ WYҠߦ
    ϊʔυ਺✖️ʁ ˞ΫϥελSFTUBSUʹΑΔ
 ɹఀࢭϝϯς ˞3PMMJOH
    SFTUBSUʹΑΔ
 ɹແఀࢭϝϯς ˞ΫϥελίϐʔεΠον
 ɹʹΑΔແఀࢭϝϯς ˞3PMMJOH
    SFTUBSUʹΑΔϝϯς༧ఆ
24. σϓϩΠ؅ཧπʔϧ ⿣ ಺੡$BQJTUSBOP
    8FC
    XSBQQFS
    ⿣ &MBTUJDTFBSDI
    EFQMPZTUBSUTUPQSFTUBSU
    ⿣ ֤छϓϥάΠϯΠϯετʔϧ
    ⿣ 1VU
    NBQQJOH
    ⿣ ΤΠϦΞε੾Γସ͑
    ⿣ ($)FBQ$MBTT
    IJTUPHSBN
    EVNQग़ྗ
    ⿣ ֤छϩά֬ೝ
    ⿣ 
26. ϞχλϦϯά ⿣ $IFDL
    $MVTUFS
    )FBMUI
    CZ
    ;BCCJY
 HSFFOZFMMPXSFE
    ⿣ 4MPX
    -PH *OEFYJOH4FBSDIJOH
    ⿣ ($
    MPH ZPVOHPME
    ⿣ $16.FNPSZ)FBQ
    ⿣ FUD
27. ͜Ε·Ͱૺ۰ͨ͠τϥϒϧ ⿣ 4QSJUCSBJO໰୊ ˞NJOJNVN@NBTUFS@OPEFTઃఆࡁΈ
    ⿣ ϚελϊʔυͰͷ'VMM
    ($ʹΑΔQJOH
    UJNFPVU
    ⿣ ϚελϊʔυͷSFTUBSUͱγϟʔυͷSFSPVUFͰճ෮ 01 02 03 04 05 06 01 03 04 05 06 02 03 04 05 06 Ϋϥελ ϊʔυ Ϛελʔϊʔυ cluster_v1 cluster_v1 cluster_v1 ϊʔυ01͔ΒݟͨΫϥελ ϊʔυ02͔ΒݟͨΫϥελ ਖ਼ৗঢ়ଶͷΫϥελ
28. ͜Ε·Ͱૺ۰ͨ͠τϥϒϧ ⿣ ૝ఆ֎ͷΠϯσοΫε
    
    EZOBNJD
    DSFBUJPO
    PG
    NBQQJOHT
    ⿣ ஌Β͵ؒʹҙਤ͍ͯ͠ͳ͍ΠϯσοΫε͕࡞ΒΕͯ͠·͍ͬͯͨ
    ⿣ BDUJPOBVUP@DSFBUF@JOEFY
    GBMTF
    ⿣ JOEFYNBQQFSEZOBNJD
    GBMTF
    ⿣ ૝ఆ֎ͷϑΟʔϧυ
    
    EZOBNJD
    DSFBUJPO
    PG
    pFMET
    ⿣ ϑΟʔϧυ௥Ճ͸Ͱ͖Δ͚Ͳফͤͳ͍
    ⿣ ҙਤ͍ͯ͠ͳ͍ϑΟʔϧυΛؚΉ+40/Λ์ΓࠐΜͩ͜ͱ͕ݪҼ
    ⿣ EZOBNJD
    TUSJDU
    ⿣ JOEFYRVFSZQBSTFBMMPX@VONBQQFE@pFMET
    USVF
    
    GBMTF
     ʹ͠ͱ͘ͱ ͞Βʹྑ͍͔΋
29. ͜Ε·Ͱૺ۰ͨ͠τϥϒϧ ⿣ 0VU0G.FNPSZ
    FYDFQUJPO
    ⿣ େ͖ΊͷΠϯσοΫεͷϨϓϦΧγϟʔυͷ͕ͭ*OJUJBMJ[JOHͷ··ఀࢭ
    ⿣ ࢭΉແ͘OVNCFS@PG@SFQMJDBΛˠˠʹͨ͠ޙɺൃੜ
    ⿣ ϊʔυͷSFTUBSUͰ෮ؼ
    ⿣ ң͕௧͘ͳΔτϥϒϧ͸ଞʹ΋
30. ෛՙܭଌνϡʔχϯά ⿣ ෛՙܭଌπʔϧ
    (BUMJOH
    +.FUFS
 IUUQTHJUIVCDPNFMBTUJDTFBSDIFMBTUJDTFBSDIUSFFNBTUFSTSDUFTUSFTPVSDFTKNFUFS
    ⿣ νϡʔχϯά ୅දతͳ΋ͷ
    ⿣ Ϋϥελ
    ⿣ EJTDPWFSZ[FONJOJNVN@NBTUFS@OPEFT
    ⿣ EJTDPWFSZ[FOGEQJOH@JOUFSWBM
    UJNFPVU
    SFUSJFT
    ⿣ ΠϯσΫγϯά
    ⿣ SFGSFTI@JOUFSWBM
    TT
    ⿣ @BMM
    
    \FOBCMFE
    
    GBMTF^
    ⿣ ݕࡧ
    ⿣
    @SPVUJOH
    \QBUI
    HSPVQ@JE^
    ⿣ "HHSFHBUJPO༻ʹCVDLFUT୯Ґʹγϟʔυ͕෼͔ΕΔΑ͏ϧʔςΟϯά
31. 5IBOLT
    2VFUJPOT
32. ྑ͘ݟΔ৘ใݯ ⿣ &MBTUJDTFBSDI
    #MPH
 IUUQXXXFMBTUJDTFBSDIPSHCMPH
    ⿣ &MBTUJDTFBSDI
    
    UIF
    EFpOJUJWF
    HVJEF
 IUUQXXXFMBTUJDTFBSDIPSHHVJEFFOFMBTUJDTFBSDIHVJEFDVSSFOUJOEFYIUNM
    ⿣ 'PVOE
 IUUQTXXXGPVOEOPEPDVNFOUBUJPO
    IUUQTXXXGPVOEOPGPVOEBUJPO
    ⿣ !KPIUBOJͷ೔ه
    OE
 IUUQCMPHKPIUBOJJOGP
    ⿣ )FMMP
    &MBTUJDTFBSDI
 IUUQTNFEJVNDPNIFMMPFMBTUJDTFBSDI
    ⿣ -VDFOF
    OJHIUMZ
    CFODINBSLT
 IUUQQFPQMFBQBDIFPSHdNJLFNDDBOEMVDFOFCFODI
33. ఏ
    ڙ גࣜձࣾυϫϯΰ υϫϯΰ͸ຊ෺ͷ ݕࡧcσʔλϚΠχϯάc8FCΞϓϦcεϚϗΞϓϦ 
 ΤϯδχΞΛืू͍ͯ͠·͢ɻ