Elastic Stack 6.5で実現する快適Kibana生活 / Comfortable Kibana Life by Using Elastic Stack 6.5

第28回 Elasticsearch勉強会の発表資料です。

勉強会の詳細はこちら
https://www.meetup.com/ja-JP/Tokyo-Elastic-Fantastics/events/257301177/

Transcript

1. &MBTUJD
   4UBDL
   Ͱ
   ࣮ݱ͢Δ
   շద,JCBOBੜ׆ @j__yama 2019/01/16 ୈ28ճElasticsearchษڧձ
2. whoami • ࢁԼ
   ৓࢘@NTTςΫϊΫϩε • Twitter: @j__yama • Favorite: Listening HR/HM , Watching Speedrun • Ҿ༻ݩɿIUUQTRJJUBDPNTFBSDI RKZBNB 35"J+
   
   "(%2
   
3. IUUQTFMBTUJDMPWFSIBUFOBCMPHKQ
4. ࠓ೔ͷςʔϚ͸
5. &MBTUJD
   4UBDL
   
   ৽ػೳ঺հ
6. ͜ͷൃද͸,JCBOB
   ͷ
   ৽ػೳΛ঺հ͠ɺ
   ͦͷັྗΛօ͞Μʹ఻͑Δ΋ͷͰ͢ʂ
7. agenda ɾKibana 6.5ͷ৽ػೳ
   ɾಛʹ঺հ͍ͨͭ͠ͷػೳ
   
   ɾ'JMF
   %BUB
   7JTVBMJ[FS
   
   ɾ4QBDFT
   ɾ͓ΘΓʹ
8. ,JCBOB
   ͷ৽ػೳ
9. Kibana 6.5ͷओͳ৽ػೳ(1/2) • ੝Γͩ͘͞Μ 4QBDFT $BOWBTͷϏϧτΠϯ 'JMF
   %BUB
   7JTVBMJ[FS *OGSB
   6*
   
   -PH
   6* #FUB #FUB ৽͍͠αϯϓϧσ ʔλ΋ͭ௥Ճ #FUB
10. Kibana 6.5ͷओͳ৽ػೳ(2/2) • ੝Γͩ͘͞Μ 3PMMVQͷ؅ཧ6* 3PMMVQ݁ՌͷՄࢹԽ %FW
    5PPMTͷػೳڧԽ 4QZύωϧ͔Β*OTQFDUPS΁ͷҠߦ ͱͯ΋શ෦͸࿩ͤͳ͍ #FUB
11. ಛʹ঺հ͍ͨͭ͠ͷػೳ
12. ಛʹ঺հ͍ͨ͠2ͭͷػೳ • File Data Visualizer • ͜Ε͔ΒElastic StackΛ࢖͍࢝ΊΔਓ޲͚ • ௒؆୯ʹElasticsearchʹσʔλ౤ೖͰ͖Δػೳ • Spaces • طʹ։ൃɾӡ༻ʹ࢖͍ͬͯΔਓ޲͚ • Visualize΍Dashboard͕ཚཱ͍ͯͯ͠؅ཧʹࠔ͍ͬͯΔਓ ΁ͷղܾࡦ
13. 'JMF
    %BUB
    7JTVBMJ[FS ʲ͜Ε͔Β&MBTUJD
    4UBDLΛ࢖͍࢝ΊΔਓ޲͚ʳ
14. File Data Visualizer • ςΩετϑΝΠϧΛKibanaը໘͔Β௚઀ Elasticsearchʹ౤ೖͰ͖Δػೳ • ରԠσʔλܗࣜ: csv, ndjson, ஶ໊ͳϩάܗࣜ • ࣗಈͰύʔεɺϚοϐϯάੜ੒΋࣮ߦ • σϑΥϧτͰ͸࠷ॳͷ1000ߦΛ෼ੳͯࣝ͠ผ
15. Before • ElasticsearchʹσʔλΛ౤ೖ͢ΔͨΊʹ͸
 ԿΒ͔ͷETLπʔϧ΍ϥΠϒϥϦΛ࢖͏ඞཁ͕ ͋ͬͨ ίϯϑΟάͷॻ͖ํͳͲ शख़͕ඞཁͰίετେ
16. After • KibanaΛද͍ࣔͯ͠Δϒϥ΢β͔Β௚઀ ElasticsearchʹσʔλΛ౤ೖͰ͖Δ ࢼ͠ʹ࢖ͬͯΈ͍ͨ࣌ʹ ͱͯ΋ศརʂ
17. Kibanaੜ׆ͷBefore / After • Before • After KibanaͰϩάͱ͔ՄࢹԽ͠ ·͠ΐ͏ʂ Ͱ͖ͳ͍ͷͰLogstash΍ Filebeatͷ࢖͍ํΛ֮͑· ͠ΐ͏ csvͷσʔλͱ͔؆୯ʹը໘ ͔Β౤ೖͰ͖·͔͢ʁ ͑ͬ...େมͦ͏...࢖͏ͷ΍Ί Α͏͔ͳ… Ͱ͖·͢ΑʂKibanaͷը໘ ͔Β͜͏΍ͬͯબ୒͍ͯ͠ ͘ͱ... ͍͍͡ΌΜʂ࢖ͬͯΈΔΑ
18. File Data Visualizerͷ࢖͍ํ 
    .BDIJOF
    -FBSOJOHΛΫϦοΫ 
    %BUB
    7JTVBMJ[FSΛΫϦοΫ 
    ը໘தԝʹ౤ೖ͍ͨ͠ϑΝΠϧΛ υϥοάΞϯυυϩοϓ͢Δ
19. File Data Visualizerͷ࢖͍ํ 
    σʔλ܏޲͕දࣔ͞ΕΔ ͷͰɺ֬ೝͨ͠Β
    JNQPSUΛΫϦοΫ͢Δ
20. File Data Visualizerͷ࢖͍ํ 
    JOEFY
    OBNFʹઃఆ͍ͨ͠
    ΠϯσοΫε໊Λೖྗ͢Δ 
    JNQPSUΛΫϦοΫ͢Δ
21. File Data Visualizerͷ࢖͍ํ ଴͍ͬͯΔͱϓϩάϨεόʔ͕ਐΈɺ
    σʔλ͕౤ೖ͞ΕΔ
22. File Data Visualizerͷ࢖͍ํ ౤ೖ࣌ʹJOEFY
    QBUUFSO͕࡞੒͞ΕΔͨΊɺ
    ͦͷ··%JTDPWFS΍7JTVBMJ[FͰදࣔͰ͖Δ
23. File Data Visualizerͷ࢖͍ํ JNQPSU
    EBUBͰ"EWBODFEʹ͢Δͱɺ ز͔ͭͷઃఆΛมߋͰ͖Δ ࢀߟ ΠϯσοΫε ઃఆ Ϛοϐϯάͷ ΧελϚΠζ σʔλՃ޻ઃఆ JOHFTU
    QJQFMJOF
24. σϞ
25. File Data VisualizerͰࢥ͏͜ͱ • Q. σʔλͷՃ޻Λ͢ΔͷʹIngest PipelineΛ࢖Θ ͳ͍ͱ͍͚ͳ͍ͷ͸େมͳͷͰ͸ʁ • A. ͦ΋ͦ΋౤ೖલʹExcelͳͲͰcsvΛՃ޻͓ͯ͠ ͚͹໰୊͋Γ·ͤΜ &MBTUJD͞Μͷ#MPHͰ΋ॳ৺ऀ޲͚ͷهࣄͰ͸ࣄલʹ &YDFMͳͲͰσʔλՃ޻͢ΔखॱͰ঺հ͞Ε͍ͯ·͢ɻ
    IUUQTXXXFMBTUJDDPCMPHBGUFSTIPDLUIFSBQZ XJUIFMBTUJDTFBSDIBOEDTWEBUBJNQPSU
26. 4QBDFT ʲطʹ։ൃɾӡ༻ʹ࢖͍ͬͯΔਓ޲͚ʳ
27. Spaces • Visualize/DashboardΛ੔ཧ੔಴Ͱ͖Δػೳ • εϖʔεͱݺ͹ΕΔྖҬΛ࡞੒͠ɺͦͷதͰVisualize΍ DashboardΛ࡞੒͢ΔͱɺͦͷεϖʔεʹډΔͱ͖͚ͩ ӾཡͰ͖ΔΑ͏ʹͳΔ ηΩϡϦςΟ ϚʔέςΟϯά ʓ º
28. Spaces • Ϣʔβʔ͸Ӿཡ࣌ʹͲͷεϖʔεΛར༻ͨ͠ ͍͔બ୒͢Δ ؂ࠪϩάΛݟ͍ͨ࣌͸
    ηΩϡϦςΟ༻ͷεϖʔε ച্৘ใΛݟ͍ͨ࣌͸
    ϚʔέςΟϯά༻ͷεϖʔε
29. Before • Visualize΍DashboardͷҰཡΛදࣔ͢Δͱଘ ࡏ͢Δશ͕ͯදࣔ͞Εͯ͠·͏ • ϫʔΫΞϥ΢ϯυͱͯ͠༻్ผʹVisualize ໊΁prefixΛ෇͚ͨΓ͍͕ͯͨ͠… • ී௨ʹԚ͍͠ɺҰཡը໘ͷݕࡧػೳ͸บ ͕͋ͬͯɺҾ͔͔ͬΒͳ͍͜ͱ΋͋Δ
30. Beforeͷྫ #FBUTͷαϯϓϧμογϡϘʔυ Λ౤ೖ͢Δͱಋೖ࣌఺͔Βʜ
31. Beforeͷྫ ϚΠφεݕࡧͰ.FUSJDCFBUͷ 7JTVBMJ[FΛඇදࣔʹ͠Α͏ͱͨ͠΋ͷͷɺҰ෦ 7JTVBMJ[Fͷ5JUMFʹ.FUSJDCFBUΛද͢Ωʔϫʔυ͕ ೖ͍ͬͯͳ͍ͨΊදࣔ͞Εͯ͠·͏ʜ
32. Beforeͷྫ ͜Ε·Ͱ͸۠ผΛ͚ͭΔͨΊʹ<1SPKFDU">ͳͲ ͷ1SFpYΛ෇͚ͯରԠ͢Δͷ͕ਫ਼Ұഋ ݕࡧ͢Ε͹Ҿ͔͚ͬΒΕΔ͕ɺ
    5JUMFΛ෇͚Δͱ͖ʹҙࣝ͢Δඞཁ͕͋Γɺ
    ίετ͕ߴ͍ ݕࡧ
33. After • ࣄલʹద੾ʹ෼ྨ͓͚ͯ͠͹ɺͦͷ࣌ʑͰڵຯ ͋ΔVisualize/DashboardͷΈΛӾཡͰ͖Δ • SpaceΛ࡞੒͓͚ͯ͠͹ɺଞͷਓͷҰཡΛԚ͢ ͜ͱͳ͘ɺςετDashboardΛ࡞੒Ͱ͖Δ • SecurityΛ૊Έ߹ΘͤΕ͹ಛఆͷਓ͚͕ͩӾཡՄ ೳͳDashboard΋࡞੒Մೳ
34. Kibanaੜ׆ͷBefore / After • Before • After ,JCBOB ,JCBOB ։ൃɾӡ༻ ηΩϡϦςΟ ϚʔέςΟϯά ؂ࠪϩά ؂ࠪϩά ΞϓϦϩά Ϣʔβʔ࢖༻܏޲ ϝτϦοΫ ϝτϦοΫ ΞϓϦϩά ύέοτ৘ใ ύέοτ৘ใ Ϣʔβʔ࢖༻܏޲ ച্ ച্
35. σϞ
36. Spacesͷ׆༻Ҋͦͷ1 νʔϜ΍ۀ຿͝ͱʹ෼͚Δ • Elastic͞Μͷઆ໌Ͱ΋Α͘࢖ΘΕΔ • ଞνʔϜͷڵຯͳ͍Visualize/DashboardΛඇදࣔʹͰ͖Δ ։ൃɾӡ༻ ηΩϡϦςΟ ϚʔέςΟϯά ӡ༻ ϝτϦοΫ ΞϓϦϩά ؂ࠪϩά ύέοτ৘ใ Ϣʔβʔ࢖༻܏޲ ച্
37. Spacesͷ׆༻Ҋͦͷ2 ςφϯτ͝ͱʹ෼͚Δ • ͓٬༷͝ͱʹҟͳΔDashboardΛఏڙͰ͖Δ • ࣮͸͜Ε·ͰετϨʔτͳํ๏Ͱ͸࣮ݱͰ͖ͳ͔ͬͨ • ςφϯτ͝ͱʹKibanaΠϯελϯεΛݐͯͯݸผ ͷ".kibana"Λઃఆͯ͠…ͳͲͷϫʔΫΞϥ΢ϯυ͕ඞཁͩͬͨ ͓٬༷" ͓٬༷# ͓٬༷$ ӡ༻ #༷ࣾݶఆ "༷ࣾݶఆ $༷ࣾݶఆ
38. Spacesͷ׆༻Ҋͦͷ3 Dashboardผʹ෼͚Δ • ։ൃத͸ಛʹVisualizeҰཡը໘͸ཚࡶʹͳΓ͕ͪͳͷΛ๷͛Δ • Space͕ཚཱ͢Δͷ͕एׯ໰୊ʁ • Dashboardͷ݅਺͕গͳ͚Ε͹͋Γ ΞϓϦϩά ϛυϧϩά ϝτϦοΫ ։ൃ "QBDIF 5PNDBU ΞϓϦ" ΞϓϦ# $16 ϝϞϦ
39. Spacesͷ׆༻Ҋͦͷ4 ։ൃऀผʹ෼͚Δ • ։ൃத͸ಛʹVisualizeҰཡը໘͸ཚࡶʹͳΓ͕ͪͳͷΛ๷͛Δ • ଞͷਓ΁ͷӨڹΛߟ͑ͣؾָʹVisualize/DashboardΛ࡞੒Ͱ͖Δ • ਓ਺͕গͳ͚Ε͹Space΋ཚཱ͠ͳ͍ "͞Μ #͞Μ $͞Μ ։ൃ ؂ࠪ ੑೳ ͓ࢼ͠ ύέοτ ςετ
40. Spacesͷ׆༻Ҋͦͷ5 DashboardͷϨϏϡΛKibanaͰߦ͏ • ςετతʹࢼ͍ͨ͠৔߹ʹ͸αϯυϘοΫεͰ࡞੒͢Δ • ϫʔΫεϖʔεͰ࡞੒͠ɺϨϏϡر๬ͷDashboard͸ϨϏϡ΁Ҡಈ • ϨϏϡ׬ྃͨ͠ΒϓϩμΫγϣϯ΁Ҡಈ αϯυϘοΫε ϫʔΫεϖʔε ϨϏϡ ։ൃ ϓϩμΫγϣϯ ࡞੒த ϨϏϡ଴ͪ ϨϏϡࡁΈ ϨϏϡࡁΈ ͓ࢼ͠
41. SpacesͰࢥ͏͜ͱ • ͋·Γʹஅઈ͠ա͗Δͱྑ͘ͳ͍͔΋ • ӡ༻ɺ։ൃɺϚʔέςΟϯάͷਓͨͪڞ௨ Ͱݟ͍ͨDashboard΋͋Δ • ·ͨɺผͷSpaceͷVisualize͸ࢀরͰ͖ͳ ͘ͳͬͯ͠·͏ͷͰ࠶ར༻ੑ΋Լ͕Δ
42. ͓ΘΓʹ
43. ͜Ε͔ΒͷշదͳKibanaੜ׆ΛΠ ϝʔδͰ͖·͔ͨ͠ʁ • File Data Visualizer • ͜Ε͔ΒElastic StackΛ࢖͍࢝ΊΔਓ͸௒؆୯ʹσʔλ౤ೖ Ͱ͖ΔΑ͏ʹͳΔʂ • ETLπʔϧͷ࢖͍ํ͸ࣗಈԽͨ͘͠ͳ͔ͬͯΒ֮͑Ε͹OK • Spaces • طʹ։ൃ/ӡ༻Ͱ࢖͍ͬͯΔਓ͸Visualize/DashboardͷҰཡ Λ੔ཧ੔಴Ͱ͖ΔΑ͏ʹͳΔʂ
44. 'JMF
    %BUB
    7JTVBMJ[FS͸ &YQFSJNFOUBMͳػೳͳͨΊɺࠓޙ ࢓༷มߋ͞ΕΔՄೳੑ͕͋Γ·͢  ·ͨɺόάΛଟؚ͘Ή͜ͱ͕༧૝͞ΕΔͨΊɺ
    ݱ࣌఺ͰϓϩμΫγϣϯͰ࢖༻͢Δ͜ͱ͸ΦεεϝͰ͖·ͤΜɻ όάΛൃݟͨ͠Β*TTVF13
45. &MBTUJD
    4UBDL
    Ͱ
    ࣮ݱ͢Δ
    շద,JCBOBੜ׆ @j__yama 2019/01/16 ୈ28ճElasticsearchษڧձ
46. ͓͠·͍
47. "QQFOEJY
48. File Data Visualizerͷ ରԠϑΝΠϧܗࣜ • ٙ໰ ճ౴ ΤϯίʔσΟϯά͸ԿʹରԠʁ UTF-8ɺBOM༗ແͷରԠ͸ʁ վߦίʔυ͸LFɺCRLFͷͲͪΒʹ΋ରԠʁ $47͸೚ҙͷ۠੾ΓจࣈʹରԠʁ ೔෇ͷରԠϑΥʔϚοτ͸ʁ ରԠ͍ͯ͠Δஶ໊ͳϩάܗࣜ͸ʁ όΠφϦϑΝΠϧʹ͸ରԠʁ
49. File Data Visualizerͷίʔυ • ΫϥΠΞϯτ(Kibana)ଆͷίʔυ • https://github.com/elastic/kibana/blob/master/x- pack/plugins/ml/server/routes/file_data_visualizer.js • αʔόʔ(Elasticsearch)ଆͷίʔυ • https://github.com/elastic/elasticsearch/tree/ master/x-pack/plugin/ml/src/main/java/org/ elasticsearch/xpack/ml/filestructurefinder
50. File Data VisualizerͷରԠ͍ͯ͠ ΔΤϯίʔσΟϯά • https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/ml/src/main/ java/org/elasticsearch/xpack/ml/filestructurefinder/FileStructureFinderManager.java
51. File Data VisualizerͷରԠ͍ͯ͠ Δ೔෇ϑΥʔϚοτ • https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/ml/src/main/ java/org/elasticsearch/xpack/ml/filestructurefinder/TimestampFormatFinder.java
52. File Data VisualizerͷରԠ͍ͯ͠ Δஶ໊ͳϩάܗࣜ • https://github.com/elastic/elasticsearch/blob/master/x-pack/plugin/ml/src/main/ java/org/elasticsearch/xpack/ml/filestructurefinder/GrokPatternCreator.java IUUQTHJUIVCDPN MPHTUBTIQMVHJOTMPHTUBTIQBUUFSOT DPSFUSFFNBTUFSQBUUFSOT
    ʹఆٛ͞Ε͍ͯΔ(SPL
    1BUUFSO໊Ͱࢀর͞Εͯ ͍Δ
53. File Data Visualizerͷ ରԠϑΝΠϧܗࣜ • ٙ໰ ճ౴ ΤϯίʔσΟϯά͸ԿʹରԠʁ 4+*4
    &6$+1
    65'ͳͲ UTF-8ɺBOM༗ແͷରԠ͸ʁ ʓ վߦίʔυ͸LFɺCRLFͷͲͪΒʹ΋ରԠʁ ʓ $47͸೚ҙͷ۠੾ΓจࣈʹରԠʁ ʓ ೔෇ͷରԠϑΥʔϚοτ͸ʁ *40ͳͲछྨ ରԠ͍ͯ͠Δஶ໊ͳϩάܗࣜ͸ʁ "QBDIF
    5PNDBUͳͲछྨ όΠφϦϑΝΠϧʹ͸ରԠʁ º
54. SpaceؒͰVisualize/Dashboard ΛҠಈͤ͞Δํ๏ • ݱঢ়͸ManagementͷSaved Object͔Β Export/Import͢Δ͔͠ͳͦ͞͏ • ҠಈݩͷSaved Object͔ΒVisualize/ DashboardΛJSONʹExport • ҠಈઌͷSaved Object͔ΒJSONΛImport
55. Dev Toolsͷػೳ֦ு • Grok DebuggerʹSyntax Highlight௥Ճ
56. Dev Toolsͷػೳ֦ு • ConsoleͰCtrl/Cmd + /͢Δͱɺ࣮ߦ͠Α͏ͱ͍ͯ͠ΔAPIͷυΩ ϡϝϯτ͕৽͍͠΢Οϯυ΢Ͱ։͘Α͏ʹͳͬͨ