Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic Stack 6.5で実現する快適Kibana生活 / Comfortable...

Avatar for j-yama j-yama
January 16, 2019

Elastic Stack 6.5で実現する快適Kibana生活 / Comfortable Kibana Life by Using Elastic Stack 6.5

第28回 Elasticsearch勉強会の発表資料です。

勉強会の詳細はこちら
https://www.meetup.com/ja-JP/Tokyo-Elastic-Fantastics/events/257301177/

Avatar for j-yama

j-yama

January 16, 2019
Tweet

More Decks by j-yama

Other Decks in Technology

Transcript

  1. whoami • ࢁԼ৓࢘@NTTςΫϊΫϩε • Twitter: @j__yama • Favorite: Listening HR/HM

    , Watching Speedrun • Ҿ༻ݩɿIUUQTRJJUBDPNTFBSDI RKZBNB 35"J+ "(%2
  2. ಛʹ঺հ͍ͨ͠2ͭͷػೳ • File Data Visualizer • ͜Ε͔ΒElastic StackΛ࢖͍࢝ΊΔਓ޲͚ • ௒؆୯ʹElasticsearchʹσʔλ౤ೖͰ͖Δػೳ

    • Spaces • طʹ։ൃɾӡ༻ʹ࢖͍ͬͯΔਓ޲͚ • Visualize΍Dashboard͕ཚཱ͍ͯͯ͠؅ཧʹࠔ͍ͬͯΔਓ ΁ͷղܾࡦ
  3. File Data Visualizer • ςΩετϑΝΠϧΛKibanaը໘͔Β௚઀ Elasticsearchʹ౤ೖͰ͖Δػೳ • ରԠσʔλܗࣜ: csv, ndjson,

    ஶ໊ͳϩάܗࣜ • ࣗಈͰύʔεɺϚοϐϯάੜ੒΋࣮ߦ • σϑΥϧτͰ͸࠷ॳͷ1000ߦΛ෼ੳͯࣝ͠ผ
  4. Kibanaੜ׆ͷBefore / After • Before • After KibanaͰϩάͱ͔ՄࢹԽ͠ ·͠ΐ͏ʂ Ͱ͖ͳ͍ͷͰLogstash΍

    Filebeatͷ࢖͍ํΛ֮͑· ͠ΐ͏ csvͷσʔλͱ͔؆୯ʹը໘ ͔Β౤ೖͰ͖·͔͢ʁ ͑ͬ...େมͦ͏...࢖͏ͷ΍Ί Α͏͔ͳ… Ͱ͖·͢ΑʂKibanaͷը໘ ͔Β͜͏΍ͬͯબ୒͍ͯ͠ ͘ͱ... ͍͍͡ΌΜʂ࢖ͬͯΈΔΑ
  5. File Data VisualizerͰࢥ͏͜ͱ • Q. σʔλͷՃ޻Λ͢ΔͷʹIngest PipelineΛ࢖Θ ͳ͍ͱ͍͚ͳ͍ͷ͸େมͳͷͰ͸ʁ • A.

    ͦ΋ͦ΋౤ೖલʹExcelͳͲͰcsvΛՃ޻͓ͯ͠ ͚͹໰୊͋Γ·ͤΜ &MBTUJD͞Μͷ#MPHͰ΋ॳ৺ऀ޲͚ͷهࣄͰ͸ࣄલʹ &YDFMͳͲͰσʔλՃ޻͢ΔखॱͰ঺հ͞Ε͍ͯ·͢ɻ IUUQTXXXFMBTUJDDPCMPHBGUFSTIPDLUIFSBQZ XJUIFMBTUJDTFBSDIBOEDTWEBUBJNQPSU
  6. Kibanaੜ׆ͷBefore / After • Before • After ,JCBOB ,JCBOB ։ൃɾӡ༻

    ηΩϡϦςΟ ϚʔέςΟϯά ؂ࠪϩά ؂ࠪϩά ΞϓϦϩά Ϣʔβʔ࢖༻܏޲ ϝτϦοΫ ϝτϦοΫ ΞϓϦϩά ύέοτ৘ใ ύέοτ৘ใ Ϣʔβʔ࢖༻܏޲ ച্ ച্
  7. ͜Ε͔ΒͷշదͳKibanaੜ׆ΛΠ ϝʔδͰ͖·͔ͨ͠ʁ • File Data Visualizer • ͜Ε͔ΒElastic StackΛ࢖͍࢝ΊΔਓ͸௒؆୯ʹσʔλ౤ೖ Ͱ͖ΔΑ͏ʹͳΔʂ

    • ETLπʔϧͷ࢖͍ํ͸ࣗಈԽͨ͘͠ͳ͔ͬͯΒ֮͑Ε͹OK • Spaces • طʹ։ൃ/ӡ༻Ͱ࢖͍ͬͯΔਓ͸Visualize/DashboardͷҰཡ Λ੔ཧ੔಴Ͱ͖ΔΑ͏ʹͳΔʂ
  8. File Data Visualizerͷ ରԠϑΝΠϧܗࣜ • ٙ໰ ճ౴ ΤϯίʔσΟϯά͸ԿʹରԠʁ UTF-8ɺBOM༗ແͷରԠ͸ʁ վߦίʔυ͸LFɺCRLFͷͲͪΒʹ΋ରԠʁ

    $47͸೚ҙͷ۠੾ΓจࣈʹରԠʁ ೔෇ͷରԠϑΥʔϚοτ͸ʁ ରԠ͍ͯ͠Δஶ໊ͳϩάܗࣜ͸ʁ όΠφϦϑΝΠϧʹ͸ରԠʁ
  9. File Data Visualizerͷίʔυ • ΫϥΠΞϯτ(Kibana)ଆͷίʔυ • https://github.com/elastic/kibana/blob/master/x- pack/plugins/ml/server/routes/file_data_visualizer.js • αʔόʔ(Elasticsearch)ଆͷίʔυ

    • https://github.com/elastic/elasticsearch/tree/ master/x-pack/plugin/ml/src/main/java/org/ elasticsearch/xpack/ml/filestructurefinder
  10. File Data Visualizerͷ ରԠϑΝΠϧܗࣜ • ٙ໰ ճ౴ ΤϯίʔσΟϯά͸ԿʹରԠʁ 4+*4 &6$+1

    65'ͳͲ UTF-8ɺBOM༗ແͷରԠ͸ʁ ʓ վߦίʔυ͸LFɺCRLFͷͲͪΒʹ΋ରԠʁ ʓ $47͸೚ҙͷ۠੾ΓจࣈʹରԠʁ ʓ ೔෇ͷରԠϑΥʔϚοτ͸ʁ *40ͳͲछྨ ରԠ͍ͯ͠Δஶ໊ͳϩάܗࣜ͸ʁ "QBDIF 5PNDBUͳͲछྨ όΠφϦϑΝΠϧʹ͸ରԠʁ º