What's new in Elastic Stack 6.3

Jun Ohtani, Community Engineer  Kosho Owa, Principal Solution Architect Elastic
Stack 6.3 ঺հ

!2 ηογϣϯʹೖΔ  ͦͷલʹɻ  QRίʔυϦʔμʔͷ͝༻ҙΛʂ

!3 bit.ly/EnqEsMeetup

!10 Elasticsearch SQL (6.3 - Experimental) ࢖͍׳ΕͨγϯλοΫεͰElasticsearchʹ໰͍߹Θͤ

!11 Elasticsearch SQL (6.3) ࢖͍׳ΕͨγϯλοΫεͰElasticsearchʹ໰͍߹Θͤ • REST API & CLI
Client (Basic License) ◦ SQLγϯλοΫεͰElasticsearchͱ΍ΓͱΓ ◦ දܗࣜͰ݁ՌΛදࣔ • Translate API (Basic License) ◦ SQLίϚϯυΛElasticsearchͷQuery DSLʹม׵ ◦ ElasticsearchͷQuery DSLγϯλοΫεʹ׳ΕΔͨΊͷศརͳํ๏ • JDBC Client (Platinum License)

!12 Data Rollups • API for creating an Elasticsearch process
to periodically store aggregate statistics • Primary benefit is space savings ◦ Faster queries ◦ Potentially less nodes to manage ◦ Smaller snapshots ◦ Longer retention times ◦ etc. • Query rolled up data and “live” data together in a single query. Rollups API (6.3 - Experimental) • ఆظతʹ౷ܭσʔλΛू໿ͯ͠อଘ͢ΔElasticsearchͷJobΛొ࿥ • ओͳར఺͸༰ྔͷ࡟ݮ • σʔλ͕গͳ͘ͳΔͨΊ ◦ Query͕ΑΓߴ଎ʹ ◦ গͳ͍ϊʔυͰσʔλΛ؅ཧ ◦ Snapshot͕ΑΓখ͘͞ ◦ σʔλͷอ࣋ظ͕ؒΑΓ௕͘ • 1ͭͷΫΤϦͰϩʔϧΞοϓͨ͠σʔλͱͯ͠ͳ͍σʔλΛ໰͍߹Θͤ ༰ྔ͕ɻɻɻ X-Pack feature (Basic, free)

!13 Raw Minute Hour Day Docs: 9,041,000 1,448,285 49,554 8,447
Size: 2.23gb 1.25gb 48.40mb 9.10mb Docs % : -83.98% -99.45% -99.91% Size %: -43.68% -97.84% -99.59% (avg ~200 docs per minute, 32 days of data, single host) (20 grouping fields, 62 numerics @ min/max/avg == 186 metrics) Rolling up Metricbeat data ༰ྔ࡟ݮͷҰྫ MetricbeatͷϩʔϧΞοϓ (ฏۉ ~200 docs/෼ɺ32೔ؒɺ1αʔόʔ) (20ݸͷάϧʔϓϑΟʔϧυɺ62ݸͷ਺஋ @ min/max/avg == 186 metrics) X-Pack feature (Basic, free)

!14 ͦͷଞʹ΋ … • Java 10 (6.3) for Elasticsearch •
Painless execute API • ML jobs Ͱ Cross Cluster Search ͕Մೳʹ(X-Pack Platinum) • ML job ͷࣄલνΣοΫ (X-Pack Platinum)

!16 Kibana Query LanguageͰࣗಈิ׬

!17 Recently Viewed ΍ Recent Time Range ௚લʹԿΛ͔ͯͨ͠ͳ?... Recently viewed
objects Recently used time range

!18 Kibana Lab Visualizations Input controls (supports chaining) Vega Plugin

!19 Index Management UI UIͰIndex؅ཧ X-Pack feature (Basic, free)

!20 License Management UI TrialΛ࢝ΊͨΓɺBasic΁໭ͨ͠ΓɺϥΠηϯεͷߋ৽͕UI͔Β

!21 ͦͷଞʹ΋… • DashboardͷύωϧͷαΠζมߋ͕ΑΓॊೈʹ • MonitoringͷػೳΛKibanaͷը໘͔Β༗ޮʹ(σϑΥϧτ͸ແޮ) • APMͱWatcherͷ࿈ܞ͕༰қʹ

!23 LogstashͷઃఆΛΑΓ؆ུԽ ྫ͑͹... Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!24 LogstashͷઃఆΛΑΓ؆ུԽ  • Distributor  Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!25 LogstashͷઃఆΛΑΓ؆ུԽ  • Output Isolator  Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!26 LogstashͷઃఆΛΑΓ؆ུԽ  • Forked Path  Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!27 LogstashͷઃఆΛΑΓ؆ུԽ  • Collector Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!28 SNMP Poller (Beta) Centralized polling of SNMP agents with
a Logstash input plugin ϕʔλ൛ͷػೳ • TCP΋͘͠͸UDPͰSNMP v1 ͱ v2c ΛϙʔϦϯά • SNMP GETs ͱ WALKs Λαϙʔτ • ఆظతʹϙʔϦϯά • MIB ϑΝΠϧΠϯϙʔτ • ޓ׵ੑ: Logstash 2.4 Ҏ্ʹରԠ Blog: https://www.elastic.co/blog/logstash-lines-support-for-tcp-and-snmpv1-in-snmp-input-plugin Routers Switches Servers SNMP Agents v0.1.0.beta1

!29 ͦͷଞʹ΋... • Persistent Queue Breaking changes • S3ͷinput/outputͰcustom endpointͱregionͷࢦఆ͕Մೳʹ
• PQͷσΟεΫͷ࢖༻ྔ΋MonitoringͰऔಘ (Basic)

!31 • at-least-once deliveryΛఏڙ  • Metricbeat΍AuditbeatͷΑ͏ͳ  λΠϓͰॏཁ • Ϧελʔτ΍ωοτϫʔΫো֐Ͱ΋ σʔλ్͕੾Εͳ͍
Disk΁Spool (Beta) ΤοδͰӬଓԽ # ____beat.yml queue: spool: file: path: "${path.data}/ spool.dat" permissions: 0600 # One- time size: 100MiB # One- time #prealloc: true # One- time write: buffer_size: 1MiB flush.timeout: 1s flush.events: 16384

!32 ௥Ճ͞ΕͨModule Filebeat Metricbeat

!33 Filebeatͷೖྗ͕ଟ༷ʹ • TCP Input ‒ SSL/TLSΛαϙʔτ • UDP Input
• Syslog Input ‒ BSD RFC3164Λαϙʔτ ‒ ϓϩτίϧ͸TCPͱUDP filebeat.inputs:  - type: tcp  max_message_size: 10MiB  host: "localhost:9000" filebeat.inputs:  - type: udp  max_message_size: 10KiB  host: "localhost:8080" filebeat.inputs:  - type: syslog  protocol.tcp:  host: "localhost:9000" filebeat.yml

!34 ͦͷଞʹ΋... • Kubernetes/DockerͷAudodiscoverػೳͷվળ • add_host_metadataͷ௥Ճ

!35 6.3.1͕ϦϦʔε  ͞Ε͍ͯ·͢ʂ

!36 Demo

What's new in Elastic Stack 6.3

What's new in Elastic Stack 6.3

Jun Ohtani

More Decks by Jun Ohtani

Other Decks in Technology

Featured

Transcript

Jun Ohtani, Community Engineer  Kosho Owa, Principal Solution Architect Elastic

!2 ηογϣϯʹೖΔ  ͦͷલʹɻ  QRίʔυϦʔμʔͷ͝༻ҙΛʂ

!3 bit.ly/EnqEsMeetup

!4

!8

!9

!10 Elasticsearch SQL (6.3 - Experimental) ࢖͍׳ΕͨγϯλοΫεͰElasticsearchʹ໰͍߹Θͤ

!11 Elasticsearch SQL (6.3) ࢖͍׳ΕͨγϯλοΫεͰElasticsearchʹ໰͍߹Θͤ • REST API & CLI

!12 Data Rollups • API for creating an Elasticsearch process

!13 Raw Minute Hour Day Docs: 9,041,000 1,448,285 49,554 8,447

!14 ͦͷଞʹ΋ … • Java 10 (6.3) for Elasticsearch •

!15

!16 Kibana Query LanguageͰࣗಈิ׬

!17 Recently Viewed ΍ Recent Time Range ௚લʹԿΛ͔ͯͨ͠ͳ?... Recently viewed

!18 Kibana Lab Visualizations Input controls (supports chaining) Vega Plugin

!19 Index Management UI UIͰIndex؅ཧ X-Pack feature (Basic, free)

!20 License Management UI TrialΛ࢝ΊͨΓɺBasic΁໭ͨ͠ΓɺϥΠηϯεͷߋ৽͕UI͔Β

!21 ͦͷଞʹ΋… • DashboardͷύωϧͷαΠζมߋ͕ΑΓॊೈʹ • MonitoringͷػೳΛKibanaͷը໘͔Β༗ޮʹ(σϑΥϧτ͸ແޮ) • APMͱWatcherͷ࿈ܞ͕༰қʹ

!22

!23 LogstashͷઃఆΛΑΓ؆ུԽ ྫ͑͹... Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!24 LogstashͷઃఆΛΑΓ؆ུԽ  • Distributor  Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!25 LogstashͷઃఆΛΑΓ؆ུԽ  • Output Isolator  Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!26 LogstashͷઃఆΛΑΓ؆ུԽ  • Forked Path  Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!27 LogstashͷઃఆΛΑΓ؆ུԽ  • Collector Pipelineؒ௨৴ (Beta) ઃఆΛΑΓγϯϓϧʹ

!28 SNMP Poller (Beta) Centralized polling of SNMP agents with

!29 ͦͷଞʹ΋... • Persistent Queue Breaking changes • S3ͷinput/outputͰcustom endpointͱregionͷࢦఆ͕Մೳʹ

!30

!31 • at-least-once deliveryΛఏڙ  • Metricbeat΍AuditbeatͷΑ͏ͳ  λΠϓͰॏཁ • Ϧελʔτ΍ωοτϫʔΫো֐Ͱ΋ σʔλ్͕੾Εͳ͍

!32 ௥Ճ͞ΕͨModule Filebeat Metricbeat

!33 Filebeatͷೖྗ͕ଟ༷ʹ • TCP Input ‒ SSL/TLSΛαϙʔτ • UDP Input

!34 ͦͷଞʹ΋... • Kubernetes/DockerͷAudodiscoverػೳͷվળ • add_host_metadataͷ௥Ճ

!35 6.3.1͕ϦϦʔε  ͞Ε͍ͯ·͢ʂ

!36 Demo