Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What's new in Elastic Stack 6.1?

Jun Ohtani
January 30, 2018

What's new in Elastic Stack 6.1?

LINE Dev Meetupで発表した資料になります。
https://line.connpass.com/event/76226/

Jun Ohtani

January 30, 2018
Tweet

More Decks by Jun Ohtani

Other Decks in Technology

Transcript

  1. about • Me, Jun Ohtani / Developer Advocate ‒ lucene-gosenίϛολʔ

    ‒ σʔλ෼ੳج൫ߏஙೖ໳ͷஶऀͷ1ਓ ‒ http://blog.johtani.info
 • Elasticsearch, founded in 2012 ‒ Products: Elasticsearch, Logstash, Kibana, Beats 
 X-Pack, Elastic Cloud
 Professional services: Support & development subscriptions ‒ Trainings, Consulting, SaaS 2
  2. 5 Brand new upgrade experience Upgrades just got oh so

    simpler Upgrading to 2.x Upgrading to 5.x Upgrading to 6.x
  3. 6 Brand new upgrade experience • New Upgrade Assistant (UI

    & API) • Zero downtime upgrades ‒Rolling restarts from latest 5.x to 6.x ‒Cross-cluster search across major version Upgrades just got oh so simpler
  4. 7 Space-saving columnar store • Better for storing sparse fields

    • Save on disk space & file system cache Tapping into Lucene 7 goodness (sparse doc value) user first middle last age phone johns Alex Smith jrice Jill Amy Rice 508.567.121 1 mt123 Jeff Twain 56 sadams Sue Adams adoe Amy Doe 31 lp12 Liz Potter
  5. 8 Much speedier sorted queries Tapping into Lucene 7 goodness

    (index sorting) Player 1 Score: 600 5.x Query for top 3 player scores Player 2 Score: 0 Player 3 Score: 200 Player 4 Score: 700 Player 5 Score: 300 Player 1907 Score: 800 ... Query for top 3 player scores ... Player 1907 Score: 800 Player 4 Score: 700 Player 1 Score: 600 Player 5 Score: 300 Player 3 Score: 200 Player 2 Score: 0 6.x Sort at index time vs. query time Optimize on-disk format for some use cases Improve query performance at the cost of index performance
  6. 9 Large Improvements to Replication • Limit syncs to only

    changed documents (instead of file-based recovery) • Fast replica recovery after temporary unavailability (network issues, etc.) • Re-sync on primary failure • Laying foundation for future big league features ‒Cross-datacenter replication ‒Changes API (tbd) New operation-based approach to recovery (sequence numbers)
  7. 10 Breaking changes • Improved tools to handle breaking changes

    ‒Deprecation logging ‒Upgrade Assistant (UI & APIs) • Refer to Release Notes for complete list • Test, test, test Because major releases is time for major cleanup
  8. 11 Simpler data models with type removal • Breaking change

    • Gradual migration path ‒ 6.0 indices can be created with only one type ‒ Existing 5.x indices using _type will continue to function • Introducing new APIs for type-less operations Say goodbye to _type confusion
  9. 12 Some interesting changes • Rename template to index_patterns in

    _template • Content-Type detection disabled • Set explicit Content-Type in request header • Deprecation of _all • _all can no longer be configured for indices in 6.0 • Use all_fields in query
  10. 13 Some interesting changes • <= 2.x indices need to

    be reindexed • Re-index into 5.x or 6.0 cluster • Deprecate Groovy, Python, Javascript lang plugin • Rewrite scripts in plainless • Java High Level REST Client • Starting from version 5.6.0 a new Java client has been released.
  11. 14 Distributed watch execution • Watches are no longer executed

    on only the master node • They are executed on nodes which hold shards of the .watches index • Configure all or specific nodes dedicated to watch execution X-Pack feature (Gold)
  12. 15 Secure all the things Default security No default passwords

    Mandatory TLS between nodes changeme X-Pack feature (Gold)
  13. 16 What's new in 6.1? • Index Splitting • Original

    primary shard is split into some primary shard in new index • Composite Aggregation • Designed to return ALL terms and sorted in ‘natural order’ • Improve indexing throughput • Simple change of _fields metafield • Scripted Similarity • Custom similarity has become much easier
  14. 18 Export saved searches to CSV with a single click

    Highly requested feature Trigger export via Watcher X-Pack feature (Basic, free)
  15. 19 Lock down edits with Dashboard Only mode Share dashboards

    without worrying about accidental changes X-Pack feature (Gold)
  16. 21

  17. 22 6.0 starts Kibana on the accessibility path • High

    contrast color scheme • Keyboard accessibility • Screen reader support • More improvements on the way Accessibility improvements
  18. 24 Kibana now supports multiple query languages • Lucene Query

    Language (default) • Kuery (off by default, experimental in 6.0) • ... perhaps others in the future We want your feedback! • Enable Kuery from Advanced Settings More ways to query with Kuery Consistent syntax and simple to get started
  19. 25 Get e-mail alerts on Cluster Alerts • Cluster Alerts

    are built-in Watches for cluster issues • Get e-mails when Cluster Alerts get triggered and resolved • Add admin e-mail in Kibana Advanced Settings
 X-Pack feature (Gold)
  20. 26 Easily create simple threshold alerts New form based UI

    for threshold alerts X-Pack feature (Gold)
  21. 31 • Run multiple, distinct workloads on a single Logstash

    JVM • Simplify dataflow logic by managing per data source logic independently • Monitor each pipeline separately with the new Pipeline Viewer Multiple Pipelines, One Logstash Logstash JDBC Pipeline Netflow Pipeline Apache Pipeline
  22. 32 • Visualize pipeline topologies as graphs
 • Reveal bottlenecks

    at the plugin level
 • Optimize dataflow with better metrics
 • Integrated with Monitoring UI Zoom in on your Pipelines Pipeline Viewer X-Pack feature (Basic, free)
  23. 33 • Manage multiple pipelines from multiple nodes in a

    single UI
 • Logstash nodes can poll and dynamically reload pipelines on configuration change
 • Secure access to configuration management with X-Pack Centrally Manage Logstash Pipelines Configuration Management X-Pack feature (Gold) Elasticsearch Kibana Logstash Apache Logstash Logstash Config Mgmt UI DevOps / Admins Auto-Update Pipelines JDBC Netflow
  24. 35 Convert ingest node to Logstash pipelines with a CLI

    tool Why Logstash? • More input sources • Multiple outputs • Richer transformations • Buffering, persistent queues Easily Migrate from Ingest Node Pipelines Ingest Node Converter $LS_HOME/bin/ingest-convert.sh --input file:///path/to/ ingest_pipeline.json --output file:///path/to/ logstash_pipeline.conf
  25. 36 File Based Ruby Scripting Support - 6.1.0 filter {

    ruby { # Cancel 90% of events path => "/etc/logstash/drop_percentage.rb" script_params => { "percentage" => 0.9 } } } def register(params) @should_reject = params["reject"] end def filter(event) return [] if event.get("message") == @should_reject event.set("field_test", rand(10)) extra_processing(event) [event] end ...
  26. 38 • New Kubernetes module in Metricbeat ‒ CPU, memory,

    bytes on network and more. • New processor to add_docker_metadata ‒ Container ID, name, image, labels • New processor to add_kubernetes_metadata ‒ Pod name, pod namespace, container name, pod labels Beats <3 containerization Monitor your Docker and Kubernetes deployments with ease
  27. 40 • Improved dashboards for Metricbeat system module • Filebeat

    NGINX module ships with Machine Learning jobs ‒ We want your feedback Better Modules
  28. 41 • Skip the hassle of parsing auditd logs ‒

    Auditbeat subscribes to the kernel directly • Reuse auditd rule formats (no need to learn new rule formats) • Plus, file integrity checks on Linux, macOS, and Windows ‒ Watch files or directories (non-recursively) for changes ‒ Report file metadata and MD5, SHA1, SHA256 hashes on changes Auditbeat - a simpler way to track audit logs An alternative to auditd on Linux
  29. 42 • Index pattern versions • Simpler configuration ‒ Module

    commands and configuration files ‒ module.d directory ‒ ./metricbeat module enable system ‒ Dashboards are easier to load and packaged with the Beat And moar awesome all around
  30. 43 In 6.1? • Docker Autodiscovery • New Metricbeat and

    Filebeat modules • Metricbeat: Graphite, HTTP server metricset, Etcd, Logstash, System uptime, Windows service, OSD tree, RabbitMQ queue metricset • Filebeat: Logstash, Postgres, Kafka • TLS support in Packetbeat
  31. Thanks for listening! Q & A We’re hiring! https://www.elastic.co/about/careers/ We’re

    helping! https://www.elastic.co/subscriptions http://training.elastic.co