Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Gearing Up for Elastic Stack, X-Pack 5.0 Releases

Gearing Up for Elastic Stack, X-Pack 5.0 Releases

Introductions for new features and breaking changes of Elasticsearch, Logstash, Kibana and Beats version 5.0 releases. Information based on 5.0.0-alpha4 releases and subject to change before the general availability.

Kosho Owa

July 14, 2016
Tweet

More Decks by Kosho Owa

Other Decks in Technology

Transcript

  1. ‹#› Kosho Owa, Solutions Architect, Elastic July 14th, 2016 Gearing

    Up for 5.0 Elastic Stack, X-Pack 5.0.0-alpha4 releases
  2. 2 Elastic Cloud Security X-Pack Kibana User Interface Elasticsearch Store,

    Index,
 & Analyze Ingest Logstash Beats + Elastic Stack Elastic Stack, X-Pack, and Cloud Alerting Monitoring Reporting Graph
  3. What’s New in Elasticsearch 5.0 • Lucene 6 • Ingest

    node • Painless scripting • Text/keyword to replace strings • Completion suggester v2 • Settings validation • Safety in production • Percolate query • Deleted Index Tombstones 3 • Indexed scripts/templates • Dots in Field Names • Cluster Allocation Explain API • Rollover/Shrink Index API • Delete by Query and more!
  4. Ingest Node • The data goes through a pipeline before

    being indexed • Popular Logstash filters (grok, split, convert and date) are implemented • Filebeat to forward a log file to Elasticsearch directly 4 PUT _ingest/pipeline/my_pipeline_id { "description" : "...", "processors": [ { "grok": { "field": "message", "patterns": ["%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} % {NUMBER:duration}"] }}]} PUT my-index/my-type/my-id?pipeline=my_pipeline_id { "message": "55.3.244.1 GET /index.html 15824 0.043" }
  5. Text/Keyword to Replace Strings • The string field datatype has

    been replaced by the text field for full text analyzed content, and the keyword field for not-analyzed exact string values • The string field to be deprecated in 6.0 • Default string mappings: 5 { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore_above": 256 } } }
  6. Safety in Production • Bootstrap Checks Must be Passed in

    Production Mode - Heap size check - File descriptor check - Memory lock check - Minimum master nodes check - Maximum number of threads check - Maximum size virtual memory check - Maximum map count check - Server JVM check • Development vs Production Mode - No external interface bound - development (default), otherwise production 6 • Docs > Elasticsearch Reference > Setup Elasticsearch > Bootstrap Checks
  7. Dots in Field Names • Treat dots in field names

    as though they were an object • The following two documents are treated as thought they were the same: 7 PUT my_index/my_type/1 { "aaa.bbb.ccc": "some_val", "aaa.ddd": "other_val" } PUT my_index/my_type/2 { "aaa": { "bbb": { "ccc": "some_val" }, "ddd": "other_val" } }
  8. Rollover Index API • Rolls an alias over to a

    new index when the existing index is considered to be too large or too old • Settings, mappings, and aliases can be specified other than using index templates 8 PUT /logs-0001 { "aliases": { "logs_write": {} }} POST logs_write/_rollover { "conditions": { "max_age": "7d", "max_docs": 1000 }, "settings": { "index.number_of_shards": 2 }}
  9. Shrink Index API • Shrinks an existing index into a

    new index with fewer primary shards • The requested number of primary shards in the target index must be a factor of the number of shards in the source index (e.g. 8 primary shards into 4, 2 or 1; 15 primary shards into 5, 3 or 1) 9 POST my_source_index/_shrink/my_target_index { "settings": { "index.number_of_replicas": 1, "index.number_of_shards": 1, "index.codec": "best_compression" }, "aliases": { "my_search_indices": {} } }
  10. Delete by Query • Delete-By-Query plugin has been removed and

    replaced by the Delete By Query API • Performs a deletion on every document that match a query 10 POST twitter/_delete_by_query { "query": { "match": { "message": "some message" } } }
  11. Field Data is Disabled in Analyzed Fields • Field data

    cache is no longer loaded to memory by default • Aggregation does not work in analyzed fields • “fielddata” must be set “true” explicitly when using 11 PUT my_index { "mappings": { "_default_" :{ "properties": { “my_text_field” : { "type": "text", "index": "analyzed", "fielddata": true } } } } }
  12. • Features: - Cluster Checkup - Reindex Helper - Deprecation

    Logging • Works on Elasticsearch 2.3.x only Migration Helper 12 • github > Elasticsearch Migration Helper • Docs Elasticsearch Reference > Breaking changes > Breaking changes in 5.0
  13. What’s New in Kibana • Flat plug-in structure • Upload

    CSV • Sense in Console • X-Pack Integration 13
  14. Upload CSV • Import a CSV file into Elasticsearch directory

    through Kibana up to 1 GB • Tips - Date/time field must look like “2016-07-12T03:47:19.744Z” or “2016-07-12” - Single title and at least 10 data lines are required - No extra spaces between columns - Save in UTF-8 14
  15. What’s New in X-Pack • Kibana in Monitoring • Reporting

    • Security - Users/Roles management - Change password 15
  16. What’s New in Logstash 5.0 • Monitoring API • Kafka

    0.9, 0.10 support • Improved Release packages • Settings file • Java event API • Plugin generator • JSON logging • Thread safe Elasticsearch output 19
  17. Monitoring API • Provides the following monitoring APIs to retrieve

    runtime metrics - Plugins API - Node Stats API - Pipeline Stats API - Hot Threads API 20 • Docs > Logstash Reference > Monitoring APIs curl "localhost:9600/?pretty" { "host" : “logstash-host.local", "version" : "5.0.0-alpha4", "http_address" : "127.0.0.1:9600", "build_date" : "2016-06-28T22:22:41+02:00", "build_sha" : "f77ec6c845d864a28862e590dff621c68f0823e4", "build_snapshot" : false }
  18. What’s New in Beats • JSON support in Filebeat •

    Integration with Ingest Node • Packetbeat IP/TCP flows • Kafka output • Winlogbeat improvements • Improved Redis output • Kibana dashboards per Beat • New directory layout • Metricbeat replaces Topbeat 21 • Support for gzip compression in Elasticsearch output • Ignore Symlinks log files
  19. Metricbeat • Contains Apache, MongoDB, MySQL, Nginx, Redis, System and

    Zookeeper modules • Developer Guide available for your own metrics 22 metricbeat.modules: #---------------------------- Apache Status Module --------------------------- - module: apache metricsets: ["status"] enabled: true period: 1s filters: hosts: ["http://127.0.0.1/"] #---------------------------- MySQL Status Module ---------------------------- - module: mysql metricsets: ["status"] enabled: true period: 2s hosts: ["root@tcp(127.0.0.1:3306)/"]