Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic 2.0

Elastic 2.0

Slide deck from a series of meetups in Denmark and Sweden in November 2015.

Elasticsearch Inc

November 26, 2015
Tweet

More Decks by Elasticsearch Inc

Other Decks in Technology

Transcript

  1. Elastic 2.0 Everything you need to know Christoph Wurm –

    Solutions Architect @ Elastic Nov 26, 2015 www.elastic.co Twitter: @elastic
  2. www.elastic.co 1 Agenda 1. Elastic and Elastic Stack 2. Elastic

    2.0 3. Use Cases from the field 4. Stuff 5. Q&A
  3. www.elastic.co 3 Downloads Mar’15 Oct’12 Apr’13 Apr’14 Oct’13 20. Millions

    of Downloads 10. 40+ Million Downloads Cumulative across Elastic products to date Jun’15 40. Sept’14
  4. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 4 Elastic stack Logstash Collect, parse and enrich data Elasticsearch Store, search, analyze Kibana Visualize and explore data Hadoop Ecosystem Hadoop connector Beats Tap into your wire data Shield Security Watcher Scheduler Marvel - Monitoring Found Scale in the cloud
  5. www.elastic.co 6 elasticsearch 2.0 • Networking • multicast removed (available

    as plugin) • will only bind to localhost by default, both ipv4 and ipv6 • new unicast node discovery § will contact hosts listed in discovery.zen.ping.unicast.hosts (should contain all master nodes) § using first 5 ports in transport.tcp.port (default 9300-9400)
  6. www.elastic.co 7 elasticsearch 2.0 • New feature: Pipeline aggregations •

    “Aggregations on the results of other aggregations” • Derivatives • Moving average • Holt Winters (prediction / anomaly detection) • Stats: Min/Max/avg • Custom
  7. www.elastic.co 11 elasticsearch 2.0 • New Features • Pipeline Aggregations

    • Query DSL/Doc Improvement • Index Compression – 10-30% • Performance & resilience • Lucene 5.2 • Update Cluster State with diffs • Doc_values by default • Sync-flush (1.6+) • Better handling for node-leave/rejoin (1.7+) • Durability-by-default • Async shard allocation (1.6+) • Breaking Backward Compatibility • Facets, Rivers – removed • Zen discovery improvements – FULL CLUSTER RESTART • Type mappings are now strict • Index segments created before ES .90.0 must be upgraded • Migration Assistant • Units are required in settings
  8. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. Dark theme Customizable maps Apps kibana 4.2: customizability
  9. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. Dark theme Offline/customizable maps Apps kibana 4.2: custom maps
  10. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. Dark theme Offline/customizable maps Apps kibana 4.2: apps
  11. www.elastic.co 16 kibana migration • Kibana 4 to Kibana 4.2

    • Support for Elasticsearch 2.x • Not backward-compatible with Elasticsearch 1.x • Dashboards are automatically migrated • Kibana 3 EOL is Nov 2015
  12. www.elastic.co 18 logstash • logstash 2.0 • compatible with Elasticsearch

    2.0 • HTTP as default transport protocol • Better shutdown process § all input plugins need to be adapted
  13. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 23 Elasticsearch @ Stagemonitor https://www.elastic.co/blog/elasticsearch-as-a-time-series-data-store
  14. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 24 Mapping { "template": "stagemonitor-metrics-*”, "settings": { "index": { "refresh_interval": "5s” } }, "mappings": { "_default_": { "dynamic_templates": [ { "strings": { "match": "*”, "match_mapping_type": "string”, "mapping": { "type": "string", "doc_values": true, "index": "not_analyzed" } } } ], "_all": { "enabled": false }, "_source": { "enabled": false },
  15. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 25 Mapping (cont.) "properties": { "@timestamp": { "type": "date", "doc_values": true }, "count": { "type": "integer", "doc_values": true, "index": "no" }, "m1_rate": { "type": "float", "doc_values": true, "index": "no" }, "m5_rate": { "type": "float", "doc_values": true, "index": "no" }, "m15_rate": { "type": "float", "doc_values": true, "index": "no" }, "max": { "type": "integer", "doc_values": true, "index": "no" }, "mean": { "type": "integer", "doc_values": true, "index": "no" }, "mean_rate": { "type": "float", "doc_values": true, "index": "no" }, "median": { "type": "float", "doc_values": true, "index": "no" }, "min": { "type": "float", "doc_values": true, "index": "no" }, "p25": { "type": "float", "doc_values": true, "index": "no" }, "p75": { "type": "float", "doc_values": true, "index": "no" }, "p95": { "type": "float", "doc_values": true, "index": "no" }, "p98": { "type": "float", "doc_values": true, "index": "no" }, "p99": { "type": "float", "doc_values": true, "index": "no" }, "p999": { "type": "float", "doc_values": true, "index": "no" }, "std": { "type": "float", "doc_values": true, "index": "no" }, "value": { "type": "float", "doc_values": true, "index": "no" }, "value_boolean": { "type": "boolean", "doc_values": true, "index": "no" }, "value_string": { "type": "string", "doc_values": true, "index": "no" } }
  16. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 26 Force merge $ curator optimize --delay 2 --max_num_segments 1 indices --older- than 1 --time-unit days --timestring %Y.%m.%d --prefix stagemonitor- metrics- • Force merge – optimize for long-term storage by merging Lucene segments on disk – happens automatically on writes – run manually after 24-48 hours – CPU intensive operation, run during off-peak hours Post Optimize: 2.2 GB => ~510 MB
  17. www.elastic.co Elastic proprietary and confidential. Please do not share or

    forward. 27 Elasticsearch @ MozDef SIEM https://www.elastic.co/elasticon/2015/sf/tackling-security-logs- with-the-elk-stack?q=mozilla Defender’s version of Metasploit and others Used in production at Mozilla, processes over 300M events per day Geolocate attackers Alerts Handle incidents in real time, collaborate with other administrators Integrates with other systems, e.g. to ban IP address ranges
  18. www.elastic.co 33 Training schedule Operations, Hands-On Workshop AMSTERDAM, Netherlands December

    1-4, 2015 Development, Operations MADRID, Spain January 19-21, 2016 Development, Operations, Kibana BERLIN, Germany January 25-28, 2016 Development, Operations, Hands- On Workshop COPENHAGEN, Denmark January 26-29, 2016 PARIS, France February 2016 LONDON, United Kingdom February 2016 AMSTERDAM, Netherlands February 2016 LONDON, United Kingdom February 2016 training.elastic.co
  19. q&a