Elasticsearch - What's new in 2.0?

A brief overview about all the new things in the upcoming Elasticsearch 2.0 release, covering new features, cleanups, stability, security and plugins. Held at the Search Meetup Munich in October 2015.

Transcript

1. Elasticsearch 2.0 What’s cooking? Alexander Reelsen @spinscale alex@elastic.co
2. Agenda Simplification Security Resiliency Features Plugins
3. About 2012 Elasticsearch got founded Series A investment Trainings Supports subscriptions
4. About 2012 Series B investment Kibana Elasticsearch for Apache Hadoop Integration Logstash Elasticsearch Clients 2013 Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
5. About 2012 Series C investment Marvel released 2013 2014
6. About 2012 Shield goes GA First user conference & rebrand Found acquired Packetbeat joins Watcher in beta 2013 2014 2015
7. About 2012 2013 2014 2015 Joined in March 2013 Working on Elasticsearch, Shield & Watcher Development, Trainings, Conferences, Support, Blog posts We're hiring...
8. Cleanups
9. Cleanups Replaced: Facets Removed: shutdown API Removed: Thrift/Memcached protocol support Removed: Rivers Removed: Bulk UDP
10. Moved into plugins Delete-by-query murmur3/size types Multicast discovery
11. Mapping changes
12. Mapping changes Fields … must be referenced by full name … names cannot be referenced using type name prefix … names cannot contain dots Types Type names cannot start with a dot Type may no longer be deleted Removed: index_analyzer & _analyzer date format changes
13. Mapping changes Fields … must be referenced by full name … names cannot be referenced using type name prefix … names cannot contain dots Types Type names cannot start with a dot Type may no longer be deleted Removed: index_analyzer & _analyzer date format changes https://www.elastic.co/blog/great-mapping-refactoring
14. Query/Filter changes
15. Query/Filter changes Queries/Filters merged into queries context decides over scoring Two-Phase Execution approximation and verification Query Caching based on usage pattern by tracking queries Using roaring bitmaps for caches
16. Query/Filter changes { "filtered" : { "query": { query definition }, "filter": { filter definition } } }
17. Query/Filter changes { "bool" : { "must": { query definition }, "filter": { filter definition } } }
18. Security
19. Security Security Manager enabled by default Very reduced permission set after startup Networking Unicast by default Binds to local interfaces only by default
20. Reliability
21. Reliability Transaction log: fsync() after each operation Multiple path data striping Cluster state diffs Doc values by default Units are required (“1m”) 1.6: Synced flush, async shard allocation fix 1.7: delayed allocation
22. Features
23. Features Pipeline Aggs Derivatives, moving avg, holt winters, time series! Index compression Structured exceptions
24. Plugins 22 plugins are part of main repo now bin/plugin install clouds-aws Shield/Watcher/Marvel releases aligned now bin/plugin install watcher Helper: Migration plugin!
25. Plugins
26. Plugins Shield document/field level security, realms API Watcher hipchat/slack actions, activate/deactivate API Marvel new UI, less but more useful metrics
27. Plugins
28. Sense is back!
29. Resources
32. https://www.elastic.co/webinars/elasticsearch-2-0-overview/
33. Resources https://imotov.github.io/presentations/what-s-new-in-elasticsearch-20.html https://www.elastic.co/blog/to-shade-or-not-to-shade https://www.elastic.co/blog/elasticsearch-unplugged https://www.elastic.co/blog/core-delete-by-query-is-a-plugin https://www.elastic.co/blog/staying-in-control-with-moving-averages-part-1 https://www.elastic.co/blog/staying-in-control-with-moving-averages-part-2 https://www.elastic.co/blog/better-query-execution-coming-elasticsearch-2-0 https://www.elastic.co/blog/out-of-this-world-aggregations https://www.elastic.co/blog/store-compression-in-lucene-and-elasticsearch https://www.elastic.co/blog/great-mapping-refactoring https://www.elastic.co/blog/frame-of-reference-and-roaring-bitmaps https://www.elastic.co/blog/elasticsearch-2-0-0-beta1-released https://www.elastic.co/blog/elasticsearch-2-0-0-beta2-released https://www.elastic.co/blog/elasticsearch-2-0-0-rc1-released https://www.elastic.co/elasticon/tour/2015/muenchen https://www.elastic.co/elasticon/conf/2016/sf
34. Q & A Thanks for listening! Alexander Reelsen @spinscale alex@elastic.co We're hiring https://www.elastic.co/about/careers We're helping https://www.elastic.co/subscriptions