Elastic{ON} 2018 - Managing the Elastic Stack in Production

There are many things to consider and many tools and techniques at your disposal when you begin running and managing the Elastic Stack in production. This talk highlights management features in Kibana and covers the most important elements for running and managing the Elastic Stack in production.

Steve Kearns| Director of Product Management | Elastic
Chris Earle | Monitoring Lead | Elastic
Chris Roberson | Senior Javascript Engineer | Elastic

Transcript

1. Product Management @skearns64 Managing the Elastic Stack In Production Steve Kearns Stack Monitoring Lead @pickypg Chris Earle Senior UI Engineer @chrisronline Chris Roberson
2. Agenda 2 1 Planning for Production 2 Detecting Problems 3 Fixing Problems
3. Planning for Production
4. Are you Single or Multi-Tenant? 4
5. Are you Single or Multi-Tenant? 5
6. Are you Single or Multi-Tenant? 6
7. What do your tenants need? 7 • Dedicated Indexes / Custom Mappings? • Dedicated Data Lifecycles? • What are Their Usage Patterns? • Service Level Agreement (hard with noisy neighbors!) • Upgrade and Maintenance • Chargeback / Showback • Security & Compliance requirements?
8. Multi-Cluster, Perhaps? 8
9. Cluster Architecture Master Data Coordinating Ingest Machine Learning Master Data Coordinating Ingest Machine Learning Master Data Coordinating Ingest Machine Learning
10. Cluster Architecture Master Data Coordinating Ingest Machine Learning
11. Cluster Architecture Master Data Coordinating Ingest Machine Learning Hot Hot Hot Warm Warm Warm
12. Indexing Strategy - Time Series Data Pro Tips • Set shard count based on expected ingest rate • For efficient search, fewer shards are better • Roll indexes based on target size, rather than daily • Shrink Indexes to reduce shard count
13. Index Lifecycle Rollover every day/size 1 2 3 4 Re-allocate indices Shrink to 1 shard Forcemerge Change # of replicas Re-allocate indices Change # of replicas Delete Hot Warm Cold Delete
14. Indexing Strategy - Non-Time Series Data Pro Tips • Plan shard count based on total docs & insert/update rate • Scale up search throughput with additional replicas • Set number_of_routing_shards to use Shard Splitting in 6.1+
15. Detecting Problems
16. In production What’s next 16 Use Case 1 2 3 4 5 Cluster Level Operational Level Data Level Data Discovery
17. Fixing Problems
18. 18 Cluster Alerts Opening Cluster Alerts Diagnosing and Fixing • Growing list • Watches under the hood • Uses monitoring data • Cluster alerts will be opened to allow more customization and collaboration • Unknown unknowns • What’s happening in Monitoring outside of alerts • What does it mean X-Pack monitoring
19. 19 More Questions? Visit us at the AMA
20. www.elastic.c o
21. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 21 Please attribute Elastic with a link to elastic.co