Elastic{ON} 2018 - “Here, Hold My Beer.” Customer Stories to Learn from As Told by Support

Transcript

1. Elastic 1 March 2018 @GeorgeKobar @jpcarey @juliD “Here, Hold My Beer.” Customer Stories to Learn from as Told by Support George, Jared & Jaleh
2. Most good stories start with, “Here, Hold my beer.” Unless it’s a story about you...
3. Our Goal To avoid this...
4. 4 And Provide This
5. In this session, members of the Elastic support team discuss the top three common customer mistakes and misconfigurations along with best practices and recommendations so you can avoid those issues yourself.
6. In this session, members of the Elastic support team discuss the top three common customer mistakes and misconfigurations along with best practices and recommendations so you can avoid those issues yourself.
7. { Common Evolution of Using Elasticsearch } 7
8. { Common Evolution of Using Elasticsearch } 8 Common Evolution of Marriage
9. { Common Evolution of Using Elasticsearch } 9 1 The Honeymoon 2 The Work Maintaining 3 4 Realization Marriage
10. { Common Evolution of Using Elasticsearch } 1 0 1 The Honeymoon Proof of Concept 2 The Work Tuning & Optimization, Growth Maintaining Business Continuity, Upgrades 3 4 Realization Life in Production Marriage
11. { Honeymoon }
12. Proof of Concept : { Honeymoon }
13. Proof of Concept : { Honeymoon }
14. { Bootstrap Checks } Heap size check - JVM min and max heap are equal File descriptor check - file descriptors are set to at least 65,536 Memory lock check - Bootstrap.memory_lock is set Maximum number of threads check - Allow elasticsearch process to create at least 2048 threads Maximum size virtual memory check - Allow unlimited address space for elasticsearch process Maximum map count check - sysctl -w vm.max_map_count=262144 Client JVM check - Make sure server JVM is running, not the client JVM Use serial collector check - Make sure -XX:+UseSerialGC is enabled System call filter check - Make sure system call filters are installed and enabled OnError and OnOutOfMemoryError checks - Disallow these settings when syscall filters are enabled Early-access check Disallow use of openJDK early access builds G1GC check - Disallow use of G1GC
15. Elastic Products { Honeymoon } 0.90 1.x 2.x 5.x 6.x 7+
16. { Common User Path } POC Production
17. { Common User Path } POC Production
18. { Realization }
41. { Shard Benchmarking } Rally (elasticsearch’s benchmarking tool) $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:1" --report-file=~/result_1p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:2" --report-file=~/result_2p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:4" --report-file=~/result_4p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:8" --report-file=~/result_8p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:12" --report-file=~/result_12p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:18" --report-file=~/result_18p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:24" --report-file=~/result_24p.txt $ esrally --distribution-version=6.2.2 --track=http_logs --car="4gheap" --track-params="number_of_shards:48" --report-file=~/result_48p.txt esrally 0.9.2 Google Compute n1-standard-4 (4 vCPUs, 15 GB memory) CPU platform: Intel Haswell 100gb SSD persistent disk CentOS Linux release 7.4.1708 (Core)
42. {Indexing}
43. {Terms}
44. { The Work }
45. { The Work } 45 Elasticsearch Has Many Uses Cases • Application Search • Enterprise Search • Business Analytics • Metrics / Operational Log Analytics • Security Analysis • https://www.elastic.co/use-cases
46. { The Work } 46 Elasticsearch Has Many Uses Cases •Search •Time Series Data • Application Search • Enterprise Search • Business Analytics • Metrics / Operational Log Analytics • Security Analysis • https://www.elastic.co/use-cases
47. { The Work } Elasticsearch Has Many Uses Cases • Search How Fast Are My Search Results? • Time Series How Fast Can I Index?
48. { The Work } Default Search Indexing Search Performance Index Performance
49. Search { The Work } Faster Search Performance
50. Search { The Work } Indexing Slower Indexing Performance Faster Search Performance
51. Faster Index Performance { The Work } Slower Search Performance Search Indexing
52. • Data Modeling • Filters!!! • Use Profiler for Query Optimization • Force Merge Static Indices Search Performance { Optimize for Search }
53. • Increase Refresh Interval >30 sec • Disable OS Swapping • Increase Buffer Index Size Index Performance { Optimize for Index }
54. { Optimize for Both } Index Performance Search Performance
55. • Rollover API - • Managing Time Based Series Data Efficiently • https://www.elastic.co/blog/managing-time-based-indices-efficiently • Aliases • Abstract or change your physical mapping without downtime • https://www.elastic.co/blog/aliases-ftw • Optimize for failure • Users commonly optimize and fine tune to what is available. • Rarely plan for node outage. N+1!!! OR N+2!!! { Other Considerations }
56. 5 6 {Maintenance}
57. { Business Continuity } Big Words for It Needs to Keep Working
58. 5 8 Beats Log Files Metrics Wire Data your(beat) Datastore Social Web APIs Sensors Kafka Redis Messaging Queue Nodes (X) Logstash Elasticsearch Kibana X-pack Authentication Notification X-pack LDAP AD SSO Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) { Maintaining The Family }
59. 59 Secure • Authentication • Secure Connections • Monitor - • Nodes, Kibana, and Logstash • Data Retention • Index Sizing { Keeping an Eye on Your Cluster } • High Availability • Disaster Recovery - Hot/Warm Site Monitoring Management HA and DR
60. 60 This is a sample image Lock it up
61. {Chose your Lock}
62. 62 This is a sample image Monitoring
63. {Don’t take your eyes off it }
64. 6 4 Cluster Alerts { Cluster Alerts }
65. 6 5 Nodes { Nodes }
66. 6 6 Logstash { Logstash }
67. 6 7 Logstash pipeline { Pipeline }
68. {Do it your way}
69. 69 This is a sample image Index Management
70. • Data Retention • Delete indices • Index size management. • Replicas (Change the number of replicas per shard for indices) • Rollover • Shrink • Rollover indices { Curator to the Rescue } 70 How can it help you
71. 71 This is a sample image High Availability Disaster Recovery
72. 72 Elasticsearch X-pack Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Zone A Zone B { High Availability } Shard Allocation Awareness Beats Log Files Metrics Wire Data your(beat) Nodes (X) Logstash Kibana Instances (X)
73. 73 { Disaster Recovery } Elasticsearch Kibana X-pack X-pack Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Elasticsearch Kibana X-pack X-pack Instances (X) Master Nodes (3) Ingest Nodes (X) Data Nodes - Hot (X) Data Nodes - Warm (X) Beats Log Files Metrics Wire Data your(beat) Datastore Social Web APIs Sensors Kafka Redis Messaging Queue Nodes (X) Logstash
74. { Upgrades } Keep it Fresh
75. 75 This is a sample image But It’s a Tough Climb
76. 76 This is a sample image Sudden leaps are more risky
77. • Rolling Upgrade • Upgrade assistant • Interactive Upgrade Guide. { We Heard You } 77 Major upgrade without downtime
78. 78 This is a sample image { Upgrade Assistant }
79. 79 This is a sample image { Cluster Checkup }
80. 80 This is a sample image { Reindex Helper }
81. 81 This is a sample image { Toggle Deprecation Logger }
82. 82 This is a sample image { Interactive Upgrade Guide }
83. 83 This is a sample image { Interactive Upgrade Guide }
84. { Backup } Don’t Lose Your Data
85. 85 Not This Way
86. { Backup Your Elasticsearch Data } Avoid relying solely on - Infrastructure strategies - OS Strategies Elasticsearch Snapshot API - Separate storage in your data Center - Cloud storage
87. { Curator to the Rescue }
88. { How to Get Help }
89. Timeshare Time
90. { How to Get Help } • Subscription Support -Consultative in Nature ◦ Dedicated Support Engineer ▪Use Case Driven Support ▪Understands Impact of Upgrade/Caveats For Your Environment • Baseline Taken -Metrics ▪Upgrade Planning ▪24/7 Break Fix For Production Issues
91. • Support • https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html • Training • Courses in your area: https://www.elastic.co/blog/category/releases • Consulting • For on-site, time-sensitive assistance • https://www.elastic.co/services_policy#upgrade-strategy { How to Get Help } 91
92. { How to Get Help } 92 ⬩ IRC on Freenode ◈ #elasticsearch, #logstash, #kibana, #beats ◈ #elastic-webinar ⬩ Slides and recording are available ⬩ Forum: https://discuss.elastic.co/ ⬩ Github: https://github.com/elastic ⬩ StackOverflow: http://stackoverflow.com/questions/tagged/e lasticsearch
93. 93 More Questions? Visit us at the AMA
94. www.elastic.c o
95. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 95 Please attribute Elastic with a link to elastic.co