Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Stop Guessing, Start Measuring: Getting Your Cl...

Stop Guessing, Start Measuring: Getting Your Cluster Size Right with Rally Benchmarks

Last year, Elastic released Rally, its homegrown benchmarking tool for Elasticsearch. In this talk, Christian and Daniel will describe how to use, extend, and configure Rally when running benchmarks for sizing, performance tuning, or capacity planning. This will include insight into Rally internals and practical examples of simulating realistic load for different use cases, as well as a discussion around methodology and analysis of results.

Christian Dahlqvist l Solutions Architect l Elastic
Daniel Mitterdorfer l Software Engineer l Elastic

Elastic Co

March 09, 2017
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Stop Guessing, Start Measuring Getting Your Cluster Size Right with

    Rally Benchmarks Christian Dahlqvist and Daniel Mitterdorfer
  2. 4

  3. 5

  4. 7

  5. • Execute benchmarks based on Elasticsearch API • Gather system

    metrics (CPU usage, disk I/O, GC ...) and attach “telemetry” for more insights • Manage and provision Elasticsearch instances • Structured storage for metrics What is Rally? 8 Macrobenchmarking for Elasticsearch Think “JMeter on Steroids”
  6. | Metric | Operation | Value | Unit | |--------------------------------:|-------------:|----------:|-------:|

    | Indexing time | | 124.712 | min | | Merge time | | 21.8604 | min | | Refresh time | | 4.49527 | min | | Merge throttle time | | 0.120433 | min | | Median CPU usage | | 546.5 | % | | Total Young Gen GC | | 72.078 | s | | Total Old Gen GC | | 3.426 | s | | Index size | | 2.26661 | GB | | Totally written | | 30.083 | GB | | … | … | … | … | | 99.9th percentile latency | index-update | 2972.96 | ms | | 99.99th percentile latency | index-update | 4106.91 | ms | | 100th percentile latency | index-update | 4542.84 | ms | | 99.9th percentile service time | index-update | 2972.96 | ms | | 99.99th percentile service time | index-update | 4106.91 | ms | | 100th percentile service time | index-update | 4542.84 | ms | Summary Report 13
  7. { "trial-timestamp": "20170223T000046Z", "@timestamp": 1487811668093, "relative-time": 150148201, "track": "geonames", "challenge":

    "append-no-conflicts-index-only", "car": "4gheap", "sample-type": "normal", "name": "disk_io_write_bytes", "value": 12355731456, "unit": "byte", "meta": { "node_name": "rally-node0", "cpu_model": "Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz", "os_name": "Linux", "os_version": "4.4.0-38-generic", "jvm_vendor": "Oracle Corporation", "jvm_version": "1.8.0_101", "distribution_version": "6.0.0-alpha1", "source_revision": "18f57c0" } } Metrics Records 14
  8. 15

  9. 1 8

  10. Why benchmark? 20 What insights are we looking for? Cluster

    size required to support use-case Optimal cluster configuration What hardware to use Cluster behaviour under varying load
  11. Search use-cases • Complex queries • Complex data models •

    Limited indexing • Latency sensitive Benchmarking and use-cases 21 Event-based use-cases • Indexing heavy • Flat data model • Analysis through Kibana • Limited other querying Common patterns
  12. Why more complex benchmarks? 22 How does different types of

    load interact? Target Indexing Rate Achieved Indexing Rate Maximum Kibana Latency Minimum Kibana Latency Average Kibana Latency
  13. 24 Data generation Simulate Kibana usage Easy to use and

    extend • Support long benchmarks • Rate-limiting • Configurable timestamp • Configurable • More realistic load patterns • Easy to get started • Run it as-is • Adapt to your scenario • Use as inspiration What do we need?
  14. • _shrink and _rollover APIs add flexibility • 8 CPU

    cores, 61GB RAM • 6 2TB disks in RAID10 => ~6TB storage • Separate instance for Rally - CPU intensive Example: Using the track to evaluate hardware 2 5 How performant are d2.2xlarge instances? Why d2 instances?
  15. Bulk index data generator 30 Unbounded volumes of access log

    data { "agent": "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0", "useragent": { "os": "Windows 8.1", "os_name": "Windows 8.1", "name": "Firefox" }, "geoip": { "country_name": "Canada", "location": [-95, 60] }, "clientip": "184.151.239.181", "referrer": "-", "request": "/favicon-16x16.png?change=123", "bytes": 1763, "verb": "GET", "response": 200, "httpversion": "1.1", "@timestamp": "2017-02-22T13:09:06.343Z", "message": "184.151.239.181 - - [2017-02-22T13:09:06.343Z] \"GET /favicon-16x16.png?change=123 HTTP/1.1\" 200 1763 \"-\" \"-\" \"Mozilla/5.0 (Windows NT 6.3; WOW64; rv:42.0) Gecko/20100101 Firefox/42.0\"" }
  16. • Content Issues Dashboard • Internal/external missing link analysis •

    Analyses subset of data • Lightweight Simulating Kibana queries 31 2 Out-of-the-box simulated Kibana dashboards • Traffic Dashboard • Traffic pattern analysis • Analyses all data • Heavyweight
  17. How should I use it? 32 Fork and extend the

    track Dynamically loads files from directories Add files with new operation and challenge definition files - no conflicts ... |-- parameter_sources | +-- [custom parameter sources] |-- runners | +-- [custom runners] |-- challenges | |-- bulk-size-evaluation.json | |-- elasticlogs-1bn-load.json | |-- shard-sizing.json | +-- my_challenges.json eventdata |-- track.json |-- track.py |-- mappings.json |-- operations | |-- indexing.json | |-- querying.json | |-- stats.json | +-- my_operations.json ...
  18. 38 More Questions? Visit us at the AMA or Discuss

    in “BoF: Benchmarking Elasticsearch” today at 12:45
  19. • “measuring tape” by Sean MacEntee: https://www.flickr.com/photos/smemon/14618772953/ (CC BY 2.0)

    • “Works Mini Cooper S DJB 93B” by Andrew Basterfield: https://www.flickr.com/photos/andrewbasterfield/4759364589/ (CC BY-SA 2.0) Image Credits 40
  20. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/

    Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 41 Please attribute Elastic with a link to elastic.co