Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elastic{ON} 2018 - Stretching the Cloud -- Fle...

Elastic Co
March 01, 2018

Elastic{ON} 2018 - Stretching the Cloud -- Flexibility in cloud deployments

New Elastic Cloud capabilities allow you to match your provisioned hardware to your use cases, making it easier to efficiently run Elasticsearch for multiple workloads such as hot/warm architectures, dedicated master nodes, and machine learning and enable Elastic Stack features like APM, Logstash, and more.

Jonathan Halterman | Cloud Engineer | Elastic
Uri Cohen | Director of Product Management | Elastic

Elastic Co

March 01, 2018
Tweet

More Decks by Elastic Co

Other Decks in Technology

Transcript

  1. Elastic{ON} 2018 Stretching the Cloud: Flexibility in Cloud Deployments Jonathan

    Halterman, Cloud Software Engineer Uri Cohen, Cloud Product Manager
  2. Agenda 2 1 Intro to Elastic Cloud [Enterprise] 2 Why

    We’re Here 4 How It Really Works 5 What’s Next 3 A Cluster’s Tale
  3. 3 Providers AWS | GCP Azure is coming soon 2

    Regions 9 AWS + 4 GCP More in the pipe 13 Clusters For any use case you can think of 10K+ Docs And growing as we speak... 300B+ Elastic Cloud in A Nutshell
  4. 5 Customers From all industries and sizes Dozens Releases 1

    major 1 minor 5 service releases 7 Clusters Managed by ECE Thousands Hosts Already running ECE in Production Hundreds Elastic Cloud Enterprise, 10 Months Post GA
  5. In Elastic Cloud, All Nodes & Hosts Are the Same

    • Every node is Master, Data and Ingest, all at the same time • They all run on the same I/O optimized setup
  6. In Elastic Cloud, All Nodes & Hosts Are the Same

    AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest AWS i3 Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest Master/Data/Ingest
  7. 12 The typical Elastic adoption pattern Start Small 1 Even

    Bigger Grow Bigger 2 3 More data More Users
  8. 13 The typical Elastic adoption pattern Start Small 1 Even

    Bigger Grow Bigger 2 3 More data More Users Do More with Your Data: ML and APM! 4 More Insights
  9. 15 Grow Your Cluster Further - Hot/Warm Master Kibana Hot

    Nodes Warm Nodes Automatic Index Curation Data Feed
  10. What We’ve Seen Multiple Node Types In the Same Cluster

    1 2 3 Place each node on different HW that’s most appropriate for it Deployment templates: a way to model and deploy an entire Elastic stack
  11. Why Different Hardware Profiles? SSDs Fast data access, fast writes.

    But $$$$ HDDs slow data access, but cheap. Good for archiving Lots of CPU Processing at Ingestion Many queries Lots of RAM Aggregations Kibana Master nodes APM
  12. AWS i3 AWS i3 AWS i3 AWS i3 AWS i3

    AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Tags and Metadata AWS i3 - High I/O AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32 instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48
  13. AWS i3 AWS i3 AWS i3 AWS i3 AWS i3

    AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Tags AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48 AWS i3 - High I/O instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32
  14. AWS i3 AWS i3 AWS i3 AWS i3 AWS i3

    AWS i3 AWS i3 AWS i3 AWS i3 AWS i3 Host Metadata AWS d2 - Dense Storage AWS c5 - High CPU AWS r4 - High Mem AWS m5 - Gen Purpose instance-type: aws.d2 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 48000GB vCPUs: 36 instance-type: aws.c5 ssd: false vpc: true foo: qux ---- RAM: 144GB Disk: 1024GB vCPUs: 36 instance-type: aws.r4 ssd: false vpc: true foo: baz ---- RAM: 244GB Disk: 1024 vCPUs: 32 instance-type: aws.m5 ssd: false vpc: true foo: bar ---- RAM: 192GB Disk: 1024 vCPUs: 48 AWS i3 - High I/O instance-type: aws.i3 ssd: true vpc: true foo: bar ---- RAM: 244GB Disk: 7600GB vCPUs: 32
  15. Choosing the Right Hardware Data - High I/O Requirements: Fast

    I/O, Ample Memory Filter: instance-type is "aws.i3" Disk:RAM ratio: 30GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4
  16. Choosing the Right Hardware Requirements: Tons of storage, Ample Memory

    Filter: instance-type is "aws.d2" Disk:RAM ratio: 100GB disk for every 1GB RAM i3 Data - High Storage m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4
  17. Choosing the Right Hardware Requirements: Ample CPU, Ample RAM Machine

    Learning Filter: instance-type is "aws.m5" Disk:RAM ratio: 5GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4
  18. Choosing the Right Hardware Requirements: Ample RAM Kibana Filter: instance-type

    is "aws.r4" Disk:RAM ratio: 4GB disk for every 1GB RAM i3 m5 c5 r4 m5 c5 d2 i3 d2 i3 c5 m5 i3 r4 d2 r4
  19. Tying It All Together: Deployment Templates Template: hot/warm Data -

    Hot Default size: 240GB Storage Number of zones: 3 Filter: instance-type is "aws.i3" Disk:RAM ratio: 30 Data - Warm Default size: 1TB Storage Number of zones: 2 Filter: instance-type is "aws.d2" Disk:RAM ratio: 100 Kibana Default size: 1GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 4 Master Default size: 2GB RAM Number of zones: 3 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Machine Learning Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.m5" Disk:RAM ratio: 4 APM Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3
  20. Tying It All Together: Deployment Templates Template: hot/warm Kibana Default

    size: 1GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 4 Master Default size: 2GB RAM Number of zones: 3 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Machine Learning Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.m5" Disk:RAM ratio: 4 APM Default size: 0GB RAM Number of zones: 1 Filter: instance-type is "aws.r4" Disk:RAM ratio: 3 Data - High Storage Default size: 1TB Storage Number of zones: 2 Filter: instance-type is "aws.d2" Disk:RAM ratio: 100 Data - High I/O Default size: 240GB Storage Number of zones: 3 Filter: instance-type is "aws.i3" Disk:RAM ratio: 30
  21. What’s In It for You • Cost effectiveness for more

    use cases: ▪ Choose the right hardware for the job ▪ e.g. for logging: o RAM:Disk ratio of ~1:100, way cheaper per GB storage • Deploy common architectural patterns in a few clicks • Bigger clusters ▪ With dedicated masters • Machine Learning, APM and Logstash* on Elastic Cloud! * Future
  22. • Roll out planned for Q2 this year, stay tuned!

    ▪ For both Elastic Cloud and Elastic Cloud Enterprise ▪ Includes I/O, storage, CPU and RAM optimized HW o Templates for hot/warm ▪ Includes Elasticsearch (with ML), Kibana, APM • Future improvement: ▪ Gradually transition data nodes to different hardware profiles o e.g. start with hot-only, move to hot/warm What’s Next
  23. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/

    Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 39 Please attribute Elastic with a link to elastic.co