Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Cooking Up Elasticsearch with Chef at elastic{on}

Martin Smith
February 19, 2016

Cooking Up Elasticsearch with Chef at elastic{on}

It’s not written by Julia Child or Gordon Ramsey, but the elasticsearch cookbook is worth it. Come hear Martin Smith — co-author of the Chef Community Cookbook for Elasticsearch — present a deep dive on automating Elasticsearch installation and configuration with Chef, and re-writing the open source Chef cookbook the last 12 months.

Martin Smith

February 19, 2016
Tweet

More Decks by Martin Smith

Other Decks in Technology

Transcript

  1. 1 Martin B. Smith, DevOps Engineer Rackspace Hosting, Inc. 19

    February, 2016 @martinb3 Cooking up Elasticsearch with Chef
  2. 3 Chef Ecosystem Desired State Centralized Resource Primitives • Idempotency

    • Declarative • Client-server model • Data & secret storage • Management tools • Ruby DSL • Multiple implementations
  3. 4 an operation that will produce the same results if

    executed once or multiple times Idempotency … desired state
  4. 5 5

  5. 6 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do

    action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb
  6. 7 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do

    version node[‘curl_version’] action :install end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb default[‘curl_version’] = ‘1.2.3’ attributes/curl.rb
  7. 8 Recipes and attributes Examples from ‘mycookbook’ package ‘curl’ do

    action :install end file ‘/etc/motd’ do content ‘Curl is installed’ end include_recipe ‘mycookbook::curl’ recipes/curl.rb recipes/default.rb
  8. 10 Elasticsearch cookbook • 2011 – Initial commits from Karel

    Minarik, ES @ 0.18.5 • 2013 – Capable of installing monit, nginx, EBS via Fog, Discovery • 2014 – Began discussing rewrites • 2015 – GCE support, and the kitchen sink… • 2015 July – released v1.0.0 A short history Source: Gray Arial 10pt
  9. 11

  10. 12 Technical debt Attribute soup No multiple ES instances Unmaintained

    features Lack of testing Problems with the ‘current’ cookbook
  11. 13 - Focus on basic features - Respecting Chef standards

    - Testability Guiding principles for rewrite/refactor - October 2014
  12. 14 Simple, declarative resources Examples for package, service, file package

    ‘curl’ service ’mysql' do action [:enable, :start] end file ’/etc/motd' do content ‘Hello World!’ end
  13. 15 Where we ended up Simple resources with no surprises

    Add appropriate user and group Install, uninstall by package or archive file (tar.gz) Configure elasticsearch.yml, parity with packages Create and start system service Install and remove plugins via bin/plugin
  14. 16 Simple, declarative resources Examples using elasticsearch cookbook elasticsearch_install 'elasticsearch'

    do type :package end elasticsearch_configure 'elasticsearch' do logging(:action => 'INFO') allocated_memory '123m' configuration('node.name' => 'arbitrary_name') end elasticsearch_service 'my_elasticsearch_node' do service_actions [:enable, :start] end
  15. 17 Where we ended up Simple resources with no surprises

    Defaults follow packages, which have evolved Recipe installs most defaults, otherwise use resources Missing the more exotic features (EBS, S3, Discovery) Still cannot handle complex cases with plugins Does support multiple instances
  16. 19 FOLLOWED of all Chef cookbooks 27th 1% 70% Elasticsearch

    cookbook TOP COOKBOOKS out of 2719 on Community ‘Supermarket’ CHEF CLIENTS are Fortune 1000 companies
  17. 20 How to get involved Logstash, Kibana, ELK stack Elastic

    stack elastic/cookbook-elasticsearch lusis/chef-logstash lusis/chef-kibana rackspace-cookbooks/elkstack
  18. 21

  19. 22 Option 3 Martin B. Smith, DevOps Engineer Rackspace Hosting,

    Inc. 19 February, 2016 @martinb3 Questions outside by AMA booth
  20. 23 Except where otherwise noted, this work is licensed under

    http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders.