Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Chef to scale MongoDB Clusters

ProdOps
April 19, 2016

Chef to scale MongoDB Clusters

The cloud presents many opportunities to use APIs for acquiring virtual hardware resources, servers, storage volumes and networking. This talk will show how to create a very big cluster of MongoDB sharded replica sets with the many considerations required for its operation coded using Chef. The talk will go over using Chef to provision AWS resources, managing secrets and shared data using data bags and vault, and the operations aspects of managing a large MongoDB cluster. These are lessons we learned creating such a cluster for a client who wanted to move from using a MongoDB as a service to their own servers in AWS.

http://www.meetup.com/Chef-Users-London/events/229071046/

ProdOps

April 19, 2016
Tweet

More Decks by ProdOps

Other Decks in Technology

Transcript

  1. .co.uk www. ### mongodb/metadata.rb depends 'apt' ### mongodb/recipes/repo.rb include_recipe 'apt'

    apt_repository 'mongodb-org' do uri 'http://repo.mongodb.org/apt/ubuntu' distribution "#{node['lsb']['codename']}/mongodb-org/3.2" components [ 'multiverse' ] keyserver 'keyserver.ubuntu.com' key 'EA312927' action :add end docs.mongodb.org/manual/tutorial/install-mongodb-on-ubuntu/ github.com/sunggun-yu/chef-mongodb3/blob/master/recipes/package_repo.rb
  2. .co.uk www. ### mongodb / attributes / default.rb include_attribute 'sysctl'

    override['sysctl']['params'] = { # more ports for open connections - 37k # 27017 is MongoDB listen port! 'net.ipv4.ip_local_port_range' => '28000 65000', # more concurrent connections when spammed 'net.ipv4.tcp_fin_timeout' => 15, # recycle sockets in time_wait state 'net.ipv4.tcp_tw_recycle' => 1, # re-use sockets in time_wait state 'net.ipv4.tcp_tw_reuse' => 1, # don't waste cycles timestamping tcp packets 'net.ipv4.tcp_timestamps' => 0, # higher limit for max socket connections (128 is default) 'net.core.somaxconn' => 37000, 'net.core.netdev_max_backlog' => 4096, 'net.ipv4.tcp_max_syn_backlog' => 4096, 'net.ipv4.tcp_keepalive_time' => 60, 'net.ipv4.tcp_keepalive_probes' => 3 'net.ipv4.tcp_keepalive_intvl' => 90, 'net.core.rmem_max' => 8388608, 'net.core.wmem_max' => 8388608 } ### mongodb / metadata.rb depends 'sysctl' ### mongodb / recipes / default.rb include_recipe 'sysctl::apply' supermarket.chef.io/cookbooks/sysctl
  3. .co.uk www. ### mongodb / attributes / default.rb include_attribute 'ulimit'

    override['ulimit']['users'] = { node['mongodb']['owner'] => { 'core_limit' => 0, 'filehandle_limit' => 20000, 'process_limit' => 10000 } } ### mongodb / metadata.rb depends 'ulimit' ### mongodb / recipes / default.rb include_recipe 'ulimit' supermarket.chef.io/cookbooks/ulimit
  4. .co.uk www. ### mongodb/recipes/default.rb include_recipe "#{cookbook_name}::repo" include_recipe "#{cookbook_name}::sysctl" package 'mongodb'

    do package_name 'mongodb-org' version node['mongodb']['version'] end template "/etc/init/mongodb.conf" do ### ubuntu upstart init file source "upstart-mongodb.conf.erb" owner 'root' group 'root' mode '0644' variables({ dbpath: node['mongodb']['dbpath'], logpath: File.dirname(node['mongodb']['logfile']), owner: node['mongodb']['owner'], group: node['mongodb']['group'], daemon: node['mongodb']['daemon'], cfgfile: node['mongodb']['configfile'] }) end chef tricks
  5. .co.uk www. if node['mongodb']['keyfile']['file'] if node['mongodb']['keyfile']['content'] keydata = node['mongodb']['keyfile']['content'] elsif

    node['mongodb']['keyfile']['databag'] mongodata = data_bag_item node['mongodb']['keyfile']['databag'], 'data' keydata = mongodata['key'] else Chef::Log.fatal 'MongoDB keyfile specified but no key data is available' end directory File.dirname(node['mongodb']['keyfile']['file']) do owner node['mongodb']['owner'] group node['mongodb']['group'] mode '0755' end file node['mongodb']['keyfile']['file'] do owner node['mongodb']['owner'] group node['mongodb']['group'] mode '0400' backup false content keydata end end chef trick
  6. .co.uk www. template node['mongodb']['configfile'] do source 'mongodb.conf.erb' owner 'root' group

    'root' mode '0644' variables({ mongos: node['mongodb']['mongos'], dbpath: node['mongodb']['dbpath'], logfile: node['mongodb']['logfile'], port: node['mongodb']['port'], auth: node['mongodb']['auth'], keyfile: node['mongodb']['keyfile']['file'], journal: node['mongodb']['journal'], lock_file: node['mongodb']['logrotate']['lock_file'], extra: node['mongodb']['extra'] }) notifies(:restart, "service[mongodb]") if node['mongodb']['config-restart'] end service 'mongodb' do provider Chef::Provider::Service::Upstart supports status: true, restart: true action [ :enable, :start ] end include_recipe "#{cookbook_name}::logrotate" chef trick
  7. .co.uk www. ### mongodb/templates/mongodb.conf.erb # http://docs.mongodb.org/manual/reference/configuration-options/ # Where to store

    the data. # Note: if you run mongodb as a non-root user (recommended) you may # need to create and set permissions for this directory manually, # e.g., if the parent directory isn't mutable by the mongodb user. <%= @dbpath ? "dbpath=#{@dbpath}" : '' %> # where to log logpath=<%= @logfile %> # ... # Disables write-ahead journaling # nojournal = true <%- if @journal -%> journal=true <%- elsif @journal == false -%> nojournal=true <%- end %> <%- unless @mongos %> smallfiles=true <%- end %>
  8. .co.uk www. template node['mongodb']['configfile'] do source 'mongodb.conf.erb' owner 'root' group

    'root' mode '0644' variables({ mongos: node['mongodb']['mongos'], dbpath: node['mongodb']['dbpath'], logfile: node['mongodb']['logfile'], port: node['mongodb']['port'], auth: node['mongodb']['auth'], keyfile: node['mongodb']['keyfile']['file'], journal: node['mongodb']['journal'], lock_file: node['mongodb']['logrotate']['lock_file'], extra: node['mongodb']['extra'] }) notifies(:restart, "service[mongodb]") if node['mongodb']['config-restart'] end ### ... mongodb/templates/mongodb.conf.erb # Disable the HTTP interface (Defaults to localhost:28017). nohttpinterface = true # Turns off server-side scripting. This will result in greatly limited # functionality #noscripting = true # Turns off table scans. Any query that would do a table scan fails. #notablescan = true # Disable data file preallocation. #noprealloc = true # Specify .ns file size for new databases. # nssize = <size> <%= @extra %>
  9. .co.uk www. ### mongodb/recipes/shard.rb include_recipe 'ebs-raid' # <== this creates

    Raid EBS volumes! replicaset_name = node['mongodb']['replicaset'] node.default['mongodb']['port'] = 27018 node.default['mongodb']['extra'] = <<-EOF replSet=#{replicaset_name} EOF include_recipe cookbook_name chef trick - shard (replica member)
  10. .co.uk www. ### mongodb/recipes/arbiter.rb replicaset_name = node['mongodb']['replicaset'] node.default['mongodb']['port'] = 27018

    node.default['mongodb']['journal'] = false node.default['mongodb']['extra'] = <<-EOF replSet=#{replicaset_name} EOF include_recipe cookbook_name chef trick - arbiter
  11. .co.uk www. ### mongodb/recipes/mongos.rb # once set, config_servers line must

    never change on a mongos if not node['mongodb']['config_servers'] config_servers = search( :node, "recipes:#{cookbook_name}\\:\\:configserver" ).map { |n| n['fqdn'] }.join(',') node.set['mongodb']['config_servers'] = config_servers end node.default['mongodb']['mongos'] = true node.default['mongodb']['auth'] = nil node.default['mongodb']['dbpath'] = nil node.default['mongodb']['journal'] = nil node.default['mongodb']['daemon'] = node['mongodb']['mongos_daemon'] node.default['mongodb']['extra'] = <<-EOF configdb = #{node['mongodb']['config_servers']} EOF include_recipe cookbook_name chef trick - mongoS server
  12. .co.uk www. ### roles/db-c0.rb name "db-c0" description "mongodb cluster config

    server c0" run_list [ "role[base]", "role[mongodb-configserver]" ] ### roles/db-s0m0.rb name "db-s0m0" description "mongodb cluster shard s0 m0" override_attributes \ mongodb: { replicaset: "graph-s0" } run_list [ "role[base]", "role[mongodb-shard]" ] chef roles ### roles/mongodb-shard.rb name "mongodb-shard" run_list [ "recipe[mongodb::shard]", "recipe[munin]", "recipe[mms-agent::remove]", "recipe[stackdriver::remove]" ]
  13. .co.uk www. ◦ RAID 10 w/ 4 x 1000 piops

    drives avg 13.4 ops/sec, 16.1 ops/sec max ◦ Single 4000 piops drive avg 21.4 ops/sec, 23.4 ops/sec max ◦ RAID 10 w/ 4 x 4000 piops drives avg 38.3 ops/sec, 43 ops/sec max EBS provisoned iops compose.io/articles/debunking-myth-of-raid-10-as-best-practice-on-aws blog.celingest.com/en/2013/02/01/benchmarking-mongodb-replica-aws/
  14. .co.uk www. Mongo MMAPv1 smallfiles & ext4 # format a

    block device if it has not been formatted before # use 'small' usage-type for ext4 (for mongod smallFiles setting) mkfs -t ext4 -T small "/dev/${ebs_device}" # add mount definition if it has not beed added before echo "/dev/${ebs_device} /data ext4 defaults,auto,noatime,noexec 0 0" >> /etc/fstab storage.mmapv1.smallFiles Type: boolean Default: False When true, MongoDB uses a smaller default file size. The storage.mmapv1.smallFiles option reduces the initial size for data files and limits the maximum size to 512 megabytes. storage.mmapv1.smallFiles also reduces the size of each journal file from 1 gigabyte to 128 megabytes. Use storage.mmapv1.smallFiles if you have a large number of databases that each holds a small quantity of data. The storage.mmapv1.smallFiles option can lead the mongod instance to create a large number of files, which can affect performance for larger databases.
  15. .co.uk www. Mongo MMAPv1 readahead buffer ebs_readahead_kb=32 ebs_device=xvdf blockdev --setra

    $ebs_readahead_kb "/dev/${ebs_device}" # persist across reboot echo 'ACTION=="add", KERNEL=="'$ebs_device'", ATTR{bdi/read_ahead_kb} ="'$ebs_readahead_kb'"' \ > /etc/udev/rules.d/85-ebs.rules For the MMAPv1 storage engine: • Ensure that readahead settings for the block devices that store the database files are appropriate. For random access use patterns, set low readahead values. A readahead of 32 (16 kB) often works well. • For a standard block device, you can run sudo blockdev --report to get the readahead settings and sudo blockdev -- setra <value> <device> to change the readahead settings. Refer to your specific operating system manual for more information.
  16. .co.uk www. Indexing notablescan Specify whether all queries must use

    indexes. If 1, MongoDB will not execute queries that require a collection scan and will return an error.
  17. .co.il www. Thank you! We invite you to join Operations

    UK Facebook group on dvps.me/OpsUK www.devopspro.co.uk