Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Load Balancing a Dynamic Infrastructure with NG...

Kevin Reedy
October 22, 2014

Load Balancing a Dynamic Infrastructure with NGINX, Chef, and Consul

Load balancing and routing traffic to a single application is easy, but sending traffic to a always-changing number of applications is quite a challenge. In the last year, Belly has migrated from a monolithic Rails app to a service-oriented architecture with over fifty applications.

In this session, we’ll talk about how to use Chef and Consul to dynamically configure NGINX to route and load balance traffic across applications. These applications can be deployed in a variety of ways, including Chef, Docker, and S3. Additionally, we’ll discuss the importance of service discovery and monitoring inter-service traffic in a service-oriented architecture.

Kevin Reedy

October 22, 2014
Tweet

More Decks by Kevin Reedy

Other Decks in Technology

Transcript

  1. Load Balancing a Dynamic Infrastructure @kevinreedy Overview •Belly’s infrastructure stack

    over time •Using Chef to configure Nginx •Using data from Consul’s service discovery to configure Nginx •Pro Tips (time permitting)
  2. Load Balancing a Dynamic Infrastructure @kevinreedy Disclaimer •Nothing in this

    talk is specific to a single tool •Chef, Puppet, Ansible, Salt, CFEngine, etc. are all great! •Consul, Etcd, Zookeeper, etc. are all great! •Use what works for you !
  3. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v0 the

    dark age •Monolithic Ruby on Rails Application •MySQL, MongoDB, Memcache •Deployed to Heroku
  4. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v1 a

    new hope •Monolithic Ruby on Rails Application •MySQL, MongoDB, Memcache •Deployed using Capistrano •~6 Servers on AWS EC2 •AWS Elastic Load Balancer
  5. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA

    •Separate the Homepage from the API •Simply separate www.bellycard.com from api.bellycard.com
  6. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA

    •Some routes on www.bellycard.com still need to route to our API servers • www.bellycard.com/api • www.bellycard.com/api-assets • www.bellycard.com/eo • www.bellycard.com/ec • www.bellycard.com/facebook/callback
  7. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA

    •AWS Elastic Load Balancer no longer gave us enough flexibility •Nginx to the rescue! •Multiple Upstreams •Rewrite Rules ! !
  8. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA

    upstream api { least_conn; server 10.0.0.101:8080; server 10.0.0.102:8080; server 10.0.0.103:8080; server 10.0.0.104:8080; server 10.0.0.105:8080; server 10.0.0.106:8080; } ! upstream homepage { least_conn; server 10.0.0.201:8080; server 10.0.0.202:8080; }
  9. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA

    server { listen 80; server_name api.bellycard.com; ! location / { proxy_pass http://api; } } ! server { listen 80; server_name www.bellycard.com; ! location / { proxy_pass http://homepage; } ! location /api { proxy_pass http://api; } }
  10. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA

    •This worked great! •So we deployed a third service •…and a fourth •…and a fifth… that was Node.js ! !
  11. Load Balancing a Dynamic Infrastructure @kevinreedy Files package 'nginx' !

    service 'nginx' do action [:start, :enable] end ! file '/usr/share/nginx/html/index.html' do content '<html> <body> <h1>Hello World!</h1> </body> </html>' end !
  12. Load Balancing a Dynamic Infrastructure @kevinreedy Templates package 'nginx' !

    service 'nginx' do action [:start, :enable] end ! g = 'Hello' w = 'Kevin' ! template '/usr/share/nginx/html/index.html' do source 'index.html.erb' variables ( 'greeting' => g, 'who' => w ) end
  13. Load Balancing a Dynamic Infrastructure @kevinreedy Cookbooks •Collection of Recipes,

    Files, Templates •And Resources, Providers, and Libraries •Tons of cookbooks at https://supermarket.getchef.com/ • nginx • mysql • java • apt/yum • ssh_known_hosts !
  14. Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example •Install Nginx

    •Configure Nginx using Chef •Use Chef to discover where our application is running •Configure Nginx to route traffic to our application
  15. Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example # Tune

    Nginx node.set['nginx']['worker_connections'] = 8192 node.set['nginx']['worker_rlimit_nofile'] = 32768 node.set['nginx']['event'] = 'epoll' node.set['nginx']['client_max_body_size'] = '4m' ! # Install Nginx include_recipe 'nginx::repo' include_recipe 'nginx' ! # Find application nodes nodes = search(:node, 'recipe:belly-api') ! # Generate site config template "#{node['nginx']['dir']}/sites-enabled/api" do source 'api.erb' variables( 'servers' => nodes ) notifies :reload, 'service[nginx]' end
  16. Load Balancing a Dynamic Infrastructure @kevinreedy Nginx Example upstream api

    { least_conn; <% @servers.each do |server| %> server <%= server['ipaddress'] %>:8080; # <%= server['name'] %> <% end %> } ! server { listen 80; server_name api.bellycard.com; ! location / { proxy_pass http://api; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Proto https; } }
  17. Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example #

    Set staging and production specific attributes if node.chef_environment == 'production' node.set['belly']['lb']['domain'] = 'bellycard.com' node.set['belly']['lb']['cert_name'] = 'wildcard_bellycard_com' node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-assets/sitemap.xml.gz' elsif node.chef_environment == 'staging' node.set['belly']['lb']['domain'] = 'bellystaging.com' node.set['belly']['lb']['cert_name'] = 'wildcard_bellystaging_com' node.set['belly']['lb']['sitemap_url'] = 'https://s3.amazonaws.com/homepage-staging-assets/sitemap.xml.gz' end ! # Install Nginx include_recipe 'nginx::repo' include_recipe 'nginx' ! # Configure Nginx include_recipe 'belly-lb::_tuning' include_recipe 'belly-lb::_logging' include_recipe 'belly-lb::_security' include_recipe 'belly-lb::_ssl-certs' ! # Add Sites include_recipe 'belly-lb::_default-site' include_recipe 'belly-lb::_api' include_recipe 'belly-lb::_platform' include_recipe 'belly-lb::_ssl-termination' include_recipe 'belly-lb::_other-sites' ! # Post-install Configuration include_recipe 'belly-lb::_route53' include_recipe 'belly-lb::_monitoring' include_recipe 'belly-lb::_ngxtop' ! # Misc # Set chef interval to 5 minutes (or keep at lessor value) if node['chef_client'] && node['chef_client']['interval'] node.set['chef_client']['interval'] = [Integer(node['chef_client']['interval']), 300].min node.set['chef_client']['splay'] = [Integer(node['chef_client']['splay']), 30].min end
  18. Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example if

    Chef::Config[:solo] Chef::Log.warn('This recipe uses search. Chef Solo does not support search.') else services = Chef::DataBagItem.load('belly-platform', 'portmap') services['services'].each do |service_name, service_attr| servers = Array.new ! search_keys = { 'name' => ['name'], 'ipaddress' => ['ipaddress'] } ! partial_search(:node, "recipe:belly-service\\:\\:#{ service_name } AND chef_environment:#{ node.chef_environment }", keys: search_keys).each do |result| servers.push( 'host' => result['ipaddress'], 'port' => service_attr['port'], 'comment' => result['name'] ) end ! unless servers.empty? # Generate site config template "#{ node['nginx']['dir'] }/sites-available/#{ service_name }" do source "simple-site.erb" mode 0444 owner "root" group "root" variables( 'service_name' => service_name, 'service_comment' => service_attr['comment'], 'domain_name' => "#{ service_name }.#{ node['belly']['lb']['domain'] }" 'servers' => servers, 'ssl_cert' => "#{ node['belly']['lb']['cert_name'] }.crt", 'ssl_key' => "#{ node['belly']['lb']['cert_name'] }.key" ) notifies :reload, 'service[nginx]' end ! # Enable site nginx_site service_name end end end
  19. Load Balancing a Dynamic Infrastructure @kevinreedy Real World Example #

    <%= @service_comment %> nginx config # Generated by Chef for <%= node.name %> # Changes will be overwritten ! upstream <%= @service_name %> { least_conn; <% @servers.each do |server| %> server <%= server['host'] %>:<%= server['port'] %> max_fails=3 fail_timeout=60 weight=1; # <%= server['comment'] %> <% end %> } ! server { listen 443 ssl; server_name <%= @domain_name %>; ssl on; ssl_certificate /etc/nginx/ssl/<%= @ssl_cert %>; ssl_certificate_key /etc/nginx/ssl/<%= @ssl_key %>; ! # enable session resumption to improve https performance # http://vincent.bernat.im/en/blog/2011-ssl-session-reuse-rfc5077.html ssl_session_cache shared:SSL:10m; ssl_session_timeout 10m; ! # enables server-side protection from BEAST attacks # http://blog.ivanristic.com/2013/09/is-beast-still-a-threat.html ssl_prefer_server_ciphers on; ! ssl_protocols TLSv1 TLSv1.1 TLSv1.2; ! # ciphers chosen for forward secrecy and compatibility # http://blog.ivanristic.com/2013/08/configuring-apache-nginx-and-openssl-for-forward-secrecy.html ssl_ciphers 'EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH EDH+aRSA RC4 AES128-SHA !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS'; ! location / { proxy_pass http://<%= @service_name %>; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-Proto https; } }
  20. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v2 SOA

    •We deployed around 40 ruby apps to our servers using Chef •All traffic to all apps was routed by Nginx, dynamically configured with Chef ! !
  21. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SOA

    All The Things •We also had around 10 Frontend Javascript applications deployed onto S3 / Cloudfront •And another 5 on Heroku ! !
  22. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SOA

    All The Things •We also had around 10 Frontend Javascript applications deployed onto S3 / Cloudfront •And another 4 on Heroku ! !
  23. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v3 SSL

    All The Things •All traffic to bellycard.com and its subdomains now passes through our load balancers •Assets are served on a cloudfront domain for performance !
  24. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v4 Docker

    •Greatly simplifies deploying applications •Separation of responsibility from development and operations •Allows developers to easily spin up dependent services on their own laptop •Speeds up deployment !
  25. Load Balancing a Dynamic Infrastructure @kevinreedy Belly’s Stack v4 Docker

    •Deploying Containers •CoreOS / Fleet •Deis •Mesos •Kubernetes •Flynn ! !
  26. Load Balancing a Dynamic Infrastructure @kevinreedy Deployment Strategies •Zero downtime

    deploys •Deploy in place •Blue / Green •Canary ! ! ! !
  27. Load Balancing a Dynamic Infrastructure @kevinreedy Consul •Distributed key value

    store •Service discovery as a first class citizen •Distributed health checks •Multiple Datacenters •Long polling for instant notification of changes •Scale out agents !
  28. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Released yesterday

    •Subscribes to changes in: •services •health check •key/value store !
  29. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Released yesterday

    •Subscribes to changes •Generates files from templates •Optionally calls a command on change !
  30. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template $ consul-template

    -consul consul-prod.example.com:8500 \ -template "/etc/consul-templates/api:/etc/nginx/sites- enabled/api:service nginx reload"
  31. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template upstream api

    { least_conn; {{range service "api"}}server {{.Address}}:{{.Port}}; {{end}} } ! upstream homepage { least_conn; {{range service "homepage"}}server {{.Address}}:{{.Port}}; {{end}} } ! server { listen 80; server_name www.bellycard.com; ! location / { proxy_pass http://homepage; } ! location /api { proxy_pass http://api; } }
  32. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template •Convergence on

    service changes is fast •Configuration Management is still needed to generate templates !
  33. Load Balancing a Dynamic Infrastructure @kevinreedy Consul Template services =

    ( JSON.parse( open("http://#{consul_host}:8500/v1/catalog/services") .read ) || {} ).keys ! # Generate nginx template for consul template template '/etc/consul-templates/services' do source 'consul-upstream.erb' variables( services: services, ssl_cert: "#{ node['belly']['lb']['cert_name'] }.crt", ssl_key: "#{ node['belly']['lb']['cert_name'] }.key" ) notifies :reload, 'service[consul-haproxy]', :delayed end ! # Configure Consul Template template '/etc/consul-template.conf' do source 'consul-conf.erb' variables( host: "#{ consul_host }:8500" source: '/etc/consul-templates/services' destination: '/etc/nginx/sites-enabled/services' reload_command: 'service nginx reload' ) notifies :reload, 'service[consul-template]', :delayed end
  34. Load Balancing a Dynamic Infrastructure @kevinreedy Chef Partial Search •Chef

    search makes a copy of the entire node, including all attributes •Partial search allows you to return only the keys you care about
  35. Load Balancing a Dynamic Infrastructure @kevinreedy Chef Partial Search #

    search nodes = search(:node, 'recipe:belly-api') ! # partial search search_keys = { 'name' => ['name'], 'ipaddress' => ['ipaddress'] } ! nodes = partial_search(:node, 'recipe:belly-api', :keys => search_keys)
  36. Load Balancing a Dynamic Infrastructure @kevinreedy DNS •Nginx resolves DNS

    in upstream blocks at config reload •Don’t do this if DNS resolution changes regularly • s3.amazonaws.com • *.herokuapp.com •Resolve in Chef and trigger Nginx reload on changes
  37. Load Balancing a Dynamic Infrastructure @kevinreedy Avoid Stale Config •We

    need a way to clean up stale sites •add :delete functions to your Chef cookbooks •render all site configs to a single file
  38. Load Balancing a Dynamic Infrastructure @kevinreedy Thanks! •Belly is hiring!

    •Slides will be posted •Keep in touch! •Twitter @kevinreedy •[email protected] •https://tech.bellycard.com !