Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reliable High-Performance HTTP Infrastructure w...

Reliable High-Performance HTTP Infrastructure with nginx and Lua

We recently replaced a proprietary API management solution with an in-house implementation built with nginx and Lua that is more robust, higher performance, and has greater visibility. Learn about our development process and the overall architecture that allowed us to write high-level code while enjoying native code performance, and how we leveraged other open source tools like Vagrant, Ansible, and OpenStack to build an automation-rich delivery pipeline. We will also take an in-depth look at our capacity management approach that differs from the rate limiting concept prevalent in the API community.

Sean Cribbs

April 11, 2016
Tweet

More Decks by Sean Cribbs

Other Decks in Technology

Transcript

  1. Reliable High-Performance HTTP Infrastructure with nginx and Lua Sean Cribbs

    Senior Principal Engineer, Comcast Cable @seancribbs
  2. CodeBig 1 API Consumer CDN • traffic shaping • caching

    • access control • rate limiting Vendor APIM
  3. CodeBig 1 API Consumer CDN • traffic shaping • caching

    • access control • rate limiting Vendor APIM Internet Comcast LB • DNS RR • VIP
  4. CodeBig 1 API Consumer CDN • traffic shaping • caching

    • access control • rate limiting Vendor APIM • DMZ intermediary • path-host mapping iAuth Internet Comcast LB • DNS RR • VIP
  5. CodeBig 1 API Consumer CDN • traffic shaping • caching

    • access control • rate limiting Vendor APIM • DMZ intermediary • path-host mapping iAuth Internet Comcast Origin APIs LB • DNS RR • VIP
  6. Lua

  7. nginx+Lua extension points init access header_filter log init_worker rewrite body_filter

    balancer ssl_certificate set content _by_lua   _by_lua_file     _by_lua_block +
  8. CodeBig Request Phases init access header_filter log Load code and

    configuration Authenticate Rate-limit Tweak request request flow
  9. CodeBig Request Phases init access header_filter log Load code and

    configuration Authenticate Rate-limit Tweak request Tweak response request flow
  10. CodeBig Request Phases init access header_filter log Load code and

    configuration Authenticate Rate-limit Tweak request Tweak response Clean up request flow
  11. local  setmetatable  =  setmetatable   local  _M  =  {}  

    function  _M:new(ctx,  conf)          local  o  =  {                  _ctx  =  ctx,                  _conf  =  conf          }          o.super  =  self          setmetatable(o,  self)          self.__index  =  self          return  o   end   function  _M:access()          return  true   end   function  _M:post_access()          -­‐-­‐  nop   end   function  _M:header_filter()          -­‐-­‐  nop   end   function  _M:log()          -­‐-­‐  nop   end   return  _M
  12. for _, name in ipairs(conf.plugins) do -- load plugin by

    fully qualified name local plugin = require(name):new(ctx, conf) -- exit immediately upon first rejection local is_ok, err = plugin:access() if not is_ok then ngx.status = err.code ngx_say(err.error) ngx.var.access_error = err.error return ngx_exit(ngx.HTTP_OK) end insert(plugins, plugin) end for _, plugin in ipairs(plugins) do plugin:post_access() end
  13. function _M.header_filter() local plugins = ngx.ctx.plugins or {} for _,

    plugin in ipairs(plugins) do plugin:header_filter() end end
  14. # nginx.conf lua_package_path '/usr/share/?/init.lua;/usr/share/?.lua;;'; lua_shared_dict memory 50M; init_by_lua_block { codebig

    = require("codebig") codebig.init(“/path/to/configs“) }; # vhost.conf location / { access_by_lua 'return codebig.access("somehost")'; header_filter_by_lua 'return codebig.header_filter()'; log_by_lua 'return codebig.log()'; }
  15. Cross-Datacenter DC1 DC2 DC3 vod vod acct acct entry-­‐vip-­‐dc1.  A

                 10.1.0.1 vod-­‐dc1.              CNAME      entry-­‐vip-­‐dc1. vod.                      CNAME      vod-­‐dc1. entry-­‐vip-­‐dc2.  A              10.2.0.1 entry-­‐vip-­‐dc3.  A              10.3.0.1 vod-­‐dc2.              CNAME      entry-­‐vip-­‐dc2. VIP VIP VIP vod-­‐dc1-­‐fo.        CNAME      entry-­‐vip-­‐dc1.
  16. client origin APIM 2 req/s 1s N = XR =

    2 req/s x 1s = 2 concurrent
  17. client origin APIM 2 req/s 10s N = XR =

    2 req/s x 10s = 20 concurrent
  18. client origin APIM 2 req/s 10s N = XR =

    2 req/s x 10s = 20 concurrent
  19. Concurrent Request Limiting lua_shared_dict            

     memory    50M;   access_by_lua          …      +1   log_by_lua                …      -­‐1
  20. Configs in VCS Playbooks in VCS config templates API configs

    vault
 (keys) vip.conf vhost.lua vhost.conf vhost.conf vhost.json nginx.conf ssh
  21. Impact index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =  request_time

     -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d)
  22. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =

     request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d)
  23. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =

     request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d) 99th
  24. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =

     request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d) 99th max
  25. Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline

    Concurrent request limiting Operational flexibility