Reliable High-Performance HTTP Infrastructure with nginx and Lua

We recently replaced a proprietary API management solution with an in-house implementation built with nginx and Lua that is more robust, higher performance, and has greater visibility. Learn about our development process and the overall architecture that allowed us to write high-level code while enjoying native code performance, and how we leveraged other open source tools like Vagrant, Ansible, and OpenStack to build an automation-rich delivery pipeline. We will also take an in-depth look at our capacity management approach that differs from the rate limiting concept prevalent in the API community.

Transcript

1. Reliable High-Performance HTTP Infrastructure with nginx and Lua Sean Cribbs Senior Principal Engineer, Comcast Cable @seancribbs
2. Background
4. Consumer
5. Internal Consumer
6. Partner Internal Consumer
7. API Management
8. API Management access control capacity management
9. CodeBig 1 API Consumer
10. CodeBig 1 API Consumer CDN • traffic shaping • caching
11. CodeBig 1 API Consumer CDN • traffic shaping • caching • access control • rate limiting Vendor APIM
12. CodeBig 1 API Consumer CDN • traffic shaping • caching • access control • rate limiting Vendor APIM Internet Comcast LB • DNS RR • VIP
13. CodeBig 1 API Consumer CDN • traffic shaping • caching • access control • rate limiting Vendor APIM • DMZ intermediary • path-host mapping iAuth Internet Comcast LB • DNS RR • VIP
14. CodeBig 1 API Consumer CDN • traffic shaping • caching • access control • rate limiting Vendor APIM • DMZ intermediary • path-host mapping iAuth Internet Comcast Origin APIs LB • DNS RR • VIP
15. Challenges
16. Challenges visibility
17. Challenges visibility responsibility
18. Challenges visibility responsibility scope
19. Challenges visibility responsibility scope latency
20. Challenges visibility responsibility scope latency security
21. CodeBig 2
22. CodeBig 2 simplify architecture
23. CodeBig 2 simplify architecture increase visibility
24. CodeBig 2 simplify architecture increase visibility use open-source tools
25. Architecture
26. custom  logic HTTP Proxy
27. Lua
28. nginx+Lua extension points init access header_filter log init_worker rewrite body_filter balancer ssl_certificate set content _by_lua   _by_lua_file     _by_lua_block +
29. CodeBig Request Phases init access header_filter log
30. CodeBig Request Phases init access header_filter log request flow
31. CodeBig Request Phases init access header_filter log Load code and configuration request flow
32. CodeBig Request Phases init access header_filter log Load code and configuration Authenticate Rate-limit Tweak request request flow
33. CodeBig Request Phases init access header_filter log Load code and configuration Authenticate Rate-limit Tweak request Tweak response request flow
34. CodeBig Request Phases init access header_filter log Load code and configuration Authenticate Rate-limit Tweak request Tweak response Clean up request flow
35. local  setmetatable  =  setmetatable   local  _M  =  {}   function  _M:new(ctx,  conf)          local  o  =  {                  _ctx  =  ctx,                  _conf  =  conf          }          o.super  =  self          setmetatable(o,  self)          self.__index  =  self          return  o   end   function  _M:access()          return  true   end   function  _M:post_access()          -­‐-­‐  nop   end   function  _M:header_filter()          -­‐-­‐  nop   end   function  _M:log()          -­‐-­‐  nop   end   return  _M
36. for _, name in ipairs(conf.plugins) do -- load plugin by fully qualified name local plugin = require(name):new(ctx, conf) -- exit immediately upon first rejection local is_ok, err = plugin:access() if not is_ok then ngx.status = err.code ngx_say(err.error) ngx.var.access_error = err.error return ngx_exit(ngx.HTTP_OK) end insert(plugins, plugin) end for _, plugin in ipairs(plugins) do plugin:post_access() end
37. function _M.header_filter() local plugins = ngx.ctx.plugins or {} for _, plugin in ipairs(plugins) do plugin:header_filter() end end
38. # nginx.conf lua_package_path '/usr/share/?/init.lua;/usr/share/?.lua;;'; lua_shared_dict memory 50M; init_by_lua_block { codebig = require("codebig") codebig.init(“/path/to/configs“) }; # vhost.conf location / { access_by_lua 'return codebig.access("somehost")'; header_filter_by_lua 'return codebig.header_filter()'; log_by_lua 'return codebig.log()'; }
39. Lua ~ 3K LoC!!
40. Intra-Datacenter VIP
41. Intra-Datacenter VIP haproxy haproxy …
42. Intra-Datacenter VIP … haproxy haproxy …
43. Intra-Datacenter VIP Origin APIs … haproxy haproxy …
44. Cross-Datacenter DC1 DC2 DC3 vod vod acct acct entry-­‐vip-­‐dc1.  A              10.1.0.1 vod-­‐dc1.              CNAME      entry-­‐vip-­‐dc1. vod.                      CNAME      vod-­‐dc1. entry-­‐vip-­‐dc2.  A              10.2.0.1 entry-­‐vip-­‐dc3.  A              10.3.0.1 vod-­‐dc2.              CNAME      entry-­‐vip-­‐dc2. VIP VIP VIP vod-­‐dc1-­‐fo.        CNAME      entry-­‐vip-­‐dc1.
45. Capacity Management
46. N = XR
47. N = XR # concurrent requests
48. N = XR # concurrent requests transaction rate
49. N = XR # concurrent requests transaction rate response time
50. N = XR # concurrent requests transaction rate response time Little’s Law
51. client origin APIM 2 req/s 1s N = XR = 2 req/s x 1s = 2 concurrent
52. client origin APIM 2 req/s 10s N = XR = 2 req/s x 10s = 20 concurrent
53. client origin APIM 2 req/s 10s N = XR = 2 req/s x 10s = 20 concurrent
54. Concurrent Request Limiting lua_shared_dict              memory    50M;   access_by_lua          …      +1   log_by_lua                …      -­‐1
55. Deployment
57. Configs in VCS Playbooks in VCS config templates API configs vault
 (keys) vip.conf vhost.lua vhost.conf vhost.conf vhost.json nginx.conf ssh
58. Results
59. Performance
60. switch Performance
61. switch mean 99th Performance
62. switch ~10x mean 99th Performance
63. Stability
64. switch Stability
65. Impact index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =  request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d)
66. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =  request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d)
67. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =  request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d) 99th
68. Impact seconds index=codebig  host=*.cimops.net  source="/var/log/nginx/access.log"  |   eval  d  =  request_time  -­‐  upstream_response_time  |  
 timechart  span=1m  perc99(d)  max(d) 99th max
69. Successes
70. Successes great performance improvements
71. Successes great performance improvements hosting ~400 endpoints
72. Successes great performance improvements hosting ~400 endpoints > 367MM requests a day
73. Successes great performance improvements hosting ~400 endpoints > 367MM requests a day prevented upstream downtime
74. Challenges
75. Challenges 3rd-party Lua ecosystem
76. Challenges 3rd-party Lua ecosystem not self-service yet
77. Challenges 3rd-party Lua ecosystem not self-service yet configuration file size
78. Challenges 3rd-party Lua ecosystem not self-service yet configuration file size kernel tuning
79. Challenges 3rd-party Lua ecosystem not self-service yet configuration file size kernel tuning owning availability
80. Conclusion
81. Conclusion NGINX + Lua for HTTP middleware
82. Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline
83. Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline Concurrent request limiting
84. Conclusion NGINX + Lua for HTTP middleware Automated deployment pipeline Concurrent request limiting Operational flexibility
85. Thanks