Practical nginx module development〜C and Lua〜

1. Practical nginx module development ʙC and Luaʙ Tatsuhiko Kubo@cubicdaiya YAPC::Asia TOKYO 2015
2. @cubicdaiya / Tatsuhiko Kubo • Software Engineer @ Mercari, Inc. • Infrastructure Engineering • Skills • C, Go, Lua, nginx, … • OSS developer • ngx_small_light, ngx_dynamic_upstream, nginx-build, slackboard, cachectl, gaurun, …
3. Activities on Github
4. Other Activities IUUQNP[BJDGNQPTUOHJOY
7. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
8. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
9. nginx • Open Source Software • HTTP server for various purpose • Static content delivery • Reverse proxy ( HTTP / FastCGI / uWSGI / … ) • Load balancer • Cache • TLS Termination / SPDY Gateway • etc… • Mail proxy ( SMTP / POP3 / IMAP ) • TCP proxy
10. nginx’s architecture • event-driven • non-blocking • asynchronous • single-threaded ( multi-threaded partially) • multi-processes ( master / wokers / …) • modular
11. nginx’s architecture • event-driven • non-blocking • asynchronous • single-threaded ( multi-threaded partially) • multi-processes ( master / wokers / …) • modular
12. nginx is module-oriented • most features are designed as module • ngx_http_core_module • ngx_http_access_module • ngx_http_log_module • ngx_http_proxy_module • ngx_http_upstream_module • etc…
13. 3rd party module for nginx • nginx-user can develop an original module • It is not required to modify source code of nginx • embed original module with ‘./configure’ $ cd nginx-1.9.x $ ./configure —-add-module=${ngx_module_path}
14. Examples
15. Example • ngx_http_empty_gif_module (Official) • return 1x1 GIF location /gif { empty_gif; } $ curl -I http://127.0.0.1/gif HTTP/1.1 200 OK Content-Length: 43 Content-Type: image/gif ... $
16. Example • ngx_small_light • Dynamic Image Transformation for nginx small_light on; location ~ small_light[^/]*/(.+)$ { set $file $1; rewrite ^ /$file; }
17. ngx_small_light • Dynamic Image Transformation for nginx • https://github.com/cubicdaiya/ngx_small_light • Provides various • Image Manipulations ( Resize / Crop / Composite / …) • Image Processing Libraries (Image Magick / Imlib2 / GD) • Image Formats (JPEG / GIF / PNG / WEBP)
18. Transform image ▪URI for resize /small_light(dw=300,dh=200)/img/image.jpg →
19. Transform image on S3 proxy from nginx to S3 small_light on; location ~ small_light[^/]*/(.+)$ { set $file $1; rewrite ^ /$file; } location /img { proxy_pass http://<s3_bucket_url>; }
20. Adoption case https://github.com/wantedly/nginx-image-server
21. Adoption case https://speakerdeck.com/yano3/dynamic-image-transformation-server-okara-beta
22. Example • ngx_dynamic_upstream • Dynamic Upstream for nginx location / { allow xxx.xxx.xxx.xxx deny all; dynamic_upstream; }
23. ngx_dynamic_upstream • Dynamic Upstream for nginx • https://github.com/cubicdaiya/ngx_dynamic_upstream • Provides HTTP APIs for manipulating upstream for nginx • It is useful for • zero-down-time deployment • blue green deployment
24. Zero-down-time deployment on
25. ɾɾɾ ɾɾɾ "1*
    "QBDIF NPE@QIQ OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN HTTPS or SPDY OHJOY
    
    OHY@EZOBNJD@VQTUSFBN HTTP Mercari frontend "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ ɾɾɾ ɾɾɾ
26. "1*
    "QBDIF NPE@QIQ Previous deployment "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU rsync rsync rsync
27. Pros / Cons • Pros • Very Very Very simple • Cons • many 500 error happens at deployment!!!!!
28. "1*
    "QBDIF NPE@QIQ Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ
29. "1*
    "QBDIF NPE@QIQ Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
30. "1*
    "QBDIF NPE@QIQ Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
31. Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1*
    "QBDIF NPE@QIQ
32. "1*
    "QBDIF NPE@QIQ Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
33. Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1*
    "QBDIF NPE@QIQ
34. "1*
    "QBDIF NPE@QIQ Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
35. Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ
36. "1*
    "QBDIF NPE@QIQ Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream
37. Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ
38. "1*
    "QBDIF NPE@QIQ Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ ※rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU
39. Current deployment ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream down "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ
40. "1*
    "QBDIF NPE@QIQ Current deployment "1*
    "QBDIF NPE@QIQ ChatOps with Slack yes OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN "1*
    "QBDIF NPE@QIQ rsync ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream EFQMPZCPU
41. Current deployment ChatOps with Slack yes EFQMPZCPU OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN OHJOY
    
    OHY@EZOBNJD@VQTUSFBN ※ rsync ——rsync-path=mercari_app_rsync(↓) #!/bin/sh mercari_app_ctl down # deactivate server on nginx upstream rsync $* # deploy mercari_app_ctl up # activate server on nginx upstream up "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ "1*
    "QBDIF NPE@QIQ
42. Hello, World!
43. Hello, World! • ngx_http_hello_world • https://github.com/cubicdaiya/ngx_http_hello_world location /hello { hello_world; } $ curl http://127.0.0.1/hello Hello, World! $
44. Layout of ngx_http_hello_world ngx_http_hello_world |———... |———config |———ngx_http_hello_world_module.c └──-...
45. config • module configuration file ngx_addon_name=ngx_http_hello_world_module HTTP_MODULES="$HTTP_MODULES \ ngx_http_hello_world_module” NGX_ADDON_SRCS="$NGX_ADDON_SRCS \ $ngx_addon_dir/ngx_http_hello_world_module.c”
46. Hello, World! • ngx_http_hello_world • return “Hello, World!” location /hello { hello_world; } $ curl http://127.0.0.1/hello Hello, World! $
47. Hello, World! • ngx_http_hello_world • return “Hello, World!” location /hello { hello_world; } $ curl http://127.0.0.1/hello Hello, World! $
48. Hello, World! • ngx_http_hello_world • return “Hello, World!” location /hello { hello_world; # original directive } $ curl http://127.0.0.1/hello Hello, World! $
49. Directive declaration struct ngx_command_s { ngx_str_t name; ngx_uint_t type; char *(*set)(ngx_conf_t *cf, ngx_command_t *cmd, void *conf); ngx_uint_t conf; ngx_uint_t offset; void *post; };
50. Directive definition(hello_world) static ngx_command_t ngx_http_hello_world_commands[] = { { ngx_string("hello_world"), NGX_HTTP_LOC_CONF|NGX_CONF_NOARGS, ngx_http_hello_world, 0, 0, NULL }, ngx_null_command };
51. Module context typedef struct { ngx_int_t (*preconfiguration)(ngx_conf_t *cf); ngx_int_t (*postconfiguration)(ngx_conf_t *cf); void *(*create_main_conf)(ngx_conf_t *cf); char *(*init_main_conf)(ngx_conf_t *cf, void *conf); void *(*create_srv_conf)(ngx_conf_t *cf); char *(*merge_srv_conf)(ngx_conf_t *cf, void *prev, void *conf); void *(*create_loc_conf)(ngx_conf_t *cf); char *(*merge_loc_conf)(ngx_conf_t *cf, void *prev, void *conf); } ngx_http_module_t;
52. Module context static ngx_http_module_t ngx_http_hello_world_module_ctx = { NULL, /* preconfiguration */ NULL, /* postconfiguration */ NULL, /* create main configuration */ NULL, /* init main configuration */ NULL, /* create server configuration */ NULL, /* merge server configuration */ NULL, /* create location configuration */ NULL /* merge location configuration */ }; In this case, all is NULL.
53. Module definition ngx_module_t ngx_http_hello_world_module = { NGX_MODULE_V1, &ngx_http_hello_world_module_ctx, /* module context */ ngx_http_hello_world_commands, /* module directives */ NGX_HTTP_MODULE, /* module type */ NULL, /* init master */ NULL, /* init module */ NULL, /* init process */ NULL, /* init thread */ NULL, /* exit thread */ NULL, /* exit process */ NULL, /* exit master */ NGX_MODULE_V1_PADDING };
54. Hook original handler static char *ngx_http_hello_world(ngx_conf_t *cf, ngx_command_t *cmd, void *conf) { ngx_http_core_loc_conf_t *clcf; clcf = ngx_http_conf_get_module_loc_conf(cf, ngx_http_core_module); clcf->handler = ngx_http_hello_world_handler; return NGX_CONF_OK; }
55. ngx_http_hello_world_handler() #define NGX_HTTP_HELLO_WORLD “Hello, World!” static ngx_int_t ngx_http_hello_world_handler(ngx_http_request_t *r) { ngx_int_t rc; ngx_chain_t out; ngx_buf_t *b; ngx_str_t body = ngx_string(NGX_HTTP_HELLO_WORLD); if (r->method != NGX_HTTP_GET && r->method != NGX_HTTP_HEAD) { return NGX_HTTP_NOT_ALLOWED; } if (r->headers_in.if_modified_since) { return NGX_HTTP_NOT_MODIFIED; } b = ngx_pcalloc(r->pool, sizeof(ngx_buf_t)); if (b == NULL) { return NGX_HTTP_INTERNAL_SERVER_ERROR; } b->pos = body.data; b->last = b->pos + body.len; b->memory = 1; b->last_buf = 1; out.buf = b; out.next = NULL; ngx_str_set(&r->headers_out.content_type, "text/plain"); r->headers_out.status = NGX_HTTP_OK; r->headers_out.content_length_n = body.len; rc = ngx_http_send_header(r); if (rc == NGX_ERROR || rc > NGX_OK || r->header_only) { return rc; }
56. Embed ngx_http_hello_world into nginx $ ./confgiure \ —-add-module=../ngx_http_hello_world $ make $ sudo make install
57. Hook details…
58. Request processing phases in nginx TFSWFS
    SFXSJUF QPTUBDDFTT MPH pOE
    DPOpH SFXSJUF QPTU
    SFXSJUF QSFBDDFTT USZpMFT DPOUFOU Request BDDFTT try_files processing access-control response body generation logging rewrite in server find location rewrite in location
59. Phase definitions typedef enum { NGX_HTTP_POST_READ_PHASE = 0, NGX_HTTP_SERVER_REWRITE_PHASE, NGX_HTTP_FIND_CONFIG_PHASE, NGX_HTTP_REWRITE_PHASE, NGX_HTTP_POST_REWRITE_PHASE, NGX_HTTP_PREACCESS_PHASE, NGX_HTTP_ACCESS_PHASE, NGX_HTTP_POST_ACCESS_PHASE, NGX_HTTP_TRY_FILES_PHASE, NGX_HTTP_CONTENT_PHASE, NGX_HTTP_LOG_PHASE } ngx_http_phases;
60. Hook handler to specified phases ngx_http_handler_pt *h; ngx_http_core_main_conf_t *cmcf; cmcf = ngx_http_conf_get_module_main_conf(cf, ngx_http_core_module); h = ngx_array_push(&cmcf->phases[NGX_HTTP_ACCESS_PHASE].handlers); if (h == NULL) { return NGX_ERROR; } *h = ngx_http_xxx_handler;
61. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
62. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
63. nginx API • memory-pool • string • array • hash-table • time • list • regular-expression • temporary-file • shared-memory • etc…
64. essence of nginx API programming • understand nginx architecture • event-driven • non-blocking • asynchronous • single-threaded • multi-process
65. essence of nginx API programming • memory management strategies • memory-pool • slab ( for shared-memory) • string with length • string is not always NULL-terminated
66. essence of nginx API programming • structs for HTTP • ngx_http_request_t • ngx_http_headers_in • ngx_http_headers_out • etc… ( See nginx/src/http/ngx_http_request.h )
67. Move to terminal…
68. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
69. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
70. Test::Nginx https://github.com/openresty/test-nginx
71. Test::Nginx • Testing framework for nginx • https://github.com/openresty/test-nginx • We are able to test nginx’s behavior with CLI • Install from CPAN $ cpanm Test::Nginx
72. test nginx’s behavior with Test::Nginx use Test::Nginx::Socket; plan tests => repeat_each() * 2 * blocks(); run_tests(); __DATA__ === TEST 1: hello world --- config location /hello { return 200 "Hello, World!\n"; } --- request GET /hello --- response_body Hello, World! --- error_code: 200
73. test nginx’s behavior with Test::Nginx use Test::Nginx::Socket; plan tests => repeat_each() * 2 * blocks(); run_tests(); __DATA__ === TEST 1: hello world --- config location /hello { return 200 "Hello, World!\n"; } --- request GET /hello --- response_body Hello, World! --- error_code: 200 run tests
74. test nginx’s behavior with Test::Nginx use Test::Nginx::Socket; plan tests => repeat_each() * 2 * blocks(); run_tests(); __DATA__ === TEST 1: hello world --- config location /hello { return 200 "Hello, World!\n"; } --- request GET /hello --- response_body Hello, World! --- error_code: 200 run tests define behavior ɾnginx.conf ɾHTTP request ɾHTTP response
75. By the way, • C programming is difficult and complicated • productivity is not high If scripting language is available on nginx, productivity would improve dramatically.
76. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
77. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
78. ngx_lua https://github.com/openresty/lua-nginx-moduleʙ
79. ngx_lua • embed Lua/LuaJIT into nginx • Lua scripting in nginx.conf • Hooking request processing phase with Lua • wrap nginx API with Lua • non-blocking APIs (e.g. ngx.location.capture())
80. • set_by_lua / set_by_lua_file • rewrite_by_lua / rewrite_by_lua_file • access_by_lua / access_by_lua_file • content_by_lua / content_by_lua_file • log_by_lua / log_by_lua_file Hook directives for request processing phases
81. • set_by_lua / set_by_lua_file • rewrite_by_lua / rewrite_by_lua_file • access_by_lua / access_by_lua_file • content_by_lua / content_by_lua_file • log_by_lua / log_by_lua_file run Lua script-string Hook directives for request processing phases
82. Hook directives for request processing phases • set_by_lua / set_by_lua_file • rewrite_by_lua / rewrite_by_lua_file • access_by_lua / access_by_lua_file • content_by_lua / content_by_lua_file • log_by_lua / log_by_lua_file run Lua script-file
83. content_by_lua <lua-script-string> location /hello { content_by_lua “ngx.say(‘Hello, World!’)”; } ▪ nginx.conf $ curl -s http://127.0.0.1/hello Hello, World! $ ▪ run
84. content_by_lua_file <lua-script-path> location /hello { content_by_lua_file /etc/nginx/lua/hello.lua; } ▪ nginx.conf ▪ /etc/nginx/lua/hello.lua ngx.say(“Hello, World!”)
85. set_by_lua $result <lua-script-string> location /yapc { set $conference YAPC::Asia; set $location Tokyo; set $year 2015; set_by_lua $result ‘return ngx.var.conference .. “ ” .. ngx.var.location .. “ ” .. ngx.var.year ‘; content_by_lua ‘ngx.say(ngx.var.result)’; } ▪ nginx.conf $ curl -s http://127.0.0.1/hello YAPC::Asia Tokyo 2015 $ ▪ run
86. Request processing phases in nginx TFSWFS
    SFXSJUF QPTUBDDFTT MPH pOE
    DPOpH SFXSJUF QPTU
    SFXSJUF QSFBDDFTT USZpMFT DPOUFOU Request BDDFTT try_files processing access-control response body generation logging rewrite in server find location rewrite in location set_by_lua*
 rewrite_by_lua* access_by_lua* content_by_lua* log_by_lua*
87. Other hook directives and phases IPPL
    OBNF QIBTF JOJU@CZ@MVB
    JOJU@CZ@MVB@pMF MPBEJOH
    OHJOYDPOG JOJU@XPSLFS@CZ@MVB
    JOJU@XPSLFS@CZ@MVB@pMF TUBSUJOH
    
    XPSLFS
    QSPDFTT IFBEFS@pMUFS@CZ@MVB
    IFBEFS@pMUFS@CZ@MVB@pMF pMUFSJOH
    IFBEFS CPEZ@pMUFS@CZ@MVB
    CPEZ@pMUFS@CZ@MVB@pMF pMUFSJOH
    CPEZ
88. ngx_lua APIs • ngx.say() / ngx.exit() / ngx.log / … • ngx.var.VARIABLE • ngx.shared.DICT • ngx.header.HEADER • etc…
89. ngx.say() ngx.say(“Hello, World!”) • output string to response body
90. ngx.exit() ngx.exit(ngx.HTTP_OK) • return HTTP status code
91. HTTP status code in ngx_lua • ngx.HTTP_OK (200) • ngx.HTTP_CREATED (201) • ngx.HTTP_MOVED_PERMANENTLY (301) • ngx.HTTP_MOVED_TEMPORARILY (302) • ngx.HTTP_NOT_MODIFIED (304) • ngx.HTTP_BAD_REQUEST (400) • ngx.HTTP_INTERNAL_SERVER_ERROR (500) • etc…
92. ngx.log() ngx.exit(ngx.ERR, “some error”) • output message to error.log • $arg1: log-level, $arg2: message
93. log-level in ngx_lua • ngx.EMERG • ngx.ALERT • ngx.CRIT • ngx.ERR • ngx.WARN • ngx.NOTICE • ngx.INFO • ngx.DEBUG
94. log-level in ngx_lua • ngx.EMERG • ngx.ALERT • ngx.CRIT • ngx.ERR • ngx.WARN • ngx.NOTICE • ngx.INFO • ngx.DEBUG log-level High Low
95. ngx.var.VARIABLE • read / overwrite nginx variables location / { set $fn “Tatsuhiko Kubo”; set $hn “@cubicdaiya”; content_by_lua ‘ local body = ngx.var.fn .. ngx.var.hn ngx.say(body) —- Tatsuhiko Kubo@cubicdaiya ‘; }
96. ngx.var.VARIABLE • creating new variable is not allowed location / { content_by_lua ‘ ngx.var.fn = “Tatsuhiko Kubo” ngx.var.hn = “@cubicdaiya” ‘; } runtime error: … variable “fn” not found for writing; maybe it is a built-in variable that is not changeable or … ɾ error.log
97. ngx.shared.DICT • dictionary with shared memory http { lua_shared_dict yapc 1m; init_by_lua ‘ ngx.shared.yapc:set(“presentator”, “@cubicdaiya”) ’; … server { … location / { content_by_lua ‘ ngx.say(ngx.shared.yapc:get(“presentator”)) ’; } } }
98. ngx.header.HEADER • response header reference and manipulation location / { content_by_lua ‘ local body = “{ \\”presentator\\” : \\”@cubicdaiya\\” }” —- assign application/json to Content-Type ngx.header.content_type = “application/json” ngx.say(body) ‘; }
99. Header reference and manipulation • ngx.req.get_headers() • ngx.req.set_header() • ngx.req.clear_header() • … • ngx.resp.get_headers() • ngx.header.HEADER • … ▪ For request-header ▪ For response-header
100. ngx.time() • return current timestamp local timestamp = ngx.time()
101. ngx_lua time APIs • ngx.time() • ngx.today() • ngx.now() • ngx.update_time() • ngx.parse_http_time() • ngx.localtime() • ngx.utctime() • ngx.cookie_time() • ngx.http_time()
102. Use ngx.time() instead of os.time() • os.time() always calls gettimeofday() • ngx.time() uses timer-cache of nginx • avoid system-call’s overhead • ngx.time() is more efficient than os.time()
103. ngx.re.match() • Regular Expression matching • powered by PCRE local matches, err = ngx.re.match(“image/webp”, “([^/]+)\/(.+)”) -— matches[0] -> image/webp -— matches[1] -> image -— matches[2] -> webp
104. ngx.re.* (Regular Expression API) • ngx.re.find() • ngx.re.match() • ngx.re.gmatch() • ngx.re.sub() • ngx.re.gsub()
105. • Use PCRE options(e.g. o, j) • o -> enable compile-once • j -> enable PCRE-JIT Optimize ngx.re.* local matches, err = ngx.re.match(“image/webp”, “([^/]+)\/(.+)”, “oj”)
106. • ngx.re is based PCRE • Powerful & Fat • Lua builtin-re(regular expression) • Simple & Light ngx.re.* VS Lua builtin-re If ‘re’ is simple, Lua builtin-re is faster.
107. ngx.location.capture() • send non-blocking sub-request to other location location /external { internal; proxy_pass http://external; } location / { content_by_lua ‘ —— non-blocking behavior local res = ngx.location.capture(“/external”) … ‘; }
108. By the way, • ngx_lua is very powerful • But setup and deployment is a little painful • Lua / LuaJIT instalation • Lua packages management
109. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
110. Agenda • nginx http module development with C • nginx API programming • Test::Nginx • ngx_lua • OpenResty
111. OpenResty http://openresty.org/
112. OpenResty • Web application framework bundled • nginx and 3rd party modules (e.g. ngx_lua) • Lua, LuaJIT • Resty modules(lua-resty-xxx…) • All-in-one superset of nginx
113. 3rd party modules • lua-nginx-module (ngx_lua) • echo-nginx-module • headers-more-nginx-module • etc…
114. Resty modules • lua-resty-core • resty-cli • lua-resty-mysql / redis / memcached • etc…
115. lua-resty-core • pure Lua partial-re-implementation of ngx_lua APIs • accelerate ngx_lua APIs with LuaJIT • because ngx_lua APIs with C is not JIT compiled http { # accelerate ngx_lua APIs init_by_lua ‘require resty.core’; … }
116. lua-resty-cli • command-line utility for OpenResty • run OpenResty with CLI • all APIs are not always enabled $ resty -e ‘ngx.say(“Hello, World!”)’ Hello, World! $
117. lua-resty-mysql / redis / memcached • client drivers for mysql / redis / memcached • non-blocking behavior • Powered by ngx.socket
118. OpenResty on
119. PascalʙMercari app-event analysis baseʙ 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ (PPHMF
     #JH
     2VFSZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events transfer utilize events utilize events transfer
120. OpenResty’s role on Pascal • High performance event logger • receive events as JSON from devices • parse JSON • utilize events for BigQuery • log events into file every event-channel
121. References • The Architecture of Open Source Applications (Volume 2): nginx • http://www.aosabook.org/en/nginx.html • Nginx Internals • http://www.slideshare.net/joshzhu/nginx-internals • Emiller’s Guide To Nginx Module Development • http://www.evanmiller.org/nginx-modules-guide.html
122. References • openresty/lua-nginx-module • https://github.com/openresty/lua-nginx-module • Using ngx_lua / lua-nginx-module in pixiv • http://www.slideshare.net/harukayon/ngx-lua-public • ngx_mruby and ngx_lua • http://qiita.com/cubicdaiya/items/3913d3dddf6999984c5b
123. Thanks!