Human & Machine

Test::Nginx 0.18 just released to CPAN

2012-02-03T15:55:00.001+08:00

I've just uploaded Test::Nginx 0.18 to CPAN:

http://search.cpan.org/dist/Test-Nginx/

It will appear on the CPAN mirror near you in the next few hours or so.

Special thanks go to all of our users :)

Here's the complete change log for this release, compared to the last CPAN release, 0.17:

feature: added new section "--- no_error_log" which could be used to specify patterns of lines that do not appear in error.log at all.
feature: added MOCKEAGAIN_VERBOSE, DYLD_INSERT_LIBRARIES, TOCKEAGAIN_WRITE_TIMEOUT_PATTERN, and LD_PRELOAD to the environments that should be kept through nginx reload and restart.
doc: added Naxsi and ngx_rds_csv to the user module list in the POD documentation.
feature: now we can use the environment TEST_NGINX_USE_VALGRIND to specify the valgrind command-line options, when set to 1 or other non-zero numbers, it is effectively equivalent to the value "--tool=memcheck --leak-check=full". To use Valgrind SGCheck to run the tests, just specify the environment TEST_NGINX_USE_VALGRIND="--tool=exp-sgcheck" first.
feature: added support for environment TEST_NGINX_VERBOSE.
feature: added support for environment TEST_NGINX_USE_HUP.
feature: added support for the TEST_NGINX_POSTPONE_OUTPUT environment.
feature: now we'll retry sending the QUIT signal for at most 5 times when the nginx process is refusing to quit. after 5 trials, KILL signal will be sent.
feature: exposes the "master_off" API which turns off the nginx master process.
feature: added new section "--- ignore_response" to test bad responses.
bugfix: Test::Nginx::Socket will retry 10 times if fail to connect because nginx with valgrind can be very slow to start in some systems.
feature: added new section name "--- error_log" to test error lines. requested by Piotr Sikora in the last year or so.
feature: added the TEST_NGINX_EVENT_TYPE environment to specify an event API type (like "epoll" or "select") when running the test suite.

This Perl module provides a test scaffold based on IO::Socket or LWP for automated testing in Nginx C module development.

This class inherits from Test::Base, thus bringing all its declarative power to the Nginx C module testing practices.

Please check out the full documentation on CPAN:

http://search.cpan.org/perldoc?Test::Nginx::Socket

All of our Nginx modules are using Test::Nginx to drive their test suites.

Please note that this is different from the Test::Nginx module developed by Maxim Dounin.

Enjoy!

ngx_openresty stable version 1.0.10.48 released!

2012-02-01T12:52:00.001+08:00

I'm happy to announce that the new stable release of ngx_openresty, 1.0.10.48, has just been kicked out of door:

http://openresty.org/#Download

This is the 3rd stable release of ngx_openresty that is based on the Nginx core 1.0.10, which is a maintenance release.

Special thanks go to all our contributors and users to help make this release happen :)

Here goes the complete change log for this release, as compared to the last stable release, 1.0.10.44, released about two weeks ago:

upgraded LuaNginxModule to 0.4.1.
- bugfix: ngx_http_lua_header_filter_init was called with an argument which actually accepts none. this could cause compilation errors at least with gcc 4.3.4 as reported in github issue #80. thanks bigplum (Simon).
- bugfix: fixed all the warnings from the clang static analyzer.
- bugfix: ngx.exit, ngx.redirect, ngx.exec, and ngx.req.set_uri(uri, true) could return (they should never return as per the documentation). this bug had appeared in ngx_lua v0.3.1rc4 and ngx_openresty 1.0.6.13. thanks @cyberty for reporting it.
- feature: allow use of the DDEBUG macro from the outside (via the -D DDEBUG=1 cc opton).
upgraded DrizzleNginxModule to v0.1.2rc6.
- bugfix: fixed all the warnings from the clang static analyzer.
- feature: allow use of the DDEBUG macro from the outside (via the -D DDEBUG=1 cc opton).
upgraded EchoNginxModule to 0.38rc1, SetMiscNginxModule to 0.22rc5, HeadersMoreNginxModule to 0.17rc1, and MemcNginxModule to 0.13rc3, to allow use of the DDEBUG macro from the outside (via the -D DDEBUG=1 cc opton).

As always, you're welcome to report bugs and feature requests either here or directly to me :)

OpenResty (aka. ngx_openresty) is a full-fledged web application server by bundling the standard Nginx core, lots of 3rd-party Nginx modules, as well as most of their external dependencies.

By taking adantage of various well-designed Nginx modules, OpenResty effectively turns the nginx server into a powerful web app server, in which the web developers can use the Lua programming language to script various existing nginx C modules and Lua modules and construct extremely high-performance web applications that is capable to handle 10K+ connections.

OpenResty aims to run your server-side web app completely in the Nginx server, leveraging Nginx's event model to do non-blocking I/O not only with the HTTP clients, but also with remote backends like MySQL, PostgreSQL, Memcached, and Redis.

You can find more details on the homepage of ngx_openresty here:

http://openresty.org

Enjoy!

ngx_lua module v0.4.1 released!

2012-02-01T12:01:00.001+08:00

I'm happy to announce the v0.4.1 release of our ngx_lua module. You can get the release tarball from the download page:

https://github.com/chaoslawful/lua-nginx-module/tags

Here's the change log compared to the last formal release, v0.4.0:

bugfix: ngx.exit, ngx.redirect, ngx.exec, and ngx.req.set_uri(uri, true) could return (they should never return as per the documentation). this bug had appeared in ngx_lua v0.3.1rc4 and ngx_openresty 1.0.6.13. thanks @cyberty for reporting it.
bugfix: ngx_http_lua_header_filter_init was called with an argument which actually accepts none. this could cause compilation errors at least with gcc 4.3.4 as reported in github issue #80. thanks bigplum (Simon).
bugfix: fixed all the warnings from the clang static analyzer.
feature: allow use of the DDEBUG macro from the outside (via the -D DDEBUG=1 C compiler opton).

Special thanks go to all of our contributors and users!

I'm currently focusing on the ngx_lua cosocket branch which will become the v0.5.x series. With the upcoming cosocket support, we'll be able to code up nonblocking network client drivers for various backend services (memcached, mysql, redis, to name a few) in pure Lua, and these drivers could be even faster than the old approach that requires combining nginx subrequests and nginx upstream modules.

This Nginx module embeds the Lua 5.1 interpreter or LuaJIT 2.0 into the nginx core and integrates the powerful Lua threads (aka Lua coroutines) into the nginx event model by means of nginx subrequests.

Unlike Apache's mod_lua and Lighttpd's mod_magnet, Lua code written atop this module can be 100% non-blocking on network traffic as long as you use the ngx.location.capture or ngx.location.capture_multi interfaces to let the Nginx core do all your requests to mysql, postgresql, memcached, redis, upstream http web services, and etc etc etc.

This module is also included and enabled by default in our ngx_openresty bundle: http://openresty.org/

You can find the complete documentation for this module on the following wiki page:

http://wiki.nginx.org/HttpLuaModule

And you always get the latest source code from the git repository here:

https://github.com/chaoslawful/lua-nginx-module

Have fun!

ngx_openresty stable version 1.0.10.44 released!

2012-01-16T22:32:00.002+08:00

I'm happy to announce that the new stable release of ngx_openresty, 1.0.10.44, has just been kicked out of door:

http://openresty.org/#Download

This is the second stable release of ngx_openresty that is based on the Nginx core 1.0.10 (and the next one will be based on the Nginx core 1.0.11+).

Special thanks go to all our contributors and users to help make this release happen :)

Here goes the complete change log for this release, as compared to the last stable release, 1.0.10.24, released about one month ago:

upgraded Mike Pall's LuaJIT to 2.0.0beta9.
- changes: http://luajit.org/changes.html
upgraded EchoNginxModule to 0.37.
- bugfix: fixed two spots that we did not check null pointers returned by the memory allocator.
- bugfix: attempt to fix places where ngx_time_update might not be compiled properly.
upgraded HeadersMoreNginxModule to 0.16.
- bugfix: more_set_input_headers and more_clear_input_headers did not handle the Accept-Encoding request headers properly. thanks 天街夜色.
- bugfix: removing builtin headers in huge request headers with 20+ entries could result in data loss. thanks Chris Dumoulin for the patch in github issue #6.
- bugfix: the more_set_input_headers directive might cause invalid memory reads because Nginx request header values must be null terminated. thanks Maxim Dounin.
upgraded LuaNginxModule to 0.4.0.
- bugfix: ngx.flush(true) could not be used before I/O calls like ngx.location.capture. this bug had appeared in v0.3.1rc34.
- bugfix: ngx.var.VARIABLE did not evaluate to nil when the Nginx variable's valid flag is 0.
- docs: various documentation improvements. thanks Nginx User.
- bugfix: there were various places where we did not check the pointer returned by the memory allocator.
- bugfix: ngx.req.get_uri_args and ngx.req.get_post_args now only parse up to 100 arguments by default. but one can specify the optional argument to these two methods to specify a custom maximum number of args. thanks Tzury Bar Yochay for reporting this.
- bugfix: ngx.req.get_headers now only parse up to 100 request headers by default. but one can specify the optional argument to this method to specify a custom maximum number of headers.
- bugfix: removing builtin headers via ngx.req.clear_header and its equivalent in huge request headers with 20+ entries could result in data loss. thanks Chris Dumoulin for the patch in github issue #6.
- bugfix: could not compile with Nginx 1.1.12+ because Nginx 1.1.12 changed its regex API. now we call PCRE API directly and require at least PCRE 8.21 for the PCRE JIT support in our ngx.re API (since PCRE 8.20 had a bug in its JIT engine that it did not honor pcre_malloc and pcre_free at all).
- bugfix: ngx.req.set_header might cause invalid memory reads because Nginx request header values must be null terminated. thanks Maxim Dounin.
- bugfix: ngx.var.VARIABLE might evaluate to nil even if there is a valid value because the Nginx variable value's valid flag might not be initialized properly. this bad had appeared in v0.3.1rc40.
- bugfix: ngx.req.set_header and ngx.req.clear_header did not handle the Accept-Encoding request headers properly. thanks 天街夜色.
upgraded EncryptedSessionNginxModule to 0.02.
- bugfix: the -lssl option broke nginx linking when --with-openssl=DIR is specified. thanks charlieyang for reporting this issue.
upgraded SetMiscNginxModule to v0.22rc4.
- bugfix: fixed one place that does not check the pointer returned by the memory allocator.
- src: converted CRLF in the source files and test files to LF.
upgraded SrcacheNginxModule to v0.13rc3.
- bugfix: fixed a regression with XssNginxModule for cache hits. this bug had appeared in v0.13rc1. thanks 万�新.
- bugfix: we did not cache the Location response header at all for 301/302 responses.
- bugfix: we should not blindly cache the Accept-Ranges: bytes response headers regardless of the actual current requests.
upgraded XssNginxModule to v0.03rc8.
- bugfix: fixed a few debug-level log messages; the old text was misleading.
upgraded Maxim Dounin's UpstreamKeepaliveNginxModule to 0.7.
- Bugfix: unbuffered connection might not be kept alive under Linux.
- Bugfix: module could not be built on Windows.
- Bugfix: module could not be built without the ngx_http_ssl_module.
- Feature: https connections support (requires patches).
- Bugfix: invalid connections might be cached.
- Bugfix: the "[alert] ... open socket ... left in connection ..." messages were logged on nginx worker process gracefull exit for each cached connection; the bug had appeared in version 0.3.
bugfix: fixed issues with relative path DIR in the --with-openssl=DIR option for ./configure.
bugfix: fixed compatibility of the packaging scripts on Darwin and *BSD. thanks nightsailer and Piotr Sikora.
bugfix: the install phony target did not depend on the all phony target in the Makefile generated by ./configure. thanks 姚伟斌 for reporting this issue.
bugfix: the ./configure script's --add-module option did not accept relative path values. thanks 姚伟斌 for the patch.
bugfix: some old version of shell cp command does not support trailing slashes in the destination argument and could break our ./configure script. thanks 姚伟斌 for reporting it.

ngx_headers_more module v0.16 released!

2012-01-16T15:05:00.001+08:00

I'm happy to announce the v0.16 release of our ngx_headers_more module with various bug fixes accumulated in the last 6 months. You can download the source code tarball from the download page below:

http://github.com/agentzh/headers-more-nginx-module/tags

Here's the complete change log for this version (as compared to the last formal release, v0.15):

bugfix: the on-demand handler/filter registration mechanism did not work fully for config reload via the HUP signal.
bugfix: when setting a multi-value response header to a single value, the single value might be repeated on each old value.
feature: added some debugging outputs that can be enabled by the --with-debug option when building nginx or ngx_openresty.
bugfix: we should set header hash using ngx_hash_key_lc, not simply to 1.
bugfix: Setting Cache-Control response headers might not work with other nginx output filter modules because we did not properly prepare the r->cache_control array at the same time.
bugfix: more_set_input_headers and more_clear_input_headers did not handle the Accept-Encoding request headers properly. thanks 天街夜色.
bugfix: the more_set_input_headers directive might cause invalid memory reads because Nginx request header values must be null terminated. thanks Maxim Dounin.
bugfix: removing builtin headers in huge request headers with 20+ entries could result in data loss. thanks Chris Dumoulin for the patch in github issue #6.

Special thanks go to all of our contributors and users!

The ngx_headers_more module allows you to add, set, or clear any output or input header that you specify. This is an enhanced version of the standard headers module because it provides more utilities like resetting or clearing "builtin headers" like Content-Type, Content-Length, and Server.

You can always get the latest source code from its project page on GitHub

http://github.com/agentzh/headers-more-nginx-module

and the full documentation from the nginx wiki

http://wiki.nginx.org/NginxHttpHeadersMoreModule

This module is included and enabled by default in our ngx_openresty bundle: http://openresty.org

Enjoy!

ngx_echo module v0.37 released!

2012-01-13T17:05:00.001+08:00

I'm happy to announce the v0.37 release of the ngx_echo module, which includes lots of bugfixes in the last 6 months.

You can get the release tarball from the download page below

https://github.com/agentzh/echo-nginx-module/tags

Here's the complete change log for this release (compared to the last release, v0.36):

bugfix: data truncation might occur with echo_after_body in not-so-good networks (we should be prepared for NGX_AGAIN returned by downstream output filters). thanks 林青 (Kindy Lin) for reporting it.
bugfix: we no longer check sync buffers for subrequests because it is incorrect.
bugfix: fixed a bug when sending out response headers: we did not take into account the NGX_ERROR error code returned by ngx_http_send_header.
bugfix: we did not work with HEAD http requests before.
bugfix: now we carefully eliminate empty flush buffers in echo_after_body to work around a long-standing bug in the standard HttpGzipModule.
bugfix: we might send empty chain link in echo_after_body because it may trigger the infamous "the http output chain is empty" alert in error.log when the standard HttpSsiModule is disabled. thanks Sparsh Gupta.
bugfix: we did not set subrequest's Content-Length request headers which could cause problems in the backends.
bugfix: echo_exec + named locations might cause weird issues and now we explicitly clear all the modules' contexts before calling ngx_http_named_location.
bugfix: echo_exec might hang when running after echo_sleep (or other I/O interruption calls): we should have called ngx_http_finalize_request on NGX_DONE to decrement r->main->count anyway.
bugfix: there was a memory issue in both echo_sleep and echo_blocking_sleep: we should not pass ngx_str_t strings to atof() which expects C strings.
bugfix: some users report that this module cannot be compiled with Nginx 1.0.x on their systems due to ngx_time_update (as in github issue #7). this is a blind attemp to fix it because we could not reproduce it on our side.
bugfix: fixed places in the source code that we did not check null pointers returned by the memory allocator.

You can also view the HTML-version of the change log here (which may look better):

http://wiki.nginx.org/HttpEchoModule#v0.37

The ngx_echo module wraps lots of Nginx internal APIs for streaming input and output, parallel/sequential subrequests, timers and sleeping, as well as various meta data accessing. You can read the full documentation on the following wiki page:

http://wiki.nginx.org/HttpEchoModule

and you can always get the latest source code from GitHub:

http://github.com/agentzh/echo-nginx-module

The ngx_echo module is included and enabled by default in our ngx_openresty bundle:

http://openresty.org

Happy echoing!

ngx_lua module v0.4.0 released!

2012-01-11T23:07:00.001+08:00

After four months' active development of our contributors, I'm happy to announce the v0.4.0 release of our ngx_lua module. You can get the release tarball from the download page:

https://github.com/chaoslawful/lua-nginx-module/tags

There's many many new features and bug fixes that you can find out in the following complete change log for this release (it is just too long to put here :P):

http://wiki.nginx.org/HttpLuaModule#v0.4.0

Special thanks go to all of our contributors and users!

This module embeds the Lua 5.1 interpreter or LuaJIT 2.0 into the nginx core and integrates the powerful Lua threads (aka Lua coroutines) into the nginx event model by means of nginx subrequests.

Unlike Apache's mod_lua and Lighttpd's mod_magnet, Lua code written atop this module can be 100% non-blocking on network traffic as long as you use the ngx.location.capture or ngx.location.capture_multi interfaces to let the Nginx core do all your requests to mysql, postgresql, memcached, redis, upstream http web services, and etc etc etc.

This module is also included and enabled by default in our ngx_openresty bundle: http://openresty.org/

You can find the complete documentation for this module on the following wiki page:

http://wiki.nginx.org/HttpLuaModule

And you always get the latest source code from the git repository here:

https://github.com/chaoslawful/lua-nginx-module

Have fun!

ngx_openresty stable version 1.0.10.24 released

2011-12-11T23:19:00.001+08:00

I'm happy to announce that the new stable release of ngx_openresty, 1.0.10.24, has just been kicked out of door:

http://openresty.org/#Download

This is the first stable release of ngx_openresty that is based on the Nginx core 1.0.10.

Special thanks go to all our contributors and users to help make this release happen :)

Here goes the complete change log for this release, as compared to the last stable release, 1.0.9.10, released nearly one month ago:

upgraded the Nginx core to 1.0.10.
upgraded LuaNginxModule to 0.3.1rc38.
- feature: added opions copy_all_vars and vars to ngx.location.capture and ngx.location.capture_multi. thanks Marcus Clyne for the patch.
- feature: added new Lua API ngx.now to return the current time (including the ms part as the decimal part). thanks 林青.
- feature: added new Lua API ngx.update_time to forcibly updating nginx's time cache.
- feature: added wait boolean argument to ngx.flush() to support synchronous flushing; ngx.flush(true) will not return until all the data has been flushed into the system send buffer or the send timeout has expired.
- feature: added constant ngx.HTTP_GATEWAY_TIMEOUT (504) per Fry-kun in github issue #73.
- feature: added new regex options "j" and "d" to ngx.re.match, ngx.re.gmatch, ngx.re.sub, and ngx.re.gsub so as to enable the PCRE JIT mode and DFA mode, respectively. thanks @姜大炮 for providing the patch.
- feature: added ngx.hmac_sha1. thanks drdrxp.
- bugfix: fixed a bad regression in ngx.location.capture_multi when the request option table is specified. this bug had appeared in ngx_lua 0.3.1rc26 and ngx_openresty 1.0.9.1.
- bugfix: now we check timed out downstream connections in our write event handler.
- bugfix: use of the ngx.re API might lead to errors like pcre_compile() failed: failed to get memory in ... due to incorrect pcre_malloc and pcre_free handling. thanks Vittly for reporting this as github issue #72.
- docs: documented the long-existent ngx.md5 and ngx.md5_bin APIs.
- docs: massive documentation improvements. thanks Nginx User.
upgraded MemcNginxModule to 0.13rc2.
- bugfix: now we use signed char explicitly instead of the vague char type which could be unsigned by default in certain systems like PowerPC. thanks Dmitry E. Oboukhov.
upgraded Redis2NginxModule to 0.08rc2.
- bugfix: when char defaults to unsigned char, the Ragel-based Redis response parser could not accept non-ascci bytes. thanks Dmitry E. Oboukhov.
upgraded RdsJsonNginxModule to v0.12rc7.
- added more debug level error log outputs to ease debugging (as discussed in github issue #2).
upgraded XssNginxModule to v0.03rc7.
- now we use the ngx_log_debugN macros to emit debugging outputs instead of info level error logging because the latter is costly in production.
- bugfix: now we use signed char explicitly instead of the vague char type which could be unsigned by default in certain systems like PowerPC. thanks Dmitry E. Oboukhov.
bugfix: fixed a serious regression for Linux AIO in nginx-1.0.10-epoll_check_stale_wev.patch, thanks Maxim Dounin.

Working on Nginx Tutorials in Chinese

2011-12-06T15:54:00.001+08:00

I've been posting Nginx tutorials in Chinese to my yet another blog site hosted by Sina Blog here:

http://blog.sina.com.cn/openresty

I'm currently posting tutorials for the "Nginx Variables" series in great details. And I'm trying to post an article every work day.

I'm hoping that I can eventually work out an Nginx book in Chinese based on these materials soon so I'm working hard and seriously.

If you cannot read Chinese, please try out translating tools like Google Translator ;) I'm very willing to find someone to translate these to English or any other languages though :)

Enjoy!

ngx_openresty 1.0.9.10 (stable) released

2011-11-16T11:28:00.001+08:00

I'm happy to announce that the new stable release of ngx_openresty, 1.0.9.10, has just been kicked out of door:

http://openresty.org/#Download

This is the first stable release of ngx_openresty that is based on the Nginx core 1.0.9.

Special thanks go to all our contributors and users to help make this release happen :)

Here goes the complete change log for this release, as compared to the last stable release, 1.0.8.26, released two weeks ago:

upgraded the Nginx core to 1.0.9.
applied the epoll_check_stale_wev patch to the Nginx 1.0.9 core. this issue affected PostgresNginxModule when connecting to a remote PostgreSQL server over a slow network. thanks @晓旭XX.
bugfix: nginx-1.0.9-variable_header_ignore_no_hash.patch might introduce a memory overflow issue in multi-header variables. thanks Markus Linnala.
bugfix: fixed the error message length while the ./configure script fails.
feature: applied a patch to add new directives log_escape_non_ascii to prevent escaping non-ascii bytes in access log variable values. requested by @姜大炮. It can be turned on and off, and default to on just as the standard Nginx version.
upgraded DrizzleNginxModule to 0.1.2rc4.
- bugfix: fixed issues with poll, rtsig, and select used by the Nginx event model by eliminating the poll syscall performed by libdrizzle. This also gives rise to a nice speedup (about 10% in simple cases).
upgraded LuaNginxModule to 0.3.1rc28.
- feature: added the ngx.encode_args method to encode a Lua code to a URI query string. thanks 郭颖 (0597虾).
- feature: ngx.location.capture and ngx.exec now supports the same Lua args table format as in ngx.encode_args. thanks 郭颖 (0597虾).
- bugfix: Cache-Control header modification might introduce empty value headers when using with the standard ngx_headers module.
- feature: added the ctx option to ngx.location.capture: you can now specify a custom Lua table to pass to the subrequest as its ngx.ctx. thanks @hugozhu.
- bugfix: fixed compatibility with nginx 0.8.54. thanks @0579虾.
upgraded HeadersMoreNginxModule to 0.16rc4.
- bugfix: Cache-Control header modification might introduce empty value headers when using with the standard ngx_headers module.
upgraded PostgresNginxModule to 0.9rc2
- bugfix: now we log an error message when the postgres_pass target is not found at all and returns 500 in this case instead of returning empty response.
- bugfix: we should no longer return NGX_AGAIN when the re-polling returns IO WAIT in case of the "connection made" state.
- feature: added some debugging outputs which be enabled by passing the --with-debug option while building Nginx or OpenResty.
- bugfix: fixed compatibility issues with Nginx 1.1.4+: ngx_chain_update_chains now requires a pool argument.
upgraded LuaRdsParserLibrary to 0.04.
- bugfix: fixed a serious memory leak reported by bearnard.
upgraded XssNginxModule to 0.03rc5.
- bugfix: the callback argument value parser did not accept JavaScript identifier names started with underscores. thanks Sam Mulube.

As always, you're welcome to report bugs and feature requests either here or directly to me :)

It'll also be highly appreciated to try out the devel releases (based on the Nginx core 1.0.10+) that are coming out later ;)

OpenResty (aka. ngx_openresty) is a full-fledged web application server by bundling the standard Nginx core, lots of 3rd-party Nginx modules, as well as most of their external dependencies.

By taking adantage of various well-designed Nginx modules, OpenResty effectively turns the nginx server into a powerful web app server, in which the web developers can use the Lua programming language to script various existing nginx C modules and Lua modules and construct extremely high-performance web applications that is capable to handle 10K+ connections.

OpenResty aims to run your server-side web app completely in the Nginx server, leveraging Nginx's event model to do non-blocking I/O not only with the HTTP clients, but also with remote backends like MySQL, PostgreSQL, Memcached, and Redis.

You can find more details on the homepage of ngx_openresty here:

http://openresty.org

Have fun!

ngx_openresty 1.0.8.26 (stable) released

2011-11-03T17:24:00.001+08:00

I'm happy to announce that the new stable release of ngx_openresty, 1.0.8.26, has just been kicked out of door:

http://openresty.org/#Download

This is the first stable release of ngx_openresty that is based on the Nginx core 1.0.8.

Here goes the complete change log for this release, as compared to the last stable release, 1.0.6.22, released nearly a month ago:

upgraded the Nginx core to 1.0.8.
upgraded LuaNginxModule to 0.3.1rc23.
- feature: added new directive lua_shared_dict: http://wiki.nginx.org/HttpLuaModule#lua_shared_dict
- feature: added Lua API for the shm-based dictionary: http://wiki.nginx.org/HttpLuaModule#ngx.shared.DICT
- feature: now we apply the patch to the nginx core so as to allow main request body modifications.
- feature: added new Lua API ngx.req.set_body_file(): http://wiki.nginx.org/HttpLuaModule#ngx.req.set_body_file
- feature: added new Lua API ngx.req.set_body_data(): http://wiki.nginx.org/HttpLuaModule#ngx.req.set_body_data
- feature: added new Lua functions ngx.req.read_body(), ngx.req.discard_body(), ngx.req.get_body_data(), and ngx.req.get_body_file(). see the docs here: http://wiki.nginx.org/HttpLuaModule#ngx.req.read_body
- feature: now we implemented ngx.req.set_uri() and ngx.req.set_uri_args() to emulate ngx_rewrite's rewrite directive (without redirect or permanent modifiers). thanks Vladimir Protasov (utros) and Nginx User.
- feature: added constant ngx.HTTP_METHOD_NOT_IMPLEMENTED (501). thanks Nginx User.
- feature: now for HTTP 1.0 requests, we disable the automatic full buffering mode if the user sets the Content-Length response header before sending out the headers. this allows streaming output for HTTP 1.0 requests if the content length can be calculated beforehand. thanks Li Ziyi.
- bugfix: now we properly support setting the Cache-Control response header via the ngx.header.HEADER interface.
- bugfix: no longer set header hash to 1. use the ngx_hash_key_lc instead.
- bugfix: now we skip rewrite phase Lua handlers altogether if ngx_rewrite's rewrite directive issue a location re-lookup by changing URIs (but not including rewrite ... break). thanks Nginx User.
- bugfix: fixed hanging issues when using ngx.exec() within rewrite_by_lua and access_by_lua. thanks Nginx User for reporting it.
- bugfix: lua_need_request_body should not skip requests with methods other than POST and PUT. thanks Nginx User.
- bugfix: ndk.set_var.DIRECTIVE had a memory issue and might pass empty argument values to the directive being called. thanks dannynoonan.
- bugfix: no longer free request body buffers that are not allocated by ourselves.
- bugfix: now we allow setting ngx.var.VARIABLE to nil.
- bugfix: now we explicitly clear all the modules' contexts before dump to named location with ngx.exec. thanks Nginx User.
upgraded EchoNginxModule to 0.37rc7.
- bugfix: fixed a memory issue in both echo_sleep and echo_blocking_sleep: we should not pass ngx_str_t strings to atof() which expects C strings.
- bugfix: now we explicitly clear all the modules' contexts before dump to named location with echo_exec.
- bugfix: bugfix: echo_exec may hang when running after echo_sleep (or other I/O interruption calls): we should have called ngx_http_finalize_request on NGX_DONE to decrement r->main->count anyway.
- bugfix: now we properly set the Content-Length request header for subrequests.
upgraded SrcacheNginxModule to 0.13rc2.
- feature: implemented response status line and general response header cachin and added new directives srcache_store_hide_header and srcache_store_pass_header to control which headers to cache and which not.
- feature: added new directive srcache_response_cache_control to control whether honor response headers Cache-Control and Expires, default on.
- feature: we disable srcache_store automatically by default when Cache-Control: max-age=0 and Expires: <date no more recently than now> are seen.
- feature: implemented builtin nginx variable $srcache_expire for automatic expiration time calculation based on response headers Cache-Control (max-age) and Expires; also added new directives srcache_max_expire and srcache_default_expire.
- feature: implemented the srcache_store_no_cache directive; now by default, we do not store responses with the header Cache-Control: no-cache into the cache.
- feature: implemented the srcache_store_no_store directive (default off). Now by default, responses with the header Cache-Control: no-store will not be stored into the cache.
- feature: implemented the srcache_store_private directive to control whether to store responses with the header Cache-Control: private.
- feature: implemented the srcache_request_cache_control directive to allow request headers Cache-Control: no-cache or Pragma: no-cache to force bypassing cache lookup. it also honors the request header Cache-Control: no-store. this directive is turned off by default.
- feature: now we check response header Content-Encoding by default and a non-empty header value will skip srcache_store; also introduced a new directive named srcache_ignore_content_encoding to ignore this response header.
- feature: implemented the srcache_methods directive to specify request methods that are cacheable, by default, only GET and HEAD are cacheable.
- bugfix: we no longer set header hash to 1; we use ngx_hash_key_lc instead.
- bugfix: when we skip srcache_fetch by means of srcache_fetch_skip, we should not automatically skip srcache_store.
- bugfix: now we ignore the Content-Length header (if any) of the main request for the subrequests.
- bugfix: there might be a segfault when failing to allocate memory in ngx_http_srcache_add_copy_chain. thanks Shaun savage.
- feature: implemented new directive srcache_store_statuses to allow the user to specify the response status code list that is to be stored into the cache.
- bugfix: we now only cache 200, 301, and 302 responses by default.
upgraded IconvNginxModule to 0.10rc5.
- bugfix: fixed -Wset-but-not-used warnings issued by gcc 4.6.0. thanks Zhi Jiale (Calio).
upgraded HeadersMoreNginxModule to 0.16rc3.
- bugfix: we should set header hash using ngx_hash_key_lc, not simply to 1.
- bugfix: fixed setting Cache-Control response headers. we should properly prepare the r->cache_control array as well.
upgraded RdsJsonNginxModule to 0.12rc6.
- bugfix: fixed compatibility with nginx 1.1.4+.
upgraded RdsCsvNginxModule to 0.04.
- bugfix: fixed compatibility issues with nginx 1.1.4+.
- optimization: now we only register our filters when rds_csv is actually used in nginx.conf.
upgraded Redis2NginxModule to 0.08rc1.
- bugfix: fixed compatibility with nginx 1.1.4+.
upgraded DrizzleNginxModule to 0.1.2rc2.
- bugfix: fixed compatibility with nginx 1.1.4+
upgraded MemcNginxModule to 0.13rc1.
- bugfix: fixed compatibility with nginx 1.1.4+.
upgraded SetMiscNginxModule to v0.22rc3.
- minor code cleanup.
applied the patch to the Nginx core that always clears all modules' contexts in ngx_http_named_location.
applied the patch for the variable-header-ignore-no-hash issue. see http://forum.nginx.org/read.php?29,216062 for details.

As always, you're welcome to report bugs and feature requests either here or directly to me :)

It'll also be highly appreciated to try out the devel releases (based on the Nginx core 1.0.9+) that are coming out later ;)

OpenResty (aka. ngx_openresty) is a full-fledged web application server by bundling the standard Nginx core, lots of 3rd-party Nginx modules, as well as most of their external dependencies.

By taking adantage of various well-designed Nginx modules, OpenResty effectively turns the nginx server into a powerful web app server, in which the web developers can use the Lua programming language to script various existing nginx C modules and Lua modules and construct extremely high-performance web applications that is capable to handle 10K+ connections.

OpenResty aims to run your server-side web app completely in the Nginx server, leveraging Nginx's event model to do non-blocking I/O not only with the HTTP clients, but also with remote backends like MySQL, PostgreSQL, Memcached, and Redis.

You can find more details on the homepage of ngx_openresty here:

http://openresty.org

Have fun!

openresty.org will be down temporarily for maintenance

2011-10-08T14:58:00.001+08:00

We're moving the sites openresty.org and agentzh.org to a Linode data center in Japan and these sites will be down for just a few hours or so on Oct 8. Sorry for the inconvenience.

Sorry for the inconvenience :)

ngx_openresty 1.0.6.22 released

2011-10-07T13:11:00.001+08:00

I'm happy to announce that the stable release of ngx_openresty, 1.0.6.22, has just been released:

http://openresty.org/#Download

Here goes the complete change log for this release, as compared to the last stable release, 1.0.6.12:

added new option -jN (e.g., -j8, -j10, and etc.) to OpenResty's ./configure script to allow parallel build of the dependencies like LuaJIT; thanks @Lance.

upgraded LuaNginxModule to v0.3.1rc8.
- exposes the CRC-32 API of the Nginx core to the Lua land, in the form of the ngx.crc32_short and ngx.crc32_long methods. thanks @Lance.
- now ngx.exec() supports lua table as the second args argument value. thanks sexybabes.
- implemented the ngx.headers_sent API to check if response headers are sent (by ngx_lua). thanks @hugozhu.
- now we also return the Last-Modified header (if any) for the subrequest response object. thanks @cyberty and sexybabes.
- fixed an issue in ngx.redirect, ngx.exit, and ngx.exec: these function calls would be intercepted by Lua pcall/xpcall because they used lua exceptions; now they use lua yield just as ngx.location.capture. thanks @hugozhu for reporting this.

OpenResty (aka. ngx_openresty) is a full-fledged web application server by bundling the standard Nginx core, lots of 3rd-party Nginx modules, as well as most of their external dependencies.

By taking adantage of various well-designed Nginx modules, OpenResty effectively turns the nginx server into a powerful web app server, in which the web developers can use the Lua programming language to script various existing nginx C modules and Lua modules and construct extremely high-performance web applications that is capable to handle 10K+ connections.

OpenResty aims to run your server-side web app completely in the Nginx server, leveraging Nginx's event model to do non-blocking I/O not only with the HTTP clients, but also with remote backends like MySQL, PostgreSQL, Memcached, and Redis.

You can find more details on the homepage of ngx_openresty here:

http://openresty.org

Have fun!

ngx_openresty 1.0.6.12 released

2011-09-21T23:28:00.001+08:00

I've just kicked ngx_openresty's 1.0.6.12 release out of the door, which is the first stable release of ngx_openresty based on the nginx 1.0.6 core:

http://openresty.org/#Download

Here goes the complete change log, as compared to the last stable release 1.0.5.1:

upgraded RdsJsonNginxModule to v0.12rc4.
- made rds_json_ret honor rds_json_success_property and rds_json_user_property. thanks Liseen Wan (万珣新)
- only register our output filters when the rds_json directive is actually used in nginx.conf.
upgraded RdsCsvNginxModule to v0.03.
- only register our output filters when the rds_csv directive is actually used in nginx.conf.

added new options --with-luajit=PATH and --with-lua51=PATH to the ./configure script. thanks NginxUser.
upgraded DrizzleNginxModule to v0.1.1.

upgraded RdsJsonNginxModule to 0.12rc3.
- implemented new directive rds_json_root.
- implemented new directive rds_json_success_property.
- implemented new directive rds_json_user_property.

upgraded LuaNginxModule to 0.3.1rc3.
- implemented and documented the API for reading response headers from within Lua: value = ngx.header.HEADER.
- fixed a bug when setting a multi-value response header to a single value (via writing to ngx.header.HEADER): the single value will be repeated on each old value.
upgraded EchoNginxModule to 0.37rc4.
- fixed a bug in echo_after_body: when network is not perfect, data truncation might occur. we should have taken into account NGX_AGAIN returned by the downstream output filters. thanks Sparsh Gupta.
upgraded HeadersMoreNginxModule to v0.16rc2.
- fixed a bug when setting a multi-value response header to a single value: the single value will be repeated on each old value.
applied the patch from Maxim Dounin to fix a bug in the standard ngx_gzip module when dealing with empty flush buffers: http://mailman.nginx.org/pipermail/nginx-devel/2011-February/000730.html
updated the no-pool-patch to eliminate the -Wset-but-not-used warnings issued by gcc 4.6.0.

upgraded LuaNginxModule to 0.3.1rc1.
- fixed a bug when the both the main request and the subrequest are POST requests with a body: we should not forward the main request's Content-Length headers to the user subrequests. thanks 朱峰.

upgraded HeadersMoreNginxModule to 0.16rc1.
- fixed on-demand hander/filter registration trick for HUP signal restarts.
- added some debugging outputs that can be enabled by the --with-debug option when building Nginx or OpenResty.
Upgraded the nginx core to 1.0.6

Enjoy!

ngx_drizzle v0.1.1: fixed segfaults on 32-bit systems

2011-09-17T13:07:00.001+08:00

I'm happy to announce that ngx_drizzle v0.1.1 has been released. You can download the latest release tarball from

https://github.com/chaoslawful/drizzle-nginx-module/downloads

This release fixed a long-standing segmentation fault on 32-bit systems, as well as some other issues. Here goes the complete change log:

fixed segmentation faults on 32-bit systems. thanks @魏世江 and @stefanli for reporting this issue.
documented the $drizzle_thread_id variable.
added lots of debug outputs (enabled by the --with-debug option while building Nginx or OpenResty), inspired by github issue #10.
fixed issues regarding defining global variables in C header files: we should have defined rds_rough_col_type_t as a type rather than a global variable. thanks @姜大炮.
documented the python -> python3 pitfall while building libdrizzle 1.0 on at least ArchLinux.
fixed the automatic libdrizzle searching algorithm in the config file: now we should look under libdrizzle-1.0/ instead. thanks 支家乐 (Calio) for reporting this issue.

ngx_drizzle is an nginx upstream module integrating libdrizzle into Nginx in a non-blocking and streamming way.

Essentially it provides a very efficient and flexible way for nginx internals to access MySQL, Drizzle, as well as other RDBMS's that support the Drizzle or MySQL wired protocol. Also it can serve as a direct REST interface to those RDBMS backends.

This module does not generate human-readable outputs, rather, in a binary format called Resty-DBD-Stream (RDS) designed by ourselves. You usually need other components, like HttpRdsJsonModule, HttpRdsCsvModule, or LuaRdsParser, to work with this module. See Output Format for details about the RDS format.

You can find the complete documentation for ngx_drizzle on Nginx Wiki:

http://wiki.nginx.org/HttpDrizzleModule

and the latest source is on GitHub:

https://github.com/chaoslawful/drizzle-nginx-module

This module is also bundled (though not enabled by default) by the ngx_openresty bundle:

http://openresty.org/

You can check out this wiki page to enable this component while building ngx_openresty:

http://openresty.org/#DrizzleNginxModule

Have fun!

My ngx_redis2 module now has a wiki page!

2011-09-06T18:54:00.001+08:00

Today I turned the good old plain-text documentation of my ngx_redis2 module into a cute wiki page:

http://wiki.nginx.org/HttpRedis2Module

I also generated a markdown version of this document with a Perl script for GitHub to display:

https://github.com/agentzh/redis2-nginx-module

This is an Nginx upstream module that makes Nginx talk to a Redis 2.x server in a non-blocking way. The full Redis 2.0 unified protocol has been implemented including the Redis pipelining support.

There's a lot of success reports of using this module in production (combined with ngx_lua and lua-redis-parser for sure), including Qunar.com. Web applications built upon these components have remarkable performance especially when LuaJIT and the TCP connection pool provided by ngx_upstream_keepalive are both enabled.

Enjoy!

The lua-rds-parser Lua library and ngx_rds_csv nginx module

2011-09-04T13:16:00.001+08:00

Recently I wrote a Lua C library named lua-rds-parser for the ngx_openresty world that we can now efficiently parse Resty DBD Stream (RDS) formatted data generated by ngx_drizzle and ngx_postgres into Lua data structures without using JSON as the intermediate data format. This can save both CPU time and memory for web applications running atop ngx_lua.

Here is the project homepage for this Lua library:

https://github.com/agentzh/lua-rds-parser

Also, there's a new Nginx filter module named ngx_rds_csv that can convert RDS output into the Comma-Separated Values (CSV) format in a streaming fashion:

https://github.com/agentzh/rds-csv-nginx-module

Both of these two components have already been included and enabled by default in the latest ngx_openresty stable release :)

Enjoy!

ngx_lua v0.3.0 released: new regex API, request args API, header filter hooks, and more

2011-09-02T16:20:00.001+08:00

After two months' active development of our contributors, I'm happy to announce the v0.3.0 release of our ngx_lua module. You can get the release tarball from the download page:

    https://github.com/chaoslawful/lua-nginx-module/downloads

This release contains a new PCRE regex API for Lua, an API for retrieving parsed URI query arguments and POST query arguments, as well as the new hooks that allow you to define nginx header filters directly in Lua. Also, the special "ngx.ctx" table is also implemented so that you can store arbitrary Lua values into this data which will have a lifetime identical to the current nginx request.

There's many more new features and bug fixes that you can find out in the following complete change log for this release:

    http://wiki.nginx.org/HttpLuaModule#v0.3.0

Special thanks go to our contributors and users, Liseen Wan, Lance, Ou Yuanning, Bertrand Mansion (golgote), James Hurst, Patrick Crosby, Bill Donahue, Jiang Dapao, and others.

This module embeds the Lua interpreter or LuaJIT into the nginx core and integrates the powerful Lua threads (aka Lua coroutines) into the nginx event model by means of nginx subrequests.

Unlike Apache's mod_lua and Lighttpd's mod_magnet, Lua code written atop this module can be 100% non-blocking on network traffic as long as you use the ngx.location.capture or ngx.location.capture_multi interfaces to let the nginx core do all your requests to mysql, postgresql, memcached, redis, upstream http web services, and etc etc etc.

This module is also included and enabled by default in our ngx_openresty bundle: http://openresty.org/

You can find the complete documentation for this module on the following wiki page:

    http://wiki.nginx.org/HttpLuaModule

And you always get the latest source code from the git repository here:

    https://github.com/chaoslawful/lua-nginx-module

Have fun!

ngx_openresty 1.0.4.2 released

2011-08-09T15:44:00.001+08:00

I'm happy to announce that ngx_openresty 1.0.4.2 has been released:

http://openresty.org/#Download

This release features the ability to be built and run on FreeBSD 8.2 and Solaris 11, even with LuaJIT enabled. Our ngx_set_misc, ngx_lua, and ngx_drizzle modules have been upgraded to their latest versions, with both bug fixes and new features. New options has also been added to the ./configure script, like --with-libpq, --with-pg_config, --with-make and --with-no-pool-patch. You can find the complete change log on the following page:

http://openresty.org/#ChangeLog1000004

The devel release 1.0.5.0rc1 is the first (pre)release that includes the latest nginx stable version 1.0.5.

OpenResty (aka. ngx_openresty) is a full-fledged web application server by bundling the standard Nginx core, lots of 3rd-party Nginx modules, as well as most of their external dependencies.

By taking adantage of various well-designed Nginx modules, OpenResty effectively turns the nginx server into a powerful web app server, in which the web developers can use the Lua programming language to script various existing nginx C modules and Lua modules and construct extremely high-performance web applications that is capable to handle 10K+ connections.

You can find out more information on the OpenResty web site

http://openresty.org/

Have fun!

A sample for a dynamic request router based on ngx_lua + ngx_redis2

2011-07-27T22:02:00.001+08:00

I wrote and tested a running sample that demonstrates how to use redis to route incoming requests to different HTTP backends based on the requests' User-Agent header, as a response to an email on the nginx mailing list:

http://openresty.org/#DynamicRoutingBasedOnRedis

Scripting the nginx core and various 3rd-party nginx modules with Lua is a lot of fun and the possibilities are unlimited :)

I'm going to add more and more samples and demo apps on the wiki there ;)

Have fun!

ngx_redis2 v0.07: nginx upstream module for the full redis 2.0 protocol

2011-07-22T12:26:00.001+08:00

Hi, folks!

I'm happy to announce that ngx_redis2 v0.07 has now been released!

You can download the release tarball from its download page:

https://github.com/agentzh/redis2-nginx-module/downloads

This is an nginx upstream module that makes nginx talk to a redis 2.x server in a non-blocking way. The full Redis 2.0 unified protocol has been implemented including the redis pipelining support.

This module returns the raw TCP response from the redis server. It's recommended to use my lua-redis-parser module (written in pure C) to parse these responses into lua data structure when combined with the ngx_lua module:

https://github.com/agentzh/lua-redis-parser

If you only want to use the "get" redis command, you can try out the ngx_redis module here:

http://wiki.nginx.org/HttpRedis

It returns the parsed content part of the redis response because only "get" is needed to implement.

Another option is to parse the redis responses on your client side yourself.

You can find the full documentation on its GitHub project page:

https://github.com/agentzh/redis2-nginx-module

Here goes the complete change log for this release:

now we have implemented redis command pipelining by using multiple redis2_query directives in a single location. also updated the document to reflect this change.
implemented the new redis2_raw_queries directive which supports multiple pipelined redis commands in a single TCP query.
fixed github issue #5: CRLF in chunk data confused the response parser when packet segmentation happens. thanks dlogar.
fixed an issue in the multi-bulk reply parser that only chunks were allowed as elements. now single line replies are all allowed, but recursive multi-bulk replies are still not allowed.
fixed an issue regarding defining global variables in C header files: we should have defined the global ngx_http_redis2_module in a single compilation unit. thanks Jiang Dapao.
documented the redis2_next_upstream directive as per Lance.
documented the limited redis pub/sub feature support.
documented the various timeout config directives. thanks Mike Ferrier.
now we only issue a warning when there is left-over bytes in the redis response stream.
added a memory allocation failure check. (calio)
fixed a typo in the source; it does not affect runtime results though. thanks calio for catching it.
fixed one spot of unused-but-set-variable warning issued by gcc 4.6.
now we allow empty parameters in the redis2_query directive.
added a performance tuning section as per HowToMeetLadies.
fixed a typo in the source code reported by Rares Mirica.

Enjoy!
-agentzh

ngx_srcache v0.12: new config directives and bug fixes

2011-07-11T12:58:00.001+08:00

I'm happy to announce that ngx_srcache v0.12 has just been released. You can download the release tarball from the download page below:

http://github.com/agentzh/srcache-nginx-module/downloads

Special thanks go to Andre (shoki) for contributing a lot for this release :)

Here goes the complete change log:

now we properly support fetch/store subrequests with internal redirection in them. main requests with internal redirection will still not be stored into the cache if there is a cache miss. thanks Liseen Wan for reporting it.
fixed spots that trigger the unused-but-set-variable warning by gcc 4.6.
added srcache_store_skip and srcache_fetch_skip directives to skip cache fetching or storing based on variables that are set and not empty nor 0. thanks Andre. Examples of using Lua to set $nocache to avoid storing URIs that contain /tmp:

set_by_lua $nocache '
if string.match(ngx.var.request_uri, "/tmp") then
return "true"
else
return ""
end';

srcache_store_skip $nocache;

added new directive srcache_store_max_size. thanks Andre.
made our filter optimization work with nginx HUP by clearing the ngx_http_srcache_used flag at nginx pre-config callback. thanks Marcus Clyne.
now we skip NULL chains in our output filters and also removed the SUBREQUEST_IN_MEMORY flag for our srcache_store subrequests because it will cause mysterious hanging issues when memcached returns CLIENT_ERROR for "get".

The ngx_srcache module provides a transparent caching layer for arbitrary nginx locations (like those use an upstream or even serve static disk files). Usually, the ngx_memc module is used together with this module to provide a concrete caching storage backend. But technically, any modules that provide a REST interface can be used as the fetching and storage subrequests used by ngx_srcache, for example, ngx_redis2.

For main requests, the srcache_fetch directive works at the end of the access phase, so the standard access module's "allow" and "deny" direcives run *before* ours, which is usually the desired behavior for security reasons.

You can find the complete documentation on the project page hosted on GitHub:

http://github.com/agentzh/srcache-nginx-module

This module has been heavily used in our production environment for more than a year, as illustrated in the following chart:

http://agentzh.org/misc/slides/perl-lz-apps/index.html#23

Several outside users have also reported success in their production use.

Also, this module is included and enabled by default in our ngx_openresty bundle: http://openresty.org

Enjoy!

ngx_memc v0.12: improvements in the last 8 months

2011-07-08T16:56:00.001+08:00

I've just released ngx_memc v0.12, which includes various improvements in the last 8 months:

http://github.com/agentzh/memc-nginx-module/downloads

Here's the complete change log for this release:

fixed the spots that trigger the unused-but-set-variable warning by gcc 4.6.
added more debug information when memcached sends back "invalid" responses.
we now document the timeout units properly. it should default to seconds.
now we use the 2-clause bsd license.
added an error message when no upstream backend is found in "memc_pass $backend".

The ngx_memc module extends the standard memcached module to support almost the whole memcached ascii protocol. And it can be used with either the standard memcached server or other backends supporting the memcached wire protocol like TokyoTyrant.

Maxim Dounin's excellent ngx_http_upstream_keepalive module can also be used with this module to provide a powerful connection pool for memcached.

See ngx_memc's wiki page for more details:

http://wiki.nginx.org/HttpMemcModule

This module is included and enabled by default in our ngx_openresty bundle: http://openresty.org

Enjoy!

ngx_echo v0.36: bugfixes in the last 5 months

2011-07-08T16:14:00.001+08:00

I'm happy to announce the v0.36 release of the ngx_echo module, which includes the various bugfixes in the last 5 months.

You can get the release tarball from the download page below

https://github.com/agentzh/echo-nginx-module/downloads

Here's the complete change log for this release:

now we back-ported the subrequest mechanism of ngx_lua to ngx_echo. this also helps some crazy test cases of deep-recursively mixing echo_location and echo_location_async pass now.
now echo_location and its friends can work with ngx_xss (as well as other output filter modules) completely. thanks wd for reporting this issue.
done some minor optimization when modifying subrequest's content-length header.
now we always pad a trailing \0 to filepath in "echo_subrequest(_async) METHOD /path -f filepath" because ngx_open_cached_file requires a C string file name. thanks dr-dr xp.
made our filter optimization works with nginx HUP by clearing the ngx_http_echo_filter_used flag at nginx pre-config callback. thanks Marcus Clyne.

The echo module wraps lots of Nginx internal APIs for streaming input and output, parallel/sequential subrequests, timers and sleeping, as well as various meta data accessing. You can read the full documentation on the following wiki page:

http://wiki.nginx.org/HttpEchoModule

and you can always get the latest source code from GitHub:

http://github.com/agentzh/echo-nginx-module

The ngx_echo module is included and enabled by default in our ngx_openresty bundle: http://openresty.org

Happy echoing!

Best regards,
-agentzh

Test::Nginx 0.16 released to CPAN

2011-07-07T17:25:00.001+08:00

I've just uploaded Test::Nginx 0.16 to CPAN:

http://search.cpan.org/dist/Test-Nginx/

It will appear on the CPAN mirror near you in the next few hours or so.

This Perl module provides a test scaffold based on non-blocking IO::Socket or LWP for automated testing in Nginx C module development.

This class inherits from Test::Base, thus bringing all its declarative power to the Nginx C module testing practices.

Please check out the full documentation on CPAN:

http://search.cpan.org/perldoc?Test::Nginx::Socket

All of our Nginx modules are using Test::Nginx to drive their test suites.

Please note that this is different from the Test::Nginx module used by Maxim Dounin.

Antoine BONAVITA and Piotr Sikora have contributed a lot to this project. Special thanks go to them :)

Here's the complete change log for this release:

fixed a bug that length(undef) produced lots of warnings on perl 5.10.x. thanks Guo Ying (shrimp).
now we automatically retry connecting for 3 times (1 sec delay) in Test::Nginx::Socket::send_request. this will help automatically recovering from random slow valgrind loading problems.
when the directories specified in --- user_files do not exist, Test::Nginx will create it automatically.
now we'll issue warning when there is extra bytes after the last chunk in the chunked response parser.

Enjoy!

ngx_headers_more v0.15: bug fixes in the past 6 months

2011-07-06T12:12:00.001+08:00

I'm happy to announce the v0.15 release of our ngx_headers_more module which features some minor optimizations and bug fixes. You can download the tarball from the download page below:

http://github.com/agentzh/headers-more-nginx-module/downloads

Here's the complete change log for this version:

now more_set_headers supports overriding charset in Content-Type. thanks ML for reporting it.
fixed an issue in more_clear_headers: we should remove all the instances of the headers specified, not only the first occurrence. thanks Li Yang for reporting it.
back-ported a bugfix from ngx_lua: in output header set, we should always set the header->hash to 1. thanks moodydeath for reporting it.
fixed a bug when clearing the Accept-Ranges header. thanks Bo Blangstrup.

ngx_rds_json v0.11: compact JSON output support

2011-07-06T11:32:00.001+08:00

I'm glad to announce that ngx_rds_json v0.11 has been released:

https://github.com/agentzh/rds-json-nginx-module/downloads

This release includes the new directive "rds_json_format compact" to emit very compact JSON like

[["id","name","age"],[1,"marry",18],[2,"bob",23],[3,"tom",9]]

instead of the more verbose default form

[{"id":1,"name":"marry","age":18},
{"id":2,"name":"bob","age":9},
{"id":3,"name":"tom","age":9}]

The ngx_rds_json module provides an output filter that can format the RDS outputs generated by ngx_drizzle and ngx_postgres modules to JSON.

You can find the complete documentation on its project page:

https://github.com/agentzh/rds-json-nginx-module

Have fun!

ngx_drizzle v0.1.0 released: now runs with mysql 5.5+ w/o patching libdrizzle

2011-07-05T18:08:00.001+08:00

I'm happy to announce the ngx_drizzle module's v0.1.0 release, which includes lots of bug fixes and new features accumulated in the last 8 months:

https://github.com/chaoslawful/drizzle-nginx-module/downloads

The most exciting thing about this release is that we can now work with the latest libdrizzle 1.0 without patches on at least Linux and Mac OS X. And we can now connect to MySQL 5.5+ and we're now more stable under load.

This is an nginx upstream module integrating libdrizzle (https://launchpad.net/drizzle ) into nginx in an non-blocking and streamming way.

Essentially it provides a very efficient and flexible way for nginx internals to access mysql, drizzle, as well as other RDBMS's that support the drizzle protocol or mysql protocol. Also it can serve as a direct REST interface to those RDBMS backends. Alternatively it can be invoked by other modules like ngx_lua via subrequests.

It also has a builtin per-worker connection pool mechanism.

Check out its project page for the full documentation:

https://github.com/chaoslawful/drizzle-nginx-module

Here goes the complete change log for this release:

now we require at least libdrizzle 1.0, which supports official mysql 5.5+ and is much more stable under load. thanks Taylor Weibley for pushing this. We no longer require patching libdrizzle any more.
fixed a compilation issue on Mac OS X: we should include drizzle.h prior to nginx headers, or we will not get the "bool" type properly installed on Mac OS X.
fixed the spots that trigger -Wunused-but-set-variable by gcc 4.6.
fixed the duplicate last chunk issue: we should not set the last_buf flag ourselves in drizzle_output.c because ngx_http_upstream already sends a last buf for us.
ported over Maxim Dounin's patch for ngx_http_upstream_keepalive connection pool fixes: we should have discarded stale read events for cached tcp connections in the pool.
implemented the new drizzle_status directive to provide connection pool status monitoring capability.
fixed a minor bug in the connection pool: we should resume the "name", "sockaddr", and "socklen" fields for the connection from the pool such that we can get more detailed error log messages with the "upstream: drizzle://ip.add.re.ss:port" bit.
implemented the $drizzle_thread_id variable which is automatically set when mysql/drizzle times out.
now we use the 2-clause bsd license.
report an error message when upstream name not found for drizzle_pass.
now we implemented the charset option for the "drizzle_server" diredctive which causes ngx_drizzle send "set names xxx" automatcially for every connection to that drizzle server.
added a lot of more documentation.

Enjoy!
-agentzh

Slides for my nginx app talk in Beijing Perl Workshop 2011

2011-07-05T17:12:00.000+08:00

I gave an nginx talk in Beijing Perl Workshop 2011, which titles "Applications of ngx_openresty and Perl in lz.taobao.com". I showed how our nginx modules are used to build our company products.

You can view the (English and Chinese) slides directly in your web browser:

http://agentzh.org/misc/slides/perl-lz-apps/

Please use the arrow keys or pageup/pagedown keys on your keyboard to switch pages.

Alternatively, you can download the slides tarball and browse offline: http://agentzh.org/misc/slides/perl-lz-apps.tar.gz

You can find the slides for my other nginx talks on the page as well:

http://agentzh.org/#Presentations

Have fun!

ngx_lua v0.2.0 is now released with lots of bug fixes

2011-07-05T13:26:00.001+08:00

After 5 months' active development, the ngx_lua module v0.2.0 release is now finally released:

https://github.com/chaoslawful/lua-nginx-module/downloads

This release contains lots of bug fixes as well as some new features. Many many thanks go to all our users and contributors worldwide, especially moodydeath :)

The ngx_lua module embeds the Lua/LuaJIT interpreter into the nginx core and integrates the powerful Lua threads (aka Lua coroutines) into the nginx event model by means of nginx subrequests.

You can get the latest source code and full documentation from the ngx_lua's project page:

https://github.com/chaoslawful/lua-nginx-module/

This module is now included in our ngx_openresty bundle: http://openresty.org/

Here goes the complete change log for this release:

now we support ngx.var[1], ngx.var[2], and etc to refer to the nginx regex capturing variables \$1, \$2, and etc in Lua. this resolved github issue #43. thanks Tobia Conforto for reporting it.
now we use the same value overriding mechanism as ngx_rewrite's set command for ngx.var.VAR = new_value. Assigning values to special variables like $limit_rate and $args should now work; also writing to built-in variables that are not changeable (like $arg_PARAMETER) will result in a 500 error page, as expected, now. thanks Richard Kearsley for reporting it.
fixed the lua_code_cache off warning when the lua_code_cache is explicitly on. thanks Feng Xingguo.
applied the patch from cyberty to add ngx.http_time() function to expose the nginx core function ngx_http_time to the Lua land.
fixed an issue on i386: we now use off_t consistently. mixing it with size_t on 32-bit systems can cause Bad Things. this fixed github issue #42. thanks moodydeath.
fixed an issue on i386: fixed a formatter mismatch issue in ngx_http_echo_adjust_subrequest. thanks Wang Bin. This caused incorrect subrequest Content-Length header when a body is specified.
now in the subrequest capturing processor, we worked around an issue in ngx_http_static_module that when it issues 301 redirect for directory access w/o a trailing slash, it does not inject r->headers_out.location into the r->headers_out.headers list. thanks moodydeath for reporting it in the discussion of github issue #41.
fixed a bug in ngx.location.capture() and ngx.location.capture_multi() that we could not capture locations with internal redirections in them. thanks moodydeath for reporting it in github issue #41.
fixed redundant last chunk issue for ngx.exec() invocation at rewrite and access phases: we should quit the current core_run_phases cycle; this also fixed github issue #40: 2 Subrequest calls when using access_by_lua, ngx.exec and echo_location.
fixed ngx.exit(status) where status >= 200 and status < 300 for access_by_lua* and rewrite_by_lua*: it should quit the whole request altegother and skip all those subsequent phase handlers (if any). thanks moodydeath for reporting it.
fixed github issue #39: setting differnt response headers in Lua with common prefix might interfere with each other. thanks moodydeath.
fixed GitHub issue #38: request headers did not forward to subrequests when the "method" or "body" option is explicitly specified by a non-nil value for ngx.location.capture(). thanks Richard Kearsley.
fixed a bug in output header set; we should always set the header->hash to 1. thanks moodydeath for reporting it.
fixed spots that trigger the "variable set but not used" warning issued by gcc 4.6.0.
now we turn the ngx.req.header table into an ngx.req.get_headers() function; we also added ngx.req.set_header(name, value) and ngx.req.clear_header(name). thanks moodydeath.
now we make ngx_devel_kit (NDK) optional. thanks Kirill A. Korinskiy.
removed a duplicate definition of the ngx_str_set macro caught by ctags; also fixed a warning thrown by gcc -O3 on Mac OS X 10.6.
added patch to use PCRE related Lua extensions in ngx_lua (chaoslawful)
now we change the way we process HTTP 1.0 requests by automatically buffering all the user outputs generated by ngx.print()/ngx.say() calls, which is much more natural than the old broken way.
fixed the "ngx.exec() after ngx.location.capture() hanging" bug for rewrite_by_lua* and access_by_lua* as well. thanks Wendal Chen.
applied a patch from moodydeath to introduce the "ngx.is_subrequest" attribute.
now we encourage use of the client_body_in_single_buffer directive instead of big client_body_buffer_size when lua_need_request_body is turned on.
fixed the config script and added extra linking options needed by LuaJIT in 64-bit Mac OS X.
fixed the zero size alert caused by ngx.print("") in Lua.
now we always allocate r->request_body for subrequests when the method option is specified for ngx.location.capture*. this prevents accidental inheritance of parent request's request body when client_body_buffer_size < client_max_body_size.

How nginx "location if" works

2011-03-08T14:35:00.001+08:00

Nginx's if directive does have some weirdness in practice. And people may misuse it when they do not have enough knowledge about its behavior. In this post, I'll analyze some examples here such that people may get some light and use it correctly.

In short, Nginx's "if" block effectively creates a (nested) location block and once the "if" condition matches, only the content handler of the inner location block (i.e., the "if" block) will be executed.

Case 1

location /proxy {
set $a 32;
if ($a = 32) {
set $a 56;
}
set $a 76;
proxy_pass http://127.0.0.1:$server_port/$a;
}

location ~ /(\d+) {
echo $1;
}

Calling /proxy gives 76 because it works in the following steps:

1. Nginx runs all the rewrite phase directives in the order that they're in the config file, i.e.,

set $a 32;
if ($a = 32) {
set $a 56;
}
set $a 76;

and $a gets the final value of 76.

2. Nginx traps into the "if" inner block because its condition $a = 32 was met in step 1.

3. The inner block does not has any content handler, ngx_proxy inherits the content handler (that of ngx_proxy) in the outer scope (see src/http/modules/ngx_http_proxy_module.c:2025).

4. Also the config specified by proxy_pass also gets inherited by the inner "if" block (see src/http/modules/ngx_http_proxy_module.c:2015)

5. Request terminates (and the control flow never goes outside of the "if" block).

That is, the proxy_pass directive in the outer scope will never run in this example. It is "if" inner block that actually serves you.

Let's see what happens when we override the inner "if" block's content handler with out own:

Case 2

location /proxy {
set $a 32;
if ($a = 32) {
set $a 56;
echo "a = $a";
}
set $a 76;
proxy_pass http://127.0.0.1:$server_port/$a;
}

location ~ /(\d+) {
echo $1;
}

You will get this while accessing /proxy:

a = 76

Looks counter-intuitive? Oh, well, let's see what's happening this time:

1. Nginx runs all the rewrite phase directives in the order that they're in the config file, i.e.,

set $a 32;
if ($a = 32) {
set $a 56;
}
set $a 76;

and $a gets the final value of 76.

2. Nginx traps into the "if" inner block because its condition $a = 32 was met in step 1.

3. The inner block does has a content handler specified by "echo", then the value of $a (76) gets emitted to the client side.

4. Request terminates (and the control flow never goes outside of the "if" block), as in Case 1.

We do have a choice to make Case 2 work as we like:

[Case 3]

location /proxy {
set $a 32;
if ($a = 32) {
set $a 56;
break;

echo "a = $a";
}
set $a 76;
proxy_pass http://127.0.0.1:$server_port/$a;
}

location ~ /(\d+) {
echo $1;
}

This time, we just add a break directive inside the if block. This will stop nginx from running the rest ngx_rewrite directives. So we get

a = 56

So this time, nginx works this way:

1. Nginx runs all the rewrite phase directives in the order that they're in the config file, i.e.,

set $a 32;
if ($a = 32) {
set $a 56;
break;
}

and $a gets the final value of 56.

2. Nginx traps into the "if" inner block because its condition $a = 32 was met in step 1.

3. The inner block does has a content handler specified by echo, then the value of $a (56) gets emitted to the client side.

4. Request terminates (and the control flow never goes outside of the "if" block), just as in Case 1.

Okay, you see how ngx_proxy module's config inheritance among nested locations take the key role here, and make you believe it works the way that you want. But other modules (like echo mentioned in one of my earlier emails) may not inherit content handlers in nested locations (in fact, most content handler modules, including upstream ones, don't).

And one must be careful about bad side effects of config inheritance of "if" blocks in other cases, consider the following example:

Case 4

location /proxy {
set $a 32;
if ($a = 32) {
return 404;
}
set $a 76;
proxy_pass http://127.0.0.1:$server_port/$a;
more_set_headers "X-Foo: $a";
}

location ~ /(\d+) {
echo $1;
}

Here, ngx_header_more's more_set_headers will also be inherited by the implicit location created by the "if" block. So you will get:

$ curl localhost/proxy
HTTP/1.1 404 Not Found
Server: nginx/0.8.54 (without pool)
Date: Mon, 14 Feb 2011 05:24:00 GMT
Content-Type: text/html
Content-Length: 184
Connection: keep-alive
X-Foo: 32

which may or may not what you want :)

BTW, the add_header directive will not emit an X-Foo header in this case, and it does not mean no directive inheritance happens here, but add_header's header filter will skip 404 responses.

You see, how tricky it is behind the scene! No wonder people keep saying "if is evil".

We've been using the ngx_lua module to do such complicated nginx.conf branching (and also the whole application's business logic) in Lua. Lua's "if" is not evil anyway.

For ngx_lua's set_by_lua directive, there's even no Lua coroutine overhead (though the overhead itself is very small).

Please note that I did not say that you should never use nginx's "if". Don't take me wrong. My motivation of writing this explanation of the underlying mechanism is to help you use it correctly and wisely ;)

I think Igor Sysoev will redesign the whole rewrite module in his nginx 2.0 devel branch. Then everything here will be changed.

P.S. This article was originally posted to this nginx mailing list thread: http://forum.nginx.org/read.php?2,174917

ngx_lua v0.1.5: ability to capture multiple parallel subrequests

2011-02-09T17:59:00.000+08:00

I'm pleased to announce that ngx_lua v0.1.5 is now released. You can download the release tarball from the download page:

    https://github.com/chaoslawful/lua-nginx-module/downloads

This version of the ngx_lua module implements the ngx.location.capture_multi method that can issue multiple nginx subrequests at the same time, capture their responses along the way, and then return the results in the same order that they're specified in the arguments.

Here is a small example:

    location ~ '^/echo/(\w+)$' {
        echo -n $1;
    }

    location = /main {
        content_by_lua '
            local res1, res2, res3 =
                ngx.location.capture_multi{
                    { "/echo/sun" },
                    { "/echo/moon" },
                    { "/echo/earth" },
                }
            ngx.say(res1.body)
            ngx.say(res2.body)
            ngx.say(res3.body)
        ';
    }

then accessing /main gives

    sun
    moon
    earth

and those three subrequests, GET /echo/sun, GET /echo/moon, and GET /echo/earth, were issued at the same time.

Below is a more complicated example that fires 2 concurrent MySQL queries to calculate the count of dogs and the count cats, respectively, at the same time, and finally output the sum of these 2 counts, i.e., the total count of all the animals.

First of all, we define the remote mysql server upstream in the main http block:

    upstream mysql_node {
        drizzle_server 10.32.136.5:3306
            user=monty password=some_pass dbname=main protocol=mysql;
        drizzle_keepalive max=100 mode=single overflow=ignore;
    }

then we define a general-purpose internal location to proxy SQL queries to remote MySQL node in our default virtual host server block:

    location = /mysql {
        internal;
        drizzle_query $echo_request_body;
        drizzle_pass mysql_node;
        rds_json on;
    }

after that, we define our main location that does the job:

    location /main {
        content_by_lua '
            local opts1 = {
                method = ngx.HTTP_POST,

body = "select sum(1) cnt from cats",

            }
            local opts2 = {
                method = ngx.HTTP_POST,

body = "select sum(1) cnt from dogs",

            }
            local res1, res2 =
                ngx.location.capture_multi{
                    { "/mysql", opts1 },
                    { "/mysql", opts2 }
                }

            local yajl = require("yajl")
            local cats = yajl.to_value(res1.body)
            local dogs = yajl.to_value(res2.body)

            local total = cats[1].cnt + dogs[1].cnt
            ngx.say("for total ", total, " animals.")
        ';
    }

For brevity, I've omitted all the error handling code like checking if res1.status is 200 and validating cats.errcode has nil values ;)

The nginx modules ngx_drizzle, ngx_rds_json, and the lua-yajl Lua library are also used in the sample above.

This example can be trivially extended to run tens or even more sql queries to different MySQL machines and combined with more PostgreSQL database servers. All in parallel and all are 100% non-blocking on network traffic.

This v0.1.5 version of ngx_lua also contains

optimizations on subrequest buffer management and thus saving much memory when using ngx.location.capture (and ngx.location.capture_multi).
fixes of the build system that we no longer require OpenSSL and ngx_lua no longer crash when used with statically linked OpenSSL. Thanks Marcus Clyne and Vladislav Manchev.
fixes of the Lua code cache handling that we no longer clear pre-loaded standard Lua packages like "table" and "string" when the Lua code cache is off.

The ngx_lua module embeds the Lua/LuaJIT interpreter into the nginx core and integrates the powerful Lua threads (aka Lua coroutines) into the nginx event model by means of nginx subrequests.

You can get the latest source code and full documentation from the ngx_lua's project page:

https://github.com/chaoslawful/lua-nginx-module/

Happy Lua and nginx.conf hacking!

ngx_echo v0.35: ability to POST/PUT local files as subrequest bodies

2011-02-07T18:58:00.001+08:00

I'm happy to announce the v0.35 release of the ngx_echo module, which features the ability to POST/PUT location files as subrequest bodies via the -f option of the echo_subrequest and echo_subrequest_async directives.

You can get the release tarball from the download page below

https://github.com/agentzh/echo-nginx-module/downloads

This new feature was implemented by Bernd Dorn. Many thanks to him ;)

Bernd uses the nginx upload module to upload big files and he wants to issue a PUT subrequest with the uploaded file to store the file somewhere else based on its hash (hash generation is done by the upload module). So it's nice to have something like a file option to replace the "-b" option in the echo_subrequest directive like this:

echo_subrequest PUT /some_upstream/$upload_file_md5 -f $upload_tmp_path

Why nginx does not have an API reference

2011-01-26T15:44:00.011+08:00

People keep asking me (and others) why there is no API reference for nginx internals. The answer is simple: it is still in flux and changing even in nginx's stable releases.

For this reason, I won't work on such a reference and I bet Igor Sysoev and Maxim Dounin will not either. (Feel free to prove me wrong! :))

Nginx 0.8.54 changed the meaning of the return values of rewrite-phase handlers, which made Valery's guide to phases out of date even before it was written:

http://www.nginxguts.com/2011/01/phases/

See my comment at the bottom of that article.

This is not the only API change that is not backward compatible, rather, just a small example among many others.

FWIW, the rewrite phase handlers do not even honor NGX_DONE nor NGX_AGAIN in the version range of 0.8.42 ~ 0.8.53. My mail to this list reporting this regression got completely ignored:

http://forum.nginx.org/read.php?29,103078,103078

Is it funny? Sure not ;)

We've been adding more and more #if/#else craps to our modules' code just in the hope of being compatible with more and more versions of nginx. Let's count how many version checks in ngx_echo:

$ ack '#if' src/|grep nginx_version|wc -l
11

and that's the cost of making ngx_echo work with nginx 0.7.21 ~ 0.9.4.

And the number is similar for ngx_lua:

$ ack '#if' src/|grep nginx_version|wc -l
10

Well, one has to invest enough time on the nginx source and trace every new release carefully enough and prepare for API changes that may make your module even fail to compile :) Most of the API changes will not appear in the change log of the new nginx releases. Someone (IIRC, it was Marcus Clyne) had requested a change log for nginx C API changes on the list, but got ignored as always :)

Anyway, this article should not be taken as a complaint, rather, it is just a report of the current status of nginx 3rd-party module/patch development. Yeah, it's the reality. If this post makes you feel the other way around, I apologize for that.

We're currently maintaining dozens of nginx 3rd-party modules most of which were born in the 0.8.x era. Just some of them have been open-sourced. The maintenance work takes time, efforts and patience :) But still we've been trying very hard to optimize this process for fun, not only for maintaining existing modules but also for creating brand new ones ;)

Hope you good luck!

ngx_headers_more v0.14: minor optimizations and fixes

2011-01-25T13:10:00.002+08:00

I'm happy to announce the v0.14 release of our ngx_headers_more module which features some minor optimizations and bug fixes. You can download the tarball from the download page below:

http://github.com/agentzh/headers-more-nginx-module/downloads

Here's the complete change log for this version

now we postpone the rewrite phase handler only once rather than on every main request previously. this will save some CPU cycles on every request if more_set_input_headers or more_clear_input_headers are used.
fixed two spots where we did not check against null pointers when out of memory.
now we use the 2-clause bsd license instead.
various coding style fixes.

ngx_lua v0.1.4: ability to disable the Lua code cache

2011-01-24T18:30:00.002+08:00

I'm delighted to announce the ngx_lua v0.1.4 release which features the ability to disable the Lua code cache to ease Lua development. You can get the release tarball from the download page:

https://github.com/chaoslawful/lua-nginx-module/downloads

By default, Lua code loaded into the ngx_lua module will be cached (the compiled/JIT'd form will be cached). So changing .lua files requires reloading or restarting the nginx server. From this release, we can use the following directive to disable the code cache:

lua_code_cache off;

Then the Lua files referenced in set_by_lua_file, content_by_lua_file, access_by_lua_file, and rewrite_by_lua_file won't be cached at all, and Lua's "package.loaded" table will be cleared at every request's entry point (such that Lua modules won't be cached either). Developers can now enjoy the PHP-way, i.e., edit-and-refresh.

But please note that Lua code inlined into nginx.conf like those specified by set_by_lua, content_by_lua, access_by_lua, and rewrite_by_lua will *always* be cached because only nginx knows how to parse nginx.conf and the only way to tell it to re-load the config file is to send a HUP signal to it or just to restart it from scratch.

For now, ngx_lua does not support the "stat" mode like Apache's mod_lua, but we will work on it in the future.

This release also contains a small over-all optimization that the keys used for Lua code cache lookups are pre-calculated at config time wherever possible.

The ngx_lua module embeds the Lua/LuaJIT interpreter into the nginx core and integrates the powerful Lua threads (aka Lua coroutines) into the nginx event model by means of nginx subrequests.

You can get the latest source code and full documentation from the ngx_lua's project page:

https://github.com/chaoslawful/lua-nginx-module/

Have fun!

ngx_lua v0.1.3: improved Solaris and Mac portability

2011-01-18T12:36:00.002+08:00

I'm delighted to announce that ngx_lua v0.1.3 has been released.

You can download the tarball from the following download page:

https://github.com/chaoslawful/lua-nginx-module/downloads

In this version, we've improved the config-time detection logic for Solaris and Mac, and there's success report on Solaris 11. Thanks the great help from Vladislav Manchev.

There's also some work for NetBSD/OpenBSD/FreeBSD systems from Piotr Sikora. *BSD folks are very welcome to try it out ;)

Our main development and testing environment is 32-bit and 64-bit Linux though. Help for porting this module to more (exotic) systems will be highly appreciated as usual :)

The ngx_lua module embeds the Lua/LuaJIT interpreter into the nginx core and integrates the powerful Lua threads (aka Lua coroutines) into the nginx event model by means of nginx subrequests.

chaoslawful is currently working on the "cosocket" branch of ngx_lua that will emulate a common set of Lua socket API that will give you totally transparently non-blocking capability out of the box by means of a brand new upstream layer atop the nginx event model and no nginx subrequest overheads.

You can get the latest source code and full documentation from the ngx_lua's project page:

https://github.com/chaoslawful/lua-nginx-module/

Happy Lua and nginx.conf hacking!

ngx_lua v0.1.2: rewrite_by_lua/access_by_lua fixes and new features

2011-01-13T20:01:00.001+08:00

I'm happy to announce the v0.1.2 release of our ngx_lua module. You

can download the release tarball from the module's download page:

http://github.com/chaoslawful/lua-nginx-module/downloads

This release contains the following changes:

Fixed a bug in rewrite_by_lua (introduced in ngx_lua v0.1.0) andaccess_by_lua (introduced in ngx_lua v0.1.1) that it always sent out default response headers even if there's no user outputs in the rewrite/access Lua handlers. Thanks Roman Vasilyev.
The results returned from "ngx.location.capture" now support multi-value headers, like "Set-Cookie". The value is a Lua(array) table that holds all the values in the order that they appear.

For instance, if the subrequest response headers contains the following lines:

   Set-Cookie: a=3
   Set-Cookie: foo=bar
   Set-Cookie: baz=blah

Then res.header["Set-Cookie"] will be evaluted to the table value {"a=3", "foo=bar", "baz=blah"}. Thanks neilljordan ( https://github.com/neilljordan/ ).

rewrite_by_lua_file, access_by_lua_file, and content_by_lua_file now support nginx variables in the path value. For example:

   # use nginx var in code path
   # WARN: contents in nginx var must be carefully filtered,
   # otherwise there'll be great security risk!
   location ~ ^/app/(.+) {
   content_by_lua_file /path/to/lua/app/root/$1.lua;
   }

   Thanks Xiaozhe Wang (chaoslawful).

Fixed support for HTTP 1.0 requests.

Sometimes you may want to use nginx's standard ngx_proxy module to proxy requests to another nginx machine configured by a location with content_by_lua. Because proxy_pass only supports the HTTP 1.0 protocol, we have to know the length of your response body and set the Content-Length header before emitting any data out. ngx_lua will automatically recognize HTTP 1.0 requests and try to send out an appropriate Content-Length header for you, at the first invocation of ngx.print() and ngx.say(), assuming all the response body data is in a single call of ngx.print() or ngx.say().

So if you want to support HTTP 1.0 clients like ngx_proxy, do not call ngx.print() or ngx.say() multiple times, try buffering the output data yourself wherever needed. This behavior is the same as the ngx_echo module.

Please also note that rewrite_by_lua will *not* work with nginx 0.8.41 ~ 0.8.53.

Because the rewrite_by_lua and access_by_lua directives are very new, be prepared to hit some hidden bugs. And you're very welcome to report any issues that you meet on your side.

ngx_lua is an nginx C module that embeds the Lua or LuaJIT interpreter into the nginx core and you can find the latest source code and the complete documentation here

https://github.com/chaoslawful/lua-nginx-module

Have fun!

My fork of ngx_eval 2011.01.12 released

2011-01-13T19:44:00.001+08:00

People have been reporting issues of my fork of ngx_eval when using it with recent versions of nginx. I've updated it to version 2011.01.12 to compatible with nginx 0.8.54+:

https://github.com/agentzh/nginx-eval-module/downloads

Now "eval_subrequest_in_memory" is off by default. Because subrequest_in_memory mode is still having issues.

Here's a small example for using ngx_eval with my ngx_redis2, an nginx upstream module that supports almost the full Redis 2.0 unified wire protocol:

location /foo {
eval_override_content_type 'application/octet-stream';
eval_subrequest_in_memory off;
eval $res {
redis2_literal_raw_query 'set one 5\r\nfirst\r\n';
redis2_pass 127.0.0.1:6379;
}
echo [$res];
}

Accessing /foo with curl yields

[+OK
]

on my side.

Please ensure that you're using nginx 0.8.54+ and the latest version of my fork of ngx_eval (at lease 2010.12.29+ is required).

If you find any issues, please let me know.

Special thanks go to Valery Kholodkov for creating ngx_eval in the first place.

P.S. Still, it's recommended to use ngx_lua module's rewrite_by_lua or access_by_lua directives combined by the "ngx.location.capture" API to do what my fork of ngx_eval has been doing.

ngx_lua v0.1.1: access_by_lua and access_by_lua_file landed

2011-01-13T19:33:00.001+08:00

After the first public release of ngx_lua, I'm proud to annouce the v0.1.1 version, which has introduced the access_by_lua and access_by_lua_file directives:

https://github.com/chaoslawful/lua-nginx-module/downloads

Now we can code up our own nginx access-phase handlers directly in pure Lua, with all the capabilities with rewrite_by_lua and content_by_lua, like firing up subrequests to other locations, reading/writing nginx variables, changing response headers, internal redirection, HTTP 301/302 redirection, throwing out 403/500/etc error pages, and etc etc etc.

Here's a small example that emulates ngx_auth_request's functionality in pure Lua:

location / {
access_by_lua '
local res = ngx.location.capture("/auth")
if res.status == ngx.HTTP_OK then
return
end
if res.status == ngx.HTTP_FORBIDDEN then
ngx.exit(res.status)
end
ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
';

# proxy_pass/fastcgi_pass/postgres_pass/...
}

which is approximately equivalent to

location / {
auth_request /auth;

# proxy_pass/fastcgi_pass/postgres_pass/...
}

except that "auth_request" runs at the beginning of the nginx access phase while "access_by_lua" runs at the end of the access phase.

ngx_lua is an nginx C module that embeds the Lua or LuaJIT interpreter into the nginx core and you can find the latest source code and the complete documentation here

https://github.com/chaoslawful/lua-nginx-module

Enjoy!

ngx_lua v0.1.0: scripting nginx with Lua

2011-01-13T19:19:00.001+08:00

I'm happy to announce the first public release of the ngx_lua module, v0.1.0:

https://github.com/chaoslawful/lua-nginx-module/downloads

ngx_lua This module embeds the Lua/LuaJIT interpreter into the nginx
core and integrates the powerful Lua threads (aka Lua coroutines) into
the nginx event model by means of nginx subrequests.

Unlike Apache's mod_lua and Lighttpd's mod_magnet, Lua code written
atop this module can be 100% non-blocking on network traffic as long
as you use the ngx.location.capture interface to let the nginx core do
all your requests to mysql, postgresql, memcached, redis, upstream
http web services, and etc etc etc (see ngx_drizzle, ngx_postgres,
ngx_memc, ngx_redis2, and ngx_proxy modules for details).

You can find the latest source code as well as the latest documentation here:

http://github.com/chaoslawful/lua-nginx-module

We've already run ngx_lua as well as dozens of other nginx C modules
in Taobao.com's web applications for months and this module is
considered production ready.

With ngx_lua, it's now possible to construct true C10K-capable web
applications because everything can be trivially made non-blocking,
not only the network traffic between the clients, but also the traffic
to the backends like RDBMS and memcached clusters. And we've been
working hard to make it a much better substitution for traditional web
development solutions like PHP, not only in terms of performance, but
also ease of use.

Enjoy!

Video for my ngx_openresty talk on ECUG 2010

2010-11-06T18:01:00.001+08:00

I gave a (Chinese) talk regarding ngx_openresty (aka nginx.conf scripting) at ECUG 2010 in the last month. Here's the video for the talk:

http://v.ku6.com/show/D00rqtnRwKzJdIsB.html

The slides can be viewed in a web browser from here:

http://agentzh.org/misc/slides/nginx-state-of-the-art/

Please use the arrow keys on your keyboard to switch pages in the slides.

Have fun!

ngx_memc v0.11 released: small bug fixes and more documentation

2010-10-28T12:14:00.001+08:00

I've just released the v0.11 version of our ngx_memc module:

http://github.com/agentzh/memc-nginx-module/tarball/v0.11

This version applies the patch from iframist, fixing the zero size buf alert in error.log when $memc_value is set to empty ("").

The ngx_memc module extends the standard memcached module to support almost the whole memcached ascii protocol. And it can be used with either the standard memcached server or other backends supporting the memcached wire protocol like TokyoTyrant.

Maxim Dounin's excellent ngx_http_upstream_keepalive module can also be used with this module to provide a powerful connection pool for memcached.

See ngx_memc's wiki page for more details:

http://wiki.nginx.org/HttpMemcModule

I've also updated the wiki page for this release and documented various directives like memc_connect_timeout, memc_read_timeout, memc_send_timeout, and memc_buffer_size. (These directives are inherited directly from the standard memcached module.)

Enjoy!

淘宝（北京）量子团队的实习机会

2010-09-26T15:10:00.001+08:00

我们是阿里巴巴淘宝网（北京）数据平台部门的量子团队（ http://lz.taobao.com ），我们正在寻找有想法有激情的同学到我们团队来实习。

我们目前的几个实习岗位近期的主要工作包括

基于 hive/hadoop 的数据仓库系统的自动化测试的方法研究和实施（Perl 5, PostgreSQL/mysql, Hive/Java）

在数据产品的在线应用（Online Application）中进行 hive 计算任务的提交和异步执行的管理

基于 nginx 和 ngx_openresty 的数据 web service 平台的完善（C, Lua, Erlang, 以及我们自主设计的 LZSQL 语言）

感兴趣的同学可以在我的个人博客上找到更多的细节：

http://agentzh.spaces.live.com

我们对学校、学历和专业都没有要求，因为我们相信真正有潜质的人并不必出自名牌大学，也不必是科班出身。我们也不会要求有很深很广的知识和技术背景，毕竟我们这里更多的是使用或者自己创造全新的知识和技术。我们看重的主要是

有很强的求知欲和好奇心，对新事物和新想法有浓厚的兴趣，并愿意为之付出很大的努力，经常自己搞点有趣的事情玩
同时有坚持力，愿意沉下心系统地学习一些学科、理论和技术细节，有耐心，能耐得住寂寞
拥有较好英语功底，可以无障碍地大量阅读英文文档和其他资料
能有较长较为稳定的实习期（6 个月以上），因为许多重要的工作需要足够长的时间才能达到足够的深度和广度

下面是我们觉得胜任此工作所必需具备的技术上的基本技能

经常在 Linux 或者其他 *NIX 环境中工作，熟悉 shell 和命令行
经常使用像 vim 和 emacs 这样的文本编辑器
有比较好的 ANSI C 编程的基础
熟悉至少一门动态的脚本语言，比如 Perl 或者 Python
曾经使用过某一种版本控制系统，例如 git 或者 subversion
熟悉至少一种 make 性质的项目构造和管理工具，例如 GNU make
熟悉 SQL 语言和某一种 RDBMS，例如 mysql 或者 PostgreSQL

我们能提供和一直倡导的主要是

比较宽松的工作氛围和学习氛围

比较细致的技术指导和支持

激进但不失必要谨慎的新技术的调研与生产应用

通过自己设计语言和工具，变无聊的体力活为有趣的创造性的事情

这个实习岗位工作会像正式员工的工作一样有趣，一样富有挑战

我们的办公地点在：北京市朝阳区东三环北街 38 号院 1 号楼泰康金融大厦 25 层。

如果你对我们这个实习岗位感兴趣，请将你的简历通过电子信箱发送到我的信箱： chunlai at taobao dot com，并确保邮件标题中出现“应聘”这两个字，便我能及时看到 :)

LZSQL compiler hacking and future VoltDB applications

2010-09-15T11:21:00.010+08:00

We've been working on a compiler for our own language named "lzsql". It's for our web service platform that drives our data product lz.taobao.com. Our "lzsql" compiler can now emit lua code that has passed lots of real world tests.

We can now decide whether to run a sql query at a remote mysql node or at the nginx core, all in the lzsql language.

For "local sql queries", we've implemented a full-fledged sql engine in pure lua. It's damn fast, especially using LuaJIT. 6k q/s for a single nginx worker process is not uncommon in our benchmark.

And we've introduced a type system in our language such that it can handle sql quoting rules automatically. The typechecker can ensure that a lzsql variable with a specific type is used correctly in the context of the sql query. The sql language is part of the language anyway. Therefore, sql injection cannot happen.

We mostly use the "local sql engine" for merging data from completely different data sources, like those from both mysql and a non-relational data source. We do have some non-relational data sources like our real time stats services and other Java-powered web services from other departments of Taobao.com.

Here's a small example:

    text $pattern;
    location $mysql_node;

    @a := select count(id) as count
          from cats
          where name contains $pattern
                group by park
          at $mysql_node;

    @b := select count(id) as count
          from other_service.some_api($pattern)
          group by park;

    return (@a union all @b);

In this sample, "other_service.some_api" is a non-blocking call to some remote non-relational data source. And the first SQL query is run on a remote mysql node specified by the variable $mysql_node while the last two are both running directly in the nginx core by our sql engine written in Lua.

The .lzsql source file is compiled down to (very compact) Lua code before deploying to our production servers. Because it is a true compiler, we use Perl 5, one of the not-so-fast scripting languages, to implement the whole compiler (approximately 3k lines of hand-written code). Perl modules like Moose and Parse::Descent has made the compiler construction process quite enjoyable :)

In the future, the lzsql compiler is also expected to optimize the sql queries automatically for specific remote sql engine, like mysql's.

The lzsql compiler will be eventually be released under an opensource license with the name "RestyScript" when we decouple those our specific business logic from the compiler. For now, we hardcode some business logic into the compiler for the sake of convenience. We're going to move them into compiler or language plugins and make the lzsql toolchain more general.

My intern students become very productive when they start using the lzsql language ;) The old system they're replacing is written in tons of ugly php, oh well ;) we've cut 90% of the codebase size and also got 20 ~ 30 times faster :D

We're also puting our heads around VoltDB, a really nice memory database. And we're also looking forward to rewriting our "real time stats services" mentioned above using VoltDB and Erlang or Lua or etc. An nginx upstream module for the VoltDB binary protocol is also on chaoslawful's and my TODO list.

The only sad part regarding VoltDB is that it's written in Java, but it's not a very big issue for us. It has some ugly limitations regarding its sql and interfaces, but we can work around those details on the level of our lzsql language and just use it (combined with java) as the runtime.

It's already starting to become more and more interesting :)

Stay tuned!

ngx_lua module updated

2010-09-05T18:42:00.001+08:00

chaoslawful++ has just added several new features to our ngx_lua module. Checkout his blog article for details:

http://chaoslawful.javaeye.com/blog/755013

ngx_lua embeds the power of the Lua language into the nginx core and can be used to construct high performance web services and web applications:

http://github.com/chaoslawful/lua-nginx-module

Happy lua hacking! :D

ngx_set_misc v0.14: extending ngx_rewrite's "set" directive

2010-09-02T16:15:00.001+08:00

I'm happy to announce the first public release of our Taobao.com ngx_set_misc module, v0.14.

ngx_set_misc is an nginx module that extends the standard ngx_rewrite module's "set" directive to support various advanced functionalities like MD5, SHA1, json/mysql/postgresql string literal quoting, URI escaping/unescaping, default variable value assignment, upstream hashing based on a custom key, base32 encoding/decoding, and more :)

Please see the project homepage for more details:

    http://github.com/agentzh/set-misc-nginx-module

And the release tarball can be downloaded from

    http://github.com/agentzh/set-misc-nginx-module/tarball/v0.14

Various (funny) use cases can be found in my "nginx.conf scripting" talk's slides:

    http://agentzh.org/misc/slides/nginx-conf-scripting/ (use the arrow keys on your keyboard to switch pages)

I must thank my colleagues shrimp and calio for their work on polishing this module in the last few months.

This module won't be possible if Marcus Clyne does not publish his crazy Nginx Development Kit (NDK) project:

    http://github.com/simpl-it/ngx_devel_kit

And it's a prerequisite for this module :)

I know that this module has a really terrible name, but it's been there for months already :P

We've been using it extensively in our products of Taobao.com. And Qunar.com is also using it heavily in their production environment.

Enjoy!

ngx_drizzle v0.0.12: better timeout control

2010-09-02T15:51:00.001+08:00

Hi, folks!

I'm delighted to announce the v0.0.12 release of ngx_drizzle, a non-blocking upstream module that helps nginx talk directly to mysql, drizzle, and sqlite3 servers (and with an optional connection pool). The project source repository and the homepage is on GitHub:

http://github.com/chaoslawful/drizzle-nginx-module

and the release tarball can be downloaded from

http://github.com/chaoslawful/drizzle-nginx-module/tarball/v0.0.12

This release has the highlight of several new config directives that control the various timeout settings used by ngx_drizzle:

drizzle_connect_timeout <time>
drizzle_send_query_timeout <time>
drizzle_recv_cols_timeout <time>
drizzle_recv_rows_timeout <time>

The default timeout values for them are all "60 s", i.e., 60 seconds, which may be too long for many real world applications.

Thanks my colleague shrimp++ for testing them and fixing bugs in the original (undocumented) implementation :)

Also thanks Piotr Sikora for his tireless improving the test suite to fit our grand test build farm and his fixes for the recent nginx 0.8.47+ releases.

As always, special thanks go to my colleague and closest friend, chaoslawful++, for creating such an excellent module in the first place ;)

FWIW, we're already using this module (combined with ngx_rds_json as well as many other modules) in production, in particular, the Taobao.com LiangZi Shop Stats website:

http://lz.taobao.com/

Have fun!
-agentzh

cheater: yet another rule-driven tool to generate random databases

2010-08-19T11:24:00.001+08:00

cheater 是我前一阵子开发的规则驱动的随机数据库生成器，是量子开发、测试工具链的组成部分。

cheater 已在量子店铺统计的前端开发中得到了广泛的使用，为前端开发提供大量的指定模式的伪造数据，从而大大减轻了对后端和真实数据的依赖，同时也可以得到比较理想的数据覆盖率。

cheater 工具的源码仓库位于下面这个位置：

http://github.com/agentzh/cheater/

相比我们 QA 部门的 xdata，Ruby 世界的 faker 和 Perl 世界的 Data::Faker 等同类型的工具，cheater 具有以下优点：

能自动处理表间的关联和外键约束，因此是真正的“数据库实例生成器”
定义了一种类 SQL 的小语言来表达期望生成的数据模型
支持强大的 {a, b, c} 离散集合，数值/时间/日期区间记法 a..b，Perl 正则表达式模式 /regex/，常量值 'string', 1.32 等方式来表达数据字段的值域
能直接生成 JSON 或者 SQL insert 语句，便于导入到 mysql/Pg 等数据库或者 hive 等其他系统

下面是一个很简单的示例，演示其基本用法。

我们首先在一个工作目录（比如 ~/work/ 下）新建一个 .cht 文件，用来描述我们想生成的数据模型。假设我们有一个 company.cht 文件：

    # Empolyee table
    table employees (
         id serial;
        name text /[A-Z]a-z{2,5} [A-Z]a-z{2,7}/ not null unique;
         age integer 18..60 not null;
        tel text /1[35]8\d{8}/;
         birthday date;
        height real 1.50 .. 1.90 not null;
         grades text {'A','B','C','D','E'} not null;
        department references departments.id;
     )

    # Department table
     table departments (
        id serial;
         name text /\w{2,10}/ not null;
    )
 
    10 employees;
    2 departments;

这里我们使用的是 cheater 自己的小语言，其含义几乎是一眼就明了的，特别地，最后两行是说随机生成符合规则的 10 条 employees 表的记录和 2 条 departments 表的记录。

然后我们使用 cht-compile 命令来编译我们的 company.cht 生成随机的数据库实例：

    $ cht-compile company.cht
    Wrote ./data/departments.rows.json
     Wrote ./data/employees.rows.json

我们看到它分别为 departments 和 employees 这两张表生成了两个 JSON 格式的数据文件。其中的 data/employees.rows.json 文件在我机器上的此次运行是这个样子的：

    $ cat data/employees.rows.json
    [["id","name","age","tel","birthday","height","grades","department"],
     ["7606","Kxhwcn Cflub",54,"15872171866","2011-04-01","1.67276","D","408862"],
     ["63649","Whf Iajgw",55,"13850771916",null,"1.65297","E","844615"],
     ["348161","Nnwe Obfkln",27,"15801601215","2011-03-06","1.69275","D","408862"],
     ["353404","Shgpak Xvqxw",28,"15816453097",null,"1.67796","A","408862"],
     ["445500","Bdt Mhepht",47,"13855517847",null,"1.89943","C","844615"],
     ["513515","Ipsa Mcbtk",25,"13874017694","2011-01-06","1.79534","A","844615"],
     ["658009","Lboe Etqo",27,null,"2011-04-14","1.85162","E","408862"],
     ["716899","Gey Elacflr",18,"15804516095","2011-02-27","1.75681","A","844615"],
     ["945911","Hsuz Qcmky",39,"13862516775","2011-05-31","1.75947","B","408862"],
     ["960643","Qbmbe Ijnbqsb",24,"15872418765","2011-04-11","1.78864","B","844615"]]

最后，为导入到关系数据库，我们可以使用 cht-rows2sql 命令将得到的 .json 数据文件转换为 .sql 文件：

    $ cht-rows2sql data/*.rows.json
    Wrote ./sql/departments.rows.sql
     Wrote ./sql/employees.rows.sql

其中的 sql/departments.rows.sql 在我这里是这样的：

    $ cat sql/departments.rows.sql
    insert into departments (id,name) values
     (408862,'dJRq7LCXL'),
    (844615,'G_m9Nkh3q');

这样我们就可以直接往 mysql 这样的数据库里导入数据了：

    $ mysql -u monty test -p < sql/departments.rows.sql

目前 cheater 仍处于比较活跃的开发阶段，缺乏比较完整的文档。最完整的文档是它的自动化测试集：

http://github.com/agentzh/cheater/tree/master/t/

点开其中的 .t 文件，便可以看到一个个的声明性的测试用例

如果您在使用过程中发现任何 bug 或者有任何功能提议，请在 GitHub 上创建相应的 ticket:

http://github.com/agentzh/cheater/issues

Enjoy!

A patch for libdrizzle to fix issues on Mac OS X

2010-08-10T11:30:00.001+08:00

libdrizzle is an excellent piece of software but we've noticed that it does not compile on Mac OS X due to its use of the new bool type in C:

/usr/local/include/libdrizzle/result.h:69: error: syntax error before 'drizzle_result_eof'

The following small patch for libdrizzle 0.8 fixes this (as well as another bug regarding streaming parsing on the TCP protocol level, as reported by my colleague chaoslawful++ months ago):

http://agentzh.org/misc/nginx/libdrizzle-0.8-parsebug_and_mac_fixes.patch

I'm looking forward to the next libdrizzle release or I'll have to keep my patch for my ngx_drizzle module's users ;)

ngx_chunkin v0.20: fixed some memory bugs and added support for chunked PUT

2010-08-03T11:24:00.001+08:00

I've just pushed a new release (v0.20) for my ngx_chunkin module:

http://github.com/agentzh/chunkin-nginx-module/tarball/v0.20

Here's the changes included in this version:

fixed a bug that may read incomplete chunked body. thanks Gong Kaihui (龚开晖).
fixed various memory issues in the implementation which may cause nginx processes to crash.
added support for chunked POST requests.
now we always require "error_page 411 @resume" and no default (buggy) magic any more. thanks Gong Kaihui (龚开晖).

This module adds HTTP 1.1 chunked input support for Nginx without the need of patching the Nginx core.

Behind the scene, it registers an access-phase handler that will eagerly read and decode incoming request bodies when a "Transfer-Encoding: chunked" header triggers a 411 error page in Nginx. For requests that are not in the chunked transfer encoding, this module is a "no-op".

To enable the magic, just turn on the chunkin config option and define a custom 411 error_page using chunkin_resume, like this:

     server {
     chunkin on;

     error_page 411 = @my_411_error;
     location @my_411_error {
     chunkin_resume;
     }

     ...
     }

See its wiki page for the full documentation:

http://wiki.nginx.org/NginxHttpChunkinModule

Note that nginx 0.8.41 and 0.7.67 are confirmed to work with this version. Nginx releases newer than 0.8.41 will not work due to this
issue:

http://forum.nginx.org/read.php?29,103078

There's no reply from the nginx author Igor Sysoev yet :(

Enjoy!

Update: I've just kicked ngx_chunkin v0.21 out of the door, which applies a patch from Gong Kaihui that fixed a small bug:

   http://wiki.nginx.org/NginxHttpChunkinModule#v0.21

http://github.com/agentzh/chunkin-nginx-module/tarball/v0.21

ngx_lua now has (basic) subrequest support

2010-06-24T12:52:00.001+08:00

Last night's ngx_lua hackathon has been proven extremely fruitful. chaoslawful and I didn't stop coding until midnight, and successfully finished the first draft of the most tricky bit in ngx_lua, that is, transparent non-blocking IO interface (or nginx subrequest interface) on the Lua land.

The following test case is now passing:

location /other {
echo "hello, world";
}

# transparent non-blocking I/O in Lua
location /lua {
content_by_lua '
local res = ngx.location.capture("/other")
if res.status == 200 then
ngx.echo(res.body)
end';
}

And on the client side:

$ curl 'http://localhost/lua'
hello, world

In the /other location, we can actually have drizzle_pass, postgres_pass, memcached_pass, proxy_pass, or any other content handler configuration.

Here's a more amusing example to do "recursive subrequest":

location /recur {
       content_by_lua '
           local num = tonumber(ngx.var.arg_num) or 0
           ngx.echo("num is: ", num, "\\n")

           if num > 0 then
               res = ngx.location.capture("/recur?num=" .. tostring(num - 1))
               ngx.echo("status=", res.status, " ")
               ngx.echo("body=", res.body)
           else
               ngx.echo("end\\n")
           end
           ';
   }

Here's the output on the client side:

    $ curl 'http://localhost/recur?num=3'
    num is: 3
    status=200 body=num is: 2
    status=200 body=num is: 1
    status=200 body=num is: 0
    end

So...time to replace our PHP code in the business with nginx.conf + Lua scripts!

We'd make the first public release of ngx_lua when its implementation and API become solid enough ;)

ngx_xss v0.02: fixed a bug that prevents responses from being gzipped

2010-06-22T17:49:00.001+08:00

I'm glad to announce the v0.02 release of the ngx_xss module:

http://github.com/agentzh/xss-nginx-module/tarball/v0.02

This module provides native cross-site scripting (XSS) support in nginx, and cross-site GET via JSONP in particular. Please visit the project homepage for more details:

http://github.com/agentzh/xss-nginx-module

This release fixes a nasty bug in Content-Type header handling. The previous version does not clear r->headers_out.content_type_lowcase which sadly prevents responses from being compressed by the ngx_http_gzip_filter_module if configured.

Thanks my teammate kindy++ for catching it in our production environment :P

ngx_drizzle v0.0.11 and ngx_rds_json v0.09: significant performance boost

2010-06-21T17:48:00.001+08:00

I'm happy to announce that ngx_drizzle v0.0.11 and ngx_rds_json v0.09 are finally out:

http://github.com/chaoslawful/drizzle-nginx-module/tarball/v0.0.11

http://github.com/agentzh/rds-json-nginx-module/tarball/v0.09

ngx_drizzle is an upstream module that talks to mysql, drizzle, and sqlite3 by libdrizzle, and generates output in a binary format known as RDS (Resty DBD Stream), just like Piotr Sikora's ngx_postgres module.

ngx_rds_json is an output filter module that converts the RDS outputs of ngx_drizzle and ngx_postgres, to plain JSON text.

The highlight of these releases are significant performance boost due to extensive refactoring and optimizations in these two modules recently.

One can observe hundreds of times of improvement for big responses above 300KB for both ngx_drizzle + ngx_rds_json and ngx_postgres + ngx_rds_json combination.

We even observed that ngx_drizzle + ngx_rds_json achieved 128MB/sec (Yes, 1024Mbit/sec!) for a 380KB data-set query while connecting to a simple mysql server, about 3 ~ 4 times as fast as php + libmysql in an identical setting.

Technically, we (partially) adapted the "fixed-size bufs" model used in ngx_http_gzip_filter_module (and elsewhere in the nginx core) in both of our modules, effectively eliminating lots of unnecessary packet splitting and buffer allocations, which contributes most of the performance boost. And we'd like ngx_postgres to apply this technique in the near future too (for a typical 470 KB data-set query, ngx_drizzle + ngx_rds_json is now more than 50% faster than ngx_postgres + ngx_rds_json).

We also introduce the drizzle_buffer_size and rds_json_buffer_size directives to allow the user adjust the size of each buf that is used in the output emitter:

drizzle_buffer_size 4k;
rds_json_buffer_size 4k;

The default setting is the page-size, usually 4k ~ 8k. The bigger the buffer size, the less streamming the output will be.

At last but not least, this release of ngx_drizzle also includes various new features ported from ngx_postgres, like the "method-specific queries" support in the drizzle_query directive.

Here's an example that implements a full RESTful interface to a mysql backend using a single nginx location, and no "if hacks":

location ~ '^/cat/(\d+)' {
set $id $1;
set_form_input $name;
set_quote_sql_str $quoted_name $name;

drizzle_query HEAD GET "select * from cats where id=$id";
drizzle_query DELETE "delete from cats where id=$id";
drizzle_query POST "insert into cats (name) values($quoted_name)";

drizzle_pass my_mysql_backend;
}

There's also an example for ngx_postgres in my slides:

http://agentzh.org/misc/slides/recent-dev-nginx-conf/#18

Have fun!

Recent developments in nginx.conf scripting

2010-06-21T11:33:00.000+08:00

Last Saturday I gave a talk in Beijing OpenParty's monthly meetup regarding our recent developments in nginx.conf scripting with the highlight of new modules like ngx_postgres, ngx_form_input, ngx_srcache, ngx_encrypted_session, and ngx_lua.

Here's the slides that I used in this event:

http://agentzh.org/misc/slides/recent-dev-nginx-conf/

Please use the arrow keys or page-up/page-down keys on your keyboard to navigate through them.

Special thanks go to Piotr Sikora for his preview of these slides before the talk and lots of his good suggestions.

Feedback and comments are very welcome as usual :)

Enjoy!

ngx_openresty 端午节大优化

2010-06-18T15:41:00.001+08:00

端午节前后我对 ngx_drizzle 和 ngx_rds_json 进行了深入重构和优化，现在对于几百 KB 的大结果查询，比一周前提升了几百倍的性能。现在对于一个典型的 380 KB 的大结果集的 mysql 查询，10 并发，单机可达 430+ q/s，128 MB 的传输速率，20 ms 平均响应时间 [1]。

nginx 这一侧没有使用任何针对结果集的缓存，mysql 那一侧倒是有可能用到了其自己的内存缓存，呵呵。现在我们的性能终于是 php + libmysql 的 4 倍左右了。

主要的改动就是重新设计了缓冲区的管理模型，采用了类似 ngx_http_gzip_filter_module 的定长 buf 链和回收机制，后面有机会再详细介绍一下，呵呵。

值得一提的是，编译 nginx 时 gcc -O1 会比 -O0 提升 50%；-O2 比 -O1 只提升几个百分点的样子。

ngx_rds_json 的优化也让 ngx_postgres 和它一起工作时，性能提高了一二百倍 :)

另外，git HEAD 中还分别给这两个模块新增了一条配置指令，一是 drizzle_buffer_size，一是 rds_json_buffer_size. 二者的默认值都是 page size, 即一般是 4k 或者 8k, 对于大结果集，适当调大有助于进一步提升性能，但 buffer size 设得越大，流输出的特性就越少。

明天 OpenParty 的 talk 完了之后，我再发布这两个模块的新版本 :)

Stay tuned~

[1] 测试机器的配置是 4 核的 Intel(R) Xeon(R) CPU 5130 @ 2.00GHz, 4 GB RAM.

ngx_headers_more v0.10: ability to remove a header completely

2010-06-06T20:46:00.001+08:00

I'm happy to announce that the v0.10 release of ngx_headers_more has just landed:

http://github.com/agentzh/headers-more-nginx-module/tarball/v0.10

This module allows you to add, set, or clear any output or input header that you specify. Please see the full documentation for more details.

From this release, the more_clear_headers and more_clear_input_headers directives can remove a specified header completely.

In earlier releases, clearing out a header usually just clears the header's value, not its key. For instance,

more_clear_headers 'Server';

yields

Server:

in the response headers. Now, it'll be erased completely.

Non-standard headers like X-Foo can be totally erased as well.

The same change applies to more_clear_input_headers, where we can erase an input header in the request object before forwarding it to a content handler like proxy_pass or fastcgi_pass.

You're very welcome to send bug report (if any!) or wishlist to us at GitHub Issues.

Enjoy!

ngx_echo v0.32: various memory issue fixes inspired by valgrind

2010-06-04T12:08:00.001+08:00

I'm happy to announce the v0.32 release of the ngx_echo module:

http://github.com/agentzh/echo-nginx-module/tarball/v0.32

This module wraps lots of Nginx internal APIs for streaming input and output, parallel/sequential subrequests, timers and sleeping, as well as various meta data accessing. Basically it provides various utilities that help testing and debugging of other modules by trivially emulating different kinds of faked subrequest locations.

This release fixes several memory issues reported by valgrind's memcheck tool and below is the complete change log for this version:

http://wiki.nginx.org/NginxHttpEchoModule#v0.32

We've just integrated valgrind support into our test scaffold Test::Nginx which is used by all of our nginx module projects. Now running a particular module's test suite with valgrind's memcheck is as simple as

cd test
TEST_NGINX_USE_VALGRIND=1 prove -r t

This facility also helps spotting quite a few memory-related issues in several other modules developed by ourselves :)

Enjoy!

The upcoming nginx.conf scripting talk at OpenParty

2010-06-03T11:00:00.001+08:00

I'm going to talk about nginx.conf scripting in this month's OpenParty meetup:

http://app.beijing-open-party.org/topic/10

You're very welcome to attend my talk:

http://app.beijing-open-party.org/

See you there ;)

ngx_headers_more v0.09: wildcard support in more_clear_headers

2010-06-02T20:09:00.000+08:00

I'm delighted to announce the v0.09 release of the ngx_headers_more module:

http://github.com/agentzh/headers-more-nginx-module/tarball/v0.09

This module allows you to add, set, or clear any output or input headers that you specify. This is an enhanced version of the standard headers module because it provides more utilities like resetting or clearing "builtin headers" like Content-Type, Content-Length, and Server.

This release features the wildcard (*) support in the more_clear_headers directive. For example, the following directive effectively clears any output headers starting by X-Hidden-:

more_clear_headers 'X-Hidden-*';

Thanks our new contributor Bernd Dorn for implementing this :)

See the complete change log for this version if you're interested:

http://wiki.nginx.org/NginxHttpHeadersMoreModule#v0.09

Also, in the previous version, i.e., v0.08, Bernd Dorn also introduced the -r option to the more_set_input_headers directive to replace the value of the input header specified if and only if that header actually exists.

You can find the complete documentation of this module on the nginx wiki site:

http://wiki.nginx.org/NginxHttpHeadersMoreModule

Enjoy!

Optimizing ngx_openresty for big responses

2010-06-01T14:04:00.001+08:00

(Piotr Sikora blames me for posting my previous article in Chinese. Hence the following English transcript.)

I've been spotting bottlenecks in ngx_drizzle and ngx_rds_json for responses with big data sets. Now it's finally 1 times faster than php + libmysql for huge mysql responses of 380+ KB. The former can reach 250+ q/s using ab -c50 while the latter merely gets 120+ q/s.

Earlier last week, for requests of such big responses, ngx_drizzle + ngx_rds_json merely got 12 ~ 14 q/s, just 1/10 of the that of php + libmysql, which drove me crazy.

This serious performance issue mainly comes from the "strict context-free streamming output model" introduced into ngx_drizzle and ngx_rds_json around the beginning of this year, due to my laziness. This has been proven to be a really bad idea because it will split the data buffer into many many little pieces and even worse, it sets buf->flush to true which effectively enables auto-flushing. Sigh. For that specific 380 KB result set query mentioned above, it will result in 10000+ times of writev syscalls and slowness compared to even php :)

I've already turned off buf->flush in ngx_drizzle v0.0.11rc1 and ngx_rds_json v0.08.

calio++ will refactor the bufs and chains model used by these two modules this month, so as to achieve the best performance here.

For little data sets (less than 500 bytes), ngx_drizzle's qps is always one or two magnitude higher than php + libmysql, without doubt. Thousands of q/s is not uncommon at all.

A good news is that, qunar.com is already using ngx_postgres + ngx_rds_json to access PostgreSQL in production for several online services. And according to their report, they can achieve 7k ~ 8k q/s per machine without caching the result sets. Their nginx worker processes have run flawlessly for 12 days and never crash.

We are also preparing to push ngx_drizzle online for our own data product.

Stay tuned~~

ngx_openresty 优化进行时

2010-06-01T11:28:00.001+08:00

这几日我一直在优化 ngx_drizzle 和 ngx_rds_json. ngx_drizzle 对于 380 KB 这样的大结果集终于比 php + libmysql 快一倍以上了，前者是 250+ q/s 单机，后者是 120+ q/s.

在上周的时候，这种大结果集的单次查询，ngx_drizzle + ngx_rds_json 只有 php + libmysql 的 1/10 的 q/s 性能，只有 12 ~ 14 q/s，当时我都快疯了，哈哈！

小结果集 ngx_drizzle 比 php + libmysql 高一个数量级以上没什么悬念，呵呵。前者都是千级的 qps.

原先对大结果集查询有性能瓶颈的主要原因是我年初做 ngx_drizzle 和 ngx_rds_json 的时候为偷懒，实现了一种上下文无关的严格流式输出模型，导致数据切片过细，并且置了 buf->flush，导至 writev 系统调用次数过多（在上面那个 380 k 输出的测试用例中，最初有 10000+ 次调用）。确实，下游输出的 tcp packet 的大小应该和上游 mysql 的 tcp packet 的这个"上下文"有关 :) 我已经让 calio 同学在下个月重构 ngx_drizzle 和 ngx_rds_json 的输出组件的内存管理模型。

一个好消息是，qunar.com 已经在生产上使用 ngx_postgres + ngx_rds_json 来访问他们的 PostgreSQL 数据库后端，据他们的报告，单机性能可达 7k ~ 8k q/s，并且已经在线上稳定运行 12 天了，nginx worker 进程从未掉过一个，呵呵，也没有观察到内存泄漏 :D

我们自己的数据产品也在紧锣密鼓地准备把 ngx_drizzle 推上线 ;)

ngx_echo v0.31: more sequential subrequest fixes inspired by ngx_srcache

2010-05-26T10:33:00.001+08:00

I'm glad to release ngx_echo v0.31 which include various fixes on sequential subrequests inspired by the ngx_srcache development:

http://github.com/agentzh/echo-nginx-module/tarball/v0.31

Here's the (boring) technical change log:

the echo wev handler should not proceed if it is still waiting for some sequential subrequest or has just processed one to avoid bouncing issues.
fixed a segfault for echo_exec for 0.7.x: we should check r->done before proceeding.
no longer explicitly set r->write_event_handler to ngx_http_request_empty_handler because it's totally wrong for the state machine.
fixed the sequential subrequest model bugs: we should ensure the pr->write_event_handler gets called immediately after the post_subrequest callback when the subrequest finalizes.

BTW, ngx_srcache is near the corner of its first public release:

http://github.com/agentzh/srcache-nginx-module

calio++ is currently working on the last missing bit, i.e., response headers caching :)

ngx_rds_json v0.07: fixed boolean values for PosgreSQL

2010-05-26T10:08:00.000+08:00

I'm pleased to announce that ngx_rds_json v0.07 has been released:

http://github.com/agentzh/rds-json-nginx-module/tarball/v0.07

ngx_rds_json is an output filter module that can format the RDS (Resty DBD Stream) outputs of the ngx_drizzle and ngx_postgres modules to JSON text.

This release includes the patch from Tom Tuling, that fixes the boolean values in the result sets emitted by PostgreSQL, i.e., values like "f" and "t". Thanks Tom Tuling :)

FWIW, my friend xunxin++ has already put ngx_postgres + ngx_rds_json into production :)

A Simple ngx_lua Example for the Future

2010-05-18T10:51:00.001+08:00

One of my nginx module users is asking for a "get_once" command support for the ngx_memc module. But I'd rather do such specific things on the ngx_lua level. Here's a quick example for this task:

location /get_once {
content_by_lua "
-- read the memcached key from the nginx variable $uri
key = ngx.var.uri;

-- do memcached get
res = ngx.location.capture('/memc?cmd=get&key=' .. key);

-- forward the memcached get response to the downstream
ngx.res.status = res.status;
ngx.res.out(res.body);

if res.status ~= 404 then
-- delete the key and discard the response
ngx.location.capture('/memc?cmd=delete&key=' .. key);
end
";
}

location /memc {
internal;
set $memc_key $arg_key;
set $memc_cmd $arg_cmd;
memc_pass 127.0.0.1:11211;
}

Furthermore, we can use the content_by_lua_file directive if we'd put the lua code into a separate .lua file ;)

It's worth mentioning that the ngx.location.capture method runs in completely non-blocking fashion. Thanks to Lua's coroutine support :D

And yes! At the moment, ngx_lua does not have content_by_lua and content_by_lua_file yet. But we're already working on them and they'll come soon.

Because Lua can run pretty fast especially when JIT is enabled (even compared to plain C), I think we do not have much performance penalty here :)

Do you like it?

ngx_echo v0.29: major core refactoring and more robust sequential subrequests

2010-05-05T17:25:00.001+08:00

I'm happy to announce the v0.29 release of the ngx_echo module:

http://github.com/agentzh/echo-nginx-module/tarball/v0.29

As mentioned in some other threads on the nginx mailing list, I've completed the big refactoring of the ngx_echo module's core in this version to reflect my latest understanding (hopefully being quite
complete and correct already) of the nginx internals.

Now the implementation of echo_subrequest, echo_location, echo_sleep, and echo_read_request_body directives has been massively rewritten. I'm trying to set up a design pattern for nginx content handlers that require to do various kinds of non-blocking I/O during its lifetime (similar to upstream modules but for different tasks like subrequests and others).

For sequential subrequests issued by the echo module's content handler, we now use a totally different approach.

Instead of issuing subrequests directly in our post_subrequest callback (as fed into the ngx_http_subrequest call), we now postpone firing the subrequests in a custom write event handler which will be called automatically once the current subrequest in question gets completely finalized (that post_subrequest callback is called by ngx_http_finalize_request of the subrequest in question).

This works because the parent request will always be waken up once its subrequest completes. (This is done by ngx_http_finalize_request via posting the parent request into the "posted requests" queue and the posted requests will be always called at the end of the top-level read/write event handler, i.e., ngx_http_request_handler).

This solves a lot of issues like mangled r->main->count and the following alert message when ginx is configured with the --with-debug option, for instance,

2010/05/05 16:46:14 [alert] 23853#0: *1 http finalize non-active request: "/main?" ...

We'll apply this technique to the upcoming ngx_lua and ngx_srcache modules soon :)

P.S. See the ngx_echo module's wiki page for more information:
http://wiki.nginx.org/NginxHttpEchoModule

ngx_echo v0.28: support for the -n and -- options

2010-05-03T17:49:00.001+08:00

I'm happy to announce the v0.28 release of our ngx_echo module:

http://github.com/agentzh/echo-nginx-module/tarball/v0.28

ngx_echo is a powerful nginx module that wraps lots of Nginx internal APIs for streaming input and output, parallel/sequential subrequests, timers and sleeping, as well as various meta data accessing. Basically it provides various utilities that help testing and debugging of other modules by trivially emulating different kinds of faked subrequest locations.

The highlight of this version is the introduction of new options "-n" and "--" for the "echo", "echo_before_body", and "echo_after_body" directives.

Now we can suppress the trailing newline character in the "echo" output by using the -n option, as in

location /echo {
echo -n "hello, ";
echo "world";
}

Accessing /echo gives

$ curl 'http://localhost/echo'
hello, world

Leading -n in variable values won't take effect and will be emitted literally, as in

location /echo {
set $opt -n;
echo $opt "hello, ";
echo "world";
}

This gives the following output

$ curl 'http://localhost/echo'
-n hello, world

One can also output leading -n literals and other options using the special -- option like this

location /echo {
echo -- -n is an option;
}

which yields

$ curl 'http://localhost/echo'
-n is an option

Please see the documentation for more details:

http://wiki.nginx.org/NginxHttpEchoModule#echo

I must thank Dylan Stamat and kindy for prompting me to add the -n option. They have even provided patches on their own but their patches required more treatment before merging into the mainstream :)

In the next release, i.e., v0.29, I'll refactor the implementation of the echo_location, echo_subrequest, and echo_sleep as promised on the nginx mailing list.

Enjoy!

Looking for nginx C developers

2010-04-20T10:00:00.001+08:00

We're looking for (senior) nginx C developers to work with us in Taobao.com's data department here in Beijing.

Now we do have some full-time job headcount (yes, finally!). If you're interested, please send your resume to agentzh at gmail dot com.

We can promise that most of the nginx modules that you write here can be opensourced under the BSD license with the company's full permissions.

Waiting to hear from you ;)

ngx_drizzle v0.0.8 and ngx_rds_json v0.05 released

2010-04-16T16:41:00.001+08:00

On behalf of the ngx_dbd team, I'm happy to announce the new releases of the ngx_drizzle and ngx_rds_json modules.

Here's the download links for these modules:

ngx_drizzle v0.0.8
http://github.com/chaoslawful/drizzle-nginx-module/tarball/v0.0.8
ngx_rds_json v0.05
http://github.com/agentzh/rds-json-nginx-module/tarball/v0.05

ngx_drizzle is a non-blocking upstream module for mysql, drizzle, and
sqlite. Its project page is on GitHub:

http://github.com/chaoslawful/drizzle-nginx-module

ngx_rds_json is an output filter module that can format the Resty DBD
Stream (RDS) outputs of ngx_drizzle and other database driver modules
to JSON. Its project page is also on GitHub:

http://github.com/agentzh/rds-json-nginx-module

Below is a brief list of (major) changes since their last releases:

Fixed a nasty bug in ngx_drizzle's connection pool. Now the drizzle_keepalive directive's overflow=reject option works for real now. Thanks Piotr Sikora for reporting this issue. (agentzh)
Updated the Resty DBD Stream (RDS) format to version 0.0.3 which specifies that SQL NULL field values are represented by setting field length == (uint32_t) -1 and empty trailing data section. Thanks Piotr Sikora for reporting this issue. (agentzh)
ngx_drizzle and ngx_json now both supports RDS v0.0.3 and can differentiate SQL NULL values and SQL empty strings (""). (agentzh)
Now we postpone the header sending action after we parse the RDS header. So errors in RDS headers can result in proper 500 response headers. Thanks Piotr Sikora for suggesting this. (agentzh)
Fixed a small bug in ngx_rds_json that may cause random hang. (This bug also appeared in my fork of ngx_eval, fixed there as well.) (agentzh)
ngx_drizzle now returns 500 Internal Server Error instead of 502 Bad Gateway for bad SQL queries, bad tables, DB connection failures. (Piotr Sikora)
Fixed a bug that could cause double free in ngx_drizzle. (Piotr Sikora)
ngx_drizzle now works with libdrizzle 0.7 and 0.8.

Piotr Sikora does have more fixes and/or improvements in his ngx_postgres module that is expected to be back-ported to ngx_drizzle some time in the near future :)

P.S. The original creator of the ngx_drizzle project, chaoslawful, is currently working on ngx_lua and he's got something runnable already. We'll make the first public release of ngx_lua in the near future. Stay tuned!

Slides for my talk on nginx.conf scripting

2010-04-10T21:56:00.001+08:00

I gave a talk on nginx.conf scripting in today's PerlChina monthly meeting. Here's the (English) slides that I used:

http://agentzh.org/misc/slides/nginx-conf-scripting/nginx-conf-scripting.html

Web browsers other than IE can be used to view these slides. Use your mouse's left and right keys or the arrow keys on your keyboard to move forwards and backwards.

Comments will be highly appreciated as usual ;)

Have fun!

set_encode_base32 and set_decode_base32 for nginx.conf

2010-03-23T10:44:00.001+08:00

I've just implemented the set_encode_base32 and set_decode_base32 in our ngx_set_misc module.

Here's a small example from its test suite:

    location /bar {
        set $a '"hello, world!\nhiya"';
        set_encode_base32 $a;
        set $b $a;
        set_decode_base32 $b;

        echo $a;
        echo $b;
    }

Performing GET /bar yields the following response body

    49k6ar3cdsm20trfe9m6888ad1knio92
    "hello, world!
    hiya"

The first line is the Base32 form for the text spanning the last two lines. See RFC 3548 for more details on the Base32 encoding if you're not familiar with it ;)

A lot of thanks go to chaoslawful++ for implementing the encode_base32 and decode_base32 functions.

Have fun :)

Setting up our pure JS blog app atop ngx_openresty

2010-03-19T14:20:00.001+08:00

We've been working very hard on ngx_openresty in the last few months. For now our blog site written in pure JavaScript has been ported to the new ngx_openresty and it runs *very* fast.

It should be first mentioned that there's not a top-level nginx module named ngx_openresty. Physically ngx_openresty is just a set of loosely coupled nginx modules that can also be used independently.

Here's a brief explanation on how to set it up on your machine. It has been tested on Linux but should also work on *BSD or Mac. If it does not work for you, just ask in the OpenResty mailing list ;)

Grab the various nginx 3rd-party modules:
   ngx_echo http://github.com/agentzh/echo-nginx-module
   ngx_xss http://github.com/agentzh/xss-nginx-module
   NDK http://github.com/agentzh/ngx_devel_kit
   ngx_set_misc http://github.com/agentzh/set-misc-nginx-module
   ngx_rds_json http://github.com/agentzh/rds-json-nginx-module
   ngx_drizzle http://github.com/chaoslawful/drizzle-nginx-module
Grab the latest nginx core:
http://nginx.org/
(Last tested version is 0.8.33.)
Build the nginx executable with these modules enabled:

   ./configure \
   --add-module=/path/to/eval-nginx-module \
   --add-module=/path/to/echo-nginx-module \
   --add-module=/path/to/xss-nginx-module \
   --add-module=/path/to/ndk-nginx-module \
   --add-module=/path/to/set-misc-nginx-module \
   --add-module=/path/to/rds-json-nginx-module \
   --add-module=/path/to/drizzle-nginx-module

Please note that the order of the --add-module option is very important for those filter modules among them.
Set up your mysql database for the blog app using the following SQL file:

   http://github.com/agentzh/ngx_openresty/blob/master/demo/Blog/misc/mysql/init-db.sql

I assume you use the following command to execute the sql against your mysql instance:

   mysql -h localhost -p -u monty test < init-db.sql

where your db user name is "monty" and the test db name is "test".

There's some row data for the posts and comments tables that you can import into your "test" database:

   http://agentzh.org/misc/blog-rows.sql

This is a mysqldump on my side. (Be careful with charsets.)
Grab the following nginx.conf file from my site:

http://agentzh.org/misc/nginx.conf

Edit the following snippet if your mysql database has a different set of connection option:

   upstream backend {
        drizzle_server 127.0.0.1:3306 dbname=test password=some_pass user=monty protocol=mysql;
      drizzle_keepalive max=400 overflow=ignore;
   }

Here I assume your test database is listening to 127.0.0.1:3306 and your db name is "test", your user is "monty" and your password is "some_pass".

Start your nginx compiled earlier using this config file.
Download the blog site written in pure JavaScript and HTML

http://agentzh.org/misc/blog-site-for-ngx-openresty.tar.gz

Unpack it and edit the following line in blog.js if your nginx is not listening to localhost:8080:

var host = 'http://localhost:8080';
Open the index.html in the blog site directory with your favorite web browser and it should get alive ;)

For now, blog post search and rss feed are not implemented. But the former is coming soon because my ngx_array_var module is mostly ready now ;)

There's also plans to port this site over to the PostgreSQL backend. Piotr Sikora has completed the ngx_postgres module and it's compatible with ngx_openresty :)

Companion.JS rocks in IE JS debugging

2010-03-16T09:57:00.001+08:00

I was having a hard time in debugging JavaScript for Internet Explorer (IE). Luckily I've found the Companion.JS addon and it works great in at least IE 7 and very easy to install on a plain Windows :D

The following article gives a detailed explanation on this handy tool's motivation and installation:

http://www.grepsedia.com/development/debugging-javascript-in-ie-without-visual-studios

With this addon, I quicky located a nasty issue in jQuery's $.html method that fails to construct DOM subtrees in IE yesterday. By setting a node's innerHTML attribute directly, I finally got my web app running happily in both IE 7 and IE 6 :D

No pain any more.

Start tweeting

2010-02-05T14:25:00.001+08:00

I've just registered a twitter account and here's my page

http://twitter.com/agentzh

Fee free to follow me. And I'll try sticking to English in my messages ;)

一个关于动态应用单核千级 rps 的传说

2010-02-03T12:20:00.001+08:00

我们在过去的四个月中，已经开发了 8 个 nginx C 模块（第 8 个这两天刚放到 GitHub 上面，名为 ngx_set_misc ），后面还会有更多更精彩的模块甚至应用面世。这些模块的功能看起来很碎很零散，但其实是一个更宏伟的项目的组成部分。这个项目叫做 ngx_openresty.

我们的一个方向是几个客户端 .js 文件 + 几个 .html/.css 文件 + nginx.conf 搞定一个完整的交互式 web 应用，或者至少是一个很复杂应用的比较大的一部分。

不幸的是，业界的一些同仁对这种应用开发模式产生了质疑，认为我们对 nginx 的各种应用层面的扩展让 nginx 自身变得臃肿和低效。本文旨对这些质疑进行一次非正式的驳斥。目的不在于驳人面子和树己威风，而是为了争取更多的朋友加入我们在 nginx 领域的努力。

我们已经在 ngx_openresty 原型的基础上，使用 nginx.conf 和纯客户端 JavaScript 开发了一个比较完整的 blog 应用（源码在这里： http://github.com/agentzh/ngx_openresty/tree/master/demo/Blog/ ）

在我们的高并发实测中，单核能扛千级 rps，小请求在 keepalive 下几百并发也可以扛到 ~ 6000 rps，且禁用了 nginx 级别的结果集 cache，内存、CPU 等资源的占用只是 php 等传统应用的一个零头，呵呵。其主要原因是：

都是认真书写的 C 代码来完成原先 PHP 等代码来完成的事情，
绝大部分应用都是 I/O 密集型的，所以让 nginx 的高效的事件模型来统一调度整个应用的所有网络 I/O 操作（包括到 DB 等后端的），只会进一步提高性能，而非降低。相比之下，用 PHP 等脚本语言访问 mysql，一定会阻塞当前 PHP 进程或者线程，而 PHP 解释器的阻塞开销是非常大的。
我们总是可以通过 LVS + 多机 nginx 来线性 scale. 总体资源是一个常量，不会因为多启几个进程和线程就会增加，只要不是浪费。阻塞在 I/O 操作上、在多进程多线程间无谓地切换和争用、在后端应用和 nginx 之间通过 socket 复制请求和应答数据，等等等等，这些才是浪费。
通过 fastcgi 等协议，PHP 这些脚本语言不免要自己再解析一遍 HTTP 头和 URL 参数啥的，还要自己做 URL 分发；而 nginx 其实一般都已经解析和处理过了。
根据模版渲染 HTML 展现的工作现改由客户浏览器去做了，又进一步节约了服务器端的计算。如果一秒有几千请求，在 cache miss 的情况下，就可以节约几千次 HTML 渲染，这累积起来，相比生成 JSON 是相当可观的，呵呵。

如果还有我没有想到的理由，请来信补充，呵呵。

其实我们的方向不算特别新奇，Apache 2.2 早在几年前就提供了强大的 apr-dbd API 和 mod_dbd 模块来提供从 server 到 mysql/pgsql/oracle/odbc 这些数据库后端的直连服务。

我们在雅虎工作的时候，也曾经通过 Apache 2.2 mod_dbd + PostgreSQL 后端搭建过单机几千 rps 的服务。即使在那个 DB I/O 阻塞 apache worker 线程的场景中，我们也没觉得 Apache 在线上的性能差到哪儿去了，呵呵。

当然，我们有机会在 nginx 世界中做得比 Apache dbd 更好 ;) nginx 的内部架构比 Apache2 优秀多了，这是真的，呵呵。

所以还是面对现实，让我们一起拥抱高简洁和高性能吧！呵呵！

ngx_drizzle v0.0.7: now running on *BSD

2010-02-01T17:17:00.001+08:00

We're proud to announce the v0.0.7 release for the ngx_drizzle module.

ngx_drizzle is an upstream nginx module that can help nginx talk directly to mysql and other RDBMS backends that support the mysql or drizzle TCP protocol.

This release includes patches from our new contributor Piotr Sikora to make this module work with kqueue and other event models on *BSD systems. (Now select/poll/epoll/kqueue all have been tested, but for rgsig, the server will hang.)

Also, this release fixed a long-standing bug captured by Piotr that the nginx process may crash when DB queries complete in a single run of upstream function calls under high traffic.

We have also tested this module with the recent new release of libdrizzle, v0.7.

Release tarballs can be downloaded below

http://github.com/chaoslawful/drizzle-nginx-module/downloads

Enjoy!

ngx_memc v0.06: new directive memc_flags_to_last_modified

2010-01-29T15:28:00.001+08:00

I'm pleased to announce the v0.06 release of the ngx_memc module:

http://wiki.nginx.org/NginxHttpMemcModule

This release has the highlight of a new directive named "memc_flags_to_last_modified".

If this directive is turned on, then for memcached get operations, ngx_memc will read the memcached flags as epoch seconds and set it as the value of the Last-Modified header.

For conditional GET requests, it will signal nginx's not_modified_filter module to return the "304 Not Modified" response to save bandwidth.

Here's a small sample config that I've tested with ngx_memc v0.06:

# read the memcached flags into the Last-Modified header
# to respond 304 to conditional GET
location /memc {
set $memc_key $arg_key;

memc_pass 127.0.0.1:11984;

memc_flags_to_last_modified on;
}

My GET request sets the following header:

If-Modified-Since: Thu, 28 Jan 2010 12:09:23 GMT

And the key stored in memcached has the flags 1264680563. Then we get

$ curl -H 'If-Modified-Since: Thu, 28 Jan 2010 12:09:23 GMT' \
-I 'http://localhost:1984/memc?key=foo'
HTTP/1.1 304 Not Modified
Server: nginx/0.8.32
Date: Fri, 29 Jan 2010 07:10:52 GMT
Last-Modified: Thu, 28 Jan 2010 12:09:23 GMT
Connection: keep-alive

How about setting the flag with ngx_memc too? Well, I'm going to implement the $echo_time and $echo_http_time variables in our ngx_echo module so that we can have

set $memc_flags $echo_time;
add_header Last-Modified $echo_http_time;

for memcached storage operations set/add/etc. That is, $echo_time will return the epoch seconds while $echo_http_time returns the textual representation in the HTTP date format. Patches welcome! Volunteers welcome!

Have fun!

ngx_xss: Native support for cross-site scripting in an nginx

2010-01-26T18:33:00.000+08:00

I'm delighted to announce the first release of our new module, ngx_xss. This output filter module adds native support for simple cross-site AJAX to the nginx server. Currently only cross-site GET is implemented, but cross-site POST support is on our TODO list.

Here's a small example using our ngx_echo module together:

   location /foo {
    default_type "application/json";
    echo '{"errcode":400,"errstr":"Bad Request"}';

    xss_get on; # enable cross-site GET support
    xss_callback_arg callback; # use $arg_callback
}

Then accessing /foo?callback=blah gives the following response:

blah({"errcode":400,"errstr":"Bad Request"}
);

And the ultimate response Content-Type is set to "application/x-javascript", which can be overridden by the "xss_output_type" directive like this

xss_output_type text/javascript;

By default, the ngx_xss module filter will skip responses with Content-Type set to anything other than "application/json". If that's not what you want, you can use the "xss_input_types" directive to override that:

   xss_input_types text/plain text/css;

This module can also be chained with other output filters like ngx_rds_json:

xss_get on;
xss_callback_arg _callback;

location /query {
     drizzle_query "select name from products limit 0, 10";
     drizzle_backend my_mysql;

     rds_json on;
}

Then you can expect something like this when doing GET /query?_callback=OpenResty.callback[32]

OpenResty.callback[32]([{"name":"Bike"},{"name":"Book"}]);

Be careful with the order of output filters while building nginx. The ngx_rds_json filter expects a valid binary stream in the RDS format while the ngx_xss filter expects some JSON text. If you don't take the order right, you'll see your ngx_xss settings get completely ingored in the final responses. Because the ngx_xss filter sees RDS first and it ignores it due to its "application/x-resty-dbd-stream" content type.

Below is the correct nginx configure command if you want to use ngx_xss and ngx_rds_json together:

./configure \
   --add-module=/path/to/xss-nginx-module \
   --add-module=/path/to/rds-json-nginx-module \
   # more options omitted here...

You see, the order of adding output filters on the configure time is just the reversed order that the output filters are actually applied at runtime. Generally speaking, the nginx output filter chain is a stack, not a queue ;)

Only a very limited set of callback values is allowed to prevent JavaScript injection. Valid callback values can be expressed using the following (Ragel) grammar:

identifier = [$A-Za-z] [$A-Za-z0-9_]*;

index = [0-9]* '.' [0-9]+
      | [0-9]+
      ;

main := identifier ( '.' identifier )*
      ('[' index ']')?

This is exactly the Ragel grammar used to generate the C validator used by the ngx_xss module itself.

Here goes the project home page & code repository:

http://github.com/agentzh/xss-nginx-module

as well as the download page for release tarballs:

http://github.com/agentzh/xss-nginx-module/downloads

Enjoy!

ngx_rds_json: help ngx_drizzle and other DBD modules to emit JSON data

2010-01-19T16:18:00.001+08:00

I'm happy to announce the first release of our ngx_rds_json module that can convert Resty DBD Streams (RDS) to JSON.

As some of you might have noticed, the mysql/drizzle DBD driver module ngx_drizzle generates a specific binary stream in a format known as RDS that is defined by ourselves. We introduced RDS just because we don't want to bind to a specific textual format and makes internal data exchange or conversion unnecessarily hard.

As web app developers, we're certainly more interested in more popular textual formats like JSON, YAML, CSV, or even HTML. This module does the job of formating RDS to JSON in a truly streaming fashion.

The project is hosted on GitHub as our other nginx modules:

http://github.com/agentzh/rds-json-nginx-module/

Release tarballs can be downloaded from here

http://github.com/agentzh/rds-json-nginx-module/downloads

Just as the ngx_drizzle module, this module is now considered highly experimental, but it's maturing very rapidly because it's part of our Taobao.com's company $project. If you have found any bugs, or any wishlist, please send us mails or create tickets on GitHub.

Here's some typical use cases drawn from ngx_rds_json's test suite ( http://github.com/agentzh/rds-json-nginx-module/blob/master/test/t/sanity.t ):

Use Case 1

mysql db init

create table cats (id integer, name text);
insert into cats (id) values (2);
insert into cats (id, name) values (3, 'bob');

nginx.conf

upstream backend {
      drizzle_server 127.0.0.1:3306 dbname=test
   password=some_pass user=monty
   protocol=mysql;
}
server {
      ...
      location /mysql {
      drizzle_pass backend;
      drizzle_query 'select * from cats';
      rds_json on;
      }
}

request

GET /mysql

response

   [{"id":2,"name":null},{"id":3,"name":"bob"}]

Use Case 2

mysql db init

(ditto)

nginx.conf

upstream backend {
# ditto
}
server {
      ...
      location /mysql {
      if ($arg_name ~ '[^A-Za-z0-9]') {
      return 400;
      }
      drizzle_pass backend;
      drizzle_query "update cats set name='$arg_name' where name='$arg_name'";
      rds_json on;
      }
}

request

GET /mysql?name=bob

response

{"errcode":0,"errstr":"Rows matched: 1 Changed: 0 Warnings: 0"}

Use Case 3

mysql db init

    create table foo (id serial, flag bit);

nginx.conf

    upstream backend {
      # ditto
    }
    server {
      ...
      location /mysql {
      if ($arg_bit !~ '^[01]$') {
      return 400;
      }
      drizzle_pass backend;
      drizzle_query "insert into foo (flag) values ($arg_bit);";
      rds_json on;
      }
    }

request

GET /mysql?bit=1

response

   {"errcode":0,"insert_id":1,"affected_rows":1}

You'll see fancier (working) use cases in the test suite, by combining ngx_echo module's echo_location or echo_location_async directive. Parallel SQL queries can be very useful for certain applications.

As a side note, chaoslawful++ is working on ngx_rds_tt2 module which will allow us to use Perl TT2's template language to specify custom output formater for RDSs. Here's a quick example that will work very soon:

location /myxml {
   drizzle_query 'select * from products';
   drizzle_pass my_mysql;

   echo_before_body '<?xml version="1.0"?>';
   echo_before_body '<pie>';

   rds_tt2_line_template
'<slice title="[% title | xml %]" color="[% color | xml %]>'
   '[% count %]'
'</slice>';

   echo_after_body '</pie>';
}

And you will get streaming output as well, just buffered on the data line level. (in contrast, ngx_json_rds does not buffer data for large fields like BLOBs).

And there will be ngx_srcache that can allow you to buffer database output by ngx_memc + memcached as well ;)

Stay tuned!

实例分享：利用 Erlang 编写的 TCP Proxy 工具 etcproxy 定位 ngx_drizzle 模块的一个 bug

2010-01-06T18:38:00.001+08:00

阿哲老师（chaoslawful）的 Erlang 版 TCP proxy 工具 etcproxy，刚刚成功地帮我定位了 ngx_drizzle 模块中的几处隐秘的 bug，哈哈！

原来我在 ngx_http_drizzle_output.c 的函数组中的 size 局部变量都忘了初始化为 0 了，哈哈！这种未初始的情形只有在 field 的 packet 被拆分时才会出现。我不用在测试 DB 里准备很大的 blob 数据来测试此种情况，而且非常可靠，每次都能精确复现场景。

定位 bug 的具体过程是这样的：我启动 etcproxy，让它的上游是本地 mysql server 监听的 3307，它自己监听 3306 端口，然后我让 nginx 去连 3306. 这样 etcproxy 就会开始作连接和数据流中传。它提供了一些选项（目前是写在代码内部的宏定义）。我是这样使用的，先设置了按每字节来拆分 packet 数据，即 mysql 发送的 packet 都被强制拆成了一个字节一个 packet. 同时我设置了相邻发送的 packet 之间的延时为 1ms，这样这些小 packet 在 localhost 方式发送时，不会被 kernel 自动合并。于是 nginx 得到的 packet 也是一个字节，再一个字节的。

于是这就触发了 libdrizzle 这个流式 mysql 客户端的 field 分割逻辑，即对于结果集中的一个 field，会分多份返回给我的 ngx_drizzle 模块，而我的模块对于多片的 offset + len 的 field 值接收代码，正好有刚才提到的 size 未初始化为 0 的 bug，于是出现了 size 为 4 这样的随机数的情形。这样最终导致 buffer 使用发生错误，并被我自己的一个 buffer 指针断言给捕捉到了，抛到了 nginx 的 error.log 里；与此同时，我的 Test::Nginx 测试台也发现 nginx 的输出发生了截断，结果不是期望，并在终端上向我报了错误，我也就第一时间看到了。

其实它还有一个功能，我还没有及使用。那就是在指定的读或者写的 data 偏移量处发生超时或者关闭上下游的 TCP 连接。这个也可以精确地测试我的非阻塞客户端的状态机的一些特殊方面，主要就是在特定状态下，超时和连接出错的处理代码，是否如期望那般运行了，哈哈哈。比如在接受 SQL 查询结果的 columns 部分的超时定时器是否如期望的工作，接受 rows 时的超时定时器是否也如期望般的工作了。

今晚我决定给自己做点好吃的，庆祝一下这个 bug 的顺利定位和修复，呵呵！

关于 ngx_drizzle 和 ngx_rds 的一些随机的想法。。。

2010-01-06T12:39:00.001+08:00

我终于想到了为我的 ngx_drizzle 模块再添加一条 drizzle_type 指令，这样用户可以自定义嵌入 SQL 的参数类型和正则约束，比如：

drizzle_type email '^[-A-Za-z0-9._]+\@\w+\.(?:com|cn|net)' quote=on;
drizzle_quote $arg_email email;
drizzle_query "select * from users where email=$arg_email";

这样内建类型 column, table, int, bool 啥的也可以让用户自己覆盖了，哈哈！ngx_drizzle 其实已经拥有了 OpenResty View API 的主要功能了，哈哈哈，而且更灵活，更高效！

我刚才吃早饭的时候，还想到可以为咱的二进制的 Resty DBD Stream (RDS) 格式再引入两种非结构化的查询结果类型

一是结果是单个 field value 的，
二是结果是多个 field value 的线性列表

这样可以在 ngx_rds 模块中通过 output filter 对 RDS 结果集进行 Excel 风格的“单元格”或者“单元格区域”的 pick 操作。只要我们的 ngx_rds_json 等数据格式化器也支持这两种新类型，就可以生成更直接的 JSON 表示，从而简化在 ngx_eval 这样的 subrequest in memory 的上下文中的 if + 正则等判断处理。

我同时还想到了通过更高层的 merger 来融合多个 RDS 流。比如每个流分别来自不同 mysql 机器的 select + order by 查询，然后 merger 再把这些 RDS 结果进行流式的全局的 order by. 在牺牲流处理并限定各个 RDS 支流的情况下，还可以进行多 RDS 支流的全局 join/group by 等操作。

这些都将是 ngx_rds 模块的功能，同时支持 RDS picker 和 merger :D

为 nginx 正名，为 Igor 申冤

2010-01-04T11:48:00.001+08:00

刚才在和 cnhackTNT 聊到 ngx_drizzle 模块未来的若干种极酷的玩法的时候，不由地开始感慨 nginx 社区大部分用户其实并没有真正理解 nginx 的设计真谛。包括目前网上流行的那些 nginx 的玩法，本身都是直接违反作者 Igor Sysoev 老大的设计初衷和本意的。

所以从这个意义上讲，Igor 是孤独的。这也是为什么 Igor 有时终于忍不住在 nginx 列表里揭斯底里地说他压根原本就不想支持 fastcgi，他压根就不想把 nginx 的 proxy 模块做成第二个 Squid。当然了，Igor 自己玩的 ngx_perl 模块也违反了 nginx 的设计原则，包括他自己说要做的 ngx_v8 模块。不过我相信 Igor 这两个模块只是玩玩而已，耍酷之类。

真正对 nginx 有意义的脚本语言引擎，是那些支持 C 级别的 coroutine 的解释器，perl 和 v8 都不满足这一基本条件。（ coco lua 是可以的！）nginx 的核心是非阻塞，是的，再说一遍，非阻塞。Fastcgi 协议的另一边永远是阻塞的，ngx_perl 和 ngx_v8 中的脚本代码运行时永远是阻塞的，ngx_proxy 协议的另一头经常也是阻塞的，所以说它们没有本质的意义和价值，不会带来性能的真正飞跃。

Igor 在 nginx core 中设计了极为精巧的非阻塞的编程模型，无论是 subrequest, 还是 upstream，但极少有人了解这些，
极少人有欣赏这些，更少有人懂得去利用好这些。在非阻塞和并发 I/O 的上下文中，编程模型和传统的代码是很不一样的
就像 JS 的许多写法和 perl 里极不相同一样，其道理本身倒不难理解，但对于"精巧"的感悟，则需要对 core 的更多了解 ;)
基于此，所以我说 Igor 是孤独的，所以我说 nginx 社区许多朋友其实不懂高并发，不懂高性能，不懂 C10K，呵呵。

当然了，这一切我也是在晓哲老师过去几个月每日一课的耐心点拨下，在我手抄了 nginx core 中的大量 C 源之后，我才开悟的。需要我们的更多教程，我们需要向世人揭示这种力量，这种神奇。Igor 老大的表达能力貌似非常有限，至少在英语表达方面，这不得不说是一件憾事。但幸运的是他还是可以用很漂亮的 C 代码来传递和实践他的那些极好的 idea. 我必须说这些 idea 是我在 openresty 中的一些引以为豪的想法的高级发展形式。所以我第一眼看到，我就有着某种强烈的共鸣
即使当时我还没有足够了解其细节，呵呵。Igor 万岁！阿哲万岁！

ngx_drizzle: make nginx talk directly to mysql, drizzle, and sqlite3 by libdrizzle

2009-12-31T19:13:00.000+08:00

This is the last day in 2009 and I'm too impatient to hold the first release (version v0.0.1) of our ngx_drizzle module, an upstream module that talks directly to RDBMS backends like mysql, drizzle, and the drizzle server shipped with libdrizzle for sqlite3.

This module is initially started by my friend and colleague, chaoslawful. His Chinese name is 王晓哲 :) He did the initial (most difficult) work all in his own time :) I also shamelessly borrowed a lot of code from Igor's ngx_http_upstream.c and ngx_http_memcached_module.c in the nginx 0.8.30 core, as well as Maxim Dounin's excellent upstream_keepalive module. These parts of code are copyrighted by these authors, respectively.

This module is still at its very early phase of development and considered highly experimental. But you're encouraged to test it out on your side and report any quirks that you experience :)

Here's some sample configurations:

http {
...

upstream cluster {
# simple round-robin
drizzle_server 127.0.0.1:3306 dbname=test
password=some_pass user=monty protocol=mysql;
drizzle_server 127.0.0.1:1234 dbname=test2
password=pass user=bob protocol=drizzle;
}

upstream backend {
drizzle_server 127.0.0.1:3306 dbname=test
password=some_pass user=monty protocol=mysql;
}

server {
location /mysql {
set $my_sql 'select * from cats';
drizzle_query $my_sql;

drizzle_pass backend;
}
...
}
}

Essentially it provides a very efficient and flexible way for nginx internals to access mysql, drizzle, sqlite3, as well as other RDBMS's that support the drizzle protocol or mysql protocol. Also it can serve as a direct REST interface to those RDBMS backends.

It also has a builtin per-worker connection pool mechanism borrowed from Maxim Dounin's upstream_keepalive module.

Here's a sample configuration:

upstream backend {
drizzle_server 127.0.0.1:3306 dbname=test
password=some_pass user=monty protocol=mysql;
drizzle_keepalive max=100 mode=single overflow=reject;
}

You may wonder why it will be useful for your PHP/Python/Perl/Java applications fastcgi'd or proxied by nginx. Well, we'll work out an ngx_accel_subrequest module some time in the future to allow these backend apps directly issue subrequests by means of the special X-Accel-Subrequest header and continuation passing style.

Unlike the current X-Accel-Redirect trick we're already familiar with, X-Accel-Subrequest is more like a function invocation that issues one (or multiple parallel subrequests), and eventually *returns* back when the resulting data the subrequest obtains is ready and gives control back to your backend apps to go on processing the data.

Sadly this mdoule does not have a wiki page yet, just the source repository on GitHub:

http://github.com/chaoslawful/drizzle-nginx-module

I've done some work in the README file there. I promise I'll work on the wiki doc in another day ;)

Release tarballs can be downloaded from the page below

http://github.com/chaoslawful/drizzle-nginx-module/downloads

I've also cc'd the nginx-devel mailing list in hope to get interested developers to join this project's development. __YOU__ are always welcomed!

Happily, we've already got a working framework for integrating third-party libraries like libdrizzle and libpq into nginx's upstream system as long as the libraries meet the following prerequisites:

It exposes the underlying socket fd.
It has a truly non-blocking API for its operations.

libdrizzle and libpq meet these conditions while oracle's OCI library needs some hack for the requirement #1. So, you can expect ngx_oracle and even ngx_pgsql to be announced here in the next few weeks or so, and you'll see even more! If you'd join the fun, please don't hesitate to drop us a line :)

Happy new year!

P.S. I was hoping to release v0.0.1 by this Christmas, but missed the deadline because I had spent too much time on fixing weird bugs in the ngx_chunkin module :P

The year 2009 is my first year in the nginx community. It's really live and has a lot of fun. I'm going to do more nginx C hacking in the next year! :D

Test::Nginx::LWP and Test::Nginx::Socket are now on CPAN

2009-12-08T13:36:00.001+08:00

I've just released the Perl modules Test::Nginx::LWP and Test::Nginx::Socket as a single Test-Nginx distribution to CPAN (it may still require some more time to reach the CPAN mirror near you):

http://search.cpan.org/perldoc?Test::Nginx::LWP

http://search.cpan.org/perldoc?Test::Nginx::Socket

The latter is still a hack using while (1) with non-blocking sysread/syswrite. I'll rewrite it using IO::Select at some point in the future.

These two Test::Base-style modules are driving the test suites of the following Nginx C modules:

ngx_echo: http://wiki.nginx.org/NginxHttpEchoModule
ngx_headers_more: http://wiki.nginx.org/NginxHttpHeadersMoreModule
ngx_chunkin: http://wiki.nginx.org/NginxHttpChunkinModule
ngx_memc: http://wiki.nginx.org/NginxHttpMemcModule

And our ngx_drizzle module, started by my friend and colleague, chaoslawful++, will soon be on that list as well ;)

Happy testing nginx modules with Perl!

ngx_memc: an extended version of ngx_memcached that supports set, add, delete, and many more commands

2009-12-06T18:27:00.001+08:00

I'm happy to announce the first release of the ngx_memc module, an extended version of the standard "memcached" module that supports almost the whole memcached TCP protocol:

http://wiki.nginx.org/NginxHttpMemcModule

I know the state of the art in the nginx community is to manipulate contents in memcached from within a backend (fastcgi) application while just use the nginx memcached module as a frontend that simply "serves" the content to the outside world. But read on...

Our motivation here, however, is to build an efficient and flexible memcached TCP client component for nginx itself so that we can reuse it in an non-blocking way by means of an nginx subrequest or a "fakerequest" [1].

As always, special thanks go to Igor Sysoev for all his heavy lifting work already done in ngx_http_upstream and ngx_http_memcached_module in the nginx core.

At last but not least, thanks my current employer, Taobao.com, for allowing me to opensource this work, as well as other nginx modules I have already announced or am going to announce here ;)

Enjoy!

References
[1] See http://github.com/srlindsay/nginx-independent-subrequest

Major updates to ngx_chunkin: lots of bug fixes and beginning of keep-alive support

2009-12-04T18:13:00.000+08:00

Prompted by the bug reports from one of my users, the J guy, I've made lots of improvements into the ngx_chunkin module:

http://wiki.nginx.org/NginxHttpChunkinModule

Please see the change log for the new v0.07 release:

http://wiki.nginx.org/NginxHttpChunkinModule#v0.07

It's worth special mentioning that it's really a shame that older versions do not work with chunked data with non-ascii octets. This was caused by incorrect alphtype definition in my Ragel spec. Thanks J for reporting it.

Also, I've introduced a new directive spelled "chunkin_keepalive", which will make ngx_chunkin work in the non-pipelined keep-alive context. Preliminary support for HTTP 1.1 pipelining is also introduced but has not been tested very well. (For technical details, see this nginx-devel thread )

As before, this module is still considered experimental but you're encouraged to try it out and report any issues that you encounter. I promise I'll fix bugs as fast as I can :)

Happy chunking nginx!

The "headers more" module: scripting input and output filters in your Nginx config file

2009-11-18T18:09:00.001+08:00

I've been working madly on the "headers more" module:

http://github.com/agentzh/headers-more-nginx-module

And got everything I want working now. It also has a nice wiki page (which also has brief explanation of the underlying implementation):

http://wiki.nginx.org/NginxHttpHeadersMoreModule

Our buzzword is that it can rewrite the "Server" output header dynamically! See this:

   location /foo {
more_set_headers   "Server: $arg_server";
   }

Then GET /foo?server=Foo will get a response with the "Server: Foo" header set ;)

Input headers can be trivially rewritten as well, including the "Host" header:

    more_set_input_headers   "Host: some-other-host";

Well, the full practical power of this module is out of my current imagination. If you have some crazy uses, please drop me a line ;)

Happy Nginx hacking!

The "chunkin" module: Experimental chunked input support for Nginx

2009-11-15T18:25:00.001+08:00

Pushed by those cutting-edge users on the list, I've quickly worked out "chunkin" module which adds HTTP 1.1 chunked input support for Nginx without the need to patching the core:

http://github.com/agentzh/chunkin-nginx-module

This module registers an access-phase handler that will eagerly read and decode incoming request bodies when a "Transfer-Encoding: chunked" header triggers a 411 error page in Nginx (hey, that's what you have to pay for avoiding patching the core ;)). For requests that are not in the "chunked" transfer encoding, this module is a "no-op".

To enable the magic, just turn on the "chunkin" config option like this:

chunkin on;
location /foo { ... }
....

No other modification is required in your nginx.conf file. (The "chunkin" directive is not allowed to use in the location block BTW.)

This module is still considered highly experimental and there must be some serious bugs lurking somewhere. But you're encouraged to play and test it in your non-production environment and report any quirks to me :)

Efforts have been made to reduce data copying, dynamic memory allocation, and other performance tuning, thus unfortunately raising the risk of potential buffer handling bugs caused by premature optimizations :P

This module is not supposed to be merged into the Nginx core because I've used Ragel to generate the chunked encoding parser for joy :)

The following Nginx versions have been successfully tested by this module's (very limited) test suite:

0.8.0 ~ 0.8.24
0.7.21 ~ 0.7.63

The test suite definitely needs more test cases and the code is hacky in various places. If you're willing to contribute, feel free to ask me for a commit bit in a private email :)

Hacking on the Nginx echo module

2009-10-15T11:01:00.001+08:00

Over the recent weeks, I've been reading a lot of Nginx and its modules' C source code and it's really enjoyable. I've got lots of good ideas in implementing the next generation of OpenResty server based on the Nginx architecture. Well, it's currently my full-time $work anyway.

For the sake of testing other modules, experimenting the Nginx internals, and for fun, I've started my first Nginx module named "echo":

http://github.com/agentzh/echo-nginx-module

It's already quite usable, and it also has a declarative test suite based on Perl's Test::Base. At the moment, LWP is used for simplicity and it's rather weak in testing streaming behavior of Nginx (I'm using "curl" to test these aspects manually for now). I'm considering coding up my own Perl HTTP client library based on IO::Select and IO::Socket (there might be already one around?).

I'm intentionally heavily commenting my C source in this "echo" module in the hope that newcomers would find it a "live tutorial" or something like that. I'll write more about the details here in subsequent posts. After all, it's able to do a lot more thingies other than just "echo" stuffs directly, such as sleeping and flushing output buffer. And it will be capable of outputing subrequests' responses as well.

Happy hacking Nginx C modules and stay tuned!

Slides for my perl testing & VDOM.pm talks in Beijing Perl Workshop 2009

2009-09-20T23:40:00.001+08:00

I really enjoyed the talks in BJPW2009. Here's the slides for my two talks in this conference:

"A Tour of Perl Testing":
Browse URL (Firefox required): http://agentzh.org/misc/slides/BJPW2009/perl-testing.xul
Download URL (Open perl-testing.xul after extraction): http://agentzh.org/misc/slides/BJPW2009.tar.gz
"Introduction to our VDOM & vdom-webkit cluster"
Browse URL (Firefox required): http://agentzh.org/misc/slides/taobao-fe/vdomwebkit.xul
Download URL (Open vdomwebkit.xul after extration): http://agentzh.org/misc/slides/taobao-fe.tar.gz

Hopefully you'll find them interesting ;)

A plan for nginx-openresty

2009-09-14T18:30:00.001+08:00

Now that I've joined Taobao.com's SDS department and will be able to work on OpenResty in a full time fashion, I've just worked out a (somewhat) detailed plan for the next generation of the OpenResty server. Well, sorry, this draft is in Chinese since my $manager reads Chinese better:

http://www.pgsqldb.org/mwiki/index.php/Nginx_openresty_plan (still being actively updated)

After discussing with my friend and colleague chaoslawful++ for possible designs of a high performance implementation of the OpenResty server, we finally decided to rewrite OpenResty.pm in pure C in the form of an nginx module.

Here's some highlights of the Chinese project plan given above:

* Nginx-openresty will remain fully opensource.
* We want to take full advantage of Nginx's event modle and asynchronous I/O and we don't want backend requests blocking as well.
* We want to integrate Coco Lua into nginx's event model leveraging coco's C level coroutines, thus leading to transparent asynchronous I/O on the Lua land. For example, consider the following Lua code

res = http.get('http://www.taobao.com')

will automatically yield the current "lua thread" and register a socket fd to nginx's underlying epoll/kqueue/select/etc model and return control to nginx to do other things. Once data arrives at the socket fd, nginx will get informed and accumulated the response data. When the requested data is completely ready, nginx will resume the pending lua session and the Lua function "http.get" will successfully return.
* We want Lua to become the first-class embedded langauge on the service level and we'd abandon the restyscript language.
* The first two backends we want to implement for nginx-openresty are mysql and Oracle (and Hive afterwards) because these are heavily used in our department's production.
* We'll only allow the HTTP POST method to emulate modification methods like PUT and DELETE. GET will no longer be allowed here to reduce XSS attach risk.
* Password login method will require the client to request a random number from the OpenResty server and use it as a salt to encrypt her password using multi-pass MD5, as in

 passwd = md5(passwd)
 for 1..2 do
    passwd = md5(passwd + salt)
 done

* For the REST interfce, we will introduce Type API to allow user use Lua snippet to define new parameter types for Views and Actions.
* For RDBMS backends, Views and Actions can use the "map_to" attribute to map automatically to underlying DB functions and stored procedures.
* Views and Actions' definitions (ako "queries") will specify in the backend's own query language (like PL/SQL, PL/PgSQL, and etc.). OpenResty willl only recognize special interpolated parameters in the form of $(param_name type=xxx checker=xxx default=xxx ...).
* Views and Actions will support "cached" and "expire" attributes to allow caching of result data set, and "async" attribute to allow time-consuming backend queries to submit to remote queues like memcacheq and eventually run by async daemon workers. The original View invoker will immediately get a job ID for his request (unless the task queue is full) and poll OpenResty's Job API for the status of his task and get the final results when the task is marked "done".
* View/Action parameters can specify types, Lua-specified checkers, and default values. For example:

{"name":"my_view","query":
"select * from animals where age > $(age type=integer checker='return age >= 0 and age <= 100')"}

* Filter, Template, and Trigger APIs will also be introduced :)

I'll create a git repository on GitHub for the nginx-openresty's development. Participation will always be appreciated :) I'll keep you posted.

Slides for my VDOM + WebKit talk

2009-09-04T16:04:00.001+08:00

I gave a presentation on VDOM + WebKit to the Taobao.com Search Frontend Team this morning. The slides are based on my talk in April's Beijing Perl Workshop, but with notable updates to reflect recent changes in the last few months:

http://agentzh.org/misc/slides/taobao-fe/vdomwebkit.xul (Firefox required to open this link)

Be patient when it's downloading big images while browsing the slides, or you can download the whole tarball to your local side, unpack the package, open the vdomwebkit.xul in it, and browse the slides locally:

http://agentzh.org/misc/slides/taobao-fe.tar.gz

Recent major development regarding our browser-based web scraping clusters are:

We've switched from the Visual DOM Firefox extension completely to the VDOM Browser based on QtWebKit for hunter development.
We've switched from OpenResty + Pg to our queue-size-ware version of memcacheq to coordinate the whole cluster.
We've extensively used our new "VDOM spectroscapy algorithm" to establish corronspondence between text nodes in similar page regions.

I'll use almost the same slides for my 3rd (lightening) talk at the annual Beijing Perl Workshop conference weeks later :)

Our queue-size-aware version of memcacheq

2009-09-03T18:19:00.001+08:00

Xunxin++ and I have been working on a fork of memcacheq (originally just within the company), adding support for queue length constraint.

In our scenario, a pipelined webpage information extraction cluster based on apple's WebKit core, it's important to limit the queue's length and to make the queue "inform" the queue item producers by some way in case the queue is full.

We're not sure if it's worth merging back to the mainstream version because this new addition adds some cost (though the cost is low).

Here goes the project page on GitHub, with more explanation of the details (I won't repeat them here ;)):

http://github.com/agentzh/memcacheq/tree/master

The newly added code is also licensed under the same license as the mainstream memcacheq.

Enjoy :)

I'll talk in the upcoming Beijing Perl Workshop 2009 event

2009-09-02T17:29:00.001+08:00

I've submitted 3 talk proposals to this year's upcoming Beijing Perl Workshop conference scheduled at Sep 19. I'll publish my slides for my talks later here for your preview :)

The 3 talks are

Wonders of Perl automated testing
Inventing mini languages in Perl
Web Scraping based on Perl + WebKit + VDOM

If you feel like attending the conference, please register in the conference site below:

http://conference.perlchina.org/bjpw2009/

Don't forget to specify a T-shirt size in your profile setting there so that we can prepare a T-shirt for you in this event (well, it's for free!).

See you there ;)

OpenResty.pm has been moved to GitHub

2009-05-11T16:02:00.001+08:00

As some of you may have already noticed, I've moved the source repository of OpenResty.pm from the good old OpenFoundry to GitHub:

http://github.com/agentzh/openresty/tree/master

Feel free to branch it or ask me for a commit bit if you don't have one ;)

I'll destroy the stuffs in the old "openapi" repository on openfoundry and leave a note there to avoid potential confusion.

Mailing list for OpenResty

2009-05-11T14:51:00.001+08:00

After releasing several new releases of OpenResty.pm to CPAN, I created a mailing list for OpenResty users/developers on Google Groups:

http://groups.google.com/group/openresty?hl=en

This is for both OpenResty.pm and mod_openresty. You're very welcome to join us there ;) There's also a #openresty on freenode but it's been very quiet :P

Text::SmartLinks: The Perl 6 love for Perl 5

2009-04-28T14:05:00.001+08:00

I'm so glad to find this blog post while browsing the Iron Man planet:

http://szabgab.com/blog/2009/04/1240827553.html

Three years ago, I wrote the smartlinks.pl script to integrate the Pugs test suite with the Perl 6 Synopses documentation.

Gábor Szabó now has done an excellent job in refactoring and packaging the tool into a general-purpose CPAN module. It had been my TODO until I was caught by accumulated schoolwork :P

Enjoy his (well, also our) Text::SmartLinks module!

http://search.cpan.org/perldoc?Text::SmartLinks

SSH::Batch: Treating clusters as maths sets and intervals

2009-04-23T10:23:00.001+08:00

System administration is also part of my $work. Playing with a (big) bunch of machines without a handy tool is painful. So I refactored some of our old scripts and released SSH::Batch, a collection of useful parallel ssh scripts, to CPAN:

    http://search.cpan.org/dist/SSH-Batch/

SSH::Batch allows you to name your clusters using variables and interval/set syntax in your ~/.fornodesrc config file. For instance:

    $ cat ~/.fornodesrc
    A=foo[01-03].com bar.org
    B=bar.org baz[a-b,d,e-g].cn foo02.com
    C={A} * {B}
    D={A} - {B}

where cluster C is the intersection set of cluster A and B while D is those machines in A but not in B.

And then you can query machine host list by using SSH::Batch's fornodes script:

   $ fornodes '{C}'
   bar.org
   foo02.com

   $ fornodes '{D}'
   foo01.com
   foo03.com

Furthermore, to run a command on a cluster by the concurrency of 6:

   atnodes 'ls -lh' '{A} + {B}' my.more.com -c 6

Or upload a local file to the remote cluster:

  tonodes ~/my.tar.gz '{A} / {B}' :/tmp/

There's also a key2nodes script to push SSH public keys to remote machines ;)

A colleague in Alibaba B2B is already using it. And one of my teammates is going to use it to operate on those thousands of machines in our instance of the YST (Yahoo! Search Technology) cluster and I'm ready to receive more feedback from him ;)

Have fun :)

My VDOM.pm & WebKit Cluster Talk at the April Meeting of Beijing Perl Workshop

2009-04-10T11:51:00.001+08:00

Last night I gave a talk to our PerlChina folks at the April meeting in the Flow Bar. Here's the slides that I used:

Firefox XUL format: Browse in Firefox, Download tarball
PDF format
PPT format

The XUL format is the best among the three ;)

Just as the topic of the talk suggests, we're migrating from Firefox clusters to WebKit ones. I'll post more details here in the near future.

Enjoy!