Repository files navigation

ngx_memc - An extended version of the standard memcached module that supports set, add, delete, and many more memcached commands.

This module is not distributed with the Nginx source. Seethe installation instructions.

• Version
• Synopsis
• Description
  • Keep-alive connections to memcached servers
  • How it works
• Memcached commands supported
  • get $memc_key
  • set $memc_key $memc_flags $memc_exptime $memc_value
  • add $memc_key $memc_flags $memc_exptime $memc_value
  • replace $memc_key $memc_flags $memc_exptime $memc_value
  • append $memc_key $memc_flags $memc_exptime $memc_value
  • prepend $memc_key $memc_flags $memc_exptime $memc_value
  • delete $memc_key
  • delete $memc_key $memc_exptime
  • incr $memc_key $memc_value
  • decr $memc_key $memc_value
  • flush_all
  • flush_all $memc_exptime
  • stats
  • version
• Directives
  • memc_pass
  • memc_cmds_allowed
  • memc_flags_to_last_modified
  • memc_connect_timeout
  • memc_send_timeout
  • memc_read_timeout
  • memc_buffer_size
  • memc_ignore_client_abort
• Installation
  • For Developers
• Compatibility
• Community
  • English Mailing List
  • Chinese Mailing List
• Report Bugs
• Source Repository
• Changes
• Test Suite
• TODO
• Getting involved
• Author
• Copyright & License
• See Also

This document describes ngx_memcv0.19 released on 19 April 2018.

 # GET /foo?key=dog # # POST /foo?key=cat # Cat's value... # # PUT /foo?key=bird # Bird's value... # # DELETE /foo?key=Tigerlocation /foo{set$memc_key$arg_key;     # $memc_cmd defaults to get for GET,     #   add for POST, set for PUT, and     #   delete for the DELETE request method.     memc_pass127.0.0.1:11211;}

 # GET /bar?cmd=get&key=cat # # POST /bar?cmd=set&key=dog # My value for the "dog" key... # # DELETE /bar?cmd=delete&key=dog # GET /bar?cmd=delete&key=doglocation /bar{set$memc_cmd$arg_cmd;set$memc_key$arg_key;set$memc_flags$arg_flags; # defaults to 0set$memc_exptime$arg_exptime; # defaults to 0     memc_pass127.0.0.1:11211;}

 # GET /bar?cmd=get&key=cat # GET /bar?cmd=set&key=dog&val=animal&flags=1234&exptime=2 # GET /bar?cmd=delete&key=dog # GET /bar?cmd=flush_alllocation /bar{set$memc_cmd$arg_cmd;set$memc_key$arg_key;set$memc_value$arg_val;set$memc_flags$arg_flags; # defaults to 0set$memc_exptime$arg_exptime; # defaults to 0     memc_cmds_allowed getset add delete flush_all;     memc_pass127.0.0.1:11211;}

http{     ...upstream backend{server127.0.0.1:11984;server127.0.0.1:11985;}server{location /stats{set$memc_cmd stats;             memc_pass backend;}         ...}}   ...

 # read the memcached flags into the Last-Modified header # to respond 304 to conditional GETlocation /memc{set$memc_key$arg_key;     memc_pass127.0.0.1:11984;     memc_flags_to_last_modified on;}

location /memc{set$memc_key foo;set$memc_cmd get;     # access the unix domain socket listend by memcached     memc_pass unix:/tmp/memcached.sock;}

This module extends the standardmemcached module to support almost the wholememcached ascii protocol.

It allows you to define a customREST interface to your memcached servers or access memcached in a very efficient way from within the nginx server by means of subrequests orindependent fake requests.

This module is not supposed to be merged into the Nginx core because I've usedRagel to generate the memcached response parsers (in C) for joy :)

If you are going to use this module to cache location responses out of the box, trysrcache-nginx-module with this module to achieve that.

When used in conjunction withlua-nginx-module, it is recommended to use thelua-resty-memcached library instead of this module though, because the former is much more flexible and memory-efficient.

Back to TOC

You needHttpUpstreamKeepaliveModule together with this module for keep-alive TCP connections to your backend memcached servers.

Here's a sample configuration:

http{upstream backend{server127.0.0.1:11211;       # a pool with at most 1024 connections       # and do not distinguish the servers:keepalive1024;}server{         ...location /memc{set$memc_cmd get;set$memc_key$arg_key;             memc_pass backend;}}}

Back to TOC

It implements the memcached TCP protocol all by itself, based upon theupstream mechanism. Everything involving I/O is non-blocking.

The module itself does not keep TCP connections to the upstream memcached servers across requests, just like other upstream modules. For a working solution, see sectionKeep-alive connections to memcached servers.

Back to TOC

The memcached storage commandsset,add,replace,prepend, andappend uses the$memc_key as the key,$memc_exptime as the expiration time (or delay) (defaults to 0),$memc_flags as the flags (defaults to 0), to build the corresponding memcached queries.

If$memc_value is not defined at all, then the request body will be used as the value of the$memc_value except for theincr anddecr commands. Note that if$memc_value is defined as an empty string (""), that empty string will still be used as the value as is.

The following memcached commands have been implemented and tested (with their parameters marked by correspondingnginx variables defined by this module):

Back to TOC

Retrieves the value using a key.

location /foo{set$memc_cmd'get';set$memc_key'my_key';       memc_pass127.0.0.1:11211;add_header X-Memc-Flags$memc_flags;}

Returns200 OK with the value put into the response body if the key is found, or404 Not Found otherwise. Theflags number will be set into the$memc_flags variable so it's often desired to put that info into the response headers by means of the standardadd_header directive.

It returns502 forERROR,CLIENT_ERROR, orSERVER_ERROR.

Back to TOC

To use the request body as the memcached value, just avoid setting the$memc_value variable:

   # POST /foo   # my value...location /foo{set$memc_cmd'set';set$memc_key'my_key';set$memc_flags12345;set$memc_exptime24;       memc_pass127.0.0.1:11211;}

Or let the$memc_value hold the value:

location /foo{set$memc_cmd'set';set$memc_key'my_key';set$memc_flags12345;set$memc_exptime24;set$memc_value'my_value';       memc_pass127.0.0.1:11211;}

Returns201 Created if the upstream memcached server repliesSTORED,200 forNOT_STORED,404 forNOT_FOUND,502 forERROR,CLIENT_ERROR, orSERVER_ERROR.

The original memcached responses are returned as the response body except for404 NOT FOUND.

Back to TOC

Similar to theset command.

Back to TOC

Similar to theset command.

Back to TOC

Similar to theset command.

Note that at least memcached version 1.2.2 does not support the "append" and "prepend" commands. At least 1.2.4 and later versions seem to supports these two commands.

Back to TOC

Similar to theappend command.

Back to TOC

Deletes the memcached entry using a key.

location /fooset$memc_cmd delete;set$memc_key my_key;       memc_pass127.0.0.1:11211;}

Returns200 OK if deleted successfully,404 Not Found forNOT_FOUND, or502 forERROR,CLIENT_ERROR, orSERVER_ERROR.

The original memcached responses are returned as the response body except for404 NOT FOUND.

Back to TOC

Similar to thedelete $memc_key command except it accepts an optionalexpiration time specified by the$memc_exptime variable.

This command is no longer available in the latest memcached version 1.4.4.

Back to TOC

Increments the existing value of$memc_key by the amount specified by$memc_value:

location /foo{set$memc_cmd incr;set$memc_key my_key;set$memc_value 2;       memc_pass127.0.0.1:11211;}

In the preceding example, every time we access/foo will cause the value ofmy_key increments by2.

Returns200 OK with the new value associated with that key as the response body if successful, or404 Not Found if the key is not found.

It returns502 forERROR,CLIENT_ERROR, orSERVER_ERROR.

Back to TOC

Similar toincr $memc_key $memc_value.

Back to TOC

Mark all the keys on the memcached server as expired:

location /foo{set$memc_cmd flush_all;       memc_pass127.0.0.1:11211;}

Back to TOC

Just likeflush_all but also accepts an expiration time specified by the$memc_exptime variable.

Back to TOC

Causes the memcached server to output general-purpose statistics and settings

location /foo{set$memc_cmd stats;       memc_pass127.0.0.1:11211;}

Returns200 OK if the request succeeds, or 502 forERROR,CLIENT_ERROR, orSERVER_ERROR.

The rawstats command output from the upstream memcached server will be put into the response body.

Back to TOC

Queries the memcached server's version number:

location /foo{set$memc_cmd version;       memc_pass127.0.0.1:11211;}

Returns200 OK if the request succeeds, or 502 forERROR,CLIENT_ERROR, orSERVER_ERROR.

The rawversion command output from the upstream memcached server will be put into the response body.

Back to TOC

All the standardmemcached module directives in nginx 0.8.28 are directly inherited, with thememcached_ prefixes replaced bymemc_. For example, thememcached_pass directive is spelledmemc_pass.

Here we only document the most important two directives (the latter is a new directive introduced by this module).

Back to TOC

syntax:memc_pass <memcached server IP address>:<memcached server port>

syntax:memc_pass <memcached server hostname>:<memcached server port>

syntax:memc_pass <upstream_backend_name>

syntax:memc_pass unix:<path_to_unix_domain_socket>

default:none

context:http, server, location, if

phase:content

Specify the memcached server backend.

Back to TOC

syntax:memc_cmds_allowed <cmd>...

default:none

context:http, server, location, if

Lists memcached commands that are allowed to access. By default, all the memcached commands supported by this module are accessible.An example is

location /foo{set$memc_cmd$arg_cmd;set$memc_key$arg_key;set$memc_value$arg_val;        memc_pass127.0.0.1:11211;        memc_cmds_allowed get;}

Back to TOC

syntax:memc_flags_to_last_modified on|off

default:off

context:http, server, location, if

Read the memcached flags as epoch seconds and set it as the value of theLast-Modified header. For conditional GET, it will signal nginx to return304 Not Modified response to save bandwidth.

Back to TOC

syntax:memc_connect_timeout <time>

default:60s

context:http, server, location

The timeout for connecting to the memcached server, in seconds by default.

It's wise to always explicitly specify the time unit to avoid confusion. Time units supported are "s"(seconds), "ms"(milliseconds), "y"(years), "M"(months), "w"(weeks), "d"(days), "h"(hours), and "m"(minutes).

This time must be less than 597 hours.

Back to TOC

syntax:memc_send_timeout <time>

default:60s

context:http, server, location

The timeout for sending TCP requests to the memcached server, in seconds by default.

It is wise to always explicitly specify the time unit to avoid confusion. Time units supported are "s"(seconds), "ms"(milliseconds), "y"(years), "M"(months), "w"(weeks), "d"(days), "h"(hours), and "m"(minutes).

This time must be less than 597 hours.

Back to TOC

syntax:memc_read_timeout <time>

default:60s

context:http, server, location

The timeout for reading TCP responses from the memcached server, in seconds by default.

It's wise to always explicitly specify the time unit to avoid confusion. Time units supported are "s"(seconds), "ms"(milliseconds), "y"(years), "M"(months), "w"(weeks), "d"(days), "h"(hours), and "m"(minutes).

This time must be less than 597 hours.

Back to TOC

syntax:memc_buffer_size <size>

default:4k/8k

context:http, server, location

This buffer size is used for the memory buffer to hold

• the complete response for memcached commands other thanget,
• the complete response header (i.e., the first line of the response) for theget memcached command.

This default size is the page size, may be4k or8k.

Back to TOC

syntax:memc_ignore_client_abort on|off

default:off

context:location

Determines whether the connection with a memcache server should be closed when a client closes a connection without waiting for a response.

This directive was first added in thev0.14 release.

Back to TOC

You're recommended to install this module (as well as the Nginx core and many other goodies) via theOpenResty bundle. See theinstallation steps forOpenResty.

Alternatively, you can compile this module into the standard Nginx source distribution by hand:

Grab the nginx source code fromnginx.org, for example,the version 1.13.6 (seenginx compatibility), and then build the source with this module:

 wget'http://nginx.org/download/nginx-1.13.6.tar.gz' tar -xzvf nginx-1.13.6.tar.gzcd nginx-1.13.6/# Here we assume you would install you nginx under /opt/nginx/. ./configure --prefix=/opt/nginx \     --add-module=/path/to/memc-nginx-module make -j2 make install

Download the latest version of the release tarball of this module frommemc-nginx-module file list.

Starting from NGINX 1.9.11, you can also compile this module as a dynamic module, by using the--add-dynamic-module=PATH option instead of--add-module=PATH on the./configure command line above. And then you can explicitly load the module in yournginx.conf via theload_moduledirective, for example,

load_module /path/to/modules/ngx_http_memc_module.so;

Back to TOC

The memached response parsers were generated byRagel. If you want toregenerate the parser's C file, i.e.,src/ngx_http_memc_response.c, use the following command from the root of the memc module's source tree:

 $ ragel -G2 src/ngx_http_memc_response.rl

Back to TOC

The following versions of Nginx should work with this module:

• 1.17.x (last tested: 1.17.4)
• 1.16.x
• 1.15.x (last tested: 1.15.8)
• 1.14.x
• 1.13.x (last tested: 1.13.6)
• 1.12.x
• 1.11.x (last tested: 1.11.2)
• 1.10.x
• 1.9.x (last tested: 1.9.15)
• 1.8.x
• 1.7.x (last tested: 1.7.10)
• 1.6.x
• 1.5.x (last tested: 1.5.12)
• 1.4.x (last tested: 1.4.4)
• 1.2.x (last tested: 1.2.9)
• 1.1.x (last tested: 1.1.5)
• 1.0.x (last tested: 1.0.10)
• 0.9.x (last tested: 0.9.4)
• 0.8.x (last tested: 0.8.54)
• 0.7.x >= 0.7.46 (last tested: 0.7.68)

It's worth mentioning that some 0.7.x versions older than 0.7.46 might also work, but I can't easily test them because the test suite makes extensive use of theecho module'secho_location directive, which requires at least nginx 0.7.46 :)

Earlier versions of Nginx like 0.6.x and 0.5.x willnot work.

If you find that any particular version of Nginx above 0.7.46 does not work with this module, please considerreporting a bug.

Back to TOC

Back to TOC

Theopenresty-en mailing list is for English speakers.

Back to TOC

Theopenresty mailing list is for Chinese speakers.

Back to TOC

Although a lot of effort has been put into testing and code tuning, there must be some serious bugs lurking somewhere in this module. So whenever you are bitten by any quirks, please don't hesitate to

1. create a ticket on theissue tracking interface provided by GitHub,
2. or send a bug report or even patches to thenginx mailing list.

Back to TOC

Available on github atopenresty/memc-nginx-module.

Back to TOC

The changes of every release of this module can be obtained from the OpenResty bundle's change logs:

http://openresty.org/#Changes

Back to TOC

This module comes with a Perl-driven test suite. Thetest cases aredeclarative too. Thanks to theTest::Base module in the Perl world.

To run it on your side:

 $ PATH=/path/to/your/nginx-with-memc-module:$PATH prove -r t

You need to terminate any Nginx processes before running the test suite if you have changed the Nginx server binary.

EitherLWP::UserAgent orIO::Socket is used by thetest scaffold.

Because a single nginx server (by default,localhost:1984) is used across all the test scripts (.t files), it's meaningless to run the test suite in parallel by specifying-jN when invoking theprove utility.

You should also keep a memcached server listening on the11211 port at localhost before running the test suite.

Some parts of the test suite requires modulesrewrite andecho to be enabled as well when building Nginx.

Back to TOC

• add support for the memcached commandscas,gets andstats $memc_value.
• add support for thenoreply option.

Back to TOC

You'll be very welcomed to submit patches to theauthor or just ask for a commit bit to thesource repository on GitHub.

Back to TOC

Yichun "agentzh" Zhang (章亦春)<agentzh@gmail.com>, OpenResty Inc.

This wiki page is also maintained by the author himself, and everybody is encouraged to improve this page as well.

Back to TOC

The code base is borrowed directly from the standardmemcached module in the Nginx core. This part of code is copyrighted by Igor Sysoev and Nginx Inc.

Copyright (c) 2009-2018, Yichun "agentzh" Zhang (章亦春)agentzh@gmail.com, OpenResty Inc.

This module is licensed under the terms of the BSD license.

Redistribution and use in source and binary forms, with or withoutmodification, are permitted provided that the following conditionsare met:

• Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
• Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOTLIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FORA PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHTHOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITEDTO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, ORPROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OFLIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDINGNEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THISSOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Back to TOC

• The original announcement email on the nginx mailing list:ngx_memc: "an extended version of ngx_memcached that supports set, add, delete, and many more commands"
• My slides demonstrating various ngx_memc usage:http://agentzh.org/misc/slides/nginx-conf-scripting/nginx-conf-scripting.html#34 (use the arrow or pageup/pagedown keys on the keyboard to swith pages)
• The latestmemcached TCP protocol.
• Thengx_srcache module
• Thelua-resty-memcached library based on thelua-nginx-module cosocket API.
• The standardmemcached module.
• Theecho module for Nginx module's automated testing.
• The standardheaders module and the 3rd-parthheaders-more module.