Nginx image processing server with OpenResty and Lua

Today I'll be showing you how to create a fast on the fly image processing server.The whole system can be created in less than 100 lines of code.

We'll be usingOpenResty, an enhanced distribution of Nginx. We'll alsoneed to write a little bit ofLua to get all the functionality we want.Lastly, we'll be using thisLua ImageMagick binding. If you're notfamiliar with any of these that’s OK, I'll be showing you how to get everythingrunning from scratch.

The code

Because the entire system isn’t that many lines of code I'll show everythingfirst and you can read on if you want to learn more about how it works.

You can also find the code in thisGitrepository.

nginx.conf

# These three directives should be tweaked for productionerror_logstderrnotice;daemonoff;events{}http{include/usr/local/openresty/nginx/conf/mime.types;server{listen80;location@image_server{content_by_lua_file"serve_image.lua";}location~^/images/(?<sig>[^/]+)/(?<size>[^/]+)/(?<path>.*\.(?<ext>[a-z_]*))${rootcache;set_md5$digest"$size/$path";try_files/$digest.$ext@image_server;}}}

serve_image.lua

localsig,size,path,ext=ngx.var.sig,ngx.var.size,ngx.var.path,ngx.var.extlocalsecret="hello_world"-- signature secret keylocalimages_dir="images/"-- where images come fromlocalcache_dir="cache/"-- where images are cachedlocalfunctionreturn_not_found(msg)ngx.status=ngx.HTTP_NOT_FOUNDngx.header["Content-type"]="text/html"ngx.say(msgor"not found")ngx.exit(0)endlocalfunctioncalculate_signature(str)returnngx.encode_base64(ngx.hmac_sha1(secret,str)):gsub("[+/=]",{["+"]="-",["/"]="_",["="]=","}):sub(1,12)endifcalculate_signature(size.."/"..path)~=sigthenreturn_not_found("invalid signature")endlocalsource_fname=images_dir..path-- make sure the file existslocalfile=io.open(source_fname)ifnotfilethenreturn_not_found()endfile:close()localdest_fname=cache_dir..ngx.md5(size.."/"..path).."."..ext-- resize the imagelocalmagick=require("magick")magick.thumb(source_fname,size,dest_fname)ngx.exec(ngx.var.request_uri)

What’s an image processing server?

An image processing server is a web application that is concerned with takingan image path along with a set of manipulation instructions and returning themanipulated image.

A good example is user avatar images. If you let your users upload their ownimages then the images probably come in a handful of different sizes andformats. When displaying the image you might have it on many different pagesand require many different sizes. In order to avoid resizing up front you canuse an image processing server to get the image sizes you want on demand justby requesting a special URL.

Additionally, if the URL is requested multiple times the resized image should becached so it can be returned to the user instantly.

Initial thoughts

The first step to this project is to design a URL structure. In this tutorial I'lluse the following format:

/images/SIGNATURE/SIZE/PATH

Given an image,leafo.jpg, and a desired size,100x100, we might requestthe URL:

/images/abcd123/100x100/leafo.png

You'll notice I've included a section for a signature in the URL. We'll beusing some basic cryptography to ensure a stranger can’t request images of anysize. This is an important thing to consider as image processing can take a lot ofCPU power. If someone were to write a malicious script that iterates over alarge quantity of image sizes they could max out your CPU in an attempt toperform aDenial-of-service attack.

The signature is the result of a cryptographic function run on a portion of theURL (the size and path) and a secret key. To verify that the URL of theresized image is valid you need to perform a simple assertion:

assert(calculate_signature("100x100/leafo.png")=="abcd123")

Lastly, it’s worth mentioning image sources and caching. For simplicity theimages will be loaded directly from local disk. Although it’s perfectlypossible to load them from external places, like S3 or other URLs, it wont becovered in this tutorial.

Modified images will be cached to disk, and cache expiration wont be covered.This is perfectly fine for most cases.

Installation requirements

You can download the latest version of OpenResty from here:http://openresty.org/#Download

Installation is simple, after extracting the archive just run

$ ./configure --with-luajit$ make$ make install

You can find more detailed installation instructionson the officialsite.

On my system this places OpenResty at/usr/local/openresty/, so when it comestime to run Nginx we'll be running:/usr/local/openresty/nginx/sbin/nginx

OpenResty comes with Lua, so the last component is theImageMagickbinding.

If you're familiar with Lua already, you can use LuaRocks to do the install:

luarocks install magick

Nginx configuration

Our Nginx configuration is concerned with serving cached images or executingour Lua script for un-cached images. It’s listed in full at the top of thepost, but here I'll step all the pieces explain their roles.

error_logstderrnotice;daemonoff;events{}

These are basic settings that I use for doing Nginx configuration development.I leave most settings default but I disable the daemon and make sureinformation is printed to standard out. When deploying your Nginx applicationyou'll want to spend some time adding some additional directives to make sureit can run as fast as possible.

http{include/usr/local/openresty/nginx/conf/mime.types;

Thehttp block defines our HTTP settings, the only thing to be done here isinclude themime.types file. This is a file that comes with Nginx. When Nginxserves a file from disk is uses the mime types configuration file to correctlyset theContent-type header.

server{listen80;

Theserver block configures our server. For illustrative purposes I'vebound to port 80 (even though port 80 is the default). In the server block wedeclarelocation blocks, which are destinations for requests.

location@image_server{content_by_lua_file"serve_image.lua";}

The first location defined is called@image_server. The@ signifies a namedlocation. A named location can not be externally accessed by any URL, it canonly be called upon by other locations. We'll execute this location whenthe file we want doesn’t exist in the cache.

location~^/images/(?<sig>[^/]+)/(?<size>[^/]+)/(?<path>.*\.(?<ext>[a-z_]*))${

This location matches our image URLs. The regular expression here is quite bigso let’s step through it.

Following the wordlocation is~, this instructs Nginx to perform casesensitive regular expression matching against the incoming request path.

As I mentioned before our URL structure is:

/images/SIGNATURE/SIZE/PATH

And the regular expression is:

^/images/(?<sig>[^/]+)/(?<size>[^/]+)/(?<path>.*\.(?<ext>[a-z_]*))$

^/images/ will match the left hand side of the path. Following that is aseries of named capture groups that match each part of the request pathseparated by/. Although it would be possible to shorten the regularexpression by avoiding the named captures, I decided to use them becausethey'll help with the readability of our script.

$(?<sig>[^/]+) is a named capture group. It captures as many characters as itcan that aren’t/ and assigns it to the namesig. Thesize named captureworks the same.

(?<path>.*\.(?<ext>[a-z_]*))$ captures the path of our image, which iseverything else in the URL. It contains an inner named capture group to extractthe extension of the image.

Named capture groups are interesting in Nginx becuase their results can be useddirectly as variables. For example, if we were using theecho module, wecould just dump the extension of the request usingecho $ext; in the Nginxconfiguration.

Now the body of the location:

rootcache;set_md5$digest"$size/$path";try_files/$digest.$ext@image_server;

This will only execute if the regular expression from above matches.

rootcache;

This sets the directory of the cache where files will be searched usingtry_files.

set_md5$digest"$size/$path";

This calculates a hash of the image path and size. The MD5 digest is used as thename of the file in the cache.

try_files/$digest.$ext@image_server;

Finally we usetry_files to either load the existing image from the cache orpass the request off to the@image_server location we defined earlier.

Lua script

Our Lua script runs when the@image_server location is executed. Its job isto verify the signature, ensure the image exists, then resize and serve theimage.

The script should be saved in the current directory of Nginx, normally next tothenginx.conf. I've also called the scriptserve_image.lua. It must matchwhat is referenced in the configuration.

localsig,size,path,ext=ngx.var.sig,ngx.var.size,ngx.var.path,ngx.var.extlocalsecret="hello_world"-- signature secret keylocalimages_dir="images/"-- where images come fromlocalcache_dir="cache/"-- where images are cached

The first step is to set some variables that will be used. The named capturegroup variables are pulled in as local variables to make their access moreconvenient.

localfunctionreturn_not_found(msg)ngx.status=ngx.HTTP_NOT_FOUNDngx.header["Content-type"]="text/html"ngx.say(msgor"not found")ngx.exit(0)end

If an invalid URL is accessed the server should gracefully show a 404 message.The functionreturn_not_found sets the correct status code, prints amessage and exits.

localfunctioncalculate_signature(str)returnngx.encode_base64(ngx.hmac_sha1(secret,str)):gsub("[+/=]",{["+"]="-",["/"]="_",["="]=","}):sub(1,12)end

This is the function that signs our URL using the secret key. I've opted totake the first 12 characters of the base64 encoded result of the HMAC-SHA1.

Additionally I usegsub to translate characters that have special meanings inURLs to avoid any potential URL encoding issues.

Now that everything has been declared we can continue on with the logic of thefile.

ifcalculate_signature(size.."/"..path)~=sigthenreturn_not_found("invalid signature")end

Here we verify that the signature is correct. If it’s not what we expect basedon the rest of the URL then a 404 is returned.

localsource_fname=images_dir..path-- make sure the file existslocalfile=io.open(source_fname)ifnotfilethenreturn_not_found()endfile:close()

These lines of check for the existence the file. In Lua we can check if a fileis readable by trying to open it. If the file can’t be opened we abort. Wedon’t need to read the file here so we close.

localdest_fname=cache_dir..ngx.md5(size.."/"..path).."."..ext-- resize the imagelocalmagick=require("magick")magick.thumb(source_fname,size,dest_fname)

dest_fname is set to the same hashed name we searched for in our Nginxconfiguration. The file can be picked up automatically by Nginxtry_fileson any subsequent requests.

Now that the request has been verified it’s time to do the resize. We pass thesize string directly into Magick’sthumbfunction. This gives us nice syntaxfor various types of resizes and crops, like100x100 for a resize, or10x10+5+5 for a crop.

ngx.exec(ngx.var.request_uri)

Now that the image is written we are ready to display it to the browser. HereI've trigger a request to the current location,request_uri. Normally thiswould trigger a loop error but, because we've written the cached file,try_files will return the file and skip the Lua script.

Running the server

Now were ready to try it out. We'll run Nginx isolated in its own directory.This directory should start out withnginx.conf,serve_image.lua, and animages directory.

Before starting the server you should place some images in theimagesdirectory.

You should be inside of the directory where we want the server to run beforerunning the following commands.

Create the cache directory:

$ mkdir cache

Initialize some files our configuration requires for starting:

$ mkdir logs$ touch logs/error.log

Now start the server:

$ /usr/local/openresty/nginx/sbin/nginx -p"$(pwd)" -c"nginx.conf"

Assuming the server has started we can now access the server. For example, ifyou have an imageleafo.jpg you might resize it by going to the followingURL:http://localhost/images/LMzEhc_nPYwX/80x80/leafo.jpg.

Final notes

That’s all there is to it. With some minor tweaks to the initialization innginx.conf your server is ready to go live.

There are a couple additional things you could also do:

If you already have an Nginx installation you could integrate this code into itso you don’t have to run separate Nginx processes.

If you are using the image server with another web application you'll need towrite thecalculate_signature function inside of your application so you cangenerate valid URLs.

If you're concerned about the cache taking up too much space with unused imagesizes you could look into creating a system that deletes unused cached entries.

Thanks for reading, leave a comment if you any suggestions or are confusedabout anything.

A Lua module that provides LuaJIT FFI to MagickWand, the image library included in Image Magick.