Repository files navigation

Hiredis-cluster is a C client library for cluster deployments of theRedis database.

Hiredis-cluster is usingHiredis for theconnections to each Redis node.

Hiredis-cluster is a fork of Hiredis-vip, with the following improvements:

• The C libraryhiredis is an external dependency rather than a builtin partof the cluster client, meaning that the latesthiredis can be used.
• Support for SSL/TLS introduced in Redis 6
• Support for IPv6
• Support authentication using AUTH
• Uses CMake (3.11+) as the primary build system, but optionally Make can be used directly
• Code style guide (using clang-format)
• Improved testing
• Memory leak corrections and allocation failure handling
• Low-level API for sending commands to specific node

• Redis Cluster

  • Connect to a Redis cluster and run commands.

• Multi-key commands

  • SupportMSET,MGET andDEL.
  • Multi-key commands will be processed and sent to slot owning nodes.(This breaks the atomicity of the commands if the keys reside on differentnodes so if atomicity is important, use these only with keys in the samecluster slot.)

• Pipelining

  • Send multiple commands at once to speed up queries.
  • Supports multi-key commands described in above bullet.

• Asynchronous API

  • Send commands asynchronously and let a callback handle the response.
  • Needs an external event loop system that can be attached using an adapter.

• SSL/TLS

  • Connect to Redis nodes using SSL/TLS (supported from Redis 6)

• IPv6

  • Handles clusters on IPv6 networks

Prerequisites:

• A C compiler (GCC or Clang)
• CMake and GNU Make (but seeAlternative build using Makefiledirectly below for how to buildwithout CMake)
• hiredis >= v1.0.0; downloaded automatically bydefault, seebuild options to disable.
• libevent (libevent-dev in Debian); can be avoidedif building without tests (DISABLE_TESTS=ON)
• OpenSSL (libssl-dev in Debian) if building with TLS support

Hiredis-cluster will be built as a shared librarylibhiredis_cluster.so andit depends on the hiredis shared librarylibhiredis.so.

When SSL/TLS support is enabled an extra librarylibhiredis_cluster_ssl.sois built, which depends on the hiredis SSL support librarylibhiredis_ssl.a.

A user project that needs SSL/TLS support should link to bothlibhiredis_cluster.soandlibhiredis_cluster_ssl.so to enable the SSL/TLS configuration API.

$ mkdir build;cd build$ cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_SSL=ON ..$ make

The following CMake options are available:

• DOWNLOAD_HIREDIS
  • OFF CMake will search for an already installed hiredis (for example thethe Debian packagelibhiredis-dev) for header files and linkage.
  • ON (default) hiredis will be downloaded fromGithub, built and installed locally inthe build folder.
• ENABLE_SSL
  • OFF (default)
  • ON Enable SSL/TLS support and build its tests (also affect hiredis whenDOWNLOAD_HIREDIS=ON).
• DISABLE_TESTS
  • OFF (default)
  • ON Disable compilation of tests (also affect hiredis whenDOWNLOAD_HIREDIS=ON).
• ENABLE_IPV6_TESTS
  • OFF (default)
  • ON Enable IPv6 tests. Requires that IPv6 issetup in Docker.
• ENABLE_COVERAGE
  • OFF (default)
  • ON Compile using build flags that enables the GNU coverage toolgcovto provide test coverage information. This CMake option also enables a newbuild targetcoverage to generate a test coverage report usinggcovr.
• USE_SANITIZERCompile using a specific sanitizer that detect issues. The value of thisoption specifies which sanitizer to activate, but it depends on support in thecompiler.Common option values are:address,thread,undefined,leak

Options needs to be set with the-D flag when generating makefiles, e.g.

cmake -DENABLE_SSL=ON -DUSE_SANITIZER=address ..

The build uses CMake'sfind_packageto search for ahiredis installation. CMake will search for ahiredisinstallation in the default paths, searching for a file calledhiredis-config.cmake.The default search path can be altered viaCMAKE_PREFIX_PATH oras described in the CMake docs; a specific path can be set using a flag like:-Dhiredis_DIR:PATH=${MY_DIR}/hiredis/share/hiredis

Seeexamples/using_cmake_separate/build.sh orexamples/using_cmake_externalproject/build.sh for alternative CMake builds.

The list of commands and the position of the first key in the command line isdefined incmddef.h which is included in this repo. It has been generatedusing the JSON files describing the syntax of each command in the Redisrepository, which makes sure hiredis-cluster supports all commands in Redis, atleast in terms of cluster routing. To add support for custom commands defined inRedis modules, you can regeneratecmddef.h using the scriptgencommands.py.Use the JSON files from Redis and any additional files on the same format asarguments to the script. For details, see the comments insidegencommands.py.

When a simpler build setup is preferred a provided Makefile can be used directlywhen building. A benefit of this, instead of using CMake, is that it also providesa static library, a similar limitation exists in the CMake files in hiredis v1.0.0.

The only option that exists in the Makefile is to enable SSL/TLS support viaUSE_SSL=1

By default the hiredis library (and headers) installed on the system is used,but alternative installations can be used by defining the compiler flagsCFLAGS andLDFLAGS.

Seeexamples/using_make/build.sh for anexample build using an alternative hiredis installation.

Build failures likehircluster_ssl.h:33:10: fatal error: hiredis/hiredis_ssl.h: No such file or directoryindicates that hiredis is not installed on the system, or that a givenCFLAGS is wrong.Use the previous mentioned build example as reference.

Prerequisites:

• Perl withJSON module.Can be installed usingsudo cpan JSON.
• Docker

Some tests needs a Redis cluster and that can be setup by the make targetsstart/stop. The clusters will be setup using Docker and it may take a whilefor them to be ready and accepting requests. Runmake start to start theclusters and then wait a few seconds before runningmake test.To stop the running cluster containers runmake stop.

$ make start$ maketest$ make stop

If you want to set up the Redis clusters manually they should run on localhostusing following access ports:

The functionredisClusterContextInit is used to create aredisClusterContext.The context is where the state for connections is kept.

The functionredisClusterSetOptionAddNodes is used to add one or many Redis Cluster addresses.

The functionsredisClusterSetOptionUsername andredisClusterSetOptionPassword are used to configure authentication, causingthe AUTH command to be sent on every new connection to Redis.

For more options, see the filehircluster.h.

The functionredisClusterConnect2 is used to connect to the Redis Cluster.

TheredisClusterContext struct has an integererr field that is non-zero when the connection isin an error state. The fielderrstr will contain a string with a description of the error.After trying to connect to Redis usingredisClusterContext you should check theerr field to seeif establishing the connection was successful:

redisClusterContext*cc=redisClusterContextInit();redisClusterSetOptionAddNodes(cc,"127.0.0.1:6379,127.0.0.1:6380");redisClusterConnect2(cc);if (cc!=NULL&&cc->err) {printf("Error: %s\n",cc->errstr);// handle error}

There is a hook to get notified when certain events occur.

intredisClusterSetEventCallback(redisClusterContext*cc,void(fn)(constredisClusterContext*cc,intevent,void*privdata),void*privdata);

The callback is called withevent set to one of the following values:

• HIRCLUSTER_EVENT_SLOTMAP_UPDATED when the slot mapping has been updated;
• HIRCLUSTER_EVENT_READY when the slot mapping has been fetched for the firsttime and the client is ready to accept commands, useful when initiating theclient withredisClusterAsyncConnect2() where a client is not immediatelyready after a successful call;
• HIRCLUSTER_EVENT_FREE_CONTEXT when the cluster context is being freed, sothat the user can free the event privdata.

There is a hook to get notified about connect and reconnect attempts.This is useful for applying socket options or access endpoint information for a connection to a particular node.The callback is registered using the following function:

intredisClusterSetConnectCallback(redisClusterContext*cc,void(fn)(constredisContext*c,intstatus));

The callback is called just after connect, before TLS handshake and Redis authentication.

On successful connection,status is set toREDIS_OK and the redisContext(defined in hiredis.h) can be used, for example, to see which IP and port it'sconnected to or to set socket options directly on the file descriptor which canbe accessed asc->fd.

On failed connection attempt, this callback is called withstatus set toREDIS_ERR. Theerr field in theredisContext can be used to find outthe cause of the error.

The functionredisClusterCommand takes a format similar to printf.In the simplest form it is used like:

reply=redisClusterCommand(clustercontext,"SET foo bar");

The specifier%s interpolates a string in the command, and usesstrlen todetermine the length of the string:

reply=redisClusterCommand(clustercontext,"SET foo %s",value);

Internally, hiredis-cluster splits the command in different arguments and willconvert it to the protocol used to communicate with Redis.One or more spaces separates arguments, so you can use the specifiersanywhere in an argument:

reply=redisClusterCommand(clustercontext,"SET key:%s %s",myid,value);

Commands will be sent to the cluster node that the client perceives handling the given key.If the cluster topology has changed the Redis node might respond with a redirection errorwhich the client will handle, update its slotmap and resend the command to correct node.The reply will in this case arrive from the correct node.

If a node is unreachable, for example if the command times out or if the connecttimes out, it can indicated that there has been a failover and the node is nolonger part of the cluster. In this case,redisClusterCommand returns NULL andsetserr anderrstr on the cluster context, but additionally, hirediscluster schedules a slotmap update to be performed when the next command issent. That means that if you try the same command again, there is a good chancethe command will be sent to another node and the command may succeed.

Hiredis-cluster supports mget/mset/del multi-key commands.The command will be splitted per slot and sent to correct Redis nodes.

Example:

reply=redisClusterCommand(clustercontext,"mget %s %s %s %s",key1,key2,key3,key4);

When there is a need to send commands to a specific node, the following low-level API can be used.

reply=redisClusterCommandToNode(clustercontext,node,"DBSIZE");

This function handles printf like arguments similar toredisClusterCommand(), but willonly attempt to send the command to the given node and will not perform redirects or retries.

If the command times out or the connection to the node fails, a slotmap updateis scheduled to be performed when the next command is sent.redisClusterCommandToNode also performs a slotmap update if it has previouslybeen scheduled.

To disconnect and free the context the following function can be used:

voidredisClusterFree(redisClusterContext*cc);

This function closes the sockets and deallocates the context.

The functionredisClusterGetReply is exported as part of the Hiredis API and can be usedwhen a reply is expected on the socket. To pipeline commands, the only things that needsto be done is filling up the output buffer. For this cause, the following commands can be used thatare identical to theredisClusterCommand family, apart from not returning a reply:

intredisClusterAppendCommand(redisClusterContext*cc,constchar*format, ...);intredisClusterAppendCommandArgv(redisClusterContext*cc,intargc,constchar**argv);/* Send a command to a specific cluster node */intredisClusterAppendCommandToNode(redisClusterContext*cc,redisClusterNode*node,constchar*format, ...);

After calling either function one or more times,redisClusterGetReply can be used to receive thesubsequent replies. The return value for this function is eitherREDIS_OK orREDIS_ERR, wherethe latter means an error occurred while reading a reply. Just as with the other commands,theerr field in the context can be used to find out what the cause of this error is.

voidredisClusterReset(redisClusterContext*cc);

Warning: You must callredisClusterReset function after one pipelining anyway.

Warning: CallingredisClusterReset without pipelining first will reset all Redis connections.

The following examples shows a simple cluster pipeline:

redisReply*reply;redisClusterAppendCommand(clusterContext,"SET foo bar");redisClusterAppendCommand(clusterContext,"GET foo");redisClusterGetReply(clusterContext,&reply);// reply for SETfreeReplyObject(reply);redisClusterGetReply(clusterContext,&reply);// reply for GETfreeReplyObject(reply);redisClusterReset(clusterContext);

Hiredis-cluster comes with an asynchronous cluster API that works with many event systems.Currently there are adapters that enables support forlibevent,libev,libuv,gliband Redis Event Library (ae). For usage examples, see the test programs with the differentevent librariestests/ct_async_{libev,libuv,glib}.c.

The hiredis library has adapters for additional event systems that easily can be adaptedfor hiredis-cluster as well.

There are two alternative ways to initiate a cluster client which also determineshow the client behaves during the initial connect.

The first alternative is to use the functionredisClusterAsyncConnect, which initiallyconnects to the cluster in a blocking fashion and waits for the slotmap before returning.Any command sent by the user thereafter will create a new non-blocking connection,unless a non-blocking connection already exists to the destination.The function returns a pointer to a newly createdredisClusterAsyncContext struct anditserr field should be checked to make sure the initial slotmap update was successful.

// Insufficient error handling for brevity.redisClusterAsyncContext*acc=redisClusterAsyncConnect("127.0.0.1:6379",HIRCLUSTER_FLAG_NULL);if (acc->err) {printf("error: %s\n",acc->errstr);exit(1);}// Attach an event engine. In this example we use libevent.structevent_base*base=event_base_new();redisClusterLibeventAttach(acc,base);

The second alternative is to useredisClusterAsyncContextInit andredisClusterAsyncConnect2which avoids the initial blocking connect. This connection alternative requires an attachedevent engine whenredisClusterAsyncConnect2 is called, but the connect and the initialslotmap update is done in a non-blocking fashion.

This means that commands sent directly afterredisClusterAsyncConnect2 may failbecause the initial slotmap has not yet been retrieved and the client doesn't know whichcluster node to send the command to. You may use theeventCallbackto be notified when the slotmap is updated and the client is ready to accept commands.An crude example of using the eventCallback can be found inthis testcase.

// Insufficient error handling for brevity.redisClusterAsyncContext*acc=redisClusterAsyncContextInit();// Add a cluster node address for the initial connect.redisClusterSetOptionAddNodes(acc->cc,"127.0.0.1:6379");// Attach an event engine. In this example we use libevent.structevent_base*base=event_base_new();redisClusterLibeventAttach(acc,base);if (redisClusterAsyncConnect2(acc)!=REDIS_OK) {printf("error: %s\n",acc->errstr);exit(1);}

UseredisClusterSetEventCallback withacc->ccas the context to get notified when certain events occur.

Because the connections that will be created are non-blocking,the kernel is not able to instantly return if the specifiedhost and port is able to accept a connection.Instead, use a connect callback to be notified when a connectionis established or failed.Similarily, a disconnect callback can be used to be notified abouta disconnected connection (either because of an error or per user request).The callbacks are installed using the following functions:

intredisClusterAsyncSetConnectCallback(redisClusterAsyncContext*acc,redisConnectCallback*fn);intredisClusterAsyncSetDisonnectCallback(redisClusterAsyncContext*acc,redisConnectCallback*fn);

The callback functions should have the following prototype,aliased toredisConnectCallback:

void(constredisAsyncContext*ac,intstatus);

Alternatively, ifhiredis >= v1.1.0 is used, you set a connect callbackthat will be passed a non-constredisAsyncContext* on invocation (e.g.to be able to set a push callback on it).

intredisClusterAsyncSetConnectCallbackNC(redisClusterAsyncContext*acc,redisConnectCallbackNC*fn);

The callback function should have the following prototype,aliased toredisConnectCallbackNC:

void(redisAsyncContext*ac,intstatus);

On a connection attempt, thestatus argument is set toREDIS_OKwhen the connection was successful.The file description of the connection socket can be retrievedfrom a redisAsyncContext asac->c->fd.On a disconnect, thestatus argument is set toREDIS_OKwhen disconnection was initiated by the user,orREDIS_ERR when the disconnection was caused by an error.When it isREDIS_ERR, theerr field in the context can be accessedto find out the cause of the error.

You don't need to reconnect in the disconnect callback.Hiredis-cluster will reconnect by itself when the next command for this Redis node is handled.

Setting the connect and disconnect callbacks can only be done once per context.For subsequent calls it will returnREDIS_ERR.

In an asynchronous cluster context, commands are automatically pipelined due to the nature of an event loop.Therefore, unlike the synchronous API, there is only a single way to send commands.Because commands are sent to Redis Cluster asynchronously, issuing a command requires a callback functionthat is called when the reply is received. Reply callbacks should have the following prototype:

void(redisClusterAsyncContext*acc,void*reply,void*privdata);

Theprivdata argument can be used to carry arbitrary data to the callback from the point wherethe command is initially queued for execution.

The most commonly used functions to issue commands in an asynchronous context are:

intredisClusterAsyncCommand(redisClusterAsyncContext*acc,redisClusterCallbackFn*fn,void*privdata,constchar*format, ...);intredisClusterAsyncCommandArgv(redisClusterAsyncContext*acc,redisClusterCallbackFn*fn,void*privdata,intargc,constchar**argv,constsize_t*argvlen);intredisClusterAsyncFormattedCommand(redisClusterAsyncContext*acc,redisClusterCallbackFn*fn,void*privdata,char*cmd,intlen);

These functions works like their blocking counterparts. The return value isREDIS_OK when the commandwas successfully added to the output buffer andREDIS_ERR otherwise. When the connection is beingdisconnected per user-request, no new commands may be added to the output buffer andREDIS_ERR isreturned.

If the reply for a command with aNULL callback is read, it is immediately freed. When the callbackfor a command is non-NULL, the memory is freed immediately following the callback: the reply is onlyvalid for the duration of the callback.

All pending callbacks are called with aNULL reply when the context encountered an error.

When there is a need to send commands to a specific node, the following low-level API can be used.

intredisClusterAsyncCommandToNode(redisClusterAsyncContext*acc,redisClusterNode*node,redisClusterCallbackFn*fn,void*privdata,constchar*format, ...);intredisClusterAsyncCommandArgvToNode(redisClusterAsyncContext*acc,redisClusterNode*node,redisClusterCallbackFn*fn,void*privdata,intargc,constchar**argv,constsize_t*argvlen);intredisClusterAsyncFormattedCommandToNode(redisClusterAsyncContext*acc,redisClusterNode*node,redisClusterCallbackFn*fn,void*privdata,char*cmd,intlen);

These functions will only attempt to send the command to a specific node and will not perform redirects or retries,but communication errors will trigger a slotmap update just like the commonly used API.

Asynchronous cluster connections can be terminated using:

voidredisClusterAsyncDisconnect(redisClusterAsyncContext*acc);

When this function is called, connections arenot immediately terminated. Instead, newcommands are no longer accepted and connections are only terminated when all pending commandshave been written to a socket, their respective replies have been read and their respectivecallbacks have been executed. After this, the disconnection callback is executed with theREDIS_OK status and the context object is freed.

There are a few hooks that need to be set on the cluster context object after it is created.See theadapters/ directory for bindings tolibevent and a range of other event libraries.

AredisClusterNodeIterator can be used to iterate on all known master nodes in a cluster context.First it needs to be initiated usingredisClusterInitNodeIterator() and then you can repeatedlycallredisClusterNodeNext() to get the next node from the iterator.

voidredisClusterInitNodeIterator(redisClusterNodeIterator*iter,redisClusterContext*cc);redisClusterNode*redisClusterNodeNext(redisClusterNodeIterator*iter);

The iterator will handle changes due to slotmap updates by restarting the iteration, but on the newset of master nodes. There is no bookkeeping for already iterated nodes when a restart is triggered,which means that a node can be iterated over more than once depending on when the slotmap update happenedand the change of cluster nodes.

Note that whenredisClusterCommandToNode is called, a slotmap update canhappen if it has been scheduled by the previous command, for example if theprevious call toredisClusterCommandToNode timed out or the node wasn'treachable.

To detect when the slotmap has been updated, you can check if the iterator'sslotmap version (iter.route_version) is equal to the current cluster context'sslotmap version (cc->route_version). If it isn't, it means that the slotmaphas been updated and the iterator will restart itself at the next call toredisClusterNodeNext.

Another way to detect that the slotmap has been updated is toregister an eventcallback and look for the eventHIRCLUSTER_EVENT_SLOTMAP_UPDATED.

This library usesrandom() while selectinga node used for requesting the cluster topology (slotmap). A user should seedthe random number generator usingsrandom()to get less predictability in the node selection.

Hiredis-cluster uses hiredis allocation structure with configurable allocation and deallocation functions. By default they just point to libc (malloc,calloc,realloc, etc).

If you have your own allocator or if you expect an abort in out-of-memory cases,you can configure the used functions in the following way:

hiredisAllocFuncsmyfuncs= {    .mallocFn=my_malloc,    .callocFn=my_calloc,    .reallocFn=my_realloc,    .strdupFn=my_strdup,    .freeFn=my_free,};// Override allocators (function returns current allocators if needed)hiredisAllocFuncsorig=hiredisSetAllocators(&myfuncs);

To reset the allocators to their default libc functions simply call:

hiredisResetAllocators();