Developers familiar with Aerospike will know that the vast majority of the API calls take a“policy” as a parameter. This policy is optional but can have a significant impact on the application. There are a number of parameters in policies that can be confusing todevelopers, and even creating the policies is frequently done incorrectly. This blog will attempt to clear up some of this confusion.

What are policies?

Each API call in Aerospike typically takes a policy as a parameter. For example, consider a simple get in Java:

client.put(null,newKey("test","testSet",1),newBin("name","Tim"),newBin("age",312));

The first parameter here is the policy; in this case, it is an instance of theWritePolicy class. Ifnull is specified, then the default policy is used. Policies typically contain fields that fall into one of three categories:

• Networking control: How should the API call behave if something goes wrong? For example, if the server cannot be reached, should the call be retried on a different server?

• Application behavior: Parameters that control how the API call should behave in various application-driven scenarios. For example, Aerospike does an “upsert” operation by default on aput, selecting to either update or insert the record depending on whether it already exists or not. However, the application might require the record to be inserted only, failing if the record already exists in the database.

• Filtering: Aerospike supports filters on operations, allowing almost all operations to become conditional. For example, an operation can be specified to retrieve a record if theamount bin in that record is greater than a thousand.

These three categories are all in the same policy object, with the networking control defined on thePolicy class, which is a superclass of other policies, such asWritePolicy. Note that thePolicy class is used for read operations without a subclass; there is noReadPolicy class.

Networking control policies

The policy superclass defines a number of parameters related to networking. These are critical to optimizing application behavior but are often poorly understood. Let’s take a look at these and the interaction between them.

These networking fields are:

• connectTimeout
• maxRetries
• replica
• sleepBetweenRetries
• socketTimeout
• timeoutDelay
• totalTimeout

A diagram is helpful in understanding the interaction between these fields:

The diagram shows a timeline of what happens when an API call is submitted to the Aerospike Client Library (ACL) and can be read from left to right.

The first thing that happens is the ACL needs a connection to the server which holds the data. If this is a simple get or put, this will be a connection to a single server, but if it’s a complex operation such as a batch or query, multiple servers might be involved. The ACL has connection pools to each server node, so acquiring this connection is typically a matter of borrowing one from the pool, which is almost instantaneous. However, if the pool has no available connections, a new connection must be established, which can take some time. This is especially true if the connection is over TLS as the handshake between the client and server on a TLS connection can be slow.

This connection establishment process has its own timeout field,connectTimeout. This field is in milliseconds and includes both the connection establishment as well as user authentication (if any). By default, this is set to zero, which is usually a reasonable default – this uses the lesser of thesocketTimeout andtotalTimeout, or 2,000 milliseconds if both are zero. If the application can tolerate extra time when a connection is established, especially when using TLS, then this can be set to a non-zero value. For example, if you want a largeconnectTimeout and a smallsocketTimeout to allow for protracted SSL handshakes but fast application responses, setting this value to something like 5,000 milliseconds works well.

Once a connection has been retrieved, the API call is sent to the server. There are two timeouts compared at this point, thesocketTimeout and thetotalTimeout. Both are in milliseconds, and thesocketTimeout reflects how long it takes to wait for data from this API call before retrying or failing. ThetotalTimeout specifies the maximum time the call can take, including retries. If the time remaining in thetotalTimeout is less than thesocketTimeout, then thetotalTimeout is used as thesocketTimeout for the call.

In Figure 1, thesocketTimeout is less than thetotalTimeout, so the ACL waits until one of the following occurs:socketTimeout milliseconds have elapsed, the connection returns a timeout, or the call succeeds. If the call was unsuccessful in this period, the ACL checks themaxRetries setting to see if any retries remain. For reads, this value defaults to two, giving it a total of three tries – the initial attempt plus the two retries. For writes, this value defaults to zero, so no retries are attempted.

Figure 1 depicts a read with the default two retries, so once the originalsocketTimeout expires, the API call does not return to the application. Instead, the ACL waits forsleepBetweenRetries milliseconds and then submits the call again.

This second call may or may not go to the same server. This is controlled by thereplica policy setting, which defaults toSEQUENCE. Sequence means that the first attempt is made to the node containing the master record, but on failure, the retry is made to a replica. This can be very useful in a read-sensitive use case, for example. If thesocketTimeout is set to a short interval (say 20 milliseconds) and there are retries with a replica policy of SEQUENCE if the master node is busy and cannot respond in time or has fallen over. Still, if the cluster has not detected this yet, the retry will go to the replica and may return data within the application’s SLA.

If this second call again times out aftersocketTimeout milliseconds, a second delay ofsleepBetweenRetry milliseconds is performed then the second retry (3rd total attempt) is performed.

Let’s take a look at where this leaves us in Figure 1:

The ACL is about to start the second retry, but in this case, the remaining time in thetotalTimeout is less than thesocketTimeout. Hence the API call will timeout before thesocketTimeout elapses when the remaining time in thetotalTimeout has elapsed.

Note that once thetotalTimeout has elapsed, the client API call fails, even if further retries are available.

Timeout delay

The only timeout setting not covered in the above is thetimeoutDelay setting. This is not strictly related to the API behavior but rather the system's efficiency. As the above shows, when the ACL submits the call to the server node, and no response is received withinsocketTimeout milliseconds, either the call is abandoned, or a retry is performed. But what happens to the connection on which that original request was placed?

By default, that connection is closed, destroying the connection and not placing it back in the connection pool. This is because it is possible that the server simply did not respond in time, and it might respond some time later. If the connection was reused instead of being closed, this server response might confuse the communication of the reusing call.

However, as discussed above, connection establishment can be expensive. Closing connections typically means that this connection must be re-established later, which can impact the application.

ThetimeoutDelay parameter specifies a time interval in milliseconds for those connections to be kept alive. If the server responds within that timeout, the socket is drained and then returned to the connection pool. If the timeout is exceeded, then the connection is closed.

Note that there is a cost of using thetimeoutDelay as the ACL must keep track of which connections have timed out. It is useful in situations where timeouts may occur frequently (for example, aggressive read settings) and connection establishment is expensive.

Default policies

Now that the timeout settings are understood, we can discuss default policies. The ACL defines default values of the various policy attributes, which users can customize to suit their application instead of repeatedly modifying the defaults in the application code. Default policies are set up when the ACL is created and are used whennull is passed to the policy parameter in the API call. Most often, the default policies are customized with the network settings discussed above.

Here is an example which creates a defaultwritePolicy with asocketTimeout of 2,000 milliseconds:

WritePolicywritePolicyDefault=newWritePolicy();writePolicyDefault.socketTimeout=2000;ClientPolicyclientPolicy=newClientPolicy();clientPolicy.writePolicyDefault=writePolicyDefault;IAerospikeClientclient=newAerospikeClient(clientPolicy,"127.0.0.1",3000);

Note that thewritePolicyDefault is set on theClientPolicy instance before the ACL is created. When passed to the ACL constructor, a copy of the default policies is made so no changes are possible to the default policies.

After executing this code, passingnull to a write API call will result in the ACL retrieving and using the defaultWritePolicy. Hence, the following two calls are identical:

client.put(null,key,newBin("name","joe"));client.put(client.getWritePolicyDefault(),key,newBin("name","joe"));

Creating policies

Timeout settings are typically set on default policies and reused multiple times across the application. However, application-level settings are done as needed by the application on a per-call basis. Consider a check-and-set scenario where a record has been read from the database, changes made, and then those changes pushed back to the database, but only if the record has not been changed since it was read:

Recordrecord=client.get(null,key);...WritePolicywritePolicy=client.getWritePolicyDefault();writePolicy.generation=record.generation;writePolicy.generationPolicy=GenerationPolicy.EXPECT_GEN_EQUAL;client.put(writePolicy,key,newBin("name","joe"));

This code attempts to solve the requirements, getting thedefaultWritePolicy from the ACL and then setting the generation andgenerationPolicy correctly. However, this code is very dangerous and will create issues in the application. There is just onedefaultWritePolicy on the ACL, and thegetWritePolicyDefault()call will return a reference to that object, not a copy of that object. Hence, setting the generation values on that policy will affect all calls that use that default policy on all threads, almost certainly causing many calls to fail with an exception.

Another common pattern to solve this problem is:

WritePolicywritePolicy=newWritePolicy();writePolicy.generation=record.generation;writePolicy.generationPolicy=GenerationPolicy.EXPECT_GEN_EQUAL;client.put(writePolicy,key,newBin("name","joe"));

While this code is an improvement on the previous snippet, it too, has a problem. In this case, a uniqueWritePolicy instance has been created, preventing issues with this call affecting other calls. However, using newWritePolicy() does not pick up the settings of the default write policy, so settings changed on this default policy, such as thesocketTimeout will not apply to this new policy.

Thecorrect way to create a new policy to override fields is as follows:

WritePolicywritePolicy=newWritePolicy(client.getWritePolicyDefault());writePolicy.generation=record.generation;writePolicy.generationPolicy=GenerationPolicy.EXPECT_GEN_EQUAL;client.put(writePolicy,key,newBin("name","joe"));

In this case, a newWritePolicy is created, but it gets a copy of all the settings off the default write policy. This object is local to this API call, so there is no contention with other calls.

Summary

Aerospike’sPolicy classes are both powerful and flexible, allowing fine-grained, per-call control over both network and application-level settings. However, it is important to understand what the settings do to correctly control the API call. Network settings are often overlooked during the application development phase and are frequently poorly understood. A thorough understanding of them is critical to developing applications that perform well not just in the happy-day scenario but also in the face of network issues or server failures.

Developers who are new to Aerospike should understand this simple pattern to create new policies. It will ensure the correct execution of the API without affecting other concurrent calls and unintended surprises.