The Standard Tier of Memorystore for Redis provides the ability to scale yourapplication's read queries using read replicas. This page assumes you arefamiliar with the different MemorystoreRedis tier capabilities.

Read replicas allow you to scale your read workload by querying the replicas. Aread endpoint is provided to make it easier for the applications to distributequeries across replicas. For more information, seeScaling reads with read endpoint.

Note: The read replicas are maintained using asynchronous replication. Ifapplications require "read your write" consistency then querying from readreplicas can return inconsistent data because the replica can lag by a fewseconds. If you require "read your write" consistency you should query using theprimary endpoint.

For instructions on managing a Redis instance with read replicas, seeManage read replicas.

Use cases for read replicas

Session store, leaderboard, recommendation engine, and other use cases requirethe instance to be highly available. For these use cases there are many morereads than writes, and these use cases are generally able to tolerate some stalereads. In cases such as these, it makes sense to leverage read replicas toincrease the availability and scalability of the instance.

Read replica behavior

• Read replicas are not enabled on Standard Tier instances by default.
• Once read replicas are enabled on an instance read replicas can no longer bedisabled for that instance.
• Standard Tier instances can have 1 to 5 read replicas.
• Theread endpointprovides a single endpoint for distributing queries across replica nodes.
• Read replicas are maintained usingRedis asynchronous replication.

Caveats and limitations

• Read replicas are only supported for instance sizes with nodes >= 5 GB.
• Read replicas can only be enabled on instances that use Redis version 5.0 orhigher.
• If you designate a zone and alternate zone for provisioning nodes,Memorystore uses those zones for the first and second nodes inthe instance. After that, Memorystore selects the zones for allremaining nodes provisioned for the instance.
• You must provision the instance with a CIDR IP address range of/28 orgreater. Larger range sizes like/27 and/26 are valid. Smaller rangeslike/29 are not supported for this feature.

Architecture

Whenenabling read replicas,you specify the number of replicas you want in the instance.Memorystore automatically distributes the primary and read replicanodes across available zones in a region.

Note: Memorystore does not charge for cross-zone data transfer fromMemorystore, but you are charged for cross-zone data transfer from theapplication VM to Memorystore instances. Seepricingfor details.

Each instance has a primary endpoint and read endpoint. The primary endpointalways directs traffic to the primary node, while the read endpointautomatically load-balances read queries across available replicas.

The Memorystore for Redis health monitoring service monitors theinstance and is responsible for detecting any failure of the primary node,and elects a replica as the new primary and initiates an automatic failover tothe new primary.

Failovers for instances with read replicas

When a primary fails, the Memorystore health monitoring serviceinitiates the failover and the new primary is made available for both reads andwrites. The failover usually completes in under 30 seconds.

When a failover happens the primary endpoint automatically redirects trafficto the new primary, however, all client connections to the primary endpoint aredisconnected during a failover. Applications with connection retry logic willautomatically reconnect once the new primary is online. Some of the clientconnections to the read endpoint also undergo disconnects from the read replicathat is promoted to primary during failover. Connections to remaining replicascontinue to be served during a failover. On retry the connections are redirectedto the new replicas.

When a failover occurs, due to the asynchronous nature of replication, replicascan have different replication lag. However, the failover process does a besteffort to fail over to the replica with the least lag. This helps minimize theamount of data loss and reduction to read throughput during a failover. Thenewly promoted primary can be in the same zone or a different zone as the formerprimary. A replica is selected to be the new primary if it is in the same zoneas the former primary and has the least lag. If not, a replica from a differentzone can become the new primary.

Since the replication is asynchronous, there is always the possibility ofreading stale data during a failover. Also, while the new primary is beingpromoted, some writes to the instance may be lost. Applications shouldbe able to deal with this behavior.

Redis makes a best effort to avoid the other replicas requiring a full syncduring failover, but it can happen in rare scenarios. A full sync can take a fewminutes to an hour depending on the write rate and the size of the data set thatis being replicated. During this time replicas undergoing a full syncare unavailable for reads. Once the synchronization is complete the replicas canbe accessed for reads.

Failure modes for read replicas

Instances with read replicas can run into various failures and unhealthyconditions that impact the application. The behavior varies depending on whetherthe instance has one replica, or two or more replicas. The section outlines somecommon failure modes and outlines the behavior of the instance during theseconditions.

Replica is unavailable

• When a replica fails for any reason, the replica is marked as unavailable andall connections to the replica are terminated after a certain timeout. Oncethe replica is recovered, new connections are routed to the restored replica.The time to recover a replica varies depending on the failure mode.

• If a zone failure occurs, then Memorystore for Redis doesn't recover thereplica until the zone becomes available.

Zone failure

• If the zone where the primary is located fails, the primary automaticallyfails over to a replica in another zone. If the instance only has one replica,then the read endpoint is unavailable for the duration of the zone outage. Ifthe instance has more than one replica, replicas outside of the affected zoneare available for reads

• If the zone where one or more of the replicas is located fails, those replicasare unavailable for the duration of the zone failure. If there is a two zonefailure and there are two or more replicas, the replica with the least lag inthe remaining zones is promoted to the primary. Any remaining replicas in theunaffected zones are available for reads.

Network partition

A network partition is a scenario where nodes stay running but cannot reach allclients, zones, or peer nodes. Memorystore uses a quorum basedsystem to prevent isolated nodes from serving writes. In case of a networkpartition, any primary in a minority partition self demotes. The majoritypartition (if one exists) elects a new primary if it does not already have one.Isolated replicas continue to serve reads. However, they may become stale ifthey cannot sync from the primary.

To determine if the link is broken, monitor themaster_link_down_since_secondsandoffset_diff metrics to identify isolated nodes.

Full sync

When a replica falls too far behind the primary it triggers a full sync whichcopies an entire snapshot from the primary to a replica. This operation can takeanywhere from minutes up to an hour in the worst case. A full sync does notcause an instance failure, however during this time the replica undergoing thefull sync is not available for reads and the primary experiences higher CPU andmemory utilization.

Primary endpoint returns READONLY

Your writes to the primary endpoint of a Memorystore for Redis instance withread replicas may unexpectedly receive-READONLY You can't write against a readonly replica. errors. We recommend closing and recreating the connections tothe instance. In most cases, restarting the client application can mitigate theissue. If these options are not feasible or the behavior persists, pleasecontact the Google Cloud Support team.

Scaling reads with the read endpoint

Read replicas allow applications to scale their reads by reading from thereplicas. Applications can connect to read replicas via the read endpoint.

Read endpoint

The read endpoint is an IP address that your application connects to. Itevenly load balances connections across the replicas in the instance.Connections to the read replica can send read queries, but not write queries.Every Standard Tier instance that has read replicas enabled has a read endpoint.For instructions on viewing your instance's read endpoint, seeView read replica information for your instance.

Behavior of the read endpoint

• The read endpoint automatically distributes connections across all availablereplicas. Connections are not directed to the primary.
• A replica is considered available as long as it is able to serve clienttraffic. This does not include times when a replica is undergoing afull-sync with its primary.
• A replica with a high replication lag continues to serve traffic. Applicationswith high write volume can read stale data from a replica serving high writes.
• In the event a replica node becomes the primary, connections to that node areterminated and new connections are redirected to a new replica node.
• Individual connections to the read endpoint target the same replica for thelifetime of the connection. Different connections from the same client hostare not guaranteed to target the same replica node.

Read consistency

Read replicas are maintained using native OSS Redis asynchronous replication.Due to the nature of asynchronous replication it is possible that the replicalags behind the primary. Applications with constant writes that are alsoreading from the replica should be able to tolerate inconsistent reads.

If the application requires "read your write" consistency, we recommended usingthe primary endpoint for both writes and reads. Using the primary endpointensures that the reads are always directed to the primary. Even in this case itis possible to have stale reads after a failover.

Setting TTLs on the keys on the primary ensures expired keys are not read fromeither the primary or the replica. This is because Redis ensures an expired keycannot be read from the replica.

Behavior of enabling read replicas on an existing instance

• Enabling read replicas is an exclusiveoperation, meaning you cannot perform otherupdate operationinstance modifications as a part of the same operation that enables readreplicas.

• Enabling read replicas on an existing Redis instance requires you to allocatea valid secondary IP address range for node placement. This must be aClassless Inter-Domain Routing (CIDR) range of size/28, regardless of thesize of the existing IP address range that's allocated toMemorystore for Redis.

  • You must provide the additional IP range when enabling read replicas forthe Redis instance. You can either choose a specific range, or letMemorystore automatically select one for you.

• The read/write IP address for your instance does not change when enabling readreplicas. Theread endpointIP address is located in the original range allocated for yourMemorystore instance, not the additional range you provide whenenabling read replicas.

• To find the new read endpoint,view read replica information for your instance after the operationto enable read replicas completes.

Scaling an instance

You can scale the number of read replicas for your instance, and you can alsothe modify the node size:

• For instructions on adding and removing nodes, seeAdd or remove replica nodes from your Redis instance.

• For instructions on scaling the size of Redis nodes, seeScale the size of Redis nodes.

We recommend that you scale your instance during a period of low read and writetraffic to minimize impact to the application.

Adding a new replica results in additional load on the primary while the replicaperforms a full sync. When adding nodes, existing connections are not affectedor transferred. Once the new replica is available it starts to receiveconnections from the endpoint and serves reads. Removing a replica closes anyactive connections routed to that replica. The client application should beconfigured to automatically reconnect to the read endpoint to reestablishconnections to the remaining replicas.

Best practices

Memory management

Redis does not allow client writes to exceed themaxmemory limitof the instance. However, overhead such as fragmentation, replication buffers,and expensive commands like EVAL can increase memory utilization past thislimit. In these cases Memorystore fails writes until the memorypressure is reduced. SeeMemory management best practicesfor more details.

If Memorystore is undergoing a BGSAVE operation due to an exportor full sync replication and an OOM condition occurs, the child process iskilled. In this case the BGSAVE operation fails and the Redis node serverremains available.

To guarantee replication and snapshot creation under all circumstances we advisekeeping memory utilization less than 50% during important operations like export,scaling, etc. You can manually triggerexportorfailoverto see the performance impact of these operations.

CPU management

Memorystoreprovides metricson CPU usage and the connection count for each node. We recommend that youallocate enough overhead so that the loss of a single availability zone can betolerated. The ideal target may vary based on the number of replicas and usagepatterns but a good starting point is to keep replica CPU usage below 50%.

Individual nodes may experience high usage if client usage patterns areunbalanced, or if failover operations result in an unbalanced connectiondistribution. In this case, we recommend that you periodically close yourconnections to allow Memorystore to automatically rebalanceconnections. Memorystore does not rebalance open connections.

Connection balance management

Anytime a node's connections are closed clients have to reconnect, typically byenabling auto reconnect on the client library of your choice. When the node isreintroduced existing connections are not rerouted, however new connections arerouted to the new node. Clients can periodically kill connections to ensure theyare balanced across available nodes.

Replication lag management

It is possible for replicas to lag behind especially when the write rate is veryhigh. In such scenarios the replica continues to be available for reads. Inthis circumstance, reads from the replica can be stale and the applicationshould be able to handle this, or the high write rate should be addressed.

What's next

• Learn how tomanage read replicas.
• Learn aboutexporting data from a Redisinstance.
• Learn abouthigh availability for Memorystore for Redis.