This page gives an overview ofmanual failover for Memorystore for Redis. Tolearn how to perform a failover, seeInitiating a manual failover.

What is a manual failover?

A standard tier Memorystore for Redis instance uses a replica node to back upthe primary node. A normal failover occurs when the primary node becomesunhealthy, causing the replica to be designated as the new primary. A manualfailover differs from a normal failover because you initiate it yourself. Formore information on how Memorystore for Redis replication works, seeHigh availability.

Why initiate a manual failover?

Initiating a manual failover allows you to test how your application responds toa failover. This knowledge can ensure a smoother failover process if anunexpected failover occurs later on.

Optional data protection mode

The two available data protection modes are:

• limited-data-loss mode (default).
• force-data-loss mode.

To set the data protection mode, use one of the following commands:

gcloud redis instances failover INSTANCE_NAME --data-protection-mode=limited-data-loss

or

gcloud redis instances failover INSTANCE_NAME --data-protection-mode=force-data-loss

How data protection modes work

Thelimited-data-loss mode minimizes data loss by verifying that thedifference in data between the primary and replica is below 30 MB beforeinitiating the failover. The offset on the primary is incremented for each byteof data that must be synchronized to its replicas. In thelimited-data-lossmode, the failover will abort if the greatest offset delta between the primaryand each replica is 30MB or greater. If you can tolerate more data loss and wantto aggressively execute the failover, try setting the data protection mode toforce-data-loss.

Theforce-data-loss mode employs a chain of failover strategies toaggressively execute the failover. It does not check the offset delta betweenthe primary and replicas before initiating the failover; you can potentiallylose more than 30MB of data changes.

Bytes pending replication metric

Thebytes pending replication metric tells you how many remaining bytes thereplica needs to copy before the primary is fully backed up. You may observean increase in bytes pending as the primary replicates to the replica duringa failover. If the failover is triggered by hardware error, you may observe empty in bytes pending replication as the offset value could not be obtained until the new replica repaired from host error.

You can access thismetric in the Google Cloud console on the instance details page. To view theinstance details page, click the instance id in your project'sinstances listpage.

Alternatively, access the Metrics Explorer for yourproject, and search for theredis.googlapis.com/replication/offset_diffmetric.

When to run a manual failover

Manual failovers using the defaultlimited-data-loss protection mode onlysucceed if thebytes pending replication metric is less than 30MB. If youwant to run a manual failover withbytes pending replication higher than30MB, use theforce-data-loss protection mode.

If you are trying to preserve as much data as possible, temporarily stop yourapplication from writing to the Redis instance, and wait to run your manualfailover until thebytes pending replication metric is as low as you deemacceptable.

Potential issues blocking a manual failover

• Running a manual failover on a Basic Tier instance does not work because BasicTier instances do not have replicas to which the primary can failover.

• If your Redis instance is unhealthy, then a limited-data-loss manual failoveroperation fails because it is blocked for data-loss minimization.

• If you are running a Lua script that is executing indefinitely, then you mustuseforce-data-loss to initiate a failover. In this situation alimited-data-loss failover operation will not complete successfully.

• If your instance has incomplete operations pending, such as scaling orupdating, the manual failover operation is blocked. You must wait until yourinstance is in theREADY state to run a manual failover.

Client application connection

When your primary node fails over to the replica, existing connections toMemorystore for Redis are dropped. However, on reconnect, your application isautomatically redirected to the new primary node using the same connection stringor IP address.

Verifying a manual failover

You can verify the success of a manual failover operation with theGoogle Cloud console orgcloud.

Google Cloud console verification

Before you start a manual failover, go to the Memorystore for Redisinstanceslist page,and click the name of your instance.

Then, in theConfiguration tab, next toPrimary Location, view which zoneyour primary node is in. Make a note of the zone. Check this page again when youcomplete your manual failover to confirm that the primary node switched zones.

Cloud Monitoring verification

To view the metrics for a monitored resource by using theMetrics Explorer, do the following:

1. In the Google Cloud console, go to theleaderboard Metrics explorer page:

   Go toMetrics explorer

   If you use the search bar to find this page, then select the result whose subheading isMonitoring.

2. In the toolbar of the Google Cloud console, select your Google Cloud project. ForApp Hub configurations, select the App Hub host project or the app-enabled folder's management project.
3. In theMetric element, expand theSelect a metric menu, enterNode role in the filter bar, and then use the submenus to select a specific resource type and metric:
   1. In theActive resources menu, selectCloud Memorystore Redis.
   2. In theActive metric categories menu, selectreplication.
   3. In theActive metrics menu, selectNode role.
   4. ClickApply.

4. To add filters, which remove time series from the query results, use theFilter element.

5. To combine time series, use the menus on theAggregation element. For example, to display the CPU utilization for your VMs, based on their zone, set the first menu toMean and the second menu tozone.

   All time series are displayed when the first menu of theAggregation element is set toUnaggregated. The default settings for theAggregation element are determined by the metric type you selected.

6. For quota and other metrics that report one sample per day, do the following:
   1. In theDisplay pane, set theWidget type toStacked bar chart.
   2. Set the time period to at least one week.

The Cloud Monitoring chart represents the primary and replica nodes with twolines. When a node's line has a value of zero on the chart, it is the replicanode. When a node's line has a value of one on the chart, it is the primary node.The chart represents a failover by showing how the lines switch from one tozero, and zero to one, respectively.

gcloud verification

Before you initiate a manual failover, use the following command to check whichzone your primary node is in:

gcloud redis instances describe [INSTANCE_ID] --region=[REGION]

Your primary node is in the zone labeledcurrentLocationId. Make a note of thezone.

After you complete a manual failover, you can confirm that your primary nodeswitched to a new zone by running thegcloud redis instances describe commandagain and checking that thecurrentLocationId changed zones.

Additionally, thelocationId label tells you the zone in which you originallyprovisioned your primary node. ThealternativeLocationId label tells you thezone in which system originally provisioned your replica node. Each time afailover occurs the primary and replica switch between these two zones. However,the zones associated withlocationId andalternativeLocationId do notchange.