Regional Persistent Disk and Hyperdisk Balanced High Availability arestorage options that let you implement high availability (HA) services inCompute Engine. Regional Persistent Disk and Hyperdisk Balanced High Availability synchronouslyreplicate data between two zones in the same region and ensure HA for diskdata for up to one zonal failure.

Building high availability services with regional disks

This section explains how you can build HA services withregional Persistent Disk orHyperdisk Balanced High Availability disks.

Design considerations

Before you start designing a HA service, understand the characteristicsof the application, the file system, and the operating system. Thesecharacteristics are the basis for the design and can rule out variousapproaches. For example, if an application does not support application-levelreplication, some corresponding design options are not applicable.

Similarly, if the application, the file system, or the operating system are notcrash tolerant, then usingregional Persistent Disk orHyperdisk Balanced High Availability disks, or even zonal disk snapshots, might not bean option.Crash tolerance is defined as the ability to recover from anabrupt termination without losing or corrupting datathat was already committed to a disk prior to the crash.

Consider the following when designing for high availability:

• The effect on the application of using Hyperdisk Balanced High Availability, regional Persistent Disk, or other solutions.
• Disk write performance.
• The service recovery time objective - how quickly your service must recover from a zonal outage and the SLA requirements.
• The cost to build a resilient and reliable service architecture.
• For more information about region-specific considerations, seeGeography and regions.

In terms of cost, use the following options for synchronous andasynchronous application replication:

• Use two instances of the database and VM. In this case the followingitems determine the total cost:

  • VM instance costs
  • Persistent Disk orHyperdisk costs
  • Costs of maintaining application replication

• Use a single VM with synchronously replicated disks. To achieve highavailability with aregionalPersistent Disk or Hyperdisk Balanced High Availability disk, use the same VM instanceand disk components as the previous option, but also include a synchronouslyreplicated disk. Regional Persistent Disks and Hyperdisk Balanced High Availability disks are double thecost per byte compared to zonal disks because they are replicated in twozones.

  However, using synchronously replicated disks might reduceyour maintenance cost because the data is automatically written to tworeplicas without the requirement of maintaining application replication.

• Don't start the secondary VM until failover is required. You can reducehost costs even more by starting the secondary VM only on demand duringfailover rather than maintaining the VM as an active standby VM.

Compare cost, performance, and resiliency

The following table highlights the trade-offs in cost, performance, andresiliency for the different service architectures.

HA service architecture	Zonal disk snapshots	Application level synchronous	Application level asynchronous	Regional disks
Protects against application, VM, zone failure*
Mitigation against application corruption (Example: application crash-intolerance)		†	†
Cost	$	$$ Two instances of the database or VM running, application replication setup and maintenance, cross zone networking.	$$ Two instances of the database or VM running, application replication setup and maintenance, cross zone networking.	$1.5x - $$ The costs are the same as application replication if you use an active standby VM. You can achieve a lower cost by spinning up a backup VM on demand during failover. There is no charge for cross zone networking between disk replicas.
Application performance	No effect	Trade-off on application performance with synchronous replication	No effect	No effect for most applications
Suited for application with low RPO requirement (Very low tolerance for data loss)	Data loss depending on when the snapshot was taken	No data loss‡	Data loss because replication is asynchronous	No data loss
Storage recovery time from disaster#	O(minutes)	O(seconds)	O(seconds)	O(seconds)—to force attach the disk to a standby VM instance

^* Using regional disks or snapshots is notsufficient to protect from and mitigate against failures and corruptions.Your application, file system, and possibly other software components must becrash consistent or use some sort ofquiescing.

^† The replication of some applications providesmitigation against some application corruptions. For example, MySQL primaryapplication corruption doesn't cause its replica VM instances to becomecorrupted as well. Review your application's documentation for details.

^‡ Data loss means unrecoverable loss of datacommitted to persistent storage. Any non-committed data is still lost.

^# Failover performance doesn't include file systemcheck and application recovery and load after failover.

Building HA database services using regional disks

This section covers high level concepts for building HA solutionsfor stateful database services (MySQL, Postgres, etc.) usingCompute Engine withRegional Persistent Disks and Hyperdisk Balanced High Availability disks.

If there are broad outages in Google Cloud, for example, if a whole regionbecomes unavailable, your application might become unavailable. Depending onyour needs, considercross-regional replication techniquesorAsynchronous Replication for even higher availability.

Database HA configurations typically have at least two VM instances. Preferablythese VM instances are part of one or more managed instance groups:

• A primary VM instance in the primary zone
• A standby VM instance in a secondary zone

A primary VM instance has at least two disks: a boot disk, and aregional disk. The regional disk contains database data and any other mutabledata that should be preserved to another zone in case of an outage.

A standby VM instance requires a separate boot disk to be able to recover fromconfiguration-related outages, which could result from an operating systemupgrade, for example. Also, you can't force attach a boot disk to anotherVM during a failover.

The primary and standby VM instances are configured to use a load balancer withthe traffic directed to the primary VM based on health check signals. Thedisaster recovery scenario for dataoutlines other failover configurations, which might be more appropriate foryour scenario.

Challenges with database replication

The following table lists some common challenges with setting up and managingapplication synchronous or semi-synchronous replication (like MySQL) and howthey compare to synchronous disk replication withRegional Persistent Diskand Hyperdisk Balanced High Availability disks.

Challenges	Application synchronous or semi-synchronous replication	Synchronous disk replication
Maintaining stable replication between primary and failover replica.	There are a number of things that could go wrong and cause a VM instance to fall out of HA mode: Misconfiguration of replication parameters such as SSL certificate mismatch or missing ACL on the primary side. High load on primary VM instance causing failover replica to be unable to keep up. Bugs causing replication issues such as application issues, OS misconfiguration, or Docker failure. Infrastructure failures like CPU contention, frozen VM, or intermediate network interruption.	Storage failures are handled by Regional Persistent Disk and Hyperdisk Balanced High Availability disks. This happens transparently to the application except for a possible fluctuation in the disk's performance. There must be user-defined health checks to reveal any application or VM issues and trigger failover.
The end-to-end failover time is longer than expected.	The time taken for the failover operation doesn't have an upper bound. Waiting for all transactions to be replayed (step 2 above) could take an arbitrarily long time, depending on the schema and the load on the database.	Regional Persistent Disk and Hyperdisk Balanced High Availability disks provide synchronous replication, so the failover time is bounded by the sum of the following latencies: Creation of a secondary VM, unless there is already a standby VM instance available. Force attaching a regional disk. Application initialization.
Split-brain	To avoidsplit-brain, both approaches require provisions to ensure that there is only one primary at a time.

Sequence of read and write operations to disks

In determining the read and write sequences, or the order in which data is readfrom and written to disk, the majority of the work is done by the disk driver inyour VM. As a user, you don't have to deal with the replication semantics, andcan interact with the file system as usual. The underlying driver handles thesequence for reading and writing.

By default, a Compute Engine VM withRegional Persistent Disk orHyperdisk Balanced High Availability operates in full replication mode, where requeststo read or write from disk are sent to both replicas.

In full replication mode, the following occurs:

• When writing, a write request tries to write to both replicas andacknowledges when both writes succeed.
• When reading, the VM sends a read request to both replicas, and returns theresults from the one that succeeds. If the read request times out, anotherread request is sent.

If a replica falls behind or fails to acknowledge that the read or writerequests completed, then thereplica stateis updated.

Health checks

Thehealth checks used by theload balancer are implemented by the health check agent. The health check agentserves two purposes:

1. The health check agent resides within the primary and secondary VMs tomonitor the VM instances and communicate with the load balancer to directtraffic. This works best when configured withinstance groups.
2. The health check agent syncs with the application-specific regional controlplane and makes failover decisions based on control plane behavior. Thecontrol plane must be in a zone that differs from the VM instancewhose health it is monitoring.

The health check agent itself must be fault tolerant. For example, notice that,in the image that follows, the control plane is separated from the primary VMinstance, which resides in zoneus-central1-a, while the standby VM residesin zoneus-central1-f.

What's next

• Learn how tocreate and manage regional disks.
• Learn aboutAsynchronous Replication.
• Learn how toconfigure a SQL Server failover cluster instance for disks in multi-writer mode.
• Learn how tobuild scalable and resilient web applications on Google Cloud.
• Review thedisaster recovery planning guide.