Astateful managed instance group(stateful MIG) preserves the unique state of each virtual machine (VM)instance—including VM name, attached persistent disks, IP addresses,and/or metadata—on machine restart, recreation, auto-healing, or update.

This page describes how stateful MIGs work. SeeConfiguring stateful MIGsto learn how to set up a stateful MIG.

Overview of how stateful MIGs work

A MIG is considered stateful if you have created astateful configuration.

Youcreate a stateful configurationby setting a non-empty stateful policy and/or one or more non-emptyper-instance configurations:

• Astateful policydefines items that you want to preserve for all instances in your MIG.
• Aper-instance configurationdefines items to preserve for a specific VM instance.

The configuration is effective after you or the MIGapplies it:

• A MIG automatically applies your stateful policy configuration to new andexisting instances.
• When creating or updating per-instance configurations, you can choose whether toapply the new configuration manually or have it applied automatically.

After the stateful configuration (stateful policy and/or per-instance configurations)is applied, you canverifyit by inspecting thepreserved stateof each managed instance.

Subsequent changes to your MIG's stateful configuration or size (for example,decreasing the MIG's size, or deleting or abandoning instances from the MIG) canaffect the preserved states ofthe instances.

How stateful configuration is applied to managed instances

Your stateful configuration is effective after you or the MIG applies it.Applying stateful configuration to a MIG's instances depends on theconfiguration:

• Stateful policy: The MIG automatically applies your stateful policyconfiguration to new and existing instances.
• Per-instance configurations: When creating or updating per-instance configurations,you can choose whether to apply the new configuration manually or have itapplied automatically.

How stateful policy updates are applied to instances

When you create or update astateful policy,for example add or remove a stateful disk, the MIG applies your stateful policyconfiguration to all managed instances in the group automatically andasynchronously. A MIG also automatically applies your stateful policyconfiguration to new instances during their creation, for example, when a MIG'ssize is increased or when you create instances in the MIG manually.

After the configuration is applied, you can see the effect of the update in eachmanaged instance'spreserved state from policy.

Updates to a stateful policy do not disrupt running VMs.

When you update a stateful policy to add a stateful disk, the MIG updates eachVMresource,changing the value of the disk'sautoDelete flag(instances.disks[].autoDelete):

• The MIG setsautoDelete toFALSE for disks that you configure asstateful. This prevents deletion of that disk on instance recreation byautohealing, updating, or manual recreation.
• The MIG setsautoDelete to match your instance templateconfiguration (instanceTemplates.disks[].autoDelete)for all disks that are to be stateless.

Changing the value of theautoDelete flag does not disrupt a running VM.

How per-instance configuration updates are applied to instances

When you create or update a per-instance configuration, you can choosewhether to apply the new configuration manually or automatically. For moreinformation, seeApplying stateful configuration from per-instance configurations.

The following table shows thedisruption levels that are required to apply different per-instance configuration updates to a VM:

When applying an updated per-instance configuration to the corresponding VM,the MIG performs the following actions depending on which stateful items areupdated:

• Adds (or removes) disks, IP addresses, or metadata to thepreserved state from configuration in the corresponding managed instance.
• Attaches (or detaches) disks that are not defined by the instance templateto the VM.
• Sets (or removes) metadata key-value pairs that are specific for the VM.
• Assigns (or removes) specified IP addresses to the VM instance.

After a per-instance configuration is applied to a corresponding managed VM, youcan see the effect of the update in the instance'spreserved state from configuration.

Preserved state of a managed instance

When it is applied, the MIG translates your instance template andstateful configuration into a "preserved state" foreachmanaged instance.

You canview the preserved stateby inspecting a managed instance.

The MIG maintains these preserved states automatically, and the MIGautomatically and asynchronously applies this state to eachcorresponding actual VM instance in the MIG.

The preserved state describes which individual items (persistentdisks, IP addresses, metadata) are stateful for a given instance:

The preserved state generated based on a stateful policy is storedseparately from the preserved state generated based on a per-instanceconfiguration. The MIG combines both of them when recreating a VM, withthe preserved state from a per-instance configuration taking priority.

Preserved state according to stateful policy

A stateful policy specifies items, present in all instances anddefined in the MIG's instance template, to preserve individually foreach VM instance in a MIG.

When applied, the MIG translates the stateful policy intoinstance-specific preserved states(managedInstances[].preservedStateFromPolicy). The MIG maintainsthese preserved states automatically.

The following example shows a MIG with two VM instances that use a statefuldisk defined in a stateful policy that applies to every instance. Thereare no per-instance configurations in this example.

The preceding figure shows a MIG with two instances:

• The instance template defines a boot disk with device name,boot-disk,and a disk with device name,data-disk, for all instancesin the MIG.
• The stateful policy declaresdata-disk as stateful. The boot diskremains stateless. Note that the disk with device name,data-disk,must be and is defined by the instance template.
• After the configuration is applied, the MIG translates the stateful policy intoinstance-specific preserved states for each managed instance. The preservedstates instruct the MIG to preserve the diskdata-disk-1 for VMinstancenode-1 and the diskdata-disk-2 for the instancenode-2,because both of these disks have device namedata-disk configuredin the stateful policy.
• This example has no per-instance configurations.

Preserved state according to per-instance configuration

A per-instance configuration specifies items that must be preserved for a particularVM. These items don't have to be defined in the MIG's instancetemplate.

When applied, the MIG translates each per-instance configuration into a preservedstate (preservedStateFromConfig) for the corresponding instance.

The following example shows a MIG with two VM instances for which statefulmetadata and disks are defined in per-instance configurations (PICs) for everyinstance. There is no stateful policy in this example.

In the preceding figure:

• The instance template defines a boot disk with device nameboot-diskfor all instances in the MIG. The boot disk is stateless for all VMsin the MIG.
• Per-instance configurations define the states to preserve for two instances inthe MIG:node-1 andnode-2.
  • For thenode-1 instance, the per-instance configuration defines a diskmy-legacy-1 with device namelegacy-disk and metadatanode-id:xyz273.
  • For thenode-2 instance, the per-instance configuration defines a diskmy-logs-1 with device namelogs-disk and metadatanode-id:pqr851.
• After the configuration is applied, the MIG automatically translates theper-instance configurations into preserved states for each managed instance. Thepreserved states instruct the MIG to attach and preserve the following:
  • Persistent diskmy-legacy-1 and metadatanode-id:xyz273 for VMnode-1
  • Persistent diskmy-logs-1 and metadatanode-id:pqr851 for VMnode-2
• This example has no stateful policy.

Note that the disks and metadata in the preserved state from per-instanceconfigurations are not defined by the instance template in this example; instead, theyare defined by the per-instance configurations only. This is because the configurationthat you specify in a per-instance configuration is specific for a particular VM,which means it does not have to be present in the instance template.

Per-instance configurations have priority over stateful policy and instance template

You can configure both a stateful policy and one or more per-instance configurationsin one MIG. For example, in a stateful policy, you can define stateful disksthat are present in all instances, and in per-instance configurations, you can defineinstance-specific metadata.

A managed instance's per-instance configuration takes priority over conflictingconfiguration in the instance template or in a stateful policy.

If you apply a per-instance configuration to add a disk or a network interfacethat is already defined in a stateful policy, the MIG stores the statefulconfiguration for that disk or network interface in the managed instance'spreserved state from per-instance configuration (preservedStateFromConfig) andremoves the conflicting entries from itspreserved state from policy(preservedStateFromPolicy). The MIG mustrefresh the VM if the new preserved state is different from the previous one. Therefresh could result in metadata change, external IP address change, or a diskswap to detach the disk from the last preserved state configuration and attachthe disk specified in the new preserved state configuration.

In the following example, the per-instance configuration for VM instancenode-1redefines:

• The preserved state for the disk with device namelogs-disk,originally defined in the stateful policy
• The value for metadata keylogmonth, originally defined in theinstance template.

In the preceding figure:

• The instance template defines:
  • Three disks for all instances in the MIG, with device namesboot-disk,data-disk,logs-disk.
  • Metadata common to all instances:logmonth:jan.
• The stateful policy declares that disks with device namesdata-diskandlogs-disk are stateful; the boot disk remains stateless.
• A per-instance configuration for the instancenode-1 redefines:
  • Stateful configuration for a disk with device namelogs-disk: This instructs the MIG to attach the diskpd-logs-feb tonode-1 under thelogs-disk device name.
  • Metadata, defined in the instance template, with key valuelogmonth:jan: This instructs the MIG to set valuelogmonth:feb tonode-1.
• After youapply,the configuration, the MIG automatically translates thestateful policy and per-instance configuration into an instance-specific preservedstate, stored in the managed instance.
  • The preserved state from policy instructs the MIG to preserve thediskdata-disk-1 for the VMnode-1. Note that the preservedstate from policy does not include stateful configuration for thedisk with device namelogs-disk because this configuration isoverridden by the configuration forlogs-disk in the per-instanceconfiguration.
  • The preserved state from configuration instructs the MIG to attach andpreserve persistent disklogs-disk and to set and preservemetadatalogmonth:feb for VM instancenode-1. Note thatthe preserved state from configuration overrides the configuration forlogs-disk from the stateful policy and overrides metadatalogmonth:jan from the instance template.

How removing a resource from a stateful policy affects preserved state

If you remove a resource configuration from your stateful policy, the MIGautomatically removes the correspondingpreservedStateFromPolicy forall managed instances. The compute resources remain attached to the instances,but they are no longer stateful.

In the following example, removing a disk from the stateful policy leads toremoval of that disk from preserved states from policy in all managed VMs. Thosedisks remain attached to their VMs, but they are no longer stateful and might bedeleted and recreated on next VM recreation.

If the same item, for example, a stateful persistent disk, is present both inthe stateful policy and a per-instance configuration, and you remove its statefulconfiguration from the stateful policy only, the MIG doesn't remove it from theper-instance configuration. For the corresponding VM, the configured resourceremains stateful.

In the following example, removing the disk from the stateful policy does notlead to removal of the disk from the per instance configuration. The diskremains stateful because it is still a part of preserved state from configuration.

How removing items from per-instance configurations affects preserved state

If you remove the stateful configuration from a per-instance configuration, and applythe change, the MIG automatically removes the stateful configuration from thepreserved state from configuration (preservedStateFromConfig) in the correspondingmanaged instance. The compute resources that are no longer part of any preservedstate become stateless.

How removing stateful disks configuration from per-instance configurations affects preserved state

If you remove a stateful disk from a per-instance configuration and apply the changeto the associated VM instance, the MIG does the following:

• The disk configuration is removed from the instance'spreserved state from configuration.
• If a disk with the same device name is defined in the instance templatebut not configured in a stateful policy, then the disk stays attached tothe given VM. However, the disk becomes stateless for the givenVM and it might be recreated according to the instance templateconfiguration on the next VM recreation, autohealing, or update event.
• If a disk with the same device name is not defined in the instancetemplate, then it is automatically detached from the VM immediatelyupon application of the updated per-instance configuration to the associatedVM, regardless of its auto-delete configuration.
• If a disk with the same device name is configured in a stateful policy,then its stateful policy configuration is translated into thepreserved state from policyfor the given managed instance, and the disk remains stateful.

In the following example, removing a blue and a green disk fromnode-1'sper-instance configuration leads to removal of both disks from thenode-1 managedinstance's preserved state from configuration.

• The blue disk remains attached to thenode-1 VM instance, but it is nowstateless and can be recreated on the next VM recreation according tothe instance template configuration.
• The green disk is detached from thenode-1 VM instance because theinstance template does not define a disk with the same device name.

How removing stateful metadata from per-instance configurations affects preserved state

Removing stateful metadata from a per-instance configuration and applying the changecauses the MIG to immediately remove that stateful metadata from thecorresponding managed instance's preserved state:

• If you defined metadata with the same key in the instance template,the MIG immediately applies the value from the instance template to theinstance.
• If the metadata with the same key is not defined in the instancetemplate, the MIG immediately removes the key value from the instance.

In the following example, removingmode:dev andid:xyz273 metadatafromnode-1's per-instance configuration leads to automatic removal of both key-valuepairs from thenode-1 managed instance's preserved state from configuration.

• mode:dev is replaced by the instance template'smode:testin the VM.
• id:xyz273 is removed from the VM immediately because theinstance template does not have metadata with the same keyid toreplace it with.

How removing stateful IP configuration from per-instance configurations affects preserved state

Removing the internal IP configuration from per-instance configuration makes theIP address for this VM stateless. No automated changes are performed on this VM,but the IP address can change after the VM is recreated, updated, or autohealed.

Fallback to stateful policy

If you remove the stateful configuration of a resource from a per-instanceconfiguration, and you configured the same resource in the stateful policy,then the resource remains stateful according to the stateful policy.

The MIG automatically removes the item's stateful configuration from thepreservedStateFromConfigand adds it to thepreservedStateFromPolicyfor the corresponding managed instance.

In the following example, removing a disk fromnode-1's per-instance configurationdoes not lead to removal of the disk from the stateful policy. The diskremains stateful according to the stateful policy:

• The MIG automatically removes the disk from thepreserveStateFromConfigfor thenode-1 managed instance because the disk is no longer partof its per-instance configuration.
• The MIG automatically adds the disk to thepreserveStateFromPolicy for thenode-1 managed instance because the stateful policy configuration is stillin place and is no longer in conflict with thenode-1 per-instance configuration.

Feedback

We want to learn about your use cases, challenges, and feedback about statefulMIGs. Please share your feedback with our team atmig-discuss@google.com.

What's next

• Learnhow different configurations and MIG actions affect preserved state.
• For information about how to support stateful workloads by preservinginstance names, persistent disks, and metadata in managed instances, seeConfiguring stateful MIGs.
• Learn more aboutwhen to use stateful MIGs.
• Learn how toMigrate an existing workload to a stateful MIG.
• Learn more aboutMIGs.
• Work withmanaged instances.