This document provides recommendations to help you maintain continuous accessto Google Cloud resources. Business continuity aims to ensure that yourorganization can maintain essential operations, even during disruptions likeoutages or disasters. This goal includes continued employee access whencritical services and infrastructure are unavailable.

This document is intended for security or reliability professionals who areresponsible for Identity and Access Management (IAM) and for the maintenance of secureaccess to Google Cloud. This document assumes that you're already familiarwith Cloud Identity, Google Workspace, and IAM management.

To help you prepare for outages and ensure continuous access, this documentoutlines the following recommended steps that you can implement. You can chooseto do all or some of these steps, but we recommend that you implement them inthe following order.

1. Set up emergency access: Enable last-resort access toGoogle Cloud resources.

   We recommend that you set up emergency access for all of yourGoogle Cloud organizations, regardless of your individual businesscontinuity requirements.

2. Provide authentication alternatives for critical users: If yourorganization usessingle sign-on (SSO),any disruption that affects your external identityprovider (IdP) can affect employees' ability to authenticate and useGoogle Cloud.

   To reduce the overall impact of an IdP disruption on your organization,provide an authentication alternative for business-critical users tocontinue to access Google Cloud resources.

3. Use a backup IdP: To let all users access Google Cloudresources during an IdP disruption, you can maintain a fallback IdP.

   A fallback IdP can help to further minimize the impact of a disruption,but this option might not be cost-effective for every organization.

The following sections describe these recommended steps and best practices.

Set up emergency access

The purpose of emergency access is to enable last-resort access toGoogle Cloud resources and prevent situations in which you might loseaccess entirely.

Emergency access users are characterized by the following properties:

• They're users that you create in your Cloud Identity orGoogle Workspace account.
• They have thesuper admin privilege,which provides users with sufficient access to resolve any misconfigurationthat affects your Cloud Identity, Google Workspace, orGoogle Cloud resources.
• They're not associated with a specific employee in the organization andare exempt from theJoiner, Mover, and Leaver (JML) lifecycle of regular user accounts.
• They're exempt from SSO.

The following sections describe the recommended best practices to follow whenyou manage and secure emergency access users.

Create emergency access users for every environment

For Google Cloud environments that host production workloads, emergencyaccess is critical. For Google Cloud environments that are used fortesting or staging purposes, a loss of access can still be disruptive.

To ensure continuous access to all of your Google Cloud environments,create and maintain emergency access users in Cloud Identity orGoogle Workspace for each environment.

Ensure emergency access redundancy

A single emergency access user is a single point of failure. In this scenario,a broken security key, a lost password, or an account suspension can disruptaccess to an account. To mitigate this risk, you can create more than oneemergency access user for each Cloud Identity orGoogle Workspace account.

Emergency access users are highly privileged, so don't create too many of them.For most organizations, we recommend a minimum of two and a maximum of fiveemergency access users for each Cloud Identity orGoogle Workspace account.

Use a separate organizational unit for emergency access users

Emergency access users require a special configuration and they aren't subject tothe JML lifecycle that you might follow for other user accounts.

To keep emergency access users separate from regular user accounts, use adedicated organizational unit (OU) for emergency access users. A separate OUlets you apply custom configurations to only emergency users.

Use FIDO security keys for 2-step verification

UseFast IDentity Online (FIDO)security keys for 2-step verification.

Because emergency access users are highly privileged users in yourCloud Identity or Google Workspace account, you must protectthese users by using 2-step verification.

Among the 2-step verification methods that Cloud Identity andGoogle Workspace support, we recommend that you use FIDO security keys.This method provides protection against phishing and strong security. To ensurethat all of your emergency access users use FIDO security keys for 2-stepverification, do the following:

• In the OU that contains your emergency access users,configure 2-step verification to allow only security keys as the authentication method.
• For all emergency access users, enable 2-step verification.
• For each emergency access user, enroll two or more FIDO security keys.

When you enroll multiple keys for each user, you help mitigate the risk oflosing access due to a broken security key. You also increase the likelihoodthat the user can access at least one of the keys in an emergency.

It's acceptable to use the same set of security keys for multiple emergencyaccess users. However, it's better to use different security keys for eachemergency access user.

Use physical security controls to protect credentials and security keys

When you store the credentials and security keys of emergency access users, youmust balance strong protection with availability during an emergency:

• Prevent unauthorized personnel from being able to accessemergency access user credentials. Emergency access users must use thesecredentials only in an emergency.
• Ensure that authorized personnel can access the credentialswith minimal delay during an emergency.

We recommend that you don't rely on a software-based password manager. Instead,it's better to rely on physical security controls to protect the credentials andsecurity keys of emergency access users.

When you choose which physical security controls to apply, consider thefollowing:

• Improve availability:
  • Store copies of passwords in multiple physical locations, such as inmultiple security vaults in different offices.
  • Enroll multiple security keys for each emergency access user, and storeone key in each relevant office location.
• Improve security: Store the password and security keys in differentlocations.

Avoid automation for password rotation

It might seem beneficial to automate the password rotation for emergency accessusers. However, this automation might increase the risk of a securitycompromise. Emergency access users have super admin privileges. To rotate thepassword of a super-admin user, automation tools or scripts must also havesuper-admin privileges. This requirement can cause the tools to be attractivetargets for attackers.

To ensure that you don't weaken your overall security posture, don't useautomation to rotate the passwords.

Use strong passwords

To help protect emergency access users, make sure that they use along and strong password.To enforce a minimum level of password complexity, use a dedicated OU asdescribed earlier, and implementpassword requirements.

Unless you rotate passwords manually,disable password expiration for all of the emergency access users.

Exclude an emergency access user from access policies

During an emergency, context-aware access policies might cause a situationwhere even an emergency access user can't access certain resources. To mitigatethis risk,exclude at least one emergency access user from all of the access levels in your access policies.

These exemptions help you ensure that at least one of your emergency accessusers has continuous access to resources. In the event of an emergency or amisconfigured context-aware access policy, these emergency access users canmaintain their access.

Set up alerts for emergency access user events

Any emergency access user activity outside of an emergency event likelyindicates suspicious behavior. To be notified about any events related toactivity from emergency access users,create a reporting rule in the Google Admin console. When you create a reporting rule, you can setconditions such as the following:

• Data source: User log events.

• Attributes in theCondition builder tab: Use attributes andoperators to create a filter for the OU that contains your emergency accessusers and the events.

  For example, you can set attributes and operators to create a filter that'ssimilar to the following conditional statements:

  Actor organizational unit Is /PrivilegedAND(Event Is Successful login OR Event Is Failed login OR Event Is Accountpassword change)

• Threshold: Every 1 hour when count > 0

• Action: Send email notifications

• Email recipients: Select a group that contains the relevant membersof your security team

Provide authentication alternatives for critical users

If your organization uses SSOto let employees authenticate to Google services, then the availability of yourthird-party IdP becomes critical. Any disruption to your IdPcan prevent employees from accessing essential tools and resources.

Although emergency accesshelps you ensure continuous administrative access, it doesn't address the needsof employees during an IdP outage.

To reduce the potential effect of an IdP interruption, you can configure yourCloud Identity or Google Workspace account to use anauthentication fallback for critical users. You can use the following fallbackplan:

• During normal operations, you let users authenticate by using SSO.
• During an IdP outage, you selectively disable SSO for these criticalusers and let them authenticate by using Google sign-in credentials, whichyou provision in advance.

Note: To selectively disable SSO for privileged users, a user must havesuper admin privileges. This action might alsorequire multi-party approval.

The following sections describe the recommended best practices when you letcritical users authenticate during external IdP outages.

Focus on privileged users

In order for critical users to authenticate during an IdP outage, the users musthave valid Google sign-in credentials like the following:

• A password with a security key for second-factor authentication.
• A passkey.

When you provision Google sign-in credentials for users who normally use SSO,you might increase the operational overhead and user friction in the followingways:

• You might not be able to synchronize user passwords automatically, dependingon your IdP. Therefore, you might have to ask users to set a passwordmanually.
• You might need to request that users register a passkey or enroll in2-step verification. This step isn't usually required for SSO users.

To balance the benefits of uninterrupted access to Google services with theextra overhead, focus on privileged and business-critical users. These users aremore likely to benefit significantly from uninterrupted access, and they mightbe only a fraction of your overall user base.

Use the opportunity to enable post-SSO verification

When you provision alternative authentication for privileged users, anunintended result might be additional overhead. To help offset this overhead,you can also enable post-SSO verification for these users.

By default, when you set up SSO for your users, they aren't required to perform2-step verification. Although this practice is convenient, if the IdP iscompromised, any user who doesn't have post-SSO verification enabled can becomea target forcredential forgery attacks.

Post-SSO verification helps you mitigate the potential effect of an IdP compromise because users mustperform 2-step verification after each SSO attempt. If you provision Googlesign-in credentials for privileged users, post-SSO verification can help improvethe security posture of these user accounts without additional overhead.

Use a separate OU for privileged users

Privileged users who can authenticate during external IdP outages require aspecial configuration. This configuration differs from the configuration forregular users and for emergency access users.

To help you keep privileged users separate from these other user accounts, usea dedicated OU for privileged users. This separate OU helps you apply custompolicies such as post-SSO verification to only these privileged users.

A separate OU also helps you selectively disable SSO for privileged usersduring an IdP outage. To disable SSO for the OU, you canmodify the SSO profile assignments.

Use a backup IdP

When youprovide authentication alternatives for critical users during IdP outages, you help reduce the effect of that IdP outage on yourorganization. However, this mitigation strategy might not be sufficient tomaintain full operational capacity. Many users might still be unable to accessessential applications and services.

To further reduce the potential effect of an IdP outage, you can fail over to abackup IdP. You can use the following backup plan:

• During normal operations, you let users authenticate by using SSO andyour primary IdP.
• During an IdP outage, you change the SSO configuration of yourCloud Identity or Google Workspace account to switch to thebackup IdP.

The backup IdP doesn't need to be from the same vendor. When you create abackup IdP, use a configuration that matches the configuration of your primaryIdP. To ensure that the backup IdP lets all of your users authenticate andaccess Google services, the backup IdP must use an up-to-date copy of theprimary IdP's user base.

A backup IdP can help provide comprehensive contingency access. However, youmust weigh these advantages against the additional risks that a backup IdP mightintroduce. These potential risks include the following:

• If the backup IdP has weaker security than the primary IdP, the overallsecurity posture of your Google Cloud environment might also beweaker during a failover.
• If the primary IdP and backup IdP differ inhow they issue SAML assertions,the IdP might put users at risk of spoofing attacks.

Note: To change the SSO configuration to switch to the backup IdP, a usermust have super admin privileges. This action might alsorequire multi-party approval.

The following sections describe the recommended best practices when you use abackup IdP for contingency access.

Create a separate SAML profile for the backup IdP

Cloud Identity and Google Workspace let you create multipleSAML profiles. Each SAML profile can refer to a different SAML IdP.

To minimize the amount of work that's required to fail over to the backup IdP,prepare a SAML profile for the backup IdP in advance:

• Create separate SAML profiles for your primary IdP and for your backup IdP.
• ConfigureSSO profile assignments to assign only the primary IdP's SAML profile during normal operations.
• Modify SSO profile assignments to use the backup IdP's SAML profileduring an IdP outage. Don't change the individual SAML profile settings.

Use an existing on-premises IdP

You don't need to provision an additional IdP to serve as the backup. Instead,check whether you can use an existing on-premises IdP for this purpose. For example,your organization might use Active Directory as its authoritative source foridentities, and it might also useActive Directory Federation Services (AD FS) for SSO.In this scenario, you might be able to use AD FS as the backup IdP.

This reuse approach can help you limit cost and maintenance overhead.

Prepare the backup IdP to handle the required load

When you switch authentication to the backup IdP, it must handle all of theauthentication requests that your primary IdP normally handles.

When you deploy and size a backup IdP, remember that the number of expectedrequests depends on the following factors:

• The number of users in your Cloud Identity orGoogle Workspace account.
• The configuredGoogle Cloud session length.

For example, if the session length is between 8 and 24 hours, theauthentication requests might spike during the morning hours when employeesbegin their workday.

Test the failover procedure periodically

To help ensure that the SSO failover process works reliably, you mustperiodically verify the process. When you test the failover procedure, do thefollowing:

• Manually modify theSSO profile assignment of one or more OUs or groups to use the backup IdP.
• Verify that SSO with the backup IdP works as expected.
• Verify that signing certificates are up-to-date.

What's next

• Review thesecurity best practices for administrator accounts.
• Learn more aboutbest practices for federating Google Cloud with an external identity provider.
• For more reference architectures, diagrams, and best practices, explore theCloud Architecture Center.

Contributors

Author:Johannes Passing | Cloud Solutions Architect

Other contributor:Ido Flatow | Cloud Solutions Architect