Exporting and importing data

You can use the Firestore managed export and import service torecover from accidental deletion of data and to export data for offlineprocessing. You can export all documents or just specific collection groups. Likewise,you can import all data from an export or only specific collection groups. Dataexported from one Firestore database can be imported into anotherFirestore database. You can alsoloadFirestore exports into BigQuery.

This page describes how to export and import Firestore documents usingthe managed export and import service andCloud Storage. TheFirestore managed export and import service is available throughthegcloudcommand-line tool and the FirestoreAPI (REST,RPC).

1. Enable billing for your Google Cloud project. Only Google Cloud projects with billing enabled can use the export and import functionality.Note: Firebase projects must be on theBlaze plan to use the managed export and import service. Enabling billing for the Google Cloud automatically upgrades your Firebase project to the Blaze plan.
2. Create a Cloud Storage bucket for your project in a location nearyour Firestore database location. You cannot use a Requester Pays bucket for export and import operations.

3. Make sure your account has the necessary permissions for Firestore and Cloud Storage.If you are the project owner, your account has the required permissions. Otherwise, the following roles grant the necessary permissions for export and import operations and for access to Cloud Storage:

   • Firestore roles:Owner,Cloud Datastore Owner, orCloud Datastore Import Export AdminNote: These Datastore roles also grant permissions in Firestore.

   • Cloud Storage roles:Owner orStorage Admin

Service agent permissions

Export and import operations use a Firestore service agent toauthorize Cloud Storage operations. The Firestore service agentuses the following naming convention:

Firestore service agent

service-PROJECT_NUMBER@gcp-sa-firestore.iam.gserviceaccount.com

To learn more about service agents, seeService agents.

The Firestore service agent requires access to theCloud Storage bucket used in an export or import operation.If yourCloud Storage bucket is in the same project as your Firestoredatabase, then the Firestore service agent can access the bucket by default.

If the Cloud Storage bucket is in another project, then youmust give the Firestore service agent access to the Cloud Storagebucket.

Assign roles to the service agent

You can use thegsutil command-line tool toassign one of the roles below. For example, to assign the Storage Admin roleto the Firestore service agent, run the following:

gsutiliamchserviceAccount:service-PROJECT_NUMBER@gcp-sa-firestore.iam.gserviceaccount.com:roles/storage.admin\gs://[BUCKET_NAME]

ReplacePROJECT_NUMBER with your project number, whichis used to name your Firestore service agent. To view theservice agent name, seeView service agent name.

Alternatively, you canassign this role using the Google Cloud console.

View service agent name

You can view the account that your import and export operations use to authorizerequests from theImport/Export page in the Google Cloud console. You can alsoview whether your database uses the Firestoreservice agent or the legacy App Engine service account.

1. In the Google Cloud console, go to theDatabases page.

   Go to Databases

2. Select the required database from the list of databases.

3. In the navigation menu, clickImport/Export.

4. View the authorization account next to theImport/Export jobs run as label.

The service agent needs theStorage Admin role for the Cloud Storagebucket to be used for the export or import operation.

Set upgcloud for your project

You can initiate import and export operations through the Google Cloud console orthegcloud command-line tool. To usegcloud, set up the command-line tooland connect to your project in one of the following ways:

• Accessgcloud from the Google Cloud Platform console usingCloud Shell.

  Start Cloud Shell

  Make suregcloud is configured for the correct project:

  gcloudconfigsetproject[PROJECT_ID]

• Install and initialize the Google Cloud SDK.

Export data

An export operation copies documents in your database to a set of files in aCloud Storage bucket. Note that an export is not an exact database snapshot takenat the export start time. An export may include changes made while the operationwas running.

• Import only specific collection groups
• Load Firestore data into BigQuery

Export all documents

  gcloud firestore export gs://[BUCKET_NAME] \  --database=[DATABASE]

Export specific collection groups

gcloud firestore export gs://[BUCKET_NAME] \--collection-ids=[COLLECTION_GROUP_ID_1],[COLLECTION_GROUP_ID_2] \--database=[DATABASE]

gcloud firestore export gs://[BUCKET_NAME] \--collection-ids=reviews \--database='cymbal'

Export from a PITR timestamp

You can export your database to Cloud Storage fromPITR data.You can export PITR data where the timestamp is a whole minute timestamp withinthe past seven days, but not earlier than theearliestVersionTime. If data nolonger exists at the specified timestamp, the export operation fails.

The PITR export operation supports all filters, including exporting alldocuments and exporting specific collection groups.

Note the following points before exporting PITR data:

• Specify the timestamp inRFC 3339 format. For example,2023-05-26T10:20:00.00Z.
• Make sure that the timestamp you specify is a whole minute timestampwithin the past seven days, but not earlier than theearliestVersionTime. If data no longer exists at the specifiedtimestamp, an error is generated.
• You are not charged for a failed PITR export.

gcloudfirestoreexportgs://[BUCKET_NAME_PATH]\--snapshot-time=[PITR_TIMESTAMP]

Import data

Once you have export files in Cloud Storage, you can import documents in thosefiles back into your project or to another project. Note the following pointsabout import operations:

• When you import data, the required indexes are updated using your database'scurrent index definitions. An export does not contain index definitions.

• Imports do not assign new document IDs. Imports use the IDs captured atthe time of the export. As a document is being imported, its ID is reservedto prevent ID collisions. If a document with the same ID already exists, theimport overwrites the existing document.

• If a document in your database is not affected by an import, it will remainin your database after the import.

• Import operations do not trigger Cloud Functions.Snapshot listenersdo receive updates related to import operations.

• The.overall_export_metadata file name must match the name of its parentfolder:

  gs://BUCKET_NAME/OPTIONAL_NAMESPACE_PATH/PARENT_FOLDER_NAME/PARENT_FOLDER_NAME.overall_export_metadata

  If you move or copy the output files of an export, keep thePARENT_FOLDER_NAME and.overall_export_metadata file name thesame.

Import all documents from an export

gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/ --database=[DATABASE]

gcloud firestore import gs://my-bucket/2017-05-25T23:54:39_76544/ --database='cymbal'

Import specific collection groups

  gcloud firestore import gs://[BUCKET_NAME]/[EXPORT_PREFIX]/ \  --collection-ids=[COLLECTION_GROUP_ID_1],[COLLECTION_GROUP_ID_2] \  --database=[DATABASE]

Import a PITR export

Use the steps inImport all documents to import your exporteddatabase. If any document already exists in your database, it will beoverwritten.

Managing export and import operations

After you start an export or import operation, Firestore assignsthe operation a unique name. You can use the operation name to delete,cancel, or status check the operation.

Operation names are prefixed withprojects/[PROJECT_ID]/databases/(default)/operations/,for example:

projects/my-project/databases/(default)/operations/ASA1MTAwNDQxNAgadGx1YWZlZAcSeWx0aGdpbi1zYm9qLW5pbWRhEgopEg

However, you can leave out the prefix when specifying an operation name forthedescribe,cancel, anddeletecommands.

List all export and import operations

gcloud firestore operations list

Check operation status

gcloud firestore operations describe [OPERATION_NAME]

Estimate the completion time

A request for the status of a long-running operation returns the metricsworkEstimated andworkCompleted. Each of these metrics is returned in bothnumber of bytes and number of entities:

• workEstimated shows the estimated total number of bytes and documents anoperation will process. Firestore might omit this metric if itcannot make an estimate.

• workCompleted shows the number of bytes and documents processed so far.After the operation completes, the value shows the total number ofbytes and documents that were actually processed, which might be larger than thevalue ofworkEstimated.

DivideworkCompleted byworkEstimated for a rough progress estimate. Thisestimate might be inaccurate, because it depends on delayed statisticscollection.

Cancel an operation

gcloud firestore operations cancel [OPERATION_NAME]

Cancelling a running operation does not undo the operation. A cancelled exportoperation will leave documents already exported in Cloud Storage, and a cancelledimport operation will leave in place updates already made to your database. Youcannot import a partially completed export.

Delete an operation

Use thegcloud firestore operations delete command to removean operation from the list of recent operations. This command will not deleteexport files from Cloud Storage.

gcloud firestore operations delete [OPERATION_NAME]

Billing and pricing for export and import operations

You are required to enable billing for your Google Cloud project before you usethe managed export and import service.

Export and import operations are chargedfor document reads and writes at the rates listed inFirestore pricing. Export operations incur one readoperation per document exported. Import operations incur one write operation perdocument imported.

Output files stored in Cloud Storage count towards yourCloud Storage data storage costs.

Export or import operations will not trigger yourGoogle Cloud budget alerts until after completion. Export and importoperations will not affect the usage shown in the usage section of the console.

Viewing export and import costs

Export and import operations apply thegoog-firestoremanaged:exportimportlabel to billed operations. In theCloud Billing reports page,you can use this label to view costs related to import and export operations:

You can load data from a Firestore export into BigQuery,but only if you specified acollection-ids filter. SeeLoading data from Firestore exports.

BigQuery column limit

BigQuery imposes a limit of 10,000 columns per table.Firestore export operations generate a BigQuery tableschema for each collection group. In this schema, each unique field name withina collection group becomes a schema column.

If a collection group's BigQuery schema surpasses 10,000 columns, theFirestore export operation attempts to stay under the column limitby treating map fields as bytes. If this conversion brings thenumber of columns below 10,000, you can load the data intoBigQuery, but you cannot query the subfields within the map fields.If the number of columns still exceeds 10,000, the export operation does notgenerate a BigQuery schema for the collection group and you cannot loadits data into BigQuery.

Export format and metadata files

The output of a managed export uses theLevelDB log format.

Metadata files

An export operation creates a metadata file for each collection groupyou specify. Metadata files are typically namedALL_NAMESPACES_KIND_[COLLECTION_GROUP_ID].export_metadata.

The metadata files are protocol buffers and you can decode them with theprotoc protocol compiler.For example, you can decode a metadata file to determine the collection groupsthe export files contain:

protoc --decode_raw < export0.export_metadata

Service agent migration

Firestore uses a Firestore service agent to authorize importand export operations instead of using the App Engine service account.The service agent and service account use the following naming conventions:

Firestore service agent

service-PROJECT_NUMBER@gcp-sa-firestore.iam.gserviceaccount.com

Firestore previously used the App Engine default serviceaccount instead of the Firestore service agent. If your databasestill uses the App Engine service account to import or export data, werecommend that you follow the instructions in this section to migrateto using the Firestore service agent.

App Engine service account

PROJECT_ID@appspot.gserviceaccount.com

The Firestore service agent is preferable because it is specificto Firestore. The App Engine service account is sharedby more than one service.

1. In the Google Cloud console, go to theDatabases page.

   Go to Databases

2. Select the required database from the list of databases.

3. In the navigation menu, clickImport/Export.

4. View the authorization account next to theImport/Export jobs run as label.

If your project does not use the Firestore service agent, youcan migrate to the Firestore service agent using either ofthese techniques:

• Migrate a project by checking and updating Cloud Storage bucket permissions (recommended).
• Add an organization-wide policy constraint that affects all projects withinthe organization.

The first of these techniques is preferable because it localizes the scope ofeffect to a single Firestore project. The second technique is notpreferred because it doesn't migrate existing Cloud Storage bucketpermissions. It does, however, offer security compliance at the organizationlevel.

Migrate by checking and updating Cloud Storage bucket permissions

The migration process has two steps:

1. Update Cloud Storage bucket permissions. See the following section fordetails.
2. Confirm migration to the Firestore service agent.

Service agent bucket permissions

For any export or import operations that use a Cloud Storage bucket inanother project, you must grant the Firestore service agentpermissions for that bucket. For example, operations that move data to anotherproject need to access a bucket in that other project. Otherwise, theseoperations fail after migrating to the Firestore serviceagent.

Import and export workflows that stay within the same project do not requirechanges to permissions. The Firestore service agent can accessbuckets in the same project by default.

Update the permissions for Cloud Storage buckets from other projects to giveaccess to theservice-PROJECT_NUMBER@gcp-sa-firestore.iam.gserviceaccount.comservice agent. Grant the service agent theFirestore Service Agent role.

TheFirestore Service Agent role grants read and write permissions for aCloud Storage bucket. If you need to grant only read or only writepermissions, use acustom role.

The migration process described in the following sectionhelps you identify Cloud Storagebuckets that might require permission updates.

Migrate a project to the Firestore Service Agent

Complete the following steps to migrate from the App Engine service account tothe Firestore service agent. Once completed, the migration can't beundone.

1. In the Google Cloud console, go to theDatabases page.

   Go to Databases

2. Select the required database from the list of databases.

3. In the navigation menu, clickImport/Export.

4. If your project has not yet migrated to the Firestore service agent, you see a banner describing the migration and aCheck Bucket Status button. The next step helps you identify and fix potential permission errors.

   ClickCheck Bucket Status.

   A menu appears with the option to complete your migration and a list of Cloud Storage buckets. It may take a few minutes for the list to finish loading.

   This list includes buckets which were recently used in import and export operations, but do not currently give read and write permissions to the Firestore service agent.

5. Take note of the principal name of your project's Firestore service agent. The service agent name appears under theService agent to give access to label.

6. For any bucket in the list that you will use for future import or export operations, complete the following steps:

   1. In this bucket's table row, clickFix. This opens that bucket's permissions page in a new tab.

   2. ClickAdd.
   3. In theNew principals field, enter the name of your Firestore service agent.
   4. In theSelect a role field, selectService Agents > Firestore Service Agent.
   5. ClickSave.
   6. Return to the tab with the Firestore Import/Export page.
   7. Repeat these steps for other buckets in the list. Make sure to view all the pages of the list.

7. ClickMigrate to Firestore Service Agent. If you still have buckets with failed permission checks, you need to confirm your migration by clickingMigrate.

   An alert informs you when your migration completes. Migration can't be undone.

View migration status

To verify your project's migration status:

1. In the Google Cloud console, go to theDatabases page.

   Go to Databases

2. Select the required database from the list of databases.

3. In the navigation menu, clickImport/Export.

4. Look for the principal next to theImport/Export jobs run as label.

   If the principal isservice-PROJECT_NUMBER@gcp-sa-firestore.iam.gserviceaccount.com,then your project has already migrated to the Firestoreservice agent. The migration can't be undone.

   If the project has not been migrated, a banner appears at the top of the pagewith aCheck Bucket Status button. SeeMigrate to the Firestore service agentto complete the migration.

Add an organization-wide policy constraint

• Set the following constraint in your organization's policy:

  Require Firestore Service Agent for import/export (firestore.requireP4SAforImportExport).

  This constraint requires import and export operations to use theFirestore service agent to authorize requests.To set this constraint, seeCreating and managing organization policies.

Applying this organizational policy constraint does not automatically grant theappropriate Cloud Storage bucket permissions for theFirestore service agent.

If the constraint creates permission errors for any import or export workflows,you can disable it to go back to using default service account.After youcheck and update Cloud Storage bucketpermissions, you can enable the constraint again.