Use a managed import to set up replication from external databases

MySQL  |  PostgreSQL  |  SQL Server

This page describes how to set up and use a managed import for data whenreplicating from an external server to Cloud SQL.

You must complete all the steps on this page. When finished, you canadminister and monitor the source representation instance the same way as youwould any other Cloud SQL instance.

Before you begin

Before you begin, complete these steps:

  1. Configure the external server.

  2. Create the source representation instance.

  3. Set up the Cloud SQL replica.

Verify your replication settings

After your setup is complete, ensure that the Cloud SQL replicacan replicate from the external server.

The following external sync settings must be correct.

  • Connectivity between the Cloud SQL replica and external server
  • Replication user privileges
  • Version compatibility
  • The Cloud SQL replica is not already replicating

To verify these settings, open a Cloud Shell terminal and enter the followingcommands:

curl

gcloud auth loginACCESS_TOKEN="$(gcloud auth print-access-token)"curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \     --header 'Content-Type: application/json' \     --data '{         "syncMode": "SYNC_MODE",         "syncParallelLevel": "SYNC_PARALLEL_LEVEL",         "selectedObjects": "SELECTED_OBJECTS"       }' \     -X POST \     https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT_ID/instances/REPLICA_INSTANCE_ID/verifyExternalSyncSettings

example

gcloudauthloginACCESS_TOKEN="$(gcloud auth print-access-token)"curl--header "Authorization: Bearer ${ACCESS_TOKEN}" \--header 'Content-Type: application/json' \--data '{"syncMode":"online","syncParallelLevel":"optimal","selectedObjects":[{"database":"db1"},{"database":"db2"}]}'\-XPOST\https://sqladmin.googleapis.com/sql/v1beta4/projects/myproject/instances/myreplica/verifyExternalSyncSettings

These calls return a list of typesql#externalSyncSettingErrorList.

If the list is empty, then there are no errors. A response without errors appears like this:

{"kind":"sql#externalSyncSettingErrorList"}
PropertyDescription
SYNC_MODEEnsure that you can keep the Cloud SQL replica and the external serverin sync after replication is set up. Sync modes includeEXTERNAL_SYNC_MODE_UNSPECIFIED,ONLINE, andOFFLINE.
SYNC_PARALLEL_LEVEL

Verify the setting that controls the speed at which data from tables of a database are transferred. The following values are available:

  • min: Takes the lowest amount of compute resources on the database. This is the slowest speed for transferring data.
  • optimal: Provides a balanced performance with an optimal load on the database.
  • max: Provides the highest speed for transferring data, but this might cause an increased load on the database.

Note: The default value for this parameter isoptimal because this setting provides a good speed to transfer the data and it has a reasonable impact on the database. We recommend that you use this value.

SELECTED_OBJECTSA comma-separated list of databases that you're migrating from the source representation instance to the destination Cloud SQL instance. If you don't use this parameter or provide an empty list as the value for the parameter, then all databases are migrated from the source to the destination.
PROJECT_IDThe ID of your Google Cloud project.
REPLICA_INSTANCE_IDThe ID of your Cloud SQL replica.

Update a source representation instance

To update the databases that you want to migrate from the source representationinstance to the destination Cloud SQL instance, update the list ofdatabase names that are associated with theselectedObjects parameter. If youdon't use this parameter or provide an empty list as the value for the parameter,then all databases are migrated from the source to the destination.

source.json

{"name":"SOURCE_NAME","region":"REGION","databaseVersion":"DATABASE_VERSION","onPremisesConfiguration":{"selectedObjects":"SELECTED_OBJECTS","username":"USERNAME","password":"PASSWORD"}}

example

//exampleofsource.jsonforexternalserverthat//-initiatesreplicationfromaCloudSQLmanagedimport//-doesn'tuseSSL/TLS{"name":"cloudsql-source-instance","region":"us-central1","databaseVersion":"POSTGRES_9_6","onPremisesConfiguration":{"selectedObjects":[{"database":"db1"},{"database":"db3"}],"username":"newReplicationUser","password":"525#@%*@"}}
PropertyDescription
SOURCE_NAMEThe name of the source representation instance.
REGIONTheregion where thesource representation instance resides.
DATABASE_VERSIONThe database version running on yourexternal server. The options arePOSTGRES_9_6,POSTGRES_10,POSTGRES_11,POSTGRES_12,POSTGRES_13,POSTGRES_14,POSTGRES_15,POSTGRES_16, orPOSTGRES_17.
SELECTED_OBJECTSAn updated comma-separated list of databases thatyou're migrating from the source representation instanceinstance to the destination Cloud SQLinstance.
USERNAMEThe replication user account on the external server.
PASSWORDThe password for the account.

Then, to modify the source representation instance in Cloud SQL, opena Cloud Shell terminal and enter the following commands:

curl

gcloud auth loginACCESS_TOKEN="$(gcloud auth print-access-token)"curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \     --header 'Content-Type: application/json' \     --data @JSON_PATH \     -X PATCH \     https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT_ID/instances/SOURCE_NAME

example

gcloudauthloginACCESS_TOKEN="$(gcloud auth print-access-token)"curl--header "Authorization: Bearer ${ACCESS_TOKEN}" \--header 'Content-Type: application/json' \--data @./source.json \-XPATCH\https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances/cloudsql-source-instance
PropertyDescription
JSON_PATHThe path to theJSON file that contains therequest data for the external server.
PROJECT_IDThe ID of your Google Cloud project.
SOURCE_NAMEThe name of the source representation instance.

Start replication on the external server

After verifying that you can replicate from the external server, start the replication. The speed for performing the replication for the initial import process is up to 500 GB per hour. However, this speed can vary based on the machine tier, data disk size, network throughput, and nature of your database.

curl

gcloud auth loginACCESS_TOKEN="$(gcloud auth print-access-token)"curl --header "Authorization: Bearer ${ACCESS_TOKEN}" \     --header 'Content-Type: application/json' \     --data '{         "syncMode": "SYNC_MODE",         "skipVerification": "SKIP_VERIFICATION",         "syncParallelLevel": "SYNC_PARALLEL_LEVEL"       }' \     -X POST \     https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT_ID/instances/REPLICA_INSTANCE_ID/startExternalSync

example

gcloudauthloginACCESS_TOKEN="$(gcloud auth print-access-token)"curl--header "Authorization: Bearer ${ACCESS_TOKEN}" \--header 'Content-Type: application/json' \--data '{"syncMode":"online","syncParallelLevel":"optimal"}'\-XPOST\https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances/replica-instance/startExternalSync
PropertyDescription
SYNC_MODEVerify that you can keep the Cloud SQL replica and the external serverin sync after replication is set up.
SKIP_VERIFICATIONWhether to skip the built-in verification step before syncing your data. This parameter is recommended only if you have alreadyverified your replication settings.
SYNC_PARALLEL_LEVEL

Provide a setting that controls the speed at which data from tables of a database are transferred. The following values are available:

  • min: Takes the lowest amount of compute resources on the database. This is the slowest speed for transferring data.
  • optimal: Provides a balanced performance with an optimal load on the database.
  • max: Provides the highest speed for transferring data, but this might cause an increased load on the database.

Note: The default value for this parameter isoptimal because this setting provides a good speed to transfer the data and it has a reasonable impact on the database. We recommend that you use this value.

PROJECT_IDThe ID of your Google Cloud project.
REPLICA_INSTANCE_IDThe ID of your Cloud SQL replica.

Monitor the migration

Once you start replication from the external server, you need to monitorreplication. To learn more, seeMonitoring replication.You can then complete your migration.

Troubleshoot

Consider the following troubleshooting options:

IssueTroubleshooting
Read replica didn't start replicating on creation.There's probably a more specific error in the log files.Inspect the logs in Cloud Logging to find the actual error.
Unable to create read replica - invalidFlagValue error.One of the flags in the request is invalid. It could be a flag you provided explicitly or one that was set to a default value.

First, check that the value of themax_connections flag is greater than or equal to the value on the primary.

If themax_connections flag is set appropriately,inspect the logs in Cloud Logging to find the actual error.

Unable to create read replica - unknown error.There's probably a more specific error in the log files.Inspect the logs in Cloud Logging to find the actual error.

If the error is:set Service Networking service account as servicenetworking.serviceAgent role on consumer project, then disable and re-enable theService Networking API. This action creates the service account necessary to continue with the process.

Disk is full.The primary instance disk size can become full during replica creation. Edit the primary instance to upgrade it to a larger disk size.
Disk space increases significantly.A slot that's not actively used to track data causes PostgreSQL tohold onto WAL segments indefinitely, causing the disk space to grow indefinitely. If you usethelogical replication and decoding features in Cloud SQL, replication slots are created and dropped automatically. Unused replication slots can be detected by querying thepg_replication_slotssystem view and filtering on theactive column. Unusedslots can be dropped to remove WAL segments using thepg_drop_replication_slot command.
The replica instance is using too much memory.The replica uses temporary memory to cache often-requested read operations, which can lead it to use more memory than the primary instance.

Restart the replica instance to reclaim the temporary memory space.

Replication stopped.The maximum storage limit was reached and automatic storage increase isn't enabled.

Edit the instance to enableautomatic storage increase.

Replication lag is consistently high.The write load is too high for the replica to handle. Replication lag takes place when the SQL thread on a replica is unable to keep up with the IO thread. Some kinds of queries or workloads can cause temporary or permanent high replication lag for a given schema. Some of the typical causes of replication lag are:
  • Slow queries on the replica. Find and fix them.
  • All tables must have a unique/primary key. Every update on such a table without a unique/primary key causes full table scans on th replica.
  • Queries likeDELETE ... WHERE field < 50000000 cause replication lag with row-based replication since a huge number of updates are piled up on the replica.

Some possible solutions include:

  • Edit the instance to increase the size of the replica.
  • Reduce the load on the database.
  • Send read traffic to the read replica.
  • Index the tables.
  • Identify and fix slow write queries.
  • Recreate the replica.
Errors when rebuilding indexes in PostgreSQL 9.6.You get an error from PostgreSQL informing you that you need to rebuild a particular index. This can be done only on the primary instance. If you create a new replica instance, you soon get the same error again.Hash indexes are not propagated to replicas in PostgreSQL versions below 10.

If you must use hash indexes, upgrade to PostgreSQL 10+. Otherwise, if you also want to use replicas, don't use hash indexes in PostgreSQL 9.6.

Query on the primary instance is always running.After creating a replica, the querySELECT * from pg_stat_activity where state = 'active' and pid = XXXX and username = 'cloudsqlreplica' is expected to run continuously on your primary instance.
Replica creation fails with timeout.Long-running uncommitted transactions on the primary instance can cause read replica creation to fail.

Recreate the replica after stopping all running queries.

If the primary instance and the replica havedifferent vCPU sizes, then there might be query performance issues because the query optimizer takes vCPU sizes into account.

To resolve this issue, complete the following steps:

  1. Turn on thelog_duration flag and set thelog_statement parameter toddl. This provides you with both the queries and the run time on the database. However, depending on your workload, this might cause performance issues.
  2. On both the primary instance and the read replica, runexplain analyze for the queries.
  3. Compare the query plan and check for differences.

If this is a specific query, then modify the query. For example, you can change the order of the joins to see if you get better performance.

Review your replication logs

When youverify your replication settings, logs areproduced.

You can view these logs by following these steps:

  1. Go to the Logs Viewer in the Google Cloud console.

    Go to the Logs Viewer

  2. Select the Cloud SQL replica from theInstance dropdown.
  3. Select thereplication-setup.log log file.

If the Cloud SQL replica is unable to connect to the external server,confirm the following:

  • Any firewall on the external server is configured to allow connections from the Cloud SQL replica'soutgoing IP address.
  • Your SSL/TLS configuration is correct.
  • Your replication user, host, and password are correct.

What's next

Except as otherwise noted, the content of this page is licensed under theCreative Commons Attribution 4.0 License, and code samples are licensed under theApache 2.0 License. For details, see theGoogle Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2025-07-14 UTC.