This page describes the process for setting up external server replication usinga custom import. These steps are the best option when you need to replicatefrom a large external database.

You must complete all the steps on this page. When finished, you canadminister and monitor the replica the same way as you would any otherCloud SQL instance.

This process is supported only for external servers that are configured to use global transaction identifier (GTID)-based replication. Before replication can be initiated, you need to load data from the external server into the Cloud SQL replica. If you don't use GTID-based replication, then Cloud SQL can't identify the exact binary log position from which to begin replication. If you can't use GITD-based replication, then you need to configure your dump tool to institute a global read-only lock during the dump process.

Before you begin

Before you start, you should haveconfigured the external server,created thesource representation instance,and set up theCloud SQL replica.

Update permissions for the replication user

The replication user on the external server is configured toaccept connections from any host (%). You should update this user accountso that it can only be used with the Cloud SQL replica.Open a terminal on the source database server and enter these commands:

UPDATEmysql.userSETHost='NEW_HOST'WHEREHost='OLD_HOST'ANDUser='USERNAME';GRANTREPLICATIONSLAVE,EXECUTEON*.*TO'GCP_USERNAME'@'HOST';FLUSHPRIVILEGES;

UPDATEmysql.userSETHost='192.0.2.0'WHEREHost='%'ANDUser='replicationUser';GRANTREPLICATIONSLAVE,EXECUTEON*.*TO'gcp_user'@'gmail.com';FLUSHPRIVILEGES;

Set up the Cloud SQL replica as a primary instance

Because Cloud SQL replica instances are read-only, in order to performa custom import, you need topromote the Cloud SQL replicato a standalone instance. Once the initial data import is complete, you demotethe instance back to a replica.

Perform a custom dump and import

In this section, we show you how to create the dump file and import it intothe eventual Cloud SQL replica usingmydumper or themysqldumpclient utilities.

When you dump the data, you might need to exclude MySQL generic databases,includingmysql, andsys, if they exist on the source instance. Otherwise,the data import fails. SeeHow to exclude (or include) databases?.

Usemydumper andmyloader

To create a dump file and import it to Cloud SQL:

1. Create a dump file of the external server database usingmydumper.

   $mydumper-uUSERNAME-pPASSWORD\--threads=16-o./backup\-hHOST\--no-locks\--regex'^(?!(mysql\.|sys\.))'

   Property	Description
   USERNAME	The name of the replication user account or useraccount on the external server that has database readpermissions.
   PASSWORD	Replication user password.
   HOST	The IPv4 or DNS address for the external server.

2. Import the data into the Cloud SQL instance usingmyloader.

   $myloader-uREPLICA_USERNAME-pREPLICA_PASSWORD\--threads=16\-d./backup-hHOST-o

   Property	Description
   REPLICA_USERNAME	The user account on the Cloud SQL instance.
   REPLICA_PASSWORD	Cloud SQL instance user password.
   HOST	The IPv4 for the Cloud SQL instance.

3. Write down the GTID or binlog information of the data dump. You need thisinformation when configuring the replication with the stored procedures.

   To get the GTID or binlog information of the data dump, run the followingcommand:

   sudocat./backup/metadata

Usemysqldump

1. Create a dump usingmysqldump:

   mysqldump

   mysqldump\--host=EXTERNAL_HOST\--port=EXTERNAL_PORT\--user=USERNAME\--password=PASSWORD\--databases=DATABASE_LIST\--hex-blob\--master-data=EXTERNAL_DATA\--no-autocommit\--default-character-set=utf8mb4\--single-transaction\GTID_PURGED\ADD_DROP_TABLE\ROUTINES\COMPRESS\GZIP

   Property	Description
   EXTERNAL_HOST	The IPv4 or DNS address for the external server.
   EXTERNAL_PORT	The port for the external server.If the external server is hosted on Cloud SQL,this is3306.
   USERNAME	The name of the replication user account or useraccount on the external server that has database readpermissions.
   USER_PASSWORD	Replication user password.
   DATABASE_LIST	Space-separated list of all databases on theexternal server, except for the system databases(sys,mysql,performance_schema, andinformation_schema). Use theSHOW DATABASESMySQL command to list your databases.
   EXTERNAL_DATA	If your external server does not supportGTID, and you have permission to access the global read lockon it, use--master-data=1. Otherwise, don't use thisproperty.
   GTID_PURGED	If your external server supports GTID, use--set-gtid-purged=on; otherwise, don't use this property.
   ADD_DROP_TABLE	If you want to add aDROP TABLE statement beforeeachCREATE TABLE statement, include--add-drop-table.
   ROUTINES	If you want to show stored routines, such asprocedures and functions, in the output for dumpeddatabases, include--routines.
   COMPRESS	If you want to compress all information sent betweenthe Cloud SQL replica and the external server,use--compress.
   GZIP	If you want to compress the dump file even more, use| gzip. If your database contains data that doesnot compress well, such as binary incompressible dataor JPG images, don't use this.

   example

   mysqldump\--host=192.0.2.1\--port=3306\--user=replicationUser\--password\--databasesguestbookjournal\--hex-blob\--master-data=1\--no-autocommit\--default-character-set=utf8mb4\--single-transaction\--compress\|gzip

2. Write down the GTID or binlog information of the data dump. You need thisinformation to configure the replication with the Cloud SQL storedprocedures.

   For the GTID, look for a line similar to the following:

   SET@@GLOBAL.GTID_PURGED='32eb1e6a-17b6-11ea-a39e-06d94ac3ec98:1-33496';

   For the binlog, look for a line similar to the following:

   CHANGEMASTERTOMASTER_LOG_FILE='mysql-bin-changelog.033877',MASTER_LOG_POS=360;

3. Remove the following lines in the dump file that require super privileges.Since Cloud SQL users don't have super privileges, these lines causethe import to fail.

   For GTID-based replication: Remove the SET GTID_PURGED statement along withthe session variable setting statement in the dump. For example:

   ...SET@MYSQLDUMP_TEMP_LOG_BIN=@@SESSION.SQL_LOG_BIN;SET@@SESSION.SQL_LOG_BIN=0;...SET@@GLOBAL.GTID_PURGED='32eb1e6a-17b6-11ea-a39e-06d94ac3ec98:1-33496';...SET@@SESSION.SQL_LOG_BIN=@MYSQLDUMP_TEMP_LOG_BIN;

   For binlog-based replication, remove the CHANGE MASTER statement. Forexample:

   ...CHANGEMASTERTOMASTER_LOG_FILE='mysql-bin-changelog.033877',MASTER_LOG_POS=360;...

4. Import the data into the Cloud SQL replica, using themysql CLI:

   mysql

   mysql-hREPLICA_HOST-uREPLICA_USER\-pREPLICA_DATABASE_NAMERESULT_FILE

   Property	Description
   REPLICA_HOST	Host on which MySQL server is located.
   REPLICA_USER	MySQL user name to use when connectingto the server.
   REPLICA_DATABASE_NAME	Name of the database where the data islocated.
   RESULT_FILE	Name of the dump file to import.

   example

   mysql-h255.255.255.255-ureplica_username-preplica_db <result.sql

You can also import the dump file using a Google Cloud bucket. SeeImporting data from a SQL dump file into Cloud SQL.

Demote the Cloud SQL instance

To demote the Cloud SQL instance to a Cloud SQL replica, use thedemoteMastermethod on the instance.

1. Prepare a request JSON file with the name of the instance you want to demote.

   Source JSON

   {"demoteMasterContext":{"masterInstanceName":SOURCE_REPRESENTATION_INSTANCE_NAME,"skipReplicationSetup":true}}

   Property	Description
   SOURCE_REPRESENTATION_INSTANCE_NAME	The name of the sourcerepresentation instance.

   example

   {"demoteMasterContext":{"masterInstanceName":"cloudsql-source-instance","skipReplicationSetup":true}}

2. Open a terminal and use the following commands to invokedemoteMaster:

   curl

   gcloudauthloginACCESS_TOKEN="$(gcloudauthprint-access-token)"curl--header"Authorization: Bearer${ACCESS_TOKEN}"\--header'Content-Type: application/json'\--data@JSON_PATH\-XPOST\https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT-ID/instances/INSTANCE-NAME/demoteMaster

   Property	Description
   JSON_PATH	The path to theJSON file.
   PROJECT_ID	The ID of your project in Google Cloud.
   INSTANCE-NAME	The name of the instance to demote.

   example

   gcloudauthloginACCESS_TOKEN="$(gcloudauthprint-access-token)"curl--header"Authorization: Bearer${ACCESS_TOKEN}"\--header'Content-Type: application/json'\--data@./source.json\-XPOST\https://sqladmin.googleapis.com/sql/v1beta4/projects/MyProject/instances/cloudsql-replica-instance/demoteMaster

What you should see when you finish

To ensure your instances were set up correctly, go to theCloud SQL Instances page.

You should see your source representation instance and Cloud SQL replica.They look similar to the following:

Start replication on the Cloud SQL instance

This step uses Cloud SQL stored procedures. The Cloud SQL storedprocedures are installed after calling thedemoteMaster request. They areremoved after callingpromoteReplica. For more information, seeStored procedures for replication management.

1. Log on to the replica instance. For more information, seeConnecting using a database client from a local machine.

2. Use themysql.resetMaster stored procedure to reset replication settings.

   mysql>callmysql.resetMaster();

3. Configure the replication. This step requires the GTID or binlog informationthat you previously wrote down.

   GTID

   1. Configure thegtid_purged field with themysql.skipTransactionWithGtid(GTID_TO_SKIP)stored procedure.

   Property	Description
   GTID_TO_SKIP	TheGTID setvalue to configure.

   For example:

   mysql>callmysql.skipTransactionWithGtid('32eb1e6a-17b6-11ea-a39e-06d94ac3ec98:1-33496');

   1. Run themysql.setupExternalSourceAutoPosition(HOST,PORT,USER_NAME,USER_PASSWORD,MASTER_AUTO_POSITION,USE_SSL,USE_SSL_CLIENT_AUTH) stored procedure.

   Property	Description
   HOST	Source endpoint.
   PORT	Source port.
   USER_NAME	Source user.
   USER_PASSWORD	Source user password.
   MASTER_AUTO_POSITION	Value of themaster_auto_positionparameter. Possible values are0,1.
   USE_SSL	Whether to use SSL-basedreplication. Possible values aretrue,false. Iftrue, you needto set thecaCertificate field intheDemoteMaster request.
   USE_SSL_CLIENT_AUTH	Whether to use SSL clientauthentication. Possible values aretrue,false. Iftrue, you needto set theclientKey andclientCertificatesfields in thedemoteMasterrequest.

   mysql>callmysql.setupExternalSourceAutoPosition('1.1.1.1',3306,\'USERNAME','PASSWORD',\/*master_auto_position=*/1,false,false);\

   binlog

   Run themysql.setupExternalSource(HOST,PORT,USER_NAME,USER_PASSWORD,SOURCE_LOG_NAME,SOURCE_LOG_POS,USE_SSL,USE_SSL_CLIENT_AUTH) stored procedure.

   Property	Description
   HOST	Source endpoint.
   PORT	Source port.
   USER_NAME	Source user.
   USER_PASSWORD	Source user password.
   SOURCE_LOG_NAME	The name of the binary log on thesource database instance thatcontains the replicationinformation.
   SOURCE_LOG_POS	The location in themysql_binary_log_file_name binarylog at which replication startsreading the replication information.
   USE_SSL	Whether to use SSL-basedreplication. Possible values aretrue,false. Iftrue, you needto set thecaCertificate field intheDemoteMaster request.
   USE_SSL_CLIENT_AUTH	Whether to use SSL clientauthentication. Possible values aretrue,false. Iftrue, you needto set theclientKey andclientCertificatesfields in thedemoteMasterrequest.

   mysql>callmysql.setupExternalSource('1.1.1.1',3306,\'user_name','password','mysql-bin-changelog.033877',360,\false,false);

4. Use themysql.startReplication() stored procedure to start replication fromthe external database.

   mysql>callmysql.startReplication();

5. Verify the replication status. Make sure that both theSlave_IO_Running andSlave_SQL_Running fields sayYES.

   mysql>showslavestatus\G

   The output from this command looks similar to the following:

   ***************************1.row***************************Slave_IO_State:WaitingformastertosendeventMaster_Host:1.1.1.1Master_User:user_nameMaster_Port:3306Connect_Retry:60Master_Log_File:mysql-bin-changelog.000001Read_Master_Log_Pos:1Relay_Log_File:relay-log.000002Relay_Log_Pos:1Relay_Master_Log_File:mysql-bin-changelog.000001Slave_IO_Running:YesSlave_SQL_Running:YesReplicate_Do_DB:Replicate_Ignore_DB:Replicate_Do_Table:Replicate_Ignore_Table:Replicate_Wild_Do_Table:Replicate_Wild_Ignore_Table:mysql.%Last_Errno:0Last_Error:Skip_Counter:0Exec_Master_Log_Pos:412Relay_Log_Space:752Until_Condition:NoneUntil_Log_File:Until_Log_Pos:0Master_SSL_Allowed:NoMaster_SSL_CA_File:Master_SSL_CA_Path:Master_SSL_Cert:Master_SSL_Cipher:Master_SSL_Key:Seconds_Behind_Master:0Master_SSL_Verify_Server_Cert:NoLast_IO_Errno:0Last_IO_Error:Last_SQL_Errno:0Last_SQL_Error:Replicate_Ignore_Server_Ids:Master_Server_Id:1509941531Master_UUID:1cb2c80e-90f0-11eb-9ea3-02389b1c2e6fMaster_Info_File:mysql.slave_master_infoSQL_Delay:0SQL_Remaining_Delay:NULLSlave_SQL_Running_State:SlavehasreadallrMaster_Retry_Count:86400Master_Bind:Last_IO_Error_Timestamp:Last_SQL_Error_Timestamp:Master_SSL_Crl:Master_SSL_Crlpath:Retrieved_Gtid_Set:Executed_Gtid_Set:478af53c-bd24-11eb-be72-42010a80002a:1-226Auto_Position:01rowinset(0.00sec)

Proceed with replication

Once you start replication from the external server, you need to monitorreplication and then complete your migration. To learn more, seeMonitoring replication.

Troubleshoot

Review your replication logs

When youverify your replication settings, logs areproduced.

You can view these logs by following these steps:

1. Go to the Logs Viewer in the Google Cloud console.

   Go to the Logs Viewer

2. Select the Cloud SQL replica from theInstance dropdown.
3. Select thereplication-setup.log log file.

If the Cloud SQL replica is unable to connect to the external server,confirm the following:

• Any firewall on the external server is configured to allow connections from the Cloud SQL replica'soutgoing IP address.
• Your SSL/TLS configuration is correct.
• Your replication user, host, and password are correct.

What's next

• Learn aboutupdating an instance.
• Learn aboutmanaging replicas.
• Learn aboutmonitoring instances.
• Learn aboutpromoting your Cloud SQL replica topromote the replica to a standalone instance and stop replicating from theexternal server.