You can use the elasticsearch-dump tool to migrate the settings, mappings, and documents of indexes between Alibaba Cloud Elasticsearch clusters or between an on-premises machine and an Alibaba Cloud Elasticsearch cluster. This topic describes how to install and use elasticsearch-dump.
Background information
elasticsearch-dump is an open source tool used to migrate Elasticsearch data. For more information, see theofficial documentation of elasticsearch-dump.
Prerequisites
An Alibaba Cloud Elasticsearch cluster is created. For more information, seeCreate an Alibaba Cloud Elasticsearch cluster.
The Auto Indexing feature is enabled for the destination Alibaba Cloud Elasticsearch cluster, or an index is created in the Alibaba Cloud Elasticsearch cluster. For more information about how to enable the Auto Indexing feature, seeConfigure the YML file.
An Elastic Compute Service (ECS) instance is created. The ECS instance is used to install elasticsearch-dump. For more information, seeCreate an instance by using the wizard.
Use scenarios
elasticsearch-dump is used to migrate small volumes of data in scenarios with a small number of indexes.
Install elasticsearch-dump
Connect to the ECS instance.
For more information, seeConnect to a Linux instance by using a password or key.
Install Node.js.
Download the installation package.
wget https://nodejs.org/dist/v16.18.0/node-v16.18.0-linux-x64.tar.xzDecompress the package.
tar -xf node-v16.18.0-linux-x64.tar.xzConfigure environment variables.
If you want the environment variables to temporarily take effect, run the following command:
export PATH=$PATH:/root/node-v16.18.0-linux-x64/bin/If you want the environment variables to permanently take effect, run the following commands:
vim ~/.bash_profileexport PATH=$PATH:/root/node-v16.18.0-linux-x64/bin/source ~/.bash_profile
Install elasticsearch-dump.
npm install elasticdump -g
Examples
If the password that is used to access the source or destination Alibaba Cloud Elasticsearch cluster contains special characters such as# and$, the request is blocked by Alibaba Cloud Web Application Firewall (WAF), and an error is reported after the related command is run. For more information about the reported error and troubleshooting method, seeFAQ.
Migrate data to an Alibaba Cloud Elasticsearch cluster
Migrate the settings of an index
elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=settingsMigrate the mappings of an index
elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=mappingMigrate the documents of an index
elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=http://"<OtherName>:<OtherPassword>"@<OtherEsHost>/<OtherEsIndex> --type=data
Migrate data to your on-premises machine
Migrate the settings of an index
elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=settingsMigrate the mappings of an index
elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=mappingMigrate the documents of an index
elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> --type=dataMigrate data based on a query
elasticdump --input=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --output=<YourLocalFile> ----searchBody="<YourQuery>"
Restore data from your on-premises machine to an Alibaba Cloud Elasticsearch cluster
Restore the documents of an index
elasticdump --input=<YourLocalFile> --output=http://"<UserName>:<YourPassword>"@<YourEsHost>/<YourEsIndex> --type=dataFor more information about other data migration scenarios, see theofficial documentation of elasticsearch-dump.
The following table describes the parameters used in the preceding examples.
Parameter | Description |
<YourEsHost>/<OtherEsHost> | The internal or public endpoint of the source or destination Alibaba Cloud Elasticsearch cluster. You can obtain the internal or public endpoint of an Alibaba Cloud Elasticsearch cluster from the Basic Information page of the cluster. For more information, seeView the basic information of a cluster. Note If the server on which elasticsearch-dump is installed resides in the same virtual private cloud (VPC) as the source and destination Elasticsearch clusters, you can use the internal endpoint. Otherwise, you must use the public endpoint and configure a public IP address whitelist. For more information, seeConfigure a public or private IP address whitelist for an Elasticsearch cluster. |
<UserName>/<OtherName> | The username that is used to access the source or destination Alibaba Cloud Elasticsearch cluster. The default username for an Alibaba Cloud Elasticsearch cluster is elastic. |
<YourPassword>/<OtherPassword> | The password that is used to access the source or destination Alibaba Cloud Elasticsearch cluster. The password is specified when you create the cluster. If you forget the password, you can reset it. For more information about the procedure and precautions for resetting a password, seeReset the access password for an Elasticsearch cluster. |
<YourEsIndex>/<OtherEsIndex> | The name of the source or destination index. |
<YourLocalFile> | The on-premises path that is used to store the data to be migrated. Example: Important When you migrate data to your on-premises machine, elasticsearch-dump automatically generates the destination file in the specified path. Therefore, before you migrate data to the on-premises machine, you must make sure that the name of the destination file is unique in the related directory. |
<YourQuery> | The query statement. Example: |
FAQ
Q: What do I do if the errorError: getaddrinfo ENOTFOUND elastic is reported?
A: The HTTP authentication password contains special characters. As a result, a URL parsing error occurs. To resolve this issue, you need to add an HTTP authentication file.
Enter the username and password for accessing the Elasticsearch cluster in the authentication file
auth.ini. Example:user=elasticpassword="Es#123456"NoteThe password must be enclosed in a pair of double quotation marks (").
Run the following command and specify the authentication file by using the
--httpAuthFileparameter:elasticdump --input=http://es-*****.public.elasticsearch.aliyuncs.com:9200/customers --output=/root/customers.json --httpAuthFile=/root/auth.ini --type=settings
Q: What do I do if the errorURIError: URI malformed is reported?
A: The elasticsearch-dump plug-in cannot parse usernames and passwords that contain special characters. If the username or password of your Alibaba Cloud Elasticsearch cluster contains special characters, the system reports the preceding error. You can use one of the following methods to troubleshoot the issue:
Method 1: Remove the special characters in the username or password to change the username or password.
Method 2: Log on to the Kibana console of the cluster, create a user, grant the required permissions to the user, and then use the new user for data migration.