CN111475334A - Maintenance method, device, computer equipment and storage medium for TiDB database - Google Patents

Abstract

The invention discloses a method and a device for maintaining a TiDB database, computer equipment and a storage medium, wherein the method comprises the following steps: the method comprises the steps of collecting data in a TiDB production library through a TiDB-Binlog cluster component to obtain a binlogo file corresponding to the data in the TiDB production library, synchronizing the binlogo file to a TiDB city-wide disaster backup library, synchronizing the binlogo file in the TiDB city-wide disaster backup library to a flash NAS device by adopting a preset timing backup script, and carrying out data reduction processing on the TiDB production library based on the binlogo file stored in the flash NAS device when the abnormality of the TiDB production library is monitored, so that the problem of low backup and reduction speed caused by the fact that the data in the production library are directly stored in a filing tape in a traditional mode is solved, and the maintenance efficiency of the TiDB database is improved.

Description

Translated fromChinese

TiDB数据库的维护方法、装置、计算机设备及存储介质Maintenance method, device, computer equipment and storage medium for TiDB database

技术领域technical field

本发明涉及数据处理领域，尤其涉及一种TiDB数据库的维护方法、装置、计算机设备及存储介质。The invention relates to the field of data processing, in particular to a maintenance method, device, computer equipment and storage medium for a TiDB database.

背景技术Background technique

随着经济的快速发展，越来越多的企业机构业务范围越来越大，用户群体也越来越广，许多企业开始采用分布式的数据库进行数据存储，以便提高业务处理能力，TiDB作为是一种开源分布式关系型的数据库，其安全性较高，且兼容MySQL协议和生态，迁移便捷，运维成本极低，成为一种比较受欢迎的分布式数据库。With the rapid development of the economy, more and more enterprises and institutions have larger and larger business scope and wider user groups. Many enterprises have begun to use distributed databases for data storage in order to improve their business processing capabilities. An open source distributed relational database with high security, compatibility with MySQL protocol and ecology, easy migration, and extremely low operation and maintenance cost, making it a relatively popular distributed database.

数据库进行事务处理时，需要进行数据备份，以便在发生故障时，根据备份文件进行数据恢复还原处理，现有的TiDB数据库，通过归档磁带进行数据备份，归档磁带IO传输能力较弱，而TiDB数据库往往数据量较大，这使得数据备份和还原时间较长，导致数据库维护效率较低。When the database performs transaction processing, it is necessary to perform data backup, so that in the event of a failure, data recovery and restoration processing can be performed according to the backup file. The existing TiDB database uses archive tapes for data backup, and the archive tape IO transmission capability is weak. The amount of data is often large, which makes the data backup and restore time longer, resulting in low database maintenance efficiency.

发明内容SUMMARY OF THE INVENTION

本发明实施例提供一种TiDB数据库的维护方法、装置、计算机设备和存储介质，以提高TiDB数据库的维护效率。Embodiments of the present invention provide a TiDB database maintenance method, device, computer equipment and storage medium, so as to improve the maintenance efficiency of the TiDB database.

为了解决上述技术问题，本申请实施例提供一种TiDB数据库的维护方法，包括：In order to solve the above-mentioned technical problems, an embodiment of the present application provides a method for maintaining a TiDB database, including:

通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到所述TiDB生产库中的数据对应的binlog文件，并将所述binlog文件同步到TiDB同城灾备库；Collect the data in the TiDB production library through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production library, and synchronize the binlog file to the TiDB intra-city disaster recovery library;

采用预设的定时备份脚本，将所述TiDB同城灾备库中的binlog文件同步到闪存NAS设备；Synchronize the binlog file in the TiDB intra-city disaster recovery database to the flash NAS device by using a preset timing backup script;

若监控到TiDB生产库异常，则基于所述闪存NAS设备中存储的binlog文件，对所述TiDB生产库进行数据还原处理。If it is monitored that the TiDB production library is abnormal, data restoration processing is performed on the TiDB production library based on the binlog file stored in the flash NAS device.

可选地，所述TiDB-Binlog集群组件包括Drainer组件与pump组件，所述通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到所述TiDB生产库中的数据对应的binlog文件，并将所述binlog文件同步到TiDB同城灾备库包括：Optionally, the TiDB-Binlog cluster component includes a Drainer component and a pump component, and the data in the TiDB production library is collected through the TiDB-Binlog cluster component, the binlog file corresponding to the data in the TiDB production library is obtained, and the The binlog file synchronized to the TiDB intra-city disaster recovery database includes:

通过所述pump组件收集所述pump组件对应的TiDB节点服务器的binglog文件，作为基础文件，并将所述基础文件发送至所述Drainer组件；Collect the binglog file of the TiDB node server corresponding to the pump component through the pump component as a basic file, and send the basic file to the Drainer component;

通过所述Drainer组件接收所述基础文件并对所述基础文件进行合并，得到合并后的binglog文件；Receive the base file through the Drainer component and merge the base file to obtain a merged binglog file;

将所述合并后的binglog文件同步到所述TiDB同城灾备库。Synchronize the merged binglog file to the TiDB same-city disaster recovery database.

可选地，所述闪存NAS设备包括第一闪存设备和第二闪存设备。Optionally, the flash NAS device includes a first flash device and a second flash device.

可选地，所述采用预设的定时备份脚本，将所述TiDB同城灾备库中的binlog文件同步到闪存NAS设备包括：Optionally, using a preset timed backup script to synchronize the binlog file in the TiDB intra-city disaster recovery library to the flash NAS device includes:

将所述TiDB同城灾备库中的binlog文件移动并保存到所述第一闪存设备中；moving and saving the binlog file in the TiDB intra-city disaster recovery library to the first flash memory device;

定时对所述第一闪存设备和所述第二闪存设备进行增量和全量的差异校验，得到校验结果；Perform incremental and full difference verification on the first flash memory device and the second flash memory device periodically to obtain a verification result;

若所述校验结果为存在差异，则根据所述校验结果中的差异，采用所述第一闪存设备中的数据对所述第二闪存设备中的数据进行更新。If the verification result is that there is a difference, the data in the second flash memory device is updated by using the data in the first flash memory device according to the difference in the verification result.

可选地，所述将所述TiDB同城灾备库中的binlog文件移动并保存到所述第一闪存设备中包括：Optionally, the moving and saving the binlog file in the TiDB intra-city disaster recovery library to the first flash memory device includes:

将所述TiDB同城灾备库中的binlog文件作为待导入数据，并获取所述待导入数据对应的所述第一闪存设备的存储路径；Taking the binlog file in the TiDB intra-city disaster recovery database as the data to be imported, and acquiring the storage path of the first flash memory device corresponding to the data to be imported;

基于所述待导入数据和所述第一闪存设备，确定目标线程的数量；determining the number of target threads based on the data to be imported and the first flash memory device;

基于所述目标线程数量，构建多线程，并采用所述多线程将所述待导入数据同步到所述第一闪存设备的存储路径。Based on the target number of threads, multithreading is constructed, and the multithreading is used to synchronize the data to be imported to the storage path of the first flash memory device.

可选地，其特征在于，所述若监控到TiDB生产库异常，则基于所述闪存NAS设备中存储的binlog文件，对所述TiDB生产库进行数据还原处理包括：Optionally, it is characterized in that, if the TiDB production library is monitored abnormally, based on the binlog file stored in the flash NAS device, performing data restoration processing on the TiDB production library includes:

停止应用服务对所述TiDB生产库的写表处理，并将所述TiDB生产库的数据表中的数据清空；Stop the table writing processing of the TiDB production library by the application service, and clear the data in the data table of the TiDB production library;

从所述闪存NAS设备中获取全备文件和增备文件，作为待还原文件；Obtain full backup files and additional backup files from the flash NAS device as files to be restored;

将所述全备文件载入到TiDB生产库中的数据表中，再将增备文件载入到所述TiDB生产库中的数据表中；Load the full backup file into the data table in the TiDB production library, and then load the supplementary file into the data table in the TiDB production library;

对所述TiDB生产库中的数据表中的数据进行验证，验证通过后，确认数据还原成功，并启动应用服务。The data in the data table in the TiDB production library is verified, and after the verification is passed, it is confirmed that the data is restored successfully, and the application service is started.

为了解决上述技术问题，本申请实施例还提供一种TiDB数据库的维护装置，包括：In order to solve the above-mentioned technical problems, the embodiment of the present application also provides a maintenance device for a TiDB database, including:

数据采集模块，用于通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到所述TiDB生产库中的数据对应的binlog文件，并将所述binlog文件同步到TiDB同城灾备库；a data collection module, used for collecting data in the TiDB production library through the TiDB-Binlog cluster component, obtaining the binlog file corresponding to the data in the TiDB production library, and synchronizing the binlog file to the TiDB intra-city disaster recovery library;

文件同步模块，用于采用预设的定时备份脚本，将所述TiDB同城灾备库中的binlog文件同步到闪存NAS设备；The file synchronization module is used for synchronizing the binlog file in the TiDB intra-city disaster recovery database to the flash NAS device by using a preset timed backup script;

数据恢复模块，用于若监控到TiDB生产库异常，则基于所述闪存NAS设备中存储的binlog文件，对所述TiDB生产库进行数据还原处理。A data recovery module, configured to perform data restoration processing on the TiDB production database based on the binlog file stored in the flash NAS device if an abnormality of the TiDB production database is monitored.

可选地，所述数据采集模块包括：Optionally, the data collection module includes:

采集单元，用于通过所述pump组件收集所述pump组件对应的TiDB节点服务器的binglog文件，作为基础文件，并将所述基础文件发送至所述Drainer组件；a collection unit, configured to collect the binglog file of the TiDB node server corresponding to the pump component through the pump component as a basic file, and send the basic file to the Drainer component;

合并单元，用于通过所述Drainer组件接收所述基础文件并对所述基础文件进行合并，得到合并后的binglog文件；a merging unit, for receiving the base file through the Drainer component and merging the base file to obtain a merged binglog file;

同步单元，用于将所述合并后的binglog文件同步到所述TiDB同城灾备库。A synchronization unit, configured to synchronize the merged binglog file to the TiDB same-city disaster recovery database.

可选地，闪存NAS设备包括第一闪存设备和第二闪存设备，所述文件同步模块包括：Optionally, the flash NAS device includes a first flash memory device and a second flash memory device, and the file synchronization module includes:

同步单元，用于将所述TiDB同城灾备库中的binlog文件移动并保存到所述第一闪存设备中；a synchronization unit, configured to move and save the binlog file in the TiDB intra-city disaster recovery library to the first flash memory device;

校验单元，用于定时对所述第一闪存设备和所述第二闪存设备进行增量和全量的差异校验，得到校验结果；a verification unit, configured to periodically perform incremental and full difference verification on the first flash memory device and the second flash memory device to obtain a verification result;

更新单元，用于若所述校验结果为存在差异，则根据所述校验结果中的差异，采用所述第一闪存设备中的数据对所述第二闪存设备中的数据进行更新。an updating unit, configured to update data in the second flash memory device by using the data in the first flash memory device according to the difference in the verification result if the verification result is a difference.

可选地，同步单元包括：Optionally, the synchronization unit includes:

路径获取子单元，用于将所述TiDB同城灾备库中的binlog文件作为待导入数据，并获取所述待导入数据对应的所述第一闪存设备的存储路径；A path acquisition subunit, configured to use the binlog file in the TiDB intra-city disaster recovery database as the data to be imported, and to acquire the storage path of the first flash memory device corresponding to the data to be imported;

线程数量确定子单元，用于基于所述待导入数据和所述第一闪存设备，确定目标线程的数量；a thread number determination subunit, configured to determine the number of target threads based on the data to be imported and the first flash memory device;

多线程同步子单元，用于基于所述目标线程数量，构建多线程，并采用所述多线程将所述待导入数据同步到所述第一闪存设备的存储路径。The multi-thread synchronization subunit is configured to construct multi-threads based on the target number of threads, and use the multi-threads to synchronize the data to be imported to the storage path of the first flash memory device.

可选地，数据恢复模块包括：Optionally, the data recovery module includes:

数据清理单元，用于停止应用服务对所述TiDB生产库的写表处理，并将所述TiDB生产库的数据表中的数据清空；A data cleaning unit, used to stop the table writing processing of the TiDB production library by the application service, and clear the data in the data table of the TiDB production library;

待还原文件确定单元，用于从所述闪存NAS设备中获取全备文件和增备文件，作为待还原文件；A to-be-restored file determination unit, configured to obtain a full backup file and an additional backup file from the flash NAS device as the to-be-restored file;

文件还原单元，用于将所述全备文件载入到TiDB生产库中的数据表中，再将增备文件载入到所述TiDB生产库中的数据表中；A file restoration unit, configured to load the full backup file into the data table in the TiDB production library, and then load the additional backup file into the data table in the TiDB production library;

数据验证单元，用于对所述TiDB生产库中的数据表中的数据进行验证，验证通过后，确认数据还原成功，并启动应用服务。The data verification unit is used to verify the data in the data table in the TiDB production library, and after the verification is passed, confirm that the data is restored successfully, and start the application service.

为了解决上述技术问题，本申请实施例还提供一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，所述处理器执行所述计算机程序时实现上述TiDB数据库的维护方法的步骤。In order to solve the above technical problem, an embodiment of the present application further provides a computer device, including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor executes the computer The steps of implementing the maintenance method of the above TiDB database during the program.

为了解决上述技术问题，本申请实施例还提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时实现上述TiDB数据库的维护方法的步骤。In order to solve the above technical problem, an embodiment of the present application further provides a computer-readable storage medium, where the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, implements the steps of the above-mentioned TiDB database maintenance method .

本发明实施例提供的TiDB数据库的维护方法、装置、计算机设备及存储介质，通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到TiDB生产库中的数据对应的binlog文件，并将binlog文件同步到TiDB同城灾备库，再采用预设的定时备份脚本，将TiDB同城灾备库中的binlog文件同步到闪存NAS设备，在监控到TiDB生产库异常，基于闪存NAS设备中存储的binlog文件，对TiDB生产库进行数据还原处理，避免传统方式采用将生产库中的数据直接保存在归档磁带中导致备份和还原速度慢的问题，提高了数据库维护的效率。The TiDB database maintenance method, device, computer equipment and storage medium provided by the embodiments of the present invention collect data in the TiDB production library through the TiDB-Binlog cluster component, obtain a binlog file corresponding to the data in the TiDB production library, and convert the binlog file to the Synchronize to the TiDB intra-city disaster recovery database, and then use the preset scheduled backup script to synchronize the binlog files in the TiDB intra-city disaster recovery database to the flash NAS device. When the TiDB production database is monitored abnormally, it is based on the binlog file stored in the flash NAS device. , perform data restoration processing on the TiDB production library, avoid the problem of slow backup and restore speed caused by the traditional method of directly saving the data in the production library in the archive tape, and improve the efficiency of database maintenance.

附图说明Description of drawings

为了更清楚地说明本发明实施例的技术方案，下面将对本发明实施例的描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。In order to illustrate the technical solutions of the embodiments of the present invention more clearly, the following briefly introduces the drawings that are used in the description of the embodiments of the present invention. Obviously, the drawings in the following description are only some embodiments of the present invention. , for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative labor.

图1是本申请可以应用于其中的示例性系统架构图；FIG. 1 is an exemplary system architecture diagram to which the present application can be applied;

图2是本申请的TiDB数据库的维护方法的一个实施例的流程图；Fig. 2 is the flow chart of an embodiment of the maintenance method of the TiDB database of the present application;

图3是根据本申请的TiDB数据库的维护装置的一个实施例的结构示意图；3 is a schematic structural diagram of an embodiment of a maintenance device for a TiDB database according to the present application;

图4是根据本申请的计算机设备的一个实施例的结构示意图。FIG. 4 is a schematic structural diagram of an embodiment of a computer device according to the present application.

具体实施方式Detailed ways

除非另有定义，本文所使用的所有的技术和科学术语与属于本申请的技术领域的技术人员通常理解的含义相同；本文中在申请的说明书中所使用的术语只是为了描述具体的实施例的目的，不是旨在于限制本申请；本申请的说明书和权利要求书及上述附图说明中的术语“包括”和“具有”以及它们的任何变形，意图在于覆盖不排他的包含。本申请的说明书和权利要求书或上述附图中的术语“第一”、“第二”等是用于区别不同对象，而不是用于描述特定顺序。Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the technical field of this application; the terms used herein in the specification of the application are for the purpose of describing specific embodiments only It is not intended to limit the application; the terms "comprising" and "having" and any variations thereof in the description and claims of this application and the above description of the drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the description and claims of the present application or the above drawings are used to distinguish different objects, rather than to describe a specific order.

在本文中提及“实施例”意味着，结合实施例描述的特定特征、结构或特性可以包含在本申请的至少一个实施例中。在说明书中的各个位置出现该短语并不一定均是指相同的实施例，也不是与其它实施例互斥的独立的或备选的实施例。本领域技术人员显式地和隐式地理解的是，本文所描述的实施例可以与其它实施例相结合。Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor a separate or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

请参阅图1，如图1所示，系统架构100可以包括终端设备101、102、103，网络104和服务器105。网络104用以在终端设备101、102、103和服务器105之间提供通信链路的介质。网络104可以包括各种连接类型，例如有线、无线通信链路或者光纤电缆等等。Please refer to FIG. 1 . As shown in FIG. 1 , thesystem architecture 100 may includeterminal devices 101 , 102 , and 103 , anetwork 104 and aserver 105 . Thenetwork 104 is a medium used to provide a communication link between theterminal devices 101 , 102 , 103 and theserver 105 . Thenetwork 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

用户可以使用终端设备101、102、103通过网络104与服务器105交互，以接收或发送消息等。The user can use theterminal devices 101, 102, 103 to interact with theserver 105 through thenetwork 104 to receive or send messages and the like.

终端设备101、102、103可以是具有显示屏并且支持网页浏览的各种电子设备，包括但不限于智能手机、平板电脑、电子书阅读器、MP3播放器(Moving Picture E界面显示perts Group Audio Layer III，动态影像专家压缩标准音频层面3)、MP4(Moving PictureE界面显示perts Group Audio Layer IV，动态影像专家压缩标准音频层面4)播放器、膝上型便携计算机和台式计算机等等。Theterminal devices 101, 102, and 103 can be various electronic devices that have a display screen and support web browsing, including but not limited to smart phones, tablet computers, e-book readers, and MP3 players (Moving Picture E interface displays perts Group Audio Layer). III, Moving Picture Expert Compression Standard Audio Layer 3), MP4 (Moving PictureE interface displays perts Group Audio Layer IV, Moving Picture Expert Compression Standard Audio Layer 4) players, laptops and desktop computers, etc.

服务器105可以是提供各种服务的服务器，例如对终端设备101、102、103上显示的页面提供支持的后台服务器。Theserver 105 may be a server that provides various services, such as a background server that provides support for the pages displayed on theterminal devices 101 , 102 , and 103 .

需要说明的是，本申请实施例所提供的TiDB数据库的维护方法由服务器执行，相应地，TiDB数据库的维护装置设置于服务器中。It should be noted that the maintenance method of the TiDB database provided by the embodiment of the present application is performed by the server, and correspondingly, the maintenance device of the TiDB database is set in the server.

应该理解，图1中的终端设备、网络和服务器的数目仅仅是示意性的。根据实现需要，可以具有任意数目的终端设备、网络和服务器，本申请实施例中的终端设备101、102、103具体可以对应的是实际生产中的应用系统。It should be understood that the numbers of terminal devices, networks and servers in FIG. 1 are merely illustrative. According to implementation requirements, there may be any number of terminal devices, networks, and servers, and theterminal devices 101 , 102 , and 103 in the embodiments of the present application may specifically correspond to application systems in actual production.

请参阅图2，图2示出本发明实施例提供的一种TiDB数据库的维护方法，以该方法应用在图1中的服务端为例进行说明，详述如下：Please refer to FIG. 2. FIG. 2 shows a TiDB database maintenance method provided by an embodiment of the present invention. The method is applied to the server in FIG. 1 as an example for description. The details are as follows:

S201：通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到TiDB生产库中的数据对应的binlog文件，并将binlog文件同步到TiDB同城灾备库。S201: Collect the data in the TiDB production library through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production library, and synchronize the binlog file to the TiDB intra-city disaster recovery library.

其中，TiDB生产库是用于生产环境，并基于TiDB开源分布式关系型的数据库，TiDB是一款定位于在线事务处理/在线分析处理(HTAP：Hybrid Transactional/AnalyticalProcessing)的融合型数据库产品，实现了一键水平伸缩，强一致性的多副本数据安全，分布式事务，实时OLAP等重要特性。同时兼容MySQL协议和生态，迁移便捷，运维成本极低。Among them, the TiDB production library is used in the production environment and is based on the open source distributed relational database of TiDB. TiDB is an integrated database product positioned for online transaction processing/online analytical processing (HTAP: Hybrid Transactional/Analytical Processing). One-click horizontal scaling, strongly consistent multi-copy data security, distributed transactions, real-time OLAP and other important features. It is compatible with the MySQL protocol and ecology at the same time, with convenient migration and extremely low operation and maintenance costs.

具体地，本实施例中，包含若干个TiDB节点服务器，TiDB节点服务器在运行中，会进行数据更新(包括但不限于增加、删除、修改、查询和备份等)，通过TiDB-Binlog集群组件，获取每个TiDB节点服务器中更新的数据，并根据这些更新的数据生成对应的binlog文件，并将binlog文件同步到TiDB同城灾备库。Specifically, this embodiment includes several TiDB node servers. The TiDB node server will perform data updates (including but not limited to adding, deleting, modifying, querying, and backing up, etc.) during operation. Through the TiDB-Binlog cluster component, Obtain the updated data in each TiDB node server, generate the corresponding binlog file based on the updated data, and synchronize the binlog file to the TiDB intra-city disaster recovery database.

其中，TiDB同城灾备库是指基于TiDB的同一地区的备份数据库。The TiDB intra-city disaster recovery database refers to the TiDB-based backup database in the same region.

其中，binlog文件是一个二进制格式的日志文件，用于记录用户对数据库更新的SQL语句信息，例如更改数据库表和更改内容的SQL语句都会记录到binlog里，但是对库表等内容的查询不会记录。默认情况下，binlog文件是二进制格式的，不能使用查看文本工具的命令(比如，cat，vi等)查看，而使用mysqlbinlog解析查看。Among them, the binlog file is a log file in binary format, which is used to record the SQL statement information that the user updates to the database. For example, the SQL statement that changes the database table and the content will be recorded in the binlog, but the query of the database table and other content will not be Record. By default, the binlog file is in binary format, which cannot be viewed using commands that view text tools (such as cat, vi, etc.), but parsed and viewed using mysqlbinlog.

需要说明的是，当有数据写入到数据库时，还会同时把更新的SQL语句写入到对应的binlog文件里，使用dump备份时，只是对一段时间的数据进行全备。It should be noted that when data is written to the database, the updated SQL statement will be written to the corresponding binlog file at the same time. When using dump backup, only the data for a period of time is fully prepared.

S202：采用预设的定时备份脚本，将TiDB同城灾备库中的binlog文件同步到闪存NAS设备。S202: Use a preset scheduled backup script to synchronize the binlog file in the TiDB intra-city disaster recovery database to the flash NAS device.

现有的外挂存储根据连接的方式分为：直连式存储(Direct-Attached Storage，简称DAS)和网络化存储(Fabric-Attached Storage，简称FAS)；网络化存储根据传输协议又分为：网络接入存储(Network-Attached Storage，简称NAS)和存储区域网络(StorageArea Network，简称SAN)。The existing external storage is divided into: Direct-Attached Storage (DAS for short) and Fabric-Attached Storage (FAS for short) according to the connection method; network storage is divided into: network storage according to the transmission protocol Access storage (Network-Attached Storage, referred to as NAS) and storage area network (StorageArea Network, referred to as SAN).

其中，闪存NAS设备是连接在网络上，具备资料存储功能的装置，因此也称为“网络存储器”。它是一种特殊的专用数据存储服务器，包括存储器件(例如磁盘阵列、CD/DVD驱动器、磁带驱动器或可移动的存储介质)和内嵌系统软件，可提供跨平台文件共享功能。NAS通常在一个LAN上占有自己的节点，无需应用服务器的干预，允许用户在网络上存取数据，在这种配置中，NAS集中管理和处理网络上的所有数据，将负载从应用或企业服务器上卸载下来，有效降低总拥有成本，保护用户投资。NAS本身能够支持多种协议(如NFS、CIFS、FTP、HTTP等)，而且能够支持各种操作系统。通过任何一台工作站，采用IE或Netscape浏览器就可以对NAS设备进行直观方便的管理。其成本远远低于使用服务器存储，而效率却远远高于后者。Among them, the flash NAS device is a device connected to the network and has the function of data storage, so it is also called "network storage". It is a special dedicated data storage server that includes storage devices (such as disk arrays, CD/DVD drives, tape drives, or removable storage media) and embedded system software that provides cross-platform file sharing capabilities. NAS usually has its own node on a LAN, without the intervention of the application server, allowing users to access data on the network. In this configuration, the NAS centrally manages and processes all data on the network, offloading the load from the application or enterprise server. It can effectively reduce the total cost of ownership and protect user investment. NAS itself can support a variety of protocols (such as NFS, CIFS, FTP, HTTP, etc.), and can support various operating systems. Through any workstation, the NAS device can be managed intuitively and conveniently by using IE or Netscape browser. Its cost is much lower than using server storage, and its efficiency is much higher.

NAS设备是真正即插即用的产品，因而又被称为闪存NAS设备，一般支持多计算机平台，用户通过网络支持协议可进入相同的文档，因而NAS设备无需改造即可用于混合Unix/Windows NT局域网内，同时NAS设备的应用非常灵活，相对于其他外挂存储方式，采用闪存NAS设备的一个关键性问题，是备份过程中的带宽消耗。与将备份数据流从LAN中转移出去的存储区域网(SAN)不同，NAS仍使用网络进行备份和恢复，因而，可采用多线程的方式进行数据备份和恢复，提高数据恢复和备份的效率。NAS devices are truly plug-and-play products, so they are also called flash NAS devices. They generally support multiple computer platforms, and users can access the same files through network support protocols. Therefore, NAS devices can be used for mixed Unix/Windows NT without modification. In the local area network, the application of NAS devices is very flexible. Compared with other external storage methods, a key problem of using flash NAS devices is the bandwidth consumption during the backup process. Unlike the storage area network (SAN), which transfers the backup data stream from the LAN, NAS still uses the network for backup and recovery. Therefore, data backup and recovery can be performed in a multi-threaded manner to improve the efficiency of data recovery and backup.

其中，预设的定时备份脚本用于定时对binlog文件进行同步，例如，通过定时备份脚本，在每天2点将drainer中，创建时间为前一天内的binlog文件转移到闪存NAS设备上，预设的定时备份脚本的类型包括但不限于：shell脚本、JavaScript脚本、Lua脚本和python脚本等。Among them, the preset scheduled backup script is used to synchronize the binlog files regularly. For example, through the scheduled backup script, at 2:00 every day, the binlog files in the drainer with the creation time of the previous day are transferred to the flash NAS device. The types of scheduled backup scripts include but are not limited to: shell scripts, JavaScript scripts, Lua scripts, and python scripts.

现有的的TiDB数据库备份方法，其在生产库发生故障时，需要通过归档磁带恢复全备和增备文件到本地服务器后再导入数据库，由于需要从归档磁带恢复的数据库全备文件一般都比较大(数据库规模大小不一，一般几十G到几个T)，磁带IO能力较弱，需要耗费在获取恢复文件的时间比较长，导致应用停止时间延长，给业务系统的持续运营造成严重影响，而本实施例中，采用闪存NAS设备进行数据的备份，能极大提高数据备份恢复速度，提高TiDB数据库的维护效率。In the existing TiDB database backup method, when the production database fails, it is necessary to restore the full backup and additional backup files through the archive tape to the local server and then import the database. Because the database full backup files that need to be restored from the archive tape are generally more Large (database sizes vary, generally tens of G to several T), tape IO capability is weak, and it takes a long time to obtain recovery files, resulting in prolonged application stop time and serious impact on the continuous operation of the business system. However, in this embodiment, using a flash NAS device for data backup can greatly improve the speed of data backup and recovery and improve the maintenance efficiency of the TiDB database.

值得说明的是，闪存NAS设备不适合永久进行数据的存储，在本实施例中，在将binlog文件移动并保存到闪存NAS设备后，进而将闪存NAS设备中存储的数据同步到归档磁带中，以便数据可以被脱机保存并读取。It is worth noting that the flash NAS device is not suitable for permanent data storage. In this embodiment, after the binlog file is moved and saved to the flash NAS device, the data stored in the flash NAS device is further synchronized to the archive tape. so that the data can be saved and read offline.

其中，归档磁带又称为磁带存储器(magnetic tape storage)，是一种以磁带为存储介质，由磁带机及其控制器组成的存储设备，是计算机的一种辅助存储器。磁带机由磁带传动机构和磁头等组成，能驱动磁带相对磁头运动，用磁头进行电磁转换，在磁带上顺序地记录或读出数据。磁带存储器是计算机外围设备之一。磁带控制器是中央处理器在磁带机上存取数据用的控制电路装置。磁带存储器以顺序方式存取数据。存储数据的磁带可脱机保存和互换读出。Among them, the archive tape is also called magnetic tape storage (magnetic tape storage), which is a storage device composed of a tape drive and its controller, and is a kind of auxiliary storage of a computer. The tape drive is composed of a tape drive mechanism and a magnetic head, which can drive the magnetic tape to move relative to the magnetic head, perform electromagnetic conversion with the magnetic head, and sequentially record or read data on the magnetic tape. Tape storage is one of computer peripherals. The tape controller is a control circuit device used by the central processing unit to access data on the tape drive. Tape storage accesses data in a sequential manner. The tapes that store the data can be saved offline and read out interchangeably.

S203：若监控到TiDB生产库异常，则基于闪存NAS设备中存储的binlog文件，对TiDB生产库进行数据还原处理。S203: If it is monitored that the TiDB production database is abnormal, perform data restoration processing on the TiDB production database based on the binlog file stored in the flash NAS device.

具体地，服务端有配置应用日志监控和数据库性能指标监控，这些监控实时对在TiDB生产库的各项数据进行监控，在监控到TiDB生产库中的部分数据超出正常预设范围时，确认TiDB生产库异常，此时，将通过邮件或者电话进行告警提示，并通过闪存NAS设备中存储的binlog文件，进行数据还原处理。Specifically, the server is configured with application log monitoring and database performance indicator monitoring. These monitors monitor various data in the TiDB production library in real time. When some data in the TiDB production library is monitored to exceed the normal preset range, confirm that TiDB If the production library is abnormal, at this time, an alarm will be sent by email or phone, and the data will be restored through the binlog file stored in the flash NAS device.

其中，通过闪存NAS设备中存储的binlog文件，进行数据还原处理的具体过程，可参考后续实施例的描述，为避免重复，此处不再赘述。For the specific process of performing data restoration processing by using the binlog file stored in the flash memory NAS device, reference may be made to the description of the subsequent embodiments, and to avoid repetition, it will not be repeated here.

在本实施例中，通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到TiDB生产库中的数据对应的binlog文件，并将binlog文件同步到TiDB同城灾备库，再采用预设的定时备份脚本，将TiDB同城灾备库中的binlog文件同步到闪存NAS设备，在监控到TiDB生产库异常，基于闪存NAS设备中存储的binlog文件，对TiDB生产库进行数据还原处理，避免传统方式采用将生产库中的数据直接保存在归档磁带中导致备份和还原速度慢的问题，提高了数据库维护的效率。In this embodiment, the data in the TiDB production library is collected through the TiDB-Binlog cluster component, the binlog file corresponding to the data in the TiDB production library is obtained, the binlog file is synchronized to the TiDB intra-city disaster recovery library, and then the preset timing is used. The backup script synchronizes the binlog files in the TiDB intra-city disaster recovery database to the flash NAS device. When an abnormality is detected in the TiDB production database, data restoration is performed on the TiDB production database based on the binlog files stored in the flash NAS device, avoiding the use of traditional methods. The data in the production library is directly stored in the archive tape, which leads to the problem of slow backup and restore speed, which improves the efficiency of database maintenance.

在本实施例的一些可选的实现方式中，步骤S201中，TiDB-Binlog集群组件包括Drainer组件与pump组件，通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到TiDB生产库中的数据对应的binlog文件，并将binlog文件同步到TiDB同城灾备库包括：In some optional implementations of this embodiment, in step S201, the TiDB-Binlog cluster component includes a Drainer component and a pump component, and the TiDB-Binlog cluster component collects data in the TiDB production library to obtain the data in the TiDB production library The corresponding binlog files and the binlog files to be synchronized to the TiDB intra-city disaster recovery database include:

通过pump组件收集pump组件对应的TiDB节点服务器的binglog文件，作为基础文件，并将基础文件发送至Drainer组件；Collect the binglog file of the TiDB node server corresponding to the pump component through the pump component as the basic file, and send the basic file to the Drainer component;

通过Drainer组件接收基础文件并对基础文件进行合并，得到合并后的binglog文件；Receive the base file through the Drainer component and merge the base files to obtain the merged binglog file;

将合并后的binglog文件同步到TiDB同城灾备库。Synchronize the merged binglog file to the TiDB same-city disaster recovery database.

具体地，服务端包含多个TiDB节点服务器，每个TiDB节点服务器对应一个pump组件，TiDB节点服务器生成binglog文件后，pump组件采集该binglog文件作为基础文件，并将基础组件发送给Drainer组件，Drainer组件对接收到的所有基础文件进行合并，合并后的binglog文件，并将该文件同步到TiDB同城灾备库。Specifically, the server includes multiple TiDB node servers, and each TiDB node server corresponds to a pump component. After the TiDB node server generates a binglog file, the pump component collects the binglog file as a basic file, and sends the basic component to the Drainer component. Drainer The component merges all the basic files received, the merged binglog file, and synchronizes the file to the TiDB intra-city disaster recovery database.

容易理解地，binglog文件中的数据格式为二进制，使用binglog文件进行同步备份，能极大提高备份效率。It is easy to understand that the data format in the binglog file is binary. Using the binglog file for synchronous backup can greatly improve the backup efficiency.

其中，pump组件生成一个守护进程，在每个TiDB节点服务器的后台运行，其主要功能是实时记录TiDB节点服务器在执行数据处理时产生的Binlog文件，并顺序写入相应的存储空间。Among them, the pump component generates a daemon process, which runs in the background of each TiDB node server. Its main function is to record the Binlog files generated by the TiDB node server during data processing in real time, and sequentially write them into the corresponding storage space.

其中，Drainer组件与pump组件进行交互，收集Binlog文件。Among them, the Drainer component interacts with the pump component to collect Binlog files.

在本实施例中，通过pump组件收集pump组件对应的TiDB节点服务器的binglog文件，作为基础文件，并将基础文件发送至Drainer组件，通过Drainer组件接收基础文件并对基础文件进行合并，得到合并后的binglog文件，再将合并后的binglog文件同步到TiDB同城灾备库，实现binglog文件快速同步到TiDB同城灾备库，提高了binglog文件快速同步到TiDB同城灾备库的效率。In this embodiment, the binglog file of the TiDB node server corresponding to the pump component is collected by the pump component as the basic file, and the basic file is sent to the Drainer component, and the basic file is received and merged by the Drainer component. binglog files, and then synchronize the merged binglog files to the TiDB intra-city disaster recovery database, so that the binglog files can be quickly synchronized to the TiDB intra-city disaster recovery database, which improves the efficiency of quickly synchronizing binglog files to the TiDB intra-city disaster recovery database.

在本实施例的一些可选的实现方式中，步骤S202中，闪存NAS设备包括第一闪存设备和第二闪存设备，采用预设的定时备份脚本，将TiDB同城灾备库中的binlog文件同步到闪存NAS设备包括：In some optional implementations of this embodiment, in step S202, the flash NAS device includes a first flash memory device and a second flash memory device, and a preset timing backup script is used to synchronize the binlog files in the TiDB intra-city disaster recovery database To flash NAS devices include:

将TiDB同城灾备库中的binlog文件移动并保存到第一闪存设备中；Move and save the binlog file in the TiDB intra-city disaster recovery database to the first flash memory device;

定时对第一闪存设备和第二闪存设备进行增量和全量的差异校验，得到校验结果；Perform incremental and full difference verification on the first flash memory device and the second flash memory device regularly to obtain a verification result;

若校验结果为存在差异，则根据校验结果中的差异，采用第一闪存设备中的数据对第二闪存设备中的数据进行更新。If the verification result is that there is a difference, then according to the difference in the verification result, the data in the first flash memory device is used to update the data in the second flash memory device.

具体地，本实施例中的闪存NAS设备包括第一闪存设备和第二闪存设备，第一闪存设备为主存储设备，第二闪存设备为备用存储设备，且第二闪存设备架设在于TiDB生产库相同地区。在出现网络异常中断情况，采用与TiDB生产库的同一地域网络环境中的备用存储设备，保障在跨地域网络访问不通的情况下，也能从同城的备用存储设备上获取数据库备份文件，快速恢复数据库。Specifically, the flash NAS device in this embodiment includes a first flash memory device and a second flash memory device, the first flash memory device is a primary storage device, the second flash memory device is a backup storage device, and the second flash memory device is built in the TiDB production library same area. In the event of an abnormal network interruption, a backup storage device in the same geographical network environment as the TiDB production library is used to ensure that the database backup files can be obtained from the backup storage device in the same city for quick recovery even if the cross-regional network access is blocked. database.

进一步地，将TiDB同城灾备库中的binlog文件移动并保存到第一闪存设备中，定时对第一闪存设备和第二闪存设备进行增量和全量的差异校验，得到校验结果，在校验结果为存在差异时，根据校验结果中的差异，采用第一闪存设备中的数据对第二闪存设备中的数据进行更新，快速实现第一闪存设备和第二闪存设备的同步更新。Further, move and save the binlog file in the TiDB intra-city disaster recovery database to the first flash memory device, regularly perform incremental and full difference verification on the first flash memory device and the second flash memory device, and obtain the verification result, and then When the verification result is a discrepancy, according to the discrepancy in the verification result, the data in the first flash memory device is used to update the data in the second flash memory device, so as to quickly realize the synchronous update of the first flash memory device and the second flash memory device.

其中，增量是指最近一次更新的数据，全量是指所有数据。Among them, the increment refers to the last updated data, and the full refers to all the data.

其中，定时对第一闪存设备和第二闪存设备进行增量和全量的差异校验，得到校验结果包括：通过定时脚本比较第一闪存设备和第二闪存设备全量和增量文件差异，主要比较的点包括但不限于：文件大小，文件修改时间以及文件创建时间等。具体比较方式可采用字符串匹配的方式。Wherein, performing incremental and full difference verification on the first flash memory device and the second flash memory device regularly, and obtaining the verification result includes: comparing the full and incremental file differences between the first flash memory device and the second flash memory device through a timing script, mainly The points of comparison include, but are not limited to: file size, file modification time, and file creation time, etc. The specific comparison method can use the method of string matching.

在本实施例中，将TiDB同城灾备库中的binlog文件移动并保存到第一闪存设备中，定时对第一闪存设备和第二闪存设备进行增量和全量的差异校验，得到校验结果，在校验结果为存在差异时，根据校验结果中的差异，采用第一闪存设备中的数据对第二闪存设备中的数据进行更新，实现第一闪存设备和第二闪存设备的快速同步更新。In this embodiment, the binlog file in the TiDB intra-city disaster recovery database is moved and saved to the first flash memory device, and incremental and full difference verification is performed on the first flash memory device and the second flash memory device periodically to obtain the verification results. As a result, when there is a difference in the verification result, according to the difference in the verification result, the data in the first flash memory device is used to update the data in the second flash memory device, so as to realize the fast speed between the first flash memory device and the second flash memory device. Synchronization Update.

在本实施例的一些可选的实现方式中，将TiDB同城灾备库中的binlog文件移动并保存到第一闪存设备中包括：In some optional implementations of this embodiment, moving and saving the binlog file in the TiDB intra-city disaster recovery database to the first flash memory device includes:

将TiDB同城灾备库中的binlog文件作为待导入数据，并获取待导入数据对应的第一闪存设备的存储路径；Use the binlog file in the TiDB intra-city disaster recovery database as the data to be imported, and obtain the storage path of the first flash memory device corresponding to the data to be imported;

基于待导入数据和第一闪存设备，确定目标线程的数量；determining the number of target threads based on the data to be imported and the first flash memory device;

基于目标线程数量，构建多线程，并采用多线程将待导入数据同步到第一闪存设备的存储路径。Based on the number of target threads, multithreading is constructed, and the multithreading is used to synchronize the data to be imported to the storage path of the first flash memory device.

具体地，每个TiDB节点服务器在第一闪存设备中对应有预设的存储路径，将TiDB同城灾备库中的binlog文件作为待导入数据，并根据待导入数据对应的TiDB节点服务器，确定待导入数据对应的第一闪存设备的存储路径，并根据导入数据和第一闪存设备，确定目标线程的数量，并构建该数量的多线程进行数据同步。Specifically, each TiDB node server corresponds to a preset storage path in the first flash memory device, takes the binlog file in the TiDB intra-city disaster recovery database as the data to be imported, and determines the to-be-imported data according to the TiDB node server corresponding to the data to be imported The storage path of the first flash memory device corresponding to the imported data is imported, and the number of target threads is determined according to the imported data and the first flash memory device, and a multi-thread of the number is constructed to perform data synchronization.

其中，基于待导入数据和第一闪存设备，确定目标线程的数量，具体过程为：根据主机CPU核数设置，最开始设置最小的线程数，再慢慢成倍数放大，同时观察主机IO和网络负载，根据主机负载，动态进行导入线程数量的确定。Among them, based on the data to be imported and the first flash memory device, the number of target threads is determined. The specific process is: according to the number of CPU cores of the host, set the minimum number of threads at the beginning, and then slowly multiply it, and observe the host IO and network at the same time. Load, according to the host load, dynamically determine the number of imported threads.

在本实施例中，动态确定线程数量，并采用多线程的方式，将binlog文件同步到第一闪存设备，有利于节约系统资源，同时提高数据备份效率。In this embodiment, the number of threads is dynamically determined, and the binlog file is synchronized to the first flash memory device in a multi-thread manner, which is beneficial to saving system resources and improving data backup efficiency at the same time.

在本实施例的一些可选的实现方式中，步骤S203中，若监控到TiDB生产库异常，则基于闪存NAS设备中存储的binlog文件，对TiDB生产库进行数据还原处理包括：In some optional implementations of this embodiment, in step S203, if an abnormality of the TiDB production library is monitored, based on the binlog file stored in the flash NAS device, performing data restoration processing on the TiDB production library includes:

停止应用服务对TiDB生产库的写表处理，并将TiDB生产库的数据表中的数据清空；Stop the table writing processing of the TiDB production database by the application service, and clear the data in the data table of the TiDB production database;

从闪存NAS设备中获取全备文件和增备文件，作为待还原文件；Obtain full backup files and additional backup files from flash NAS devices as files to be restored;

将全备文件载入到TiDB生产库中的数据表中，再将增备文件载入到TiDB生产库中的数据表中；Load the full backup file into the data table in the TiDB production database, and then load the additional backup file into the data table in the TiDB production database;

对TiDB生产库中的数据表中的数据进行验证，验证通过后，确认数据还原成功，并启动应用服务。Verify the data in the data table in the TiDB production library. After the verification is passed, confirm that the data is restored successfully and start the application service.

具体地，将闪存NAS设备中存储的所有数据对应的文件作为全本文件，将最后一次备份的数据对应的文件作为赠备文件，在监控到TiDB生产库异常时，停止应用服务对TiDB生产库的写表处理，并将TiDB生产库的数据表中的数据清空，再讲全备文件和增备文件作为待还原文件，并使用待还原文件对TiDB生产库中的数据表进行数据还原，并在数据还原完成后对TiDB生产库中的数据表中的数据进行验证，验证通过后，确认数据还原成功，并启动应用服务。Specifically, the files corresponding to all the data stored in the flash NAS device are used as the full files, and the files corresponding to the last backup data are used as the spare files. When the abnormality of the TiDB production library is monitored, the application service is stopped to perform the operation on the TiDB production library. Write table processing, clear the data in the data table of the TiDB production database, and then use the full backup file and the additional backup file as the files to be restored, and use the to-be-restored file to restore the data table in the TiDB production database, and After the data restoration is completed, verify the data in the data table in the TiDB production library. After the verification is passed, confirm that the data is restored successfully and start the application service.

在本实施例中，在监控到TiDB生产库异常时，对TiDB生产库的数据表停止读写处理，并从闪存NAS设备中获取待还原文件还原到TiDB生产库的数据表中，并对还原后TiDB生产库中的数据表中的数据进行验证，验证通过后启用应用服务，实现在TiDB生产库异常时，进行自动快速还原，提高数据库还原效率。In this embodiment, when the abnormality of the TiDB production database is monitored, the read and write processing of the data table of the TiDB production database is stopped, and the files to be restored are obtained from the flash NAS device and restored to the data table of the TiDB production database, and the restoration process is performed. After the data in the data table in the TiDB production database is verified, the application service is enabled after the verification is passed, so that when the TiDB production database is abnormal, automatic and fast restoration can be performed to improve the efficiency of database restoration.

应理解，上述实施例中各步骤的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本发明实施例的实施过程构成任何限定。It should be understood that the size of the sequence numbers of the steps in the above embodiments does not mean the sequence of execution, and the execution sequence of each process should be determined by its functions and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

图3示出与上述实施例TiDB数据库的维护方法一一对应的TiDB数据库的维护装置的原理框图。如图3所示，该TiDB数据库的维护装置包括数据采集模块31、文件同步模块32和数据恢复模块33。各功能模块详细说明如下：FIG. 3 shows a principle block diagram of the maintenance device of the TiDB database corresponding to the maintenance method of the TiDB database in the above-mentioned embodiment. As shown in FIG. 3 , the maintenance device of the TiDB database includes adata acquisition module 31 , afile synchronization module 32 and adata recovery module 33 . The detailed description of each functional module is as follows:

数据采集模块31，用于通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到TiDB生产库中的数据对应的binlog文件，并将binlog文件同步到TiDB同城灾备库；Thedata collection module 31 is used to collect the data in the TiDB production library through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production library, and synchronize the binlog file to the TiDB intra-city disaster recovery library;

文件同步模块32，用于采用预设的定时备份脚本，将TiDB同城灾备库中的binlog文件同步到闪存NAS设备；Thefile synchronization module 32 is used for synchronizing the binlog file in the TiDB intra-city disaster recovery database to the flash NAS device by using a preset timed backup script;

数据恢复模块33，用于若监控到TiDB生产库异常，则基于闪存NAS设备中存储的binlog文件，对TiDB生产库进行数据还原处理。Thedata recovery module 33 is configured to perform data restoration processing on the TiDB production database based on the binlog file stored in the flash NAS device if an abnormality of the TiDB production database is monitored.

可选地，数据采集模块31包括：Optionally, thedata collection module 31 includes:

采集单元，用于通过pump组件收集pump组件对应的TiDB节点服务器的binglog文件，作为基础文件，并将基础文件发送至Drainer组件；The collection unit is used to collect the binglog file of the TiDB node server corresponding to the pump component through the pump component as the basic file, and send the basic file to the Drainer component;

合并单元，用于通过Drainer组件接收基础文件并对基础文件进行合并，得到合并后的binglog文件；The merging unit is used to receive the base files through the Drainer component and merge the base files to obtain the merged binglog file;

同步单元，用于将合并后的binglog文件同步到TiDB同城灾备库。The synchronization unit is used to synchronize the merged binglog file to the TiDB intra-city disaster recovery database.

可选地，闪存NAS设备包括第一闪存设备和第二闪存设备，文件同步模块32包括：Optionally, the flash NAS device includes a first flash memory device and a second flash memory device, and thefile synchronization module 32 includes:

同步单元，用于将TiDB同城灾备库中的binlog文件移动并保存到第一闪存设备中；The synchronization unit is used to move and save the binlog file in the TiDB intra-city disaster recovery database to the first flash memory device;

校验单元，用于定时对第一闪存设备和第二闪存设备进行增量和全量的差异校验，得到校验结果；a verification unit, configured to periodically perform incremental and full difference verification on the first flash memory device and the second flash memory device to obtain a verification result;

更新单元，用于若校验结果为存在差异，则根据校验结果中的差异，采用第一闪存设备中的数据对第二闪存设备中的数据进行更新。The updating unit is configured to update the data in the second flash memory device by using the data in the first flash memory device according to the difference in the verification result if the verification result is that there is a difference.

可选地，同步单元包括：Optionally, the synchronization unit includes:

路径获取子单元，用于将TiDB同城灾备库中的binlog文件作为待导入数据，并获取待导入数据对应的第一闪存设备的存储路径；The path obtaining subunit is used to use the binlog file in the TiDB intra-city disaster recovery database as the data to be imported, and obtain the storage path of the first flash memory device corresponding to the data to be imported;

线程数量确定子单元，用于基于待导入数据和第一闪存设备，确定目标线程的数量；a thread number determination subunit, used for determining the number of target threads based on the data to be imported and the first flash memory device;

多线程同步子单元，用于基于目标线程数量，构建多线程，并采用多线程将待导入数据同步到第一闪存设备的存储路径。The multi-thread synchronization subunit is used for constructing multi-threads based on the number of target threads, and using multi-threads to synchronize the data to be imported to the storage path of the first flash memory device.

可选地，数据恢复模块33包括：Optionally, thedata recovery module 33 includes:

数据清理单元，用于停止应用服务对TiDB生产库的写表处理，并将TiDB生产库的数据表中的数据清空；The data cleaning unit is used to stop the table writing processing of the TiDB production library by the application service, and clear the data in the data table of the TiDB production library;

待还原文件确定单元，用于从闪存NAS设备中获取全备文件和增备文件，作为待还原文件；The file to be restored determination unit is used to obtain the full backup file and the additional backup file from the flash NAS device as the file to be restored;

文件还原单元，用于将全备文件载入到TiDB生产库中的数据表中，再将增备文件载入到TiDB生产库中的数据表中；The file restoration unit is used to load the full backup file into the data table in the TiDB production database, and then load the additional backup file into the data table in the TiDB production database;

数据验证单元，用于对TiDB生产库中的数据表中的数据进行验证，验证通过后，确认数据还原成功，并启动应用服务。The data verification unit is used to verify the data in the data table in the TiDB production library. After the verification is passed, it confirms that the data is restored successfully and starts the application service.

关于TiDB数据库的维护装置的具体限定可以参见上文中对于TiDB数据库的维护方法的限定，在此不再赘述。上述TiDB数据库的维护装置中的各个模块可全部或部分通过软件、硬件及其组合来实现。上述各模块可以硬件形式内嵌于或独立于计算机设备中的处理器中，也可以以软件形式存储于计算机设备中的存储器中，以便于处理器调用执行以上各个模块对应的操作。For the specific limitation of the maintenance device of the TiDB database, reference may be made to the limitation of the maintenance method of the TiDB database above, which will not be repeated here. Each module in the above-mentioned TiDB database maintenance device may be implemented in whole or in part by software, hardware and combinations thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

为解决上述技术问题，本申请实施例还提供计算机设备。具体请参阅图4，图4为本实施例计算机设备基本结构框图。To solve the above technical problems, the embodiments of the present application also provide computer equipment. Please refer to FIG. 4 for details. FIG. 4 is a block diagram of a basic structure of a computer device according to this embodiment.

所述计算机设备4包括通过系统总线相互通信连接存储器41、处理器42、网络接口43。需要指出的是，图中仅示出了具有组件连接存储器41、处理器42、网络接口43的计算机设备4，但是应理解的是，并不要求实施所有示出的组件，可以替代的实施更多或者更少的组件。其中，本技术领域技术人员可以理解，这里的计算机设备是一种能够按照事先设定或存储的指令，自动进行数值计算和/或信息处理的设备，其硬件包括但不限于微处理器、专用集成电路(Application Specific Integrated Circuit，ASIC)、可编程门阵列(Field-Programmable Gate Array，FPGA)、数字处理器(Digital Signal Processor，DSP)、嵌入式设备等。The computer device 4 includes amemory 41, aprocessor 42, and anetwork interface 43 that communicate with each other through a system bus. It should be pointed out that the figure only shows the computer device 4 having the components connected to thememory 41, theprocessor 42, and thenetwork interface 43, but it should be understood that it is not required to implement all the components shown, and alternative implementations of more more or fewer components. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculation and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, special-purpose Integrated circuit (Application Specific Integrated Circuit, ASIC), Programmable Gate Array (Field-Programmable Gate Array, FPGA), Digital Signal Processor (Digital Signal Processor, DSP), embedded devices, etc.

所述计算机设备可以是桌上型计算机、笔记本、掌上电脑及云端服务器等计算设备。所述计算机设备可以与用户通过键盘、鼠标、遥控器、触摸板或声控设备等方式进行人机交互。The computer equipment may be a desktop computer, a notebook computer, a palmtop computer, a cloud server and other computing equipment. The computer device can perform human-computer interaction with the user through a keyboard, a mouse, a remote control, a touch pad or a voice control device.

所述存储器41至少包括一种类型的可读存储介质，所述可读存储介质包括闪存、硬盘、多媒体卡、卡型存储器(例如，SD或D界面显示存储器等)、随机访问存储器(RAM)、静态随机访问存储器(SRAM)、只读存储器(ROM)、电可擦除可编程只读存储器(EEPROM)、可编程只读存储器(PROM)、磁性存储器、磁盘、光盘等。在一些实施例中，所述存储器41可以是所述计算机设备4的内部存储单元，例如该计算机设备4的硬盘或内存。在另一些实施例中，所述存储器41也可以是所述计算机设备4的外部存储设备，例如该计算机设备4上配备的插接式硬盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。当然，所述存储器41还可以既包括所述计算机设备4的内部存储单元也包括其外部存储设备。本实施例中，所述存储器41通常用于存储安装于所述计算机设备4的操作系统和各类应用软件，例如电子文件的控制的程序代码等。此外，所述存储器41还可以用于暂时地存储已经输出或者将要输出的各类数据。Thememory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM) , Static Random Access Memory (SRAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), Programmable Read Only Memory (PROM), Magnetic Memory, Magnetic Disk, Optical Disk, etc. In some embodiments, thememory 41 may be an internal storage unit of the computer device 4 , such as a hard disk or a memory of the computer device 4 . In other embodiments, thememory 41 may also be an external storage device of the computer device 4 , such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, flash memory card (Flash Card) and so on. Of course, thememory 41 may also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, thememory 41 is generally used to store the operating system and various application software installed in the computer device 4 , such as program codes for controlling electronic files. In addition, thememory 41 can also be used to temporarily store various types of data that have been output or will be output.

所述处理器42在一些实施例中可以是中央处理器(Central ProcessingUnit，CPU)、控制器、微控制器、微处理器、或其他数据处理芯片。该处理器42通常用于控制所述计算机设备4的总体操作。本实施例中，所述处理器42用于运行所述存储器41中存储的程序代码或者处理数据，例如运行电子文件的控制的程序代码。In some embodiments, theprocessor 42 may be a central processing unit (Central Processing Unit, CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. Thisprocessor 42 is typically used to control the overall operation of the computer device 4 . In this embodiment, theprocessor 42 is configured to run the program code or process data stored in thememory 41 , for example, run the program code for controlling the electronic file.

所述网络接口43可包括无线网络接口或有线网络接口，该网络接口43通常用于在所述计算机设备4与其他电子设备之间建立通信连接。Thenetwork interface 43 may include a wireless network interface or a wired network interface, and thenetwork interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

本申请还提供了另一种实施方式，即提供一种计算机可读存储介质，所述计算机可读存储介质存储有界面显示程序，所述界面显示程序可被至少一个处理器执行，以使所述至少一个处理器执行如上述的TiDB数据库的维护方法的步骤。The present application also provides another implementation manner, which is to provide a computer-readable storage medium, where an interface display program is stored in the computer-readable storage medium, and the interface display program can be executed by at least one processor, so that all The at least one processor executes the steps of the above-mentioned TiDB database maintenance method.

通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端设备(可以是手机，计算机，服务器，空调器，或者网络设备等)执行本申请各个实施例所述的方法。From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the methods described in the various embodiments of this application.

显然，以上所描述的实施例仅仅是本申请一部分实施例，而不是全部的实施例，附图中给出了本申请的较佳实施例，但并不限制本申请的专利范围。本申请可以以许多不同的形式来实现，相反地，提供这些实施例的目的是使对本申请的公开内容的理解更加透彻全面。尽管参照前述实施例对本申请进行了详细的说明，对于本领域的技术人员来而言，其依然可以对前述各具体实施方式所记载的技术方案进行修改，或者对其中部分技术特征进行等效替换。凡是利用本申请说明书及附图内容所做的等效结构，直接或间接运用在其他相关的技术领域，均同理在本申请专利保护范围之内。Obviously, the above-described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The accompanying drawings show the preferred embodiments of the present application, but do not limit the scope of the patent of the present application. This application may be embodied in many different forms, rather these embodiments are provided so that a thorough and complete understanding of the disclosure of this application is provided. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or perform equivalent replacements for some of the technical features. . Any equivalent structure made by using the contents of the description and drawings of the present application, which is directly or indirectly used in other related technical fields, is also within the scope of protection of the patent of the present application.

Claims (10)

Translated fromChinese

1.一种TiDB数据库的维护方法，其特征在于，所述TiDB数据库的维护方法包括：1. a maintenance method of TiDB database, is characterized in that, the maintenance method of described TiDB database comprises:通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到所述TiDB生产库中的数据对应的binlog文件，并将所述binlog文件同步到TiDB同城灾备库；Collect the data in the TiDB production library through the TiDB-Binlog cluster component, obtain the binlog file corresponding to the data in the TiDB production library, and synchronize the binlog file to the TiDB intra-city disaster recovery library;采用预设的定时备份脚本，将所述TiDB同城灾备库中的binlog文件同步到闪存NAS设备；Synchronize the binlog file in the TiDB intra-city disaster recovery database to the flash NAS device by using a preset timing backup script;若监控到TiDB生产库异常，则基于所述闪存NAS设备中存储的binlog文件，对所述TiDB生产库进行数据还原处理。If it is monitored that the TiDB production library is abnormal, data restoration processing is performed on the TiDB production library based on the binlog file stored in the flash NAS device.2.如权利要求1所述的TiDB数据库的维护方法，其特征在于，所述TiDB-Binlog集群组件包括Drainer组件与pump组件，所述通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到所述TiDB生产库中的数据对应的binlog文件，并将所述binlog文件同步到TiDB同城灾备库包括：2. the maintenance method of TiDB database as claimed in claim 1, is characterized in that, described TiDB-Binlog cluster component comprises Drainer component and pump component, described by TiDB-Binlog cluster component collects the data in TiDB production library, obtains The binlog file corresponding to the data in the TiDB production library, and synchronizing the binlog file to the TiDB same-city disaster recovery library includes:通过所述pump组件收集所述pump组件对应的TiDB节点服务器的binglog文件，作为基础文件，并将所述基础文件发送至所述Drainer组件；Collect the binglog file of the TiDB node server corresponding to the pump component through the pump component as a basic file, and send the basic file to the Drainer component;通过所述Drainer组件接收所述基础文件并对所述基础文件进行合并，得到合并后的binglog文件；Receive the base file through the Drainer component and merge the base file to obtain a merged binglog file;将所述合并后的binglog文件同步到所述TiDB同城灾备库。Synchronize the merged binglog file to the TiDB same-city disaster recovery database.3.如权利要求1所述的TiDB数据库的维护方法，其特征在于，所述闪存NAS设备包括第一闪存设备和第二闪存设备。3. The TiDB database maintenance method according to claim 1, wherein the flash memory NAS device comprises a first flash memory device and a second flash memory device.4.如权利要求3所述的TiDB数据库的维护方法，其特征在于，所述采用预设的定时备份脚本，将所述TiDB同城灾备库中的binlog文件同步到闪存NAS设备包括：4. the maintenance method of TiDB database as claimed in claim 3, it is characterized in that, described adopting preset timed backup script, the binlog file in described TiDB same-city disaster recovery library is synchronized to flash memory NAS device comprises:将所述TiDB同城灾备库中的binlog文件移动并保存到所述第一闪存设备中；moving and saving the binlog file in the TiDB intra-city disaster recovery library to the first flash memory device;定时对所述第一闪存设备和所述第二闪存设备进行增量和全量的差异校验，得到校验结果；Perform incremental and full difference verification on the first flash memory device and the second flash memory device periodically to obtain a verification result;若所述校验结果为存在差异，则根据所述校验结果中的差异，采用所述第一闪存设备中的数据对所述第二闪存设备中的数据进行更新。If the verification result is that there is a difference, the data in the second flash memory device is updated by using the data in the first flash memory device according to the difference in the verification result.5.如权利要求4所述的TiDB数据库的维护方法，其特征在于，所述将所述TiDB同城灾备库中的binlog文件移动并保存到所述第一闪存设备中包括：5. The maintenance method of the TiDB database according to claim 4, wherein the moving and saving the binlog file in the TiDB same-city disaster recovery library to the first flash memory device comprises:将所述TiDB同城灾备库中的binlog文件作为待导入数据，并获取所述待导入数据对应的所述第一闪存设备的存储路径；Taking the binlog file in the TiDB intra-city disaster recovery database as the data to be imported, and acquiring the storage path of the first flash memory device corresponding to the data to be imported;基于所述待导入数据和所述第一闪存设备，确定目标线程的数量；determining the number of target threads based on the data to be imported and the first flash memory device;基于所述目标线程数量，构建多线程，并采用所述多线程将所述待导入数据同步到所述第一闪存设备的存储路径。Based on the target number of threads, multithreading is constructed, and the multithreading is used to synchronize the data to be imported to the storage path of the first flash memory device.6.如权利要求1至5任一项所述的TiDB数据库的维护方法，其特征在于，所述若监控到TiDB生产库异常，则基于所述闪存NAS设备中存储的binlog文件，对所述TiDB生产库进行数据还原处理包括：6. The method for maintaining a TiDB database according to any one of claims 1 to 5, wherein if the TiDB production library is abnormally monitored, based on the binlog file stored in the flash NAS device, the Data restoration in the TiDB production database includes:停止应用服务对所述TiDB生产库的写表处理，并将所述TiDB生产库的数据表中的数据清空；Stop the table writing processing of the TiDB production library by the application service, and clear the data in the data table of the TiDB production library;从所述闪存NAS设备中获取全备文件和增备文件，作为待还原文件；Obtain full backup files and additional backup files from the flash NAS device as files to be restored;将所述全备文件载入到TiDB生产库中的数据表中，再将增备文件载入到所述TiDB生产库中的数据表中；Load the full backup file into the data table in the TiDB production library, and then load the supplementary file into the data table in the TiDB production library;对所述TiDB生产库中的数据表中的数据进行验证，验证通过后，确认数据还原成功，并启动应用服务。The data in the data table in the TiDB production library is verified, and after the verification is passed, it is confirmed that the data is restored successfully, and the application service is started.7.一种TiDB数据库的维护装置，其特征在于，所述TiDB数据库的维护装置包括：7. a maintenance device of TiDB database, it is characterized in that, the maintenance device of described TiDB database comprises:数据采集模块，用于通过TiDB-Binlog集群组件采集TiDB生产库中的数据，得到所述TiDB生产库中的数据对应的binlog文件，并将所述binlog文件同步到TiDB同城灾备库；a data collection module, used for collecting data in the TiDB production library through the TiDB-Binlog cluster component, obtaining the binlog file corresponding to the data in the TiDB production library, and synchronizing the binlog file to the TiDB intra-city disaster recovery library;文件同步模块，用于采用预设的定时备份脚本，将所述TiDB同城灾备库中的binlog文件同步到闪存NAS设备；The file synchronization module is used for synchronizing the binlog file in the TiDB intra-city disaster recovery database to the flash NAS device by using a preset timed backup script;数据恢复模块，用于若监控到TiDB生产库异常，则基于所述闪存NAS设备中存储的binlog文件，对所述TiDB生产库进行数据还原处理。A data recovery module, configured to perform data restoration processing on the TiDB production database based on the binlog file stored in the flash NAS device if an abnormality of the TiDB production database is monitored.8.如权利要求7所述的TiDB数据库的维护装置，其特征在于，所述数据采集模块包括：8. the maintenance device of TiDB database as claimed in claim 7, is characterized in that, described data acquisition module comprises:采集单元，用于通过所述pump组件收集所述pump组件对应的TiDB节点服务器的binglog文件，作为基础文件，并将所述基础文件发送至所述Drainer组件；a collection unit, configured to collect the binglog file of the TiDB node server corresponding to the pump component through the pump component as a basic file, and send the basic file to the Drainer component;合并单元，用于通过所述Drainer组件接收所述基础文件并对所述基础文件进行合并，得到合并后的binglog文件；a merging unit, for receiving the base file through the Drainer component and merging the base file to obtain a merged binglog file;同步单元，用于将所述合并后的binglog文件同步到所述TiDB同城灾备库。A synchronization unit, configured to synchronize the merged binglog file to the TiDB same-city disaster recovery database.9.一种计算机设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如权利要求1至6任一项所述的TiDB数据库的维护方法。9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the computer program as claimed in the claims The maintenance method of the TiDB database described in any one of 1 to 6.10.一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至6任一项所述的TiDB数据库的维护方法。10. A computer-readable storage medium storing a computer program, wherein the computer program realizes the TiDB database as described in any one of claims 1 to 6 when the computer program is executed by the processor maintenance method.

Images