CN116401314A - Data processing method and device - Google Patents
Data processing method and deviceDownload PDFInfo
- Publication number
- CN116401314A CN116401314ACN202310350862.6ACN202310350862ACN116401314ACN 116401314 ACN116401314 ACN 116401314ACN 202310350862 ACN202310350862 ACN 202310350862ACN 116401314 ACN116401314 ACN 116401314A
- Authority
- CN
- China
- Prior art keywords
- database instance
- data
- written
- synchronous
- copy set
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F16/275—Synchronous replication
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Data Mining & Analysis (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本申请涉及数据处理技术领域，特别涉及一种数据处理方法及装置。The present application relates to the technical field of data processing, and in particular to a data processing method and device.
背景技术Background technique
PostgreSQL是一个功能强大的开源关系型数据库系统，它使用和扩展了SQL语言，并结合了许多功能，这些功能提供安全的数据存储以及扩展复杂数据工作负载。PostgreSQL is a powerful open source relational database system that uses and extends the SQL language and incorporates many features that provide secure data storage and scale for complex data workloads.
虽然PostgreSQL数据库在单实例的情况能够提供高效的事务处理(OLTP)服务，但是如果该单实例因为其他原因，如所在机器故障等原因，不能提供服务了，显然会带来无法估量的损失。所以在目前真正的产品化集成部署上，均会部署多个数据库实例，以此形成一个数据库集群，哪怕其中一个实例无法提供服务，其他服务也能够顺利接手，一般我们也称为数据库的高可用。尤其在当下产品系统微服务化和多节点集群部署的趋势下，如何提高产品的稳定性和持续服务能力，数据库作为其中一环比如承担着重要的作用，所以多实例化数据库部署就显得非常必要并且更加重要。Although the PostgreSQL database can provide efficient transaction processing (OLTP) services in the case of a single instance, if the single instance cannot provide services due to other reasons, such as machine failure, etc., it will obviously cause immeasurable losses. Therefore, in the current real product integration deployment, multiple database instances will be deployed to form a database cluster. Even if one of the instances cannot provide services, other services can take over smoothly. Generally, we also call it high availability of the database. . Especially under the current trend of product system micro-service and multi-node cluster deployment, how to improve product stability and continuous service capability, the database plays an important role as one of the links, so multi-instance database deployment is very necessary And more importantly.
PostgreSQL数据库本身提供了原生的流复制技术，用于高效的主数据库实例和备数据库实例间的数据复制。该流复制技术只能过手动方式进行主备之间的相关配置，并且是支持一主多备的部署模式。通过该技术一般可以实现PostgreSQL数据库的高可用模式。在一主多备的部署模式下，主和备之间的数据复制，一般有两种模式，一种是异步方式，一种是同步方式。The PostgreSQL database itself provides a native streaming replication technology for efficient data replication between the primary database instance and the standby database instance. The stream replication technology can only be manually configured between the master and backup, and it supports a deployment mode of one master and multiple backups. Generally, the high availability mode of the PostgreSQL database can be realized through this technology. In the deployment mode of one master and multiple backups, there are generally two modes for data replication between the master and the backup, one is asynchronous and the other is synchronous.
同步流复制写请求，必须等待所有数据库实例都确认之后，才能返回成功，此时如果有一台备实例故障，则会阻塞整个数据写请求，可能会造成较大的业务问题。Synchronous stream replication write requests must wait for all database instances to confirm before returning success. At this time, if a standby instance fails, the entire data write request will be blocked, which may cause major business problems.
异步流复制写请求，当在主数据库实例完成写入，则返回成功，这种情况，主库和备库通过异步方式进行数据同步，会存在数据不一致的情况，一旦存在主库故障，备库提升为主，如果有部分数据当前被提升为主库的备库还没来得及同步，会造成一定的数据丢失，可能造成不可估量的业务损失。Asynchronous stream replication write request, when the writing is completed in the primary database instance, it returns success. In this case, the primary database and the standby database synchronize data through asynchronous mode, and there will be data inconsistencies. Once the primary database fails, the standby database The promotion is the main one. If some data is currently promoted to the standby database of the main database and has not had time to synchronize, it will cause a certain amount of data loss, which may cause immeasurable business losses.
那么，如何能够在保证PostgreSQL数据库高可用的前提下，提高数据库的数据一致性，就显得尤为重要。Then, how to improve the data consistency of the database on the premise of ensuring the high availability of the PostgreSQL database is particularly important.
发明内容Contents of the invention
本申请提供了一种数据处理方法及装置。The present application provides a data processing method and device.
第一方面，本申请提供了一种数据处理方法，应用于数据库，所述方法包括：In a first aspect, the present application provides a data processing method applied to a database, the method comprising:
构建同步副本集合，所述同步副本集合包括主数据库实例和若干备数据库实例，其中，各备数据库实例与所述主数据库实例之间保持心跳检测，且心跳检测正常，若任一备数据库实例与所述主数据库之间的心跳检测异常，则将所述任一备数据库实例剔除出所述同步副本集合；Build a synchronous replica set, the synchronous replica set includes a primary database instance and several standby database instances, wherein heartbeat detection is maintained between each standby database instance and the primary database instance, and the heartbeat detection is normal, if any standby database instance and If the heartbeat detection between the primary databases is abnormal, any instance of the standby database is removed from the set of synchronous replicas;
接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例；receiving the data write request sent by the client, and based on the data write request, writing the data to be written into the primary database instance of the synchronous replica set;
采用异步流复制/同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例。The data to be written is synchronized to each standby database instance of the synchronous copy set by means of asynchronous stream replication/synchronous stream replication.
可选地，所述数据库还包括用于接入客户端的访问代理组件；用于监控同步副本集合，选举主数据库实例的哨兵组件；用于存储同步副本集合中各数据库实例的状态的后端存储组件，所述访问代理组件可访问所述后端存储组件；Optionally, the database also includes an access agent component for accessing the client; a sentinel component for monitoring the synchronous replica set and electing a master database instance; and a backend storage for storing the state of each database instance in the synchronous replica set component, the access proxy component can access the backend storage component;
构建同步副本集合的步骤包括：The steps to build a collection of synchronous replicas include:
所述哨兵组件将第一个加入所述同步副本集合的数据库实例确定为主数据库实例，将其它加入所述同步副本集合的数据库实例确定为备数据库实例，其中，任一新加入所述同步副本集合的备数据库实例与所述主数据库实例建立心跳检测机制。The sentinel component determines the first database instance that joins the synchronous replica set as the primary database instance, and determines the other database instances that join the synchronous replica set as standby database instances, wherein any new database instance that joins the synchronous replica The aggregated standby database instance establishes a heartbeat detection mechanism with the primary database instance.
可选地，接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例的步骤包括：Optionally, the step of receiving the data write request sent by the client, and based on the data write request, writing the data to be written into the primary database instance of the synchronous replica set includes:
所述访问代理组件接收客户端发送的数据写请求；The access proxy component receives the data write request sent by the client;
基于所述后端存储组件中存储的各数据库实例的状态，确定主数据库实例；determining the master database instance based on the status of each database instance stored in the back-end storage component;
将待写入数据写入所述同步副本集合的主数据库实例。Write the data to be written to the primary database instance of the set of synchronous replicas.
可选地，采用异步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例的步骤包括：Optionally, using asynchronous stream replication, the step of synchronizing the data to be written to each standby database instance of the synchronous replica set includes:
所述主数据库实例在确定待写入数据写入本地成功之后，向客户端反馈写入成功的响应，并将所述待写入数据同步至同步副本集合的各备数据库实例；After the primary database instance determines that the data to be written is successfully written locally, it feeds back a successful write response to the client, and synchronizes the data to be written to each standby database instance of the synchronization copy set;
采用同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例的步骤包括：Using synchronous stream replication, the steps of synchronizing the data to be written to each standby database instance of the synchronous copy set include:
所述主数据库实例在将待写入数据写入本地时，将所述待写入数据同步至各备数据库实例，若确定所述待写入数据均成功写入所述同步副本集合的各数据库实例，则向客户端反馈写入成功的响应。When the primary database instance writes the data to be written locally, it synchronizes the data to be written to each standby database instance, and if it is determined that the data to be written is successfully written to each database of the synchronization copy set Instance, then feedback write successful response to the client.
第二方面，本申请提供了一种数据处理装置，应用于数据库，所述装置包括：In a second aspect, the present application provides a data processing device applied to a database, the device comprising:
构建单元，用于构建同步副本集合，所述同步副本集合包括主数据库实例和若干备数据库实例，其中，各备数据库实例与所述主数据库实例之间保持心跳检测，且心跳检测正常，若任一备数据库实例与所述主数据库之间的心跳检测异常，则将所述任一备数据库实例剔除出所述同步副本集合；The construction unit is used to construct a synchronous replica set, the synchronous replica set includes a primary database instance and a number of standby database instances, wherein heartbeat detection is maintained between each standby database instance and the primary database instance, and the heartbeat detection is normal, if any If the heartbeat detection between a standby database instance and the primary database is abnormal, any standby database instance is removed from the synchronous copy set;
接收单元，用于接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例；The receiving unit is configured to receive a data write request sent by the client, and based on the data write request, write the data to be written into the master database instance of the synchronization copy set;
同步单元，用于采用异步流复制/同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例。The synchronization unit is configured to synchronize the data to be written to each standby database instance of the synchronous copy set by means of asynchronous stream replication/synchronous stream replication.
可选地，所述数据库还包括用于接入客户端的访问代理组件；用于监控同步副本集合，选举主数据库实例的哨兵组件；用于存储同步副本集合中各数据库实例的状态的后端存储组件，所述访问代理组件可访问所述后端存储组件；Optionally, the database also includes an access agent component for accessing the client; a sentinel component for monitoring the synchronous replica set and electing a master database instance; and a backend storage for storing the state of each database instance in the synchronous replica set component, the access proxy component can access the backend storage component;
构建同步副本集合时：When building a collection of synchronous replicas:
所述哨兵组件将第一个加入所述同步副本集合的数据库实例确定为主数据库实例，将其它加入所述同步副本集合的数据库实例确定为备数据库实例，其中，任一新加入所述同步副本集合的备数据库实例与所述主数据库实例建立心跳检测机制。The sentinel component determines the first database instance that joins the synchronous replica set as the primary database instance, and determines the other database instances that join the synchronous replica set as standby database instances, wherein any new database instance that joins the synchronous replica The aggregated standby database instance establishes a heartbeat detection mechanism with the primary database instance.
可选地，接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例时：Optionally, when receiving the data write request sent by the client, and based on the data write request, when writing the data to be written into the primary database instance of the synchronous replica set:
所述访问代理组件接收客户端发送的数据写请求；The access proxy component receives the data write request sent by the client;
基于所述后端存储组件中存储的各数据库实例的状态，确定主数据库实例；determining the master database instance based on the status of each database instance stored in the back-end storage component;
将待写入数据写入所述同步副本集合的主数据库实例。Write the data to be written to the primary database instance of the set of synchronous replicas.
可选地，采用异步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例时：Optionally, when synchronizing the data to be written to each standby database instance of the synchronous replica set by means of asynchronous stream replication:
所述主数据库实例在确定待写入数据写入本地成功之后，向客户端反馈写入成功的响应，并将所述待写入数据同步至同步副本集合的各备数据库实例；After the primary database instance determines that the data to be written is successfully written locally, it feeds back a successful write response to the client, and synchronizes the data to be written to each standby database instance of the synchronization copy set;
采用同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例时：When synchronous stream replication is used to synchronize the data to be written to each standby database instance of the synchronous replica set:
所述主数据库实例在将待写入数据写入本地时，将所述待写入数据同步至各备数据库实例，若确定所述待写入数据均成功写入所述同步副本集合的各数据库实例，则向客户端反馈写入成功的响应。When the primary database instance writes the data to be written locally, it synchronizes the data to be written to each standby database instance, and if it is determined that the data to be written is successfully written to each database of the synchronization copy set Instance, then feedback write successful response to the client.
第三方面，本申请实施例提供一种数据处理装置，该数据处理装置包括：In a third aspect, the embodiment of the present application provides a data processing device, the data processing device includes:
存储器，用于存储程序指令；memory for storing program instructions;
处理器，用于调用所述存储器中存储的程序指令，按照获得的程序指令执行如上述第一方面中任一项所述的方法的步骤。The processor is configured to call the program instructions stored in the memory, and execute the steps of the method according to any one of the above-mentioned first aspects according to the obtained program instructions.
第四方面，本申请实施例还提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机可执行指令，所述计算机可执行指令用于使所述计算机执行如上述第一方面中任一项所述方法的步骤。In a fourth aspect, the embodiment of the present application further provides a computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to make the computer perform the above-mentioned first A step of any one of the methods described in the aspect.
综上可知，本申请实施例提供的数据处理方法，应用于数据库，所述方法包括：构建同步副本集合，所述同步副本集合包括主数据库实例和若干备数据库实例，其中，各备数据库实例与所述主数据库实例之间保持心跳检测，且心跳检测正常，若任一备数据库实例与所述主数据库之间的心跳检测异常，则将所述任一备数据库实例剔除出所述同步副本集合；接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例；采用异步流复制/同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例。In summary, the data processing method provided by the embodiment of the present application is applied to a database, and the method includes: constructing a synchronous replica set, the synchronous replica set includes a primary database instance and several standby database instances, wherein each standby database instance is connected to Keep heartbeat detection between the primary database instances, and the heartbeat detection is normal, if the heartbeat detection between any standby database instance and the primary database is abnormal, remove the any standby database instance from the synchronization replica set ; Receive the data write request sent by the client, and based on the data write request, write the data to be written into the primary database instance of the synchronous replica set; Write data is synchronized to each standby database instance of the synchronous replica set.
采用本申请实施例提供的数据处理方法，构建同步副本集合，数据库实例以集合的方式面向客户，确保同步副本集合中各数据库实例是正常运行的，在进行数据写时，仅在同步副本集合包括的各数据库实例间进行流复制，避免在进行异步流复制时，由于在写入过程中出现数据库实例异常而导致主备数据库数据不一致，或者，在进行同步流复制时，由于任一数据库实例故障而出现阻塞数据写请求的问题。Using the data processing method provided by the embodiment of the present application, a set of synchronous replicas is constructed, and the database instances face customers as a set to ensure that each database instance in the synchronous replica set is running normally. When writing data, only when the synchronous replica set includes Stream replication is performed between each database instance to avoid data inconsistency between the primary and standby databases due to database instance exceptions during the writing process during asynchronous stream replication, or, during synchronous stream replication, due to failure of any database instance And there is a problem of blocking data write requests.
附图说明Description of drawings
为了更加清楚地说明本申请实施例或者现有技术中的技术方案，下面将对本申请实施例或者现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请中记载的一些实施例，对于本领域普通技术人员来讲，还可以根据本申请实施例的这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the application or the prior art, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments of the application or the prior art. Obviously, the accompanying drawings in the following description These are only some embodiments described in this application, and those skilled in the art can also obtain other drawings according to these drawings of the embodiments of this application.
图1为本申请实施例提供的一种数据处理方法的详细流程图；FIG. 1 is a detailed flow chart of a data processing method provided in an embodiment of the present application;
图2为本申请实施例提供的一种数据库具体架构示意图；FIG. 2 is a schematic diagram of a specific database architecture provided by an embodiment of the present application;
图3a为本申请实施例提供的一种构建同步副本集合的过程示意图；FIG. 3a is a schematic diagram of a process for constructing a synchronous replica set provided by an embodiment of the present application;
图3b为为本申请实施例提供的一种同步副本集合维护过程示意图；FIG. 3b is a schematic diagram of a synchronous replica set maintenance process provided by the embodiment of the present application;
图4为本申请实施例提供的一种数据处理装置的结构示意图；FIG. 4 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;
图5为本申请实施例提供的一种数据处理装置的硬件架构示意图。FIG. 5 is a schematic diagram of a hardware architecture of a data processing device provided by an embodiment of the present application.
具体实施方式Detailed ways
在本申请实施例使用的术语仅仅是出于描述特定实施例的目的，而非限制本申请。本申请和权利要求书中所使用的单数形式的“一种”、“所述”和“该”也旨在包括多数形式，除非上下文清楚地表示其它含义。还应当理解，本文中使用的术语“和/或”是指包含一个或多个相关联的列出项目的任何或所有可能组合。The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. As used in this application and the claims, the singular forms "a", "the" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein is meant to include any and all possible combinations of one or more of the associated listed items.
应当理解，尽管在本申请实施例可能采用术语第一、第二、第三等来描述各种信息，但这些信息不应限于这些术语。这些术语仅用来将同一类型的信息彼此区分开。例如，在不脱离本申请范围的情况下，第一信息也可以被称为第二信息，类似地，第二信息也可以被称为第一信息。取决于语境，此外，所使用的词语“如果”可以被解释成为“在……时”或“当……时”或“响应于确定”。It should be understood that although terms such as first, second, and third may be used in the embodiment of the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present application, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, furthermore, the use of the word "if" could be interpreted as "at" or "when" or "in response to a determination."
示例性的，参阅图1所示，为本申请实施例提供的一种数据处理方法的详细流程图，该方法应用于数据库，该方法包括以下步骤：Exemplarily, refer to FIG. 1, which is a detailed flow chart of a data processing method provided by an embodiment of the present application. The method is applied to a database, and the method includes the following steps:
步骤100：构建同步副本集合，所述同步副本集合包括主数据库实例和若干备数据库实例。Step 100: Build a synchronous replica set, the synchronous replica set includes a primary database instance and several standby database instances.
其中，各备数据库实例与所述主数据库实例之间保持心跳检测，且心跳检测正常，若任一备数据库实例与所述主数据库之间的心跳检测异常，则将所述任一备数据库实例剔除出所述同步副本集合。Wherein, heartbeat detection is maintained between each standby database instance and the primary database instance, and the heartbeat detection is normal. If the heartbeat detection between any standby database instance and the primary database is abnormal, the Eliminate the set of synchronous replicas.
本申请实施例中，数据库包括同步副本集合(由多个数据库实例组成)，用于接入客户端的访问代理组件；用于监控同步副本集合中各数据库实例状态，选举主数据库实例的哨兵组件；用于存储同步副本集合中各数据库实例的状态的后端存储组件，所述访问代理组件可访问所述后端存储组件。In the embodiment of the present application, the database includes a synchronous copy set (composed of multiple database instances), which is used to access the access agent component of the client; for monitoring the status of each database instance in the synchronous copy set, and electing the sentinel component of the master database instance; A back-end storage component for storing the state of each database instance in the synchronization replica set, and the access proxy component can access the back-end storage component.
示例性的，参阅图2所示，为本申请实施例提供的一种数据库具体架构示意图。以数据库为PostgreSQL为例进行说明。PostgreSQL包含访问代理Proxy(代理Proxy集群)、数据库实例Keeper(多个数据库实例组成同步副本集合/集群)以及哨兵Sentinel组件等三个组件，所有的客户端访问均通过Proxy代理进行数据库实例的事务读写操作，一个数据库实例Keeper组件包含一个具体的PostgreSQL实例，而哨兵Sentinel组件监控整个集群，计算出整个集群视图(包含数据库实例主/备库信息)，同时将集群视图存储至状态后端存储组件中(比如，etcd、consul等)。Proxy代理可访问后端存储组件，实时了解数据库实例主/备库信息。For example, refer to FIG. 2 , which is a schematic diagram of a specific database architecture provided by an embodiment of the present application. The database is PostgreSQL as an example for illustration. PostgreSQL includes three components: access proxy Proxy (proxy proxy cluster), database instance Keeper (multiple database instances form a synchronous copy set/cluster) and sentinel Sentinel components. All client access is performed through the Proxy agent for transactional reading of database instances. For write operations, a database instance Keeper component contains a specific PostgreSQL instance, while the Sentinel component monitors the entire cluster, calculates the entire cluster view (including database instance master/standby database information), and stores the cluster view to the state backend storage component at the same time Medium (for example, etcd, consul, etc.). Proxy agents can access the back-end storage components to learn about the primary/standby database information of the database instance in real time.
本申请实施例中，在构建同步副本集合时，一种较佳地实现方式为：所述哨兵组件将第一个加入所述同步副本集合的数据库实例确定为主数据库实例，将其它加入所述同步副本集合的数据库实例确定为备数据库实例，其中，任一新加入所述同步副本集合的备数据库实例与所述主数据库实例建立心跳检测机制。In this embodiment of the application, when constructing a synchronous replica set, a preferred implementation method is: the sentinel component determines the first database instance to be added to the synchronous replica set as the primary database instance, and the other The database instance of the synchronous replica set is determined as a standby database instance, wherein any standby database instance newly added to the synchronous replica set establishes a heartbeat detection mechanism with the primary database instance.
示例性的，参阅图3a所示，为本申请实施例提供的一种构建同步副本集合的过程示意图，哨兵组件在检测到第一个加入集群的PG(数据库实例)时，将该PG确定为PG主，此时，同步副本集合包括{PG主}；当第二个PG加入集群时，将该PG确定为PG备(PG备1)，PG备1与PG主建立心跳检测机制，PG备1周期性向PG主发送心跳，PG主检测心跳不超期，说明PG备1状态正常(可用)，此时，同步副本集合包括{PG主、PG备1}；当第三个PG加入集群时，将该PG确定为PG备(PG备2)，PG备2与PG主建立心跳检测机制，PG备2周期性向PG主发送心跳，PG主检测心跳不超期，说明PG备2状态正常(可用)，此时，同步副本集合包括{PG主、PG备1、PG备2}。Exemplarily, refer to FIG. 3 a , which is a schematic diagram of a process of constructing a synchronous replica set provided by an embodiment of the present application. When the Sentinel component detects the first PG (database instance) that joins the cluster, it determines the PG as PG master, at this time, the set of synchronous replicas includes {PG master}; when the second PG joins the cluster, determine the PG as the PG backup (PG backup 1), PG backup 1 establishes a heartbeat detection mechanism with the PG master, and the PG backup 1 Periodically send heartbeats to the PG master, and the PG master detects that the heartbeat does not expire, indicating that the status of PG backup 1 is normal (available). At this time, the synchronization replica set includes {PG master, PG backup 1}; when the third PG joins the cluster, Determine the PG as PG backup (PG backup 2), PG backup 2 establishes a heartbeat detection mechanism with the PG master, PG backup 2 periodically sends heartbeats to the PG master, and the PG master detects that the heartbeat does not expire, indicating that the PG backup 2 is in normal state (available) , at this time, the synchronous replica set includes {PG master, PG backup 1, PG backup 2}.
示例性的，参阅图3b所示，为本申请实施例提供的一种同步副本集合维护过程示意图，当PG主检测到PG备1的心跳超期时，将PG备1剔除出同步副本集合，也即同步副本集合包括{PG主、PG备1}。相应的，哨兵组件检测到集群状态发生变化，PG备1异常，将PG备1剔除出集群，更新后端存储中维护的集群视图，此时，代理组件从后端存储获取更新后的集群视图，后续的客户端读写操作，就仅针对PG主、PG备1进行。For example, refer to Figure 3b, which is a schematic diagram of a synchronous replica set maintenance process provided by the embodiment of the present application. When the PG master detects that the heartbeat of PG standby 1 has expired, it removes PG standby 1 from the synchronous replica set, and also That is, the synchronous replica set includes {PG master, PG backup 1}. Correspondingly, the sentinel component detects that the cluster state changes, and PG standby 1 is abnormal, removes PG standby 1 from the cluster, and updates the cluster view maintained in the backend storage. At this time, the proxy component obtains the updated cluster view from the backend storage , subsequent client read and write operations are only performed on PG master and PG backup 1.
步骤110：接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例。Step 110: Receive the data write request sent by the client, and based on the data write request, write the data to be written into the primary database instance of the synchronous replica set.
本申请实施例中，接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例时，一种较佳地实现方式为：In the embodiment of the present application, when receiving the data write request sent by the client, and based on the data write request, when writing the data to be written into the primary database instance of the synchronization replica set, a better implementation method is as follows:
所述访问代理组件接收客户端发送的数据写请求；基于所述后端存储组件中存储的各数据库实例的状态，确定主数据库实例；将待写入数据写入所述同步副本集合的主数据库实例。The access proxy component receives the data write request sent by the client; determines the primary database instance based on the status of each database instance stored in the back-end storage component; writes the data to be written into the primary database of the synchronous copy set instance.
步骤120：采用异步流复制/同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例。Step 120: Using asynchronous stream replication/synchronous stream replication, synchronize the data to be written to each standby database instance of the synchronous replica set.
具体地，本申请实施例中，在采用异步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例时，一种较佳地实现方式为：Specifically, in the embodiment of the present application, when using asynchronous stream replication to synchronize the data to be written to each standby database instance of the synchronous replica set, a better implementation method is as follows:
所述主数据库实例在确定待写入数据写入本地成功之后，向客户端反馈写入成功的响应，并将所述待写入数据同步至同步副本集合的各备数据库实例。After determining that the data to be written is successfully written locally, the primary database instance feeds back a successful write response to the client, and synchronizes the data to be written to each standby database instance in the synchronization replica set.
由上可知，主数据库实例在确定待写入数据已成功写入本地之后，即可直接向客户端反馈写入成功的反馈，此时，主数据库实例根据同步副本集合当前包括的，状态正常的备数据库实例进行同步流复制，即便在同步流复制过程中，任一备数据库实例异常，主数据库实例会将该异常的付数据库实例剔除出同步副本集合，也即，同步副本集合包括的各备数据库实例均会成功写入待写入数据。由于出现异常的，可能会导致主备数据库实例数据不一致的备数据库实例已剔除出同步副本集合，只有同步副本集合中的数据库实例时面向用户的，那么，就不会出现同步流复制，导致主备数据库实例数据不一致的情况。It can be seen from the above that after the primary database instance confirms that the data to be written has been successfully written locally, it can directly report the successful writing feedback to the client. The standby database instance performs synchronous streaming replication. Even if any standby database instance is abnormal during the synchronous streaming replication process, the primary database instance will remove the abnormal secondary database instance from the synchronous replica set, that is, each standby database instance included in the synchronous replica set The database instance will successfully write the data to be written. Due to abnormalities, the standby database instance that may cause data inconsistency between the primary and standby database instances has been removed from the synchronization replica set. Only the database instances in the synchronization replica set are user-oriented. The instance data of the standby database is inconsistent.
进一步地，本申请实施例中，在采用同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例时，一种较佳地实现方式为：Further, in the embodiment of the present application, when synchronous stream replication is used to synchronize the data to be written to each standby database instance of the synchronous copy set, a better implementation method is:
所述主数据库实例在将待写入数据写入本地时，将所述待写入数据同步至各备数据库实例，若确定所述待写入数据均成功写入所述同步副本集合的各数据库实例，则向客户端反馈写入成功的响应。When the primary database instance writes the data to be written locally, it synchronizes the data to be written to each standby database instance, and if it is determined that the data to be written is successfully written to each database of the synchronization copy set Instance, then feedback write successful response to the client.
也就是说，主数据库实例在将待写入数据写入本地的同时，将待写入数据同步至同步副本集合中的其它备数据库实例，在确定同步副本集合包括的各备数据库实例反馈的写入成功的响应，且在本地也写入成功之后，向客户端反馈写入成功的响应。也就是说，在向备数据库实例同步待写入数据的过程中，任一备数据库实例异常，该异常的备数据库实例也会被剔除出同步副本集合，不会出现由于一个备数据库实例异常而导致待写入数据写入该异常的备数据库实例时失败，最终阻塞整个数据写请求，造成较大的业务问题。That is to say, when the primary database instance writes the data to be written locally, it also synchronizes the data to be written to other standby database instances in the synchronization replica set. Enter a successful response, and after the local is also successfully written, feedback the successful response to the client. That is to say, during the process of synchronizing data to be written to the standby database instance, if any standby database instance is abnormal, the abnormal standby database instance will also be removed from the synchronization replica set, and there will be no failure due to a standby database instance exception. As a result, the data to be written fails when it is written to the abnormal standby database instance, and eventually the entire data write request is blocked, causing major business problems.
示例性的，参阅图4所示，为本申请实施例提供的一种数据处理装置的结构示意图，该装置应用于数据库，该装置包括：For example, refer to FIG. 4, which is a schematic structural diagram of a data processing device provided by an embodiment of the present application. The device is applied to a database, and the device includes:
构建单元40，用于构建同步副本集合，所述同步副本集合包括主数据库实例和若干备数据库实例，其中，各备数据库实例与所述主数据库实例之间保持心跳检测，且心跳检测正常，若任一备数据库实例与所述主数据库之间的心跳检测异常，则将所述任一备数据库实例剔除出所述同步副本集合；The
接收单元41，用于接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例；The receiving
同步单元42，用于采用异步流复制/同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例。The
可选地，所述数据库还包括用于接入客户端的访问代理组件；用于监控同步副本集合，选举主数据库实例的哨兵组件；用于存储同步副本集合中各数据库实例的状态的后端存储组件，所述访问代理组件可访问所述后端存储组件；Optionally, the database also includes an access agent component for accessing the client; a sentinel component for monitoring the synchronous replica set and electing a master database instance; and a backend storage for storing the state of each database instance in the synchronous replica set component, the access proxy component can access the backend storage component;
构建同步副本集合时：When building a collection of synchronous replicas:
所述哨兵组件将第一个加入所述同步副本集合的数据库实例确定为主数据库实例，将其它加入所述同步副本集合的数据库实例确定为备数据库实例，其中，任一新加入所述同步副本集合的备数据库实例与所述主数据库实例建立心跳检测机制。The sentinel component determines the first database instance that joins the synchronous replica set as the primary database instance, and determines the other database instances that join the synchronous replica set as standby database instances, wherein any new database instance that joins the synchronous replica The aggregated standby database instance establishes a heartbeat detection mechanism with the primary database instance.
可选地，接收客户端发送的数据写请求，并基于所述数据写请求，将待写入数据写入所述同步副本集合的主数据库实例时：Optionally, when receiving the data write request sent by the client, and based on the data write request, when writing the data to be written into the primary database instance of the synchronous replica set:
所述访问代理组件接收客户端发送的数据写请求；The access proxy component receives the data write request sent by the client;
基于所述后端存储组件中存储的各数据库实例的状态，确定主数据库实例；determining the master database instance based on the status of each database instance stored in the back-end storage component;
将待写入数据写入所述同步副本集合的主数据库实例。Write the data to be written to the primary database instance of the set of synchronous replicas.
可选地，采用异步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例时：Optionally, when synchronizing the data to be written to each standby database instance of the synchronous replica set by means of asynchronous stream replication:
所述主数据库实例在确定待写入数据写入本地成功之后，向客户端反馈写入成功的响应，并将所述待写入数据同步至同步副本集合的各备数据库实例；After the primary database instance determines that the data to be written is successfully written locally, it feeds back a successful write response to the client, and synchronizes the data to be written to each standby database instance of the synchronization copy set;
采用同步流复制的方式，将所述待写入数据同步至所述同步副本集合的各备数据库实例时：When synchronous stream replication is used to synchronize the data to be written to each standby database instance of the synchronous replica set:
所述主数据库实例在将待写入数据写入本地时，将所述待写入数据同步至各备数据库实例，若确定所述待写入数据均成功写入所述同步副本集合的各数据库实例，则向客户端反馈写入成功的响应。When the primary database instance writes the data to be written locally, it synchronizes the data to be written to each standby database instance, and if it is determined that the data to be written is successfully written to each database of the synchronization copy set Instance, then feedback write successful response to the client.
以上这些单元可以是被配置成实施以上方法的一个或多个集成电路，例如：一个或多个特定集成电路(Application Specific Integrated Circuit，简称ASIC)，或，一个或多个微处理器(digital singnal processor，简称DSP)，或，一个或者多个现场可编程门阵列(Field Programmable Gate Array，简称FPGA)等。再如，当以上某个单元通过处理元件调度程序代码的形式实现时，该处理元件可以是通用处理器，例如中央处理器(CentralProcessing Unit，简称CPU)或其它可以调用程序代码的处理器。再如，这些单元可以集成在一起，以片上系统(system-on-a-chip，简称SOC)的形式实现。The above units may be one or more integrated circuits configured to implement the above method, for example: one or more specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), or, one or more microprocessors (digital signal processor, DSP for short), or, one or more Field Programmable Gate Arrays (Field Programmable Gate Array, FPGA for short), etc. For another example, when one of the above units is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (Central Processing Unit, CPU for short) or other processors that can call program codes. For another example, these units can be integrated together and implemented in the form of a system-on-a-chip (SOC for short).
进一步地，本申请实施例提供的数据处理装置，从硬件层面而言，所述数据处理装置的硬件架构示意图可以参见图5所示，所述数据处理装置可以包括：存储器50和处理器51，Further, for the data processing device provided in the embodiment of the present application, from the hardware level, the hardware architecture diagram of the data processing device can be referred to as shown in FIG. 5, and the data processing device can include: a
存储器50用于存储程序指令；处理器51调用存储器50中存储的程序指令，按照获得的程序指令执行上述方法实施例。具体实现方式和技术效果类似，这里不再赘述。The
可选地，本申请还提供一种数据库，包括用于执行上述方法实施例的至少一个处理元件(或芯片)。Optionally, the present application further provides a database, including at least one processing element (or chip) for executing the foregoing method embodiments.
可选地，本申请还提供一种程序产品，例如计算机可读存储介质，该计算机可读存储介质存储有计算机可执行指令，该计算机可执行指令用于使该计算机执行上述方法实施例。Optionally, the present application further provides a program product, such as a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to cause the computer to execute the foregoing method embodiments.
这里，机器可读存储介质可以是任何电子、磁性、光学或其它物理存储装置，可以包含或存储信息，如可执行指令、数据，等等。例如，机器可读存储介质可以是：RAM(RadomAccess Memory，随机存取存储器)、易失存储器、非易失性存储器、闪存、存储驱动器(如硬盘驱动器)、固态硬盘、任何类型的存储盘(如光盘、dvd等)，或者类似的存储介质，或者它们的组合。Here, a machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium can be: RAM (Radom Access Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage driver (such as hard disk drive), solid-state hard disk, any type of storage disk ( such as CD, DVD, etc.), or similar storage media, or a combination thereof.
上述实施例阐明的系统、装置、模块或单元，具体可以由计算机芯片或实体实现，或者由具有某种功能的产品来实现。一种典型的实现设备为计算机，计算机的具体形式可以是个人计算机、膝上型计算机、蜂窝电话、相机电话、智能电话、个人数字助理、媒体播放器、导航设备、电子邮件收发设备、游戏控制台、平板计算机、可穿戴设备或者这些设备中的任意几种设备的组合。The systems, devices, modules, or units described in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer, which may take the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, e-mail device, game control device, etc. desktops, tablets, wearables, or any combination of these.
为了描述的方便，描述以上装置时以功能分为各种单元分别描述。当然，在实施本申请时可以把各单元的功能在同一个或多个软件和/或硬件中实现。For the convenience of description, when describing the above devices, functions are divided into various units and described separately. Of course, when implementing the present application, the functions of each unit can be implemented in one or more pieces of software and/or hardware.
本领域内的技术人员应明白，本申请的实施例可提供为方法、系统、或计算机程序产品。因此，本申请可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本申请实施例可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
本申请是参照根据本申请实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述的。应理解可以由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其它可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其它可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.
而且，这些计算机程序指令也可以存储在能引导计算机或其它可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或者多个流程和/或方框图一个方框或者多个方框中指定的功能。Moreover, these computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, The instruction means implements the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.
这些计算机程序指令也可装载到计算机或其它可编程数据处理设备上，使得在计算机或者其它可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其它可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operational steps are performed on the computer or other programmable equipment to produce computer-implemented processing, so that the information executed on the computer or other programmable equipment The instructions provide steps for implementing the functions specified in the flow diagram procedure or procedures and/or block diagram procedures or blocks.
以上所述仅为本申请的较佳实施例而已，并不用以限制本申请，凡在本申请的精神和原则之内，所做的任何修改、等同替换、改进等，均应包含在本申请保护的范围之内。The above is only a preferred embodiment of the application, and is not intended to limit the application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the application should be included in the application. within the scope of protection.
Claims (10)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310350862.6ACN116401314A (en) | 2023-03-28 | 2023-03-28 | Data processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310350862.6ACN116401314A (en) | 2023-03-28 | 2023-03-28 | Data processing method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116401314Atrue CN116401314A (en) | 2023-07-07 |
Family
ID=87013718
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310350862.6APendingCN116401314A (en) | 2023-03-28 | 2023-03-28 | Data processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116401314A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050210073A1 (en)* | 2004-03-19 | 2005-09-22 | Takashi Oeda | System executing log data transfer synchronously and database data transfer asynchronously |
| US20110099420A1 (en)* | 2009-10-26 | 2011-04-28 | Macdonald Mcalister Grant Alexander | Failover and recovery for replicated data instances |
| CN109918360A (en)* | 2019-02-28 | 2019-06-21 | 携程旅游信息技术（上海）有限公司 | Database platform system, creation method, management method, equipment and storage medium |
| CN112217859A (en)* | 2020-08-28 | 2021-01-12 | 厦门快商通科技股份有限公司 | Load balancing method, system, mobile terminal and storage medium |
| CN115129779A (en)* | 2021-03-24 | 2022-09-30 | 腾讯科技（深圳）有限公司 | Database synchronization method, device and readable medium |
- 2023
- 2023-03-28CNCN202310350862.6Apatent/CN116401314A/enactivePending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20050210073A1 (en)* | 2004-03-19 | 2005-09-22 | Takashi Oeda | System executing log data transfer synchronously and database data transfer asynchronously |
| US20110099420A1 (en)* | 2009-10-26 | 2011-04-28 | Macdonald Mcalister Grant Alexander | Failover and recovery for replicated data instances |
| CN109918360A (en)* | 2019-02-28 | 2019-06-21 | 携程旅游信息技术（上海）有限公司 | Database platform system, creation method, management method, equipment and storage medium |
| CN112217859A (en)* | 2020-08-28 | 2021-01-12 | 厦门快商通科技股份有限公司 | Load balancing method, system, mobile terminal and storage medium |
| CN115129779A (en)* | 2021-03-24 | 2022-09-30 | 腾讯科技（深圳）有限公司 | Database synchronization method, device and readable medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9575849B2 (en) | Synchronized backup and recovery of database systems | |
| CN115292407B (en) | Synchronization method, device and storage medium | |
| US8874508B1 (en) | Systems and methods for enabling database disaster recovery using replicated volumes | |
| EP2052337B1 (en) | Retro-fitting synthetic full copies of data | |
| JP2025522399A (en) | End-to-end resumability of cross-region replication with new replication | |
| WO2019154394A1 (en) | Distributed database cluster system, data synchronization method and storage medium | |
| CN107402722B (en) | A data migration method and storage device | |
| US9996427B2 (en) | Parallel backup for distributed database system environments | |
| US20100023564A1 (en) | Synchronous replication for fault tolerance | |
| US8412674B2 (en) | Replication resynchronization | |
| US20240211488A1 (en) | Transaction commitment systems, methods, and apparatuses based on distributed database systems | |
| CN111314479A (en) | Data processing method and equipment | |
| US11157511B2 (en) | Physical replication of database | |
| US12007857B2 (en) | Non-blocking backup in a log replay node for tertiary initialization | |
| CN113987078B (en) | Data synchronization method, device and computer readable storage medium | |
| CN109783204A (en) | Distributed transaction processing method, device and storage medium | |
| US11748215B2 (en) | Log management method, server, and database system | |
| CN103970834A (en) | Recovery method for incremental data synchronization fault in isomerous database synchronizing system | |
| WO2019109257A1 (en) | Log management method, server and database system | |
| CN117785546A (en) | Database backup method, system and computing device cluster | |
| US20240152429A1 (en) | Recoverable Processes | |
| US8918364B1 (en) | Online mirror state transitioning in databases | |
| CN116401314A (en) | Data processing method and device | |
| US11507545B2 (en) | System and method for mirroring a file system journal | |
| CN107357536A (en) | Distributed memory system data modification write method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |