CN115168440A - Data reading and writing method, distributed storage system, device, equipment and storage medium - Google Patents
Data reading and writing method, distributed storage system, device, equipment and storage mediumDownload PDFInfo
- Publication number
- CN115168440A CN115168440ACN202210649472.4ACN202210649472ACN115168440ACN 115168440 ACN115168440 ACN 115168440ACN 202210649472 ACN202210649472 ACN 202210649472ACN 115168440 ACN115168440 ACN 115168440A
- Authority
- CN
- China
- Prior art keywords
- storage node
- target
- storage
- identification information
- target data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2471—Distributed queries
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Probability & Statistics with Applications (AREA)
- Fuzzy Systems (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本公开涉及人工智能领域，具体涉及云计算、云存储和内容分发技术，可应用在智能云场景下，尤其涉及一种数据读写方法、分布式存储系统、装置、设备和存储介质。The present disclosure relates to the field of artificial intelligence, in particular to cloud computing, cloud storage and content distribution technologies, which can be applied in intelligent cloud scenarios, and in particular to a data reading and writing method, a distributed storage system, an apparatus, a device and a storage medium.
背景技术Background technique
分布式存储技术是指将数据分散存储在多台机器上。典型的分布式存储系统包括：客户端、元数据服务器和数据存储服务器。数据存储服务器用于存储用户数据；元数据服务器用于存储用户数据的元数据，元数据例如为用户数据的存储位置信息；客户端是用户使用分布式存储系统的入口，基于用户的读写请求消息对目标数据进行读写操作。Distributed storage technology refers to the decentralized storage of data on multiple machines. A typical distributed storage system includes: client, metadata server and data storage server. The data storage server is used to store user data; the metadata server is used to store the metadata of the user data, such as the storage location information of the user data; the client is the user's entrance to the distributed storage system, based on the user's read and write requests Messages perform read and write operations on target data.
相关技术中，客户端可以从元数据服务器获取目标数据的存储位置信息，基于存储位置信息对目标数据进行读写操作。In the related art, the client can obtain the storage location information of the target data from the metadata server, and perform read and write operations on the target data based on the storage location information.
发明内容SUMMARY OF THE INVENTION
本公开提供了一种数据读写方法、分布式存储系统、装置、设备和存储介质。The present disclosure provides a data reading and writing method, a distributed storage system, an apparatus, a device and a storage medium.
根据本公开的一方面，提供了一种数据读写方法，包括：响应于针对目标数据的读写请求消息，确定所述目标数据所属的存储池内的多个存储节点，所述多个存储节点包括所述目标数据所在的目标存储节点，且，所述多个存储节点中的至少一个存储节点内存储所述目标存储节点的标识信息；查询所述多个存储节点，以获取所述目标存储节点的标识信息；基于所述目标存储节点的标识信息，将所述读写请求消息发送至所述目标存储节点，以对所述目标存储节点内的所述目标数据进行读写操作。According to an aspect of the present disclosure, a method for reading and writing data is provided, comprising: in response to a read-write request message for target data, determining a plurality of storage nodes in a storage pool to which the target data belongs, the plurality of storage nodes Including the target storage node where the target data is located, and at least one storage node in the plurality of storage nodes stores the identification information of the target storage node; querying the plurality of storage nodes to obtain the target storage node The identification information of the node; based on the identification information of the target storage node, the read and write request message is sent to the target storage node, so as to perform read and write operations on the target data in the target storage node.
根据本公开的另一方面，提供了一种分布式存储系统，包括：客户端和多个存储节点；所述多个存储节点包括目标数据所在的目标存储节点，且，所述多个存储节点中的至少一个存储节点内存储所述目标存储节点的标识信息；所述客户端，用于响应于针对所述目标数据的读写请求消息，确定所述目标数据所属的存储池内的所述多个存储节点；查询所述多个存储节点，以获取所述目标存储节点的标识信息；以及，基于所述目标存储节点的标识信息，将所述读写请求消息发送至所述目标存储节点，以对所述目标存储节点内的所述目标数据进行读写操作。According to another aspect of the present disclosure, a distributed storage system is provided, including: a client and multiple storage nodes; the multiple storage nodes include a target storage node where target data is located, and the multiple storage nodes The identification information of the target storage node is stored in at least one storage node of storage nodes; query the multiple storage nodes to obtain the identification information of the target storage node; and send the read and write request message to the target storage node based on the identification information of the target storage node, to perform read and write operations on the target data in the target storage node.
根据本公开的另一方面，提供了一种数据读写装置，包括：确定模块，用于响应于针对目标数据的读写请求消息，确定所述目标数据所属的存储池内的多个存储节点，所述多个存储节点包括所述目标数据所在的目标存储节点，且，所述多个存储节点中的至少一个存储节点内存储所述目标存储节点的标识信息；查询模块，用于查询所述多个存储节点，以获取所述目标存储节点的标识信息；读写模块，用于基于所述目标存储节点的标识信息，将所述读写请求消息发送至所述目标存储节点，以对所述目标存储节点内的所述目标数据进行读写操作。According to another aspect of the present disclosure, a data reading and writing device is provided, comprising: a determining module for determining, in response to a read-write request message for target data, a plurality of storage nodes in a storage pool to which the target data belongs, The plurality of storage nodes include a target storage node where the target data is located, and at least one storage node in the plurality of storage nodes stores identification information of the target storage node; a query module is configured to query the target storage node. a plurality of storage nodes, to obtain the identification information of the target storage node; a read-write module, configured to send the read-write request message to the target storage node based on the identification information of the target storage node, so as to Read and write operations are performed on the target data in the target storage node.
根据本公开的另一方面，提供了一种电子设备，包括：至少一个处理器；以及与所述至少一个处理器通信连接的存储器；其中，所述存储器存储有可被所述至少一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行如上述任一方面的任一项所述的方法。According to another aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores information that can be used by the at least one processor Instructions that are executed, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any of the preceding aspects.
根据本公开的另一方面，提供了一种存储有计算机指令的非瞬时计算机可读存储介质，其中，所述计算机指令用于使所述计算机执行根据上述任一方面的任一项所述的方法。According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to cause the computer to perform any one of the preceding aspects. method.
根据本公开的另一方面，提供了一种计算机程序产品，包括计算机程序，所述计算机程序在被处理器执行时实现根据上述任一方面的任一项所述的方法。According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method according to any of the preceding aspects.
根据本公开的技术方案，可以提高读写操作的可用性。According to the technical solutions of the present disclosure, the availability of read and write operations can be improved.
应当理解，本部分所描述的内容并非旨在标识本公开的实施例的关键或重要特征，也不用于限制本公开的范围。本公开的其它特征将通过以下的说明书而变得容易理解。It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.
附图说明Description of drawings
附图用于更好地理解本方案，不构成对本公开的限定。其中：The accompanying drawings are used for better understanding of the present solution, and do not constitute a limitation to the present disclosure. in:
图1是根据本公开第一实施例的示意图；1 is a schematic diagram according to a first embodiment of the present disclosure;
图2是本公开实施例对应的应用场景的示意图；2 is a schematic diagram of an application scenario corresponding to an embodiment of the present disclosure;
图3是本公开实施例中存储节点所存储的副本信息的示意图；3 is a schematic diagram of replica information stored by a storage node in an embodiment of the present disclosure;
图4是根据本公开第二实施例的示意图；4 is a schematic diagram according to a second embodiment of the present disclosure;
图5是本公开实施例中获取目标存储节点的标识信息的示意图；5 is a schematic diagram of acquiring identification information of a target storage node in an embodiment of the present disclosure;
图6是根据本公开第三实施例的示意图；6 is a schematic diagram according to a third embodiment of the present disclosure;
图7是根据本公开第四实施例的示意图；FIG. 7 is a schematic diagram according to a fourth embodiment of the present disclosure;
图8是用来实现本公开实施例的数据读写方法的电子设备的示意图。FIG. 8 is a schematic diagram of an electronic device used to implement the data reading and writing method according to an embodiment of the present disclosure.
具体实施方式Detailed ways
以下结合附图对本公开的示范性实施例做出说明，其中包括本公开实施例的各种细节以助于理解，应当将它们认为仅仅是示范性的。因此，本领域普通技术人员应当认识到，可以对这里描述的实施例做出各种改变和修改，而不会背离本公开的范围和精神。同样，为了清楚和简明，以下的描述中省略了对公知功能和结构的描述。Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.
相关技术中，客户端可以从元数据服务器获取目标数据的存储位置信息，基于存储位置信息对目标数据进行读写操作。目标数据的存储位置信息具体为目标数据所在的目标存储节点的标识信息。In the related art, the client can obtain the storage location information of the target data from the metadata server, and perform read and write operations on the target data based on the storage location information. The storage location information of the target data is specifically the identification information of the target storage node where the target data is located.
但是，若元数据服务器发生故障，客户端无法获取目标数据的存储位置信息，进而无法对目标数据进行读写操作，因此，客户端从元数据服务器获取目标数据的存储位置信息的方式，可用性较差。However, if the metadata server fails, the client cannot obtain the storage location information of the target data, and thus cannot perform read and write operations on the target data. Therefore, the method for the client to obtain the storage location information of the target data from the metadata server is more usable. Difference.
为了提高读写操作的可用性，本公开提供如下实施例。In order to improve the availability of read and write operations, the present disclosure provides the following embodiments.
图1是根据本公开第一实施例的示意图，本实施例提供一种数据读写方法，该方法包括：FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure. This embodiment provides a method for reading and writing data, and the method includes:
101、响应于针对目标数据的读写请求消息，确定所述目标数据所属的存储池内的多个存储节点，所述多个存储节点包括所述目标数据所在的目标存储节点，且，所述多个存储节点中的至少一个存储节点内存储所述目标存储节点的标识信息。101. In response to a read/write request message for target data, determine multiple storage nodes in the storage pool to which the target data belongs, where the multiple storage nodes include the target storage node where the target data is located, and the multiple storage nodes include: The identification information of the target storage node is stored in at least one of the storage nodes.
102、查询所述多个存储节点，以获取所述目标存储节点的标识信息。102. Query the multiple storage nodes to obtain identification information of the target storage node.
103、基于所述目标存储节点的标识信息，将所述读写请求消息发送至所述目标存储节点，以对所述目标存储节点内的所述目标数据进行读写操作。103. Based on the identification information of the target storage node, send the read and write request message to the target storage node, so as to perform read and write operations on the target data in the target storage node.
本实施例的数据读写方法可以由分布式存储系统中的客户端执行。The data reading and writing method in this embodiment may be executed by a client in a distributed storage system.
读写请求消息可以来自于用户终端，即，用户需要对目标数据进行读写操作时，可以通过用户终端向分布式存储系统的客户端发送读写请求消息。用户终端可以包括：个人电脑(Personal Computer、PC)、移动设备、智能家居设备、智能家居设备、可穿戴式设备等，移动设备例如包括手机、便携式电脑、平板电脑等，智能家居设备例如包括智能音箱、智能电视等，可穿戴式设备例如包括智能手表、智能眼镜等。The read/write request message may come from the user terminal, that is, when the user needs to perform read/write operations on the target data, the read/write request message may be sent to the client of the distributed storage system through the user terminal. The user terminal may include: a personal computer (Personal Computer, PC), a mobile device, a smart home device, a smart home device, a wearable device, etc. The mobile device includes, for example, a mobile phone, a portable computer, a tablet computer, etc., and the smart home device includes, for example, a smart Speakers, smart TVs, etc., wearable devices include smart watches, smart glasses, etc., for example.
目标数据，是指用户待读写操作的数据，例如，网页数据、游戏数据、音视频数据等。The target data refers to the data to be read and written by the user, for example, web page data, game data, audio and video data, and so on.
读写请求消息中可以包含目标数据的标识信息，以表明是针对目标数据的读写请求消息。The read/write request message may contain identification information of the target data to indicate that it is a read/write request message for the target data.
目标数据的标识信息，例如为统一资源定位器(Uniform Resource Locator，URL)地址，或者，也可以为目标数据所在的应用(Application，APP)的名称等。The identification information of the target data is, for example, a Uniform Resource Locator (Uniform Resource Locator, URL) address, or may also be the name of an application (Application, APP) where the target data is located.
客户端内可以预先配置数据与存储池(pool)之间的映射关系，基于该映射关系，可以确定目标数据所属的存储池。The mapping relationship between the data and the storage pool (pool) can be preconfigured in the client, and based on the mapping relationship, the storage pool to which the target data belongs can be determined.
其中，每个存储池内可以包括多个存储节点，存储节点可以具体为服务器、磁盘等，每个存储池内的多个存储节点是可更新的。Wherein, each storage pool may include multiple storage nodes, and the storage nodes may specifically be servers, disks, etc., and the multiple storage nodes in each storage pool can be updated.
客户端可以获取存储池与存储池内的多个存储节点之间的映射关系，因此，基于该映射关系，可以确定目标数据所属的存储池内的多个存储节点。The client can obtain the mapping relationship between the storage pool and multiple storage nodes in the storage pool. Therefore, based on the mapping relationship, multiple storage nodes in the storage pool to which the target data belongs can be determined.
由于多个存储节点中的至少一个存储节点内存储目标存储节点的标识信息，因此，可以通过对多个存储节点进行查询，以获取目标存储节点的标识信息。Since the identification information of the target storage node is stored in at least one of the multiple storage nodes, the identification information of the target storage node can be acquired by querying the multiple storage nodes.
其中，存储节点可以用互联网协议地址(Internet Protocol，IP)地址和端口号，或者，编号，或者，名称等进行标识。The storage node may be identified by an Internet Protocol (Internet Protocol, IP) address and a port number, or a number, or a name, and the like.
以用IP地址和端口号进行标识为例，目标存储节点的标识信息为目标存储节点的IP地址和端口号。Taking the identification by an IP address and a port number as an example, the identification information of the target storage node is the IP address and port number of the target storage node.
获取目标存储节点的标识信息后，可以基于该标识信息将读写请求消息发送至对应的目标存储节点，对目标数据进行读写操作。After acquiring the identification information of the target storage node, a read and write request message may be sent to the corresponding target storage node based on the identification information, and the target data can be read and written.
读写操作例如可以包括：读取、写入、增加、修改、删除等操作。The read and write operations may include, for example, operations such as reading, writing, adding, modifying, and deleting.
本实施例中，通过查询存储节点，以获取目标存储节点的标识信息，可以在元服务器不可用时依然可以获取到目标存储节点的标识信息，进而可以基于目标存储节点的标识信息将读写请求消息发送至目标存储节点，以对目标存储节点内的目标数据进行读写操作，提高读写操作的可用性。In this embodiment, by querying the storage node to obtain the identification information of the target storage node, the identification information of the target storage node can still be obtained when the meta server is unavailable, and then the read and write request messages can be read and written based on the identification information of the target storage node. It is sent to the target storage node to perform read and write operations on the target data in the target storage node to improve the availability of read and write operations.
为了更好地理解本公开实施例，对本公开实施例的应用场景进行说明。In order to better understand the embodiments of the present disclosure, application scenarios of the embodiments of the present disclosure are described.
图2是本公开实施例对应的应用场景的示意图。本实施例以基于Raft的分布式存储系统为例。Raft是一种简单方便易于理解的分布式算法，主要解决了分布式中的一致性问题。FIG. 2 is a schematic diagram of an application scenario corresponding to an embodiment of the present disclosure. This embodiment takes a Raft-based distributed storage system as an example. Raft is a simple, convenient and easy-to-understand distributed algorithm that mainly solves the problem of consistency in distributed.
如图2所示，基于Raft的分布式存储系统包括：客户端(client)201、元数据服务器202和存储节点(blockserver)，多个存储节点可以组成存储池203。As shown in FIG. 2 , the Raft-based distributed storage system includes: a client (client) 201 , a
分布式存储场景下，一份数据可以复制为多份，每份数据可以称为一个副本，多个副本组成一个复制组(Replicate Group，RG)，每个副本存储在一个存储节点上。In a distributed storage scenario, a piece of data can be replicated into multiple copies, each of which can be called a copy, and multiple copies form a Replicate Group (RG), and each copy is stored on a storage node.
组成RG的多个副本可以包括主副本(leader)和从副本(follower)，主副本为一个，从副本为多个，例如，每个RG包括一个主副本和两个从副本。The multiple replicas constituting an RG may include a master replica (leader) and a slave replica (follower), with one master replica and multiple slave replicas, for example, each RG includes one master replica and two slave replicas.
如图2所示，不同的RG分别用rg0、rg1、rg2等表示，以rg0为例，同一个RG的副本用rg0_#1、rg0_#2、rg0_#3表示。As shown in Figure 2, different RGs are represented by rg0, rg1, rg2, etc., respectively. Taking rg0 as an example, the copies of the same RG are represented by
多个存储节点可以组成一个存储池(pool)，图2中以一个存储池内包括第一存储节点、第二存储节点、第三存储节点和第四存储节点为例。A plurality of storage nodes may form a storage pool (pool). In FIG. 2 , a storage pool including a first storage node, a second storage node, a third storage node and a fourth storage node is taken as an example.
用户需要对目标数据进行读写操作时，可以通过用户终端发送读写请求消息，读写请求消息中可以包含目标数据的标识信息。When the user needs to perform read and write operations on the target data, a read and write request message may be sent through the user terminal, and the read and write request message may include identification information of the target data.
客户端内可以预先配置数据的标识信息与存储池的标识信息之间的映射关系，因此，基于该映射关系，可以确定目标数据所属的存储池。The mapping relationship between the identification information of the data and the identification information of the storage pool can be pre-configured in the client. Therefore, based on the mapping relationship, the storage pool to which the target data belongs can be determined.
客户端还可以定期从元数据服务器获取存储池与存储池内的多个存储节点之间的映射关系，之后再基于该映射关系，可以确定目标数据所属的存储池内的多个存储节点。The client can also periodically obtain the mapping relationship between the storage pool and multiple storage nodes in the storage pool from the metadata server, and then, based on the mapping relationship, can determine multiple storage nodes in the storage pool to which the target data belongs.
针对不同的目标数据，若目标数据所属的存储池为同一个，则在相同周期内，即使目标数据不同，目标数据所属的存储池内的多个存储节点也是相同的。例如，如图2所示，目标数据为rg0或rg1或rg2，由于rg0～rg2对应相同的存储池，则这些目标数据对应的多个存储节点均为图2所示的第一存储节点～第五存储节点。For different target data, if the target data belongs to the same storage pool, in the same period, even if the target data is different, the multiple storage nodes in the storage pool to which the target data belongs are the same. For example, as shown in Figure 2, the target data is rg0, rg1 or rg2. Since rg0 to rg2 correspond to the same storage pool, the multiple storage nodes corresponding to these target data are the first storage node to the second storage node shown in Figure 2. Five storage nodes.
针对同一个RG，该RG内的各个副本所在的存储节点上可以存储主副本和从副本的信息，该信息用于指示哪个副本为主副本，哪个副本为从副本，以及，用于指示主副本以及从副本所在的存储节点。以rg0为例，如图2所示，rg0的三个副本分别位于第一存储节点、第三存储节点和第四存储节点上，则第一存储节点、第三存储节点和第四存储节点内均存储rg0的所有副本的信息。For the same RG, information about the master copy and the slave copy can be stored on the storage node where each copy in the RG is located, and the information is used to indicate which copy is the master copy, which copy is the slave copy, and is used to indicate the master copy. and the storage node where the slave replica is located. Taking rg0 as an example, as shown in Figure 2, the three copies of rg0 are located on the first storage node, the third storage node and the fourth storage node respectively, then the first storage node, the third storage node and the fourth storage node Both store information on all replicas of rg0.
其中，rg0的所有副本的信息可以如图3所示，可以记录各个副本是主副本或从副本，以及，主副本和从副本所在的存储节点的信息。Wherein, the information of all the replicas of rg0 may be as shown in FIG. 3 , which may record whether each replica is a master copy or a slave copy, and the information of the storage nodes where the master copy and the slave copy are located.
若目标数据是包括多个副本的数据，例如RG中的某个副本，则目标存储节点是指主副本所在的存储节点。以rg0为例，假设rg0_#1为主副本，由于rg0_#1位于第一存储节点上，则针对目标数据是rg0时，目标存储节点是第一存储节点。If the target data is data including multiple copies, such as a copy in the RG, the target storage node refers to the storage node where the primary copy is located. Taking rg0 as an example, assuming that
另外，为了保持数据一致性，还可以对从副本进行读写操作，以rg0为例，假设rg0_#2和rg0_#3是从副本，由于rg0_#2位于第三存储节点上，rg0_#3位于第四存储节点上，客户端除了将读写请求消息发送至主副本所在的第一存储节点上，对主副本进行读写操作之外，还可以将读写请求消息发送至从副本所在的第二存储节点和第三存储节点上，对从副本进行读写操作。从而可以保持rg0的各个副本的数据一致性。In addition, in order to maintain data consistency, you can also perform read and write operations on the slave copy. Take rg0 as an example, assuming that
结合上述应用场景，本公开实施例还提供一种数据读写方法。In combination with the above application scenarios, an embodiment of the present disclosure further provides a method for reading and writing data.
图4是根据本公开第二实施例的示意图，本实施例提供一种数据读写方法，该方法包括：FIG. 4 is a schematic diagram according to a second embodiment of the present disclosure. This embodiment provides a method for reading and writing data, and the method includes:
401、客户端，基于预设周期从元数据服务器获取存储池与存储池内的多个存储节点之间的映射关系。401. A client, based on a preset period, obtains a mapping relationship between a storage pool and multiple storage nodes in the storage pool from a metadata server.
其中，由于存储池与其内的存储节点之间的映射关系的更改并不频繁，因此，预设周期可以不用很频繁，例如，预设周期为1小时，从而可以降低对元数据服务器的干扰。Among them, since the mapping relationship between the storage pool and the storage nodes in it is not changed frequently, the preset period may not be very frequent, for example, the preset period is 1 hour, so as to reduce the interference to the metadata server.
可以理解的是，在一些场景下，若存储池与其内的存储节点的关系更为固定，则可以直接将该映射关系配置在客户端内。It can be understood that, in some scenarios, if the relationship between the storage pool and the storage node in it is more fixed, the mapping relationship can be directly configured in the client.
本实施例中，通过基于预设周期从元数据服务器获取上述的映射关系，可以保证映射关系的准确度，提高读写操作的可靠性。In this embodiment, by acquiring the above-mentioned mapping relationship from the metadata server based on a preset period, the accuracy of the mapping relationship can be ensured, and the reliability of read and write operations can be improved.
402、客户端，接收到用户终端发送的读写请求消息后，判断本地或元数据服务器内是否存储目标存储节点的标识信息，若是，执行403，否则，执行404。402. The client, after receiving the read and write request message sent by the user terminal, determines whether the identification information of the target storage node is stored locally or in the metadata server, and if so, executes 403; otherwise, executes 404.
其中，用户需要对目标数据进行读写操作时，可以通过用户终端发送读写请求消息，读写请求消息中可以包含目标数据的标识信息。Wherein, when the user needs to perform read and write operations on the target data, a read and write request message may be sent through the user terminal, and the read and write request message may include identification information of the target data.
目标数据，是指用户待读写操作的数据，例如，网页数据、游戏数据、音视频数据等。The target data refers to the data to be read and written by the user, for example, web page data, game data, audio and video data, and so on.
目标数据的标识信息，例如为URL地址等。The identification information of the target data, such as a URL address, etc.
为了提高读写效率，若客户端获取过目标数据，客户端会在本地缓存目标数据的标识信息以及目标存储节点的标识信息之间的映射关系，之后可以基于本地缓存的信息直接获取目标存储节点的标识信息，而不需要向元数据服务器或者存储节点进行查询。In order to improve the efficiency of reading and writing, if the client has obtained the target data, the client will cache the mapping relationship between the identification information of the target data and the identification information of the target storage node locally, and then can directly obtain the target storage node based on the locally cached information. It does not need to query the metadata server or storage node.
若客户端未缓存目标数据对应的目标存储节点的标识信息，可以向元数据服务器进行查询。If the client does not cache the identification information of the target storage node corresponding to the target data, it can query the metadata server.
其中，目标数据的分布情况是变化较为频繁的，以rg0为例，rg0具体分布在哪个或哪些存储节点上是变化较为频繁的，为了保证准确性，需要在接收到读写请求消息后确定当前的目标数据所在的存储节点。Among them, the distribution of target data changes frequently. Taking rg0 as an example, which storage node or nodes where rg0 is distributed changes frequently. The storage node where the target data resides.
一般来讲，元数据服务器内会记录当前的目标数据所在的存储节点的信息，因此，客户端可以向元数据服务器查询目标存储节点的标识信息。Generally speaking, the metadata server records the information of the storage node where the current target data is located. Therefore, the client can query the metadata server for the identification information of the target storage node.
但是，若元数据服务器发生故障后，客户端无法从元数据服务器获取到目标存储节点的标识信息。However, if the metadata server fails, the client cannot obtain the identification information of the target storage node from the metadata server.
403、客户端，从本地或者元数据服务器，获取目标存储节点的标识信息。403. The client obtains the identification information of the target storage node from the local or the metadata server.
404、客户端，基于存储池与存储池内的多个存储节点之间的映射关系，确定所述目标数据所属的存储池内的多个存储节点。404. The client, based on the mapping relationship between the storage pool and the multiple storage nodes in the storage pool, determines multiple storage nodes in the storage pool to which the target data belongs.
其中，以图2所示的结构为例，目标数据所属的存储池内的多个存储节点分别为第一存储节点～第四存储节点。Wherein, taking the structure shown in FIG. 2 as an example, the multiple storage nodes in the storage pool to which the target data belongs are respectively the first storage node to the fourth storage node.
405、客户端，依次向所述多个存储节点中的单个存储节点发送第一查询请求消息，所述第一查询请求消息用于查询所述单个存储节点内是否存储所述目标数据。405. The client sends a first query request message to a single storage node among the plurality of storage nodes in sequence, where the first query request message is used to query whether the target data is stored in the single storage node.
其中，客户端依次向单个存储节点发送查询请求消息，是指按设定顺序对每个存储节点进行查询，避免遗漏，例如，可以向第一存储节点发送查询请求消息，若第一存储节点内未存储目标数据，则向第二存储节点发送查询请求消息，若第二存储节点内未存储目标数据，则向第三存储节点发送查询请求消息，依此类推。Wherein, the client sends a query request message to a single storage node in turn, which means to query each storage node in a set order to avoid omissions. For example, a query request message can be sent to the first storage node. If the target data is not stored, a query request message is sent to the second storage node, if the target data is not stored in the second storage node, a query request message is sent to the third storage node, and so on.
为了区分，用于查询是否存储目标数据的查询请求消息可以称为第一查询请求消息。For distinction, a query request message for querying whether target data is stored may be referred to as a first query request message.
406、接收到查询请求消息的存储节点，判断自身是否存储所述目标数据，若是，执行408，否则执行407。406 . The storage node that has received the query request message determines whether it stores the target data, and if so, executes 408 , otherwise executes 407 .
其中，若目标数据存在多个副本，则任一副本所在的存储节点内均存储目标数据。Wherein, if there are multiple copies of the target data, the target data is stored in the storage node where any copy is located.
以目标数据是rg0为例，rg0对应三个副本，分别是rg0_#1、rg0_#2和rg0_#3，相应的存储节点是第一存储节点、第四存储节点和第三存储节点，则表明第一存储节点、第四存储节点和第三存储节点内均存储目标数据。Taking the target data as rg0 as an example, rg0 corresponds to three copies, namely
407、接收到第一查询请求消息的存储节点，向客户端发送第一查询响应消息，所述第一查询响应消息用于表明未存储所述目标数据。407. The storage node that receives the first query request message sends a first query response message to the client, where the first query response message is used to indicate that the target data is not stored.
其中，与第一查询请求消息对应的响应可以称为第一查询响应消息。The response corresponding to the first query request message may be referred to as a first query response message.
结合图2所示的架构，例如，目标数据是rg1，客户端向第一存储节点发送第一查询请求消息，第一查询请求消息中包含rg1的标识信息，由于第一存储节点内未存储rg1的任一副本，此时，第一存储节点发送的第一查询响应消息中可以包含用于表明未存储目标数据的指示信息(如no exist)。Combined with the architecture shown in Figure 2, for example, the target data is rg1, the client sends a first query request message to the first storage node, and the first query request message contains the identification information of rg1, because rg1 is not stored in the first storage node. In this case, the first query response message sent by the first storage node may include indication information (such as no exist) for indicating that the target data is not stored.
之后，客户端可以向下一个存储节点继续发送第一查询请求消息，直至查询到用于表明存储所述目标数据的第一查询响应消息。After that, the client may continue to send the first query request message to the next storage node until the first query response message indicating that the target data is stored is obtained.
本实施例中，通过依次向单个存储节点发送第一查询请求消息，直至接收到用于表明存储所述目标数据的第一查询响应消息，可以基于所述用于表明存储所述目标数据的第一查询响应消息，获取目标存储节点的标识信息，进而可以基于目标存储节点的标识信息对目标数据进行读写操作，保证读写操作的正常执行。In this embodiment, by sequentially sending a first query request message to a single storage node until a first query response message indicating storage of the target data is received, the first query response message indicating storage of the target data can be A query response message is sent to obtain the identification information of the target storage node, and then read and write operations can be performed on the target data based on the identification information of the target storage node to ensure normal execution of the read and write operations.
408、接收到第一查询请求消息的存储节点，向客户端发送第一查询响应消息，所述第一查询响应消息用于表明存储所述目标数据。408. The storage node that receives the first query request message sends a first query response message to the client, where the first query response message is used to indicate that the target data is stored.
结合图2所示的架构，例如，目标数据是rg0，客户端向第一存储节点发送第一查询请求消息，第一查询请求消息中包含rg0的标识信息，由于第一存储节点内存储有rg0，此时，第一存储节点发送的第一查询响应消息中可以包含用于表明存储目标数据的指示信息(如exist)。With reference to the architecture shown in Figure 2, for example, the target data is rg0, the client sends a first query request message to the first storage node, and the first query request message contains the identification information of rg0, because the first storage node stores rg0 , at this time, the first query response message sent by the first storage node may include indication information (eg exist) for indicating the storage target data.
409、客户端，基于所述第一查询响应消息，获取所述目标存储节点的标识信息。409. The client, based on the first query response message, obtains the identification information of the target storage node.
其中，针对408的存储节点是主副本所在的存储节点或者从副本所在的存储节点，第一查询响应消息中可以包含不同内容，进而可以采用不同的方式获取目标存储节点标识信息。具体内容可以参见后续描述。Wherein, for the storage node in 408 is the storage node where the master copy is located or the storage node where the slave copy is located, the first query response message may contain different contents, and then the identification information of the target storage node may be obtained in different ways. For details, please refer to the subsequent description.
另外，客户端获取到目标存储节点的标识信息后，还可以存储目标数据与目标存储节点之间的映射关系，从而后续还需要对该目标数据进行读写操作时，可以直接基于客户端本地缓存的映射关系获取目标存储节点的标识信息。In addition, after the client obtains the identification information of the target storage node, it can also store the mapping relationship between the target data and the target storage node, so that when the target data needs to be read and written in the future, it can be directly based on the client's local cache. to obtain the identification information of the target storage node.
410、客户端，基于所述目标存储节点的标识信息，将所述读写请求消息发送至所述目标存储节点，以对所述目标存储节点内的所述目标数据进行读写操作。410. The client, based on the identification information of the target storage node, sends the read and write request message to the target storage node, so as to perform read and write operations on the target data in the target storage node.
其中，针对目标数据存在多个副本的情况，目标存储节点包括：主副本所在的存储节点，以及从副本所在的存储节点，之后，可以向主副本和从副本所在的存储节点均发送读写请求消息，以对各个副本所在的存储节点内的目标数据对应的各个副本进行读写操作。Among them, for the situation where there are multiple copies of the target data, the target storage node includes: the storage node where the primary copy is located, and the storage node where the secondary copy is located, and then, read and write requests can be sent to both the storage node where the primary copy and the secondary copy are located. message to perform read and write operations on each copy corresponding to the target data in the storage node where each copy is located.
例如，目标数据是rg0，rg0的各个副本分别位于第一存储节点、第三存储节点和第四存储节点上，因此，结合图2，客户端可以分别向第一存储节点、第三存储节点和第四存储节点发送读写请求消息，以对rg0的各个副本(rg0_#1、rg0_#3、rg0_#2)进行读写操作。For example, the target data is rg0, and each copy of rg0 is located on the first storage node, the third storage node and the fourth storage node, respectively. Therefore, in conjunction with FIG. 2, the client can send the first storage node, the third storage node and The fourth storage node sends a read/write request message to perform read/write operations on each copy of rg0 (
其中，如图5所示，基于第一查询响应消息获取目标存储节点的标识信息可以包括：Wherein, as shown in FIG. 5 , obtaining the identification information of the target storage node based on the first query response message may include:
501、存储目标数据的存储节点，判断自身是否为所述目标数据对应的主副本所在的存储节点，若是，执行502，否则，执行504。501. The storage node storing the target data determines whether it is the storage node where the primary copy corresponding to the target data is located, if so, execute 502; otherwise, execute 504.
其中，以目标数据是rg0为例，rg0的各个副本所在的存储节点内均会记录rg0的所有副本的信息，例如，哪个副本是主副本，哪个副本是从副本，主副本以及从副本所在的存储节点的标识信息等，因此，存储目标数据的存储节点基于记录的目标数据对应的所有副本的信息，可以确定自身是主副本所在的存储节点或者从副本所在的存储节点，并且若自身是从副本所在的存储节点，还可以获取到主副本所在的存储节点的标识信息。Among them, taking the target data as rg0 as an example, the storage node where each copy of rg0 is located will record the information of all copies of rg0, for example, which copy is the master copy, which copy is the slave copy, where the master copy and the slave copy are located The identification information of the storage node, etc. Therefore, the storage node storing the target data can determine that it is the storage node where the master copy is located or the storage node where the slave copy is located based on the information of all copies corresponding to the recorded target data. The storage node where the replica is located can also obtain the identification information of the storage node where the primary replica is located.
502、存储目标数据的存储节点，向客户端反馈第一查询响应消息，所述第一查询响应消息中包含所述目标数据对应的所有副本所在的存储节点的标识信息。502. The storage node storing the target data feeds back a first query response message to the client, where the first query response message includes identification information of the storage node where all copies corresponding to the target data are located.
503、客户端，从所述第一查询响应消息中获取所述目标数据对应的所有副本所在的存储节点的标识信息，作为目标存储节点的标识信息。503. The client obtains, from the first query response message, the identification information of the storage nodes where all the replicas corresponding to the target data are located, as the identification information of the target storage node.
例如，以目标数据是rg0为例，第一存储节点反馈的第一查询响应消息中不仅可以包含用于指示存储目标数据的指示信息，还可以包含rg0的所有副本所在的存储节点的标识信息，即第一存储节点的标识信息、第三存储节点的标识信息和第四存储节点的标识信息，之后，客户端将第一存储节点的标识信息、第三存储节点的标识信息和第四存储节点的标识信息作为目标存储节点的标识信息。For example, taking the target data as rg0 as an example, the first query response message fed back by the first storage node may include not only the indication information for indicating the storage of the target data, but also the identification information of the storage node where all copies of rg0 are located, That is, the identification information of the first storage node, the identification information of the third storage node and the identification information of the fourth storage node, after that, the client sends the identification information of the first storage node, the identification information of the third storage node and the fourth storage node The identification information of the target storage node is used as the identification information of the target storage node.
本实施例中，若查询到主副本所在的存储节点，主副本所在的存储节点可以将所有副本所在的存储节点的标识信息反馈给客户端，保证标识信息的准确度和获取效率，进而提高读写操作的准确性和效率。In this embodiment, if the storage node where the primary copy is located is queried, the storage node where the primary copy is located can feed back the identification information of the storage nodes where all the copies are located to the client, so as to ensure the accuracy and acquisition efficiency of the identification information, thereby improving the readability Accuracy and efficiency of write operations.
504、存储目标数据的存储节点，向客户端反馈第一查询响应消息，所述第一查询响应消息中包含所述目标数据对应的主副本所在的存储节点的标识信息。504. The storage node storing the target data feeds back a first query response message to the client, where the first query response message includes identification information of the storage node where the primary copy corresponding to the target data is located.
505、客户端，从所述第一查询响应消息中获取所述主副本所在的存储节点的标识信息，基于所述主副本所在的存储节点的标识信息，向所述主副本所在的存储节点发送第二查询请求消息，所述第二查询请求消息用于查询所述主副本所在的存储节点内存储的目标存储节点的标识信息。505. The client obtains the identification information of the storage node where the primary copy is located from the first query response message, and sends the identification information to the storage node where the primary copy is located based on the identification information of the storage node where the primary copy is located. A second query request message, where the second query request message is used to query the identification information of the target storage node stored in the storage node where the primary replica is located.
结合图2所示的架构，以目标数据是rg2为例，假设rg2的主副本是rg2_#1，从副本是rg2_#2和rg2_#3，由于第一存储节点存储的是rg2_#2，是从副本，则第一存储节点反馈的第一查询响应消息中不仅可以包含用于指示存储目标数据的指示信息，还包含rg2对应的主副本(即rg2_#1)所在的存储节点(即第三存储节点)的标识信息。Combined with the architecture shown in Figure 2, taking the target data as rg2 as an example, it is assumed that the master copy of rg2 is
客户端从第一查询响应消息中获取第三存储节点的标识信息后，可以向第三存储节点发送第二查询请求消息。After acquiring the identification information of the third storage node from the first query response message, the client may send a second query request message to the third storage node.
506、主副本所在的存储节点，响应于第二查询请求消息，向客户端反馈第二查询响应消息，所述第二查询响应消息中包含所述目标数据对应的所有副本所在的存储节点的标识信息。506. The storage node where the primary replica is located, in response to the second query request message, feeds back a second query response message to the client, where the second query response message includes the identifiers of the storage nodes where all replicas corresponding to the target data are located information.
例如，针对目标数据rg2，假设第三存储节点内存储的目标存储节点的标识信息分别是第一存储节点的标识信息、第二存储节点的标识信息和第三存储节点的标识信息，则第二查询响应消息中可以包含上述三个存储节点的标识信息。For example, for the target data rg2, assuming that the identification information of the target storage node stored in the third storage node is the identification information of the first storage node, the identification information of the second storage node and the identification information of the third storage node, then the second storage node The query response message may contain the identification information of the above three storage nodes.
由于主副本所在的存储节点中的信息是最新的，因此，从主副本所在的存储节点获取目标存储节点标识信息，可以保证所获取的目标存储节点的标识信息是最新的，提高准确性，进而保证读写操作的准确性。Since the information in the storage node where the primary copy is located is the latest, obtaining the identification information of the target storage node from the storage node where the primary copy is located can ensure that the acquired identification information of the target storage node is up-to-date, improve the accuracy, and further improve the accuracy. Ensure the accuracy of read and write operations.
507、客户端，从所述第二查询响应消息中获取所述目标数据对应的所有副本所在的存储节点的标识信息，作为目标存储节点的标识信息。507. The client obtains, from the second query response message, the identification information of the storage nodes where all the replicas corresponding to the target data are located, as the identification information of the target storage node.
例如，针对rg2，可以获取目标存储节点的标识信息为：第一存储节点的标识信息、第二存储节点的标识信息和第三存储节点的标识信息。For example, for rg2, the identification information of the target storage node may be obtained as follows: identification information of the first storage node, identification information of the second storage node, and identification information of the third storage node.
本实施例中，若查询到从副本所在的存储节点，客户端还向主副本所在的存储节点查询目标存储节点的标识信息，可以保证获取到最新的标识信息，保证标识信息的准确度，进而提高读写操作的准确性。In this embodiment, if the storage node where the secondary copy is located is queried, the client also queries the storage node where the primary copy is located for the identification information of the target storage node, which can ensure the acquisition of the latest identification information, ensure the accuracy of the identification information, and further Improve the accuracy of read and write operations.
图6是根据本公开第三实施例的示意图，本实施例提供一种分布式存储系统，该系统600包括：客户端601和多个存储节点602。FIG. 6 is a schematic diagram according to a third embodiment of the present disclosure. This embodiment provides a distributed storage system. The
所述多个存储节点602包括目标数据所在的目标存储节点，且，所述多个存储节点中的至少一个存储节点内存储所述目标存储节点的标识信息；所述客户端601用于响应于针对所述目标数据的读写请求消息，确定所述目标数据所属的存储池内的所述多个存储节点；查询所述多个存储节点，以获取所述目标存储节点的标识信息；以及，基于所述目标存储节点的标识信息，将所述读写请求消息发送至所述目标存储节点，以对所述目标存储节点内的所述目标数据进行读写操作。The
本实施例中，通过查询存储节点，以获取目标存储节点的标识信息，可以在元服务器不可用时依然可以获取到目标存储节点的标识信息，进而可以基于目标存储节点的标识信息将读写请求消息发送至目标存储节点，以对目标存储节点内的目标数据进行读写操作，提高读写操作的可用性。In this embodiment, by querying the storage node to obtain the identification information of the target storage node, the identification information of the target storage node can still be obtained when the meta server is unavailable, and then the read and write request messages can be read and written based on the identification information of the target storage node. It is sent to the target storage node to perform read and write operations on the target data in the target storage node to improve the availability of read and write operations.
一些实施例中，所述系统600还包括：元数据服务器。In some embodiments, the
所述元数据服务器用于记录存储池与存储池内的多个存储节点之间的映射关系；所述客户端601进一步用于：基于预设周期，从所述元数据服务器获取所述映射关系；以及，若所述客户端本地未缓存所述目标存储节点的标识信息，且，从所述元数据服务器未获取所述目标存储节点的标识信息，基于所述映射关系，确定所述多个存储节点。The metadata server is configured to record the mapping relationship between the storage pool and multiple storage nodes in the storage pool; the
本实施例中，通过基于预设周期从元数据服务器获取上述的映射关系，可以保证映射关系的准确度，提高读写操作的可靠性。In this embodiment, by acquiring the above-mentioned mapping relationship from the metadata server based on a preset period, the accuracy of the mapping relationship can be ensured, and the reliability of read and write operations can be improved.
一些实施例中，所述客户端601进一步用于：依次向所述多个存储节点中的单个存储节点发送第一查询请求消息，所述第一查询请求消息用于查询所述单个存储节点内是否存储所述目标数据，直至接收到第一查询响应消息，所述第一查询响应消息用于表明存储所述目标数据；In some embodiments, the
基于所述第一查询响应消息，获取所述目标存储节点的标识信息。Based on the first query response message, the identification information of the target storage node is acquired.
本实施例中，通过依次向单个存储节点发送第一查询请求消息，直至接收到用于表明存储所述目标数据的第一查询响应消息，可以基于所述用于表明存储所述目标数据的第一查询响应消息，获取目标存储节点的标识信息，进而可以基于目标存储节点的标识信息对目标数据进行读写操作，保证读写操作的正常执行。In this embodiment, by sequentially sending a first query request message to a single storage node until a first query response message indicating storage of the target data is received, the first query response message indicating storage of the target data can be A query response message is sent to obtain the identification information of the target storage node, and then read and write operations can be performed on the target data based on the identification information of the target storage node to ensure normal execution of the read and write operations.
一些实施例中，所述目标数据存在多个副本，所述多个副本包括：主副本和从副本；所述目标存储节点包括：所述主副本所在的存储节点，以及，所述从副本所在的存储节点；所述多个存储节点包括：存储所述目标数据的存储节点；所述存储所述目标数据的存储节点用于：若所述存储所述目标数据的存储节点是所述主副本所在的存储节点，将所述目标数据对应的所有副本所在的存储节点的标识信息携带在所述第一查询响应消息中；所述客户端601进一步用于：从所述第一查询响应消息中，获取所述所有副本所在的存储节点的标识信息，作为所述目标存储节点的标识信息。In some embodiments, there are multiple copies of the target data, and the multiple copies include: a master copy and a slave copy; the target storage node includes: a storage node where the master copy is located, and a storage node where the slave copy is located. The multiple storage nodes include: a storage node storing the target data; the storage node storing the target data is used for: if the storage node storing the target data is the primary copy The storage node where the target data is located, carry the identification information of the storage node where all the copies corresponding to the target data are located in the first query response message; the
本实施例中，若查询到主副本所在的存储节点，主副本所在的存储节点可以将所有副本所在的存储节点的标识信息反馈给客户端，保证标识信息的准确度和获取效率，进而提高读写操作的准确性和效率。In this embodiment, if the storage node where the primary copy is located is queried, the storage node where the primary copy is located can feed back the identification information of the storage nodes where all the copies are located to the client, so as to ensure the accuracy and acquisition efficiency of the identification information, thereby improving the readability Accuracy and efficiency of write operations.
一些实施例中，所述目标数据存在多个副本，所述多个副本包括：主副本和从副本；所述目标存储节点包括：所述主副本所在的存储节点，以及，所述从副本所在的存储节点；所述多个存储节点包括：存储所述目标数据的存储节点；所述存储所述目标数据的存储节点用于：若所述存储所述目标数据的存储节点是所述从副本所在的存储节点，将所述目标数据对应的主副本所在的存储节点的标识信息携带在所述第一查询响应消息中；所述客户端601进一步用于：从所述第一查询响应消息中，获取所述主副本所在的存储节点的标识信息，基于所述主副本所在的存储节点的标识信息，向所述主副本所在的存储节点发送第二查询请求消息，所述第二查询请求消息用于触发所述主副本所在的存储节点产生第二查询响应消息，所述第二查询响应消息中包含所述目标数据对应的所有副本所在的存储节点的标识信息；从所述主副本所在的存储节点发送的所述第二查询响应消息中，获取所述所有副本所在的存储节点的标识信息，作为所述目标存储节点的标识信息。In some embodiments, there are multiple copies of the target data, and the multiple copies include: a master copy and a slave copy; the target storage node includes: a storage node where the master copy is located, and a storage node where the slave copy is located. The plurality of storage nodes include: a storage node storing the target data; the storage node storing the target data is used for: if the storage node storing the target data is the slave replica the storage node where the target data is located, carry the identification information of the storage node where the primary copy corresponding to the target data is located in the first query response message; the
本实施例中，若查询到从副本所在的存储节点，客户端还向主副本所在的存储节点查询目标存储节点的标识信息，可以保证获取到最新的标识信息，保证标识信息的准确度，进而提高读写操作的准确性。In this embodiment, if the storage node where the secondary copy is located is queried, the client also queries the storage node where the primary copy is located for the identification information of the target storage node, which can ensure the acquisition of the latest identification information, ensure the accuracy of the identification information, and further Improve the accuracy of read and write operations.
图7是根据本公开第四实施例的示意图，本实施例提供一种数据读写装置，该装置700包括：确定模块701、查询模块702和读写模块703。FIG. 7 is a schematic diagram according to a fourth embodiment of the present disclosure. This embodiment provides a data reading and writing device. The
确定模块701用于响应于针对目标数据的读写请求消息，确定所述目标数据所属的存储池内的多个存储节点，所述多个存储节点包括所述目标数据所在的目标存储节点，且，所述多个存储节点中的至少一个存储节点内存储所述目标存储节点的标识信息；查询模块702用于查询所述多个存储节点，以获取所述目标存储节点的标识信息；读写模块703用于基于所述目标存储节点的标识信息，将所述读写请求消息发送至所述目标存储节点，以对所述目标存储节点内的所述目标数据进行读写操作。The determining
本实施例中，通过查询存储节点，以获取目标存储节点的标识信息，可以在元服务器不可用时依然可以获取到目标存储节点的标识信息，进而可以基于目标存储节点的标识信息将读写请求消息发送至目标存储节点，以对目标存储节点内的目标数据进行读写操作，提高读写操作的可用性。In this embodiment, by querying the storage node to obtain the identification information of the target storage node, the identification information of the target storage node can still be obtained when the meta server is unavailable, and then the read and write request messages can be read and written based on the identification information of the target storage node. It is sent to the target storage node to perform read and write operations on the target data in the target storage node to improve the availability of read and write operations.
一些实施例中，所述方法应用于分布式存储系统的客户端；所述分布式存储系统还包括：元数据服务器，所述元数据服务器内记录存储池与存储池内的多个存储节点之间的映射关系；所述装置还包括：获取模块，用于基于预设周期，从所述元数据服务器获取所述映射关系；所述确定模块701进一步用于：若所述客户端本地未缓存所述目标存储节点的标识信息，且，从所述元数据服务器未获取所述目标存储节点的标识信息，基于所述映射关系，确定所述目标数据所属的存储池内的多个存储节点。In some embodiments, the method is applied to a client of a distributed storage system; the distributed storage system further includes: a metadata server, in which the metadata server records between a storage pool and multiple storage nodes in the storage pool. The device further includes: an obtaining module, configured to obtain the mapping relationship from the metadata server based on a preset period; the determining
本实施例中，通过基于预设周期从元数据服务器获取上述的映射关系，可以保证映射关系的准确度，提高读写操作的可靠性。In this embodiment, by acquiring the above-mentioned mapping relationship from the metadata server based on a preset period, the accuracy of the mapping relationship can be ensured, and the reliability of read and write operations can be improved.
一些实施例中，所述查询模块702进一步用于：依次向所述多个存储节点中的单个存储节点发送第一查询请求消息，所述第一查询请求消息用于查询所述单个存储节点内是否存储所述目标数据，直至接收到第一查询响应消息，所述第一查询响应消息用于表明存储所述目标数据；基于所述第一查询响应消息，获取所述目标存储节点的标识信息。In some embodiments, the
本实施例中，通过依次向单个存储节点发送第一查询请求消息，直至接收到用于表明存储所述目标数据的第一查询响应消息，可以基于所述用于表明存储所述目标数据的第一查询响应消息，获取目标存储节点的标识信息，进而可以基于目标存储节点的标识信息对目标数据进行读写操作，保证读写操作的正常执行。In this embodiment, by sequentially sending a first query request message to a single storage node until a first query response message indicating storage of the target data is received, the first query response message indicating storage of the target data can be A query response message is sent to obtain the identification information of the target storage node, and then read and write operations can be performed on the target data based on the identification information of the target storage node to ensure normal execution of the read and write operations.
一些实施例中，所述目标数据存在多个副本，所述多个副本包括：主副本和从副本；所述目标存储节点包括：所述主副本所在的存储节点，以及，所述从副本所在的存储节点；所述查询模块702进一步用于：针对产生所述第一查询响应消息的存储节点是所述主副本所在的存储节点，所述第一查询响应消息中包含所述目标存储节点的标识信息，执行：从所述第一查询响应消息中获取所述目标存储节点的标识信息。In some embodiments, there are multiple copies of the target data, and the multiple copies include: a master copy and a slave copy; the target storage node includes: a storage node where the master copy is located, and a storage node where the slave copy is located. The
本实施例中，若查询到主副本所在的存储节点，主副本所在的存储节点可以将所有副本所在的存储节点的标识信息反馈给客户端，保证标识信息的准确度和获取效率，进而提高读写操作的准确性和效率。In this embodiment, if the storage node where the primary copy is located is queried, the storage node where the primary copy is located can feed back the identification information of the storage nodes where all the copies are located to the client, so as to ensure the accuracy and acquisition efficiency of the identification information, thereby improving the readability Accuracy and efficiency of write operations.
一些实施例中，所述目标数据存在多个副本，所述多个副本包括：主副本和从副本；所述目标存储节点包括：所述主副本所在的存储节点，以及，所述从副本所在的存储节点；所述查询模块702进一步用于：针对产生所述第一查询响应消息的存储节点是所述从副本所在的存储节点，所述第一查询响应消息中获取所述主副本所在的存储节点的标识信息，执行：基于所述主副本所在的存储节点的标识信息，向所述主副本所在的存储节点发送第二查询请求消息，所述第二查询请求消息用于查询所述主副本所在的存储节点内存储的目标存储节点的标识信息；接收所述主副本所在的存储节点发送的第二查询响应消息，所述第二查询响应消息中包含所述目标存储节点的标识信息；从所述第二查询响应消息中，获取所述目标存储节点的标识信息。In some embodiments, there are multiple copies of the target data, and the multiple copies include: a master copy and a slave copy; the target storage node includes: a storage node where the master copy is located, and a storage node where the slave copy is located. The
本实施例中，若查询到从副本所在的存储节点，客户端还向主副本所在的存储节点查询目标存储节点的标识信息，可以保证获取到最新的标识信息，保证标识信息的准确度，进而提高读写操作的准确性。In this embodiment, if the storage node where the secondary copy is located is queried, the client also queries the storage node where the primary copy is located for the identification information of the target storage node, which can ensure the acquisition of the latest identification information, ensure the accuracy of the identification information, and further Improve the accuracy of read and write operations.
可以理解的是，本公开实施例中，不同实施例中的相同或相似内容可以相互参考。It can be understood that, in the embodiments of the present disclosure, the same or similar contents in different embodiments may refer to each other.
可以理解的是，本公开实施例中的“第一”、“第二”等只是用于区分，不表示重要程度高低、时序先后等。It can be understood that, in the embodiments of the present disclosure, "first", "second", etc. are only used for distinction, and do not indicate the level of importance, the order of time sequence, and the like.
本公开的技术方案中，所涉及的用户个人信息的收集、存储、使用、加工、传输、提供和公开等处理，均符合相关法律法规的规定，且不违背公序良俗。In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, and disclosure of the user's personal information involved are all in compliance with relevant laws and regulations, and do not violate public order and good customs.
根据本公开的实施例，本公开还提供了一种电子设备、一种可读存储介质和一种计算机程序产品。According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.
图8示出了可以用来实施本公开的实施例的示例电子设备800的示意性框图。电子设备旨在表示各种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备800还可以表示各种形式的移动装置，诸如，个人数字助理、蜂窝电话、智能电话、可穿戴设备和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和/或者要求的本公开的实现。FIG. 8 shows a schematic block diagram of an example
如图8所示，电子设备800包括计算单元801，其可以根据存储在只读存储器(ROM)802中的计算机程序或者从存储单元808加载到随机访问存储器(RAM)803中的计算机程序，来执行各种适当的动作和处理。在RAM 803中，还可存储电子设备800操作所需的各种程序和数据。计算单元801、ROM 802以及RAM 803通过总线804彼此相连。输入/输出(I/O)接口805也连接至总线804。As shown in FIG. 8 , the
电子设备800中的多个部件连接至I/O接口805，包括：输入单元806，例如键盘、鼠标等；输出单元807，例如各种类型的显示器、扬声器等；存储单元808，例如磁盘、光盘等；以及通信单元809，例如网卡、调制解调器、无线通信收发机等。通信单元809允许电子设备800通过诸如因特网的计算机网络和/或各种电信网络与其他设备交换信息/数据。Various components in the
计算单元801可以是各种具有处理和计算能力的通用和/或专用处理组件。计算单元801的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的计算单元、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。计算单元801执行上文所描述的各个方法和处理，例如数据读写方法。例如，在一些实施例中，数据读写方法可被实现为计算机软件程序，其被有形地包含于机器可读介质，例如存储单元808。在一些实施例中，计算机程序的部分或者全部可以经由ROM 802和/或通信单元809而被载入和/或安装到电子设备800上。当计算机程序加载到RAM 803并由计算单元801执行时，可以执行上文描述的数据读写方法的一个或多个步骤。备选地，在其他实施例中，计算单元801可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行数据读写方法。
本文中以上描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、芯片上系统(SOC)、复杂可编程逻辑设备(CPLD)、计算机硬件、固件、软件、和/或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和/或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。Various implementations of the systems and techniques described herein above can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOC), complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.
用于实施本公开的方法的程序代码可以采用一个或多个编程语言的任何组合来编写。这些程序代码可以提供给通用计算机、专用计算机或其他可编程负载均衡装置的处理器或控制器，使得程序代码当由处理器或控制器执行时使流程图和/或框图中所规定的功能/操作被实施。程序代码可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. The program code may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable load balancing device such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.
在本公开的上下文中，机器可读介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的程序。机器可读介质可以是机器可读信号介质或机器可读储存介质。机器可读介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.
为了提供与用户的交互，可以在计算机上实施此处描述的系统和技术，该计算机具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给计算机。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.
可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)和互联网。The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
计算机系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。服务器可以是云服务器，又称为云计算服务器或云主机，是云计算服务体系中的一项主机产品，以解决了传统物理主机与VPS服务("Virtual Private Server"，或简称"VPS")中，存在的管理难度大，业务扩展性弱的缺陷。服务器也可以为分布式系统的服务器，或者是结合了区块链的服务器。A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or a cloud host. It is a host product in the cloud computing service system to solve the traditional physical host and VPS service ("Virtual Private Server", or "VPS" for short). , there are the defects of difficult management and weak business expansion. The server can also be a server of a distributed system, or a server combined with a blockchain.
应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本发公开中记载的各步骤可以并行地执行也可以顺序地执行也可以不同的次序执行，只要能够实现本公开公开的技术方案所期望的结果，本文在此不进行限制。It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be executed in parallel, sequentially, or in different orders. As long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, there is no limitation herein.
上述具体实施方式，并不构成对本公开保护范围的限制。本领域技术人员应该明白的是，根据设计要求和其他因素，可以进行各种修改、组合、子组合和替代。任何在本公开的精神和原则之内所作的修改、等同替换和改进等，均应包含在本公开保护范围之内。The above-mentioned specific embodiments do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present disclosure should be included within the protection scope of the present disclosure.
Claims (18)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210649472.4ACN115168440A (en) | 2022-06-09 | 2022-06-09 | Data reading and writing method, distributed storage system, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210649472.4ACN115168440A (en) | 2022-06-09 | 2022-06-09 | Data reading and writing method, distributed storage system, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115168440Atrue CN115168440A (en) | 2022-10-11 |
Family
ID=83485056
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210649472.4APendingCN115168440A (en) | 2022-06-09 | 2022-06-09 | Data reading and writing method, distributed storage system, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115168440A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115686361A (en)* | 2022-10-12 | 2023-02-03 | 超聚变数字技术有限公司 | Method, system and device for reading and writing data |
| CN118567578A (en)* | 2024-07-31 | 2024-08-30 | 浙江省邮电工程建设有限公司 | A distributed metaverse storage construction method and system |
| CN119377017A (en)* | 2024-12-24 | 2025-01-28 | 苏州元脑智能科技有限公司 | Data storage method, computer equipment, storage medium and program product |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103258036A (en)* | 2013-05-15 | 2013-08-21 | 广州一呼百应网络技术有限公司 | Distributed real-time search engine based on p2p |
| CN112148206A (en)* | 2019-06-28 | 2020-12-29 | 北京金山云网络技术有限公司 | A data reading and writing method, device, electronic device and medium |
| CN113986135A (en)* | 2021-10-27 | 2022-01-28 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for processing requests |
- 2022
- 2022-06-09CNCN202210649472.4Apatent/CN115168440A/enactivePending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103258036A (en)* | 2013-05-15 | 2013-08-21 | 广州一呼百应网络技术有限公司 | Distributed real-time search engine based on p2p |
| CN112148206A (en)* | 2019-06-28 | 2020-12-29 | 北京金山云网络技术有限公司 | A data reading and writing method, device, electronic device and medium |
| CN113986135A (en)* | 2021-10-27 | 2022-01-28 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for processing requests |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115686361A (en)* | 2022-10-12 | 2023-02-03 | 超聚变数字技术有限公司 | Method, system and device for reading and writing data |
| CN118567578A (en)* | 2024-07-31 | 2024-08-30 | 浙江省邮电工程建设有限公司 | A distributed metaverse storage construction method and system |
| CN119377017A (en)* | 2024-12-24 | 2025-01-28 | 苏州元脑智能科技有限公司 | Data storage method, computer equipment, storage medium and program product |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN115168440A (en) | Data reading and writing method, distributed storage system, device, equipment and storage medium | |
| CN110134648A (en) | Log processing method, device, equipment, system, and computer-readable storage medium | |
| WO2020019724A1 (en) | Method for acquiring sensor data in server, acquisition system, and related apparatus | |
| CN113364877B (en) | Data processing method, device, electronic equipment and medium | |
| CN113656501B (en) | A data reading method, device, equipment and storage medium | |
| CN113193947B (en) | Method, apparatus, medium, and program product for implementing distributed global ordering | |
| CN115510036A (en) | Data migration method, device, equipment and storage medium | |
| CN116540938A (en) | Data reading method, device, distributed storage system, equipment and storage medium | |
| CN113722389B (en) | Data management method, device, electronic equipment and computer readable storage medium | |
| CN113986135B (en) | Method, device, device and storage medium for processing requests | |
| CN112860796B (en) | Method, device, device and storage medium for synchronizing data | |
| CN114490128A (en) | Data storage method and device under remote multi-active architecture | |
| CN117422556B (en) | Derivative transaction system, device and computer medium based on replication state machine | |
| CN115883653B (en) | Request processing method, request processing device, electronic equipment and storage medium | |
| CN117093139A (en) | Data processing method, device and system for data storage | |
| CN114461502B (en) | Model monitoring method and device | |
| CN114449031B (en) | Information acquisition method, device, equipment and storage medium | |
| CN113641688B (en) | Node updating method, related device and computer program product | |
| CN112948461B (en) | Method, apparatus, storage medium and program product for calendar data processing | |
| CN115617800A (en) | Data reading method, device, electronic device and storage medium | |
| CN115640280A (en) | Data migration method and device | |
| CN114647701A (en) | Load balancing method and device for distributed database, electronic equipment and medium | |
| CN114328731A (en) | Information processing method, device, electronic equipment and storage medium | |
| CN114442962B (en) | Data reading method, device, equipment and storage medium | |
| CN119336841B (en) | Data synchronization method, device, electronic device and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |