Disclosure of Invention
The embodiment of the invention aims to provide a data reading and writing method, a device, electronic equipment and a medium, so as to realize shortening of a reading and writing operation path and reduction of time consumed by the reading and writing operation, thereby reducing occurrence of failure of the reading and writing operation caused by overlong time consumed by the reading and writing operation. The specific technical scheme is as follows:
in a first aspect, a data read-write method is provided and applied to a terminal, and the method includes:
Determining fragments for storing the data to be read and written in a virtual disk according to a storage position for storing the data to be read and written in the virtual disk, wherein the virtual disk comprises a plurality of fragments, and each fragment is used for storing data with a specified size;
Determining virtual nodes for storing the fragmentation mapping of the data to be read and written, wherein the virtual nodes correspond to a preset number of servers in a storage cluster;
and sending an input/output (I/O) request to a server corresponding to the virtual node, wherein the I/O request is used for requesting to read and write the data to be read and written from the storage position.
Optionally, before determining the virtual disk according to the storage location of the virtual disk for storing the data to be read and written, determining the partition of the virtual disk for storing the data to be read and written, the method further includes:
And obtaining a virtual node list, wherein the virtual node list comprises hard disk identifiers corresponding to the virtual nodes and server Internet Protocol (IP) addresses to which the hard disks belong.
Optionally, the obtaining the virtual node list includes:
sending an acquisition request to a designated server, wherein the acquisition request is used for requesting to acquire the virtual node list;
And receiving the virtual node list, and establishing long connection with a server sending the virtual node list.
Optionally, the determining, according to a storage location in a virtual disk for storing data to be read and written, a slice in the virtual disk for storing the data to be read and written includes:
Determining a starting fragment corresponding to the data to be read and written according to a starting storage position for storing the data to be read and written in the virtual disk and the size of each fragment in the virtual disk;
Determining an end slice corresponding to the data to be read and written according to the starting slice and the size of the data to be read and written, and determining the starting slice, the end slice and the slices between the starting slice and the end slice as slices for storing the data to be read and written.
Optionally, after the determining the virtual node for storing the slice mapping of the data to be read and written, the method further includes:
and determining a hard disk set for storing the data to be read and written according to the virtual node list, wherein the hard disk set comprises hard disks corresponding to virtual nodes for storing the fragmentation mapping of the data to be read and written.
Optionally, the I/O request is a write request, and the sending an input/output I/O request to a server corresponding to the virtual node includes:
Determining the last hard disk of the hard disk set, and sending the write request to a server to which the last hard disk belongs, so that after the last hard disk writes the data to be read and written, synchronizing the data to be written by the last hard disk with other hard disks in the hard disk set according to each hard disk identifier in the hard disk set, wherein the write request comprises the data to be written and the identifier of the virtual node, or
And respectively sending the writing requests to the servers to which the hard disks in the hard disk set belong.
Optionally, the I/O request is a read request, and the sending the I/O request to the server corresponding to the virtual node includes:
Determining the last hard disk of the hard disk set, and sending the read request to a server to which the last hard disk belongs, wherein the read request comprises an identifier of data to be read and an identifier of the virtual node.
In a second aspect, a data read-write device is provided, applied to a terminal, and the device includes:
The determining module is used for determining fragments in the virtual disk for storing the data to be read and written according to the storage position in the virtual disk for storing the data to be read and written, wherein the virtual disk comprises a plurality of fragments, and each fragment is used for storing the data with a specified size;
The determining module is further configured to determine a virtual node for storing the fragmentation mapping of the data to be read and written, where the virtual node corresponds to a preset number of servers in a storage cluster;
And the sending module is used for sending an input/output (I/O) request to the server corresponding to the virtual node determined by the determining module, wherein the I/O request is used for requesting to read and write the data to be read and written from the storage position.
Optionally, the device further comprises an acquisition module;
The obtaining module is configured to obtain a virtual node list before determining, in the virtual disk, a storage location for storing data to be read and written according to the virtual disk, and before storing the fragments of the data to be read and written, where the virtual node list includes a hard disk identifier corresponding to each virtual node and a server internet protocol IP address to which the hard disk belongs.
Optionally, the acquiring module is specifically configured to:
sending an acquisition request to a designated server, wherein the acquisition request is used for requesting to acquire the virtual node list;
And receiving the virtual node list, and establishing long connection with a server sending the virtual node list.
Optionally, the determining module is specifically configured to:
Determining a starting fragment corresponding to the data to be read and written according to a starting storage position for storing the data to be read and written in the virtual disk and the size of each fragment in the virtual disk;
Determining an end slice corresponding to the data to be read and written according to the starting slice and the size of the data to be read and written, and determining the starting slice, the end slice and the slices between the starting slice and the end slice as slices for storing the data to be read and written.
Optionally, the determining module is further configured to determine, according to the virtual node list, a hard disk set for storing the data to be read and written after the determining the virtual node for storing the slice map of the data to be read and written, where the hard disk set includes a hard disk corresponding to the virtual node for storing the slice map of the data to be read and written.
Optionally, the I/O request is a write request, and the sending module is specifically configured to:
Determining the last hard disk of the hard disk set, and sending the write request to a server to which the last hard disk belongs, so that after the last hard disk writes the data to be read and written, synchronizing the data to be written by the last hard disk with other hard disks in the hard disk set according to each hard disk identifier in the hard disk set, wherein the write request comprises the data to be written and the identifier of the virtual node, or
And respectively sending the writing requests to the servers to which the hard disks in the hard disk set belong.
Optionally, the I/O request is a read request, and the sending module is specifically configured to:
Determining the last hard disk of the hard disk set, and sending the read request to a server to which the last hard disk belongs, wherein the read request comprises an identifier of data to be read and an identifier of the virtual node.
In a third aspect, an electronic device is provided, the electronic device including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory perform communication with each other through the communication bus;
A memory for storing a computer program;
and the processor is used for realizing any one of the data reading and writing method steps when executing the program stored in the memory.
In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where a computer program is stored, where the computer program is executed by a processor to implement any one of the above-mentioned data reading and writing method steps.
In a fifth aspect, embodiments of the present invention also provide a computer program product comprising instructions which, when run on a computer, cause the computer to perform any of the above-described data read-write method steps.
The data read-write method, the device, the electronic equipment and the medium provided by the embodiment of the invention can determine the fragments in the virtual disk for storing the data to be read-written according to the storage position in the virtual disk for storing the data to be read-written. And then determining virtual nodes for storing the fragmentation mapping of the data to be read and written, wherein the virtual nodes correspond to a preset number of servers in the storage cluster. And then sending the I/O request to the server corresponding to the virtual node. The terminal can directly interact data with the servers in the storage cluster without passing through metadata nodes, so that the embodiment of the invention can shorten the path of read-write operation and reduce the time consumed by the read-write operation, thereby reducing the occurrence of failure of the read-write operation caused by overlong time consumed by the read-write operation.
Of course, it is not necessary for any one product or method of practicing the invention to achieve all of the advantages set forth above at the same time.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a data read-write system shown in fig. 1 includes a terminal and a storage cluster.
The terminal can be an electronic device such as a mobile phone, a tablet personal computer or a computer, etc. capable of installing client software. The terminal may read data from or write data to the servers of the storage cluster.
The storage cluster comprises a plurality of servers, wherein the servers comprise a plurality of hard disks, and the hard disks are used for storing data written by the terminal.
One virtual node corresponds to one or more hard disks in the storage cluster, the hard disks corresponding to each virtual node can be different, and the number of the hard disks corresponding to each virtual node can also be different.
It should be noted that fig. 1 exemplarily shows one terminal, and two servers capable of communicating with the terminal, where the two servers are located in a storage cluster, and the number of servers included in the storage cluster in the embodiment of the present application is not limited thereto.
Fig. 2 is a flowchart of a data read-write method according to an embodiment of the present invention, where the data read-write method is applied to a terminal of the data read-write system shown in fig. 1, and referring to fig. 2, the method includes the following steps:
Step 201, determining a partition for storing the data to be read and written in the virtual disk according to the storage position for storing the data to be read and written in the virtual disk.
The virtual disk comprises a plurality of fragments, and each fragment is used for storing data with a specified size. For example, a size of 4 megabytes (Megabytes, MB) is specified.
In one embodiment, a method of determining a slice for storing data to be read and written includes the steps of:
step one, determining the initial fragment corresponding to the data to be read and written according to the initial storage position for storing the data to be read and written in the virtual disk and the size of each fragment in the virtual disk.
Optionally, before acquiring a starting storage location in the virtual disk for storing data to be read and written, the terminal may request to acquire metadata information of the virtual disk from the storage cluster. The metadata information is information describing data attributes, and the terminal may determine a starting storage location in the virtual disk for storing data to be read and written according to the metadata information. If the created virtual disk does not exist in the storage cluster, the storage cluster creates the virtual disk, and then metadata information of the virtual disk is sent to the terminal. And if the created virtual disk exists in the storage cluster, sending metadata information of the virtual disk to the terminal.
Specifically, the storage cluster of the data read-write system shown in fig. 1 judges whether a header file (for example, an AAA-header file) is stored in the storage cluster, and if the header file is stored in the storage cluster, it indicates that a created virtual disk exists in the storage cluster.
As an example, if the initial storage location of the virtual disk for storing the data to be read and written is the location of the 300 th MB of data stored in the virtual disk and the size of each slice in the virtual disk is 4MB, the terminal may determine that the initial slice corresponding to the data to be read and written is 300MB/4 mb=75, which indicates that the 300 th MB of data of the virtual disk is stored in the 75 th slice, so the initial slice is the 75 th slice of the virtual disk.
It will be appreciated that the terminal may determine, according to the size of the data stored in the virtual disk, a starting storage location in the virtual disk for storing the data to be written. For example, if 200MB of data is already stored in the virtual disk, the initial storage position of the data to be written in the virtual disk is 201MB.
The terminal can determine an initial storage position for storing the data to be read in the virtual disk according to the identification of the data to be read.
And step two, determining an ending slice corresponding to the data to be read and written according to the sizes of the starting slice and the data to be read and written, and determining the starting slice, the ending slice and the slices between the starting slice and the ending slice as slices for storing the data to be read and written.
For example, the size of the data to be read and written is 12MB, the initial slice is the 2 nd slice of the virtual disk, and the end slice is the 4 th slice of the virtual disk. The 2 nd slice of the virtual disk stores 0MB-4MB of data to be read and written, the 3 rd slice stores 5MB-8MB of data to be read and written, and the 4 th slice stores 9MB-12MB of data to be read and written. And determining the 2 nd slice, the 3 rd slice and the 4 th slice of the virtual disk as slices for storing data to be read and written.
Step 202, determining a virtual node for storing a sharded map of data to be read and written.
The virtual nodes correspond to a preset number of servers in the storage cluster. It can be understood that the server corresponding to the virtual node is a server to which the hard disk corresponding to the virtual node belongs.
Alternatively, the virtual node for storing the fragment map of the data to be read and written may be determined according to the identification of the fragment for storing the data to be read and written and the total number of virtual nodes.
For example, the identification of the shard for storing the data to be read and written is 3, the total number of virtual nodes is 10, and the identification of the virtual node for storing the shard map of the data to be read and written is 3%10=3. Wherein,% represents the remainder operation.
After determining the virtual node for storing the fragment map of the data to be read and written, a hard disk set for storing the data to be read and written may be determined according to the virtual node list. The hard disk set comprises hard disks corresponding to virtual nodes for storing the fragmentation mapping of the data to be read and written.
It can be understood that one virtual node corresponds to a plurality of hard disks in the storage cluster, in order to achieve data redundancy, each data to be written is stored as a plurality of copies when being written into the storage cluster, and each hard disk corresponding to the virtual node to which the data to be written is mapped stores one copy.
If the number of servers in the storage cluster of the data read-write system shown in fig. 1 is greater than or equal to the number of hard disks corresponding to the virtual node, the hard disks corresponding to the virtual node are respectively located in different servers, and if the number of servers in the storage cluster is smaller than the number of hard disks corresponding to the virtual node, the hard disks corresponding to the virtual node comprise the hard disks in different servers and the different hard disks of the same server.
For example, if the number of servers in the storage cluster is 2, which is server 1 and server 2, respectively, and the number of hard disks corresponding to the virtual node is 2, the hard disks corresponding to the virtual node include hard disk a in server 1 and hard disk B in server 2.
For another example, the number of servers in the storage cluster is 2, server 1 and server 2, respectively. The number of hard disks corresponding to the virtual node is 3, and the hard disks corresponding to the virtual node include one hard disk in the server 1 and two hard disks in the server 2, or two hard disks in the server 1 and one hard disk in the server 2.
Because the hard disk corresponding to the virtual node is in different servers, if one of the servers corresponding to the virtual node is down, other servers corresponding to the virtual node can respond to the I/O request sent by the terminal, so that the success rate of reading or writing data by the terminal is improved.
Step 203, an Input/Output (I/O) request is sent to a server corresponding to the virtual node.
The I/O request is used for requesting to read and write the data to be read and written from a storage position in the virtual disk, wherein the storage position is used for storing the data to be read and written.
It can be understood that the terminal reads and writes the data to be read and written from the storage position in the virtual disk, and the terminal may read and write the data to be read and written from the hard disk of the server corresponding to the virtual node mapped by the partition of the virtual disk.
In one embodiment, the terminal sends an I/O request to a server to which a hard disk corresponding to the virtual node belongs, and after the server receives the I/O request, the server writes data to be written into the hard disk corresponding to the virtual node, or reads the data to be read from the hard disk corresponding to the virtual node.
The data read-write method provided by the embodiment of the invention can determine the fragments used for storing the data to be read-written in the virtual disk according to the storage position used for storing the data to be read-written in the virtual disk. And then determining virtual nodes for storing the fragmentation mapping of the data to be read and written, wherein the virtual nodes correspond to a preset number of servers in the storage cluster. And then sending the I/O request to the server corresponding to the virtual node. The terminal can directly interact data with the servers in the storage cluster without passing through metadata nodes, so that the embodiment of the invention can shorten the path of read-write operation and reduce the time consumed by the read-write operation, thereby reducing the occurrence of failure of the read-write operation caused by overlong time consumed by the read-write operation.
In the embodiment of the present invention, before determining the slice for storing the data to be read and written in step 202, the terminal needs to acquire the virtual node list. The virtual node list includes hard disk identifiers corresponding to the virtual nodes and server internet protocol (Internet Protocol, IP) addresses to which the hard disks belong. The method for acquiring the virtual node list comprises the following steps:
Step one, an acquisition request is sent to a specified server.
The acquisition request is used for requesting to acquire a virtual node list, and the designated server is a server corresponding to one IP address in the pre-acquired configuration file. The IP addresses in the configuration file are server IP addresses of a preset number in the storage cluster. In order to avoid that a connection cannot be established with a server corresponding to an IP address in the configuration file, or that the server corresponding to the IP address in the configuration file is unstable, the configuration file may include a plurality of IP addresses. If the terminal cannot be successfully connected with the server corresponding to one IP address in the configuration file, the terminal is connected with the server corresponding to the other IP address in the configuration file. Therefore, the configuration file comprises a plurality of IP addresses, so that the situation that the terminal cannot acquire the virtual node list is reduced.
Alternatively, the terminal may invoke a software development kit (Software Development Kit, SDK) to connect to a storage cluster in the data read-write system shown in fig. 1. And then establishing long connection with the appointed server, and sending an acquisition request to the appointed server after the connection is successful.
And step two, receiving the virtual node list, and establishing long connection with a server for sending the virtual node list.
If the virtual node list is stored in the designated server, the terminal may receive the virtual node list sent by the designated server.
If the virtual node list is not stored in the designated server, the designated server queries whether the virtual node list is stored in other servers in the storage cluster of the data read-write system shown in fig. 1. If one server in the storage cluster stores the virtual node list, the server establishes long connection with the terminal and sends the virtual node list to the terminal.
Alternatively, the designated server may sequentially query other servers in the storage cluster of the data read-write system shown in fig. 1 in order of from small to large IP addresses until the server storing the virtual node list is queried.
Optionally, the manner in which the terminal sends the I/O request to the server corresponding to the virtual node in step 203 includes two cases:
in the first case, when the I/O request is a write request, the manner in which the terminal sends the I/O request to the server corresponding to the virtual node includes the following two ways:
Optionally, the manner in which the terminal sends the write request may be suitable for a scenario in which the number of I/O requests sent by the terminal is large, which may cause a network card usage bottleneck of the terminal, but is not limited thereto.
The first mode is to determine the last hard disk of the hard disk set and send a write-in request to a server to which the last hard disk belongs.
In the embodiment of the present invention, in combination with fig. 3, the terminal sends a write request to the server to which the last hard disk (e.g., hard disk 3 in fig. 3) in the hard disk set belongs, and after the last hard disk writes the data to be read and written, the server to which the last hard disk belongs may synchronize the data to be written in the last hard disk to other hard disks (e.g., hard disk 2 and hard disk 1 in fig. 3) in the hard disk set according to each hard disk identifier in the hard disk set.
The writing request comprises data to be written, identifications of virtual nodes and identifications of all hard disks in the hard disk set.
After writing the data to be written, the server can also send a confirmation message to the terminal, wherein the confirmation message is used for indicating that the hard disk successfully writes the data to be written.
Alternatively, the last hard disk of the hard disk set may be determined in order of the hard disk identifiers from large to small.
It can be understood that each server of the storage cluster includes a plurality of hard disks, and the server needs to determine the hard disk to which the data to be written is written according to the identifier of the virtual node. And each hard disk to be written with data is different, and after the data to be written is written with the last hard disk, the server to which the last hard disk belongs needs to synchronize the data to be written with other hard disks in the hard disk set according to the hard disk identifiers in the hard disk set.
It can be understood that, when the data to be written is not written in some hard disks in the hard disk set, the server to which the hard disk to which the data to be written is not written belongs may receive a read request for the data to be written, and at this time, the server cannot read the data to be written, and the terminal fails to read the data.
In order to reduce the occurrence of the above situation, the terminal may send a write request to the server to which the last hard disk belongs, so that the server to which the last hard disk belongs writes the data to be written first, and then synchronizes the data to be written to other hard disks in the hard disk set. And when the terminal reads the data, the terminal reads the data from the last hard disk. Since the last hard disk is written with the data to be written first, the terminal has the highest possibility of reading the data to be written from the last hard disk when reading the data to be written.
Optionally, to distinguish the data written by different users, the writing request may further include a user identifier. The user identifier may be an account identifier of the terminal login. It can be understood that virtual disks corresponding to different accounts can be different, and different users can be distinguished according to different accounts logged in by the terminal.
The write request may also include a virtual disk identification. It can be understood that the same account number can also correspond to a plurality of virtual disks, and different virtual disks of the same account number can be distinguished through virtual disk identifiers. For example, virtual disk identifier 11A represents virtual disk A of user 11.
Therefore, after the terminal sends the writing request to the server to which the last hard disk of the hard disk set belongs, the data to be written, which are synchronously written by all the hard disks in the hard disk set, can be realized. Compared with a mode of respectively sending the writing requests to the servers of the hard disks, the method can reduce the flow of the writing requests. The problem that the writing request cannot be sent or the writing request sending speed is low because the writing request sent by the terminal is overlarge in flow and the network card processing capacity of the terminal is limited is avoided.
Alternatively, the second mode of sending the write request by the terminal may be applicable to a scenario where the number of I/O requests sent by the terminal is small and a short I/O operation is required, but is not limited thereto.
In a second mode, referring to fig. 4, the terminal sends a write request to a server to which each hard disk (e.g., hard disk 1, hard disk 2, and hard disk 3 in fig. 4) in the hard disk set belongs. After writing the data to be written, the server can also send a confirmation message to the terminal, wherein the confirmation message is used for indicating that the hard disk successfully writes the data to be written.
The write request comprises data to be written and an identification of the virtual node. Optionally, to distinguish the data written by different users, the writing request may further include a user identifier.
Therefore, compared with the first mode, the method for respectively sending the writing requests to the servers of the hard disks in the hard disk set by the terminal reduces the time for writing the data to be written because the data to be written can be written in the hard disks in the hard disk set at the same time.
And in the second case, when the I/O request is a read request, the method for sending the read request to the server corresponding to the virtual node by the terminal comprises the steps of determining the last hard disk of the hard disk set and sending the read request to the server to which the last hard disk belongs.
The reading request comprises an identification of data to be read and an identification of the virtual node. Optionally, in order to distinguish the data read by different users, the read request may further include a user identifier.
It can be understood that after receiving the read request, the server to which the last hard disk belongs sends the data to be read to the terminal if the data to be read is stored in the server. If the data to be read is not stored in the server, the data to be read is read from other hard disks corresponding to the virtual node according to the sequence of the hard disk identifiers from large to small, and if the data to be read is read, the data to be read is sent to the terminal.
In order to more clearly explain the data reading and writing method provided by the embodiment of the present invention, another exemplary schematic diagram of a data reading and writing system is provided in the embodiment of the present invention, and the system is shown in fig. 5.
With reference to the system shown in fig. 5, the data read-write method provided by the embodiment of the invention includes the following steps:
Step one, a terminal sends an acquisition request to a designated server corresponding to an IP address in a configuration file and receives a virtual node list sent by the designated server.
For example, in connection with FIG. 5, the list of virtual nodes sent by the designated server includes virtual node 1, virtual node 2, and virtual node 3. The virtual node 1 corresponds to the hard disk X of the server 1 and the hard disk Y of the server 2, respectively.
And step two, the terminal determines the fragments for storing the data to be read and written according to the position of the data to be read and written in the virtual disk and the size of the data to be read and written, and determines the virtual nodes corresponding to the fragments for storing the data to be read and written.
For example, referring to fig. 5, the terminal determines that a virtual node corresponding to a fragment for storing data to be read and written is virtual node 1.
And thirdly, the terminal determines the hard disk corresponding to the virtual node determined in the second step according to the virtual node list.
For example, referring to fig. 5, the terminal determines that the hard disk corresponding to the virtual node 1 is the hard disk X and the hard disk Y.
And step four, the terminal writes the data to be written into each hard disk determined in the step three, or reads the data to be read from the hard disk determined in the step three.
For example, in connection with fig. 5, the terminal sends write requests to the server 1 and the server 2, respectively. After receiving the writing request, the server 1 writes the data to be written into the hard disk X, and sends a confirmation message to the terminal. After receiving the writing request, the server 2 writes the data to be written into the hard disk Y, and sends a confirmation message to the terminal.
Or the terminal sends a read request to the server 1 or the server 2. After receiving the reading, the server 1 reads the data to be read from the hard disk X and sends the data to the terminal. Or after receiving the reading request, the server 2 reads the data to be read from the hard disk Y and sends the data to the terminal.
The embodiment of the invention can also have the beneficial effects that as the storage cluster does not comprise the metadata node, compared with the storage cluster comprising the metadata node, the embodiment of the invention can reduce the storage components of the storage cluster, reduce the total number of devices of the storage cluster and save resources.
In addition, if the metadata node in the storage cluster including the metadata node fails, the terminal cannot acquire the hard disk identifier corresponding to the data to be read and written, cannot read or write the data in the hard disk corresponding to the data to be read and written, and cannot process the I/O request by the server for storing the data in the storage cluster. In the embodiment of the invention, however, the terminal does not need to acquire the hard disk corresponding to the data to be read and written from the metadata node, and the terminal can directly interact with the server for storing the data, so that the embodiment of the invention can avoid single-point faults.
In addition, in the embodiment of the invention, the terminal can directly send the writing request to the server, and the server can also send the confirmation message for indicating that the writing of the data to be written is successful to the terminal, so that the consistency of the data written by the terminal and the data stored by the hard disk is ensured, and the quality and the integrity of the stored data can be improved.
In the prior art, if one hard disk in a storage cluster fails, data stored in a large number of hard disks in the storage cluster need to be migrated.
This is because in the prior art, the identity of the virtual node is calculated to make a remainder on the total number of hard disks of the storage cluster, and the calculation result is determined as the hard disk corresponding to the virtual node. When one hard disk in the storage cluster fails, the total number of available hard disks in the storage cluster is changed, and then the hard disks corresponding to each virtual node are changed. The hard disk corresponding to each virtual node needs to be redetermined, and the data stored in the hard disk originally corresponding to each virtual node is migrated to the hard disk corresponding to the virtual node again. The data migration is huge in quantity, excessive in consumed time and unfavorable for realization.
In the embodiment of the invention, the hard disk corresponding to each virtual node is predetermined. If one hard disk of the storage cluster fails, only data stored in the failed hard disk is transferred to the other hard disk, and the hard disk corresponding to the virtual node is updated in the virtual node list. Therefore, the embodiment of the invention can also reduce the data quantity to be migrated when the storage cluster hard disk fails.
For example, the virtual node list includes a virtual node 1 and a virtual node 2, the virtual node 1 corresponds to a hard disk a and a hard disk B, and the virtual node 2 corresponds to a hard disk C. If the hard disk 1 fails, the data stored in the hard disk 1 is migrated to the hard disk C. And updating the virtual node list to be that the virtual node 1 corresponds to the hard disk C and the hard disk B, and the virtual node 2 corresponds to the hard disk C.
Corresponding to the above embodiment of the method, as shown in fig. 6, an embodiment of the present invention provides a data read-write device, which is applied to a terminal of the data read-write system shown in fig. 1, and the device includes a determining module 601 and a sending module 602.
The determining module 601 is configured to determine, according to a storage location in a virtual disk for storing data to be read and written, a slice in the virtual disk for storing the data to be read and written, where the virtual disk includes a plurality of slices, and each slice is used for storing data with a specified size;
the determining module 601 is further configured to determine a virtual node for storing a slice mapping of data to be read and written, where the virtual node corresponds to a preset number of servers in the storage cluster;
And the sending module 602 is configured to send an input/output I/O request to a server corresponding to the virtual node determined by the determining module, where the I/O request is used to request reading and writing of data to be read and written from a storage location.
Optionally, the device can further comprise an acquisition module;
The acquisition module is used for acquiring a virtual node list before determining the fragments for storing the data to be read and written in the virtual disk according to the storage position for storing the data to be read and written in the virtual disk, wherein the virtual node list comprises hard disk identifiers corresponding to the virtual nodes and the IP address of a server to which the hard disk belongs.
Optionally, the acquiring module may be specifically configured to:
sending an acquisition request to a designated server, wherein the acquisition request is used for requesting to acquire a virtual node list;
And receiving the virtual node list, and establishing long connection with a server for sending the virtual node list.
Optionally, the determining module 601 may be specifically configured to:
Determining the initial fragment corresponding to the data to be read and written according to the initial storage position for storing the data to be read and written in the virtual disk and the size of each fragment in the virtual disk;
and determining an ending slice corresponding to the data to be read and written according to the sizes of the starting slice and the data to be read and written, and determining the starting slice, the ending slice and the slices between the starting slice and the ending slice as slices for storing the data to be read and written.
Optionally, the determining module 601 may be further configured to determine, according to the virtual node list, a hard disk set for storing the data to be read and written after determining the virtual node for storing the partition map of the data to be read and written, where the hard disk set includes a hard disk corresponding to the virtual node for storing the partition map of the data to be read and written.
Optionally, the I/O request is a write request, and the sending module 602 may be specifically configured to:
Determining the last hard disk of the hard disk set, and sending a write request to a server to which the last hard disk belongs, so that after the last hard disk writes data to be read and written, synchronizing the data to be written by the last hard disk to other hard disks in the hard disk set according to the hard disk identifiers in the hard disk set, wherein the write request comprises the data to be written and the identifiers of virtual nodes, or
And sending writing requests to servers to which the hard disks in the hard disk set belong respectively.
Optionally, the I/O request is a read request, and the sending module 602 may be specifically configured to:
determining the last hard disk of the hard disk set, and sending a read request to a server to which the last hard disk belongs, wherein the read request comprises an identifier of data to be read and an identifier of a virtual node.
The embodiment of the present invention further provides an electronic device, as shown in fig. 7, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, where the processor 701, the communication interface 702, and the memory 703 perform communication with each other through the communication bus 704,
A memory 703 for storing a computer program;
The processor 701 is configured to implement the steps executed by the terminal in the above-described method embodiment when executing the program stored in the memory 703.
The communication bus mentioned above for the electronic device may be a peripheral component interconnect standard (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus or an extended industry standard architecture (Extended Industry Standard Architecture, EISA) bus, etc. The communication bus may be classified as an address bus, a data bus, a control bus, or the like. For ease of illustration, the figures are shown with only one bold line, but not with only one bus or one type of bus.
The communication interface is used for communication between the electronic device and other devices.
The Memory may include random access Memory (Random Access Memory, RAM) or may include Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the aforementioned processor.
The processor may be a general-purpose processor including a central Processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc., or may be a digital signal processor (DIGITAL SIGNAL Processing, DSP), application Specific Integrated Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components.
In yet another embodiment of the present invention, there is also provided a computer readable storage medium having stored therein a computer program which, when executed by a processor, implements the steps of any of the data read-write methods described above.
In yet another embodiment of the present invention, a computer program product containing instructions that, when run on a computer, cause the computer to perform the method of any of the above embodiments is also provided.
In the above embodiments, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present invention, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk Solid STATE DISK (SSD)), etc.
It is noted that relational terms such as first and second, and the like are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises an element.
In this specification, each embodiment is described in a related manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments in part.
The foregoing description is only of the preferred embodiments of the present invention and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.