CN109933566B - File reading and writing method, device, equipment and computer readable storage medium - Google Patents
File reading and writing method, device, equipment and computer readable storage mediumDownload PDFInfo
- Publication number
- CN109933566B CN109933566BCN201910170232.4ACN201910170232ACN109933566BCN 109933566 BCN109933566 BCN 109933566BCN 201910170232 ACN201910170232 ACN 201910170232ACN 109933566 BCN109933566 BCN 109933566B
- Authority
- CN
- China
- Prior art keywords
- file
- copy
- host
- writing
- reading
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034methodMethods0.000titleclaimsabstractdescription53
- 230000001960triggered effectEffects0.000claimsabstractdescription15
- 230000006835compressionEffects0.000claimsdescription24
- 238000007906compressionMethods0.000claimsdescription24
- 230000005540biological transmissionEffects0.000claimsdescription11
- 238000004891communicationMethods0.000description6
- 238000010586diagramMethods0.000description4
- 238000005516engineering processMethods0.000description2
- 230000003287optical effectEffects0.000description1
- 238000005192partitionMethods0.000description1
Images
Classifications
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical Field
The present invention relates to the field of distributed storage technologies, and in particular, to a method, an apparatus, a device, and a computer readable storage medium for reading and writing files.
Background
With the advent of the big data age, distributed storage applications have become more and more widespread, and in a distributed storage environment, there are multiple storage devices, and there are multiple volumes on each storage device. For data security and file access efficiency, there may be different numbers of file copies in a file, where the file copies need to be uniformly distributed on different volumes of different hosts, so that the situation that the file cannot be accessed due to a failure of a storage node can be avoided, and meanwhile, an increase in access delay of a user due to a high access amount can be avoided.
At present, when the distributed storage technology is actually applied, files need to be read and written, when the files are read, the files are generally read from the sub-volumes at random, the files are distributed on different sub-volumes of different hosts, when the files are read, the files are stored in the machine room, the files are read from other machine rooms, the bandwidth between the machine rooms is limited, when the data transmission quantity is more, the transmission delay of the data is higher, and the reading speed of the files is slower.
Therefore, how to reduce the bandwidth usage between the rooms and improve the file reading speed is a current urgent problem to be solved.
Disclosure of Invention
The invention mainly aims to provide a file reading and writing method, device and equipment and a computer readable storage medium, aiming at reducing the bandwidth utilization rate between machine rooms and improving the reading speed of files.
In order to achieve the above object, the present invention provides a method for reading and writing a file, the method for reading and writing a file comprising the steps of:
when a triggered file reading request is monitored, acquiring a copy storage route group of a corresponding file according to the file reading request, and determining a first host for triggering the file reading request;
judging whether a copy storage route belonging to the first host exists in the copy storage route group or not;
if the copy storage route belonging to the first host exists in the copy storage route group, reading a file copy from the first host;
and if the copy storage route belonging to the first host does not exist in the copy storage route group, reading the file copy across a machine room.
Preferably, before the step of reading the copy of the file across the machine room, the method further includes:
if the copy storage route group does not have the copy storage route belonging to the first host, determining a first machine room to which the first host belongs;
judging whether the copy storage route group has copy storage routes of other hosts belonging to the first machine room or not;
if the copy storage route group has the copy storage routes of the other hosts belonging to the first machine room, reading file copies from the other hosts;
if the copy storage route group does not have the copy storage routes of the rest hosts belonging to the first machine room, executing the steps: the file copy is read across the machine room.
Preferably, the file reading and writing method further includes:
when a triggered file writing request is monitored, acquiring a target file to be written, determining a second host triggering the file writing request, and determining a second machine room to which the second host belongs;
and writing the target file into the host computer of the second machine room as a file copy, and transmitting the target file to the host computers belonging to other machine rooms so as to enable the host computers of the other machine rooms to execute file writing operation.
Preferably, the step of writing the target file as a file copy into the host computer of the second machine room, and transmitting the target file to the host computers belonging to the other machine rooms, so that the host computers of the other machine rooms execute the file writing operation includes:
performing a deduplication operation on the target file, and writing the deduplicated target file into a host of the second machine room as a file copy;
transmitting the target file after the duplication removal to the hosts belonging to the other machine rooms so as to enable the hosts of the other machine rooms to execute file writing operation;
or, writing the target file into a host computer of the second machine room as a file copy, and executing compression operation on the target file;
and transmitting the compressed target file to the hosts belonging to the other machine rooms so as to enable the hosts of the other machine rooms to execute file writing operation.
Preferably, the step of writing the target file as a file copy into the host computer of the second machine room, and transmitting the target file to the host computers belonging to the other machine rooms, so that the host computers of the other machine rooms execute the file writing operation includes:
and executing the de-duplication operation on the target file, writing the de-duplicated target file into the host computer of the second machine room as a file copy, and executing the compression operation on the de-duplicated target file, or executing the compression operation on the target file, executing the de-duplication operation on the compressed target file, and writing the de-duplicated and compressed target file into the host computer of the second machine room as a file copy.
Preferably, the step of performing a compression operation on the target file after de-duplication includes:
writing the target file after the duplication removal into a host computer of the second machine room as a file copy, and calculating the byte number of the target file after the duplication removal;
judging whether the byte number is larger than or equal to a preset byte number, and executing compression operation on the target file after de-duplication when the byte number is larger than or equal to the preset byte number.
Optionally, the file reading and writing method further includes:
when the damaged file copy is monitored, determining a host to which the damaged file copy belongs, and reading an undamaged file copy;
and writing the undamaged file copy into the host to which the damaged file copy belongs.
In addition, in order to achieve the above object, the present invention also provides a file reading and writing apparatus, including:
the acquisition module is used for acquiring a copy storage route group of a corresponding file according to the file reading request when the triggered file reading request is monitored;
the determining module is used for determining a first host which triggers the file reading request;
the judging module is used for judging whether the copy storage route belonging to the first host exists in the copy storage route group or not;
the reading module is used for reading the file copy from the first host if the copy storage route belonging to the first host exists in the copy storage route group;
and the reading module is further configured to read the file copy across the machine room if the copy storage route belonging to the first host does not exist in the copy storage route group.
In addition, to achieve the above object, the present invention also provides a file reading and writing apparatus, including: the file reading and writing device comprises a memory, a processor and a file reading and writing program which is stored in the memory and can run on the processor, wherein the file reading and writing program realizes the steps of the file reading and writing method when being executed by the processor.
In addition, in order to achieve the above object, the present invention also provides a computer-readable storage medium having stored thereon a file read-write program which, when executed by a processor, implements the steps of the file read-write method as described above.
The invention provides a file reading and writing method, a device, equipment and a computer readable storage medium. By the method, when the file copy exists in the current host, the file copy is directly read from the current host, the file copy does not need to be read across the machine room, and when the file copy does not exist in the current host, the file copy is read across the machine room, so that the frequency of reading the file copy across the machine room can be effectively reduced, the bandwidth utilization rate between the machine rooms is reduced, and the reading speed of the file is improved.
Drawings
FIG. 1 is a schematic diagram of a file read-write device in a hardware operating environment according to an embodiment of the present invention;
FIG. 2 is a flowchart of a first embodiment of a method for reading and writing files according to the present invention;
FIG. 3 is a flowchart of a second embodiment of the method for reading and writing files according to the present invention;
FIG. 4 is a flowchart of a third embodiment of a method for reading and writing files according to the present invention;
FIG. 5 is a block diagram of an embodiment of a file reading/writing device according to the present invention.
The achievement of the objects, functional features and advantages of the present invention will be further described with reference to the accompanying drawings, in conjunction with the embodiments.
Detailed Description
It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the invention.
Referring to fig. 1, fig. 1 is a schematic diagram of a file read-write device of a hardware running environment according to an embodiment of the present invention.
The file read-write equipment of the embodiment of the invention can be terminal equipment such as a PC (personal computer) or a server (such as an X86 server) and the like which are provided with a virtualization platform.
As shown in fig. 1, the file read-write apparatus may include: aprocessor 1001, such as a CPU, anetwork interface 1004, auser interface 1003, amemory 1005, acommunication bus 1002. Wherein thecommunication bus 1002 is used to enable connected communication between these components. Theuser interface 1003 may include a Display, an input unit such as a Keyboard (Keyboard), and theoptional user interface 1003 may further include a standard wired interface, a wireless interface. Thenetwork interface 1004 may optionally include a standard wired interface, a wireless interface (e.g., WI-FI interface). Thememory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. Thememory 1005 may also optionally be a storage device separate from theprocessor 1001 described above.
It will be appreciated by those skilled in the art that the terminal structure shown in fig. 1 is not limiting of the terminal and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.
As shown in fig. 1, an operating system, a network communication module, a user interface module, and a file read-write program may be included in thememory 1005, which is a type of computer storage medium.
In the terminal shown in fig. 1, thenetwork interface 1004 is mainly used for connecting to a background server and performing data communication with the background server; theuser interface 1003 is mainly used for connecting a client (user side) and performing data communication with the client; and theprocessor 1001 may be used to call a file read-write program stored in thememory 1005 and perform operations in the following file read method embodiment.
Based on the hardware structure, the embodiment of the file reading and writing method is provided.
Referring to fig. 2, fig. 2 is a flowchart illustrating a first embodiment of a file reading and writing method according to the present invention.
In this embodiment, the method for reading and writing a file includes:
step S101, when a triggered file reading request is monitored, a copy storage route group of a corresponding file is obtained according to the file reading request, and a first host triggering the file reading request is determined, wherein the copy storage route group comprises storage routes of all copies of the file;
in this embodiment, the file reading and writing method is applied to a file reading and writing device, where the file reading and writing device may be selected from the devices shown in fig. 1, and the file reading and writing device is used for reading a file copy from a sub-volume of a host and writing the file copy into the sub-volume of the host. The file read-write equipment comprises a plurality of hosts, wherein the hosts are distributed in at least two different machine rooms, and the files copies are stored in the partial volumes of the hosts.
When the file reading and writing equipment monitors the triggered file reading request, the copy storage route group of the corresponding file is acquired according to the file reading request, and the host triggering the file reading request is determined and recorded as the first host. The method for acquiring the copy storage route group specifically comprises the following steps: the file read-write device reads the file identifier from the file read request and obtains the copy storage route group corresponding to the file identifier from the local. The file identifier may be a file name or a file number, which is not specifically limited in this embodiment.
Step S102, judging whether a copy storage route belonging to the first host exists in the copy storage route group;
in this embodiment, after determining the host that triggers the file reading request and obtaining the copy storage route group, the file reading and writing device determines whether a copy storage route belonging to the first host exists in the copy storage route group, that is, determines whether a corresponding file copy exists in the first host, specifically, obtains a file identifier from the file reading request, and determines whether a file copy corresponding to the file identifier exists in the first host.
Step S103, if the copy storage route belonging to the first host exists in the copy storage route group, reading a file copy from the first host;
step S104, if the copy storage route belonging to the first host does not exist in the copy storage route group, reading the file copy across the machine room.
In this embodiment, if there is a copy storage route belonging to the first host in the copy storage route group, that is, if there is a corresponding copy file in the first host, the file read-write device reads a copy of the file from the first host according to the copy storage route belonging to the first host, that is, directly reads the copy of the file from the local place, without crossing the machine room, if there is no copy storage route belonging to the first host in the copy storage route group, then reads the copy of the file from the machine room, that is, obtains the copy storage routes of the hosts belonging to the remaining machine rooms from the copy storage route group, and reads the copy of the file from the hosts in the remaining machine rooms based on the copy storage routes of the hosts in the remaining machine rooms.
Further, since the copy of the file is stored in not only one host in one machine room, in order to further reduce the bandwidth usage between the machine rooms, when the copy storage route belonging to the first host does not exist in the copy storage route group, the copy of the file can be read from the remaining hosts belonging to the same machine room, specifically, if the copy storage route belonging to the first host does not exist in the copy storage route group, the file reading and writing device determines the first machine room to which the first host belongs, and determines whether the copy storage route belonging to the remaining hosts of the first machine room exists in the copy storage route group, that is, determines whether the copy of the file is stored in the remaining hosts belonging to the first machine room, and if the copy storage route belonging to the remaining hosts of the first machine room exists in the copy storage route group, that is, the copy of the file is stored in the remaining hosts belonging to the first machine room, the copy of the file is read from the remaining hosts of the first machine room; and if the copy storage route group does not exist the copy storage routes of the other hosts belonging to the first machine room, namely the other hosts belonging to the first machine room do not store the file copies, reading the file copies across the machine room.
In this embodiment, when a triggered file reading request is monitored, the copy storage route group of the corresponding file is obtained, a host that triggers the file reading request is determined, then whether a copy storage route belonging to the host exists in the copy storage route group is judged, if so, a copy of the file is read from the host based on the copy storage route, and if not, the copy of the file is read across a machine room. By the method, when the file copy exists in the current host, the file copy is directly read from the current host, the file copy does not need to be read across the machine room, and when the file copy does not exist in the current host, the file copy is read across the machine room, so that the frequency of reading the file copy across the machine room can be effectively reduced, the bandwidth utilization rate between the machine rooms is reduced, and the reading speed of the file is improved.
Further, referring to fig. 3, based on the first embodiment, a second embodiment of the file reading and writing method of the present invention is proposed, which is different from the foregoing embodiment in that the file reading and writing method further includes:
step S105, when a triggered file writing request is monitored, a target file to be written is obtained, a second host triggering the file writing request is determined, and a second machine room to which the second host belongs is determined;
and step S106, writing the target file into the host machines of the second machine room as a file copy, and transmitting the target file to the host machines belonging to the other machine rooms so as to enable the host machines of the other machine rooms to execute file writing operation.
In this embodiment, when the existing writing manner of the file copy is writing the file, the file to be written is generally written on different partitions of different hosts at the same time, the file to be written is required to be synchronously transmitted to other machine rooms, the other machine rooms are synchronously written, the transmitted data amount is the product of the written data and the number of the file copies, the product is larger, that is, the transmitted data amount between the machine rooms is larger, the bandwidth between the machine rooms is limited, more time is required to be consumed for transmitting the data, the writing speed of the data is lower, in order to solve the above problem, the file copy to be written can be written in the host of the machine room where the host is located, then the file copy to be written is transmitted to the hosts of the other machine rooms, and the host of the other machine rooms synchronously writes the file copy. Specifically, when the triggered file writing request is monitored, a target file to be written is obtained, a second host triggering the file writing request is determined, a second machine room to which the second host belongs is determined, then the target file is written into the host of the second host machine room as a file copy, and the target file is transmitted to the hosts belonging to the other machine rooms so as to be used by the hosts of the other machine rooms for executing file writing operation. Through the mode, the data transmission quantity between the machine rooms can be effectively reduced, and the time consumed by data transmission can be reduced, so that the writing speed of the file is improved.
In another embodiment, since the written file and the original file may have the same data, in order to reduce the duplication of writing the same data, further increase the writing speed of the data, the file writing operation may be performed after the duplication removal process is performed on the file to be written, specifically, the file writing and reading device performs the duplication removal operation on the target file, writes the duplicated target file as a copy of the file into the host of the second machine room, and then transmits the duplicated target file to the hosts belonging to the other machine rooms, so that the hosts of the other machine rooms perform the file writing operation, that is, write the target file after the duplication is written into the corresponding sub-volumes of the host. The method for de-duplicating the target file specifically comprises the following steps: and obtaining an original file copy, and comparing the original file copy with the target file, so that the same data in the target file as the original file is removed, and the target file is de-duplicated. By de-duplicating the file copy, the data volume written into the host can be reduced, and the writing speed of the file can be further improved.
As another embodiment, since the files to be written need to be transferred across the machine rooms, and the bandwidth between the machine rooms and the data amount of the files to be written are factors affecting the transfer time, and the bandwidth between the machine rooms is generally unchanged, in order to further reduce the transfer time, the target files may be compressed and then transferred to the hosts of the other machine rooms, specifically, the file reading and writing device writes the target files as file copies in the hosts of the second machine room, performs the compression operation on the target files, and then transfers the target files after the compression operation to the hosts belonging to the other machine rooms, so that the hosts of the other machine rooms perform the file writing operation. By compressing the target file and then transmitting the compressed target file to the hosts of other machine rooms, the time consumption of data transmission can be further reduced, and the writing speed of the file is indirectly improved.
In another embodiment, since the compression of the file is performed by using a system resource, which is more occupied and is liable to cause problems, the file read/write device calculates the number of bytes of the target file after writing the target file as a copy of the file into the host of the second machine room, and determines whether the number of bytes of the target file is greater than or equal to a preset number of bytes, and if the number of bytes is greater than or equal to the preset number of bytes, performs the compression operation on the target file. It should be noted that the above-mentioned preset number of bytes can be set by those skilled in the art based on practical situations, and the present embodiment is not limited thereto. By the mode, the target file is compressed only when the target file is large, and system resources are not occupied for a long time.
In another embodiment, to further increase the writing speed of the file, the file writing and reading device may perform the de-duplication operation and the compression on the file to be written, specifically, the file writing and reading device performs the de-duplication operation on the target file, and writes the de-duplicated target file as a text copy into the host of the second machine room, and compresses the de-duplicated target file, or may perform the compression operation on the target file first, perform the de-duplication operation on the compressed target file, and write the compressed and de-duplicated target file as a file copy into the host of the second machine room, and finally transmit the de-duplicated and compressed target file to the hosts belonging to the other machine rooms, so that the hosts of the other machine rooms perform the file writing operation. Because the target file is de-duplicated, the written data can be reduced, the writing speed of the file can be directly improved, and meanwhile, after the target file is compressed, the target file is transmitted to the hosts of other machine rooms, so that the time consumption of data transmission can be further reduced, and the writing speed of the file is indirectly improved.
In another embodiment, since the compression of the file is required to occupy system resources, which occupy more system resources and are prone to cause problems, the file reading and writing device calculates the number of bytes of the target file after duplication after writing the target file after duplication as a copy of the file into the host of the second machine room, and determines whether the number of bytes is greater than or equal to a preset number of bytes, and if the number of bytes is greater than or equal to the preset number of bytes, performs the compression operation on the target file after duplication. By the mode, the target file after the duplication removal is compressed only when the target file after the duplication removal is large, and system resources are not occupied for a long time.
Further, referring to fig. 4, based on the first or second embodiment, a third embodiment of the file reading and writing method of the present invention is proposed, which is different from the foregoing embodiment in that the file reading and writing method further includes:
step S107, when the damaged file copy is monitored, determining the host to which the damaged file copy belongs, and reading the undamaged file copy;
step S108, the undamaged file copy is written into the host to which the damaged file copy belongs.
In this embodiment, during the actual running process, a certain file copy stored in the host computer may be damaged, so that the file copy needs to be reconstructed, specifically, when the file read-write device monitors the damaged file copy, the host computer to which the damaged file copy belongs is determined, the undamaged file copy is read, and the undamaged file copy is written into the host computer to which the damaged file copy belongs.
It should be noted that, the undamaged file copies may be stored in the same host, may also be stored in other hosts in the same machine room, and may also be stored in hosts in other machine rooms, when the undamaged file copies are read, there is a situation that the undamaged file copies are read across the machine room, which affects the reading speed and thus affects the reconstruction of the file copies, and when the undamaged file copies need to be read, the file reading and writing device obtains a copy storage route group, removes the copy storage route where the damaged file copies are located from the copy storage route group, obtains a target copy storage route group, and determines whether the copy storage route belonging to the host exists in the target copy storage route group, if the copy storage route belonging to the host exists in the target copy storage route group, the undamaged file copies exist in the host, and obtains the undamaged file copies from the host; if the target copy storage route group does not have the copy storage route belonging to the host, the undamaged file copy is read across the machine room.
As another implementation manner, if the copy storage route belonging to the host exists in the target copy storage route group, further determining a machine room of the host, and judging whether the copy storage route belonging to the remaining hosts of the machine room exists in the target copy storage route group, if the copy storage route belonging to the remaining hosts of the machine room exists in the target copy storage route group, reading undamaged file copies from the remaining hosts of the machine room, and if the copy storage route belonging to the remaining hosts of the machine room does not exist in the target copy storage route group, reading undamaged file copies across the machine room.
In this embodiment, when a damaged file copy is detected, the file copy may be reconstructed in time.
The invention also provides a file reading and writing device.
Referring to fig. 5, fig. 5 is a schematic block diagram of a file reading and writing device according to an embodiment of the invention.
In this embodiment, the file reading and writing device includes:
an obtainingmodule 101, configured to obtain, when a triggered file reading request is monitored, a copy storage routing group of a corresponding file according to the file reading request;
a determiningmodule 102, configured to determine a first host that triggers the file read request;
a judgingmodule 103, configured to judge whether a copy storage route belonging to the first host exists in the copy storage route group;
areading module 104, configured to read a file copy from the first host if a copy storage route belonging to the first host exists in the copy storage route group;
thereading module 104 is further configured to read the file copy across the machine room if no copy storage route belonging to the first host exists in the copy storage route group.
Further, the determiningmodule 102 is further configured to determine, if there is no copy storage route belonging to the first host in the copy storage route group, a first machine room to which the first host belongs;
the judgingmodule 103 is further configured to judge whether there are duplicate storage routes of the remaining hosts belonging to the first machine room in the duplicate storage route group;
thereading module 104 is further configured to read a file copy from the remaining hosts if there are copy storage routes of the remaining hosts belonging to the first machine room in the copy storage route group;
thereading module 104 is further configured to read the file copy across the machine room if there is no copy storage route belonging to the remaining hosts in the copy storage route group.
Further, the file reading and writing device further includes:
the obtainingmodule 101 is further configured to obtain, when a triggered file writing request is monitored, a target file to be written, determine a second host that triggers the file writing request, and determine a second machine room to which the second host belongs;
the writing module is used for writing the target file into the host computer of the second machine room as a file copy;
and the transmission module is used for transmitting the target file to the hosts belonging to the other machine rooms so as to enable the hosts of the other machine rooms to execute file writing operation.
Further, the writing module is further configured to perform a deduplication operation on the target file, and write the deduplicated target file as a file copy into a host of the second machine room;
the transmission module is further used for transmitting the target file after the duplication removal to the hosts belonging to other machine rooms so as to enable the hosts of the other machine rooms to execute file writing operation;
the writing module is further configured to write the target file as a file copy into a host of the second machine room, and perform a compression operation on the target file;
the transmission module is further configured to transmit the compressed target file to hosts belonging to other machine rooms, so that the hosts of the other machine rooms execute file writing operation.
Further, the writing module is further configured to perform a deduplication operation on the target file, write the deduplicated target file as a file copy into a host of the second machine room, and perform a compression operation on the deduplicated target file, or perform a compression operation on the target file, perform a deduplication operation on the compressed target file, and write the deduplicated and compressed target file as a file copy into the host of the second machine room;
the transmission module is further configured to transmit the target file after being subjected to the duplication and compression to hosts belonging to other machine rooms, so that the hosts of the other machine rooms execute file writing operation.
Further, the writing module is further configured to:
writing the target file after the duplication removal into a host computer of the second machine room as a file copy, and calculating the byte number of the target file after the duplication removal;
judging whether the byte number is larger than or equal to a preset byte number, and executing compression operation on the target file after de-duplication when the byte number is larger than or equal to the preset byte number.
Further, the file reading and writing device further includes:
the determiningmodule 102 is further configured to determine, when the damaged file copy is detected, a host to which the damaged file copy belongs, and read a file copy that is not damaged;
and the reconstruction module is used for writing the undamaged file copy into the host to which the damaged file copy belongs.
The embodiments of the file reading and writing device of the present invention may refer to the embodiments of the file reading and writing method, and will not be described herein.
According to the file reading and writing device provided by the invention, when the file copy exists in the current host, the file copy is directly read from the current host, the file copy does not need to be read across the machine room, and when the file copy does not exist in the current host, the file copy is read across the machine room, so that the frequency of reading the file copy across the machine room can be effectively reduced, the bandwidth utilization rate between the machine rooms is reduced, and the reading speed of the file is improved.
The invention also provides a computer readable storage medium.
The computer readable storage medium of the present invention stores a file read-write program which, when executed by a processor, implements the steps of the file read-write method as described above.
The method implemented when the file read-write program running on the processor is executed may refer to various embodiments of the file read-write method of the present invention, which are not described herein again.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or system. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or system that comprises the element.
The foregoing embodiment numbers of the present invention are merely for the purpose of description, and do not represent the advantages or disadvantages of the embodiments.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (e.g. ROM/RAM, magnetic disk, optical disk) as described above, comprising instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method according to the embodiments of the present invention.
The foregoing description is only of the preferred embodiments of the present invention, and is not intended to limit the scope of the invention, but rather is intended to cover any equivalents of the structures or equivalent processes disclosed herein or in the alternative, which may be employed directly or indirectly in other related arts.
Claims (9)
1. The file reading and writing method is characterized by comprising the following steps of:
when a triggered file reading request is monitored, acquiring a copy storage route group of a corresponding file according to the file reading request, and determining a first host for triggering the file reading request, wherein the copy storage route group comprises storage routes of all copies of the file;
judging whether a copy storage route belonging to the first host exists in the copy storage route group or not;
if the copy storage route belonging to the first host exists in the copy storage route group, reading a file copy from the first host;
if the copy storage route belonging to the first host does not exist in the copy storage route group, reading a file copy across a machine room;
when a triggered file writing request is monitored, acquiring a target file to be written, determining a second host triggering the file writing request, and determining a second machine room to which the second host belongs;
and writing the target file into the host computer of the second machine room as a file copy, and transmitting the target file to the host computers belonging to other machine rooms so as to enable the host computers of the other machine rooms to execute file writing operation.
2. The method for reading and writing files according to claim 1, further comprising, before the step of reading the copy of the file across the machine room:
if the copy storage route group does not have the copy storage route belonging to the first host, determining a first machine room to which the first host belongs;
judging whether the copy storage route group has copy storage routes of other hosts belonging to the first machine room or not;
if the copy storage route group has the copy storage routes of the other hosts belonging to the first machine room, reading file copies from the other hosts;
if the copy storage route group does not have the copy storage routes of the rest hosts belonging to the first machine room, executing the steps: the file copy is read across the machine room.
3. The method for reading and writing files according to claim 1, wherein the step of writing the target file as a file copy into the host computer of the second machine room and transmitting the target file to the host computers belonging to the remaining machine rooms for the host computers of the remaining machine rooms to perform the file writing operation includes:
performing a deduplication operation on the target file, and writing the deduplicated target file into a host of the second machine room as a file copy;
transmitting the target file after the duplication removal to the hosts belonging to the other machine rooms so as to enable the hosts of the other machine rooms to execute file writing operation;
or, writing the target file into a host computer of the second machine room as a file copy, and executing compression operation on the target file;
and transmitting the compressed target file to the hosts belonging to the other machine rooms so as to enable the hosts of the other machine rooms to execute file writing operation.
4. The method for reading and writing files according to claim 1, wherein the step of writing the target file as a file copy into the host computer of the second machine room and transmitting the target file to the host computers belonging to the remaining machine rooms for the host computers of the remaining machine rooms to perform the file writing operation includes:
performing a deduplication operation on the target file, writing the deduplicated target file into a host of the second machine room as a file copy, and performing a compression operation on the deduplicated target file, or performing a compression operation on the target file, performing a deduplication operation on the compressed target file, and writing the deduplicated and compressed target file into the host of the second machine room as a file copy;
and transmitting the target file subjected to the duplication and compression to the hosts belonging to the other machine rooms so as to enable the hosts of the other machine rooms to execute file writing operation.
5. The method for reading and writing a file as claimed in claim 4, wherein the step of performing a compression operation on the deduplicated target file comprises:
writing the target file after the duplication removal into a host computer of the second machine room as a file copy, and calculating the byte number of the target file after the duplication removal;
judging whether the byte number is larger than or equal to a preset byte number, and executing compression operation on the target file after de-duplication when the byte number is larger than or equal to the preset byte number.
6. The file reading and writing method according to any one of claims 1 to 5, characterized in that the file reading and writing method further comprises:
when the damaged file copy is monitored, determining a host to which the damaged file copy belongs, and reading an undamaged file copy;
and writing the undamaged file copy into the host to which the damaged file copy belongs.
7. A file reading and writing apparatus, characterized in that the file reading and writing apparatus includes:
the acquisition module is used for acquiring a copy storage route group of a corresponding file according to the file reading request when the triggered file reading request is monitored;
the determining module is used for determining a first host which triggers the file reading request;
the judging module is used for judging whether the copy storage route belonging to the first host exists in the copy storage route group or not;
the reading module is used for reading the file copy from the first host if the copy storage route belonging to the first host exists in the copy storage route group;
the reading module is further configured to read a file copy across a machine room if a copy storage route belonging to the first host does not exist in the copy storage route group;
the acquisition module is further used for acquiring a target file to be written when the triggered file writing request is monitored, determining a second host for triggering the file writing request, and determining a second machine room to which the second host belongs;
the writing module is used for writing the target file into the host computer of the second machine room as a file copy;
and the transmission module is used for transmitting the target file to the hosts belonging to the other machine rooms so as to enable the hosts of the other machine rooms to execute file writing operation.
8. A file reading and writing apparatus, characterized in that the file reading and writing apparatus includes: a memory, a processor and a file read-write program stored on the memory and executable on the processor, which when executed by the processor implements the steps of the file read-write method according to any one of claims 1 to 6.
9. A computer-readable storage medium, wherein a file read-write program is stored on the computer-readable storage medium, which when executed by a processor, implements the steps of the file read-write method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910170232.4ACN109933566B (en) | 2019-03-06 | 2019-03-06 | File reading and writing method, device, equipment and computer readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910170232.4ACN109933566B (en) | 2019-03-06 | 2019-03-06 | File reading and writing method, device, equipment and computer readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109933566A CN109933566A (en) | 2019-06-25 |
| CN109933566Btrue CN109933566B (en) | 2023-05-12 |
Family
ID=66986475
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910170232.4AActiveCN109933566B (en) | 2019-03-06 | 2019-03-06 | File reading and writing method, device, equipment and computer readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109933566B (en) |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013025450A (en)* | 2011-07-19 | 2013-02-04 | Nippon Telegr & Teleph Corp <Ntt> | Distributed file management system, distributed file arrangement method and program |
Family Cites Families (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7827144B1 (en)* | 2005-11-17 | 2010-11-02 | Hewlett-Packard Development Company, L.P. | Methods of reading and writing data |
| US8151068B2 (en)* | 2008-02-04 | 2012-04-03 | Microsoft Corporation | Data copy management for faster reads |
| US8959301B2 (en)* | 2012-11-09 | 2015-02-17 | International Business Machines Corporation | Accessing data in a storage system |
| JP6005122B2 (en)* | 2014-11-17 | 2016-10-12 | インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ | How to span and write files to multiple tape cartridges |
| CN106686077A (en)* | 2016-12-24 | 2017-05-17 | 上海七牛信息技术有限公司 | System and method for processing network requests across double-layer proxies of data centers in computer rooms |
| CN108197159B (en)* | 2017-12-11 | 2020-07-10 | 厦门集微科技有限公司 | Data reading and writing method and device based on distributed file system |
| CN108415986B (en)* | 2018-02-11 | 2020-10-30 | 杭州朗和科技有限公司 | Data processing method, device, system, medium and computing equipment |
- 2019
- 2019-03-06CNCN201910170232.4Apatent/CN109933566B/enactiveActive
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2013025450A (en)* | 2011-07-19 | 2013-02-04 | Nippon Telegr & Teleph Corp <Ntt> | Distributed file management system, distributed file arrangement method and program |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109933566A (en) | 2019-06-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110531940B (en) | Video file processing method and device | |
| US9690487B2 (en) | Storage apparatus and method for controlling storage apparatus | |
| US9977746B2 (en) | Processing of incoming blocks in deduplicating storage system | |
| US10817386B2 (en) | Virtual machine recovery method and virtual machine management device | |
| CA2897129C (en) | Data processing method and device in distributed file storage system | |
| US8418003B2 (en) | Log collecting apparatus, program, method, and system | |
| US8326811B2 (en) | File management method and computer system | |
| US20150113218A1 (en) | Distributed Data Processing Method and Apparatus | |
| US8521692B1 (en) | Storage system and method for controlling storage system | |
| WO2019001521A1 (en) | Data storage method, storage device, client and system | |
| EP3364303B1 (en) | Data arrangement method, storage apparatus, storage controller and storage array | |
| US20160291877A1 (en) | Storage system and deduplication control method | |
| CN105493080B (en) | Method and device for deduplication data based on context awareness | |
| CN107850983B (en) | Computer system, storage device and data management method | |
| CN110569157B (en) | Storage testing method, device, server and storage medium | |
| US11372570B1 (en) | Storage device, computer system, and data transfer program for deduplication | |
| JP2008269050A (en) | Compression control apparatus and method | |
| US11593312B2 (en) | File layer to block layer communication for selective data reduction | |
| CN114153647B (en) | Rapid data verification method, device and system for cloud storage system | |
| CN108984589A (en) | A kind of method for writing data and server | |
| US11487428B2 (en) | Storage control apparatus and storage control method | |
| CN103685359B (en) | Data processing method and device | |
| US11513739B2 (en) | File layer to block layer communication for block organization in storage | |
| CN109933566B (en) | File reading and writing method, device, equipment and computer readable storage medium | |
| CN118760398A (en) | Storage device access method and device, and computer program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |