CN111399760B - NAS cluster metadata processing method and device, NAS gateway and medium - Google Patents
NAS cluster metadata processing method and device, NAS gateway and mediumDownload PDFInfo
- Publication number
- CN111399760B CN111399760BCN201911133793.3ACN201911133793ACN111399760BCN 111399760 BCN111399760 BCN 111399760BCN 201911133793 ACN201911133793 ACN 201911133793ACN 111399760 BCN111399760 BCN 111399760B
- Authority
- CN
- China
- Prior art keywords
- metadata
- nas
- distributed cache
- file
- cache platform
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
- G06F3/0656—Data buffering arrangements
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Technical Field
The present application relates to the field of storage, and in particular, to a method and an apparatus for processing NAS cluster metadata, an NAS gateway, and a medium.
Background
NAS (Network Attached Storage) refers to a device having a data Storage function and connected to a Network. The NAS device may support various Network connection protocols, including CIFS (Common Internet File System), NFS (Network File System), and FTP (File Transfer Protocol).
The NAS cluster refers to a cluster composed of a plurality of NAS devices, and can provide NAS services with high performance, high availability, or high load balancing.
The NAS cluster includes: cloud storage, NAS gateway (NAS GATEWAY), and NAS MASTER (NAS administrator) devices. The cloud storage device provides underlying data storage functions. The NAS gateway is in interface connection with the NAS client and can provide an NAS protocol access interface for the NAS client. NAS MASTER provides unified storage and management of metadata for NAS cluster files.
When the NAS gateway receives a file writing request of the NAS client, the NAS gateway writes file data into the cloud storage device, and sends the generated metadata to the NAS MASTER device for storage. When the NAS gateway receives a file reading request of a client, the NAS gateway acquires data of a specified file from the cloud storage device, and acquires metadata from the NAS MASTER device and feeds the metadata back to the NAS client.
When a file is read each time, the NAS gateway needs to interact with NAS MASTER equipment, and metadata is obtained from NAS MASTER equipment, so that the reading speed of the metadata is greatly reduced.
Disclosure of Invention
In view of this, the present application provides a method and an apparatus for processing NAS cluster metadata, a NAS gateway, and a medium, so as to improve the metadata reading efficiency.
Specifically, the method is realized through the following technical scheme:
according to a first aspect of the present application, a method for processing metadata of an NAS cluster is provided, where the method is applied to an NAS gateway in the NAS cluster, and a distributed cache platform is set in the NAS cluster; the distributed cache platform is in butt joint with the NAS gateway; the method comprises the following steps:
when it is determined that a file is successfully written into the cloud storage device in the NAS cluster, writing metadata of the file into a memory of the distributed cache platform;
when a read request sent by an NAS client side is received, detecting whether the target metadata requested by the read request is cached in the distributed cache platform or not;
if yes, acquiring the target metadata from the memory of the distributed cache platform, and returning the acquired target metadata to the NAS client;
if not, the target metadata is obtained from the NAS MASTER device, and the obtained target metadata is returned to the NAS client.
Optionally, after the target metadata is obtained from the NAS MASTER apparatus, the method further comprises:
and storing the acquired target metadata to the distributed cache platform, and setting aging time for the target metadata.
Optionally, the method further includes:
when it is determined that the written file is successfully written into the cloud storage device in the NAS cluster, storing metadata of the written file into a persistent storage medium of a distributed cache platform;
periodically reporting the metadata of the persistent storage to NAS MASTER equipment, and after determining that the metadata is reported successfully, deleting the metadata which is reported successfully from the persistent storage medium.
Optionally, after the determining that the written file is successfully written into the cloud storage device in the NAS cluster, writing the metadata of the file into the memory of the distributed cache platform, the method further includes:
setting aging duration for the metadata of the file written into the distributed cache memory;
the method further comprises the following steps:
and periodically deleting the metadata reaching the aging duration in the memory of the distributed cache platform.
Optionally, the method further includes:
when a deletion request sent by a NAS client is received, deleting the metadata indicated by the deletion request from the memory of the distributed cache platform and the NAS MASTER.
According to a second aspect of the present application, a metadata processing apparatus for a NAS cluster is provided, where the apparatus is applied to a NAS gateway in the NAS cluster, and a distributed cache platform is set in the NAS cluster; the distributed cache platform interfaces with the NAS gateway; the device comprises:
the writing unit is used for writing the metadata of the file into the memory of the distributed cache platform when the file is determined to be successfully written into the cloud storage device in the NAS cluster;
the detection unit is used for detecting whether the target metadata requested by the read request is cached in the distributed cache platform or not when the read request sent by the NAS client side is received;
a first obtaining unit, configured to obtain the target metadata from a memory of the distributed cache platform if the target metadata is found to be valid, and return the obtained target metadata to the NAS client;
and the second acquisition unit is used for acquiring the target metadata from the NAS MASTER equipment and returning the acquired target metadata to the NAS client if the target metadata is not acquired from the NAS MASTER equipment.
Optionally, the apparatus further comprises:
and the caching unit is used for storing the acquired target metadata to the distributed caching platform and setting aging time for the target metadata.
Optionally, the apparatus further comprises:
the storage unit is used for storing the metadata of the written file to a persistent storage medium of a distributed cache platform when the written file is determined to be successfully written into the cloud storage device in the NAS cluster;
and the reporting unit is used for periodically reporting the metadata which is stored persistently to the NAS MASTER equipment and deleting the metadata which is reported successfully from the persistent storage medium after the metadata which is stored persistently is determined to be reported successfully.
Optionally, the write-in unit is further configured to set an aging duration for the metadata of the file written into the distributed cache memory;
the device further comprises:
the first deleting unit is used for periodically deleting the metadata reaching the aging duration in the memory of the distributed cache platform.
Optionally, the apparatus further comprises:
a second deleting unit, configured to delete, when receiving a deletion request sent by the NAS client, the metadata indicated by the deletion request from the memory of the distributed cache platform and the NAS MASTER.
According to a third aspect of the present application, there is provided an electronic device comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to perform the method of the first aspect.
According to a fourth aspect of the present application, there is provided a machine-readable storage medium having stored thereon machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the method of the first aspect.
As can be seen from the above description, on one hand, when a file is successfully written into a cloud storage device, the NAS gateway device writes metadata of the file into a memory of the distributed cache platform. When receiving a read request of metadata, the NAS gateway may first search the metadata requested by the read request from the memory of the distributed cache platform, and if the metadata is found, return the found metadata to the client. Since the metadata is read from the memory of the distributed cache platform, the access speed of the metadata can be greatly improved.
On the other hand, after the NAS gateway successfully writes the file into the cloud storage device, the NAS gateway also stores the metadata of the file in the persistent storage medium of the distributed cache platform, so that in the process of reporting the metadata to the NAS MASTER device, even if the distributed cache platform fails, the unreported metadata can be retrieved from the persistent storage medium after the failure of the distributed cache platform is recovered, and the unreported data is reported to the NAS MASTER device, so that the problem that the metadata to be reported is lost due to the failure of the distributed cache platform can be prevented, and the security of metadata reporting is ensured.
Drawings
FIG. 1 is an architectural diagram of a prior art NAS cluster;
FIG. 2 is an architectural diagram of a NAS cluster according to an exemplary embodiment of the present application;
FIG. 3 is a flow chart illustrating a method for metadata processing for a NAS cluster according to an exemplary embodiment of the present application;
FIG. 4 is a flow chart illustrating a method of metadata writing in accordance with an exemplary embodiment of the present application;
FIG. 5 is a flow chart illustrating a method for reading metadata according to an exemplary embodiment of the present application;
FIG. 6 is a flow diagram illustrating a metadata deletion in accordance with an exemplary embodiment of the present application;
FIG. 7 is a hardware block diagram of a NAS gateway according to an exemplary embodiment of the present application;
fig. 8 is a block diagram of a NAS cluster metadata processing apparatus according to an exemplary embodiment of the present application.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if," as used herein, may be interpreted as "at … …" or "at … …" or "in response to a determination," depending on the context.
Referring to fig. 1, fig. 1 is a schematic diagram of an architecture of a conventional NAS cluster.
The NAS cluster includes:cloud storage device 101,NAS gateway 102, andNAS MASTER device 103.
Among other things,cloud storage device 101 provides underlying data storage functionality.
TheNAS gateway 102 interfaces with theNAS client 104 and may provide an interface for NAS protocol access for theNAS client 104.
5363 and uniformly storing and managing metadata of files stored in the cloud storage device of the NAS cluster by NAS MASTER.
The metadata of the file may include file attributes (such as file size, etc.), and a storage address, an index, etc. of the file on the cloud storage device. Here, the metadata of the file is merely exemplary and is not particularly limited.
When theNAS gateway 102 receives a write request for a file from theNAS client 104, theNAS gateway 102 writes the file into thecloud storage device 101, and sends metadata generated after the file is successfully written to theNAS MASTER device 103 for storage.
When theNAS gateway 102 receives a read request for a file from theNAS client 104, theNAS gateway 102 acquires the file from thecloud storage device 101, acquires metadata of the file from theNAS MASTER device 103, and feeds back the file and the metadata of the file to theNAS client 104.
When reading the file each time, the NAS gateway needs to interact with the NAS MASTER device and obtain the metadata from the NAS MASTER device, so that the reading speed of the metadata is greatly reduced.
In view of this, the present application provides a method for processing metadata of a NAS cluster, which can effectively improve the efficiency of reading the metadata.
Referring to fig. 2, fig. 2 is a schematic diagram illustrating an architecture of a NAS cluster according to an exemplary embodiment of the present application.
On the basis of the existing NAS cluster, a distributed cache platform is configured, and the distributed cache platform can be in butt joint with an NAS gateway.
As shown in fig. 2, the NAS cluster provided in the present application includes:NAS gateway 201,cloud storage 202,NAS MASTER device 203, distributedcache platform 204, andcluster connection pool 205.
Therein, theNAS gateway 201 interfaces with theNAS client 206. TheNAS gateway 201 interfaces with acloud storage device 202. TheNAS gateway 201 interfaces withNAS MASTER devices 203. TheNAS gateway 201 also interfaces with the distributedcaching platform 204 through acluster connection pool 205.
TheNAS MASTER device 203 stores and manages metadata of files stored in the NAS cluster in a unified mode.
And the distributedcache platform 204 is used for providing a function of caching data. The distributedcache platform 204 may be built from a Redis platform.
The Redis platform is an open-source key-value storage system, and since data is stored in an internal memory in the Redis platform, the Redis platform can provide higher access speed to the outside. Therefore, the speed of accessing the cache data can be effectively improved by adopting the Redis platform to build the distributed cache platform.
In addition, the Redis platform also supports the storage of data in storage media such as a magnetic disk and the like, and realizes the persistent storage of the data.
Of course, the distributedcache platform 204 may also be built by other clusters for data caching, and here, the building manner of the distributed cache platform is only exemplarily described and is not specifically limited.
Taking the example that a distributed cache platform is built by a Redis platform, the structure of the Redis platform is introduced.
The Redis platform comprises a plurality of servers, and a plurality of Redis nodes can be deployed on each server. Of course. For the safety of data storage of the Redis platform, the nodes of the Redis platform can have two roles of a master role and a standby role. The same data is stored on the Redis master node and the Redis standby node corresponding to the Redis master node.
For example, as shown in fig. 2 in the present application.
Suppose that the Redis platform comprises 3 servers, and each server is provided with two Redis nodes in a master-slave mode. And the standby Redis node corresponding to each main Redis node is deployed on a server different from the main Redis node.
For example, a Redis master node A and a Redis standby node C are deployed on the server 1, a Redis master node B and a Redis standby node A are deployed on the server 2, and a Redis master node C and a Redis standby node B are deployed on the server 3. The Redis master node A and the Redis standby node A store the same data. The Redis master node B and the Redis standby node B store the same data. The Redis master node C and the Redis standby node C store the same data.
The advantages of deploying the master node and the standby node in this way are that: when one server is down, all Redis nodes are also deployed on other servers, so that the Redis platform can also normally provide services.
For example, still taking fig. 2 as an example, suppose that server 1 is down, and Redis master node a and Redis standby node C are not available, but server 2 is still normal, so Redis standby node a can also provide services, server 3 is still normal, redis master node C can also provide services, and since Redis node A, B, C can both provide services, the Redis platform can also provide services normally.
Here, the configuration of the Redis platform is only exemplarily described by way of example, and is not particularly limited.
It should be noted that, when the distributed cache platform is a Redis platform, storing the metadata in the memory or the persistent storage medium of the distributed cache platform described below refers to storing the metadata in the memory or the persistent storage medium of the specified Redis node according to the storage rule of the Redis cluster, and details are not described below.
Referring to fig. 3, fig. 3 is a flowchart illustrating a method for processing metadata of a NAS cluster according to an exemplary embodiment of the present application, where the method may be applied to the NAS gateway illustrated in fig. 2, and may include the following units.
Step 301: and when determining that the file is successfully written into the cloud storage device in the NAS cluster, writing the metadata of the file into a memory of the distributed cache platform.
When the NAS client needs to write a file, the NAS client may send a write request to the NAS gateway. The write request carries a file to be written by the NAS client.
After receiving a write request sent by the NAS client, the NAS gateway may write a file carried by the write request to the cloud storage device. When the NAS gateway receives a response message which is returned by the cloud storage device and is used for indicating that the file is successfully written, the NAS gateway can determine that the file is successfully written into the cloud storage device in the NAS cluster.
In addition, the response message also carries information such as file attributes of the file, storage addresses of the file on the cloud storage device, and an index, and the NAS gateway may generate metadata of the file based on the information.
After generating the metadata of the file, on one hand, the NAS gateway may write the metadata of the file into the memory of the distributed cache platform, and on the other hand, the NAS gateway may write the metadata of the file into a persistent storage medium of the distributed cache platform.
1) And writing the metadata of the file into the memory of the distributed cache platform aiming at the NAS gateway.
When the method is implemented, the NAS gateway can write the metadata of the file into the memory of the distributed cache platform and set aging duration for the metadata.
It should be noted that, on one hand, the metadata of the file is written into the memory of the distributed cache platform, so that the NAS gateway may first search the metadata requested by the read request from the memory of the distributed cache platform in the case of receiving the read request of the metadata, and if the metadata is found, return the found metadata to the client. Since the metadata is read from the memory of the distributed cache platform, the access speed of the metadata can be greatly improved. Therefore, the purpose of writing the metadata of the file into the memory of the distributed cache platform is to improve the access speed of the subsequent metadata.
On the other hand, the aging duration is set for the metadata stored in the memory of the distributed cache platform, so that the NAS gateway can delete the metadata reaching the aging duration in the memory of the distributed cache platform, thereby relieving the storage pressure of the memory of the distributed cache platform, and in addition, the metadata frequently accessed can be stored in the memory of the distributed cache platform.
2) And writing the metadata of the file into a persistent storage medium of the distributed cache platform aiming at the NAS gateway.
It should be noted that, after the NAS gateway generates the metadata of the file, the NAS gateway also needs to report the generated metadata to the NAS MASTER device, so that the NAS MASTER device can uniformly manage the metadata of the written file.
However, if a reporting mode that the NAS gateway directly reports the metadata in the memory of the distributed cache platform to the NAS MASTER device is adopted, it may happen that the metadata in the memory of the distributed cache platform is lost due to a failure of the distributed cache platform during the metadata reporting process, and since the metadata is stored in the memory, the NAS gateway cannot retrieve and report the lost data to the NAS MASTER device after the distributed cache platform is recovered.
Therefore, the reporting mode adopted by the embodiment of the application is as follows: the NAS gateway stores the metadata written into the file in a persistent storage medium of the distributed cache platform, and periodically reports the metadata in the persistent storage medium to NAS MASTER. After receiving a response message for indicating that the metadata is reported successfully, which is returned by the NAS MASTER device, the NAS gateway may determine that the metadata is reported successfully. The NAS gateway may delete the reported successful metadata from the persistent storage medium.
Since the metadata to be reported is stored in the persistent storage medium of the distributed cache platform, in the metadata reporting process, even if the distributed cache platform fails, after the failure of the distributed cache platform is recovered, the metadata which is not reported can be retrieved from the persistent storage medium, and the metadata which is not reported is reported to the NAS MASTER device, so that the problem that the metadata to be reported is lost due to the failure of the distributed cache platform can be prevented.
Step 302: when a read request sent by an NAS client side is received, whether the target metadata requested by the read request is cached in the distributed cache platform or not is detected.
Step 303: if the target metadata requested by the read request is cached in the distributed cache platform, acquiring the target metadata from a memory of the distributed cache platform, and returning the acquired target metadata to the NAS client;
step 304: and if the target metadata requested by the read request is not cached in the distributed cache platform, acquiring the target metadata from NAS MASTER equipment, and returning the acquired target metadata to the NAS client.
It should be noted that, because the metadata in the memory of the distributed cache platform is provided with the aging duration, and the NAS gateway may periodically delete the metadata reaching the aging duration, only a part of the frequently accessed metadata of the written file is stored in the distributed cache platform. Therefore, after receiving a read request sent by the NAS client, it needs to detect whether the target metadata requested by the read request is cached in the distributed cache platform.
In a specific implementation, when detecting that a user triggers an operation of reading target metadata, the NAS client may send a read request to the NAS gateway, where the read request carries an identifier of the target metadata (for example, a storage index of the target metadata, or the like).
For example, when a user selects a file on the NAS client and right-clicks to view the file attributes of the file, the NAS client may be triggered to send the read request to the NAS gateway.
Or, when a user copies a file to another place on the NAS client, the metadata of the file needs to be read first to obtain the storage address of the file on the cloud storage device, so that the file can be obtained from the cloud storage device and copied to another place. Therefore, when a user copies a file to another place on the NAS client, the NAS client is also triggered to send the read request to the NAS gateway.
The target metadata is any metadata, and is only for convenience of description. The target metadata is not specifically limited in this specification.
After receiving the read request sent by the NAS client, the NAS gateway may detect whether the target metadata requested by the read request (i.e., the target metadata indicated by the target metadata identifier carried in the read request) is stored in the memory of the distributed cache platform.
And if the target metadata requested by the read request is stored in the memory of the distributed cache platform, the NAS gateway acquires the target metadata from the memory of the distributed cache platform and returns the acquired target metadata to the NAS client.
If the target metadata requested by the read request is not stored in the memory of the distributed cache platform, the NAS gateway acquires the target metadata requested by the read request from the NAS MASTER device. The NAS gateway may return the acquired target metadata to the NAS client. In addition, the NAS gateway may further write the target metadata into a memory of the distributed cache platform, and set an aging duration for the target metadata.
As can be seen from the above description, on one hand, when the NAS gateway receives a read request for target metadata, the NAS gateway does not interact with the NAS MASTER device to obtain the target metadata according to the prior art, but checks whether the target metadata exists in the memory of the distributed cache platform, and if the target metadata exists, the target metadata can be directly read from the memory of the distributed cache platform. Since the metadata is read from the memory of the distributed cache platform, the access speed of the metadata can be greatly improved.
On the other hand, when the NAS gateway determines that the target metadata does not exist in the memory of the distributed cache platform, the NAS gateway may further obtain the target metadata from the NAS MASTER device, and write the target metadata into the memory of the distributed cache platform, so that when the NAS client accesses the target data again subsequently, the NAS gateway may directly obtain the target metadata from the memory of the distributed cache platform and return the target metadata to the NAS client.
In addition, the present specification also provides a metadata deletion method, thereby implementing deletion of metadata.
In implementation, when the NAS gateway device receives a deletion request of the NAS client, the metadata indicated by the deletion request may be deleted from the distributed cache platform and the NAS MASTER device.
For example, the deletion request carries a metadata identifier, and after receiving the deletion request, the NAS gateway device may delete the metadata indicated by the metadata identifier carried in the deletion request from the distributed cache platform and the NAS MASTER device.
It should be further noted that, since the NAS gateway needs to be docked through the cluster connection pool when being docked with the distributed cache platform, the NAS gateway needs to detect whether the cluster connection pool is initialized successfully first no matter the NAS gateway writes metadata into the distributed cache platform, or reads metadata from the distributed cache platform and deletes metadata. After the cluster connection pool is initialized successfully, metadata can be written into the distributed cache platform, read from the distributed cache platform, deleted, and the like.
As can be seen from the above description, on one hand, when a file is successfully written into a cloud storage device, the NAS gateway device writes metadata of the file into a memory of the distributed cache platform. When receiving a read request of metadata, the NAS gateway may first search the metadata requested by the read request from the memory of the distributed cache platform, and if the metadata is found, return the found metadata to the client. Since the metadata is read from the memory of the distributed cache platform, the access speed of the metadata can be greatly improved.
On the other hand, after the NAS gateway successfully writes the file into the cloud storage device, the NAS gateway also stores the metadata of the file in the persistent storage medium of the distributed cache platform, so that in the process of reporting the metadata to the NAS MASTER device, even if the distributed cache platform fails, the unreported metadata can be retrieved from the persistent storage medium after the failure of the distributed cache platform is recovered, and the unreported data is reported to the NAS MASTER device, so that the problem that the metadata to be reported is lost due to the failure of the distributed cache platform can be prevented, and the security of metadata reporting is ensured.
The NAS cluster metadata processing method provided by the present application is described in detail below by taking a distributed storage platform as a Redis platform as an example, and performing three aspects of metadata writing, metadata reading, and metadata deleting.
1. Writing of metadata
Referring to fig. 4, fig. 4 is a flowchart illustrating a metadata writing method according to an exemplary embodiment of the present application.
Step 401: when the NAS gateway receives a write request sent by the NAS client, files carried in the write request are written into the cloud storage equipment.
Step 402: and after receiving a response message which is returned by the cloud storage device and used for indicating that the file is successfully written, the NAS gateway generates the metadata of the file.
Step 403: the NAS gateway detects whether the cluster connection pool is initialized.
Step 404: if the cluster connection pool is not initialized, the NAS gateway writes the metadata of the file into NAS MASTER equipment.
Step 405: if the cluster connection pool is initialized, the NAS gateway stores the metadata of the file to a persistent storage medium of the Redis platform through the cluster connection pool, stores the metadata of the file in a memory of the Redis platform, and sets aging duration for the metadata stored in the memory.
Step 406: the NAS gateway periodically reports the metadata stored in the persistent storage medium of the Redis platform to NAS MASTER equipment in batches.
Step 407: and when a response message for indicating successful report returned by the NAS MASTER device is received, the NAS gateway deletes the metadata successfully reported from the persistent storage medium of the Redis platform.
2. Reading metadata:
referring to fig. 5, fig. 5 is a flowchart illustrating a metadata reading method according to an exemplary embodiment of the present application.
Step 501: the NAS gateway receives a read request sent by the NAS client.
Step 502: detecting whether a Redis connection pool is initialized.
Step 503: if the Redis connection pool is not initialized, the NAS gateway obtains metadata requested by the read request from the NAS MASTER device and returns the obtained metadata to the NAS client.
Step 504: if the Redis connection pool is initialized, the NAS gateway detects whether metadata requested by the read request is stored in a memory of the Redis platform;
step 505: and if the metadata requested by the read request is stored in the memory of the Redis platform, the NAS gateway acquires the metadata and returns the metadata to the NAS client.
Step 506: if the memory of the Redis platform does not have the metadata requested by the read request, the NAS gateway acquires the metadata from NAS MASTER equipment, stores the acquired metadata in the memory of the Redis platform and returns the acquired metadata to the NAS client.
3. And (3) deleting metadata:
referring to fig. 6, fig. 6 is a flow chart illustrating a metadata deletion according to an exemplary embodiment of the present application.
Step 601: the NAS gateway receives a deletion request sent by the NAS client.
Step 602: the NAS gateway detects whether the Redis connection pool is initialized.
If the Redis connection pool is initialized, then step 603 to step 604 are performed.
If the Redis connection pool is not initialized, go to step 604.
Step 603: the NAS gateway deletes the metadata indicated by the deletion request from the Redis platform memory.
Step 604: the NAS gateway deletes the metadata indicated by the deletion request from the NAS MASTER device.
In addition, the NAS gateway may periodically detect the aging duration of each metadata in the memory, and delete the metadata reaching the aging duration.
Referring to fig. 7, fig. 7 is a hardware structure diagram of a NAS gateway according to an exemplary embodiment of the present application.
The NAS gateway includes: acommunication interface 701, aprocessor 702, a machine-readable storage medium 703, and abus 704; thecommunication interface 701, theprocessor 702, and the machine-readable storage medium 703 are in communication with one another via abus 704. Theprocessor 702 may perform the NAS cluster metadata processing methods described above by reading and executing machine-executable instructions in the machine-readable storage medium 703 corresponding to NAS cluster metadata processing control logic.
The machine-readable storage medium 703 as referred to herein may be any electronic, magnetic, optical, or other physical storage device that can contain or store information such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: volatile memory, non-volatile memory, or similar storage media. In particular, the machine-readable storage medium 703 may be a RAM (random Access Memory), a flash Memory, a storage drive (e.g., a hard drive), a solid state drive, any type of storage disk (e.g., a compact disk, a DVD, etc.), or similar storage medium, or a combination thereof.
Referring to fig. 8, fig. 8 is a block diagram of a NAS cluster metadata processing apparatus according to an exemplary embodiment of the present application, where the apparatus is applicable to the NAS gateway shown in fig. 7, and a distributed cache platform is disposed in the NAS cluster; the distributed cache platform interfaces with the NAS gateway; the device comprises:
awriting unit 801, configured to write metadata of a file into a memory of the distributed cache platform when it is determined that the file is successfully written into the cloud storage device in the NAS cluster;
a detecting unit 802, configured to detect, when a read request sent by an NAS client is received, whether the target metadata requested by the read request is cached in the distributed cache platform;
a first obtainingunit 803, configured to, if yes, obtain the target metadata from the memory of the distributed cache platform, and return the obtained target metadata to the NAS client;
a second obtainingunit 804, configured to obtain the target metadata from a NAS MASTER device if the target metadata is not obtained, and return the obtained target metadata to the NAS client.
Optionally, the apparatus further comprises:
a caching unit 805 (not shown in fig. 8) configured to store the obtained target metadata to the distributed caching platform, and set an aging time for the target metadata.
Optionally, the apparatus further comprises:
a storage unit 806 (not shown in fig. 8) configured to store metadata of the written file to a persistent storage medium of a distributed caching platform when it is determined that the written file is successfully written to a cloud storage device in the NAS cluster;
a reporting unit 807 (not shown in fig. 8) configured to periodically report the persistently stored metadata to NAS MASTER equipment, and after it is determined that the metadata is reported successfully, delete the metadata that is reported successfully from the persistent storage medium.
Optionally, the writing unit 808 (not shown in fig. 8) is further configured to set an aging duration for the metadata written to the file in the distributed cache memory;
the device further comprises:
a first deleting unit 809 (not shown in fig. 8) configured to delete the metadata that reaches the aging duration in the memory of the distributed cache platform periodically.
Optionally, the apparatus further comprises:
a second deleting unit 810 (not shown in fig. 8) configured to, when receiving a deletion request sent by a NAS client, delete metadata indicated by the deletion request from the memory of the distributed cache platform and the NAS MASTER.
The implementation process of the functions and actions of each unit in the above device is specifically described in the implementation process of the corresponding step in the above method, and is not described herein again.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the scope of protection of the present application.
Claims (10)
1. The method is characterized in that the method is applied to an NAS gateway in an NAS cluster, and a distributed cache platform is arranged in the NAS cluster; the distributed cache platform interfaces with the NAS gateway; the method comprises the following steps:
when determining that a file is successfully written into the cloud storage device in the NAS cluster, writing metadata of the file into a memory of the distributed cache platform, and storing the metadata of the written file into a persistent storage medium of the distributed cache platform; periodically reporting the metadata which is stored persistently to NAS MASTER equipment, and deleting the metadata which is reported successfully from the persistent storage medium after determining that the metadata is reported successfully;
when a read request sent by an NAS client side is received, detecting whether the target metadata requested by the read request is cached in the distributed cache platform or not;
if yes, acquiring the target metadata from the memory of the distributed cache platform, and returning the acquired target metadata to the NAS client;
if not, the target metadata is obtained from the NAS MASTER device, and the obtained target metadata is returned to the NAS client.
2. The method of claim 1, wherein after the obtaining the target metadata from a NAS MASTER device, the method further comprises:
and storing the acquired target metadata to the distributed cache platform, and setting aging time for the target metadata.
3. The method of claim 1, wherein after the writing the metadata of the written file to the memory of the distributed caching platform upon determining that the file is successfully written to the cloud storage device in the NAS cluster, the method further comprises:
setting aging duration for the metadata of the file written into the distributed cache memory;
the method further comprises the following steps:
and periodically deleting the metadata reaching the aging duration in the memory of the distributed cache platform.
4. The method of claim 1, further comprising:
when a deletion request sent by a NAS client is received, deleting metadata indicated by the deletion request from a memory of the distributed cache platform and the NAS MASTER.
5. The device is characterized in that the device is applied to an NAS gateway in an NAS cluster, and a distributed cache platform is arranged in the NAS cluster; the distributed cache platform interfaces with the NAS gateway; the device comprises:
the writing unit is used for writing the metadata of the file into the memory of the distributed cache platform when the file is determined to be successfully written into the cloud storage device in the NAS cluster;
the detection unit is used for detecting whether the target metadata requested by the read request is cached in the distributed cache platform or not when the read request sent by the NAS client side is received;
a first obtaining unit, configured to obtain the target metadata from a memory of the distributed cache platform if the target metadata is found to be valid, and return the obtained target metadata to the NAS client;
a second obtaining unit, configured to obtain the target metadata from a NAS MASTER device if the target metadata is not obtained, and return the obtained target metadata to the NAS client;
the storage unit is used for storing the metadata of the written file to a persistent storage medium of a distributed cache platform when the written file is determined to be successfully written into the cloud storage device in the NAS cluster;
and the reporting unit is used for periodically reporting the metadata which is stored persistently to the NAS MASTER equipment, and deleting the metadata which is reported successfully from the persistent storage medium after the metadata is determined to be reported successfully.
6. The apparatus of claim 5, further comprising:
and the caching unit is used for storing the acquired target metadata to the distributed caching platform and setting aging time for the target metadata.
7. The apparatus of claim 5, wherein the writing unit is further configured to set an aging duration for the metadata written to the file in the distributed cache memory;
the device further comprises:
the first deleting unit is used for periodically deleting the metadata reaching the aging duration in the memory of the distributed cache platform.
8. The apparatus of claim 5, further comprising:
a second deleting unit, configured to delete, when receiving a deletion request sent by the NAS client, the metadata indicated by the deletion request from the memory of the distributed cache platform and the NAS MASTER.
9. An electronic device comprising a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor, the processor being caused by the machine-executable instructions to perform the method of any one of claims 1 to 4.
10. A machine-readable storage medium having stored thereon machine-executable instructions which, when invoked and executed by a processor, cause the processor to perform the method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911133793.3ACN111399760B (en) | 2019-11-19 | 2019-11-19 | NAS cluster metadata processing method and device, NAS gateway and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911133793.3ACN111399760B (en) | 2019-11-19 | 2019-11-19 | NAS cluster metadata processing method and device, NAS gateway and medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111399760A CN111399760A (en) | 2020-07-10 |
| CN111399760Btrue CN111399760B (en) | 2023-01-24 |
Family
ID=71433911
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911133793.3AActiveCN111399760B (en) | 2019-11-19 | 2019-11-19 | NAS cluster metadata processing method and device, NAS gateway and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111399760B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114625762A (en)* | 2020-11-27 | 2022-06-14 | 华为技术有限公司 | Metadata acquisition method, network device and system |
| CN113076298B (en)* | 2021-04-15 | 2024-10-01 | 上海卓钢链科技有限公司 | Distributed small file storage system |
| CN113296714B (en)* | 2021-06-17 | 2022-03-04 | 杭州海康威视系统技术有限公司 | Data storage system based on NAS protocol |
| CN114443595A (en)* | 2021-12-24 | 2022-05-06 | 天翼云科技有限公司 | A method and device for processing documents |
| CN114489856B (en)* | 2022-01-28 | 2025-01-10 | 中国农业银行股份有限公司 | Method, device and equipment for obtaining application configuration file |
| CN114676103A (en)* | 2022-03-04 | 2022-06-28 | 杭州宏杉科技股份有限公司 | Data storage synchronization method and device |
| CN116450694A (en)* | 2023-04-20 | 2023-07-18 | 中国建设银行股份有限公司 | Data preview method, device, equipment and storage medium |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103098443A (en)* | 2010-09-13 | 2013-05-08 | 马维尔国际贸易有限公司 | Distributed metadata cache |
| CN104267912A (en)* | 2014-09-19 | 2015-01-07 | 北京联创信安科技有限公司 | NAS (Network Attached Storage) accelerating method and system |
| CN108733507A (en)* | 2017-04-17 | 2018-11-02 | 伊姆西Ip控股有限责任公司 | The method and apparatus of file backup and recovery |
| CN109240617A (en)* | 2018-09-03 | 2019-01-18 | 郑州云海信息技术有限公司 | Distributed memory system write request processing method, device, equipment and storage medium |
| CN109254958A (en)* | 2018-10-18 | 2019-01-22 | 上海云轴信息科技有限公司 | Distributed data reading/writing method, equipment and system |
| CN109684282A (en)* | 2018-11-23 | 2019-04-26 | 华为技术有限公司 | A kind of method and device constructing metadata cache |
| CN109960662A (en)* | 2017-12-25 | 2019-07-02 | 华为技术有限公司 | A kind of memory recycling method and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10664450B2 (en)* | 2017-03-06 | 2020-05-26 | Vmware, Inc. | Decoupling the commit and replay of metadata updates in a clustered file system |
- 2019
- 2019-11-19CNCN201911133793.3Apatent/CN111399760B/enactiveActive
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103098443A (en)* | 2010-09-13 | 2013-05-08 | 马维尔国际贸易有限公司 | Distributed metadata cache |
| CN104267912A (en)* | 2014-09-19 | 2015-01-07 | 北京联创信安科技有限公司 | NAS (Network Attached Storage) accelerating method and system |
| CN108733507A (en)* | 2017-04-17 | 2018-11-02 | 伊姆西Ip控股有限责任公司 | The method and apparatus of file backup and recovery |
| CN109960662A (en)* | 2017-12-25 | 2019-07-02 | 华为技术有限公司 | A kind of memory recycling method and device |
| CN109240617A (en)* | 2018-09-03 | 2019-01-18 | 郑州云海信息技术有限公司 | Distributed memory system write request processing method, device, equipment and storage medium |
| CN109254958A (en)* | 2018-10-18 | 2019-01-22 | 上海云轴信息科技有限公司 | Distributed data reading/writing method, equipment and system |
| CN109684282A (en)* | 2018-11-23 | 2019-04-26 | 华为技术有限公司 | A kind of method and device constructing metadata cache |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111399760A (en) | 2020-07-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111399760B (en) | NAS cluster metadata processing method and device, NAS gateway and medium | |
| CN106407040B (en) | Method and system for remote data replication | |
| US6850955B2 (en) | Storage system and control method | |
| US8738857B1 (en) | System and method for improving cache performance | |
| US9507720B2 (en) | Block storage-based data processing methods, apparatus, and systems | |
| CN102782670B (en) | Memory cache data center | |
| CN104111804B (en) | A kind of distributed file system | |
| US8332600B2 (en) | Storage system and method for operating storage system | |
| CN107329704B (en) | Cache mirroring method and controller | |
| US20140236906A1 (en) | Elimination of duplicate objects in storage clusters | |
| JP5967673B2 (en) | Data maintenance methods | |
| US11314454B2 (en) | Method and apparatus for managing storage device in storage system | |
| US11321021B2 (en) | Method and apparatus of managing mapping relationship between storage identifier and start address of queue of storage device corresponding to the storage identifier | |
| CN107329708A (en) | A kind of distributed memory system realizes data cached method and system | |
| CN109254958A (en) | Distributed data reading/writing method, equipment and system | |
| US9268696B1 (en) | System and method for improving cache performance | |
| CN106445409A (en) | Distributed block storage data writing method and device | |
| EP4425320A1 (en) | Data storage system, smart network card, and computing node | |
| CN112130747A (en) | Distributed object storage system and data reading and writing method | |
| CN111796772B (en) | Cache management method, cache node and distributed storage system | |
| CN117255101A (en) | Data processing method, device, equipment and medium of distributed storage system | |
| CN105915595B (en) | Method for cluster storage system to access data and cluster storage system | |
| CN105187565A (en) | Method for utilizing network storage data | |
| US20160314051A1 (en) | Management and utilization of fault domains in distributed cache systems | |
| CN103685359B (en) | Data processing method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |