Disclosure of Invention
The technical problem to be solved by the embodiment of the invention is to overcome the defects caused by that metadata of all users are stored in the same metadata server cluster in the prior art, and provide a distributed file system, an implementation method, a management system, equipment and a medium thereof.
The embodiment of the invention solves the technical problems through the following technical scheme:
a method for implementing a distributed file system, the method comprising:
receiving a creation request for creating a file system;
creating metadata node clusters according to the creation request, wherein different file systems correspond to different metadata node clusters, and the metadata node clusters comprise a plurality of groups of members, and different group members correspond to different metadata nodes;
after the metadata node cluster is successfully created, generating a system number representing a successfully created file system;
and storing the corresponding relation between the system number and the metadata node cluster in a first database.
Preferably, the creation request includes a first creation request, and the step of creating the metadata node cluster according to the creation request includes:
and creating a first metadata node cluster according to the first creation request, wherein the metadata nodes related to the first metadata node cluster correspond to only one file system.
Preferably, the creation request includes a second creation request, and the step of creating the metadata node cluster according to the creation request includes:
And creating a second metadata node cluster according to the second creation request, wherein metadata nodes related to the second metadata node cluster correspond to at least two file systems.
Preferably, before the step of receiving a creation request to create a file system, the implementation method further includes:
receiving node information of a metadata node, wherein the node information comprises group members, available system resources, memory configuration and disk types corresponding to the metadata node;
the step of creating a metadata node cluster according to the creation request includes:
selecting a plurality of metadata nodes according to the creation request and the received node information;
sending a request for creating a new group member to each of the plurality of metadata nodes;
judging whether a new response of successful creation of the group member sent by the metadata nodes is received or not;
if yes, determining that the metadata node cluster is successfully established, wherein the metadata node cluster comprises new members successfully established;
if not, determining that the metadata node cluster is failed to be created.
Preferably, after the step of generating a system number characterizing a successfully created file system, the implementation method further includes:
Storing metadata of a file system in a metadata node cluster corresponding to a system number of the file system;
storing the corresponding relation between the metadata node cluster and the metadata of the file system in a second database;
the first database and the second database are embedded databases.
Preferably, after the step of storing metadata of a file system in a metadata node cluster corresponding to a system number of the file system, the implementation method further includes:
receiving a management request for managing metadata of a target file system, wherein the management request comprises at least one of creating, reading, updating and deleting the metadata;
determining a target metadata node cluster corresponding to the system number of the target file system;
and managing metadata in the target metadata node cluster according to the management request.
Preferably, the implementation method further comprises:
receiving a capacity expansion request for expanding the capacity of a file system to be expanded, wherein the capacity expansion request comprises capacity expansion metadata nodes for expanding capacity;
determining a metadata node cluster to be expanded corresponding to the system number of the file system to be expanded, and determining a group member corresponding to the metadata node with the highest load related to the metadata node cluster to be expanded as a target group member;
Adding a capacity expansion group member created on the capacity expansion metadata node in the metadata node cluster to be expanded;
and migrating the metadata of the file system to be expanded in the target group member to the expansion group member.
Preferably, the implementation method further comprises:
copying metadata in the second database to a second local disk at regular time to generate a second database snapshot;
reconstructing a second database according to the second database snapshot.
An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements a method of implementing any of the above-mentioned distributed file systems when executing the computer program.
A computer readable storage medium having stored thereon a computer program, characterized in that the computer program when executed by a processor implements the steps of any of the above-described methods of implementing a distributed file system.
A distributed file system, the distributed file system comprising:
the request receiving module is used for receiving a creation request for creating a file system;
the creation module is used for creating metadata node clusters according to the creation request, different file systems correspond to different metadata node clusters, wherein the metadata node clusters comprise a plurality of groups of members, and different group members correspond to different metadata nodes;
The system number generation module is used for generating a system number representing a successfully created file system after the metadata node cluster is successfully created;
and the first storage module is used for storing the corresponding relation between the system number and the metadata node cluster in a first database.
Preferably, the creation request includes a first creation request, and the creation module is specifically configured to:
and creating a first metadata node cluster according to the first creation request, wherein the metadata nodes related to the first metadata node cluster correspond to only one file system.
Preferably, the creation request includes a second creation request, and the creation module is specifically configured to:
and creating a second metadata node cluster according to the second creation request, wherein metadata nodes related to the second metadata node cluster correspond to at least two file systems.
Preferably, the distributed file system further comprises:
the node management module is used for receiving node information of the metadata node, wherein the node information comprises group members, available system resources, memory configuration and disk types corresponding to the metadata node;
the creation module comprises:
A node selection unit for selecting a plurality of metadata nodes according to the creation request and the received node information;
a creation request unit configured to send a request for creating a new group member to each of the plurality of metadata nodes;
a judging unit, configured to judge whether a response of successful creation of a new group member sent by the plurality of metadata nodes is received;
if yes, calling a first determining unit; if not, calling a second determining unit;
the first determining unit is used for determining that the metadata node cluster is successfully created, and the metadata node cluster comprises new group members which are successfully created;
the second determining unit is used for determining that the metadata node cluster creation fails.
Preferably, the distributed file system further comprises:
the metadata storage module is used for storing metadata of a file system in a metadata node cluster corresponding to a system number of the file system;
the second storage module is used for storing the corresponding relation between the metadata node cluster and the metadata of the file system in a second database;
the first database and the second database are embedded databases.
Preferably, the request receiving module is further configured to receive a management request for managing metadata of the target file system, where the management request includes at least one of creating, reading, updating, and deleting metadata;
The distributed file system further includes:
a determining module, configured to determine a target metadata node cluster corresponding to a system number of the target file system;
and the metadata management module is used for managing metadata in the target metadata node cluster according to the management request.
Preferably, the request receiving module is further configured to receive a capacity expansion request for expanding a capacity of a file system to be expanded, where the capacity expansion request includes a capacity expansion metadata node for expanding capacity;
the distributed file system further includes:
the determining module is used for determining a metadata node cluster to be expanded corresponding to the system number of the file system to be expanded, and determining a group member corresponding to the metadata node with the highest load related to the metadata node cluster to be expanded as a target group member;
the node adding module is used for adding the capacity expansion group member created on the capacity expansion metadata node in the metadata node cluster to be expanded;
and the metadata migration module is used for migrating the metadata of the file system to be expanded in the target group member to the expansion group member.
Preferably, the distributed file system further comprises:
The database snapshot module is used for copying the metadata in the second database to a second local disk at regular time to generate a second database snapshot;
and the database reconstruction module is used for reconstructing a second database according to the second database snapshot.
A management system of a distributed file system, which is characterized in that the management system comprises a client and any one of the distributed file systems;
the client is used for sending a request to a request receiving module of the distributed file system.
The embodiment of the invention has the positive progress effects that: in the embodiment of the invention, the metadata of all users are not stored in the same metadata node cluster, but the file systems of all users are distributed and corresponding to all metadata node clusters, so that the metadata of the users are physically isolated while the logical isolation is realized, and when the metadata node clusters need to be expanded, the metadata of the corresponding users on the metadata nodes only need to be migrated, thereby improving the expandability of the system.
Detailed Description
The invention is further illustrated by means of the following examples, which are not intended to limit the scope of the invention.
Example 1
The present embodiment provides a method for implementing a distributed file system, fig. 1 shows a flowchart for creating a file system in the present embodiment, and referring to fig. 1, the step of creating a file system includes:
S11, receiving node information of the metadata nodes.
The distributed file system is implemented based on metadata nodes, wherein each metadata node registers its node information when enabled, and the node information may include, but is not limited to, group members corresponding to the metadata node, available system resources, memory configuration, disk types, and the like. Step S11 is to receive node information registered by each metadata node.
S12, receiving a creation request for creating a file system.
The creation requirements of the file systems are different, wherein the creation requirements can include, but are not limited to, the number of copies, service types, available system resources, memory configuration, disk types and the like, and the creation requirements are included in the creation request of the file systems initiated by each user.
The copy number is used for determining the number of group members in the metadata node cluster corresponding to the file system to be created, and the group members are used for processing data in a certain range on the metadata nodes, so that the copy number is also the number of metadata nodes related to the metadata node cluster corresponding to the file system to be created.
The traffic types may include, but are not limited to, primary traffic and secondary traffic. The IO (Input/Output) request of the primary service is high, and it is usually required to monopolize the metadata node, i.e. the metadata node corresponds to only one group member for the primary service to use. The IO requests of the secondary services are relatively low, and the metadata node can be shared with other file systems, that is, the metadata node can comprise at least two group members, and different group members can be used by different secondary services so as to realize the sharing of the metadata node by different file systems.
S13, creating a metadata node cluster according to the creation request.
In this step, the creation request is parsed and converted into creation parameters that may include, but are not limited to, number of copies, type of service, available system resources, memory configuration, disk type, and the like.
Referring to fig. 2, step S13 may specifically include:
s131, selecting a plurality of metadata nodes according to the creation request and the received node information;
s132, respectively sending a request for creating new group members to a plurality of metadata nodes;
s133, judging whether a new response of successful creation of the group member sent by a plurality of metadata nodes is received or not;
if yes, go to step S134; if not, go to step S135;
s134, determining that the metadata node cluster is successfully created;
s135, determining that the metadata node cluster creation fails.
Specifically, in the present embodiment, a metadata node is selected based on the creation request and the respective node information in step S131. For example, the creation request requires creation of a metadata node cluster consisting of 3 group members for a primary service, selection of metadata node N from a metadata node resource pool1 、N2 、N3 Wherein the metadata node N1 、N2 、N3 Currently unoccupied, i.e., not corresponding to any group members or file systems, at metadata node N1 、N2 、N3 New group members are created respectively to construct the metadata node clusters, and metadata nodes related to the metadata node clusters correspond to only one file system. The creation request requires creating a cluster of metadata nodes consisting of 3 group members for the secondary service, selecting a metadata node N from a metadata node resource pool4 、N5 、N6 Wherein the metadata node N4 、N5 、N6 At the metadata node N, at least one of which may be currently occupied, i.e. may correspond to a group member or a file system4 、N5 、N6 And creating new group members respectively to construct the metadata node clusters, wherein metadata nodes related to the metadata node clusters correspond to at least two file systems. In this embodiment, the read node information may be stored in a cache to increase the reading speed.
In step S132, a request to create a new group member is transmitted to the selected plurality of metadata nodes, which respond to the request after receiving the request, respectively. If step S133 determines that the plurality of metadata nodes each send a response to the request that the creation of the new group member is successful, then in step S134, it is determined that the creation of the metadata node cluster is successful, and the creation of the successful metadata node cluster includes the creation of the new group member that is successful; if step S133 determines that the plurality of metadata nodes do not send a response to the request that the creation of the new group member is successful, step S135 determines that the creation of the metadata node cluster fails, and reports the failure, and in addition, the data that was not successfully created in the process may be cleaned.
In this embodiment, the consistency inside the cluster and the reasonable configuration of the metadata node resources can be realized based on the multi-raft protocol, specifically, the metadata node cluster including a plurality of group members forms a raft group (raft group), and the plurality of raft groups form a multi-raft, and the group members in the raft group do not directly communicate, but communicate via the metadata nodes corresponding to the group members in the raft group, so as to realize the consistency inside the raft group.
And S14, after the metadata node cluster is successfully created, generating a system number representing a successfully created file system.
And S15, storing the corresponding relation between the system number and the metadata node cluster in a first database.
Through the steps, the file system is successfully created, and according to the corresponding relation between the system number and the metadata node cluster, the metadata node corresponding to the file system can be determined, and then the management of metadata of the file system can be realized.
Specifically, in this embodiment, the creation request may be divided into a first creation request (primary service) and a second creation request (secondary service) according to different service types (primary service, secondary service) in the creation request, and accordingly, the metadata node cluster may include a first metadata node cluster and a second metadata node cluster, where all metadata nodes related to the first metadata node cluster correspond to only one file system, and all metadata nodes related to the second metadata node cluster correspond to at least two file systems.
Thus, referring to FIG. 3, when the creation request is a first creation request, the step of creating a file system may include:
S12A, receiving a first creation request for creating a file system;
S13A, creating a first metadata node cluster according to a first creation request;
S14A, after the first metadata node cluster is successfully created, generating a system number representing a successfully created file system;
and S15A, storing the corresponding relation between the system number and the first metadata node cluster in a first database.
Referring to fig. 4, when the creation request is a second creation request, the step of creating the file system may include:
S12B, receiving a second creation request for creating the file system.
S13B, creating a second metadata node cluster according to the second creation request.
And S14B, after the second metadata node cluster is successfully created, generating a system number representing the successfully created file system.
And S15B, storing the corresponding relation between the system number and the second metadata node cluster in a first database.
After the file system is successfully created, the implementation method of the embodiment further comprises the following steps:
storing metadata of the file system in metadata node clusters corresponding to system numbers of the file system;
And storing the corresponding relation between the metadata node cluster and the metadata of the file system in a second database.
Through the steps, firstly, the corresponding metadata node cluster is determined according to the system number of the file system, then the metadata of the file system is written into the metadata node cluster, metadata storage is realized, the corresponding relation between the metadata node cluster and the metadata of the file system is stored in the second database, and management of the metadata in the metadata node cluster can be realized based on the corresponding relation between the metadata node cluster and the metadata.
In this embodiment, the first database and the second database may be embedded databases of the embedded process itself, for example, the embedded databases may include a memory key database based on B-Tree (multiple search Tree), a memory key database based on level lbd, a SQLite (relational database management system that complies with ACID (atom, consistency, isolation, durability)), and unlike RDBMS (Relational Database Management System ) databases require additional deployment of response systems, which reduces the operation cost of the file system.
Fig. 5 shows a flowchart of managing metadata in the present embodiment, and referring to fig. 5, the steps of managing metadata include:
s21, receiving a management request for managing metadata of a target file system;
s22, determining a target metadata node cluster corresponding to the system number of the target file system;
s23, managing metadata in the target metadata node cluster according to the management request.
In this embodiment, the management request may include, but is not limited to, creating, reading, updating, and deleting metadata, specifically, determining a corresponding target metadata node cluster by a system number of the target file system, and then managing metadata in the target metadata node cluster according to the management request. For the management request of inquiring and reading the metadata, the corresponding metadata can be directly returned, and for the management request of modifying the metadata, the metadata in the metadata node cluster is modified according to the multi-raft protocol, and the modification is updated to the second database.
Fig. 6 shows a flowchart of expanding a file system in this embodiment, and referring to fig. 6, the step of expanding the file system includes:
s31, receiving a capacity expansion request for expanding the capacity of the file system to be expanded;
S32, determining a metadata node cluster to be expanded corresponding to the system number of the file system to be expanded, and determining a group member corresponding to the metadata node with the highest load related to the metadata node cluster to be expanded as a target group member;
s33, adding the capacity expansion group member created on the capacity expansion metadata node in the metadata node cluster to be expanded;
and S34, migrating the metadata of the file system to be expanded in the target group member to the expansion group member.
In this embodiment, the capacity expansion request may include a system number of the file system to be expanded and capacity expansion metadata nodes for expansion (different numbers of capacity expansion metadata nodes may be set according to actual needs). First, a metadata node cluster to be expanded is determined according to a system number, for example, the metadata node cluster to be expanded comprises a group member A located on a metadata node AM Group Member B located on metadata node BM Group member C located on metadata node CM Wherein the metadata node A is the highest loaded, the group member A is determinedM Is a target group member. There are also known capacity expansion metadata nodes D, E, F for expansion, on which expansion metadata nodes D, E, F new group members D are created, respectivelyM 、EM 、FM Group member DM 、EM 、FM Added to the metadata node cluster to be expanded. Finally, target group member AM Metadata of the file system to be expanded are respectively migrated to the group member DM 、EM 、FM Finally form and include group member AM 、BM 、CM 、DM 、EM F (F)M The metadata node cluster of the system realizes the expansion of the metadata node cluster.
The implementation method of the embodiment may further include:
and copying the metadata in the second database to a second local disk at regular time to generate a second database snapshot.
This step aims to backup the database from data loss due to disaster, failure.
The snapshot may be used for disaster recovery or node restart, and specifically, the second database may be reconstructed according to the second database snapshot.
In this embodiment, heartbeats may also be sent to the metadata nodes at regular intervals to update the node information.
In this embodiment, one file system corresponds to one user, and different file systems correspond to different metadata node clusters, so that file systems of all users are dispersed into each metadata node cluster, metadata of users are physically isolated while logical isolation is implemented, and when the metadata node clusters need to be expanded, metadata of corresponding users on metadata nodes only need to be migrated, thereby improving expandability of the system. In addition, the first and second metadata node clusters are arranged, so that competition of different users for service resources is avoided, and service quality is improved.
Example 2
The present embodiment provides an electronic device, which may be expressed in the form of a computing device (for example, may be a server device), including a memory, a processor, and a computer program stored on the memory and executable on the processor, where the processor may implement the method for implementing the distributed file system provided in embodiment 1 when executing the computer program.
Fig. 7 shows a schematic diagram of the hardware structure of the present embodiment, and as shown in fig. 7, the electronic device 9 specifically includes:
at least one processor 91, at least one memory 92, and a bus 93 for connecting the different system components (including the processor 91 and the memory 92), wherein:
the bus 93 includes a data bus, an address bus, and a control bus.
The memory 92 includes volatile memory such as Random Access Memory (RAM) 921 and/or cache memory 922, and may further include Read Only Memory (ROM) 923.
Memory 92 also includes a program/utility 925 having a set (at least one) of program modules 924, such program modules 924 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The processor 91 executes various functional applications and data processing by running a computer program stored in the memory 92, such as the implementation method of the distributed file system provided in embodiment 1 of the present invention.
The electronic device 9 may further communicate with one or more external devices 94 (e.g., keyboard, pointing device, etc.). Such communication may occur through an input/output (I/O) interface 95. Also, the electronic device 9 may communicate with one or more networks such as a Local Area Network (LAN), a Wide Area Network (WAN) and/or a public network, such as the Internet, through a network adapter 96. The network adapter 96 communicates with other modules of the electronic device 9 via the bus 93. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in connection with the electronic device 9, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, data backup storage systems, and the like.
It should be noted that although several units/modules or sub-units/modules of an electronic device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module according to embodiments of the present application. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.
Example 3
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the implementation method of the distributed file system provided in embodiment 1.
More specifically, among others, readable storage media may be employed including, but not limited to: portable disk, hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.
In a possible embodiment, the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps of the implementation method for implementing the distributed file system in embodiment 1, when said program product is run on the terminal device.
Wherein the program code for carrying out the invention may be written in any combination of one or more programming languages, which program code may execute entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device and partly on the remote device or entirely on the remote device.
Example 4
The present embodiment provides a distributed file system, fig. 8 shows a schematic block diagram of the present embodiment, and referring to fig. 8, the distributed file system 100 of the present embodiment includes:
the node management module 1 is configured to receive node information of a metadata node.
The distributed file system 100 is implemented based on metadata nodes, where each metadata node registers its node information when enabled, and the node information may include, but is not limited to, group members corresponding to the metadata node, available system resources, memory configuration, disk type, and the like. The node management module 1 is configured to receive node information registered by each metadata node.
A request receiving module 2, configured to receive a creation request for creating a file system.
The creation requirements of the file systems are different, wherein the creation requirements can include, but are not limited to, the number of copies, service types, available system resources, memory configuration, disk types and the like, and the creation requirements are included in the creation request of the file systems initiated by each user.
The copy number is used for determining the number of group members in the metadata node cluster corresponding to the file system to be created, and the group members are used for processing data in a certain range on the metadata nodes, so that the copy number is also the number of metadata nodes related to the metadata node cluster corresponding to the file system to be created.
The traffic types may include, but are not limited to, primary traffic and secondary traffic. The IO (Input/Output) request of the primary service is high, and it is usually required to monopolize the metadata node, i.e. the metadata node corresponds to only one group member for the primary service to use. The IO requests of the secondary services are relatively low, and the metadata node can be shared with other file systems, that is, the metadata node can comprise at least two group members, and different group members can be used by different secondary services so as to realize the sharing of the metadata node by different file systems.
A creation module 3, configured to create a metadata node cluster according to the creation request.
In this embodiment, the creation module 3 parses the creation request and converts it into creation parameters, which may include, but are not limited to, the number of copies, service types, available system resources, memory configuration, disk types, and the like. Referring to fig. 8, the creation module 3 may include:
a node selection unit 31 for selecting a plurality of metadata nodes according to the creation request and the received node information;
a creation request unit 32 for respectively transmitting a request for creating a new group member to the plurality of metadata nodes;
A judging unit 33, configured to judge whether a response of successful creation of a new group member sent by the plurality of metadata nodes is received;
if yes, the first determination unit 34 is called; if not, the second determination unit 35 is invoked;
a first determining unit 34, configured to determine that the metadata node cluster creation is successful;
a second determining unit 35, configured to determine that the metadata node cluster creation fails.
Specifically, in the present embodiment, the node selection unit 31 selects the metadata node based on the creation request and the respective node information. For example, the creation request requires creation of a metadata node cluster consisting of 3 group members for a primary service, selection of metadata node N from a metadata node resource pool1 、N2 、N3 Wherein the metadata node N1 、N2 、N3 Currently unoccupied, i.e., not corresponding to any group members or file systems, at metadata node N1 、N2 、N3 New group members are created respectively to construct the metadata node clusters, and metadata nodes related to the metadata node clusters correspond to only one file system. The creation request requires creating a cluster of metadata nodes consisting of 3 group members for the secondary service, selecting a metadata node N from a metadata node resource pool4 、N5 、N6 Wherein the metadata node N4 、N5 、N6 At least one of which can currently be occupied, i.e. canAt metadata node N with corresponding group members or file systems4 、N5 、N6 And creating new group members respectively to construct the metadata node clusters, wherein metadata nodes related to the metadata node clusters correspond to at least two file systems. In this embodiment, the read node information may be stored in a cache to increase the reading speed.
The creation request unit 32 communicates with a selected plurality of metadata nodes, specifically, transmits a request to create a new group member to the selected plurality of metadata nodes, respectively, which respond to the request after receiving the request. If the judging unit 33 judges that the plurality of metadata nodes send a response of successful creation of a new group member to the request, the first determining unit 34 is called to determine that the metadata node cluster is successfully created, and the metadata node cluster that is successfully created includes the new group member that is successfully created; if the judging unit 33 judges that the plurality of metadata nodes do not send a response of successful creation of a new group member to the request, the second determining unit 35 is called to determine that the creation of the metadata node cluster fails, and report the failure, and in addition, the data which is not successfully created in the process can be cleaned.
In this embodiment, the consistency inside the cluster and the reasonable configuration of the metadata node resources can be realized based on the multi-raft protocol, specifically, the metadata node cluster including a plurality of group members forms a raft group (raft group), and the plurality of raft groups form a multi-raft, and the group members in the raft group do not directly communicate, but communicate via the metadata nodes corresponding to the group members in the raft group, so as to realize the consistency inside the raft group.
And the system number generating module 4 is used for generating a system number representing a successfully created file system after the metadata node cluster is successfully created.
A first storage module 5, configured to store a correspondence between a system number and a metadata node cluster in a first database.
Therefore, the file system is successfully created, and according to the corresponding relation between the system number and the metadata node cluster, the metadata node corresponding to the file system can be determined, and then the management of metadata of the file system can be realized.
Specifically, in this embodiment, the creation request may be divided into a first creation request (primary service) and a second creation request (secondary service) according to different service types (primary service, secondary service) in the creation request, and accordingly, the metadata node cluster may include a first metadata node cluster and a second metadata node cluster, where all metadata nodes related to the first metadata node cluster correspond to only one file system, and all metadata nodes related to the second metadata node cluster correspond to at least two file systems.
Accordingly, the request receiving module 2 may receive a first creation request for creating a file system, the creating module 3 may create a first metadata node cluster according to the first creation request, the system number generating module 4 may generate a system number indicating a successfully created file system after the first metadata node cluster is successfully created, and the first storage module 5 may store a correspondence between the system number and the first metadata node cluster in the first database.
The request receiving module 2 may further receive a second creation request for creating a file system, the creation module 3 may further create a second metadata node cluster according to the second creation request, the system number generating module 4 may further generate a system number indicating a successfully created file system after the second metadata node cluster is created successfully, and the first storage module 5 may further store a correspondence between the system number and the second metadata node cluster in the first database.
The distributed file system 100 of the present embodiment may further include a metadata storage module and a second storage module. Specifically, in this embodiment, the metadata storage module is configured to determine, according to a system number of a file system, a corresponding metadata node set of the file system, and then write metadata of the file system into the metadata node cluster, so as to implement metadata storage. The second storage module is configured to store a corresponding relationship between the metadata node cluster and metadata of the file system in a second database, and can implement management of metadata in the metadata node cluster based on the corresponding relationship between the metadata node cluster and metadata.
In this embodiment, the first database and the second database may be embedded databases of the embedded process itself, for example, the embedded databases may include a memory key database based on B-Tree (multiple search Tree), a memory key database based on level lbd, a SQLite (relational database management system that complies with ACID (atom, consistency, isolation, durability)), and unlike RDBMS (Relational Database Management System ) databases require additional deployment of response systems, which reduces the operation cost of the file system.
Referring to fig. 8, the distributed file system 100 of the present embodiment may further include a determination module 6, a metadata management module 7, a node addition module 8, and a metadata migration module 10.
Specifically, in this embodiment, the request receiving module 2 is further configured to receive a management request for managing metadata of the target file system, the determining module 6 is configured to determine a target metadata node cluster corresponding to a system number of the target file system, and the metadata management module 7 is configured to manage metadata in the target metadata node cluster according to the management request.
The management request may include, but is not limited to, creating, reading, updating, and deleting metadata, specifically, determining a corresponding target metadata node cluster through a system number of the target file system, and then managing metadata in the target metadata node cluster according to the management request. For the management request of inquiring and reading the metadata, the corresponding metadata can be directly returned, and for the management request of modifying the metadata, the metadata in the metadata node cluster is modified according to the multi-raft protocol, and the modification is updated to the second database.
In this embodiment, the request receiving module 2 is further configured to receive a capacity expansion request for expanding a to-be-expanded file system, the determining module 6 is further configured to determine a to-be-expanded metadata node cluster corresponding to a system number of the to-be-expanded file system, and is further configured to determine a group member corresponding to a metadata node with a highest load related to the to-be-expanded metadata node cluster as a target group member, the node adding module 8 is configured to add, in the to-be-expanded metadata node cluster, a capacity expansion group member created on the to-be-expanded metadata node, and the metadata migration module 10 is configured to migrate metadata of the to-be-expanded file system in the target group member to the expansion group member.
The capacity expansion request includes a system number that may include a file system to be expanded and a capacity expansion metadata node for expansion (different numbers of capacity expansion metadata nodes may be set according to actual needs). First, a metadata node cluster to be expanded is determined according to a system number, for example, the metadata node cluster to be expanded comprises a group member A located on a metadata node AM Group Member B located on metadata node BM Group member C located on metadata node CM Wherein the metadata node A is the highest loaded, the group member A is determinedM Is a target group member. There are also known capacity expansion metadata nodes D, E, F for expansion, on which expansion metadata nodes D, E, F new group members D are created, respectivelyM 、EM 、FM Group member DM 、EM 、FM Added to the metadata node cluster to be expanded. Finally, target group member AM Metadata of the file system to be expanded are respectively migrated to the group member DM 、EM 、FM Finally form and include group member AM 、BM 、CM 、DM 、EM F (F)M The metadata node cluster of the system realizes the expansion of the metadata node cluster.
Referring to fig. 8, the distributed file system 100 of the present embodiment may further include a database snapshot module 11 and a database reconstruction module 12.
Specifically, the database snapshot module 11 is configured to copy metadata in the second database to the second local disk at regular time, and generate a second database snapshot.
Database snapshot module 11 is intended to back up the database from data loss due to the occurrence of disasters, failures.
The snapshot may be used for disaster recovery or node restart, and in particular, the database reconstruction module 12 may reconstruct the second database according to the second database snapshot.
In this embodiment, the node management module 1 may also send heartbeats to the metadata node at regular time to update the node information.
In this embodiment, one file system corresponds to one user, and different file systems correspond to different metadata node clusters, so that file systems of all users are dispersed into each metadata node cluster, metadata of users are physically isolated while logical isolation is implemented, and when the metadata node clusters need to be expanded, metadata of corresponding users on metadata nodes only need to be migrated, thereby improving expandability of the system. In addition, the first and second metadata node clusters are arranged, so that competition of different users for service resources is avoided, and service quality is improved.
Example 5
The present embodiment provides a management system of a distributed file system, fig. 9 shows a schematic block diagram of the present embodiment, and referring to fig. 9, the management system of the present embodiment includes a client 200 and the distributed file system 100 provided in embodiment 4.
In this embodiment, the client 200 is configured to send a request to the request receiving module 2 of the distributed file system 100, and also configured to receive data sent thereto by the distributed file system 100. In particular, the client 200 may include a fuse client and a Java SDK client.
The embodiment provides a management system of a distributed file system on the basis of embodiment 4, so as to realize management of the distributed file system of a user.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the principles and spirit of the invention, but such changes and modifications fall within the scope of the invention.