技术领域technical field
本发明涉及元数据存储技术领域，更具体地说，涉及一种元数据服务器的服务控制方法。The present invention relates to the technical field of metadata storage, and more specifically, relates to a service control method of a metadata server.
背景技术Background technique
随着物联网、云计算和移动电子商务的发展，大数据业务占整个数据库市场的比例不断提高；大数据时代的来临使数据分析和处理能力成为数据中心日益倚重的技术手段；信息规模的扩大和数据结构的多样化，使海量数据存储成为大数据分析研究的热点。With the development of the Internet of Things, cloud computing, and mobile e-commerce, the proportion of big data services in the entire database market continues to increase; the advent of the era of big data has made data analysis and processing capabilities increasingly important technical means for data centers; the expansion of information scale and The diversification of data structures makes massive data storage a hot spot in big data analysis research.
为了实现数据的快速读写，通常需要获取到数据的元数据，包括数据对应的文件名称及存储位置等信息，进而在实现数据读写时先通过元数据确定出数据的相关信息，以基于该相关信息快速定位到对应的数据。现有技术中为了实现海量数据对应的大量元数据信息，通常需要将元数据存储至元数据集群中的每个元数据服务器中，但是随着集群规模的扩大，元数据服务器发生故障的概率增加，而如果任一元数据服务器发生故障，则其存储的元数据将丢失且无法恢复，进而导致无法基于这部分元数据实现对应数据的读写，即无法实现该发生故障的元数据服务器应提供的元数据服务。In order to realize fast reading and writing of data, it is usually necessary to obtain the metadata of the data, including information such as the file name and storage location corresponding to the data. Relevant information quickly locates the corresponding data. In the prior art, in order to realize a large amount of metadata information corresponding to massive data, it is usually necessary to store metadata into each metadata server in the metadata cluster, but as the cluster scale expands, the probability of failure of the metadata server increases , and if any metadata server fails, the metadata stored in it will be lost and cannot be recovered, which will result in the inability to read and write the corresponding data based on this part of metadata, that is, it will be impossible to realize the metadata that the failed metadata server should provide. metadata service.
综上所述，现有技术中存在可能出现因元数据服务器故障导致无法实现对应元数据服务情况的问题。To sum up, in the prior art, there may be a problem that the corresponding metadata service cannot be realized due to the failure of the metadata server.
发明内容Contents of the invention
本发明的目的是提供一种元数据服务器的服务控制方法，以解决现有技术中存在的可能出现因元数据服务器故障导致无法实现对应元数据服务情况的问题。The purpose of the present invention is to provide a metadata server service control method to solve the problem in the prior art that the corresponding metadata service cannot be realized due to the failure of the metadata server.
为了实现上述目的，本发明提供如下技术方案：In order to achieve the above object, the present invention provides the following technical solutions:
一种元数据服务器的服务控制方法，包括：A service control method for a metadata server, comprising:
设置与元数据服务器集群中每个元数据服务器对应的影子节点；Set shadow nodes corresponding to each metadata server in the metadata server cluster;
控制每个所述影子节点中保存的数据与对应元数据服务器中保存的元数据信息一致；Controlling that the data stored in each shadow node is consistent with the metadata information stored in the corresponding metadata server;
如果任一所述元数据服务器发生故障成为故障服务器，则控制该故障服务器对应的影子节点基于存储的数据实现对应元数据服务。If any of the metadata servers fails and becomes a failed server, control the shadow node corresponding to the failed server to implement the corresponding metadata service based on the stored data.
优选的，控制每个所述影子节点中保存的数据与对应元数据服务器中保存的元数据信息一致，包括：Preferably, the data stored in each shadow node is controlled to be consistent with the metadata information stored in the corresponding metadata server, including:
控制每个影子节点获取对应元数据服务器的镜像文件，所述镜像文件为预先对对应元数据服务器中保存的元数据信息进行镜像得到的；Controlling each shadow node to obtain the image file of the corresponding metadata server, the image file is obtained by mirroring the metadata information stored in the corresponding metadata server in advance;
如果任一元数据服务器中保存的元数据信息相对于对应镜像文件发生变化，则将该变化对应的日志信息发送至对应的影子节点。If the metadata information stored in any metadata server changes relative to the corresponding image file, the log information corresponding to the change is sent to the corresponding shadow node.
优选的，还包括：Preferably, it also includes:
如果所述故障服务器对应的影子节点为故障节点，则将所述故障服务器对应的镜像文件及日志信息进行合并得到合并元数据信息，并在所述故障服务器重启后将所述合并元数据信息发送至所述故障服务器，以供所述故障服务器重启后基于所述合并元数据信息实现对应元数据服务。If the shadow node corresponding to the faulty server is a faulty node, merge the image file and log information corresponding to the faulty server to obtain the merged metadata information, and send the merged metadata information after the faulty server is restarted to the faulty server, so that after the faulty server is restarted, the corresponding metadata service can be implemented based on the merged metadata information.
优选的，将所述合并元数据信息发送至所述故障服务器之后，还包括：Preferably, after sending the merged metadata information to the faulty server, it also includes:
将所述故障服务器对应的镜像文件及日志信息发送至所述故障服务器，以供所述故障服务器判断所述合并元数据信息与当前接收到的镜像文件及日志信息进行数据一致性校验，并在校验通过后实现对应元数据服务。Send the image file and log information corresponding to the faulty server to the faulty server, so that the faulty server can judge the merged metadata information and the currently received image file and log information for data consistency verification, and After the verification is passed, the corresponding metadata service is implemented.
优选的，还包括：Preferably, it also includes:
定时获取每个所述元数据服务器对应的镜像文件及日志信息，并将获取时间及对应获取到的镜像文件及日志信息进行存储；Obtaining the image files and log information corresponding to each of the metadata servers regularly, and storing the acquisition time and correspondingly obtained image files and log information;
如果每个所述元数据服务器及对应影子节点均发生故障，则在各个所述元数据服务器及对应影子节点重启后将距离发生故障时刻最近的获取时刻对应的镜像文件及日志信息发送至对应的元数据服务器及影子节点。If each of the metadata servers and corresponding shadow nodes fails, after each of the metadata servers and corresponding shadow nodes is restarted, the image file and log information corresponding to the acquisition time closest to the time of failure will be sent to the corresponding Metadata server and shadow nodes.
一种元数据服务器的服务控制装置，包括：A service control device for a metadata server, comprising:
设置模块，用于设置与元数据服务器集群中每个元数据服务器对应的影子节点；A setting module, configured to set a shadow node corresponding to each metadata server in the metadata server cluster;
一致性模块，用于控制每个所述影子节点中保存的数据与对应元数据服务器中保存的元数据信息一致；A consistency module, configured to control the data stored in each shadow node to be consistent with the metadata information stored in the corresponding metadata server;
第一恢复模块，用于如果任一所述元数据服务器发生故障成为故障服务器，则控制该故障服务器对应的影子节点基于存储的数据实现对应元数据服务。The first recovery module is configured to control the shadow node corresponding to the failed server to implement the corresponding metadata service based on the stored data if any of the metadata servers fails and becomes a failed server.
优选的，所述一致性模块包括：Preferably, the consistency module includes:
一致性单元，用于：控制每个影子节点获取对应元数据服务器的镜像文件，所述镜像文件为预先对对应元数据服务器中保存的元数据信息进行镜像得到的；以及用于如果任一元数据服务器中保存的元数据信息相对于对应镜像文件发生变化，则将该变化对应的日志信息发送至对应的影子节点。The consistency unit is used to: control each shadow node to obtain the image file of the corresponding metadata server, the image file is obtained by mirroring the metadata information stored in the corresponding metadata server in advance; If the metadata information stored in the server changes relative to the corresponding image file, the log information corresponding to the change is sent to the corresponding shadow node.
优选的，还包括：Preferably, it also includes:
第二恢复模块，用于如果所述故障服务器对应的影子节点为故障节点，则将所述故障服务器对应的镜像文件及日志信息进行合并得到合并元数据信息，并在所述故障服务器重启后将所述合并元数据信息发送至所述故障服务器，以供所述故障服务器重启后基于所述合并元数据信息实现对应元数据服务。The second recovery module is configured to merge the image file and log information corresponding to the faulty server to obtain merged metadata information if the shadow node corresponding to the faulty server is a faulty node, and merge the metadata information after the faulty server is restarted The merged metadata information is sent to the faulty server, so that after the faulty server is restarted, corresponding metadata services can be implemented based on the merged metadata information.
优选的，还包括：Preferably, it also includes:
校验模块，用于将所述合并元数据信息发送至所述故障服务器之后，将所述故障服务器对应的镜像文件及日志信息发送至所述故障服务器，以供所述故障服务器判断所述合并元数据信息与当前接收到的镜像文件及日志信息进行数据一致性校验，并在校验通过后实现对应元数据服务。A verification module, configured to send the image file and log information corresponding to the faulty server to the faulty server after sending the merged metadata information to the faulty server, so that the faulty server can judge the merged The metadata information is verified for data consistency with the currently received image file and log information, and the corresponding metadata service is implemented after the verification is passed.
优选的，还包括：Preferably, it also includes:
第三恢复模块，用于：定时获取每个所述元数据服务器对应的镜像文件及日志信息，并将获取时间及对应获取到的镜像文件及日志信息进行存储；以及用于如果每个所述元数据服务器及对应影子节点均发生故障，则在各个所述元数据服务器及对应影子节点重启后将距离发生故障时刻最近的获取时刻对应的镜像文件及日志信息发送至对应的元数据服务器及影子节点。The third recovery module is used to: regularly acquire the image files and log information corresponding to each of the metadata servers, and store the acquisition time and the corresponding acquired image files and log information; and for if each of the metadata servers If both the metadata server and the corresponding shadow node fail, after each metadata server and the corresponding shadow node are restarted, the image file and log information corresponding to the acquisition time closest to the failure time will be sent to the corresponding metadata server and shadow node. node.
本发明提供了一种元数据服务器的服务控制方法及装置，其中该方法包括：设置与元数据服务器集群中每个元数据服务器对应的影子节点；控制每个所述影子节点中保存的数据与对应元数据服务器中保存的元数据信息一致；如果任一所述元数据服务器发生故障成为故障服务器，则控制该故障服务器对应的影子节点基于存储的数据实现对应元数据服务。本申请公开的上述技术方案中，为每个元数据服务器设置对应影子节点，且每个影子节点的数据与对应元数据服务器中的元数据信息保持一致，从而在元数据服务器发生故障时能够由对应影子节点代替对应元数据服务器实现对应元数据服务，从而避免了元数据因对应元数据服务器发生故障出现的丢失等情况，提高了元数据的可靠性；进一步通过上述技术方案能够在元数据服务器出现故障时快速恢复其应提供的元数据服务，保证了服务器集群中元数据服务的连续性。The present invention provides a metadata server service control method and device, wherein the method includes: setting a shadow node corresponding to each metadata server in the metadata server cluster; controlling the data stored in each shadow node and The metadata information stored in the corresponding metadata server is consistent; if any of the metadata servers fails and becomes a failed server, control the shadow node corresponding to the failed server to implement the corresponding metadata service based on the stored data. In the above technical solution disclosed in this application, a corresponding shadow node is set for each metadata server, and the data of each shadow node is consistent with the metadata information in the corresponding metadata server, so that when the metadata server fails, the The corresponding shadow node replaces the corresponding metadata server to realize the corresponding metadata service, thereby avoiding the loss of metadata due to the failure of the corresponding metadata server, and improving the reliability of metadata; further through the above technical solution, the metadata server can When a failure occurs, the metadata service it should provide can be quickly restored, which ensures the continuity of the metadata service in the server cluster.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据提供的附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only It is an embodiment of the present invention, and those skilled in the art can also obtain other drawings according to the provided drawings without creative work.
图1为本发明实施例提供的一种元数据服务器的服务控制方法的流程图；FIG. 1 is a flowchart of a service control method of a metadata server provided by an embodiment of the present invention;
图2为本发明实施例提供的一种元数据服务器的服务控制装置的结构示意图。FIG. 2 is a schematic structural diagram of a service control device for a metadata server provided by an embodiment of the present invention.
具体实施方式detailed description
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
请参阅图1，其示出了本发明实施例提供的一种元数据服务器的服务控制方法的流程图，可以包括以下步骤：Please refer to FIG. 1, which shows a flow chart of a metadata server service control method provided by an embodiment of the present invention, which may include the following steps:
S11：设置与元数据服务器集群中每个元数据服务器对应的影子节点。S11: Setting a shadow node corresponding to each metadata server in the metadata server cluster.
在元数据服务器集群中包括多个元数据服务器，每个元数据服务器均存储有其对应部分数据的元数据信息，包括这部分数据对应的文件名称、存储位置等，而对于元数据服务器与数据的对应关系可以预先根据实际需要进行设定。影子节点具体可以为一个处于备用状态下元数据服务器集群中的元数据服务器或者单独的节点，元数据服务器与影子节点一一对应。The metadata server cluster includes multiple metadata servers, and each metadata server stores the metadata information of its corresponding part of the data, including the file name and storage location corresponding to this part of the data. The corresponding relationship can be set in advance according to actual needs. Specifically, the shadow node may be a metadata server in a metadata server cluster in a standby state or a separate node, and the metadata server corresponds to the shadow node one by one.
S12：控制每个影子节点中保存的数据与对应元数据服务器中保存的元数据信息一致。S12: Control the data stored in each shadow node to be consistent with the metadata information stored in the corresponding metadata server.
控制每个影子节点中保存的数据与对应元数据服务器中保存的元数据信息始终保持一致，也即保持每个影子节点中数据与对应元数据服务器中元数据信息始终相同，但是影子节点处于备用状态，即在元数据服务器正常工作时影子节点是不提供元数据服务器，仅需要和对应元数据服务器保存的元数据信息保存一致即可，或者说影子节点为对对应元数据服务器进行的备份。Control that the data stored in each shadow node is always consistent with the metadata information stored in the corresponding metadata server, that is, the data in each shadow node is always the same as the metadata information in the corresponding metadata server, but the shadow node is in standby State, that is, when the metadata server is working normally, the shadow node does not provide a metadata server, and only needs to be consistent with the metadata information saved by the corresponding metadata server, or the shadow node is a backup for the corresponding metadata server.
S13：如果任一元数据服务器发生故障成为故障服务器，则控制该故障服务器对应的影子节点基于存储的数据实现对应元数据服务。S13: If any metadata server fails and becomes a failed server, control the shadow node corresponding to the failed server to implement the corresponding metadata service based on the stored data.
当任一元数据服务器成为故障服务器时，无法继续提供其所维护的元数据信息，影响上层文件系统的访问；此时由于其对应影子节点中的数据与故障服务器发生故障前保存的元数据信息一致，因此可以直接由影子节点替代故障服务器，并提供故障服务器应提供的元数据服务，从而实现元数据服务器的节点热备。When any metadata server becomes a faulty server, it cannot continue to provide metadata information maintained by it, which affects access to the upper file system; at this time, the data in its corresponding shadow node is consistent with the metadata information saved before the faulty server failed , so the failed server can be directly replaced by the shadow node, and the metadata service that the failed server should provide can be provided, so as to realize the node hot backup of the metadata server.
本申请公开的上述技术方案中，为每个元数据服务器设置对应影子节点，且每个影子节点的数据与对应元数据服务器中的元数据信息保持一致，从而在元数据服务器发生故障时能够由对应影子节点代替对应元数据服务器实现对应元数据服务，从而避免了元数据因对应元数据服务器发生故障出现的丢失等情况，提高了元数据的可靠性；进一步通过上述技术方案能够在元数据服务器出现故障时快速恢复其应提供的元数据服务，保证了服务器集群中元数据服务的连续性。In the above technical solution disclosed in this application, a corresponding shadow node is set for each metadata server, and the data of each shadow node is consistent with the metadata information in the corresponding metadata server, so that when the metadata server fails, the The corresponding shadow node replaces the corresponding metadata server to realize the corresponding metadata service, thereby avoiding the loss of metadata due to the failure of the corresponding metadata server, and improving the reliability of metadata; further through the above technical solution, the metadata server can When a failure occurs, the metadata service it should provide can be quickly restored, which ensures the continuity of the metadata service in the server cluster.
本发明实施例提供的一种元数据服务器的服务控制方法，控制每个影子节点中保存的数据与对应元数据服务器中保存的元数据信息一致，可以包括：A metadata server service control method provided by an embodiment of the present invention controls the data stored in each shadow node to be consistent with the metadata information stored in the corresponding metadata server, which may include:
控制每个影子节点获取对应元数据服务器的镜像文件，镜像文件为预先对对应元数据服务器中保存的元数据信息进行镜像得到的；Control each shadow node to obtain the image file of the corresponding metadata server, and the image file is obtained by mirroring the metadata information stored in the corresponding metadata server in advance;
如果任一元数据服务器中保存的元数据信息相对于对应镜像文件发生变化，则将该变化对应的日志信息发送至对应的影子节点。If the metadata information stored in any metadata server changes relative to the corresponding image file, the log information corresponding to the change is sent to the corresponding shadow node.
镜像(Mirroring)是冗余的一种类型，一个磁盘上的数据在另一个磁盘上存在一个完全相同的副本即为镜像，本申请中镜像文件即为对应元数据服务器的副本。并且在任一元数据服务器中保存的元数据信息相对于该元数据服务器的镜像文件发生变化时，可以将该变化记录为日志信息，进而将日志信息发送至对应影子节点，通过得到对应元数据服务器的镜像文件及日志信息，能够有效保证影子节点与对应元数据服务器中保存的元数据信息的一致性。Mirroring is a type of redundancy. The data on one disk has an identical copy on another disk, which is a mirror. In this application, a mirror file is a copy of the corresponding metadata server. And when the metadata information saved in any metadata server changes relative to the image file of the metadata server, the change can be recorded as log information, and then the log information is sent to the corresponding shadow node. The mirror file and log information can effectively ensure the consistency of the metadata information stored in the shadow node and the corresponding metadata server.
本发明实施例提供的一种元数据服务器的服务控制方法，还可以包括：A metadata server service control method provided in an embodiment of the present invention may further include:
如果故障服务器对应的影子节点为故障节点，则将故障服务器对应的镜像文件及日志信息进行合并得到合并元数据信息，并在故障服务器重启后将合并元数据信息发送至故障服务器，以供故障服务器重启后基于合并元数据信息实现对应元数据服务。If the shadow node corresponding to the faulty server is a faulty node, merge the image files and log information corresponding to the faulty server to obtain the merged metadata information, and send the merged metadata information to the faulty server after the faulty server is restarted, for the faulty server After restarting, the corresponding metadata service is implemented based on the merged metadata information.
如果故障服务器对应影子节点也发生了故障成为故障节点，则基于故障服务器的镜像文件及日志信息实现对应元数据服务的恢复，具体来说，可以将日志信息对应的元数据信息的变化替换镜像文件中对应部分的数据，进而得到合并元数据信息，在管理员对故障服务器实现修复操作并重启后将合并元数据信息发送至故障服务器，由此，故障服务器能够基于合并元数据信息提供对应元数据服务，从而进一步保证了元数据服务的连续性。If the shadow node corresponding to the faulty server also fails and becomes a faulty node, the corresponding metadata service can be recovered based on the mirror file and log information of the faulty server. Specifically, the mirror file can be replaced by changes in the metadata information corresponding to the log information The corresponding part of the data in the database, and then obtain the merged metadata information, and after the administrator repairs the faulty server and restarts, the merged metadata information will be sent to the faulty server, so that the faulty server can provide corresponding metadata based on the merged metadata information services, thus further ensuring the continuity of metadata services.
本发明实施例提供的一种元数据服务器的服务控制方法，将合并元数据信息发送至故障服务器之后，还可以包括：A service control method of a metadata server provided by an embodiment of the present invention may further include:
将故障服务器对应的镜像文件及日志信息发送至故障服务器，以供故障服务器判断合并元数据信息与当前接收到的镜像文件及日志信息进行数据一致性校验，并在校验通过后实现对应元数据服务。Send the image file and log information corresponding to the faulty server to the faulty server, so that the faulty server can judge and merge the metadata information with the currently received image file and log information for data consistency verification, and realize the corresponding metadata after the verification is passed. data service.
需要说明的是，为了保证合并元数据信息与对应镜像文件及日志信息的一致性，将合并元数据信息发送至重启后的故障服务器后还可以将故障服务器对应的镜像文件及日志信息发送至故障服务器，故障服务器将镜像文件及日志信息合并并确定合并所得结果与接收到的合并元数据信息是否一致，如果是，则校验通过，故障服务器基于元数据信息实现对应元数据服务，否则，则校验失败，将校验失败的信息可以发送至管理员对应管理终端，且暂时不提供元数据服务，等待管理员的处理，从而进一步保证了元数据的可靠性以及元数据服务的正常提供。It should be noted that, in order to ensure the consistency between the merged metadata information and the corresponding image file and log information, after the merged metadata information is sent to the restarted faulty server, the mirror file and log information corresponding to the faulty server can also be sent to the faulty server. Server, the faulty server merges the image file and log information and determines whether the merged result is consistent with the received merged metadata information, if yes, the verification is passed, and the faulty server implements the corresponding metadata service based on the metadata information, otherwise, then If the verification fails, the verification failure information can be sent to the corresponding management terminal of the administrator, and the metadata service will not be provided temporarily, waiting for the administrator's processing, thereby further ensuring the reliability of the metadata and the normal provision of the metadata service.
本发明实施例提供的一种元数据服务器的服务控制方法，还可以包括：A metadata server service control method provided in an embodiment of the present invention may further include:
定时获取每个元数据服务器对应的镜像文件及日志信息，并将获取时间及对应获取到的镜像文件及日志信息进行存储；Obtain the image files and log information corresponding to each metadata server regularly, and store the acquisition time and the corresponding obtained image files and log information;
如果每个元数据服务器及对应影子节点均发生故障，则在各个元数据服务器及对应影子节点重启后将距离发生故障时刻最近的获取时刻对应的镜像文件及日志信息发送至对应的元数据服务器及影子节点。If each metadata server and corresponding shadow node fails, after each metadata server and corresponding shadow node restarts, the image file and log information corresponding to the acquisition time closest to the failure time will be sent to the corresponding metadata server and Shadow nodes.
定时获取的每个元数据服务器对应的镜像文件及日志信息，包括该元数据服务器的镜像文件及截止到当前时刻其保存的元数据信息相对于镜像文件发生的每次变化对应的日志信息，从而将获取时间及获取到的上述信息进行存储，也即为上述信息增加时间戳，从而能够基于不同获取时间获取到不同的信息。如果每个元数据服务器及对应影子节点均发生故障，则需要获取距离发生上述故障的时刻最近时刻的获取时间获取的对应的镜像文件及日志信息，以使得系统重启，即元数据服务器及影子节点均重启后，元数据服务器及影子节点能够获取到故障前应具有的数据，进而使得元数据服务器正常提供元数据服务，而影子节点继续保持与元数据服务器的数据一致性，从而保证各服务器及节点恢复到故障前的一致状态，实现全局状态恢复。The image file and log information corresponding to each metadata server obtained regularly, including the image file of the metadata server and the log information corresponding to each change of the metadata information saved by the metadata server relative to the image file at the current moment, so that Storing the acquisition time and the acquired information above means adding a time stamp to the above information, so that different information can be acquired based on different acquisition times. If each metadata server and corresponding shadow node fails, you need to obtain the corresponding image file and log information obtained at the time closest to the time when the above failure occurred, so that the system can be restarted, that is, the metadata server and shadow node After restarting, the metadata server and shadow nodes can obtain the data they should have before the failure, so that the metadata server can provide metadata services normally, while the shadow nodes continue to maintain data consistency with the metadata server, thus ensuring that all servers and The node restores to the consistent state before the failure, realizing global state recovery.
需要说明的是，本发明实施例提供的上述技术方案可以应用于大数据文件系统中，该系统主要包括多个元数据服务器、多个数据服务器、共享存储池及多个客户端节点。其中，镜像文件及对应日志信息均可以存储至共享存储池中，具体来说，共享存储池可以维护有每个元数据服务器对应的日志文件，该元数据服务器产生的每次日志信息均可以存储至对应日志文件，并且为了保证镜像文件及日志信息的可靠性，可以在共享存储池中增加其对应的一个或者多个副本，以在由于某些故障导致镜像文件或日志信息丢失或损坏时由对应的副本中获取到对应数据；另外，在由共享存储池中获取镜像文件或日志信息等数据时，可以先由本地磁盘中获取，如果本地磁盘中不存在再由远程节点中获取，从而保证了数据的快速获取。It should be noted that the above-mentioned technical solutions provided by the embodiments of the present invention can be applied to a big data file system, and the system mainly includes multiple metadata servers, multiple data servers, shared storage pools, and multiple client nodes. Among them, the image file and the corresponding log information can be stored in the shared storage pool. Specifically, the shared storage pool can maintain a log file corresponding to each metadata server, and each log information generated by the metadata server can be stored To the corresponding log file, and in order to ensure the reliability of the mirror file and log information, one or more corresponding copies can be added in the shared storage pool, so that when the mirror file or log information is lost or damaged due to some faults, the The corresponding data is obtained from the corresponding copy; in addition, when data such as image files or log information is obtained from the shared storage pool, it can be obtained from the local disk first, and if the local disk does not exist, it can be obtained from the remote node, thus ensuring fast data acquisition.
本发明实施例还提供了一种元数据服务器的服务控制装置，如图2所示，可以包括：An embodiment of the present invention also provides a service control device for a metadata server, as shown in FIG. 2 , which may include:
设置模块11，用于设置与元数据服务器集群中每个元数据服务器对应的影子节点；Setting module 11, configured to set shadow nodes corresponding to each metadata server in the metadata server cluster;
一致性模块12，用于控制每个影子节点中保存的数据与对应元数据服务器中保存的元数据信息一致；Consistency module 12, used to control the data stored in each shadow node to be consistent with the metadata information stored in the corresponding metadata server;
第一恢复模块13，用于如果任一元数据服务器发生故障成为故障服务器，则控制该故障服务器对应的影子节点基于存储的数据实现对应元数据服务。The first recovery module 13 is configured to control the shadow node corresponding to the failed server to implement the corresponding metadata service based on the stored data if any metadata server fails and becomes a failed server.
本发明实施例提供的一种元数据服务器的服务控制装置，一致性模块可以包括：In a service control device for a metadata server provided in an embodiment of the present invention, the consistency module may include:
一致性单元，用于：控制每个影子节点获取对应元数据服务器的镜像文件，镜像文件为预先对对应元数据服务器中保存的元数据信息进行镜像得到的；以及用于如果任一元数据服务器中保存的元数据信息相对于对应镜像文件发生变化，则将该变化对应的日志信息发送至对应的影子节点。The consistency unit is used to: control each shadow node to obtain the image file of the corresponding metadata server, and the image file is obtained by mirroring the metadata information stored in the corresponding metadata server in advance; If the stored metadata information changes relative to the corresponding image file, the log information corresponding to the change is sent to the corresponding shadow node.
本发明实施例提供的一种元数据服务器的服务控制装置，还可以包括：A service control device for a metadata server provided in an embodiment of the present invention may further include:
第二恢复模块，用于如果故障服务器对应的影子节点为故障节点，则将故障服务器对应的镜像文件及日志信息进行合并得到合并元数据信息，并在故障服务器重启后将合并元数据信息发送至故障服务器，以供故障服务器重启后基于合并元数据信息实现对应元数据服务。The second recovery module is used for if the shadow node corresponding to the faulty server is a faulty node, then merge the image file and log information corresponding to the faulty server to obtain the merged metadata information, and send the merged metadata information to The faulty server is used to implement the corresponding metadata service based on the merged metadata information after the faulty server is restarted.
本发明实施例提供的一种元数据服务器的服务控制装置，还可以包括：A service control device for a metadata server provided in an embodiment of the present invention may further include:
校验模块，用于将合并元数据信息发送至故障服务器之后，将故障服务器对应的镜像文件及日志信息发送至故障服务器，以供故障服务器判断合并元数据信息与当前接收到的镜像文件及日志信息进行数据一致性校验，并在校验通过后实现对应元数据服务。The verification module is used to send the image file and log information corresponding to the faulty server to the faulty server after the merged metadata information is sent to the faulty server, so that the faulty server can judge the merged metadata information and the currently received mirrored file and log information The information is verified for data consistency, and the corresponding metadata service is implemented after the verification is passed.
本发明实施例提供的一种元数据服务器的服务控制装置，还可以包括：A service control device for a metadata server provided in an embodiment of the present invention may further include:
第三恢复模块，用于：定时获取每个元数据服务器对应的镜像文件及日志信息，并将获取时间及对应获取到的镜像文件及日志信息进行存储；以及用于如果每个元数据服务器及对应影子节点均发生故障，则在各个元数据服务器及对应影子节点重启后将距离发生故障时刻最近的获取时刻对应的镜像文件及日志信息发送至对应的元数据服务器及影子节点。The third recovery module is used to: regularly obtain the image file and log information corresponding to each metadata server, and store the acquisition time and the correspondingly obtained image file and log information; and for if each metadata server and If all the corresponding shadow nodes fail, after each metadata server and the corresponding shadow nodes are restarted, the image files and log information corresponding to the acquisition time closest to the failure time will be sent to the corresponding metadata servers and shadow nodes.
本发明实施例提供的一种元数据服务器的服务控制装置中相关部分的说明请参见本发明实施例提供的一种元数据服务器的服务控制方法中对应部分的详细说明，在此不再赘述。For the description of the relevant parts of the device for controlling the service of the metadata server provided by the embodiment of the present invention, please refer to the detailed description of the corresponding part of the method for controlling the service of the metadata server provided by the embodiment of the present invention, which will not be repeated here.
对所公开的实施例的上述说明，使本领域技术人员能够实现或使用本发明。对这些实施例的多种修改对本领域技术人员来说将是显而易见的，本文中所定义的一般原理可以在不脱离本发明的精神或范围的情况下，在其它实施例中实现。因此，本发明将不会被限制于本文所示的这些实施例，而是要符合与本文所公开的原理和新颖特点相一致的最宽的范围。The above description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention will not be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611092710.7ACN106789180A (en) | 2016-11-30 | 2016-11-30 | The service control method and device of a kind of meta data server |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201611092710.7ACN106789180A (en) | 2016-11-30 | 2016-11-30 | The service control method and device of a kind of meta data server |
| Publication Number | Publication Date |
|---|---|
| CN106789180Atrue CN106789180A (en) | 2017-05-31 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201611092710.7APendingCN106789180A (en) | 2016-11-30 | 2016-11-30 | The service control method and device of a kind of meta data server |
| Country | Link |
|---|---|
| CN (1) | CN106789180A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108599982A (en)* | 2018-03-14 | 2018-09-28 | 华为技术有限公司 | Data reconstruction method and relevant device |
| CN109165112A (en)* | 2018-08-16 | 2019-01-08 | 郑州云海信息技术有限公司 | A kind of fault recovery method, system and the associated component of metadata cluster |
| CN109947730A (en)* | 2017-07-25 | 2019-06-28 | 中兴通讯股份有限公司 | Metadata restoration methods, device, distributed file system and readable storage medium storing program for executing |
| CN110719326A (en)* | 2019-10-08 | 2020-01-21 | 中国电子科技集团公司第二十八研究所 | Data synchronization method in high-availability software system adapting to low-quality communication link |
| CN111159786A (en)* | 2019-12-29 | 2020-05-15 | 浪潮电子信息产业股份有限公司 | Metadata protection method and device, electronic equipment and storage medium |
| CN112506710A (en)* | 2020-12-16 | 2021-03-16 | 深信服科技股份有限公司 | Distributed file system data repair method, device, equipment and storage medium |
| WO2021189312A1 (en)* | 2020-03-25 | 2021-09-30 | Beijing Didi Infinity Technology And Development Co., Ltd. | Meta server crash recovery in object storage system using enhanced meta structure |
| CN114629907A (en)* | 2022-03-25 | 2022-06-14 | 广州助蜂网络科技有限公司 | Management and control method and device for parking lot equipment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521083A (en)* | 2011-12-13 | 2012-06-27 | 广州杰赛科技股份有限公司 | Backup method and system of virtual machine in cloud computing system |
| CN103024020A (en)* | 2012-12-05 | 2013-04-03 | 蓝盾信息安全技术股份有限公司 | Network data file storing method and device based on WEB application |
| CN103455577A (en)* | 2013-08-23 | 2013-12-18 | 中国科学院计算机网络信息中心 | Multi-backup nearby storage and reading method and system of cloud host mirror image file |
| CN103544081A (en)* | 2013-10-23 | 2014-01-29 | 曙光信息产业(北京)有限公司 | Management method and device for double metadata servers |
| CN103763383A (en)* | 2014-01-27 | 2014-04-30 | 西安雷迪维护系统设备有限公司 | Integrated cloud storage system and storage method thereof |
| CN104932841A (en)* | 2015-06-17 | 2015-09-23 | 南京邮电大学 | Saving type duplicated data deleting method in cloud storage system |
| CN105930228A (en)* | 2016-04-06 | 2016-09-07 | 广州唯品会信息科技有限公司 | Data backup method and system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102521083A (en)* | 2011-12-13 | 2012-06-27 | 广州杰赛科技股份有限公司 | Backup method and system of virtual machine in cloud computing system |
| CN103024020A (en)* | 2012-12-05 | 2013-04-03 | 蓝盾信息安全技术股份有限公司 | Network data file storing method and device based on WEB application |
| CN103455577A (en)* | 2013-08-23 | 2013-12-18 | 中国科学院计算机网络信息中心 | Multi-backup nearby storage and reading method and system of cloud host mirror image file |
| CN103544081A (en)* | 2013-10-23 | 2014-01-29 | 曙光信息产业(北京)有限公司 | Management method and device for double metadata servers |
| CN103763383A (en)* | 2014-01-27 | 2014-04-30 | 西安雷迪维护系统设备有限公司 | Integrated cloud storage system and storage method thereof |
| CN104932841A (en)* | 2015-06-17 | 2015-09-23 | 南京邮电大学 | Saving type duplicated data deleting method in cloud storage system |
| CN105930228A (en)* | 2016-04-06 | 2016-09-07 | 广州唯品会信息科技有限公司 | Data backup method and system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109947730A (en)* | 2017-07-25 | 2019-06-28 | 中兴通讯股份有限公司 | Metadata restoration methods, device, distributed file system and readable storage medium storing program for executing |
| CN109947730B (en)* | 2017-07-25 | 2024-02-02 | 中兴通讯股份有限公司 | Metadata recovery method, device, distributed file system and readable storage medium |
| CN108599982A (en)* | 2018-03-14 | 2018-09-28 | 华为技术有限公司 | Data reconstruction method and relevant device |
| CN108599982B (en)* | 2018-03-14 | 2021-02-12 | 华为技术有限公司 | Data recovery method and related equipment |
| CN109165112B (en)* | 2018-08-16 | 2022-02-18 | 郑州云海信息技术有限公司 | Fault recovery method, system and related components of metadata cluster |
| CN109165112A (en)* | 2018-08-16 | 2019-01-08 | 郑州云海信息技术有限公司 | A kind of fault recovery method, system and the associated component of metadata cluster |
| CN110719326A (en)* | 2019-10-08 | 2020-01-21 | 中国电子科技集团公司第二十八研究所 | Data synchronization method in high-availability software system adapting to low-quality communication link |
| CN110719326B (en)* | 2019-10-08 | 2021-11-12 | 中国电子科技集团公司第二十八研究所 | Data synchronization method in high-availability software system adapting to low-quality communication link |
| CN111159786B (en)* | 2019-12-29 | 2022-04-22 | 浪潮电子信息产业股份有限公司 | Metadata protection method, device, electronic device and storage medium |
| CN111159786A (en)* | 2019-12-29 | 2020-05-15 | 浪潮电子信息产业股份有限公司 | Metadata protection method and device, electronic equipment and storage medium |
| WO2021189312A1 (en)* | 2020-03-25 | 2021-09-30 | Beijing Didi Infinity Technology And Development Co., Ltd. | Meta server crash recovery in object storage system using enhanced meta structure |
| CN112506710A (en)* | 2020-12-16 | 2021-03-16 | 深信服科技股份有限公司 | Distributed file system data repair method, device, equipment and storage medium |
| CN112506710B (en)* | 2020-12-16 | 2024-02-23 | 深信服科技股份有限公司 | Distributed file system data restoration method, device, equipment and storage medium |
| CN114629907A (en)* | 2022-03-25 | 2022-06-14 | 广州助蜂网络科技有限公司 | Management and control method and device for parking lot equipment |
| Publication | Publication Date | Title |
|---|---|---|
| CN106789180A (en) | The service control method and device of a kind of meta data server | |
| US9940206B2 (en) | Handling failed cluster members when replicating a database between clusters | |
| US8260747B2 (en) | System, method, and computer program product for allowing access to backup data | |
| JP5562444B2 (en) | System and method for failing over non-cluster aware applications in a cluster system | |
| US8028192B1 (en) | Method and system for rapid failback of a computer system in a disaster recovery environment | |
| CN106776130B (en) | A log recovery method, storage device and storage node | |
| US20110099148A1 (en) | Verification Of Remote Copies Of Data | |
| US20210320977A1 (en) | Method and apparatus for implementing data consistency, server, and terminal | |
| US8266474B2 (en) | Fast cluster failure detection | |
| US8127174B1 (en) | Method and apparatus for performing transparent in-memory checkpointing | |
| US10509705B2 (en) | Application protection through a combined functionality failure manager | |
| CN105095245B (en) | Archive log synchronization method and system based on relational database | |
| US20120084260A1 (en) | Log-shipping data replication with early log record fetching | |
| JP5292351B2 (en) | Message queue management system, lock server, message queue management method, and message queue management program | |
| US11016863B2 (en) | Self-contained disaster detection for replicated multi-controller systems | |
| JP2023547949A (en) | Synchronous replication of high-throughput streaming data | |
| CN111309524A (en) | Distributed storage system fault recovery method, device, terminal and storage medium | |
| CN109165112B (en) | Fault recovery method, system and related components of metadata cluster | |
| CN106325768A (en) | Dual storage system and method | |
| JP5292350B2 (en) | Message queue management system, lock server, message queue management method, and message queue management program | |
| CN107526652B (en) | A data synchronization method and storage device | |
| CN104133742A (en) | Data protection method and device | |
| CN108319522A (en) | A method of reinforcing distributed memory system reliability | |
| CN110858168A (en) | Cluster node fault processing method and device and cluster node | |
| CN107066355A (en) | The backup server system and its building method of a kind of High Availabitity |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20170531 | |
| RJ01 | Rejection of invention patent application after publication |