CN110795404A - Hadoop distributed file system and operation method and repair method thereof - Google Patents
Hadoop distributed file system and operation method and repair method thereofDownload PDFInfo
- Publication number
- CN110795404A CN110795404ACN201911056278.XACN201911056278ACN110795404ACN 110795404 ACN110795404 ACN 110795404ACN 201911056278 ACN201911056278 ACN 201911056278ACN 110795404 ACN110795404 ACN 110795404A
- Authority
- CN
- China
- Prior art keywords
- name node
- new
- name
- edit log
- hadoop distributed
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Quality & Reliability (AREA)
- Human Computer Interaction (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明涉及数据处理技术领域，尤其涉及一种Hadoop分布式文件系统及其运行方法、修复方法。The invention relates to the technical field of data processing, in particular to a Hadoop distributed file system and its operation method and repair method.
背景技术Background technique
Hadoop集群的存储系统是Hadoop分布式文件系统(Hadoop Distributed FileSystem，HDFS)，HDFS可以有效的解决海量数据的存储和管理难题，具体为：将固定于某个地点的某个文件系统，扩展到任意多个地点、多个文件系统，众多的节点组成一个文件系统网络。每个节点可以分布在不同的地点，通过网络进行节点间的通信和数据传输。The storage system of the Hadoop cluster is the Hadoop Distributed File System (HDFS). HDFS can effectively solve the storage and management problems of massive data. Multiple locations, multiple file systems, and numerous nodes form a file system network. Each node can be distributed in different locations, and the communication and data transmission between nodes are carried out through the network.
在使用HDFS时，无需关心数据是存储在哪个节点上、或者是从哪个节点获取的，只需要像使用本地文件系统一样管理和存储文件系统中的数据。When using HDFS, you don't need to care which node the data is stored on, or from which node it is obtained, you only need to manage and store the data in the file system as if you were using the local file system.
发明内容SUMMARY OF THE INVENTION
本发明的实施例提供一种Hadoop分布式文件系统及其运行方法、修复方法，可以更高频且更多的保存元数据序列，增强数据的安全性。Embodiments of the present invention provide a Hadoop distributed file system, an operation method and a repair method thereof, which can save metadata sequences more frequently and more, and enhance data security.
为达到上述目的，本发明的实施例采用如下技术方案：To achieve the above object, the embodiments of the present invention adopt the following technical solutions:
一方面，提供一种Hadoop分布式文件系统，包括：第一名称节点、与所述第一名称节点连接的第二名称节点；所述第一名称节点，用于存储元数据文件；所述元数据文件包括镜像文件和编辑日志；所述第一名称节点还用于每隔预设的第一时长，将所述镜像文件与所述编辑日志合并，形成新的镜像文件，并启用新的编辑日志；所述第二名称节点，用于每隔预设的第一时长，对所述第一名称节点上新的镜像文件备份；还用于每隔预设的第二时长，对所述第一名称节点上的所述编辑日志备份；所述第二时长小于所述第一时长；其中，所述第一名称节点和所述第二名称节点分别部署于不同的主机。In one aspect, a Hadoop distributed file system is provided, comprising: a first name node and a second name node connected to the first name node; the first name node is used to store metadata files; the metadata The data file includes a mirror file and an edit log; the first name node is also used to merge the mirror file and the edit log every preset first time length to form a new mirror file and enable new edits log; the second name node is used for backing up the new image file on the first name node every preset first duration; it is also used for every preset second duration, the The editing log backup on a name node; the second duration is shorter than the first duration; wherein, the first name node and the second name node are respectively deployed on different hosts.
再一方面，提供一种如上所述的Hadoop分布式文件系统的运行方法，包括：启动第一名称节点，加载元数据文件，同时启动第二名称节点；所述元数据文件包括镜像文件和编辑日志；每隔预设的第一时长，所述第一名称节点将所述镜像文件与所述编辑日志合并，形成新的所述镜像文件，同时启用新的编辑日志；所述第二名称节点对所述第一名称节点上新的所述镜像文件备份；每隔预设的第二时长，所述第二名称节点对所述第一名称节点上的所述编辑日志备份；所述第二时长小于所述第一时长；其中，所述第一名称节点与所述第二名称节点分别部署于不同主机。In another aspect, a method for operating a Hadoop distributed file system as described above is provided, including: starting a first name node, loading a metadata file, and starting a second name node at the same time; the metadata file includes an image file and an editor log; every preset first duration, the first name node merges the mirror file and the edit log to form a new mirror file, and enables a new edit log at the same time; the second name node Back up the new image file on the first name node; every preset second time period, the second name node backs up the edit log on the first name node; the second name node The duration is shorter than the first duration; wherein, the first name node and the second name node are respectively deployed on different hosts.
另一方面，提供一种如上所述的Hadoop分布式文件系统的修复方法，包括：停止受损的第一名称节点，同时停止第二名称节点；备份所述第二名称节点上的元数据文件；所述元数据文件包括所述第二名称节点最后一次从受损的所述第一名称节点上备份的镜像文件和编辑日志；准备部署新的第一名称节点的主机；部署新的第一名称节点的主机与部署所述受损的第一名称节点、部署所述第二名称节点的主机均不同；配置所述主机的名称、IP地址、互信登录、运行环境和集群；对新的所述第一名称节点执行格式化；将备份的所述第二名称节点上的镜像文件和编辑日志，发送至新的所述第一名称节点；修改新的所述第一名称节点的元数据序号。In another aspect, a method for repairing a Hadoop distributed file system as described above is provided, comprising: stopping the damaged first name node and simultaneously stopping the second name node; and backing up the metadata file on the second name node ; The metadata file includes the mirror file and edit log that the second name node backed up from the damaged first name node for the last time; prepare to deploy the host of the new first name node; deploy the new first name node The host of the name node is different from the host on which the damaged first name node and the second name node are deployed; configure the name, IP address, mutual trust login, operating environment and cluster of the host; Execute formatting on the first name node; send the backed up mirror file and edit log on the second name node to the new first name node; modify the metadata sequence number of the new first name node .
可选地，在修改新的所述第一名称节点的元数据序号之后，还包括：启动新的所述第一名称节点和对应的Hadoop分布式文件系统；执行Hadoop分布式文件系统检查。Optionally, after modifying the metadata sequence number of the new first name node, the method further includes: starting the new first name node and the corresponding Hadoop distributed file system; and performing a Hadoop distributed file system check.
又一方面，提供一种Hadoop分布式文件系统，包括：第一名称节点、与所述第一名称节点连接的第二名称节点；所述第一名称节点，用于存储元数据文件；所述元数据文件包括镜像文件和编辑日志；所述第一名称节点还用于每隔预设的第一时长，启用新的编辑日志；所述第二名称节点，用于每隔预设的第一时长，对所述第一名称节点上原有的所述镜像文件和原有的所述编辑日志备份，将备份的所述镜像文件和所述编辑日志合并，形成新的所述镜像文件，并发送回所述第一名称节点，替换所述第一名称节点原有的镜像文件；所述第二名称节点，还用于每隔预设的第二时长，对所述第一名称节点上的所述编辑日志备份，形成中间编辑日志；所述第二时长小于所述第一时长；其中，所述第一名称节点和所述第二名称节点分别部署于不同的主机。In yet another aspect, a Hadoop distributed file system is provided, including: a first name node, a second name node connected to the first name node; the first name node is used to store metadata files; the The metadata file includes a mirror file and an edit log; the first name node is also used to enable a new edit log every preset first duration; the second name node is used to every preset first Duration, back up the original image file and the original edit log on the first name node, merge the backed up image file and the edit log to form a new image file, and send Go back to the first name node, and replace the original image file of the first name node; the second name node is also used for every preset second time period. The editing log is backed up to form an intermediate editing log; the second duration is shorter than the first duration; wherein, the first name node and the second name node are respectively deployed on different hosts.
又一方面，提供一种如上所述的Hadoop分布式文件系统的运行方法，包括：启动第一名称节点，加载元数据文件，同时启动第二名称节点；所述元数据文件包括镜像文件和编辑日志；每隔预设的第一时长，所述第一名称节点启动新的编辑日志；所述第二名称节点对所述第一名称节点上原有的所述镜像文件和原有的所述编辑日志备份，将备份的所述镜像文件与所述编辑日志合并，形成新的所述镜像文件，并发送回所述第一名称节点，替换所述第一名称节点原有的镜像文件；每隔预设的第二时长，所述第二名称节点对所述第一名称节点上的所述编辑日志备份，形成中间编辑日志；所述第二时长小于所述第一时长；其中，所述名称节点与所述第二名称节点分别部署于不同主机。In another aspect, a method for operating a Hadoop distributed file system as described above is provided, comprising: starting a first name node, loading a metadata file, and starting a second name node at the same time; the metadata file includes an image file and an editor log; every preset first duration, the first name node starts a new edit log; the second name node records the original image file and the original edit log on the first name node log backup, merge the backed up image file with the edit log to form a new image file, and send it back to the first name node to replace the original image file of the first name node; every other For a preset second duration, the second name node backs up the edit log on the first name node to form an intermediate edit log; the second duration is less than the first duration; wherein the name The node and the second name node are respectively deployed on different hosts.
又一方面，提供一种如上所述的Hadoop分布式文件系统的修复方法，包括：停止受损的第一名称节点，同时停止第二名称节点；备份所述第二名称节点上的元数据文件；所述元数据文件包括所述第二名称节点最后一次从受损的所述第一名称节点上备份的镜像文件和编辑日志，以及最后一次形成的中间编辑日志；准备部署新的第一名称节点的主机；部署新的第一名称节点的主机与部署所述受损的第一名称节点、部署所述第二名称节点的主机均不同；配置所述主机的名称、IP地址、互信登录、运行环境和集群；对新的所述第一名称节点执行格式化；将备份的所述第二名称节点上的镜像文件、中间编辑日志和中间编辑日志合并，生成新的镜像文件，发送至新的所述第一名称节点；修改新的所述第一名称节点的元数据序号。In yet another aspect, a method for repairing a Hadoop distributed file system as described above is provided, comprising: stopping the damaged first name node and simultaneously stopping the second name node; and backing up the metadata file on the second name node ; The metadata file includes the mirror file and the edit log that the second name node backed up from the damaged first name node for the last time, and the intermediate edit log formed for the last time; Prepare to deploy a new first name The host of the node; the host where the new first name node is deployed is different from the host where the damaged first name node and the second name node are deployed; the name, IP address, mutual trust login, Run the environment and cluster; perform formatting on the new first name node; merge the mirror file, the intermediate edit log and the intermediate edit log on the backed up second name node, generate a new image file, and send it to the new the first name node; modify the metadata sequence number of the new first name node.
可选地，在修改新的所述第一名称节点的元数据序号之后，还包括：启动新的所述第一名称节点和对应的Hadoop分布式文件系统；执行Hadoop分布式文件系统检查。Optionally, after modifying the metadata sequence number of the new first name node, the method further includes: starting the new first name node and the corresponding Hadoop distributed file system; and performing a Hadoop distributed file system check.
又一方面，提供一种计算机设备，包括存储单元和处理单元；所述存储单元中存储可在所述处理单元上运行的计算机程序并存储结果；所述处理单元执行所述计算机程序时实现如上所述的Hadoop分布式文件系统的运行方法，和/或如上所述的Hadoop分布式文件系统的修复方法。In another aspect, a computer device is provided, comprising a storage unit and a processing unit; the storage unit stores a computer program that can be run on the processing unit and stores results; the processing unit implements the above when executing the computer program The operating method of the Hadoop distributed file system, and/or the above-mentioned repairing method of the Hadoop distributed file system.
又一方面，提供一种计算机可读介质，其存储有计算机程序，所述计算机程序被处理器执行时实现如上所述的Hadoop分布式文件系统的运行方法，和/或如上所述的Hadoop分布式文件系统的修复方法。In yet another aspect, a computer-readable medium is provided, which stores a computer program, and when the computer program is executed by a processor, implements the above-mentioned operating method of the Hadoop distributed file system, and/or the above-mentioned Hadoop distribution Repair method of the file system.
本发明的实施例提供一种Hadoop分布式文件系统及其运行方法、修复方法，在第一名称节点每隔预设的第一时长，将镜像文件与编辑日志合并，形成新的镜像文件，并启用新的编辑日志的基础上，通过与第一名称节点连接的第二名称节点，每隔预设的第一时长，对第一名称节点上新的镜像文件备份，以及每隔预设的第二时长，对第一名称节点上的编辑日志备份，第二时长小于第一时长，使得编辑日志进行了更高频的备份，保存了更多的元数据序列，增强了Hadoop分布式文件系统中数据的安全性。Embodiments of the present invention provide a Hadoop distributed file system, an operation method, and a repair method thereof. At a first name node, every preset first time period, a mirror file and an edit log are merged to form a new mirror file, and a new mirror file is formed. On the basis of enabling the new edit log, through the second name node connected to the first name node, every preset first time period, backup the new image file on the first name node, and every preset first name node. The second duration is for backing up the edit log on the first name node. The second duration is shorter than the first duration, so that the edit log is backed up more frequently, more metadata sequences are saved, and the Hadoop distributed file system is enhanced. Data security.
附图说明Description of drawings
为了更清楚地说明本发明实施例或现有技术中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to explain the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained according to these drawings without creative efforts.
图1为现有的一种Hadoop分布式文件系统的结构示意图；1 is a schematic structural diagram of an existing Hadoop distributed file system;
图2为本发明的实施例提供的一种Hadoop分布式文件系统的运行方法的流程示意图；2 is a schematic flowchart of a method for operating a Hadoop distributed file system provided by an embodiment of the present invention;
图3为本发明的实施例提供的一种Hadoop分布式文件系统的修复方法的流程示意图；3 is a schematic flowchart of a method for repairing a Hadoop distributed file system according to an embodiment of the present invention;
图4为本发明的实施例提供的再一种Hadoop分布式文件系统的修复方法的流程示意图；4 is a schematic flowchart of still another method for repairing a Hadoop distributed file system according to an embodiment of the present invention;
图5为本发明的实施例提供的再一种Hadoop分布式文件系统的运行方法的流程示意图；FIG. 5 is a schematic flowchart of still another method for running a Hadoop distributed file system according to an embodiment of the present invention;
图6为本发明的实施例提供的另一种Hadoop分布式文件系统的修复方法的流程示意图；6 is a schematic flowchart of another method for repairing a Hadoop distributed file system according to an embodiment of the present invention;
图7为本发明的实施例提供的又一种Hadoop分布式文件系统的修复方法的流程示意图。FIG. 7 is a schematic flowchart of still another method for repairing a Hadoop distributed file system according to an embodiment of the present invention.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
现有技术中，如图1所示，Hadoop分布式文件系统通常包括：第一名称节点(NameNode)、与第一名称节点连接的第二名称节点(Secondary Name Node)，与第一名称节点连接的多个数据节点(Data Node)。In the prior art, as shown in FIG. 1, a Hadoop distributed file system generally includes: a first name node (NameNode), a second name node (Secondary Name Node) connected to the first name node, and connected to the first name node. of multiple Data Nodes.
第一名称节点用于存储元数据文件；元数据文件包括镜像文件和编辑日志。The first namenode is used to store metadata files; metadata files include mirror files and edit logs.
其中，元数据定义为：描述数据的数据，主要是描述数据属性的信息，用于支持如指示存储位置、历史数据、资源查找、文件记录等功能。元数据是一种电子式目录。镜像文件为所有元数据序列化后形成的文件；编辑日志记录客户端更新元数据的每一步操作。Among them, metadata is defined as: data describing data, mainly information describing data attributes, and used to support functions such as indicating storage location, historical data, resource search, and file recording. Metadata is an electronic catalog. The image file is a file formed by serializing all metadata; the edit log records every step of the client to update the metadata.
第一名称节点还用于元数据(Meta data)管理，例如查询、修改；以及负责客户端对文件的访问。The first name node is also used for metadata (Meta data) management, such as query and modification; and is responsible for client access to files.
第二名称节点是第一名称节点的冷备份。The second namenode is a cold backup of the first namenode.
数据节点为HDFS提供数据块存储数据；数据节点通常以机架的形式组织，机架通过一个交换机将所有系统连接起来。Data nodes provide HDFS with block storage data; data nodes are usually organized in racks that connect all systems through a switch.
需要说明的是，由于第一名称节点是整个Hadoop分布式文件系统的核心，一旦发生损坏，将危及到整个Hadoop分布式文件系统的数据安全，由此，为了保证第一名称节点的安全，增设第二名称节点定时备份第一名称节点的元数据文件。若名称节点损坏，第二名称节点不代替第一名称节点工作，但是第二名称节点上存储有第一名称节点的一些信息，可发送回第一名称节点，由此，减少第一名称节点损坏造成的损失。It should be noted that since the first name node is the core of the entire Hadoop distributed file system, once it is damaged, it will endanger the data security of the entire Hadoop distributed file system. Therefore, in order to ensure the security of the first name node, an additional The second namenode periodically backs up the metadata file of the first namenode. If the name node is damaged, the second name node does not work in place of the first name node, but some information of the first name node is stored on the second name node and can be sent back to the first name node, thereby reducing the damage of the first name node losses caused.
然而，随着Hadoop分布式文件系统的规模越来越大，第一名称节点会变得非常繁忙，使得元数据文件变得非常大，不适合进行高频次的备份，增加负担。但是，若在备份间隔时间较长，在间隔期间，第一名称节点突然损坏，将会导致第一名称节点从上一次备份到损坏期间的所有元数据改动序列丢失。However, as the scale of the Hadoop distributed file system becomes larger and larger, the first name node will become very busy, making the metadata file very large, which is not suitable for frequent backups and increases the burden. However, if the backup interval is long, during the interval, the first name node is suddenly damaged, which will cause the loss of all metadata change sequences of the first name node from the last backup to the damage period.
基于上述问题，本发明的实施例提供一种Hadoop分布式文件系统，包括：第一名称节点、与第一名称节点连接的第二名称节点。Based on the above problems, embodiments of the present invention provide a Hadoop distributed file system, including: a first name node and a second name node connected to the first name node.
第一名称节点，用于存储元数据文件；元数据文件包括镜像文件和编辑日志。The first name node is used to store metadata files; metadata files include image files and edit logs.
第一名称节点还用于每隔预设的第一时长，将镜像文件与编辑日志合并，形成新的镜像文件；同时，启用新的编辑日志。The first name node is also used for merging the image file and the edit log every preset first time period to form a new image file; at the same time, enabling a new edit log.
其中，启用的新的编辑日志是空文件。where the new edit log enabled is an empty file.
第二名称节点，用于每隔预设的第一时长，对第一名称节点上新的镜像文件备份；还用于每隔预设的第二时长，对第一名称节点上的编辑日志备份；第二时长小于第一时长。The second name node is used for backing up the new image file on the first name node every preset first time period; it is also used for backing up the edit log on the first name node every preset second time period ; The second duration is less than the first duration.
其中，第一名称节点和第二名称节点分别部署于不同的主机。Wherein, the first name node and the second name node are respectively deployed on different hosts.
需要说明的是，第二名称节点对第一名称节点的元数据文件进行备份时，同时会删除上一次备份的元数据文件。It should be noted that when the second name node backs up the metadata file of the first name node, the metadata file of the last backup will be deleted at the same time.
示例的，若预设的第一时长为1小时，预设的第二时长为10分钟，例如2点时，第一名称节点将镜像文件与编辑日志合并，形成新的镜像文件A2，同时，启用新的编辑日志B2，第二名称节点对新的镜像文件A2和新的编辑日志B2备份。For example, if the preset first duration is 1 hour, and the preset second duration is 10 minutes, for example, at 2 o'clock, the first name node merges the image file with the edit log to form a new image file A2 , and at the same time , enable the new edit log B2 , and the second name node backs up the new mirror file A2 and the new edit log B2 .
2点10分时，第二名称节点对第一名称节点上的编辑日志B2′(此时，该编辑日志B2′中存储的是2点到2点10分期间元数据的改动序列)备份，同时，删除上一次2点时备份的编辑日志B2。At 2:10, the second name node edits the log B2 ' on the first name node (at this time, the edit log B2 ' stores the metadata change sequence from 2:00 to 2:10) The backup, at the same time, deletes the edit log B2 that was last backed up at 2 o'clock.
2点20分时，第二名称节点对第一名称节点上的编辑日志B2″(此时，该编辑日志B″中存储的是2点到2点20分期间元数据的改动序列)备份，同时，删除上一次2点10分备份的编辑日志B2′。At 2:20, the second name node backs up the edit log B2 " on the first name node (at this time, the edit log B " stores the sequence of changes in metadata from 2:00 to 2:20) , and at the same time, delete the edit log B2 ' that was backed up last time at 2:10.
依次类推，直至3点时，第一名称节点将镜像文件A2和编辑日志B2″″′(此时，该编辑日志中存储的是2点到3点期间元数据的改动序列)合并，形成新的镜像文件A3，同时，启用新的编辑日志B3。第二名称节点对第一名称节点上的新的镜像文件A3和新的编辑日志B3备份，同时，删除2点时备份的镜像文件A2和2点50时备份的编辑日志B2″″′。And so on, until 3 o'clock, the first name node merges the mirror file A2 and the edit log B2 ""' (at this time, the edit log stores the metadata change sequence from 2 o'clock to 3 o'clock), A new mirror file A3 is formed, and at the same time, a new edit log B3 is enabled. The second name node backs up the new mirror file A3 and the new edit log B3 on the first name node, and at the same time, deletes the mirror file A2 backed up at 2:00 and the edit log B2 ” backed up at 2:50. "'.
通过在第二名称节点对于第一名称节点上的镜像文件和编辑日志备份时，设置不同的间隔时长，使得相对于镜像文件，较小的编辑日志能单独进行更高频的备份，保存更多的元数据序列，进而减少第一名称节点损坏造成的损失。By setting different interval lengths when the second name node backs up the mirror file and edit log on the first name node, compared with the mirror file, the smaller edit log can be backed up more frequently and save more The metadata sequence, thereby reducing the loss caused by the damage of the first name node.
本发明的实施例提供一种Hadoop分布式文件系统，在第一名称节点每隔预设的第一时长，将镜像文件与编辑日志合并，形成新的镜像文件，并启用新的编辑日志的基础上，通过与第一名称节点连接的第二名称节点，每隔预设的第一时长，对第一名称节点上新的镜像文件备份，以及每隔预设的第二时长，对第一名称节点上的编辑日志备份，第二时长小于第一时长，使得编辑日志进行了更高频的备份，保存了更多的元数据序列，增强了Hadoop分布式文件系统中数据的安全性。The embodiment of the present invention provides a Hadoop distributed file system, in the first name node every preset first time period, the mirror file and the editing log are merged to form a new mirror file, and the basis for enabling the new editing log , through the second name node connected to the first name node, every preset first time period, back up the new image file on the first name node, and every preset second time period, backup the first name node For the editing log backup on the node, the second duration is shorter than the first duration, so that the editing log is backed up more frequently, more metadata sequences are saved, and the data security in the Hadoop distributed file system is enhanced.
本发明的实施例还提供Hadoop分布式文件系统的运行方法，如图2所示，包括：Embodiments of the present invention also provide a method for running a Hadoop distributed file system, as shown in FIG. 2 , including:
S10、启动第一名称节点，加载元数据文件，同时启动第二名称节点。S10. Start the first name node, load the metadata file, and start the second name node at the same time.
元数据文件包括镜像文件和编辑日志。Metadata files include image files and edit logs.
需要说明的是，若第一名称节点第一次启动，则启用新的镜像文件和编辑日志，若不是第一启动，则直接加载镜像文件和编辑日志。It should be noted that if the first name node is started for the first time, a new image file and edit log are enabled, and if it is not the first startup, the image file and edit log are directly loaded.
S11、每隔预设的第一时长，第一名称节点将镜像文件与编辑日志合并，形成新的镜像文件，同时启用新的编辑日志。S11. Every preset first time period, the first name node merges the image file and the edit log to form a new image file, and simultaneously enables a new edit log.
S12、每隔预设的第一时长，第二名称节点对第一名称节点上新的镜像文件备份。S12. Every preset first time period, the second name node backs up the new image file on the first name node.
S13、每隔预设的第二时长，第二名称节点对第一名称节点上的编辑日志备份。S13. Every preset second time period, the second name node backs up the edit log on the first name node.
其中，第二时长小于第一时长。第一名称节点与第二名称节点分别部署于不同主机。Wherein, the second duration is shorter than the first duration. The first name node and the second name node are respectively deployed on different hosts.
本发明的实施例提供的Hadoop分布式文件系统的运行方法与上述Hadoop分布式文件系统具有相同的有益效果，在此不再赘述。The operation method of the Hadoop distributed file system provided by the embodiment of the present invention has the same beneficial effects as the above-mentioned Hadoop distributed file system, which will not be repeated here.
本发明的实施例还提供Hadoop分布式文件系统的修复方法，如图3所示，包括：Embodiments of the present invention also provide a method for repairing a Hadoop distributed file system, as shown in FIG. 3 , including:
S20、停止受损的第一名称节点，同时停止第二名称节点。S20. Stop the damaged first name node, and at the same time stop the second name node.
可以理解的是，停止受损的第一名称节点和第二名称节点的相关服务，防止Hadoop分布式文件系统的数据块发生变化。Understandably, stopping the related services of the compromised first namenode and second namenode prevents changes to the data blocks of the Hadoop distributed file system.
S21、备份第二名称节点上的元数据文件。S21. Back up the metadata file on the second name node.
元数据文件包括第二名称节点最后一次从受损的第一名称节点上备份的镜像文件和编辑日志。The metadata file includes the last backup image file and edit log of the second namenode from the compromised first namenode.
可以理解的是，由于对镜像文件和编辑日志备份的间隔时长不同，最后一次从受损的第一名称节点上备份镜像文件的时间和最后一次备份编辑日志的时间不同。It is understandable that, due to the difference in the interval between backing up the mirror file and the edit log, the time of the last backup of the mirror file from the damaged first name node is different from the time of the last backup of the edit log.
示例的，若预设的第一时长为1小时，预设的第二时长为10分钟，例如2点时，第一名称节点将镜像文件与编辑日志合并，形成新的镜像文件A2，同时，启用新的编辑日志B2，第二名称节点对新的镜像文件A2和新的编辑日志B2备份；2点10分时，第二名称节点对第一名称节点上的编辑日志B2′备份，此时，该编辑日志B2′中存储的是2点到2点10分期间元数据的改动序列，同时，删除上一次2点时备份的编辑日志B2。For example, if the preset first duration is 1 hour, and the preset second duration is 10 minutes, for example, at 2 o'clock, the first name node merges the image file with the edit log to form a new image file A2 , and at the same time , enable the new edit log B2 , the second name node backs up the new mirror file A2 and the new edit log B2 ; at 2:10, the second name node backs up the edit log B2 on the first name node 'Backup, at this time, the edit log B2 ' stores the metadata change sequence from 2:00 to 2:10, and at the same time, deletes the last backup edit log B2 at 2:00.
若2点19分第一名称节点损坏，则第二名称节点上最后一次从受损的第一名称节点上备份的元数据文件指的是2点时备份的镜像文件A2，以及2点10分时备份的编辑日志B2′。If the first namenode is damaged at 2:19, the last metadata file backed up on the second namenode from the damaged first namenode refers to the mirror file A2 backed up at 2:00, and the image file A 2 backed up at 2:10 Edit log B2 ' for time-sharing backup.
S22、准备部署新的第一名称节点的主机。S22. Prepare to deploy the host of the new first name node.
部署新的第一名称节点的主机与部署受损的第一名称节点、部署第二名称节点的主机均不同。The host on which the new first namenode is deployed is different from the host on which the compromised first namenode and the second namenode are deployed.
S23、配置主机的名称、IP地址、互信登录、运行环境和集群。S23, configure the name, IP address, mutual trust login, operating environment and cluster of the host.
S24、对新的第一名称节点执行格式化。S24. Perform formatting on the new first name node.
需要说明的是，执行格式化后新的第一名称节点上的元数据序号被清零。It should be noted that, after the formatting is performed, the metadata sequence number on the new first name node is cleared to zero.
S25、将备份的第二名称节点上的镜像文件和编辑日志，发送至新的第一名称节点。S25. Send the mirror file and edit log on the backup second name node to the new first name node.
示例的，根据S21中的示例可知，第二名称节点上最后一次从受损的第一名称节点上备份的元数据文件指的是2点时备份的镜像文件A2，以及2点10分时备份的编辑日志B2′，将该2点时的镜像文件A2，以及2点10分时备份的编辑日志B2′，发送至第一名称节点。Illustratively, according to the example in S21, it can be known that the last metadata file backed up on the second name node from the damaged first name node refers to the image file A2 backed up at 2:00, and the image file A 2 backed up at 2:10. The backed up edit log B2 ′ is sent to the first name node by the mirror file A2 at 2:00 and the backed up edit log B2 ′ at 2:10.
S26、修改新的第一名称节点的元数据序号。S26. Modify the metadata sequence number of the new first name node.
可以理解的是，手动修改新的第一名称节点的元数据序号，使得该元数据序号可以与发送回的编辑日志中的元数据序列进行衔接，从而，在新的第一名称节点进行运行时，继续扩展整个元数据序列。It can be understood that the metadata sequence number of the new first name node is manually modified, so that the metadata sequence number can be connected with the metadata sequence in the sent back edit log, so that when the new first name node runs , which continues to expand the entire metadata sequence.
可选地，在S26之后，如图4所示，Hadoop分布式文件系统的修复方法还包括：Optionally, after S26, as shown in FIG. 4, the method for repairing the Hadoop distributed file system further includes:
S27、启动新的第一名称节点和对应的Hadoop分布式文件系统。S27. Start the new first name node and the corresponding Hadoop distributed file system.
S28、执行Hadoop分布式文件系统检查。S28. Execute the Hadoop distributed file system check.
若对Hadoop分布式文件系统进行检查后，确认修复，则修复完成，若检查还未修复，则再次执行S22～S26的步骤，重新进行修复。If the repair is confirmed after checking the Hadoop distributed file system, the repair is completed. If the inspection has not been repaired, steps S22 to S26 are performed again to repair.
本发明的实施例还提供一种Hadoop分布式文件系统，包括：第一名称节点、与第一名称节点连接的第二名称节点。Embodiments of the present invention also provide a Hadoop distributed file system, including: a first name node, and a second name node connected to the first name node.
第一名称节点，用于存储元数据文件；元数据文件包括镜像文件和编辑日志；第一名称节点还用于每隔预设的第一时长，启用新的编辑日志。The first name node is used to store the metadata file; the metadata file includes the mirror file and the edit log; the first name node is also used for enabling a new edit log every preset first time period.
第二名称节点，用于每隔预设的第一时长，对第一名称节点上原有的镜像文件和原有的编辑日志备份，将备份的镜像文件和编辑日志合并，形成新的镜像文件，并发送回第一名称节点，替换第一名称节点原有的镜像文件。The second name node is used to back up the original image file and the original edit log on the first name node every preset first duration, and merge the backed up image file and edit log to form a new image file, And send it back to the first name node to replace the original image file of the first name node.
第二名称节点，还用于每隔预设的第二时长，对第一名称节点上的编辑日志备份，形成中间编辑日志；第二时长小于第一时长。The second name node is also used for backing up the editing log on the first name node every preset second duration to form an intermediate editing log; the second duration is shorter than the first duration.
其中，第一名称节点和第二名称节点分别部署于不同的主机。Wherein, the first name node and the second name node are respectively deployed on different hosts.
示例的，若预设的第一时长为1小时，预设的第二时长为15分钟，例如1点时，第一名称节点启动新的编辑日志b1；第二名称节点对第一名称节点上原有的镜像文件和原有的编辑日志备份，将备份的镜像文件和编辑日志合并，形成新的镜像文件a1，并发送回第一名称节点，替换第一名称节点原有的镜像文件，同时，第二名称节点对第一名称节点上新的编辑日志b1备份，形成中间编辑日志c1。For example, if the preset first duration is 1 hour, and the preset second duration is 15 minutes, for example, at 1 o'clock, the first name node starts a new edit log b1 ; backup the original image file and the original edit log, merge the backed-up image file and edit log to form a new image file a1 , and send it back to the first name node to replace the original image file of the first name node, At the same time, the second name node backs up the new edit log b1 on the first name node to form an intermediate edit log c1 .
1点15分时，第二名称节点对第一名称节点上的编辑日志b1′(此时，该编辑日志b1′中存储的是1点到1点15分期间元数据的改动序列)备份，形成中间编辑日志c1′，同时，删除上一次备份的中间编辑日志c1。At 1:15, the second name node edits the log b1 ' on the first name node (at this time, the edit log b1 ' stores the metadata change sequence from 1:00 to 1:15) Backup to form an intermediate edit log c1 ′, and at the same time, delete the intermediate edit log c1 of the last backup.
1点30分时，第二名称节点对第一名称节点上的编辑日志b1″(此时，该编辑日志b1″中存储的是1点到1点30分期间元数据的改动序列)备份，形成中间编辑日志c1″，同时，删除上一次备份的中间编辑日志c1′。At 1:30, the second name node edits the log b1 ″ on the first name node (at this time, the edit log b1 ″ stores the metadata change sequence from 1:00 to 1:30) Backup to form an intermediate edit log c1 ″, and at the same time, delete the intermediate edit log c1 ′ of the last backup.
依次类推，直至2点时，第一名称节点启动新的编辑日志b2；第二名称节点对第一名称节点上的镜像文件a1和编辑日志b1″′(此时，该编辑日志b1″′中存储的是1点到2点期间元数据的改动序列)备份，将备份的镜像文件a1和编辑日志b1″′合并，形成新的镜像文件a2，并发送回第一名称节点，替换第一名称节点原有的镜像文件a1，同时，第二名称节点对第一名称节点上新的编辑日志b2备份，形成中间编辑日志c2。By analogy, until 2 o'clock, thefirst name node startsa new edit log b2 ;1 "' stores the metadata change sequence from 1:00 to 2:00) backup, merge the backup image file a1 and the edit log b1 "' to form a new image file a2 , and send it back to the first The name node replaces the original image file a1 of the first name node, and at the same time, the second name node backs up the new edit log b2 on the first name node to form an intermediate edit log c2 .
通过在第二名称节点对于第一名称节点上的镜像文件和编辑日志备份时，设置不同的间隔时长，使得相对于镜像文件，较小的编辑日志能单独进行更高频的备份，保存更多的元数据序列，进而减少第一名称节点损坏造成的损失。By setting different interval lengths when the second name node backs up the mirror file and edit log on the first name node, compared with the mirror file, the smaller edit log can be backed up more frequently and save more The metadata sequence, thereby reducing the loss caused by the damage of the first name node.
本发明的实施例还提供一种Hadoop分布式文件系统，第一名称节点每隔预设的第一时长，启用新的编辑日志；通过与第一名称节点连接的第二名称节点，每隔预设的第一时长，对第一名称节点上原有的镜像文件和原有的编辑日志备份，将备份的镜像文件和编辑日志合并，形成新的镜像文件，并发送回第一名称节点，替换第一名称节点原有的镜像文件。此外，还通过每隔预设的第二时长，第二名称节点对第一名称节点上的编辑日志备份，形成中间编辑日志；由于第二时长小于第一时长，使得编辑日志单独进行了更高频的备份，保存了更多的元数据序列，增强了Hadoop分布式文件系统中数据的安全性。The embodiment of the present invention also provides a Hadoop distributed file system, the first name node enables a new edit log every preset first time length; through the second name node connected to the first name node, every preset time Set the first duration, back up the original image file and the original edit log on the first name node, merge the backed up image file and edit log to form a new image file, and send it back to the first name node to replace the first name node. An original image file of the name node. In addition, the second name node backs up the edit log on the first name node every preset second time length to form an intermediate edit log; since the second time length is smaller than the first time length, the edit log is independently performed higher. Frequent backups save more metadata sequences and enhance the security of data in the Hadoop distributed file system.
本发明的实施例还提供Hadoop分布式文件系统的运行方法，如图5所示，包括：Embodiments of the present invention also provide a method for running a Hadoop distributed file system, as shown in FIG. 5 , including:
S100、启动第一名称节点，加载元数据文件，同时启动第二名称节点。S100. Start the first name node, load the metadata file, and start the second name node at the same time.
元数据文件包括镜像文件和编辑日志。Metadata files include image files and edit logs.
S110、每隔预设的第一时长，第一名称节点启动新的编辑日志。S110. Every preset first time period, the first name node starts a new edit log.
S120、每隔预设的第一时长，第二名称节点对第一名称节点上原有的镜像文件和原有的编辑日志备份，将备份的镜像文件与编辑日志合并，形成新的镜像文件，并发送回第一名称节点，替换第一名称节点原有的镜像文件。S120. Every preset first time period, the second name node backs up the original image file and the original edit log on the first name node, merges the backed up image file and the edit log to form a new image file, and Send it back to the first name node, replacing the original image file of the first name node.
S130、每隔预设的第二时长，第二名称节点对第一名称节点上的编辑日志备份，形成中间编辑日志。S130. The second name node backs up the edit log on the first name node every preset second time period to form an intermediate edit log.
其中，第二时长小于第一时长。第一名称节点与第二名称节点分别部署于不同主机。Wherein, the second duration is shorter than the first duration. The first name node and the second name node are respectively deployed on different hosts.
本发明的实施例提供的Hadoop分布式文件系统的运行方法与上述Hadoop分布式文件系统具有相同的有益效果，在此不再赘述。The operation method of the Hadoop distributed file system provided by the embodiment of the present invention has the same beneficial effects as the above-mentioned Hadoop distributed file system, which will not be repeated here.
本发明的实施例还提供Hadoop分布式文件系统的修复方法，如图6所示，包括：Embodiments of the present invention also provide a method for repairing a Hadoop distributed file system, as shown in FIG. 6 , including:
S200、停止受损的第一名称节点，同时停止第二名称节点。S200. Stop the damaged first name node, and stop the second name node at the same time.
S210、备份第二名称节点上的元数据文件。S210. Back up the metadata file on the second name node.
元数据文件包括第二名称节点最后一次从受损的第一名称节点上备份的镜像文件和编辑日志，以及最后一次形成的中间编辑日志。The metadata file includes the last backup image file and edit log of the second namenode from the compromised first namenode, and the last formed intermediate edit log.
可以理解的是，由于对镜像文件和编辑日志备份，和对编辑日志备份形成中间编辑日志的间隔时长不同，所以，最后一次从受损的第一名称节点上对镜像文件和编辑日志备份的时间和最后一次对编辑日志备份，形成中间编辑日志的时间不同。It is understandable that, since the interval between the backup of the image file and the edit log is different from that of the backup of the edit log to form the intermediate edit log, the last time the image file and the edit log were backed up from the damaged first name node. The time to form the intermediate edit log is different from the last backup of the edit log.
示例的，若预设的第一时长为1小时，预设的第二时长为15分钟，例如1点时，第一名称节点启动新的编辑日志b1；第二名称节点对第一名称节点上原有的镜像文件和原有的编辑日志备份，将备份的镜像文件和编辑日志合并，形成新的镜像文件a1，并发送回第一名称节点，替换第一名称节点原有的镜像文件，同时，第二名称节点对第一名称节点上新的编辑日志b1备份，形成中间编辑日志c1。For example, if the preset first duration is 1 hour, and the preset second duration is 15 minutes, for example, at 1 o'clock, the first name node starts a new edit log b1 ; backup the original image file and the original edit log, merge the backed-up image file and edit log to form a new image file a1 , and send it back to the first name node to replace the original image file of the first name node, At the same time, the second name node backs up the new edit log b1 on the first name node to form an intermediate edit log c1 .
若1点25分第一名称节点损坏，则第二名称节点上最后一次从受损的第一名称节点上备份的元数据文件指的是1点时备份的镜像文件a0和编辑日志b0，以及1点15分时对编辑日志b1′(此时，该编辑日志b1′存储的是1点到1点15分期间元数据的改动序列)备份，形成的中间编辑日志c1′。If the first namenode is damaged at 1:25, the last metadata file on the second namenode backed up from the damaged first namenode refers to the mirror file a0 and the edit log b0 backed up at 1:00 , and back up the edit log b1 ' at 1:15 (at this time, the edit log b1 ' stores the metadata change sequence from 1:00 to 1:15) to form an intermediate edit log c1 ' .
S220、准备部署新的第一名称节点的主机。S220. Prepare to deploy the host of the new first name node.
部署新的第一名称节点的主机与部署受损的第一名称节点、部署第二名称节点的主机均不同。The host on which the new first namenode is deployed is different from the host on which the compromised first namenode and the second namenode are deployed.
S230、配置主机的名称、IP地址、互信登录、运行环境和集群。S230, configure the name, IP address, mutual trust login, operating environment and cluster of the host.
S240、对新的第一名称节点执行格式化。S240. Format the new first name node.
S250、将备份的第二名称节点上的镜像文件、编辑日志和中间编辑日志合并，生成新的镜像文件，发送至新的第一名称节点。S250. Combine the mirror file, edit log, and intermediate edit log on the backup second name node to generate a new mirror file, and send it to the new first name node.
示例的，根据S210中的示例可知，第二名称节点上最后一次从受损的第一名称节点上备份的元数据文件指的是1点时备份的镜像文件a0和编辑日志b0，以及1点15分时对编辑日志b1′(此时，该编辑日志b1′存储的是1点到1点15分期间元数据的改动序列)备份，形成的中间编辑日志c1′。Illustratively, according to the example in S210, it can be known that the last metadata file on the second name node backed up from the damaged first name node refers to the mirror file a0 and the edit log b0 backed up at 1:00, and Back up the edit log b1 ' at 1:15 (at this time, the edit log b1 ' stores the metadata change sequence from 1:00 to 1:15) to form an intermediate edit log c1 '.
将该镜像文件a0和编辑日志b0、以及中间编辑日志c1′合并，相当于先将镜像文件a0和编辑日志b0合并，生成1点时对应镜像文件a1，再将1点时对应镜像文件a1和中间编辑日志c1′合并，再生成新的镜像文件，发送至新的第一名称节点。Merging the image file a0 with the edit log b0 and the intermediate edit log c1 ′ is equivalent to merging the image file a0 with the edit log b0 first, and generating 1 point corresponds to the image file a1 , and then 1 point When the corresponding image file a1 and the intermediate edit log c1 ′ are merged, a new image file is generated and sent to the new first name node.
S260、修改新的第一名称节点的元数据序号。S260. Modify the metadata sequence number of the new first name node.
可以理解的是，手动修改新的第一名称节点的元数据序号，使得该元数据序号可以与发送回的镜像文件中的元数据序列进行衔接，从而，在新的第一名称节点进行运行时，继续扩展整个元数据序列。It can be understood that the metadata sequence number of the new first name node is manually modified, so that the metadata sequence number can be connected with the metadata sequence in the image file sent back, so that when the new first name node runs , which continues to expand the entire metadata sequence.
可选地，在S260之后，如图7所示，Hadoop分布式文件系统的修复方法还包括：Optionally, after S260, as shown in FIG. 7, the method for repairing the Hadoop distributed file system further includes:
S270、启动新的第一名称节点和对应的Hadoop分布式文件系统。S270. Start a new first name node and a corresponding Hadoop distributed file system.
S280、执行分布式文件系统数据块检查。S280. Perform a distributed file system data block check.
若检查确认修复，则修复完成，若检测未修复，则再次执行S220～S260的步骤，重新进行修复。If the inspection confirms the repair, the repair is completed. If the inspection is not repaired, the steps of S220 to S260 are performed again to perform the repair again.
本发明的实施例还提供一种计算机设备，包括存储单元和处理单元；所述存储单元中存储可在所述处理单元上运行的计算机程序并存储结果；所述处理单元执行所述计算机程序时实现如上所述的Hadoop分布式文件系统的运行方法，和/或如上所述的Hadoop分布式文件系统的修复方法。An embodiment of the present invention also provides a computer device, including a storage unit and a processing unit; the storage unit stores a computer program that can be run on the processing unit and stores results; when the processing unit executes the computer program The above-mentioned operation method of the Hadoop distributed file system, and/or the above-mentioned repair method of the Hadoop distributed file system is realized.
本发明的实施例还一种计算机可读介质，其存储有计算机程序，所述计算机程序被处理器执行时实现如上所述的Hadoop分布式文件系统的运行方法，和/或如上所述的Hadoop分布式文件系统的修复方法。An embodiment of the present invention is also a computer-readable medium, which stores a computer program, and when the computer program is executed by a processor, implements the above-mentioned operation method of the Hadoop distributed file system, and/or the above-mentioned Hadoop Repair methods for distributed file systems.
以上所述，仅为本发明的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应以所述权利要求的保护范围为准。The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention. should be included within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.
Claims (10)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911056278.XACN110795404B (en) | 2019-10-31 | 2019-10-31 | Hadoop distributed file system and operation method and repair method thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911056278.XACN110795404B (en) | 2019-10-31 | 2019-10-31 | Hadoop distributed file system and operation method and repair method thereof |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110795404Atrue CN110795404A (en) | 2020-02-14 |
| CN110795404B CN110795404B (en) | 2023-04-07 |
Family
ID=69440772
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911056278.XAActiveCN110795404B (en) | 2019-10-31 | 2019-10-31 | Hadoop distributed file system and operation method and repair method thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110795404B (en) |
Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140019405A1 (en)* | 2012-07-13 | 2014-01-16 | Facebook Inc. | Automated failover of a metadata node in a distributed file system |
| US20140019495A1 (en)* | 2012-07-13 | 2014-01-16 | Facebook Inc. | Processing a file system operation in a distributed file system |
| CN103544045A (en)* | 2013-10-16 | 2014-01-29 | 南京大学镇江高新技术研究院 | HDFS-based virtual machine image storage system and construction method thereof |
| US20150074052A1 (en)* | 2012-10-30 | 2015-03-12 | Vekatachary Srinivasan | Method and system of stateless data replication in a distributed database system |
| CN104572357A (en)* | 2014-12-30 | 2015-04-29 | 清华大学 | Backup and recovery method for HDFS (Hadoop distributed filesystem) |
| US9367579B1 (en)* | 2005-02-23 | 2016-06-14 | Veritas Technologies Llc | System and method for maintaining a file change log within a distributed file system |
| CN107122238A (en)* | 2017-04-25 | 2017-09-01 | 郑州轻工业学院 | Efficient iterative Mechanism Design method based on Hadoop cloud Computational frame |
| CN107506260A (en)* | 2017-07-27 | 2017-12-22 | 南京南瑞集团公司 | A kind of dynamic division database incremental backup method |
| CN108804253A (en)* | 2017-05-02 | 2018-11-13 | 中国科学院高能物理研究所 | A kind of concurrent job backup method for mass data backup |
| CN109189480A (en)* | 2018-07-02 | 2019-01-11 | 新华三技术有限公司成都分公司 | File system starts method and device |
| CN109328344A (en)* | 2016-06-23 | 2019-02-12 | 电子湾有限公司 | File system image processing system |
| CN109407977A (en)* | 2018-09-25 | 2019-03-01 | 佛山科学技术学院 | A kind of big data distributed storage management method and system |
| CN110362381A (en)* | 2019-06-21 | 2019-10-22 | 深圳市汇川技术股份有限公司 | HDFS cluster High Availabitity dispositions method, system, equipment and storage medium |
- 2019
- 2019-10-31CNCN201911056278.XApatent/CN110795404B/enactiveActive
Patent Citations (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9367579B1 (en)* | 2005-02-23 | 2016-06-14 | Veritas Technologies Llc | System and method for maintaining a file change log within a distributed file system |
| US20140019495A1 (en)* | 2012-07-13 | 2014-01-16 | Facebook Inc. | Processing a file system operation in a distributed file system |
| US20140019405A1 (en)* | 2012-07-13 | 2014-01-16 | Facebook Inc. | Automated failover of a metadata node in a distributed file system |
| US20150074052A1 (en)* | 2012-10-30 | 2015-03-12 | Vekatachary Srinivasan | Method and system of stateless data replication in a distributed database system |
| CN103544045A (en)* | 2013-10-16 | 2014-01-29 | 南京大学镇江高新技术研究院 | HDFS-based virtual machine image storage system and construction method thereof |
| CN104572357A (en)* | 2014-12-30 | 2015-04-29 | 清华大学 | Backup and recovery method for HDFS (Hadoop distributed filesystem) |
| CN109328344A (en)* | 2016-06-23 | 2019-02-12 | 电子湾有限公司 | File system image processing system |
| CN107122238A (en)* | 2017-04-25 | 2017-09-01 | 郑州轻工业学院 | Efficient iterative Mechanism Design method based on Hadoop cloud Computational frame |
| CN108804253A (en)* | 2017-05-02 | 2018-11-13 | 中国科学院高能物理研究所 | A kind of concurrent job backup method for mass data backup |
| CN107506260A (en)* | 2017-07-27 | 2017-12-22 | 南京南瑞集团公司 | A kind of dynamic division database incremental backup method |
| CN109189480A (en)* | 2018-07-02 | 2019-01-11 | 新华三技术有限公司成都分公司 | File system starts method and device |
| CN109407977A (en)* | 2018-09-25 | 2019-03-01 | 佛山科学技术学院 | A kind of big data distributed storage management method and system |
| CN110362381A (en)* | 2019-06-21 | 2019-10-22 | 深圳市汇川技术股份有限公司 | HDFS cluster High Availabitity dispositions method, system, equipment and storage medium |
Non-Patent Citations (9)
| Title |
|---|
| BENJAMIN ERB 等: "Log Pruning in Distributed Event-sourced Systems", 《ACM》* |
| HILMI EGEMEN CIRITOGLU 等: "Investigation of Replication Factor for Performance Enhancement in the Hadoop Distributed File System", 《ACM》* |
| ZLINGH: "利用SecondaryNameNode文件恢复Namenode-实践可行", 《HTTPS://WWW.SHUZHIDUO.COM/A/GKZ12B12JR/》* |
| 不积小流无以成江河: "Secondary NameNode究竟是做什么的", 《HTTPS://WWW.ITDAAN.COM/BLOG/2017/06/09/48E32DF97ED6F538FAA80CCCB40F3A1D.HTML》* |
| 刘涌等: "关于Hadoop中HDFS的研究", 《电脑知识与技术》* |
| 王胜杰等: "一种Ceph分布式块存储的持续数据保护方法", 《网络安全技术与应用》* |
| 谢磊等: "一种基于LRFU缓存替换策略的HDFS客户端本地缓存设计与实现", 《计算机应用与软件》* |
| 陆永祯 等: "《SQL Server 2000数据库管理和程序开发一周通》", 30 April 2005* |
| 马建红等: "基于HDFS的创新知识云平台存储架构的研究与设计", 《计算机应用与软件》* |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110795404B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12260102B2 (en) | Distributed storage method and device | |
| US10956374B2 (en) | Data recovery method, apparatus, and system | |
| JP4304194B2 (en) | File system management apparatus and method | |
| CN112463450B (en) | Incremental backup management method, system, electronic equipment and storage medium | |
| CN105824723B (en) | The method and system that a kind of data to publicly-owned cloud storage account are backed up | |
| CN113934707B (en) | Cloud native database, database expansion method, database reduction method and device | |
| CN110928728A (en) | Virtual machine copying and switching method and system based on snapshot | |
| US20250190447A1 (en) | Replica expansion methods, apparatuses, and systems for distributed graph database | |
| CN110442645B (en) | Data indexing method and device | |
| CN113254394B (en) | Snapshot processing method, system, equipment and storage medium | |
| CN109753381B (en) | Continuous data protection method based on object storage | |
| CN112231150B (en) | Method and device for recovering fault database in database cluster | |
| CN114528260A (en) | File access request processing method, electronic equipment and computer program product | |
| CN112579550A (en) | Metadata information synchronization method and system of distributed file system | |
| CN113722157B (en) | A virtual machine data management method, apparatus, device and medium | |
| EP2372552B1 (en) | Automated relocation of in-use multi-site protected data storage | |
| CN107783826B (en) | A virtual machine migration method, device and system | |
| CN110825758B (en) | Transaction processing method and device | |
| CN113467719A (en) | Data writing method and device | |
| CN110795404B (en) | Hadoop distributed file system and operation method and repair method thereof | |
| CN112000850A (en) | Method, device, system and equipment for data processing | |
| CN112083885A (en) | Data migration method and device, electronic equipment and storage medium | |
| CN106354830B (en) | Method and device for data synchronization between database cluster nodes | |
| CN106407320B (en) | File processing method, device and system | |
| CN109241071A (en) | A kind of Android database upgrade method, apparatus and terminal |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |