CN112800019A - Data backup method and system based on Hadoop distributed file system - Google Patents
Data backup method and system based on Hadoop distributed file systemDownload PDFInfo
- Publication number
- CN112800019A CN112800019ACN202110233087.7ACN202110233087ACN112800019ACN 112800019 ACN112800019 ACN 112800019ACN 202110233087 ACN202110233087 ACN 202110233087ACN 112800019 ACN112800019 ACN 112800019A
- Authority
- CN
- China
- Prior art keywords
- hdfs
- file
- data
- folder
- client
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
- G06F11/1446—Point-in-time backing up or restoration of persistent data
- G06F11/1448—Management of the data involved in backup or backup restore
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
- G06F16/134—Distributed indices
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/44—Program or device authentication
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/64—Protecting data integrity, e.g. using checksums, certificates or signatures
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Computer Hardware Design (AREA)
- Software Systems (AREA)
- Quality & Reliability (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Storage Device Security (AREA)
Abstract
Translated fromChinese
Description
Technical Field
The invention relates to a data backup method and system, in particular to a data backup method and system based on a Hadoop distributed file system.
Background
The big data exists in the fields of physics, biology, environmental ecology and the like and the industries of military, finance, communication and the like for a long time, but attracts people's attention due to the development of the internet and the information industry in recent years. With the rapid development and popularization and application of computers and information technologies, big data shows the advantages of the computers and the information technologies, the scale of an industry application system is rapidly enlarged, and data generated by industry application is explosively increased.
Hadoop realizes a Distributed File System (HDFS) with high fault tolerance, and is used for solving the problems of low hardware, capability of building a telescopic super-large cluster, realization of storage and access of large data volume and the like. With the technology becoming more mature and the system becoming more stable, manufacturers such as Cloudera and hotsonnorks successively put forward big data solutions based on the Hadoop architecture, and more enterprises use Hadoop as the basic platform of the company. In the case of a Hadoop cluster in a catastrophic situation, a method for protecting and recovering data from a remote place and an external place is lacked in the market. The invention aims to deal with the occurrence of the cluster disaster.
Disclosure of Invention
The purpose of the invention is as follows: the invention aims to provide a method and a system for efficiently and safely backing up data by using a third-party medium based on a Hadoop distributed file system.
The technical scheme is as follows: according to the data backup method based on the Hadoop distributed file system, the folder is backed up in a snapshot mode through the HDFS client, the time point snapshot of the folder is generated through the client, and the data in the folder is stored to an external storage medium.
The method comprises the following steps:
(1) creating an HDFS client;
(2) creating a folder snapshot: reading snapshot information of a current cluster through an HDFS client, generating a read-only time point snapshot for a backup folder, and not copying any data block;
(3) data backup: establishing connection between an external storage medium and an HDFS file system, reading file contents, and writing the file contents into the external storage medium;
(4) metadata element backup: and reading the metadata of the file/folder through the HDFS client, connecting a remote storage index database, and writing the metadata into the file index database.
The step (1) comprises the following steps:
(11) downloading an HDFS service client and user credentials on a Hadoop management system interface, acquiring HDFS configuration information and Kerberos authentication information, and placing configuration on an agent node;
(12) and issuing a backup job through the agent framework, reading the acquired HDFS configuration and Kerberos authentication in the job, and creating the HDFS client.
In the step (2), if the folder is an incremental backup, comparing the previous snapshot with the next snapshot, and acquiring the modification information of the folder.
The external storage medium of the step (2) is a remote storage medium.
The metadata of the step (4) includes attribute information of the distributed file and storage location information of the distributed file in the external storage medium.
Further comprising the steps of: when a file in the system is damaged or lost, the HDFS client selectively restores a part of the file by acquiring the position information of the file in the external storage medium without restoring the whole snapshot.
The data backup system based on the Hadoop distributed file system comprises an HDFS system and a storage server connected with the HDFS system, wherein the storage server comprises a storage medium and a file index database; the storage medium is used for storing system file data, and the file index database is used for storing system file metadata; the HDFS system further comprises a backup server, wherein the backup proxy node is provided with a proxy service, the server downloads Hadoop configuration and Kerberos user authentication, an HDFS client is established through the proxy service, and the HDFS client interacts with the Hadoop cluster.
Has the advantages that: compared with the prior art, the invention has the following remarkable advantages:
the method for protecting the folder data under the Hadoop platform builds an agent environment at a remote node by downloading corresponding configuration, supports Kerberos authentication, remotely connects a Hadoop cluster through a Hadoop API, does not influence the service, improves the backup efficiency and reliability, and realizes the backup protection of the data in a distributed file system (HDFS). The invention can improve the safety of data in the HDFS, prevent Hadoop clusters from disasters, automatically and quickly recover system data, and protect the integrity and consistency of company data. Backup of the entire distributed file system, or portions of the distributed file system, is supported.
Drawings
FIG. 1 is a schematic flow chart of the present invention.
Detailed Description
The technical scheme of the invention is further explained by combining the attached drawings.
The Hadoop Distributed File System (HDFS) of the present invention comprises: backup proxy node, storage server. The backup server is an independent machine, can be a physical machine or a virtual machine, an agent service is installed on a backup agent node, Hadoop configuration and Kerberos user authentication are downloaded, an HDFS client is established through the agent service, and interaction is carried out between the HDFS client and a Hadoop cluster. The storage server comprises a storage medium and a file index database. The storage medium is used for storing system file data, and the file index database is used for storing system file metadata.
The HDFS cluster is composed of one Namenode and a certain number of dataodes. The Namenode is a central server responsible for managing the namespace (namespace) of the file system and client access to files. The dataode in a cluster is typically one node and is responsible for managing the storage on the node where it resides. HDFS has a namespace of the file system on which users can store data in the form of files. Internally, a file is divided into one or more data blocks, which are stored on a set of dataodes. The Namenode performs namespace operations for the file system, such as opening, closing, renaming files or directories. It is also responsible for determining the mapping of data blocks to specific dataode nodes. The dataode is responsible for processing read and write requests of the file system client. And creating, deleting and copying the data blocks under the unified scheduling of the Namenode.
As shown in fig. 1, the method of the present invention specifically includes the following steps:
1. the proxy service is installed. Downloading the proxy service installation package on the proxy node, and configuring Hadoop client parameters, wherein the steps comprise: download Hadoop configuration, Kerberos authentication, etc.
2. And starting the proxy service, reading Hadoop client parameters, and creating an HDFS client.
3. And connecting the Hadoop cluster through the HDFS client to interact. The method comprises the steps of obtaining snapshot list information and file metadata information of the distributed file system.
4. A snapshot is created. The HDFS client is connected with nodes of NameNode in the Hadoop cluster, creates snapshots for the folders, opens up a new space in the system for storing modified files, records the positions of all file blocks of the folders at a certain time point, records information such as lists and sizes of the file blocks and does not copy Chinese file block data of the DataNode. The snapshot backup is to copy the position information of the file block in time, and the operation is instantaneous, so that the data cannot be modified; and the subsequent data backup reads the file data to an external medium through the position information of the file block in the snapshot.
5. And backing up the data. The HDFS client is connected with the Hadoop cluster NameNode node, a file block list (DataNode information) of files in the snapshot is obtained, then the HDFS client is connected with the DataNode, the backup file data is read, and the backup file data is backed up to an external storage medium.
6. And backing up the metadata. And the HDFS client is connected with the Hadoop cluster NameNode node, acquires the metadata of the snapshot directory and the storage position of the file in the external storage medium, and stores the file in a file index library of an external storage server.
7. Recovery after data corruption, loss, or mishandling. The HDFS client selectively restores a part of files by acquiring the position information of the files in the external storage medium without restoring all snapshots, so that the problem of low data restoration speed is solved. The data in the external medium is directly read and written into the HDFS folder, a folder is usually newly built, the data is directly restored to the newly built path, and if the newly built path does not exist, the original data can be covered.
Claims (8)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110233087.7ACN112800019A (en) | 2021-03-03 | 2021-03-03 | Data backup method and system based on Hadoop distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110233087.7ACN112800019A (en) | 2021-03-03 | 2021-03-03 | Data backup method and system based on Hadoop distributed file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112800019Atrue CN112800019A (en) | 2021-05-14 |
Family
ID=75816340
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110233087.7APendingCN112800019A (en) | 2021-03-03 | 2021-03-03 | Data backup method and system based on Hadoop distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112800019A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113434606A (en)* | 2021-06-30 | 2021-09-24 | 青岛海尔科技有限公司 | Data import method, device, equipment and medium |
| CN113821490A (en)* | 2021-08-24 | 2021-12-21 | 济南浪潮数据技术有限公司 | A data synchronization method and device |
| CN114153842A (en)* | 2021-11-12 | 2022-03-08 | 广东广信通信服务有限公司 | Cross-platform data processing method, system, equipment and medium |
| CN114185484A (en)* | 2021-11-04 | 2022-03-15 | 福建升腾资讯有限公司 | Method, device, equipment and medium for clustering document storage |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104714858A (en)* | 2013-12-13 | 2015-06-17 | 中国移动通信集团公司 | Data backup method, data recovery method and device |
| CN112214357A (en)* | 2020-10-30 | 2021-01-12 | 上海爱数信息技术股份有限公司 | HDFS data backup and recovery system and backup and recovery method |
- 2021
- 2021-03-03CNCN202110233087.7Apatent/CN112800019A/enactivePending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104714858A (en)* | 2013-12-13 | 2015-06-17 | 中国移动通信集团公司 | Data backup method, data recovery method and device |
| CN112214357A (en)* | 2020-10-30 | 2021-01-12 | 上海爱数信息技术股份有限公司 | HDFS data backup and recovery system and backup and recovery method |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113434606A (en)* | 2021-06-30 | 2021-09-24 | 青岛海尔科技有限公司 | Data import method, device, equipment and medium |
| CN113821490A (en)* | 2021-08-24 | 2021-12-21 | 济南浪潮数据技术有限公司 | A data synchronization method and device |
| CN114185484A (en)* | 2021-11-04 | 2022-03-15 | 福建升腾资讯有限公司 | Method, device, equipment and medium for clustering document storage |
| CN114153842A (en)* | 2021-11-12 | 2022-03-08 | 广东广信通信服务有限公司 | Cross-platform data processing method, system, equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12130708B2 (en) | Cloud-based air-gapped data storage management system | |
| US11093336B2 (en) | Browsing data stored in a backup format | |
| US20210263888A1 (en) | User-centric interfaces for information management systems | |
| US20230205648A1 (en) | Enhanced file indexing, live browsing, and restoring of backup copies of virtual machines and/or file systems by populating and tracking a cache storage area and a backup index | |
| US7953945B2 (en) | System and method for providing a backup/restore interface for third party HSM clients | |
| JP5247202B2 (en) | Read / write implementation on top of backup data, multi-version control file system | |
| US9632713B2 (en) | Secondary storage editor | |
| CN104040481B (en) | Method and system for fusing, storing and retrieving incremental backup data | |
| CN102597983B (en) | Backup using metadata virtual hard drives and differential virtual hard drives | |
| WO2021263224A1 (en) | Incremental backup to object store | |
| JP5145098B2 (en) | System and method for directly exporting data from a deduplication storage device to a non-deduplication storage device | |
| US8688645B2 (en) | Incremental restore of data between storage systems having dissimilar storage operating systems associated therewith | |
| JP5731000B2 (en) | Method and system for performing individual restore of a database from a differential backup | |
| EP3796174B1 (en) | Restoring a database using a fully hydrated backup | |
| US8433863B1 (en) | Hybrid method for incremental backup of structured and unstructured files | |
| US8874517B2 (en) | Summarizing file system operations with a file system journal | |
| CN112800019A (en) | Data backup method and system based on Hadoop distributed file system | |
| US20160162364A1 (en) | Secondary storage pruning | |
| US9043280B1 (en) | System and method to repair file system metadata | |
| US10387381B1 (en) | Data management using an open standard file system interface to a storage gateway | |
| CN102388369B (en) | Lifecycle of granular application data from a single backup | |
| US20230153010A1 (en) | Pruning data segments stored in cloud storage to reclaim cloud storage space | |
| US10628298B1 (en) | Resumable garbage collection | |
| WO2006107394A2 (en) | Production server to data protection server mapping | |
| CN114026545A (en) | Snapshots for replication at any point in time |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20210514 |