Background technique
With internet, the fast development of Web2.0, Web application is created that tens billion of small documents；People upload magnanimityPhoto, video, music, Facebook increases billions of contents newly daily, and people send hundreds billion of envelope Emails daily.It is counted according to IDC, future data between 10 years will increase by 44 times, will be added to 35ZB to the year two thousand twenty global metadata.Mass data collectionCommon feature mainly include 3 points: one is be more than 80% data being non-structured；The second is data volume is continuing notIncrease disconnectedly；The third is these data need to store for a long time, and non-hot data can be also accessed randomly.
In face of so huge data volume, block storage (SAN) and the file storage (NAS) for only having PB grades of extended capabilities are aobviousIt is helpless to obtain some: a LUN capacity of usual block storage (SAN) only counts TB.Under single file system optimal behavior patternThe quantity of documents of support is usually only in million ranks.People need a kind of storage system of completely new framework, this storage systemHigh scalability is needed to have, can satisfy the demand of extension of the people to memory capacity TB to EB scale.
Object storage is come into being, and is a kind of object-based storage equipment, and it is special to have both SAN high speed directly access diskThe Distributed sharing feature of point and NAS, has intelligence, ability of self-management, real by web services agreement (such as: REST, SOAP)The read-write of existing object and the access of storage resource.Object storage in 2002 is used for the financial service of government regulations mandate, healthThus the data filing scene of the industries such as medical treatment, object storage have the gene of backup filing；Amazon publication in 2006REST, SOAP access interface that AWS, S3 are serviced and its used become the fact that object storage standard.Amazon S3 is successfully pairAs cloud service gene is injected in storage.
Object storage system includes two kinds of data descriptions: container (Bucket), object (Object).Container and object haveOne globally unique ID.Object storage manages all data using compressed structure, and user or application pass through access code(AccessKey) after authenticating, it is only necessary to container and object and relevant data (Data), metadata can be accessed according to ID(metadata) and object properties (Attribute).
Data can be fragmented when storage system is written, to realize concurrent reading and writing.In different object storage systems,The algorithm of fragment is different, and metadata and permission control information definition and storage format it is also different.So when needingStorage system is replaced, when migrating to data, it is necessary to file be read out from former object storage system first, then be written newStorage system, rather than simply to hard disk carry out data copy migration.
Using by hand before the method for Data Migration is migrated before specifically including that system switching by tool at present, system switchesTyping.Wherein:
Referred to before system switching by tool migration: before system switching, developing a program in former storage systemHistorical data is extracted, is converted, and is loaded into new system.This method is that Data Migration is main, it is most efficiently square to be alsoMethod.But these tools require specific development, and there are various limitations, cannot function as an independent software product.This method is the most universal and efficiently method of application, but to the more demanding of initial data.
Referred to before system switching using manual typing: before system switching, organizing related operation maintenance personnel by storing data handWork export, is then introduced into new system.Although this method does not require initial data too much, the manpower of consumption,Material resources are bigger, while error rate is also relatively high.It is primarily adapted for use in the data that can not be transformed into new system and new system enablesThe occasion of Shi Bixu and the data that former storage system can not provide, can be used as the useful supplement of first method.
As it can be seen that the method for above data migration has certain defect, the method for Data Migration need further to be developed andIt is perfect.
Summary of the invention
The technical problem to be solved by the present invention is to aiming at the above shortcomings existing in the prior art, provide it is a kind of based on pairAs the method for the Data Migration of storage and its system of the Data Migration stored based on object, the data that should be stored based on object are movedThe method of shifting can effectively shield the difference of former storage system and purpose storage system, have under different operating system removablePlant property, it is easy to use, versatility is high.
Solve the method that technical solution used by present invention problem is the Data Migration based on object storage, packetInclude step:
Step S1) according to data-level, the data are extracted from former storage system, and by the user of the data, holdDevice, object and corresponding rules of competence information generate corresponding property file；
Step S2) the parsing property file, the instruction in the property file is executed, it will according to the data-level of settingNew storage system, and the user for the data, container, the corresponding rules of competence of object setting are written in the data.
Preferably, step S1) it specifically includes:
Step S11) first order catalogue is established, by all meta-data preservations to be migrated under the first order catalogue；
Step S12) second level catalogue is established, a file is established with the ID of user, by user account, password, quotaInformation is converted to user attribute file；
Step S13) third level catalogue is established, under the second level catalogue of corresponding user, a text is established with the ID of containerPart folder, is converted to a container attribute file for the authority information of container and metadata information；
Step S14) fourth stage catalogue is established, it reads all objects under corresponding container and is saved in the third level of the containerUnder catalogue, the authority information of object and metadata information are converted into object properties file.
Preferably, step S2) it specifically includes:
Step S21) all user attribute files of traversal second level catalogue, it is established and original according to the user attribute fileThe identical user account of storage system, password and quota；
Step S22) name that third level catalogue All Files press from both sides is traversed, according to the container attribute file under the userCorresponding container is established, and corresponding permission is arranged to the container；
Step S23) traversal all object properties files of fourth stage catalogue, according to the object properties file under the userCorresponding object is established, and corresponding permission is arranged to the object.
Preferably, the content of the user attribute file, the container attribute file and the object properties file is equalFor xml formatted data.
Preferably, the data extracted from former storage system, and correspond to the user property text of the dataPart, the container attribute file, the object properties file are temporarily stored into temporary memory space.
A kind of system of the Data Migration based on object storage, including data lead-out unit, Date Conversion Unit and dataImport unit, in which:
The data lead-out unit, for extracting the data from former storage system according to data-level；
The Date Conversion Unit is used for user, container, object and corresponding rules of competence information in the dataGenerate corresponding property file；
The data import unit executes the instruction in the property file for parsing the property file, according to settingNew storage system is written in the data by fixed data-level, and corresponding for user, container, object setting in the dataThe rules of competence.
Preferably, the Date Conversion Unit includes user information conversion module, information of container conversion module and objectInfo conversion module, in which:
The user information conversion module is used for user in the data and the generation pair of corresponding rules of competence informationThe property file answered, the property file are user attribute file；
The information of container conversion module is used for container in the data and the generation pair of corresponding rules of competence informationThe property file answered, the property file are container attribute file；
The object information conversion module is used for object in the data and the generation pair of corresponding rules of competence informationThe property file answered, the property file are object properties file.
Preferably, the data import unit includes user information parsing module, information of container parsing module and objectInformation analysis module, in which:
The user information parsing module, for parsing user and corresponding permission from the user attribute fileRule；
The information of container parsing module, for parsing container and corresponding permission from the container attribute fileRule；
The object information parsing module, for parsing object and corresponding permission from the object properties fileRule.
Preferably, the content of the user attribute file, the container attribute file and the object properties file is equalFor xml formatted data.
It preferably, further include temporary memory space, the temporary memory space is taken out from former storage system for savingThe data taken, and correspond to the user attribute file, the container attribute file, the object category of the dataProperty file.
The beneficial effects of the present invention are:
The method of the Data Migration based on object storage can effectively shield former storage system and purpose storage systemDifference has portability under different operating system, only needs the operation maintenance personnel of storage system that can carry out in the methodData Migration has the characteristics such as easy to use, versatility is high, participates in without user, is transparent to user；
Correspondingly, data should be may be implemented between two storage mediums based on the system for the Data Migration that object storesConvenient migration guarantees the accuracy of data.
Specific embodiment
Technical solution in order to enable those skilled in the art to better understand the present invention, with reference to the accompanying drawing and specific embodiment partyFormula makees into one the method for the Data Migration stored the present invention is based on object and its system of the Data Migration based on object storageStep detailed description.
The advantages of object storage system (Object-Based Storage System) combines NAS and SAN, has simultaneouslyThere is the high speed of SAN directly to access and the advantages such as the data sharing of NAS, provides high reliability, professional platform independence and limitless volumesWith the storage architecture of the data sharing of the safety of scalability.
In the present embodiment, be arranged with specific reference to the property of object-based storage storage element data export level,Derived content and data is needed to import the method etc. of new system.Under normal conditions, the composition member of Object-based storage systemElement includes: object (Object), container (bucket), metadata (meta data) and the rules of competence (ACL), in which:
Object (Object): object is the basic unit that data store in system, and an object is with URI (UniformResource Identifier, i.e. uniform resource identifier), data content and several attributes are to the data structure pair of compositionThe data cell that user data is recorded.The URI of all objects is globally unique, can uniquely navigate to the object according to URI.And file or block are used in traditional storage system as basic storage cell, also need to track always in block storage system beEach piece of attribute in system, object safeguard the attribute of oneself by communicating with storage system.
Container (bucket): the logical space of storage object in object storage system.Object must belong to a container；HoldDevice is used as object in container totally to carry out counting the entrance with access control simultaneously.Container with URI and several attributes to composition,The URI of all containers is globally unique, and URI can be navigated to uniquely with the container.
Metadata (meta data): the data of description container or object mainly describe container or object properties.TheseAttribute information can define RAID parameter, data distribution and service quality file-based etc..
The rules of competence (ACL): object and container support five class permissions respectively, as shown in table 1:
The permission of table 1 object and container
In object-based data-storage system, as shown in Figure 1, object storage organization is that have two-stage mesh under a userDirectory structures.It uses compressed structure (second-level directory) to manage all data, to being stored in the resource pool of multiple objects, firstGrade is container (container A, container B etc.), and the second level is object (object a, object b, object c etc.).
The method of Data Migration based on object storage in the present embodiment is totally divided into two steps, the first step be by metadata andPermission exports to file from former storage system, and new storage system is written in derived file by second step, and to the power of metadataLimit is configured.
In data migration process, different memory spaces is arranged in data export and data importing, and data are stored from originalSpace export, is directed into new memory space.As shown in Fig. 2, should be based on the specific steps of the method for the Data Migration that object storesIt is as follows:
Step S1) according to data-level, extract data from former storage system, and by the user of data, container, object withAnd corresponding rules of competence information generates corresponding property file.
Extract data from former storage system, using former storage system interface read user, container, object basic letterBreath, and the user of data, container, object and corresponding rules of competence information are generated into corresponding property file.Usual situationUnder, the interface of different storage system encapsulation is different, including reading and writing of files, newly-built user, checks permission etc..In the stepIn, data are exported from former storage system, and the metadata of user, container and object and authority information are converted into xml format textPart includes executable instruction in xml formatted file.Specifically include following sub-step:
Step S11) first order catalogue is established, all metadata to be migrated will be stored under the catalogue.
In this step, first order catalogue is only used for the place of specified data export storage or system, user can basesActual conditions flexibly define.
Step S12) second level catalogue is established, a file is established with the ID of user, by user account, password, quotaInformation exports as a file.
In this step, second level catalogue is established, establishes a file with the account ID of user；Then by user'sID, password, quota information export as a user attribute file, and name is that the name of user adds " .info ", and the content of file isThe data of xml format.
The format information memory of user is such as:
<user>
<command>create user</command>
<ID>ACCOUNT_TEST</ID>
<PublicKey>********</PublicKey>
<SecretKey>****</SecretKey>
<Quota>20480M</Quota>
</user>
Step S13) third level catalogue is established, under the second level catalogue of corresponding user, a text is established with the ID of containerPart folder, exports as a file for the authority information of container and metadata information.
In this step, third level catalogue is established under the second level catalogue of corresponding user, is built with the ID of container bucketFound a file；Then the authority information of container bucket and metadata information are exported as into a container attribute file, textThe name of part is that the ID of container adds " .acl ", and content is the data of xml format.This document illustrates the container with owner labelOwning user.
The format information memory of container is such as:
<bucket>
<command>create a bucket</command>
<Owner><ID>ACCOUNT_TEST</ID><DisplayName>admin</DisplayName></Owner>
<Grant><ID>ACCOUNT_TEST</ID><Permission>FULL_CONTROL</Permission></Grant>
</bucket>
Wherein<grant><iD>aCCOUNT_TEST</ID><permission>fULL_CONTROL</Permission></Grant>expression is configured each permission of container, and FULL_CONTROL indicates ACCOUNT_TEST user to the containerBucket possesses whole controls.
Step S14) fourth stage catalogue is established, it reads all objects under corresponding container and is saved in the third level of the containerUnder catalogue, the authority information of object and metadata information are exported as into a file.
In this step, fourth stage catalogue is established.All objects under corresponding container are read, the third of the container is saved inUnder grade catalogue；Then the authority information of object and metadata information are exported as into an object properties file, the name of file isThe ID of object adds " .acl ", and content is the data of xml format.Every an object corresponds to an object property file, this document ownerLabel illustrates the owning user of the object.
The format information memory of object is such as:
<object>
<command>create a object</command>
<Owner><ID>ACCOUNT_TEST</ID><DisplayName>admin</DisplayName></Owner>
<Grant><ID>ACCOUNT_TEST</ID><Permission>FULL_CONTROL</Permission></Grant>
</object>
Wherein<grant><iD>aCCOUNT_TEST</ID><permission>fULL_CONTROL</Permission></Grant>expression is configured each permission of object, and FULL_CONTROL indicates ACCOUNT_TEST user to the objectObject possesses whole controls.
By the above sub-step, i.e., the data of different platform are converted, are indicated by available xml formatted file.It is thereinObject information element is nested in information of container element, and information of container element is nested in user profile element.Finally, derivedFile directory is illustrated in fig. 2 shown below.
Step S2) parsing property file, the instruction in property file is executed, is write data into according to the data-level of settingNew storage system, and the user for data, container, the corresponding rules of competence of object setting.
In this step, the xml data file generated in step S1) is parsed, instruction therein is then executed, will be based onNew storage system is written in the data of object storage.Xml formatted file is parsed, derived data are imported into new storage system and are hadBody includes following sub-step:
Step S21) all user attribute files of traversal second level catalogue, it is established according to user attribute file and is stored with formerThe identical user account of system, password and quota.
In this step, all user attribute files of traversal second level catalogue establish the use as former storage systemFamily account, password and quota.
Step S22) name that third level catalogue All Files press from both sides is traversed, it is established under the user according to container attribute fileCorresponding container, and corresponding permission is arranged to the container.
In this step, the name of traversal third level catalogue All Files folder, i.e., the name of all container bucket, at thisCorresponding container bucket is established under user；Then, all container attribute files of traversal third level catalogue, i.e. container bucketAuthority information file parses the xml data of this document, and corresponding permission is arranged to container bucket.
Step S23) traversal all object properties files of fourth stage catalogue, it is established under the user according to object properties fileCorresponding object, and corresponding permission is arranged to the object.
Fourth stage catalogue is the last layer catalogue, only object (i.e. final file).In this step, the fourth stage is traversedObject object is written under corresponding user and container bucket all object properties files of catalogue；Then, traversal theAll object properties files of level Four catalogue, i.e. object object authority information file, parse the xml data of this document, to thisCorresponding permission is arranged in object object.
So far, Data Migration has been fully completed.Since object stores compatible Amazon S3 interface, so user is without moreChange the program code of oneself business, it is only necessary to which the IP for accessing former storage system and port are changed to IP and the port of new storage system?.
In the method for the Data Migration based on object storage, the data extracted from former storage system, and it is correspondingThe user attribute file of data, container attribute file, object properties file be temporarily stored into temporary memory space.As shown in figure 3, baseThe bibliographic structure derived from the data of object storage is level Four, for the content in corresponding property file, first order catalogue withoutIt is so-called, only refer to the current derived place to be stored of data；Second level catalogue is the directory attribute of user, third level catalogueIt is container attribute, fourth stage catalogue is object properties.
In the method for the Data Migration of the present embodiment, based on object storage Data Migration, introduce classification storage and it is rightMigration mechanism is answered, the specific Data Migration rule for reading with the process of migrating data and replacing object storage system are providedThen: data first being exported to the space temporarily stored from former storage system, then data are imported into new storage system.Due to there is a property file of corresponding permission and metadata under every first class catalogue, successively it is read out according to directory levelIt is imported with corresponding migration, the correctness of data can be effectively ensured.
The method of Data Migration provided by the invention based on object storage, to base when can be used for replacing object storage systemIt is migrated in the data of object.The method of the Data Migration based on object storage can effectively shield former storage system and meshStorage system difference, under different operating system have portability, only need the operation maintenance personnel of storage system according to thisMethod can carry out Data Migration, have the characteristics such as easy to use, versatility is high, participates in without user, is transparent to user.
Correspondingly, as another aspect of the present invention, as shown in figure 4, a kind of Data Migration based on object storage isSystem, including data lead-out unit 1, Date Conversion Unit 2 and data import unit 3, in which:
Data lead-out unit 1, for extracting data from former storage system according to data-level；
Date Conversion Unit 2 is used for user, container, object and the generation pair of corresponding rules of competence information in dataThe property file answered；
Data import unit 3 executes the instruction in property file, according to the data level of setting for parsing property fileNew storage system is not write data into, and sets the corresponding rules of competence for user, container, object in data.
Preferably, Date Conversion Unit 2 includes user information conversion module 21, information of container conversion module 22 and objectInfo conversion module 23, in which:
User information conversion module 21, for user in data and corresponding rules of competence information to be generated corresponding categoryProperty file, the property file be user attribute file；
Information of container conversion module 22, for container in data and corresponding rules of competence information to be generated corresponding categoryProperty file, the property file be container attribute file；
Object information conversion module 23, for object in data and corresponding rules of competence information to be generated corresponding categoryProperty file, the property file be object properties file.
Wherein, data import unit 3 includes user information parsing module 31, information of container parsing module 32 and object informationParsing module 33, in which:
User information parsing module 31, for parsing user and the corresponding rules of competence from user attribute file；
Information of container parsing module 32, for parsing container and the corresponding rules of competence from container property file；
Object information parsing module 33, for parsing object and the corresponding rules of competence from object properties file.
Preferably, the content of user attribute file, container attribute file and object properties file is xml format numberAccording to.
It further, should further include temporary memory space based on the system for the Data Migration that object stores, interim storage is emptyBetween for saving the data extracted from former storage system, and correspond to the user attribute file of data, container attribute file,Object properties file.
The system of the Data Migration based on object storage may be implemented convenient between two storage mediums of data and moveIt moves, guarantees the accuracy of data.
It is understood that the principle that embodiment of above is intended to be merely illustrative of the present and the exemplary implementation that usesMode, however the present invention is not limited thereto.For those skilled in the art, essence of the invention is not being departed fromIn the case where mind and essence, various changes and modifications can be made therein, these variations and modifications are also considered as protection scope of the present invention.