Summary of the invention
The embodiment of the present invention provides a kind of date storage method, server and system, for storing client by fileRealize object-based storage.
In a first aspect, providing a kind of date storage method, it is applied to data-storage system, the data-storage system includesDocument storage server, protocol conversion function library and distributed file system Ceph cluster, the Ceph cluster include at least oneA Ceph server, any one Ceph server is used for storage object data at least one described Ceph server, describedMethod includes:
The document storage server receives the storage file request that Network File System NFS client is sent；It is described to depositStoring up file request includes the first file data packet stored based on file；
The document storage server parses the storage file request, obtains the first file data packet；
The document storage server calls the protocol conversion function library that the first file data packet is converted into theAn object data packet；The first object data packet is the data packet stored based on object；
The document storage server calls the protocol conversion function library that the first object data packet is stored in instituteIt states in Ceph cluster.
Optionally, the method also includes:
The document storage server receives the reading file request that the NFS client is sent；The reading file is askedIt asks for reading the object data stored in the Ceph cluster；
The document storage server calls the protocol conversion function library according to the reading file request from describedThe object data to be read is searched in Ceph cluster, and generates the second object data packet；
The document storage server calls the protocol conversion function library that the second object data packet is converted into theTwo file data packets；
The second file data packet is sent to the NFS client by the document storage server.
Optionally, the method also includes:
The document storage server receives configuration file, and the configuration file is for configuring the document storage serverThe Ceph server in the Ceph cluster that can be accessed, and for configuring described in the document storage server callingThe path of protocol conversion function library；
The document storage server parses the configuration file, and constructs file according to the configuration file and store roadDiameter；Wherein, file store path described in the NFS client carry.
Second aspect, provides a kind of document storage server, and this document storage server includes:
Receiving unit, for receiving the storage file request of Network File System NFS client transmission；The storage fileRequest includes the first file data packet stored based on file；
Resolution unit obtains the first file data packet for parsing the storage file request；
The first file data packet is converted into first pair for invocation protocol transfer function library by function library call unitImage data packet；The first object data packet is the data packet stored based on object；
The function library call unit is also used to that the protocol conversion function library is called to deposit the first object data packetStorage is in distributed file system Ceph cluster；The Ceph cluster includes at least one Ceph server, it is described at least oneAny one Ceph server is used for storage object data in Ceph server.
Optionally, the server further includes transmission unit；
The receiving unit is also used to receive the reading file request that the NFS client is sent；The reading file is askedIt asks for reading the object data stored in the Ceph cluster；
The function library call unit, be also used to call the protocol conversion function library according to the reading file request fromThe object data to be read is searched in the Ceph cluster, and generates the second object data packet；And the agreement is called to turnIt changes function library and the second object data packet is converted into the second file data packet；
The transmission unit, for the second file data packet to be sent to the NFS client.
Optionally,
The receiving unit is also used to receive configuration file, and the configuration file is for configuring the file storage serviceThe Ceph server in the Ceph cluster that device can access, and the institute called for configuring the document storage serverState the path of protocol conversion function library；
The resolution unit is also used to parse the configuration file, and constructs file according to the configuration file and store roadDiameter；Wherein, file store path described in the NFS client carry.
The third aspect provides a kind of data-storage system, including document storage server, protocol conversion function library and distributionFormula file system Ceph cluster, the Ceph cluster include at least one Ceph server, at least one described Ceph serverIn any one Ceph server be used for storage object data；
The document storage server, for receiving the storage file request of Network File System NFS client transmission, andThe storage file request is parsed, the first file data packet is obtained；；The storage file request includes being carried out based on fileFirst file data packet of storage；
The protocol conversion function library, for the first file data packet to be converted into the first object data packet；It is describedFirst object data packet is the data packet stored based on object, and the first object data packet is stored in the CephIn cluster；
The Ceph cluster, for storing the first object data packet.
Optionally,
The document storage server is also used to receive the reading file request that the NFS client is sent；The readingFile request is for reading the object data stored in the Ceph cluster；
The protocol conversion function library is also used to be searched from the Ceph cluster according to the reading file request and be wantedThe object data of reading, and generate the second object data packet；And call the protocol conversion function library by second objectData packet is converted into the second file data packet；
The document storage server is also used to the second file data packet being sent to the NFS client.
Fourth aspect provides a kind of computer installation, and described device includes at least one processor, and the processor is used forThe step of date storage method such as first aspect offer is realized when executing the computer program stored in memory.
5th aspect, provides a kind of computer readable storage medium, is stored thereon with computer program, the computer journeyThe step of date storage method such as first aspect offer is realized when sequence is executed by processor.
In embodiments of the present invention, the document storage server in data-storage system is receiving file storage requestAfterwards, then the file data packet that file storage request includes can be converted into depositing based on object with invocation protocol transfer function libraryThe object data packet of storage, and in the Ceph cluster for including to data-storage system by the object data packet storage after conversion, with realityThe effect that client realizes object storage is now stored by file.It wherein, is object data due to being stored in Ceph cluster, togetherThe client that sample can be stored by object is read and identification, in this way, then file data and object data can be made to be shared, andAnd due to object storage can self the advantages of so that the maintenance of data is more simple.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present inventionIn attached drawing, technical scheme in the embodiment of the invention is clearly and completely described.
The technical background of the embodiment of the present invention is described below.
In current Ceph system, file storage and object storage can only carry out object by dedicated storage clientStorage, for example, file storage can only be carried out by Ceph file system (File System, FS), object storage can only pass through S3The objects such as client store client and carry out, and the data stored cannot be mutually distinguishable, it is clear that are unfavorable for the shared of data.
In consideration of it, the embodiment of the present invention provides a kind of date storage method, and in the method, the text in data-storage systemAfter receiving file storage request, then file can be stored request with invocation protocol transfer function library include part storage serverFile data packet be converted into the object data packet that can be stored based on object, and by the object data packet storage after conversion to numberIn the Ceph cluster for including according to storage system, the effect that client realizes object storage is stored by file to realize.Wherein, byStoring in Ceph cluster is object data, and client can be equally stored by object and reads and identifies, in this way, can then makeFile data and object data are shared, and due to object storage can self the advantages of so that the dimension of dataIt protects more simple.
It referring to Figure 1, is the application scenarios of the embodiment of the present invention.Wherein, file storage client can be stored in dataStoring data and reading data in system.Data-storage system includes document storage server, protocol conversion function library and CephCluster.Wherein, Ceph cluster includes multiple Ceph servers, each Ceph server can include multiple disks, so thatObtaining each Ceph server may be used to storage object data.Certainly, data-storage system can also include that other are possibleFunctional module etc., the embodiment of the present invention is without limitation.
Fig. 2 is referred to, the embodiment of the present invention provides a kind of date storage method, and this method can be applied to shown in FIG. 1In application scenarios.This method comprises:
Step 201: document storage server receives the storage file that Network File System file storage client is sent and asksIt asks；Storage file request includes the first file data packet stored based on file；
Step 202: document storage server parses storage file request, obtains the first file data packet；
Step 203: the first file data packet is converted into first pair by document storage server invocation protocol transfer function libraryImage data packet；First object data packet is the data packet stored based on object；
Step 204: the first object data packet is stored in Ceph collection by document storage server invocation protocol transfer function libraryIn group.
In the embodiment of the present invention, when needing to carry out the storage of data, client can be stored by file and be deposited to dataStorage system sends storage file request.Specifically, the data of file storage client storage are file data, that is to say, that storageThe data packet to be stored for including in file request is file data packet.Wherein, file storage client can be network textPart system (Network File System, NFS) client, is also possible to CephFS client, it is, of course, also possible to be otherPossible file stores client, and the present invention is without limitation.It is right by taking file storage client is NFS client as an example belowThe technical solution of the embodiment of the present invention is described in detail.
In the embodiment of the present invention, after NFS client sends storage file request, the file in data-storage system is depositedStorage server can receive storage file request.Wherein, when file storage client is NFS client, then file is depositedStorage server can be that can be compatible with the server of NFS protocol used by NFS client, such as document storage server can be withFor NFS-GANESHA server.
Specifically, document storage server can parse storage file request according to NFS protocol, to obtain the storage fileRequest the first file data packet to be stored.It certainly, then can be according to phase when file storage client is other clientsCorresponding agreement is parsed, for example, when file storage client is CephFS client, then it can be according to CephFS agreementIt is parsed, the embodiment of the present invention is without limitation.
In the embodiment of the present invention, the store function as achieved by Ceph cluster is object storage, the obtainedOne file data packet can not be stored directly, and hence it is also possible to pass through document storage server invocation protocol transfer function libraryFor the first file data packet be converted into the first object data packet.Wherein, protocol conversion function library can provide for Ceph systemLibrgw base library, Librgw base library is the library of object compatible based on S3 and Swift.
Specifically, when the first file data packet is converted in document storage server invocation protocol transfer function library, it canNFS protocol used by the first file data packet is converted into the agreement that object storage client can identify.Such as objectWhen storage client is S3 client, then it can will be converted into using the first file data packet of NFS protocol using S3 agreementFirst object data packet.Fig. 3 is referred to, is the schematic diagram that NFS protocol and S3 agreement carry out protocol conversion, wherein assisted using NFSThe NFS protocol data packet of view generally includes four parts, i.e. field 1 shown in Fig. 3, field 2, field 3 and data field, wordFor carry file or file name etc. where the file to be stored or be read, field 2 indicates section 1 for carryingThe file operation code of the NFS protocol data packet operation to be carried out, field 3 are used to carry the data of the NFS protocol data packetLength, data field are then the data that NFS protocol data packet carries.It also may include four using the S3 protocol data packet of S3 agreementA part, i.e. field 1, field 2, field 3 and data field, but the field 1 in S3 protocol data packet to be stored for carryingPosition or object oriented etc. where the object either read, field 2 indicate that the S3 protocol data packet will be into for carryingThe Object Operations code of capable operation, field 3 are used to carry the data length of the S3 protocol data packet, and data field is then S3 agreementThe data that data packet carries.It, can be by calling Librgw base library to field 1, field 2 and field when carrying out protocol conversion3 modify, to complete the conversion of agreement.
In the embodiment of the present invention, after the first object data packet after being converted, document storage server then can be withFirst object data packet is stored in Ceph cluster by invocation protocol transfer function library.Wherein, when carrying out protocol conversion, theFile in one file data packet can be broken down into one or more object, thus when being stored, each objectIt can be mapped to a placement group (Placement Group, PG), each PG can also be mapped to group objects storage equipment(Object Storage Device, OSD), one of OSD is main storage, and remaining OSD is that backup storage is setIt is standby, it is interrelated between each OSD, and mutually monitor respective state.Object storage device mentioned here then can be CephCeph server in cluster.Disk or Ceph service due to the characteristic of itself of object storage, in Ceph serverAfter device breaks down, the Backup Data being stored in other Ceph servers is equally intact, prevents the loss of data.
In the embodiment of the present invention, Fig. 4 is referred to, other than it can store client by file and carry out object storage,Client directly can also be stored by object to be stored.Specifically, can store client by object will directly be wantedThe object data packet of storage is sent to Rados gateway (Rados Gateway, RGW) server, and then passes through RGW server tuneObject data packet is stored in Ceph cluster with the primary object base library LibRados that Ceph system provides.Wherein, objectStorage client and RGW server between follow object storage agreement, such as object storage client be S3 client when, objectThe agreement followed between storage client and RGW server is then S3 agreement.
Object stores client other than being able to carry out object storage, can also carry out the reading of object.Specifically, logicalThe file data for crossing file storage client storage is stored in Ceph cluster as object data, then object clientThe data that file storage client is stored can be then read from Ceph cluster, so that either file stores clientThe file data of storage or the object data of object storage client storage are held, object stores client can be from CephIt is read in cluster, is more advantageous to the shared of data.
In the embodiment of the present invention, file stores client other than it can carry out the storage of data, can also be countedAccording to reading.When file storage client wants will be read out the data stored in Ceph cluster, can be stored by fileClient sends to data-storage system and reads file request.
After document storage server in data-storage system receives reading file request, then it can parse the readingFile request, to know the reading file request data content to be read.File server may call upon protocol conversionFunction library is searched in Ceph cluster according to the data content of acquisition to be read, and obtains the number of objects to be readAccording to being packaged into the second object data packet.Wherein, read object data is deposited in addition to can be by file storage client hereObject data after the file data conversion of storage can also be the object data of object storage client storage.
In the embodiment of the present invention, since the data that the second object data packet includes are object data, file stores clientCan not Direct Recognition, therefore the second object data packet can also be converted into the second number of files by invocation protocol transfer function libraryAccording to packet, content included by the second file data packet is the file data that client Direct Recognition can be stored by file, becauseAfter the second file data packet after being converted, the second file data packet can be then sent to by document storage server for thisFile stores client.
The implementing procedure of the data-storage system of the embodiment of the present invention is explained below.
In the embodiment of the present invention, it is necessary first to construct Ceph cluster.Specifically, Ceph can be installed in storage serverSoftware, and start the service processes that the software runs the software, in software operation, the disk in storage server is passed throughCeph software is managed, in this way, storage server can then become the Ceph server of the embodiment of the present invention.It is soft by CephPart can communicate with each other between different Ceph servers, so as to achieve the effect that distributed storage.
After the building of Ceph cluster is completed, then the information by the Ceph cluster of building is needed to notify to file storage serviceDevice, therefore can will be in the configuration file of the message composition of Ceph cluster to document storage server.It is also desirable to be fileStorage server configures the information for the protocol conversion function library that its needs calls, such as the calling path letter of protocol conversion function libraryBreath, and hence it is also possible to the information of protocol conversion function library is also programmed into together in the configuration file of document storage server.WhenIt so, can also include other possible configuration informations in configuration file, this can carry out reasonable according to actual applicable casesSelection, the embodiment of the present invention are without limitation.
After the configuration file of document storage server writes completion, then configuration file can be sent to file storageConfiguration file is e.g. copied directly in document storage server by server, naturally it is also possible to will be configured by networkFile is sent to document storage server.Document storage server can parse configuration file, and according to parsing result structureIt builds for the store path of file storage client carry.
In the embodiment of the present invention, file stores the store path of the above-mentioned building of client carry and passes through after carry is completedThe data of file storage client storage can be then stored in Ceph cluster.Wherein, store path direction can be used for file storageIt is accessible to the corresponding file system of the store path to store client by file for the file system of client storing dataSystem, and can in this document system storage file, create directory, finally, in the file that is stored of file storage clientIt can be stored in Ceph cluster with the catalogue of creation.That is, what user carried out in file storage client for fileOperation can all be mapped as the operation carried out for corresponding object in data-storage system.
In conclusion the document storage server in data-storage system of the embodiment of the present invention is receiving file storage requestLater, then it can be converted into the file data packet that file storage request includes that can be based on object with invocation protocol transfer function libraryThe object data packet of storage, and in the Ceph cluster for including to data-storage system by the object data packet storage after conversion, withIt realizes and the effect that client realizes object storage is stored by file.It wherein, is object data due to being stored in Ceph cluster,Client can be equally stored by object to read and identify, in this way, then file data and object data can be made to be shared,And due to object storage can self the advantages of so that the maintenance of data is more simple.
In addition, file storage client is stored in the object data in Ceph cluster can be stored client reading by objectIt takes, corresponding, the object data that object storage client is stored in Ceph cluster equally can store client by fileIt is read, wherein it is object data that it is read, which to store client, for object, and it is after converting that it is read, which to store client, for fileFile data.Meanwhile the migration of storage system also more convenient in this way, for example, having more the condition of prospect in the storage of current objectUnder, client can be stored by file and read file data from original file data storage system, through the inventionFile data is converted into object data storage to provided by the embodiment of the present invention by date storage method provided by embodimentIn Ceph cluster, and then it can realize and original file data storage system is converted into the object data based on Ceph clusterStorage system.
Fig. 5 is referred to, the same inventive concept based on embodiment shown in Fig. 2, one embodiment of the invention provides a kind of textPart storage server 50, this document storage server include:
Receiving unit 501, for receiving the storage file request of Network File System NFS client transmission；Storage fileRequest includes the first file data packet stored based on file；
Resolution unit 502 obtains the first file data packet for parsing storage file request；
First file data packet is converted into first pair for invocation protocol transfer function library by function library call unit 503Image data packet；First object data packet is the data packet stored based on object；
Function library call unit 503 is also used to invocation protocol transfer function library for the first object data packet and is stored in distributionIn formula file system Ceph cluster；Ceph cluster includes at least one Ceph server, any at least one Ceph serverOne Ceph server is used for storage object data.
Optionally, server further includes transmission unit 504；
Receiving unit 501 is also used to receive the reading file request of NFS client transmission；File request is read for readingTake the object data stored in Ceph cluster；
Function library call unit 503 is also used to invocation protocol transfer function library according to reading file request from Ceph clusterThe middle lookup object data to be read, and generate the second object data packet；And invocation protocol transfer function library is by second pairImage data packet is converted into the second file data packet；
Transmission unit 504, for the second file data packet to be sent to NFS client.
Optionally,
Receiving unit 501, is also used to receive configuration file, and configuration file can be accessed for configuration file storage serverCeph cluster in Ceph server, and for configuration file storage server call protocol conversion function library roadDiameter；
Resolution unit 502 is also used to parse configuration file, and constructs file store path according to configuration file；Wherein,NFS client carry file store path.
The equipment can be used for executing method provided by embodiment shown in Fig. 2, therefore, for each function of the equipmentThe function etc. that module can be realized can refer to the description of embodiment shown in Fig. 2, seldom repeat.Not due to transmission unit 504It is essential functional module, therefore shown in dotted line in Fig. 5.
One embodiment of the invention provides a kind of data-storage system, which includes document storage server, protocol conversionFunction library and distributed file system Ceph cluster, Ceph cluster include at least one Ceph server, at least one Ceph clothesAny one Ceph server is used for storage object data in business device；
Document storage server for receiving the storage file request of Network File System NFS client transmission, and parsesStorage file request, obtains the first file data packet；；Storage file request includes the first number of files stored based on fileAccording to packet；
Protocol conversion function library, for the first file data packet to be converted into the first object data packet；First object dataPacket is the data packet stored based on object, and the first object data packet is stored in Ceph cluster；
Ceph cluster, for storing the first object data packet.
Optionally,
Document storage server is also used to receive the reading file request of NFS client transmission；File request is read to be used forRead the object data stored in Ceph cluster；
Protocol conversion function library is also used to search the number of objects to be read from Ceph cluster according to reading file requestAccording to, and generate the second object data packet；And the second object data packet is converted into the second file by invocation protocol transfer function libraryData packet；
Document storage server is also used to the second file data packet being sent to NFS client.
The equipment can be used for executing method provided by embodiment shown in Fig. 2, therefore, for each function of the equipmentThe function etc. that module can be realized can refer to the description of embodiment shown in Fig. 2, seldom repeat.The data before for example,Storage system.
Fig. 6 is referred to, one embodiment of the invention also provides a kind of computer installation, which includes at least oneProcessor 601, at least one processor 601 is for realizing the embodiment of the present invention when executing the computer program stored in memoryThe step of date storage method of offer.
Optionally, at least one processor 601 specifically can be central processing unit, application-specific integrated circuit (English:Application Specific Integrated Circuit, referred to as: ASIC), it can be one or more for controlling journeyThe integrated circuit that sequence executes can be use site programmable gate array (English: Field Programmable GateArray, referred to as: FPGA) exploitation hardware circuit, can be baseband processor.
Optionally, at least one processor 601 may include at least one processing core.
Optionally, which further includes memory 602, memory 602 may include read-only memory (English:Read Only Memory, referred to as: ROM), random access memory (English: Random Access Memory, referred to as: RAM)And magnetic disk storage.Memory 602 is used to store data required when the operation of at least one processor 601.The number of memory 602Amount is one or more.Wherein, memory 602 is shown together in Fig. 6, but it is understood that memory 602 is not essentialFunctional module, therefore it is shown in dotted line in Fig. 6.
In embodiments of the present invention, it should be understood that disclosed device and method, it can be real by another wayIt is existing.For example, apparatus embodiments described above are merely indicative, for example, the division of the unit or unit, onlyA kind of logical function partition, there may be another division manner in actual implementation, for example, multiple units or components can combine orPerson is desirably integrated into another system, or some features can be ignored or not executed.Another point, shown or discussed is mutualBetween coupling, direct-coupling or communication connection can be through some interfaces, the INDIRECT COUPLING or communication link of equipment or unitIt connects, can be electrical or other forms.
Each functional unit in embodiments of the present invention can integrate in one processing unit or each unit can alsoTo be independent physical module.
If the integrated unit is realized in the form of SFU software functional unit and sells or use as independent productWhen, it can store in a computer readable storage medium.Based on this understanding, the technical solution of the embodiment of the present inventionAll or part can be embodied in the form of software products, which is stored in a storage mediumIn, including some instructions use so that a computer equipment, such as can be personal computer, server or network are setStandby etc. or processor (processor) performs all or part of the steps of the method described in the various embodiments of the present invention.And it is above-mentionedStorage medium include: general serial bus USB (Universal Serial Bus flash drive), mobile hard disk, onlyRead memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic diskOr the various media that can store program code such as CD.
The above, above embodiments are only described in detail to the technical solution to the application, but the above implementationThe method that the explanation of example is merely used to help understand the embodiment of the present invention, should not be construed as the limitation to the embodiment of the present invention.ThisAny changes or substitutions that can be easily thought of by those skilled in the art, should all cover the embodiment of the present invention protection scope itIt is interior.