Summary of the invention
For the problems of the prior art, the present invention provides a kind of data archiving method, system, equipment and storages to be situated betweenMatter achievees the purpose that database data is filed by the way of executing filing task automatically, meanwhile, the distributed storage mould of useThe flexibility of type improves the variation of the efficiency and compatible data business model of data storage.
The embodiment provides a kind of data archiving methods, comprising the following steps:
Obtain the performance parameter of business datum library facilities；
Judge whether the performance parameter meets default archive condition；
If it is satisfied,
The type of service that each traffic table to be archived is then obtained from database obtains filing according to the type of service and matchesIt sets, the archival configuration is provided with the corresponding relationship of each traffic table and filing table in this filing task；
Filing instruction is generated according to the archival configuration；
It executes the filing instruction and each traffic table of database is stored to the filing into HBase distributed memory systemTable.
Preferably, further comprising the steps of during executing filing instruction:
Obtain the performance parameter of business datum library facilities；
Judge whether the performance parameter meets default archive condition；
If conditions are not met, then stopping executing the filing instruction.
Preferably, the performance parameter is the one or more of TPS, active thread number or CPU usage.
Preferably, the data archiving method further includes following steps:
Delete stored each traffic table data into HBase distributed memory system in database.
Preferably, stored each traffic table data into distributed memory system in the deletion database, sentenceThe time that each traffic table of breaking is formed deletes in database if the traffic table forms the time earlier than at the first timeThe stored traffic table data into distributed memory system.
Preferably, the archival configuration is additionally provided with each traffic table corresponding relationship in filing table, the filing of generationInstruction further includes that the instruction of index information is generated according to the filing table stored into distributed memory system.
Preferably, the data archiving method further includes data query step.
Preferably, the data query step specifically comprises the following steps:
Inquiry request is received, according to the filing table of index information inquiry storage.
The embodiments of the present invention also provide a kind of data archiving systems, including profiling module, configuration module and execution mouldBlock, in which:
The profiling module obtains the performance parameter of business datum library facilities；
The profiling module judges whether the performance parameter meets default archive condition；
If it is satisfied, then the profiling module obtains the type of service of data to be archived from database, according to the industryService type obtains archival configuration from the configuration module, and the archival configuration is provided with each traffic table in this filing taskWith the corresponding relationship of filing table；
The profiling module generates filing instruction according to the archival configuration；
The execution module executes the filing instruction and stores each traffic table of database to HBase distributed storageFiling table in system.
Preferably, the filing instruction is additionally provided with each traffic table corresponding relationship in filing table, and the execution module is alsoIndex information is generated with instructing according to the filing of generation.
It preferably, further include enquiry module, in which:
The enquiry module is for receiving inquiry request, according to the data of index information inquiry storage.
The embodiments of the present invention also provide a kind of data filing equipment, comprising:
Processor；
Memory, wherein being stored with the executable instruction of the processor；
Wherein, the processor is configured to execute the step of the data archiving method via the executable instruction is executedSuddenly.
The embodiments of the present invention also provide a kind of computer readable storage mediums, for storing program, which is characterized in thatDescribed program is performed the step of realizing the data archiving method.
Data archiving method of the invention is stored the data in database by the way of executing filing task automaticallyInto distributed memory system, using the characteristic of distributed data base dynamic column, multiple traffic table model combinations can be supported to oneIn a distribution table, and new field can be arbitrarily extended, both save memory space, also simplified routine access dataLogic；Data after being filed using the present invention establish index in combination with inquiry business scene, to support the multidimensional of data after filingDegree inquiry, to provide compatible data library data and file the inquiry operation of data.
Specific embodiment
Example embodiment is described more fully with reference to the drawings.However, example embodiment can be with a variety of shapesFormula is implemented, and is not understood as limited to example set forth herein；On the contrary, thesing embodiments are provided so that the disclosure will moreFully and completely, and by the design of example embodiment comprehensively it is communicated to those skilled in the art.Described feature, knotStructure or characteristic can be incorporated in any suitable manner in one or more embodiments.
In addition, attached drawing is only the schematic illustrations of the disclosure, it is not necessarily drawn to scale.Identical attached drawing mark in figureNote indicates same or similar part, thus will omit repetition thereof.Some block diagrams shown in the drawings are functionEnergy entity, not necessarily must be corresponding with physically or logically independent entity.These function can be realized using software formEnergy entity, or these functional entitys are realized in one or more hardware modules or integrated circuit, or at heterogeneous networks and/or placeThese functional entitys are realized in reason device device and/or microcontroller device.
Fig. 1 be one embodiment of the invention data archiving method flow chart, specifically, the data archiving method include withLower step:
S100, the performance parameter for obtaining business datum library facilities；The frequency for obtaining performance parameter can be set in systems.
S200, judge whether the performance parameter meets default archive condition；Performance parameter herein includes TPSOne kind or more of (transaction per second, the transactions of execution per second), active thread number or CPU usage etc.Kind.
For example, CPU usage is exactly the cpu resource that the program run occupies, and indicates business datum library facilities at certainThe case where operation program at a time point.CPU usage is higher, illustrates that equipment has run more programs on this time,Otherwise fewer program is run.The height of utilization rate and the CPU power of equipment itself have direct relation.Similarly, in order to mentionThe efficiency of business datum library facilities is risen, current equipment has all generallyd use the multithreading of multi-threading parallel process service request simultaneouslyTupe is sent out, multiple service requests are handled within the same period, at the service request to enhance serverReason ability, but due to being limited by business datum library facilities process performance and system resource, equipment can be used in execution businessThe sum of the thread of parallel processing is requested to be limited.Therefore, in the embodiment of the present invention, by obtaining CPU usage, TPSAnd/or the performance parameters such as active thread number can preset above-mentioned performance parameter different threshold values according to different business, it canTo provide to be more than or less than certain threshold values as the condition for meeting system and executing automatically filing task.
If S300, meeting default archive condition, the type of service of each traffic table to be archived is obtained from database,Archival configuration is obtained according to the type of service, the archival configuration is provided with each traffic table and filing in this filing taskThe corresponding relationship of table；Filing instruction is generated according to the archival configuration；The filing instruction is executed by each business of databaseTable stores the filing table into HBase distributed memory system.
The corresponding relationship of each traffic table and filing table is according to each industry in this filing task of archival configuration settingThe corresponding business of table of being engaged in needs, i.e. type of service, and preset, and different traffic tables and different filing tables are corresponding.TogetherOne traffic table can also change according to demand and change with filing table to relationship.The corresponding pass of each traffic table and filing tableThe convenient dynamic change of system, has benefited from the dynamic column of the HBase distributed memory system utilized in narration behind the present inventionCharacteristic.
For example, such as:
The first table of commodity includes field: [commodity code, product name, the type of merchandise]；
Item property table is the second table of commodity, including field: [mobile phone shell color, operating system, mobile phone brand]；
Commodity other information is commodity third table, including field: [place of production, the date of production]；
If needed with a business, filing table is to merge the first table of commodity and the second table of commodity or commodity firstTable and commodity third table, the then field of commodity filing table after filing can include: [commodity code, product name, commodity classType, mobile phone shell color, operating system, mobile phone brand], and [commodity code, product name, the type of merchandise, the place of production are dispatched from the factory dayPhase].
Filing table can even is that the first table of commodity, in the second table of commodity and commodity third table arbitrary fields groupIt closes.The application can be referred to by the way that filing is set dynamically, that is, the corresponding relationship of each traffic table Yu filing table is set dynamically, obtainsThe filing table of arbitrary fields combination.Therefore, it is more suitable for changeable business demand.
After profiling module reads archival configuration, filing instruction is generated, specifically, filing instruction may include:
The corresponding relationship of the corresponding each traffic table of this data filing task and filing table；
The corresponding each traffic table data area of this data filing task；
The identifier of this data filing task；
Current database time etc..
Each traffic table stores the filing table into HBase distributed memory system, and HBase distributed memory system is differentIt is the distributed memory system for being suitable for unstructured data in general relational database；Meanwhile and one be based onColumn rather than the PostgreSQL database based on capable mode.Therefore, the flexibility of the distributed storage model of use is high, can support moreTraffic table model combination is opened into a distributed table, and can arbitrarily extend new field, to improve data storageEfficiency, meanwhile, and adapted to business model and changed fast business scenario.
The present invention utilizes the characteristic of HBase dynamic column, does not need to define filing table model in advance.It can be incited somebody to action in archival configurationThe traffic table that multiple needs are filed corresponds in a HBase table, in filing implementation procedure according to filing instruction in HBaseCreation filing table, i.e. the application are convenient to need to be merged into a HBase filing according to business for the field of relevant traffic tableIn table.Therefore, the application can reduce the coupled relation of data bank service table Yu filing table, reduce redundant data, deposit to reduceSpace is stored up, data access logic is simplified.
When the performance parameter of business datum library facilities meets default archive condition, i.e. system execution S300 step, hereinIn the process, profiling module still will acquire the performance parameter of service database equipment, and whether meet default filing to performance parameterCondition makes a decision, and when performance parameter no longer meets default archive condition, then can stop executing filing instruction.Actual operationIn, while stopping executing filing instruction, system also records data filing process.Filing process can be this data filing and appointThe identifier of business, record data filing process are to write the identifier of this data filing task in control file, Radix Angelicae SinensisWhen shelves task is restored again after stopping, it being filed with reference to process is filed accordingly in control file.
The data archiving method of one embodiment of the invention further includes data query step, and data query step, which specifically includes, to be connectInquiry request is received, according to the data of index information inquiry storage.In S300 step, the archival configuration further includes settingThere is each traffic table corresponding relationship in filing table, the filing instruction of generation further includes according to storing to distributed memory systemIn filing table generate index information instruction.In one embodiment, this step is exactly to generate each traffic table while filingQuery SQL, the data that SQL query goes out are assembled according to corresponding relationship and are saved in filing table.
Similarly, different business corresponds to different index informations, for example, it is desired to inquire its of commodity according to goods numberHis relevant information then needs the index information of other relevant informations of creation Customs Assigned Number and commodity；When needing to be updated according to commodityBetween inquire data, create the index information of other relevant informations of commodity renewal time and commodity.The present invention can be directed to not of the same trade or businessBusiness creates different search indexes, to support the various dimensions inquiry of filing data.Generation index information step is subsequent inquiryStep is prepared.
Above-mentioned index information can be generated when executing filing task, can also be according to regenerating after filing.
For stored each traffic table data into distributed memory system, usually will be deleted in embodiment, orPerson is in deleting database before stored each traffic table data into distributed memory system, and system judges data shapeAt time, if each traffic table data formed the time earlier than at the first time, delete, i.e., filing data within a certain period of timeIt can still be stored in original database, can set at the first time is one month.
For data archiving method of the invention, when carrying out data query, the inquiry of compatible filing data.According to industryThe query demand of business system is combined judgement to time range, the querying condition etc. of inquiry, if the data generated in the recent period,Including still not deleted data after the data and filing do not filed, then it can directly access the database and be inquired；If having returnedThe data of shelves then inquire data by the business index information that configuration module creates from distributed memory system.
Data archiving method of the invention can read to avoid data filing step database the increase of pressure, in businessPhase at a low ebb or system are not in automatic service database tasks in the state of oepration at full load, are conducive to not to systemExisting device resource is adequately utilized under the premise of impacting.
Fig. 2 is the schematic diagram of the data archiving system of one embodiment of the invention, which includes profiling module 11, configuration mouldBlock 12 and execution module 13, in which:
The profiling module 11 obtains the performance parameter of business datum library facilities；
The profiling module 11 judges whether the performance parameter meets default archive condition；
If it is satisfied, then the profiling module 11 obtains the type of service of data to be archived from database, according to describedType of service obtains archival configuration from the configuration module 12, and the archival configuration is provided with each industry in this filing taskBusiness table and the corresponding relationship for filing table；
The profiling module 11 generates filing instruction according to the archival configuration；
13 modules of the execution, which execute filing instruction and store each traffic table of database to HBase distribution, depositsFiling table in storage system.
The filing instruction is additionally provided with each traffic table corresponding relationship in filing table, and the execution module 13 also has rootIndex information is generated according to the filing instruction of generation.In order to realize the query steps in data archiving method, an embodimentData archiving system further includes enquiry module, and the enquiry module is inquired for receiving inquiry request according to the index informationThe data of storage.
The electronic equipment 600 of this embodiment according to the present invention is described referring to Fig. 3.The electronics that Fig. 3 is shownEquipment 600 is only an example, should not function to the embodiment of the present invention and use scope bring any restrictions.
As shown in figure 3, electronic equipment 600 is showed in the form of universal computing device.The component of electronic equipment 600 can wrapInclude but be not limited to: at least one processing unit 610, at least one storage unit 620, connection different platform component (including storageUnit 620 and processing unit 610) bus 630, display unit 640 etc..
Wherein, storage unit is stored with program code, and program code can be executed with unit 610 processed, so that processing is singleMember 610 executes various exemplary implementations according to the present invention described in this specification above-mentioned electronic prescription circulation processing method partThe step of mode.For example, processing unit 610 can execute step as shown in fig. 1.
Storage unit 620 may include the readable medium of volatile memory cell form, such as Random Access Storage Unit(RAM) 6201 and/or cache memory unit 6202, it can further include read-only memory unit (ROM) 6203.
Storage unit 620 can also include program/utility with one group of (at least one) program module 62056204, such program module 6205 includes but is not limited to: operating system, one or more application program, other program mouldsIt may include the realization of network environment in block and program data, each of these examples or certain combination.
Bus 630 can be to indicate one of a few class bus structures or a variety of, including storage unit bus or storageCell controller, peripheral bus, graphics acceleration port, processing unit use any bus structures in a variety of bus structuresLocal bus.
Electronic equipment 600 can also be with one or more external equipments 700 (such as keyboard, sensing equipment, bluetooth equipmentDeng) communication, can also be enabled a user to one or more equipment interact with the electronic equipment 600 communicate, and/or with makeAny equipment (such as the router, modulation /demodulation that the electronic equipment 600 can be communicated with one or more of the other calculating equipmentDevice etc.) communication.This communication can be carried out by input/output (I/O) interface 650.Also, electronic equipment 600 can be withBy network adapter 660 and one or more network (such as local area network (LAN), wide area network (WAN) and/or public network,Such as internet) communication.Network adapter 660 can be communicated by bus 630 with other modules of electronic equipment 600.It shouldUnderstand, although not shown in the drawings, other hardware and/or software module can be used in conjunction with electronic equipment 600, including but unlimitedIn: microcode, device driver, redundant processing unit, external disk drive array, RAID system, tape drive and numberAccording to backup storage platform etc..
The embodiment of the present invention also provides a kind of computer readable storage medium, and for storing program, program is performed realizationThe step of sorting data archiving method.In some possible embodiments, various aspects of the invention are also implemented as oneThe form of kind program product comprising program code, when program product is run on the terminal device, program code is for making endEnd equipment executes described in this specification above-mentioned electronic prescription circulation processing method part various exemplary realities according to the present inventionThe step of applying mode.
Refering to what is shown in Fig. 4, describing the program product for realizing the above method of embodiment according to the present invention800, can using portable compact disc read only memory (CD-ROM) and including program code, and can in terminal device,Such as it is run on PC.However, program product of the invention is without being limited thereto, in this document, readable storage medium storing program for executing can be withTo be any include or the tangible medium of storage program, the program can be commanded execution system, device or device use orIt is in connection.
Program product can be using any combination of one or more readable mediums.Readable medium can be readable signal JieMatter or readable storage medium storing program for executing.Readable storage medium storing program for executing for example can be but be not limited to electricity, magnetic, optical, electromagnetic, infrared ray or partly leadSystem, device or the device of body, or any above combination.More specific example (the non exhaustive column of readable storage medium storing program for executingTable) it include: the electrical connection with one or more conducting wires, portable disc, hard disk, random access memory (RAM), read-only storageDevice (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc read only memory (CD-ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.
Computer readable storage medium may include in a base band or as carrier wave a part propagate data-signal,In carry readable program code.The data-signal of this propagation can take various forms, including but not limited to electromagnetic signal,Optical signal or above-mentioned any appropriate combination.Readable storage medium storing program for executing can also be any readable Jie other than readable storage medium storing program for executingMatter, the readable medium can send, propagate or transmit for by instruction execution system, device or device use or and itsThe program of combined use.The program code for including on readable storage medium storing program for executing can transmit with any suitable medium, including but notIt is limited to wireless, wired, optical cable, RF etc. or above-mentioned any appropriate combination.
The program for executing operation of the present invention can be write with any combination of one or more programming languagesCode, programming language include object oriented program language-Java, C++ etc., further include conventional processFormula programming language-such as " C " language or similar programming language.Program code can be calculated fully in userIt executes in equipment, partly execute on a user device, executing, as an independent software package partially in user calculating equipmentUpper part executes on a remote computing or executes in remote computing device or server completely.It is being related to remotely countingIn the situation for calculating equipment, remote computing device can pass through the network of any kind, including local area network (LAN) or wide area network(WAN), it is connected to user calculating equipment, or, it may be connected to external computing device (such as utilize ISPTo be connected by internet).
In conclusion the present invention provides a kind of data archiving method, system, equipment and storage medium, data filing sideMethod includes: to obtain the performance parameter of business datum library facilities；Judge whether the performance parameter meets default archive condition；IfMeet, then obtain the type of service of data to be archived from database, archival configuration is obtained according to the type of service, it is described to returnShelves configuration is provided with the corresponding relationship of each traffic table and filing table in this filing task, is returned according to archival configuration generationShelves instruction executes the filing instruction for the data of database and stores the filing table into distributed memory system, the distributionFormula storage system is HBase system.Data archiving method of the invention, by the way of executing filing task automatically, by dataIt stores into distributed memory system, HBase has the characteristic of dynamic column, multiple traffic table model combinations can be supported to one pointIn cloth table, and new field can be arbitrarily extended, the coupled relation of data bank service table Yu filing table can be reduced, reduced superfluousRemainder evidence simplifies data access logic to reduce memory space；Using the data after present invention filing, in combination with inquiry industryScene of being engaged in establishes index, to support the various dimensions inquiry of data after filing, to provide compatible data library data and filing dataInquiry operation.
The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be said thatSpecific implementation of the invention is only limited to these instructions.It is obvious to a person skilled in the art that the application be not limited to it is above-mentionedThe details of exemplary embodiment, and without departing substantially from spirit herein or essential characteristic, it can be with others toolBody form realizes the application.It therefore, in all respects, the embodiments should be taken as exemplary, and is non-limitProperty processed, scope of the present application is indicated by the appended claims rather than the foregoing description, it is intended that claim will be fallen inAll changes in the meaning and scope of equivalency are included in the application.It should not be by any appended drawing reference in claimIt is construed as limiting the claims involved.Furthermore, it is to be understood that one word of " comprising " does not exclude other units or steps, odd number is not excluded for multipleNumber.The multiple units or device stated in device claim can also by a unit or device by software or hardware comeIt realizes.First, second equal words are used to indicate names, and are not indicated any particular order.