The data summarization method and device and electronic equipment of distributed data baseTechnical field
This specification embodiment is related to Internet technical field more particularly to a kind of data summarization side of distributed data baseMethod and device and electronic equipment.
Background technique
Data summarization refers to the detailed business data stored in database, according to one or more specified dimension intoThe process of row polymerization collect statistics.
In the related art, data summarization usually in some preset time point disposably to the data in preset duration intoRow summarizes.By taking financial industry as an example, it usually needs the finance data that can be generated on the day of in day terminal hour disposably carries out data remittanceAlways.However, gradually increasing with data volume, data summarization executes time-consuming increasingly longer；And when data summarization executes failureWhen, it is necessary to re-start data summarization.
Accordingly, it is desirable to provide a kind of data summarization scheme more efficiently.
Summary of the invention
The data summarization method and device and electronic equipment for a kind of distributed data base that this specification embodiment provides:
It is described according to this specification embodiment in a first aspect, provide a kind of data summarization method of distributed data baseMethod includes:
After meeting preset condition, data summarization instruction is generated；
Obtain the tables of data quantity N that point library table mode of the distributed data base has；
N number of tables of data is divided into M subtask；Wherein, the M is less than or equal to N, and each subtask is corresponding at leastOne tables of data；
Data summarization is carried out to data in the corresponding tables of data in each subtask parallel；
The summarized results for merging each subtask obtains total summarized results.
According to the second aspect of this specification embodiment, a kind of Data Transform Device of distributed data base is provided, it is describedDevice includes:
Generation unit generates data summarization instruction after meeting preset condition；
Acquiring unit obtains the tables of data quantity N that point library table mode of the distributed data base has；
N number of tables of data is divided into M subtask by split cells；Wherein, the M is less than or equal to N, and every height is appointedIt is engaged in corresponding at least one tables of data；
Parallel collection unit carries out data summarization to data in the corresponding tables of data in each subtask parallel；
Merge collection unit, merges the summarized results of each subtask, obtain total summarized results.
According to the third aspect of this specification embodiment, a kind of electronic equipment is provided, comprising:
Processor；
Memory for storage processor executable instruction；
Wherein, the processor is configured to the data summarization method of any of the above-described distributed data base.
This specification embodiment provides a kind of data summarization scheme of distributed data base, is based on distributed data baseIn point library divide the data isolation of table model, the traversal of multiple tables of data is summarized and is packaged into multiple subtasks, it is then parallel rightThe individual data summarization of each corresponding tables of data progress in subtask.Summarized by parallel data, greatly reduces data summarizationThe time needed.
Detailed description of the invention
Fig. 1 is the schematic diagram of point library table mode for the distributed data base that this specification provides；
Fig. 2 is the flow chart of the data summarization method for the distributed data base that one embodiment of this specification provides；
Fig. 3 is the schematic diagram for the multi-threaded parallel aggregation process that one embodiment of this specification provides；
Fig. 4 is the hardware structure diagram of the Data Transform Device for the distributed data base that one embodiment of this specification provides；
Fig. 5 is the module diagram of the Data Transform Device for the distributed data base that one embodiment of this specification provides.
Specific embodiment
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related toWhen attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodimentDescribed in embodiment do not represent all embodiments consistent with this specification.On the contrary, they are only and such as instituteThe example of the consistent device and method of some aspects be described in detail in attached claims, this specification.
It is only to be not intended to be limiting this explanation merely for for the purpose of describing particular embodiments in the term that this specification usesBook.The "an" of used singular, " described " and "the" are also intended to packet in this specification and in the appended claimsMost forms are included, unless the context clearly indicates other meaning.It is also understood that term "and/or" used herein isRefer to and includes that one or more associated any or all of project listed may combine.
It will be appreciated that though various information may be described using term first, second, third, etc. in this specification, butThese information should not necessarily be limited by these terms.These terms are only used to for same type of information being distinguished from each other out.For example, not taking offIn the case where this specification range, the first information can also be referred to as the second information, and similarly, the second information can also be claimedFor the first information.Depending on context, word as used in this " if " can be construed to " ... when " or" when ... " or " in response to determination ".
With the high speed development of business, traditional database can no longer meet the continuous growth of data volume, distributed dataThe mode of table is divided in library by using point library, can solve high concurrent to the access pressure of database.Point library divides table to refer to physicsDatabase is virtually the tables of data that several are mutually isolated.As shown in Figure 1, a point library table mode for a kind of distributed data base showsIt is intended to, distributed data base is made of n database DB, and each database D B is also divided into 10 points of tables.With 10 libraries 100Table, i.e. distributed data base have 10 databases, and each database has also been divided into 10 different tables of data.
It in the prior art, is usually all by data in all tables of data when carrying out data summarization to distributed data baseSuccessively traversal carries out data summarization.This, serial data summarization mode has that data summarization efficiency is lower.For example,With gradually increasing for data volume, data summarization executes time-consuming increasingly longer；On the other hand, when data summarization executes failure,Data summarization must be carried out to the data in all tables of data again, cause the waste of time cost.
To solve the above-mentioned problems, present description provides a kind of data summarization method of distributed data base, below may be usedTo refer to example introduction shown in Fig. 2, this method can apply the server-side in progress data summarization, the method may includeFollowing steps:
Step 110: after meeting preset condition, generating data summarization instruction.
Server-side can run a monitoring module, and it is default whether certain indexs for monitoring distributed database meetCondition；After meeting preset condition, instruction can be summarized with automatically generated data, so that server-side is directed to distributed data baseCarry out data summarization.
In general, the data summarization instruction can specify one or more dimension summarized for carrying out polymerization.
In one embodiment, it is possible to specify need to carry out the time range of the data of data summarization, such as the specified same day is newlyThe data of increasing.
In one embodiment, it is possible to specify need to carry out the range of attributes of the data of data summarization, such as specify and summarize oneCertain several attribute in data.For example, the detail of a transaction data may include transaction serial number, user name, transactionThe amount of money, account balance, loco, exchange hour etc., if business need needs to carry out data summarization to transaction amount,It can specify in data summarization instruction and data summarization only carried out to " transaction amount " this attribute in data.
In one embodiment, described after meeting preset condition, data summarization instruction is generated, is specifically included:
Judge whether current time reaches predetermined time；
In the case where whether reaching predetermined time at the current time, data summarization instruction is generated.
Wherein, the predetermined time can be a pre-set empirical value.In general, can be carried out according to business demandConfiguration.For example, needing to carry out the business of data summarization for the Sino-Japan terminal hour of financial industry, the predetermined time can refer to daily24 points, i.e., when reaching at 24 at current time, instruction can be summarized with automatically generated data.
In one embodiment, the method also includes:
Check whether full storage is into the distributed data base for data newly-increased in preset duration；
In the case where whether reaching predetermined time at the current time, data summarization instruction is generated, is specifically included:
In the case where whether the current time reaches predetermined time and integrity checking passes through, data summarization is generatedInstruction.
Wherein, the preset duration can be a pre-set empirical value.In general, can be carried out according to business demandConfiguration.For example, needing to carry out the business of data summarization for data newly-increased on the day of financial industry, the preset duration be can beRefer to 0 point of duration arrived between aforementioned predetermined time；In the case where predetermined time is 24 point, the preset duration can be 24A hour, that is to say, that data summarization processing can be carried out to the data increased newly in 24 nearest hours.
It, can be to avoid in the incomplete situation of data, only at the current time by carrying out data integrity inspectionWhether reach predetermined time and in the case that integrity checking passes through, can just automatically generate data summarization instruction.
In one embodiment, the method also includes:
It checks in the data increased newly in preset duration with the presence or absence of the data for hanging untreated state.
In the case where whether reaching predetermined time at the current time, data summarization instruction is generated, is specifically included:
Whether reach predetermined time at the current time and there is no in the case where the data for hanging untreated state,Generate data summarization instruction.
Wherein, the preset duration is as previously mentioned, details are not described herein again.In practical applications, certain data can be byIt being temporarily suspended in a variety of causes (such as error in data), such case is referred to as to hang, and the data being suspended can not be operated,Only after administrator has handled the data of the suspension, the data can just be released and be hung.Therefore, it needs to carry out data on the day ofWhen there are the data for hanging untreated state in the data summarized, data summarization cannot be executed, even if progress data summarizationIt can cause to summarize unsuccessfully due to there are the data of suspension.Therefore, it is necessary to carry out data suspension inspection.Only when described currentIt carves and whether reaches predetermined time and there is no in the case where the data for hanging untreated state, can just automatically generate dataSummarize instruction.
Step 120: obtaining the tables of data quantity N that point library table mode of the distributed data base has；
For distributed data base, divide under the table mode of library, each library have independent memory,The hardware resources such as CPU will not interfere with each other between different libraries, and also difference interferes with each other between the different tables under same library.For distributed data base, divide under the table mode of library, there is preferable data isolation between data.
In practical applications, it if data isolation is poor, when parallel work-flow data, needs to mark the data operatedNote.Label not only needs to access database, but also label processing needs to invade service logic, specific to need on initial data basisOne attribute of upper addition, the value of the attribute is for indicating whether to mark.Have preferably between data under the table mode of library due to dividingData isolation, therefore, in subsequent progress parallel processing, each subtask only need to pay close attention to itself for tables of data i.e.Can, it does not need to mark processed data.The access operation of database when on the one hand, due to avoiding label, has saved accessTime consumed by database；On the other hand, more meet woth no need to invade service logic without making marks also and do not change original numberAccording to data summarization principle.
Step 130: N number of tables of data is divided into M subtask；Wherein, the M is less than or equal to N, each subtaskAt least one corresponding tables of data.
In one embodiment, it can be divided according to the corresponding subtask of 1 tables of data.
For example, this 100 tables of data can be divided into 100 when distributed data table uses 10 100 table of librarySubtask, the corresponding tables of data in each subtask.
In one embodiment, in practical applications, N number of tables of data can be drawn according to the data volume in tables of dataIt is divided into M subtask.
For example, when data volume is more in some tables of data, where the corresponding tables of data quantity in subtask canWith relatively smaller.When data volume is less in some tables of data, where the corresponding tables of data quantity in subtask can phaseTo more.In short, can be by the equal or close of the actual data volume distribution in each subtask.In this manner it is ensured that parallelWhen data summarization, it is almost the same to summarize duration needed for each subtask, and it is too long to avoid summarizing due to some subtask duration,The case where its subtask waits occurs.
In one embodiment, M subtask can also be split as according to library situation is divided.For example, when 10 100 table of library,Can be with resolution for 10 subtasks, and each subtask corresponds to 10 tables of data in the library.
It should be noted that how N number of tables of data is divided into M subtask, can specifically carry out according to actual needsIt flexibly divides, above-described embodiment is only that some examples are not defined step 130.
Step 140: data summarization being carried out to data in the corresponding tables of data in each subtask parallel.
Server-side can be substantially reduced by carrying out data summarization to data in the corresponding tables of data in each subtask parallelOverall data summarizes required duration.It illustrates, it is assumed that data summarization is carried out to same distributed data base, it is assumed that existing stringA length of T1 when needed for the data summarization of line mode；And when using the data summarization of parallel mode shown in this specification, subtaskWhen quantity is M, required duration is most short can be with are as follows: T2=T1/M.As it can be seen that relative to conventional serial mode, using parallel modeWhen, M times can be promoted and summarize efficiency.
In one embodiment, the step 140 carries out data remittance to data in the corresponding tables of data in each subtask parallelAlways, it specifically includes:
The subtask is distributed to multithreading, via multi-threaded parallel to data in the corresponding tables of data in each subtaskCarry out data summarization.
By multithreading, may be implemented to carry out data remittance to data in the corresponding tables of data in each subtask parallelAlways.
It is illustrated in figure 3 the schematic diagram of multi-threaded parallel aggregation process, n thread parallel is to the number in n database DBData summarization is carried out according to table.
In one embodiment, described that the subtask is distributed to multithreading, it specifically includes:
Based on preset distribution policy, the subtask is distributed to multithreading.
The distribution policy includes at least one of random distribution, layout distribution.
In practical applications, distribution policy will affect the efficiency that final data summarizes.It can be with for different usage scenariosSet different distribution policies.For example, random distribution is suitable for all scenes with the good feature of versatility.Layout distribution,Library table mode can be divided for feature, the ways of distribution of preparatory Arranging Characteristics, so as to summarize efficiency more high for overall dataEffect.
Step 150: merging the summarized results of each subtask, obtain total summarized results.
This specification embodiment provides a kind of data summarization scheme of distributed data base, is based on distributed data baseIn point library divide the data isolation of table model, the traversal of multiple tables of data is summarized and is packaged into multiple subtasks, it is then parallel rightThe individual data summarization of each corresponding tables of data progress in subtask.Summarized by parallel data, greatly reduces data summarizationThe time needed improves data summarization efficiency.
In one embodiment, the summarized results of the multithreading caches in memory；
The summarized results for merging each subtask, obtains total summarized results, specifically includes:
The summarized results for merging each subtask cached in memory, obtains total summarized results.
Through this embodiment, it is only necessary to each summarized results of caching in memory be integrated, due in memoryIt completes, without accessing distributed data base, so as to further promoting data summarization efficiency.
Corresponding with the data summarization method embodiment of aforementioned distributed data base, this specification additionally provides distributed numberAccording to the embodiment of the Data Transform Device in library.Described device embodiment can by software realization, can also by hardware orThe mode of software and hardware combining is realized.It taking software implementation as an example, is by equipment where it as the device on a logical meaningProcessor by computer business program instruction corresponding in nonvolatile memory be read into memory operation formed.From hardFor part level, as shown in figure 4, for a kind of hardware structure diagram of this specification Data Transform Device place equipment, in addition to Fig. 4 instituteExcept the processor, network interface, memory and the nonvolatile memory that show, equipment in embodiment where device generally according toThe data summarization actual functional capability of the distributed data base can also include other hardware, repeat no more to this.
Fig. 5 is referred to, is the module map of the Data Transform Device for the distributed data base that one embodiment of this specification provides,Described device has corresponded to embodiment illustrated in fig. 2, and described device includes:
Generation unit 310 generates data summarization instruction after meeting preset condition；
Acquiring unit 320 obtains the tables of data quantity N that point library table mode of the distributed data base has；
N number of tables of data is divided into M subtask by split cells 330；Wherein, the M is less than or equal to N, every heightTask corresponds at least one tables of data；
Data in the corresponding tables of data in each subtask are carried out data summarization parallel by parallel collection unit 340；
Merge collection unit 350, merges the summarized results of each subtask, obtain total summarized results.
In an optional embodiment:
The generation unit 310, specifically includes:
Judgment sub-unit, judges whether current time reaches predetermined time；
Subelement is generated, in the case where whether reaching predetermined time at the current time, generates data summarization instruction.
In an optional embodiment:
Described device further include:
First checks subelement, and checking the data increased newly in preset duration, whether full storage is to the distributed data baseIn；
Subelement is being generated, is being specifically included:
In the case where whether the current time reaches predetermined time and integrity checking passes through, data summarization is generatedInstruction.
In an optional embodiment:
Described device further include:
Second checks subelement, checks in the data increased newly in preset duration with the presence or absence of the number for hanging untreated stateAccording to.
Subelement is being generated, is being specifically included:
Whether reach predetermined time at the current time and there is no in the case where the data for hanging untreated state,Generate data summarization instruction.
In an optional embodiment:
The parallel collection unit 340, specifically includes:
The subtask is distributed to multithreading, via multi-threaded parallel to data in the corresponding tables of data in each subtaskCarry out data summarization.
In an optional embodiment:
The summarized results caching of the multithreading is in memory；
The merging collection unit 350, specifically includes:
The summarized results for merging each subtask cached in memory, obtains total summarized results.
In an optional embodiment:
It is described that the subtask is distributed to multithreading, it specifically includes:
Based on preset distribution policy, the subtask is distributed to multithreading.
In an optional embodiment:
The distribution policy includes at least one of random distribution, layout distribution.
In an optional embodiment:
By split cells 330, specifically include:
According to the data volume in tables of data, N number of tables of data is divided into M subtask.
System, device, module or the unit that above-described embodiment illustrates can specifically realize by computer chip or entity,Or it is realized by the product with certain function.A kind of typically to realize that equipment is computer, the concrete form of computer canTo be personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media playIn device, navigation equipment, E-mail receiver/send equipment, game console, tablet computer, wearable device or these equipmentThe combination of any several equipment.
The function of each unit and the realization process of effect are specifically detailed in the above method and correspond to step in above-mentioned apparatusRealization process, details are not described herein.
For device embodiment, since it corresponds essentially to embodiment of the method, so related place is referring to method realityApply the part explanation of example.The apparatus embodiments described above are merely exemplary, wherein described be used as separation unitThe unit of explanation may or may not be physically separated, and component shown as a unit can be or can also be withIt is not physical unit, it can it is in one place, or may be distributed over multiple network units.It can be according to actualThe purpose for needing to select some or all of the modules therein to realize this specification scheme.Those of ordinary skill in the art are notIn the case where making the creative labor, it can understand and implement.
Figure 5 above describes inner function module and the structural representation of Data Transform Device, substantial executing subjectIt can be a kind of electronic equipment, comprising:
Processor；
Memory for storage processor executable instruction；
Wherein, the processor is configured to:
After meeting preset condition, data summarization instruction is generated；
Obtain the tables of data quantity N that point library table mode of the distributed data base has；
N number of tables of data is divided into M subtask；Wherein, the M is less than or equal to N, and each subtask is corresponding at leastOne tables of data；
Data summarization is carried out to data in the corresponding tables of data in each subtask parallel；
The summarized results for merging each subtask obtains total summarized results.
Optionally, described after meeting preset condition, data summarization instruction is generated, is specifically included:
Judge whether current time reaches predetermined time；
In the case where whether reaching predetermined time at the current time, data summarization instruction is generated.
Optionally, further includes:
Check whether full storage is into the distributed data base for data newly-increased in preset duration；
In the case where whether reaching predetermined time at the current time, data summarization instruction is generated, is specifically included:
In the case where whether the current time reaches predetermined time and integrity checking passes through, data summarization is generatedInstruction.
Optionally, further includes:
It checks in the data increased newly in preset duration with the presence or absence of the data for hanging untreated state.
In the case where whether reaching predetermined time at the current time, data summarization instruction is generated, is specifically included:
Whether reach predetermined time at the current time and there is no in the case where the data for hanging untreated state,Generate data summarization instruction.
Optionally, described that data summarization is carried out to data in the corresponding tables of data in each subtask parallel, it specifically includes:
The subtask is distributed to multithreading, via multi-threaded parallel to data in the corresponding tables of data in each subtaskCarry out data summarization.
Optionally, the summarized results of the multithreading caches in memory；
The summarized results for merging each subtask, obtains total summarized results, specifically includes:
The summarized results for merging each subtask cached in memory, obtains total summarized results.
Optionally, described that the subtask is distributed to multithreading, it specifically includes:
Based on preset distribution policy, the subtask is distributed to multithreading.
Optionally, the distribution policy includes at least one of random distribution, layout distribution.
Optionally, N number of tables of data is divided into M subtask, specifically included:
According to the data volume in tables of data, N number of tables of data is divided into M subtask.
In the embodiment of above-mentioned electronic equipment, it should be appreciated that the processor can be central processing unit (English:Central Processing Unit, referred to as: CPU), can also be other general processors, digital signal processor (English:Digital Signal Processor, referred to as: DSP), specific integrated circuit (English: Application SpecificIntegrated Circuit, referred to as: ASIC) etc..General processor can be microprocessor or the processor is also possible toAny conventional processor etc., and memory above-mentioned can be read-only memory (English: read-only memory, abbreviation:ROM), random access memory (English: random access memory, abbreviation: RAM), flash memory, hard disk or solidState hard disk.The step of method in conjunction with disclosed in the embodiment of the present invention, can be embodied directly in hardware processor and execute completion, orHardware and software module combination in person's processor execute completion.
All the embodiments in this specification are described in a progressive manner, same and similar portion between each embodimentDividing may refer to each other, and each embodiment focuses on the differences from other embodiments.It is set especially for electronicsFor standby embodiment, since it is substantially similar to the method embodiment, so being described relatively simple, related place is referring to method realityApply the part explanation of example.
Those skilled in the art after considering the specification and implementing the invention disclosed here, will readily occur to this specificationOther embodiments.This specification is intended to cover any variations, uses, or adaptations of this specification, these modifications,Purposes or adaptive change follow the general principle of this specification and undocumented in the art including this specificationCommon knowledge or conventional techniques.The description and examples are only to be considered as illustrative, the true scope of this specification andSpirit is indicated by the following claims.
It should be understood that this specification is not limited to the precise structure that has been described above and shown in the drawings,And various modifications and changes may be made without departing from the scope thereof.The range of this specification is only limited by the attached claimsSystem.