Background technique
In " big data " epoch, with 3S technology answering in vehicle location, intelligent management, field of vehicle control extensivelyWith car networking is come into being.Car networking is that collection data acquisition, data communication, real time monitoring, data processing and data mining are oneThe computer real time information system of body.The processing of real time data is the core content of whole system work.The processing of real time dataThe real time data generated towards real-time acquisition or analytical calculation is needed, treatment process requires very strong timeliness and reliability,The historical data of magnanimity can be also generated simultaneously.These data can manage department for vehicle and provide by excavation and analytical calculationScientific, real-time, accurate traffic-information service increases enterprise income.Therefore, this history data store process for magnanimityReal-time and reliability requirement are also just higher and higher.
Currently, car networking will generate the data of magnanimity during vehicle, road are interacted with people.How under prescribed conditionsThere is provided in real time specifically, accurate and reliable vehicle operation and status information, improve car networking mass data transfers storageReal-time and reliability, it is particularly difficult.This is primarily present the defect of following several respects:
1) data normalization problem.Due to car networking terminal generate data have " multi-source heterogeneous " characteristic (such as picture,Text, video, audio etc.), general software tool is difficult to comprehensively be competent at the works such as the extraction, management and processing of these dataMake.
2) dirty data problem.Since data volume is very big, inevitably will appear as caused by the factors such as terminal, artificial, environmentA large amount of data quality problem.For example data are imperfect, unusual, redundancy and inconsistent etc..The problematic data of these qualityIt is referred to as " junk data ".
3) data storing reliability problem.The integrality of data is very important for a user.Only to completeData analyzed, user is likely to obtain accurate objective result.At this stage, the transmission speed of car networking real time dataFastly, data volume is big, but there is also very big defects for the real-time of data storage procedure and reliability.
Summary of the invention
In order to solve the above-mentioned technical problems, the present invention provides a kind of new car networking real-time data processing method and it isSystem.
A kind of car networking real-time data processing method provided by the invention, comprising the following steps:
Monitoring step carries out real-time monitoring to the terminal data of car networking, wherein being based on time difference and fixed frequency phaseIn conjunction with monitoring policy dynamically adjust monitoring time interval；
Review step examines the data monitored in real time, rejects stale data therein；
Processing step handles the data examined in real time, cleans and repair wrong data therein；
Storing step carries out real-time storage to processed data.
According to an embodiment of the invention, in the monitoring step, the monitoring that is combined based on time difference with fixed frequencyStrategy are as follows:
Start monitoring data after first time starts, and records the last renewal time t of the data0, then underThe time point of one secondary monitoring data is the last renewal time t0With preset stationary monitoring cycle T0The sum of, if at this timePresent system time is Ts, then △ t=t is divided between monitoring time0+T0-Ts；
In time point t0+T0Monitoring data again: if the data not yet update, automatically adjust what distance monitored next timeMonitoring time interval △ t ' enables △ t ' < △ t, then waits and monitoring next time, continues to judge whether data have been updated, soRepeatedly until monitoring that data update；Wherein, once data update, monitoring time interval is re-set as preset fixed prisonSurvey cycle T0。
According to an embodiment of the invention, judging data based on the property of data and life span in the review stepIt is whether expired, and then reject stale data.
According to an embodiment of the invention, for the data of effective limitation, if the life span of data be greater than it is preset mostBig real-time time delay value, then judge data for stale data；Wherein, the life span subtracts number equal to the system actual treatment timeAccording to the last renewal time.
According to an embodiment of the invention, in the processing step, cleaned in real time using window sliding technology and fix errorsData.
According to an embodiment of the invention, in the storing step, using the memory module of cyclic buffer to processedData carry out real-time storage.
In addition, the present invention also provides a kind of car networking real-time data processing systems, comprising:
Data real-time monitoring layer, for car networking terminal data carry out real-time monitoring, wherein based on time difference withThe monitoring policy that fixed frequency combines dynamically adjusts monitoring time interval；
Data examine layer in real time, for being examined in real time the data monitored, reject stale data therein；
Generating date layer cleans and repairs error number therein for being handled in real time the data examinedAccording to；
Data real-time storage layer, for carrying out real-time storage to processed data.
According to an embodiment of the invention, the data real-time monitoring layer use is mutually tied based on time difference with fixed frequencyThe monitoring policy of conjunction are as follows:
Start monitoring data after first time starts, and records the last renewal time t of the data0, then underThe time point of one secondary monitoring data is the last renewal time t0With preset stationary monitoring cycle T0The sum of, if at this timePresent system time is Ts, then △ t=t is divided between monitoring time0+T0-Ts；
In time point t0+T0Monitoring data again: if the data not yet update, automatically adjust what distance monitored next timeMonitoring time interval △ t ' enables △ t ' < △ t, then waits and monitoring next time, continues to judge whether data have been updated, soRepeatedly until monitoring that data update；Wherein, once data update, monitoring time interval is re-set as preset fixed prisonSurvey cycle T0。
According to an embodiment of the invention, the data examine layer based on the property of data and life span to judge number in real timeAccording to whether expired, and then reject stale data.
According to an embodiment of the invention, the generating date layer cleans and repairs in real time mistake using window sliding technologyAccidentally data.
Compared with prior art, one or more embodiments of the invention can have following advantage:
The invention proposes a kind of new car networking real-time data processing methods and system.This method and its system are mainly usedIn be directed to the collected real time data of car networking terminal, to collected real-time dynamic data carry out real-time update, data examination,Cleaning treatment and missing repairing etc., to improve the quality of data.Moreover, also realizing magnanimity history by introducing " double buffering " technologyData real-time storage.The present invention can not only provide in real time specifically, accurate and reliable vehicle operation and status information,And also improve car networking mass data transfers storage real-time and guarantee the integrality of data, so as to be gone through to magnanimityHistory data excavate and analysis meter calculates a large amount of useful informations, real-time for driver, vehicle management company, vehicle supervision departmentScientific, real-time, accurate traffic-information service is provided, and then improves transport services and intelligent management.
Other features and advantages of the present invention will be illustrated in the following description, also, partly becomes from specificationIt obtains it is clear that understand through the implementation of the invention.The objectives and other advantages of the invention can be by specification, rightSpecifically noted structure is achieved and obtained in claim and attached drawing.
Specific embodiment
To make the object, technical solutions and advantages of the present invention clearer, below in conjunction with attached drawing, from the angle of system architectureThe present invention is described in further detail.
First embodiment
Fig. 1 is the composition schematic diagram of car networking real-time data processing system provided in an embodiment of the present invention.It can be with from Fig. 1Find out, which mainly examines that layer 200, generating date layer 300, data are real by data real-time monitoring layer 100, data in real timeWhen accumulation layer 400 module composition.Wherein:
Data real-time monitoring layer 100, the module of the level, which is mainly responsible for, carries out real-time monitoring to car networking terminal data source,The real-time monitoring to terminal data real-time update situation is realized, to examine that layer 200 provides " real-time " data in real time for data.ItsIn, in order to reach the monitoring effect of precise and high efficiency, present invention preferably employs one kind to be combined based on time difference with fixed frequencyMonitoring policy dynamically adjust monitoring time interval.Particular content is as follows:
Assuming that system starts monitoring data transmission server after first time starts, and record data on server " mostNearly primary renewal time ".For a certain data, it is assumed that its last renewal time is t0, then time point t0In additionThe stationary monitoring cycle T of systemic presupposition0It can be obtained by the time point t that system wants monitoring data transmission server next time0+T0,So if present system time is Ts, then need to wait for a period of time △ t=t for system0+T0-TsJust needMonitoring data transmission server again.This period △ t is known as monitoring time interval.It should be noted that in the present inventionIn, this monitoring time interval △ t is not unalterable with regard to this.On the contrary, system then can be according to data more newsDynamically adjust monitoring time interval.For example, when system is in time point t0+T0When monitoring data transmission server, the data are foundIt not yet updates, then illustrates that the data are not updated but may will be updated.Shorten the time to improve efficiency, system will be adjusted automaticallyPitch is from the monitoring time interval △ t ' monitored next time, when being set to more shorter than last monitoring time interval △ tBetween section, i.e. then △ t ' < △ t waits and monitoring next time, continue to judge whether data have been updated.Repeatedly, until monitoringIt is updated to data, and once monitors that data update, system then needs to reply at monitoring time interval again as preset fixationMonitoring cycle T0。
Data examine layer 200 in real time, and monitor layer 100 monitors real-time when the module of the level is mainly responsible for logarithm factuallyData are examined in real time.Due to and the not all data real-time monitored be all it is intact, handle these numberAccording to also needing to examine these data before.Examine that the purpose of data is intended to reject the data or not of those apparent damagesThe source data of the mistake such as complete data, in order to avoid waste system resource, the system of even resulting in report an error, while can also avoid becauseFail to handle some significant datas in time and cause immeasurable loss.Therefore, in the present embodiment, for it is this kind of notHaving the data of " real-time ", data examine several " non real-time " problems common mainly in combination with current car networking of layer 200 in real time,It is divided into following three kinds of situations according to " property " and " validity period " of Various types of data to be handled:
1) if data are significant data and there is still a need for processing in time still in the data in validity period, are illustrated；It is this kind ofData can be the vehicle body security system warning data of for example vehicle；Although it does not have " real-time ", it is contemplated that havingIt may be that accident is still fermenting the situation having not occurred, therefore this class data file is still particularly significant, need " real-time " processing；
2) data have not needed to carry out " real-time " processing, and however, there remains saved as historical data data；For this kind of data, system can choose to be dealt with again during idle time, can effectively improve working efficiency；This kind of data can be withIt is for example data such as amount of fuel, speed, engine speed in driving conditions；
3) data are not in validity period, also do not need to be saved as data information；This kind of data have not neededIt is handled again, accident early warning data more for example having occurred and that etc..
Therefore, data examine that layer 200 will judge whether data are expired based on the property of data and life span in real time, intoAnd reject stale data.Specifically, it is necessary first to judge whether the data have that " data are effective according to the property of each dataThe limitation of phase ".Such as the above-mentioned first kind and secondary sources.Wherein, for the data of effective limitation (i.e. validity period limit),System is its setting " maximum real-time time delay value ".When judging, by " life span " of data and " maximum real-time time delay value " intoRow compares, if " life span " of data is greater than " maximum real-time time delay value ", illustrates that the data are out of date and does not need to carry out againProcessing, system reject the data, and otherwise the data can be transmitted to generating date layer 300 by system.Wherein, data is " rawDeposit the time " it is equal to the system actual treatment time and subtracts renewal time of data the last time.
Generating date layer 300, the module of the level are mainly responsible for cleaning and fix errors data in real time.Passing through numberAfter the heavy selection for examining layer 200 when factually, data are delivered to generating date layer 300.On the one hand, as previously mentioned, numberIt examines that layer 200 only eliminates stale data therein according to " property " and " life span " of data when factually, therefore may be used alsoIt can need further to reject there is also other abnormal data.On the other hand, after rejecting these abnormal datas, in order to keepThe integrality of data processing, information analysis, it is also necessary to which the data of missing are compensated.
In the present embodiment, generating date layer 300 is preferably cleaned and fix errors using window sliding technology in real timeData.According to acquisition terminal data discrete characteristic, some data is related with sequence data before.It therefore, can be withCurrent collected data and front and back one piece of data are placed in the window defined using " window sliding " technology, using beforeRelated data to determine whether exist mistake data.Certainly, according to the passage of acquisition time, this window can be always in cunningDynamic, each window sliding just represents new data of having come in, and produces a new data segment.
Data real-time storage layer 400, the module of the level are mainly responsible for the real-time storage of car networking data, the content of storageCover the data such as car networking real time data, historical data, user data and system setting, to carry out unified maintenance to itWith management, it is reached for the analysis of system and calculates to provide the purpose that all kinds of basic datas are supported.
In the present embodiment, data real-time storage layer 400 is preferably located using cyclic buffer mode to by data in real timeIt manages the processed data of layer 300 and carries out real-time storage.In this regard, needing to divide in the memory modules of data real-time storage layer 400The equal buffer area of two pieces of capacity, and the two buffer areas is identical to the priority of data buffer storage.Specific cachingOperating process refers to embodiment two.
Certainly, the composition of above system can be without being limited thereto, can also further expand as needed, no longer superfluous hereinIt states.
Second embodiment
In order to illustrate more clearly of technical solution of the present invention, letter is made to the working method of above system with reference to the accompanying drawingSingly introduce.It is already mentioned above, in vehicle operation, due to the running environment of vehicle complexity, the number that is often abnormalAccording to these abnormal datas include the data containing error, random error data, there are also the data for occurring to lose or be distorted, thereforeIt needs to carry out these data to reject or appropriate fill a vacancy.On the other hand, although sometime putting the collected vehicle of terminalCarrying data is a discrete data variable, but some data in an actually time series, the data and whenBetween in sequence before data be associated.Therefore, using relevant historical data before, can data to missing orThe abnormal data of rejecting is predicted, is filled up, to obtain normal data sequence, and then is provided specifically, accuratelyWith the operation of reliable vehicle and status information.Fig. 2 is the car networking real-time data processing system processing number of the embodiment of the present invention twoAccording to method flow diagram.Wherein:
After terminal data by examining in real time, need to be cleaned in real time.Here " real-time " cleaning refers to will be at that timeAcquire currently processed, data acquiring frequency is very short, usually primary in 1s or so acquisition.But this does not represent one number of acquisitionAccording to a data will be handled, because not knowing that collected individual data is normal or abnormal data,One is also needed to be delayed herein, between usually 20s to 60s.Then it needs one to define, be judged using " window sliding "Whether abnormal data is deposited.According to the passage of acquisition time, this window will slided always, and window sliding just representsNew data produce a new data segment, this that the cleaning that data are exactly real-time is handled by sliding window.Specifically:
1) terminal sends data in real time；
2) sliding window is inquired；
3) judge whether data are normal:
If normal, be delayed 20s, and window slides into next data；
Otherwise, data exception executes step 4)；
4) data are handled, that is, utilizes data exception processing routine cleaning data and repairing missing data；
5) data acquisition system that obtains that treated；
6) return step 2), continue to judge.
About data real-time storage.As previously mentioned, car networking terminal data acquisition transmission speed is fast, data volume is big, in order toIt keeps the real-time of high-speed data processing process and guarantees the integrality of data, the data real-time storage layer of system introduces " doubleBuffer area " technology namely cyclic buffer mode create the identical buffer area of two buffer levels.When wherein first slowAfter rushing area and having expired, system control goes to second buffer area, and the data of first buffer area are then transmitted simultaneously during this periodIt stores in disk database.Detailed process is as follows:
When starting to receive with storing data, first first buffering area is written in collected data by system；Work as first buffering areaIt writes completely, again second buffering area is written in collected data by system, and application program can according to need the first buffering of taking-up at this timeThe data in area execute storage operation；After second buffering area is fully written, system can return to the section start of first buffering area, with coveringFirst buffering area is written in freshly harvested data by the mode of legacy data, and the data that application program takes out second buffering area at this time executeStorage operation；Repeatedly.
To sum up, car networking real-time data processing method provided by the invention and system, can be vehicle management departmentThere is provided in real time specifically, accurate and reliable vehicle operation and status information.
The above, specific implementation case only of the invention, scope of protection of the present invention is not limited thereto, any ripeThose skilled in the art are known in technical specification of the present invention, modifications of the present invention or replacement all should be in the present inventionProtection scope within.