CN102081641A - Design of GPU-based real-time data stream high-speed matching analysis system - Google Patents

Abstract

Translated fromChinese

本发明提供一种基于GPU(图形处理单元)的实时数据流高速匹配分析系统的设计，尤其是在利用GPU提高实时数据流高速匹配分析的效率方面做出了贡献。它至少包括以下部分：一套数据流接收装置，一套海量数据阶段化存储装置，一套完整的实时数据流高速匹配分析系统，以及一套移动智能终端；其中，数据流接收装置接收监测数据，并将数据存储到分布式的工作站中，海量数据阶段化存储装置负责把工作站上的数据分类规整并阶段化存储，之后当接收到查询实时数据匹配分析请求时，系统首先检查用户是否已授权，确认无误后将在实时数据流高速匹配分析系统中进行匹配分析处理，分析结果由无线通讯硬件负责，与相应的代理商协定后，发送到便携式终端。本发明能够有效地在便携装置上对海量数据流的监控及分析，分析效率较高，特别适合外出作业的工程人员。

The invention provides a design of a GPU (Graphics Processing Unit)-based high-speed matching analysis system for real-time data streams, and especially contributes to improving the efficiency of high-speed matching analysis of real-time data streams by utilizing the GPU. It includes at least the following parts: a set of data stream receiving device, a set of massive data staged storage device, a complete set of real-time data stream high-speed matching analysis system, and a set of mobile intelligent terminal; among them, the data stream receiving device receives monitoring data , and store the data in the distributed workstations. The mass data staged storage device is responsible for classifying and storing the data on the workstations in stages. Afterwards, when receiving a real-time data matching analysis request, the system first checks whether the user is authorized. After confirmation, the matching analysis process will be carried out in the real-time data stream high-speed matching analysis system. The analysis results will be in charge of the wireless communication hardware, and will be sent to the portable terminal after agreement with the corresponding agent. The invention can effectively monitor and analyze massive data streams on a portable device, has high analysis efficiency, and is especially suitable for engineering personnel who go out to work.

Description

Design based on the real time data stream high speed The matching analysis system of GPU

The invention provides the design of a kind of real time data stream high speed The matching analysis system, especially utilize GPU to improve the design of real time data stream matching inquiry speed.

Background technology

Real time data stream detects and is stored as the bottleneck of the business that detects for data stream, how to design a kind of brand-new mass data stage memory storage, this apparatus features is the data in the distributed work station regularly can be carried out classified finishing, and the staff is needed the real time data stream quick calling of match query.And expired data stream stores into independently and prepares against the function of doing long term data excavation rational analysis in the massage storage in the memory mechanism.

The monitoring means of real time data stream is brought in constant renewal in and to the continuous demand of result data degree of accuracy, make to real time data stream and national standard accurately, The matching analysis becomes the professional problem that needs solution of current real time data stream monitoring fast.Compare with traditional data structure, characteristics such as it is real-time, continuous, endless that data stream has, these singularity of data stream have brought new opportunities and challenges for the research of the information processing technology and application. and the characteristics of data stream have determined that its main match query mode is the continuous-query coupling.Continuous-query is meant execution continuously for a long time after an inquiry is established, along with the arrival of new data will constantly produce new match query result.Because data stream is the set of series of discrete state, if the data base querying matching way that adopts tradition to calculate based on CPU, be difficult to obtain fast match query result accurately under a lot of querying conditions, because traditional data stream storage mode all deposits mass data flow in storage mediums such as disk, when carrying out the match query order, need carry out a large amount of I/O operations, its response speed is slow, is difficult to adapt to the requirement of real-time match query.In fact the data of latest stage are more important than historical data, so the data of the latest stage often that people are concerned about in the data stream The matching analysis.

Summary of the invention

The invention provides the design of a kind of real time data stream high speed The matching analysis system, particularly, can be used on the portable terminal real time data stream being carried out the high speed The matching analysis based on the design of the real time data stream high speed The matching analysis system of GPU.

Native system mainly comprises the data stream receiving trap, mass data stage memory storage, the complete real time data stream high speed The matching analysis system of a cover, wireless telecommunications system, and mobile intelligent terminal; Implementation procedure is such, observation satellite is passed to ground data stream receiving station with observed result as data stream, the station spreads Monitoring Data to data stream receiving station, the data stream receiving trap of these two kinds of forms has received Monitoring Data stream, and data stream is stored into the workstation of distribution various places, mass data stage memory storage is responsible for the regular and stage storage the data qualification on the workstation, afterwards when receiving the The matching analysis request of mobile intelligent terminal, system checks at first whether the user is authorized to, confirming will carry out from the system of real time data stream high speed The matching analysis after errorless The matching analysis handles, after The matching analysis result is responsible for by wireless telecommunications system and corresponding commission merchant reaches an agreement on, send on the mobile intelligent terminal equipment.The present invention can carry out The matching analysis to real time data stream effectively on portable unit, The matching analysis speed is very fast, accuracy rate is higher, the engineering staff of the operation that is particularly suitable for going out.

Description of drawings

Fig. 1 is the synoptic diagram of disclosed real time data stream high speed The matching analysis system.

Fig. 2 is the inquiry frame diagram of disclosed real time data stream high speed The matching analysis system.

Fig. 3 is the querying flow figure of disclosed real time data stream high speed The matching analysis system.

Fig. 4 is the inside configuration synoptic diagram of disclosed real time data stream high speed The matching analysis system.

Embodiment

Technological difficulties to be solved by this invention be how satisfy to the real time data matching inquiry provide at a high speed, the data stream The matching analysis mechanism of pinpoint accuracy, to tackle the application demand of various data streaming service information.

As Fig. 1, provide the real time data flow receiving device to comprise various different classes of monitoring devices, detect equipment such as data sink, workstation.Monitoring device has observation satellite, the station etc., has different monitoring equipments towards different field; Each monitoring equipment all has corresponding with it data stream receiving trap, detects DRS, station DRS etc. such as satellite.These devices provide the most original measured data for us, these data can be according to instruction storage in distributed workstation, the data of this moment are not through unified data preprocessing process, because data volume is huge, so will take a large amount of stand-by period of user when carrying out the The matching analysis instruction, be not suitable for outworker's query calls at any time.Improved method adopts mass data stage memory storage and based on the real time data stream high speed The matching analysis system of GPU, to storing and computing velocity all is once tremendous improvement and optimization.

Mass data stage memory storage is used for preserving being stored in the distributed data base of data stream that each workstation sends and stageization.This is comprising the pre-service to raw data, data qualification, the operation of warehouse-in.The data pre-service is responsible for the data that transmit are carried out operations such as data cleansing, denoising, and the purpose in this step is to reduce data redundancy, data inconsistency, avoids reading invalid dirty data.Data qualification is that raw data is classified by attribute information and format differences, is different classes of data such as station data and satellite monitoring data, must be stored in the different databases.Data qualification finishes, and next is exactly these data are carried out in-stockroom operation.

Real time data stream high speed The matching analysis system based on GPU mainly comprises intermediary device, matched-field processors.The query requests of system receiving terminal is obtained data from heterogeneous database.Query Result must just can turn back on the portable terminal by wireless telecommunications hardware and the agential transfer of wireless telecommunications.The wireless telecommunications hardware is here realized Query Result is passed to the commission merchant, is responsible for transferring electromagnetic signal to digital signal by the commission merchant, sends to terminal request.

As Fig. 2, the real time data stream high speed The matching analysis system based on GPU among the present invention mainly is made up of intermediary device, matched-field processors.Intermediary device receives user's query requests, is responsible for local heterogeneous data source mutual by intermediary device.System offers global schema of user, and the data of mass data stage memory storage are carried out matching inquiry.Matched-field processors is carried out query requests, and the result is returned to intermediary device, and intermediary device carries out after the overall treatment result being returned to the user.Wherein intermediary device is made up of integrated manager and query processor, and bookkeepings such as the responsible increase to distribution local data of integrated manager, deletion can easily be managed each data source by global map, realize the data integration management; Be responsible for local data simultaneously and arrive the integrated of global data.Matched-field processors is responsible for accepting user's query requests, and Query Result is returned.

Continuing referring to Fig. 3, is the system handles process flow diagram to the data continuous query, mainly comprises following steps: at first open the web browser in the terminal (following is example with the mobile phone), open corresponding website; Then carry out query manipulation; System checks earlier whether this user has authority, if not then getting back to login interface; If this user has authority, then enter real time data stream high speed The matching analysis system and carry out query manipulation, system returns to terminal with Query Result, finishes.

Continue referring to Fig. 4, this figure is based on the core processing unit figure of the data stream matched-field processors of GPU among Fig. 2.This module is the nucleus module of system, is based on GPU and carries out high-performance calculation, is divided into data stream Uniform Scaling processing, Normalization processing and three steps of data stream matching treatment.The data stream of in calculator memory data stream being carried out that UniformScaling handles, Normalization handles and will change is sent in the GPU video memory, the video memory of GPU uses Scaling Subsequence Matrix memorying data flow and utilizes the moving window model and standardized data stream carries out the high speed The matching analysis, after at last Query Result being gathered, feed back to the user., because the characteristics of GPU are the intensive computings at a large amount of data of the same type, so the advantage that the numerical evaluation of matrix-type is operated makes the matched data flow operation more quick, matching precision is higher.

In sum, the present invention is based on the design of the real time data stream high speed The matching analysis system of GPU, Monitoring Data by various monitoring device aggregated data stream types, and through storing in the mass data stage memory storage after the data pre-service, portable terminal can be carried out corresponding real time data stream matching request by authority at any time.

The above only is an illustrative, but not is restricted.This description should not be construed as limitation of the present invention, and any spirit of the present invention and category of not breaking away from all should be contained within the application range.

Claims (8)

Translated fromChinese

1.基于GPU的实时数据流高速匹配分析系统的设计，其特征在于，应该包括有：1. The design of the real-time data flow high-speed matching analysis system based on GPU is characterized in that it should include:一套数据流接收装置，用来接收由监测所得的海量卫星监测数据和台站数据；A set of data stream receiving device is used to receive massive satellite monitoring data and station data obtained by monitoring;一套海量数据阶段化存储装置，该装置与分布式的工作站相连接，负责定期将工作站数据归类整理，并阶段化存储；A set of staged storage device for massive data, which is connected to distributed workstations and is responsible for regularly sorting and storing data from workstations;一套实时数据流高速匹配分析系统，供接收移动智能终端的匹配分析请求，从海量数据阶段化存储装置中与标准数据数值进行匹配，最后根据授权情况决定是否返回分析结果；A set of real-time data stream high-speed matching analysis system for receiving matching analysis requests from mobile smart terminals, matching with standard data values from massive data staged storage devices, and finally deciding whether to return the analysis results according to the authorization status;一套无线通讯设备，实时数据流高速匹配分析系统产生的分析结果需要通过无线通讯器材传输到便携设备上，无线通讯硬件起到信号中转的作用；A set of wireless communication equipment, the analysis results generated by the real-time data flow high-speed matching analysis system need to be transmitted to portable devices through wireless communication equipment, and the wireless communication hardware plays the role of signal relay;一套移动智能终端，可便携式设备。A set of mobile smart terminals, portable devices.2.根据权利要求1所述的数据流接收装置，其特征在于该数据流接收装置是由台站、卫星监测结果接收装置等接收装置组成的多方位的监测及收集装置。2. The data stream receiving device according to claim 1, characterized in that the data stream receiving device is a multi-directional monitoring and collecting device composed of receiving devices such as stations and satellite monitoring result receiving devices.3.根据权利要求1所述的海量数据阶段化存储装置，其特征在于该海量数据阶段化存储装置将阶段性的存储从各工作站中接收到最新的数据流。3. The mass data staged storage device according to claim 1, characterized in that the mass data staged storage device receives the latest data flow from each workstation for staged storage.4.根据权利要求1所述的实时数据流高速匹配分析系统，其特征在于异构地理数据查询系统包括以下部分组成：4. The real-time data flow high-speed matching analysis system according to claim 1, wherein the heterogeneous geographical data query system comprises the following components:一套中介器模块，接收用户的查询请求，把当前时刻之前的固定时长内的数据以数据流的形式调用并传给分析模块，最后中介器在分析模块进行综合处理后把结果返回给用户。A set of intermediary module receives the user's query request, calls the data within a fixed period of time before the current time in the form of data stream and transmits it to the analysis module, and finally the intermediary returns the result to the user after comprehensive processing by the analysis module.一套匹配处理器模块，负责将实时被查询数据与其对应的国家标准利用GPU强大的并行计算能力进行匹配分析，分析其相似匹配的程度，并把结果转换后返回给中介器。A set of matching processor modules is responsible for matching and analyzing the real-time queried data with the corresponding national standards using the powerful parallel computing capability of the GPU, analyzing the degree of similar matching, and returning the results to the intermediary after conversion.5.根据权利要求4所述的实时数据流高速匹配分析系统，其特征在于中介器由请求接收管理器和匹配处理器两部分组成。5. The real-time data flow high-speed matching analysis system according to claim 4, characterized in that the intermediary is composed of a request receiving manager and a matching processor.6.根据权利要求4所述的实时数据流高速匹配分析系统，其特征在于匹配处理器模块将两数据流进行高速分析计算匹配，并执行分析结果格式转换。6. The real-time data stream high-speed matching analysis system according to claim 4, characterized in that the matching processor module performs high-speed analysis, calculation and matching of two data streams, and performs format conversion of the analysis results.7.根据权利要求1所述的无线通讯硬件，其特征在于将查询分析结果转化成易于无线传输的格式，可以通过代理商转为电磁信号。7. The wireless communication hardware according to claim 1, characterized in that the query and analysis results are converted into a format that is easy for wireless transmission, and can be converted into electromagnetic signals through an agent.8.根据权利要求1所述的可便携式设备，其特征在于它是可移动的，智能手机或PDA。 8. The portable device according to claim 1, characterized in that it is a mobile, smartphone or PDA. the

Images