A kind of distributed meteorological data life cycle building and monitoring methodTechnical field
The present invention relates to the building of meteorological data life cycle and monitoring technology field, specially a kind of distributed meteorological numberAccording to life cycle building and monitoring method.
Background technique
In the development process of Meteorological Modernization, various real-time observed datas can be more and more in the future, in addition to numberAccording to accuracy other than, the timeliness of data is required also higher and higher, data age is that reflection operation system design is reasonableThe important indicator of property and data-handling efficiency.Carrying out real time monitoring to data is to ensure the effective means of data age.
Process meteorological data logic is relatively easy before, usually can be completed in unified node and environment, thus supervisesControl work is also based only on a certain database or certain server carries out, and does not need in face of being scattered in different data processing nodeStorage and computing resource, without the data relationship and monitoring timeliness for considering complexity.It is rapid with meteorological data data volumeIncreasing, many meteorological datas are also required to handle using distributed computing, this is just needed in face of complicated network environment, and byIn reasons such as data safeties, data and process flow dispersion are often located in different network environments, traditional technology can notMeet new business demand, thus need to develop design it is a kind of based on distributed storage and calculate, can ensure service invariant,All kinds of meteorological datas of real-time dynamic monitoring arrive report, the monitoring system of warehouse situation and each node operation processing status.
Summary of the invention
The purpose of the present invention is to provide a kind of distributed meteorological data life cycle building and monitoring methods, to solveExisting technological deficiency and inaccessiable technical requirements.
To achieve the above object, the invention provides the following technical scheme: a kind of distributed meteorological data life cycle structureIt builds and monitoring method, this method is by data resource acquisition, synchronization program, distributed treatment, distributed storage, message module, prisonModule section composition is controlled, method includes the following steps: step S1: user's configuration data download items in Web page；StepS2: synchronization program consumes message, and it is synchronous to carry out data；Step S3: the synchronized program transportation of data to data service platform is distributedFormula storage；Step S4: user handles plug-in unit in Web page configuration data, is plug-in card configuration data to be processed, required CPUWith memory source, output result；Step S5: scheduler executes plug-in unit according to configuration；Step S6: data processed result, branch are obtainedHold specific service.
Preferably, S1 is included the next steps: step S101: configuration access data source, including data source server address orData code and interface, access way, update the frequency, filename filtering rule at account information；Step S102: configuration access numberAccording to attribute, including data format, time-zone information, returns to format, whether compresses covering；Step S103: configuration message format, includingTopic field, using coding, data encoding, error coded；Step S104: database, distribution are stored in respectively according to data attributeFormula file system is stored in object or file system with logic partitioning mode.
Preferably, S3 is the following steps are included: step S301: the essential information of configuration task, description including task, allPerson, function, the description using data, output data description；Step S302: plug-in unit is designed according to service logic and executes sequence, is led toPulling mode arrangement plug-in unit sequence is crossed, and connects plug-in unit, completes the topological diagram between plug-in unit；Step S303: it is being held according to plug-in unitThe coupled modes of plug-in unit are arranged in dependence in row, and there are four types of coupled modes: data coupling, logic coupling, Global-Coupling,Message coupling；Step S304: being arranged the executive plan of task, mainly the execution time, if repeats, is finally stored in configurationIn task flow, it can be changed according to business and update, adjust task plug-in unit, executive plan.
Preferably, the data resource offer includes the meteorological data from Intranet, FTP, DMZ, public network memory node；InstituteSynchronization program is stated by, to the transmission of data, the plug set will be right after the unified management of all kinds of plug-in units in different network environmentsMeteorological data parsing, operation and processing are equipped with the reception to information in whole system production process in the message module and disappearTake interface, the monitoring module monitors the building of entire data life period by message.
Preferably, the message-driven builds the Web service page by agency, includes in page marshal data access taskData source address sends message after the completion of data downloading, synchronization program listens to downloading and completes message, is connected to Intranet by gatewayData transmission is completed with DMZ；Data are stored by the plug set that data service platform is developed based on Distributed Architecture again, are transportedIt calculates, the work of processing various aspects；Operating status message, the exception information generated in data life period, synchronized program reachService is provided in the database of ECS network segment deployment.
Compared with prior art, beneficial effects of the present invention are as follows:
1. the monitoring data transmission experience of the comprehensive previous single node of this method, to data resource, service system, computing environment and is answeredImmediately monitoring is carried out with health status, Dynamically Announce service success or failure recognizes feature, provides Data View abundant,And to service unusual condition alert, in addition, monitoring system be also responsible for collect fault message, associated upstream and downstream environment withAnd procedure information, and then improve service quality；
Calculation one is deposited 2. realizing by the plug set being equipped with, incorporates storage and calculating, changes current data storage and meterThe mode that point counting is not individually present and works, distributed computing resource distribution, program scheduler, makes user can handle a large amount of numberAccording to without considering resource problem, the data processing plug-in unit of extension can be coupled, state is mutually obtained by message mechanism, realizes and insertsCollaborative work between part；
3. dynamically configuring to data resource by Web, become source, characteristic, storage mode, scheduling information of data etc.Must be more clear with it is easy to maintain；
4. solving imeliness problem using message mechanism, traditional automatic regular polling mode is replaced by message-driven；
5. realizing that information of the data file under complex environment is collected, the data obtained on different nodes are monitored using message, are answeredWith, node resource, program health status information, message synchronization is realized by gateway or agency between different segment；
6. devising the Index Establishments such as integrated application rule agreement, data encoding, data generation time, network topology data flowThe method of Cheng Zidong building, can be directly viewable the overview of data, and quickly to catastrophic failure and should carry out fault tracingAnd association analysis, to realize the effective monitoring of data flow under distributed environment.
Detailed description of the invention
Fig. 1 is the method for the present invention comprising modules and work flow diagram；
Fig. 2 is that data life period of the present invention constructs flow chart；
Fig. 3 is the synchronous downloading flow chart of message-driven of the present invention.
Specific embodiment
Below in conjunction with the present invention, technical scheme in the embodiment of the invention is clearly and completely described, it is clear thatDescribed embodiments are only a part of the embodiments of the present invention, instead of all the embodiments.Based on the implementation in the present inventionExample, every other embodiment obtained by those of ordinary skill in the art without making creative efforts belong toThe scope of protection of the invention.
The present invention provides a kind of technical solution: a kind of distributed meteorological data life cycle building and monitoring method, shouldMethod is made of data resource acquisition, synchronization program, distributed treatment, distributed storage, message module, monitoring module part,Method includes the following steps: step S1: user's configuration data download items in Web page；Step S2: synchronization program disappearsIt is synchronous to carry out data for breath；Step S3: the synchronized program transportation of data to data service platform distributed storage；Step S4: userPlug-in unit is handled in Web page configuration data, is plug-in card configuration data to be processed, required CPU and memory source, output knotFruit；Step S5: scheduler executes plug-in unit according to configuration；Step S6: obtaining data processed result, supports specific service.
S1 is included the next steps: step S101: configuration access data source, including data source server address or data codeWith interface, account information, access way, the update frequency, filename filtering rule；Step S102: configuration access data attribute, packetData format is included, time-zone information, format is returned, whether compresses covering；Step S103: configuration message format, including topic wordSection, using coding, data encoding, error coded；Step S104: database, distributed document are stored in respectively according to data attributeSystem is stored in object or file system with logic partitioning mode.
S3 is the following steps are included: step S301: the essential information of configuration task, description, the owner including task, functionEnergy, the description using data, output data description；Step S302: plug-in unit is designed according to service logic and executes sequence, passes through draggingMode arranges plug-in unit sequence, and connects plug-in unit, completes the topological diagram between plug-in unit；Step S303: in commission according to plug-in unitThe coupled modes of plug-in unit are arranged in dependence, and there are four types of coupled modes: data coupling, logic coupling, Global-Coupling, message couplingIt closes；Step S304: being arranged the executive plan of task, mainly the execution time, if repeats, configuration is finally stored in task flowIn, it can be changed according to business and update, adjust task plug-in unit, executive plan.
Data resource offer includes the meteorological data from Intranet, FTP, DMZ, public network memory node；The synchronization programBy, to the transmission of data, the plug set is by the unified management of all kinds of plug-in units later to meteorological data solution in different network environmentsAnalysis, operation and processing, the interior reception being equipped with to information in whole system production process of the message module and consumption interface, it is describedMonitoring module monitors the building of entire data life period by message.
Message-driven builds the Web service page by agency, includes data source address in page marshal data access task,Message is sent after the completion of data downloading, synchronization program listens to downloading and completes message, is connected to Intranet by gateway and DMZ completes numberAccording to transmission；Data are stored by the plug set that data service platform is developed based on Distributed Architecture again, operation, processing each sideThe work in face；Operating status message, the exception information generated in data life period, synchronized program reach the deployment of ECS network segmentDatabase in service is provided.
Embodiment one
Meteorological data life cycle described in the present embodiment constructs and to monitor a kind of distributed method meteorological in the present embodimentThe step of data life period building is with monitoring is as follows:
Step S1: user's configuration data download items in Web page；
Step S2: user handles plug-in unit in Web page configuration data, is plug-in card configuration data to be processed, required CPU and interiorDeposit resource, output result；
Step S3: scheduler executes plug-in unit according to configuration；
Step S4: obtaining data processed result, supports specific service.
Embodiment two
Meteorological data life cycle described in the present embodiment constructs and to monitor a kind of distributed method meteorological in the present embodimentThe step of data life period building is with monitoring is as follows:
Step S1: user's configuration data download items in Web page；
Step S101: configuration access data source, including data source server address or data code and interface, account information, connectEnter mode, update the frequency, filename filtering rule；
Step S102: configuration access data attribute, including data format, time-zone information, return format, whether compress covering；
Step S103: configuration message format, including topic field, using coding, data encoding, error coded；
Step S104: being stored in database, distributed file system according to data attribute respectively or is stored in pair with logic partitioning modeAs or file system in；
Step S2: synchronization program consumes message, and it is synchronous to carry out data；
Step S3: the synchronized program transportation of data to data service platform distributed storage；
Step S301: the essential information of configuration task, description, the owner, function including task, the description, defeated using dataData describe out；
Step S302: designing plug-in unit according to service logic and execute sequence, arranges plug-in unit sequence by pulling mode, and connect and insertPart completes the topological diagram between plug-in unit；
Step S303: according to the dependence of plug-in unit in commission, the coupled modes of plug-in unit are set, and there are four types of coupled modes: numberAccording to coupling, logic coupling, Global-Coupling, message coupling；
Step S304: being arranged the executive plan of task, mainly the execution time, if repeats, configuration is finally stored in taskIn stream, it can be changed according to business and update, adjust task plug-in unit, executive plan；
Step S4: user handles plug-in unit in Web page configuration data, is plug-in card configuration data to be processed, required CPU and interiorDeposit resource, output result；
Step S5: scheduler executes plug-in unit according to configuration；
Step S6: obtaining data processed result, supports specific service.
The present invention uses message driving mechanism: message mechanism is applied to meteorological data life cycle and constructs and monitors, it canThe imeliness problem for effectively solving meteorological data realizes messaging and transmitting under complex network environment by agency or gatewayData, and support real-time Transmission, encrypted transmission.About to meteorological data coding, using message formats such as coding, error message codingsIt is fixed, monitoring system by monitoring various types of messages, it can be achieved that monitoring to data Life cycle, including data mode, running environment,Using health；The method that the present invention uses automatic flow building: since stream compression link is complicated, meteorological data is various informative,Become unrealistic by manual maintenance data flow, for this case, establishes the method that data flow constructs automatically；It is firstFirst, meteorological data has stringent treatment process from access, transmission, processing, storage to service, in order to ensure the building of automatic flowProcess, detailed design integrated application rule agreement, message format agreement, such as data encoding, data file state, upper layer linkCoding, lower layer's link coding.Secondly, according to service logic relationship, building meteorological data, agency, product or the example section of servicePoint.Transferred the registration of Party membership, etc. from one unit to another according to the data encoding, data mode, data generation time, the node-flow that are provided in message and network topology etc.Index formed data flow network, finally in the form of data topology figure access, processing, production and the application of display data it is completeWhole life cycle；The monitoring data transmission experience of the comprehensive previous single node of this method, to data resource, service system, operation ringBorder and application health status carry out immediately monitoring, and Dynamically Announce service success or failure recognizes feature, provides number abundantAccording to view, and to service unusual condition alert, in addition, monitoring system be also responsible for collect fault message, it is associated up and downLantern border and procedure information, and then improve service quality；It is realized by the plug set being equipped with and deposits calculation one, incorporate storageAnd calculating, it changes current data storage and calculates the mode for independently existing and working, distributed computing resource distribution, journeySequence scheduling makes user can handle a large amount of data without considering resource problem, can couple the data processing plug-in unit of extension, leads toIt crosses message mechanism and mutually obtains state, realize the collaborative work between plug-in unit；Replace traditional automatic regular polling side by message-drivenFormula；Realize that information of the data file under complex environment is collected, monitored using message the data obtained on different nodes, using,Node resource, program health status information realize message synchronization by gateway or agency between different segment；Synthesis is devised to answerThe method constructed automatically with the Index Establishments such as regular agreement, data encoding, data generation time, network topology data flow,It can be directly viewable the overview of data, and quickly to catastrophic failure and fault tracing and association analysis should be carried out, thus realThe effective monitoring of data flow under existing distributed environment.
It is obvious to a person skilled in the art that invention is not limited to the details of the above exemplary embodiments, Er QieIn the case where without departing substantially from spirit or essential attributes of the invention, the present invention can be realized in other specific forms.Therefore, no matterFrom the point of view of which point, the present embodiments are to be considered as illustrative and not restrictive, and the scope of the present invention is by appended powerBenefit requires rather than above description limits, it is intended that all by what is fallen within the meaning and scope of the equivalent elements of the claimsVariation is included within the present invention.
In addition, it should be understood that although this specification is described in terms of embodiments, but not each embodiment is only wrappedContaining an independent technical solution, this description of the specification is merely for the sake of clarity, and those skilled in the art shouldIt considers the specification as a whole, the technical solutions in the various embodiments may also be suitably combined, forms those skilled in the artThe other embodiments being understood that.