A kind of lightweight data processing method based on chain of messagesTechnical field
The present invention relates to distributed computing technology field, particularly relate to a kind of lightweight data processing side based on chain of messagesMethod.
Background technology
One distributed system needs to process different types of message, and the process logic of message was both had something in common, and also couldNeed special process logic according to the difference of type of message.We typically to process the general character of message using message-oriented middlewareProblem, its process step is such that
1st, typically there are three roles in message-oriented middleware: message-oriented middleware, consumer, ISP；
2nd, consumer's assembling request of data message is sent to message；
3rd, customer requests' message is transmitted to each ISP and monitors queue by message-oriented middleware, and ISP receivesRequested service message；
4th, the asynchronous response message returning request message of ISP；
5th, the response message of customer requests' message is returned to the response queue of corresponding consumer, consumer by message-oriented middlewareReceive the response message of request, and do respective handling；
6th, the data needed for ISP generates consumer, and send message to message-oriented middleware；
7th, notice message is transmitted to each consumer and monitors queue by message-oriented middleware, and consumer receives notice business reportLiterary composition；
8th, the asynchronous response message returning notice message of consumer.
But, message-oriented middleware there is also problem below in the little distributed environment of size of message:
1st, message-oriented middleware is an independent message engine, can take part of server resource, for size of message notIt is not necessary that increasing such expense for big Distributed Application, and introduce the role of the third party, message (message) is passingTemporary through an excessive layer queue during defeated, the real-time of message can be lowered.
2nd, message-oriented middleware does not have the treatment mechanism of flexible design to message processing logic, once message processing logic becomesChange, need to recompile realization.
3rd, message-oriented middleware does not design the special handling logic mechanism to message, to the exploitation of particular service logic notProfit.
Content of the invention
Present invention solves the technical problem that being to provide a kind of lightweight data processing method based on chain of messages, solve meshThe problem that the Message Processing that front universal information middleware exists is dumb, development efficiency is low, message real-time is low；
The technical scheme that the present invention solves above-mentioned technical problem is:
Including step:
Upstream message enters source processing chain module after being received from message channel；
After message sequentially passes through the process of the up process logic of each processor on chain, it is cached in upstream message queue；
Upstream message queue management thread takes out message from message queue and puts into message threads pond；
Message is parsed into object or structure by message threads pond；
Message observer decides whether received message according to the type of message object or structure, and carries out special industry to itBusiness logical process；
Process response or user's request changes into message through message resolution module and is cached in downstream message queue；
Message queue is saved in downstream message in queue to be sent；
In queue to be sent, message is fallen out inbound message process chain in order, sequentially passes through the descending place of each processor on chainSend to message channel after reason logical process；
Described message is divided into upstream message and downstream message；Upstream message refers to that system receives from message channel and needsCarry out the request of the corresponding command operation, downstream message then refer to command operation processed after response or user need sendThe request gone out；
Described source processing chain is the chain type set of Message Processing instrument, at upstream message process chain and downstream messageReason chain；The Message Processing instrument that upstream message process chain includes has message unpacking, message decompression, message deciphering and message authentication；The Message Processing instrument that downstream message process chain includes then has message authentication, message encryption, message compression and message to beat accordinglyBag；
Described message threads pond refers to the set of each Message Processing thread, have by message be parsed into each object orThe function of person's structure；
Described message observer refers to carry out the interface of various different disposal for the message object parsingObject, including daily record observer, audit observation person, virtual machine observer, logical volume observer；Message observer adopts observerModel Design is realized, and same message is supported successively by multiple observer's reception processings；Observer Pattern is one in Programming DesignPlant programming realization method.
The tool set of described described upstream message process chain and downstream message process chain supports dynamic scalable.
Described up-downgoing message queue supports internal memory form database form；
Described internal memory form message queue refers to apply for that in internal memory a certain size space, as message format, hasThe characteristic of efficiency high；
Described database form refers to first be saved in message data base, waits for the arrival of news and processes the scheduling of thread, hasPersistence characteristic, supports that service fault recovers the feature that message is processed again.
Described message observer's particular service logical process refers to carry out log recording similar to daily record observer, auditObserver is audited, and virtual machine observer carries out creating virtual machine, the logical process of closing virtual machine business.
The present invention program has the beneficial effect that:
1st, the framework of the present invention is lightweight, less to the expense of the extra increase of system, and the real-time of message is high.
2nd, the framework of the present invention is high flexible, and the process logic in source processing chain can be passed through to configure to carry out on demandIncrease and decrease, need not recompile and can run；Increase the special handling logic to message to be relatively easy to.
3rd, the method for the present invention can develop distributed treatment application at short notice, and the processor in source processing chain is allIt is the general purpose module processing logic development according to general character, only needing to be configured or done a small amount of coding in actual use isCan.
4th, the quality of distributed application development can be improved, increase CSAT.Due to applying this framework can accomplish not compileWrite code or only write very small amount code, therefore decrease the mistake encoding generation in development process, improve software quality.
Brief description
The present invention is further described below in conjunction with the accompanying drawings:
Fig. 1 is flow chart of the present invention；
Fig. 2 processes framework exploded view for upstream data of the present invention；
Specific embodiment
As shown in figure 1, the present invention is comprised the steps: based on the lightweight data processing method of chain of messages
Upstream message enters source processing chain module after being received from message channel；
After message sequentially passes through the process of the up process logic of each processor on chain, it is cached in upstream message queue；
Upstream message queue management thread takes out message from message queue and puts into message threads pond；
Message is parsed into object or structure by message threads pond；
Message observer decides whether received message according to the type of message object or structure, and carries out special industry to itBusiness logical process；
Process accordingly or user's request changes into message through message resolution module and is cached in downstream message queue；
Message queue is saved in downstream message in queue to be sent；
In queue to be sent, message is fallen out inbound message process chain in order, sequentially passes through the descending place of each processor on chainSend to message channel after reason logical process.
Described message is divided into upstream message and downstream message；Upstream message refers to that system receives from message channel and needsCarry out the request of the corresponding command operation, downstream message then refer to command operation processed after response or user need sendThe request gone out.
Source processing chain is the chain type set of Message Processing instrument, processes including upstream message process chain and downstream messageChain；The Message Processing instrument that upstream message process chain includes has message unpacking, message decompression, message deciphering and message authentication etc.；The Message Processing instrument that downstream message process chain includes then has message authentication, message encryption, message compression and message to beat accordinglyBag etc.；
Message threads pond refers to the set of each Message Processing thread, has and for message to be parsed into each object or structureThe function of body；
Message observer refers to carry out the interface object of various different disposal, bag for the message object parsingInclude daily record observer, audit observation person, virtual machine observer, logical volume observer etc.；Message observer adopts Observer PatternDesign is realized, and same message is supported successively by multiple observer's reception processings；, Observer Pattern is a kind of in Programming Design volumeJourney implementation method.
Up-downgoing source processing chain tool set supports dynamic scalable.
Up-downgoing message queue supports internal memory form database form；Internal memory form message queue refers to Shen in internal memoryPlease a certain size space as message format, there is the characteristic of efficiency high；Database form refers to first be saved in message numberAccording to storehouse, wait for the arrival of news and process the scheduling of thread, there is persistence characteristic, support that service fault recovers the spy that message is processed againPoint.
Message observer's particular service logical process refers to carry out log recording, audit observation person similar to daily record observerAudited, virtual machine observer carries out creating virtual machine, closes the business logic processing such as virtual machine.
In conjunction with shown in Fig. 2, in data processing shelf, source processing chain is in the stage of text message turnover, comprises to solveThe processing procedures such as bag, decompression, deciphering, certification, each processing procedure is a handler, and each handler needs to inheritMsghandler base class, the design of msghandler base class is as follows:
Each message processing procedure on demand dynamic scalable be assembled into upstream message process chain incomemsgchainWith downstream message process chain outcomemsgchain.Assembling process is exactly to process handler for each to call unified interfaceSetsuccessor, being allowed in succession link together becomes a source processing chain, and link assembling process is as follows:
Upstream message process chain construction process is as follows:
Downstream message process chain construction process is as follows:
Downstream message process chain has carried out encryption, pressurization, compressed logic process successively to message.
The same with source processing chain, message queue is also designed to upstream message queue and downstream message queue, message teamRow management Similar integral imqmng interface, Interface design is as follows:
The team that goes out of queue has two lines thread management respectively.Incomemqpollthread and outcomemqpollthread.Their two work are exactly constantly to take out message from queue, then put thread pool execution into.Thread pool is accomplished by
Message observer receives specific message object and structure, carries out specific business logic processing.Realization sideIn case, message observer inherits commandobserver abstract class, and its code is as follows:
All message observers are managed collectively by message observer management class, and implementation method employs ObserverFormula, code is as follows: