Movatterモバイル変換


[0]ホーム

URL:


CN103580960B - Online pipe network anomaly detection system based on machine learning - Google Patents

Online pipe network anomaly detection system based on machine learning
Download PDF

Info

Publication number
CN103580960B
CN103580960BCN201310581956.0ACN201310581956ACN103580960BCN 103580960 BCN103580960 BCN 103580960BCN 201310581956 ACN201310581956 ACN 201310581956ACN 103580960 BCN103580960 BCN 103580960B
Authority
CN
China
Prior art keywords
data
virtual machine
pipe network
abnormity detecting
abnormity
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201310581956.0A
Other languages
Chinese (zh)
Other versions
CN103580960A (en
Inventor
陈尊裕
张得志
李丹
胡斯洋
龙圣
郑思明
吴珏其
周振邦
李维海
王红旗
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Foshan Science And Technology Co Ltd
Original Assignee
Foshan Luosixun Environmental Protection Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Foshan Luosixun Environmental Protection Technology Co ltdfiledCriticalFoshan Luosixun Environmental Protection Technology Co ltd
Priority to CN201310581956.0ApriorityCriticalpatent/CN103580960B/en
Publication of CN103580960ApublicationCriticalpatent/CN103580960A/en
Application grantedgrantedCritical
Publication of CN103580960BpublicationCriticalpatent/CN103580960B/en
Activelegal-statusCriticalCurrent
Anticipated expirationlegal-statusCritical

Links

Landscapes

Abstract

The invention discloses an online pipe network anomaly detection system based on machine learning. The online pipe network anomaly detection system comprises a data collection unit, a data distribution unit and a plurality of anomaly detection units. The data collection unit is used for collecting real-time data of an online pipe network, merging the real-time data according to position areas and grouping the real-time data into different data packages. The data distribution unit is used for receiving the data packages, extracting data elements from the data packages and dividing the data packages into a plurality of data subsets after formatting the data packages. The anomaly detection units are used for receiving the data subsets in a one-to-one correspondence mode and predicating anomalism of the data subsets based on a semi-supervised machine learning framework. The anomaly detection units can be used for carrying out parallel data processing, and data transmission can be carried out among the anomaly detection units through an MPI. The online pipe network anomaly detection system can meet the requirements of the online anomaly detection units based on machine learning for usability of a server, and can prevent extra hardware on standby in an idle state from being introduced in.

Description

A kind of online pipe network abnormity detecting system based on machine learning
Technical field
The present invention relates to a kind of facility pipe network monitoring technology, be specifically related to a kind of based on machine learning at spoolNet abnormity detecting system.
Background technology
The development of sensor technology makes sensor can realize high space-time accuracy parameters measuring at environmental area.PassThe time series data that sensor is collected constantly inputs in bin, forms data stream.With waterworks operationAs a example by, sensing data can include each hydraulic parameters and water quality index.These data can be used for abnormal shapeCondition detections etc., it differentiates data exception by historical pattern or model prediction.Unusual condition can be pipelineReveal or contamination accident.The geographical scale of pipeline is big, by Changes in weather, seasonal variations, festivals or holidays and societyIt is high by water feature complexity that district's population structure change is caused so that manual method this work not competent.Therefore machine learning techniques based on historical data is the unique feasible scheme of online abnormality detection.Machine learningTechnology can be roughly divided into three classes: (a) clear data analysis classes;(b) rule-based class;C () is based on physics mouldType class, classification foundation be dependent on which kind of parameter follow the tracks of, predict current and future sensing data tendency andAssociation between each group data.First, abnormality detection system is based on normal system or sensor-based system historical dataSet a benchmark.Hereafter, any activity deviating from this benchmark will be considered exception.
Additionally, due to need to distinguish real abnormal data and non-abnormal data (false alarm), we still need toWant a set of calculating system based on replicanism Yu redundancy strategy, support continuous on-line data acquisition and perform numberAccording to parser.
Summary of the invention
For above not enough, it is an object of the invention to provide online pipe network abnormity detecting based on machine learningSystem, meets based on machine based on the virtualization of multiserver host hardware and Publish-subscribe data distribution strategyThe availability requirement to server of the online abnormity detecting unit of device study, avoids introducing what the free time awaited orders simultaneouslyUnnecessary hardware.
For realizing object above, the technical scheme that this invention takes is:
A kind of online pipe network abnormity detecting system based on machine learning, comprising:
Data acquisition unit, for gathering the real time data of described online pipe network, and depends on described real time dataMerge according to the band of position and be grouped into different packets;
File distributing unit, is used for receiving described packet, and extracts data element from packet, thenMultiple data subset it is divided into after packet is formatted process;
Multiple abnormity detecting unit, receive corresponding data subset for one_to_one corresponding, and to described dataCollection carries out abnormity prediction, the plurality of abnormity detecting unit panel data based on semi-supervised learning frameworkProcess and carry out data transmission by MPI each other.
Described abnormity detecting unit is installed on virtual machine, the corresponding abnormity detecting unit of each virtual machine.
Described online pipe network abnormity detecting system based on machine learning farther includes multiple server host,Server host is connected with each other by fully connected topology in LAN, and each server host is equipped with more than oneCore processor, multiple virtual machines that described polycaryon processor is divided on same server host according to thread,Wherein, first thread is designated as virtual machine dom0, and other thread is divided into virtual machine dom U, described voidPlan machine dom0 is for accessing the hardware of server host and interacting with virtual machine dom U, describedVirtual machine dom U is used for installing abnormity detecting unit, the virtual machine dom U of the server host of each operationOther server hosts run are provided with corresponding backup.
Described abnormity detecting unit includes:
Prediction module, for setting up forecast model according to multiple regression equation, is as good as reason to provide in hypothesisThe actuarial prediction data of data subset expecting varialbe state under condition, described prediction module and with other abnormity detectingUnit carries out the exchange of actuarial prediction data;
Analyze module, be used for receiving described actuarial prediction data, according to described actuarial prediction data estimation from numberThe regression parameter of data subset next time obtained according to Dispatching Unit, with data subset next time described in calculatingPredictive value, described data subset next time has identical time step and consistent pipe network with historical dataBackground;
Judge module, according to predictive value and the actual value of data subset next time, to described data next timeThe abnormity of collection judges;
Decision-making module, for receiving the abnormity judged result that judge module is made, and according to described abnormityDescribed forecast model is made renewal by judged result.
Described prediction module is set up the method for forecast model and is comprised the following steps:
Step 11, carry out the simulation of data model according to certain some parameter situation over time in online pipe network:
Xi(t+1)=Fi(X (t), X (t-1), X (t-2) ... X (t-n)) (1)
Wherein: FiBeing the forecast model of i-th abnormity detecting unit, i is positive integer, and is not more than abnormal detecingSurvey the sum of unit, XiIt is the input data of i-th abnormity detecting unit, wherein, X (t), X (t-1), X (t-2) ...X (t-n) is historical data, Xi(t+1) it is data subset next time;
Step 12, based on multiple regression equation build forecast model:
Xi(t+1)=Ai0*Xi(t)+Ai1*Xi(t-1)+... Ain*Xi(t-n)+Ci(2)
Wherein: Ai0、Ai1、...AinFor forecast model FiRegression parameter, CiFor i-th abnormity detecting unitRandom error parameter, described prediction module by MPI by CiCarry out the exchange of actuarial prediction data;
Step 13, solve random error parameter Ci
Ci=Σj≠inAij0*Xj(t)+Σj≠inAij1*Xj(t-1)+...+Σj≠inAijn*Xj(t-n)---(3)
In formula (3), Aij0、Aij1、...AijnFrom normal data bag, Auto-matching obtains.
The method that the abnormity of described data subset next time is judged by described judge module comprises the following steps:
Step 31, compare Xi(t+1) predictive value and the difference of measured value;
Step 32, collection historical data set up data base Xi(t), Xi(t-1) ... Xi(t-P), wherein, PFor the empirical parameter of the time relationship of i-th abnormity detecting unit, P is positive integer;
Step 33, structure historical data base sample { Xi(t), Xi(t-1) ... Xi(t-P) this sample }, is calculatedThis standard deviation scope;
Step 34, relatively described difference and standard deviation scope:
If difference is less than standard deviation scope, it is judged that module then returns a negative acknowledge character (NAK) to decision-making module, ifWhat the judge module being provided with abnormity detecting unit fed back to decision-making module is all negative acknowledge character (NAK), and decision-making module then willAll of Xi(t+1) it is stored in data base, and indicates the up-to-date sample of corresponding judge module and database synchronizationData and regression parameter, to be ready for use on prediction Xi(t+2);
Prediction Xi(t+2) time, if difference is more than the sample { X updatedi(t+1) Xi(t), Xi(t-1) ... Xi(t-P+1) standard deviation scope }, corresponding judge module then returns a signal certainly to decision-making module, certainlyPlan module in data base by Xi(t+2) be labeled as anomalous event, decision-making module by this judge module of instruction according toDatabase update regression parameter, but use old sample { Xi(t+1) Xi(t), Xi(t-1) ... Xi(t-P+1) }Definition standard deviation, for Xi(t+3) exception judges.
Described online pipe network abnormity detecting system based on machine learning farther includes Network Attached Storage listUnit, for storing the mirror image copies of the historical data of all virtual machines and online pipe network, each abnormity detectingUnit all may have access to the data in this Network Attached Storage unit, and virtual machine dom0 connects the virtual of its correspondenceMachine dom U and the communication of Network Attached Storage unit.
The method of described backup is: believed by the virtual machine dom U test point on the server host of each operationCease and be distributed backup according to the loading condition of other server hosts run, to realize optimal balance fortuneLine mode, automatically generates an inquiry table after backup, described inquiry table is used for defining primary fault virtual machine dom UThe migration node of backup, in order to perform dynamic migration when virtual machine dom U or server host break down.
Each virtual machine dom0 arranges backup manager, for by void corresponding for this virtual machine dom0The health status of plan machine dom U arranges into inventory, and virtual machine dom0 passes through backup manager according to inquiry table quiltBody plan goes to process the backup of fault virtual machine.
The method of described file distributing unit distribution data subset comprises the following steps:
Described file distributing unit receive packet and in packet owing to measuring, the reason that sends or collectAnd the interference information even error message produced filters;
Extract the data element in packet, packet is converted to consolidation form;
Packet is divided into corresponding number and ensures equilibrium according to subset, the data in data subset,
Data subset is encrypted, and by publish-subscribe architecture, data subset is distributed to abnormity detecting listUnit.
The present invention compared with prior art, has the advantage that
1, by machine learning online pipe network carried out abnormity detecting unit, thus provide and be as good as reason assumingUnder condition, the statistical distribution prediction of distributed network expecting varialbe state, improves the anomalous identification rate of online pipe network,Save substantial amounts of manpower simultaneously.
2, abnormity detecting unit parallel time processes, and reduces cpu resource competition, meets serverAvailability requirement, avoids introducing the idle unnecessary hardware awaited orders simultaneously.
3, need not rebuild data transfer application interface and deacclimatize data transmission inside and outside different server main frameAgreement with communication control.Sensing number it is far smaller than in the network of rivers by the computing relay between dom0 and dom UAccording to transformation period.
4, each server host has been not required to single disk, and virtual machine epigraph is stored in NASOn, it can be accessed by any physical machine.In this case, any virtual machine can be in any physical machineRun without again and on local disk, carry out backup.
5, virtual machine acquisition testing point is copied on another server host to complete dynamic migration.IfOne or more data processing module faults, each malfunctioning module will be by by multiserver main frame virtual platformThe copy come into force replaces.
6, in failover, even if not having fault to have moved the virtual machine on different server main frameTo resume operation from up-to-date test point.All operation times of operating system include that the TCP of activity connectsCan preserve.The process being currently running will be carried out as usual, and all files, network state and disk all will keepWhole property.
Accompanying drawing explanation
Fig. 1 is the network architecture of high availability facility pipe network abnormity detecting of the present invention;
Fig. 2 is parallel type online abnormity detecting framework;
Fig. 3 is that multiserver mainframe virtualization envisions framework;
Fig. 4 is the framework that multinuclear process thread is divided into the different virtual machine on same server main frame;
Fig. 5 is to manage the method for the high availability server of executed in parallel online abnormity detecting algorithm to retouchState.
Detailed description of the invention
With detailed description of the invention, present disclosure is described in further details below in conjunction with the accompanying drawings.
Embodiment
The present embodiment as a example by the abnormity detecting of water supply network, other online pipe network such as electric power, telecommunications, network,Communication, heating power, combustion gas etc. are similar with its method, repeat no more here.
Fig. 1 is the network architecture of high availability facility pipe network abnormity detecting.In each group, sensor can beHydraulic data or water quality data sensor, the sensing data in immediate geographic location passes through data acquisition listUnit is grouped together as packet and sends.File distributing unit receives the measurement data of sensor, by numberAccording to be converted into meet subscriber's later stage process require form and issue.The server host at manipulation center is in officeTerritory net (LAN) is connected with each other by fully connected topology.Virtual machine (vm) migration on different server main frameFirst-selected mesh topology framework.Network Attached Storage unit is connected to all physical server hosts by LANOn.
Various sensors and instrument in water supply network monitoring system constantly gather data.Data can comprise waterForce data (such as flow velocity, flow, hydraulic pressure, water level etc.) and water quality data (include free chlorine, turbidity, pH,Electrical conductivity, oxidation-reduction potential and total organic nitrogen etc.).The public network of rivers can be detected by analyzing these indexsIn pipe leakage and contamination accident.Due to for the facility regular jobs such as water tank, pump, gate, water sourceAnd the seasonal variations of closing water, or water requirements fluctuation etc., in above-mentioned water distribution system, index changes the most greatly.It would therefore be desirable to incident detection system distinguishes conventional change and the unusual condition of sensing data.
Data acquisition unit includes SCADA system (supervisory control and data acqui sition system) and RTUs(RemoteTerminal Units, remote control terminal), SCADA system is to collect real-time transport net sensing dataA kind of canonical system.In SCADA system of the present invention (Fig. 1), we pass through region RTUsThis locality sensing data is merged packet.RTUs function is by Data Digital, according to categories of sensors and collectionTime adds time tag etc..Digitized sensor data are then sent to data collection server, and this process canRealized by closed circuit industrial network, such as Modbus, Lonworks, or BACnet.
File distributing unit is based on publish-subscribe architecture.File distributing unit as file distributing unit from conveyingNetwork sensing data extracts data element, and converts them to consolidation form.Due to measure, send orThe reason collected and the interference information even mistake that produces will be filtered in advance.File distributing unit tissue latticeFormulaization receives data in case processing further.After encryption, data are by open TCP/IP Ethernet transmissionThe terminal receiving different-format water quality data to each, in the present invention, sends the data after converted formIn the abnormity detecting unit of operation center.Publish-subscribe host-host protocol includes data set X=(X1,X2... Xm) decomposition rule, such as X1Send to virtual machine #1, X2Send to virtual machine#2 ... XmSend to virtual machine #m.
Exactly, all transport net sensing datas are based on the band of position and are divided into different packets.Sensing data in each packet can be waterpower or water quality data.File distributing unit is with abnormity detecting listThe IP address of unit is packet name.As in figure 2 it is shown, be parallel type online abnormity detecting framework.All differentOften detecting unit will run in multiserver host virtual machine system, and duplication is made mistakes different by virtual machine monitorOften detecting unit, therefore, abnormity detecting unit can recover from single virtual machine fault.
The abnormity detecting unit of operation center uses panel data tupe.System in the present invention is sameShi Zhihang multiple abnormity detecting algorithm, each algorithm processes certain subset in sensing data bag respectively, theseSubset enters operation center with the form of independent packet.Need to pass through between each abnormity detecting unitMessage Passing Interface(MPI) mutually transmit data.The abnormity detecting program of abnormity detecting unitC language or Fortran can be used to write, can run on linux system.When abnormity detecting program is with CWhen language is write, MPI is one group of function in C language.When writing with Fortran language, MPI and useIn the subprogram (Fortran language compilation) exchanging data in different processes.
The detailed description of an algorithm be presented herein below:
The data being input in abnormity detecting unit cover the running status of whole pipe network.These data be bySensor measurement and obtain.Data base will the last state of real-time update pipe network.
In pipe network, certain some parameter situation over time is simulated by data model, as follows
X (t+1)=F (X (t), X (t-1), X (t-2) ... X (t-n)),
Wherein X (t) is the parameters measured by each sensor.F is forecast model, reads from data baseHistorical data X (t), X (t-1), X (t-2) ..., speculate next time point t+1's according to the observed result of historyX value.Under normal circumstances, based on multiple regression equation
X(t+1)=A0*X(t)+A1*X(t-1)+...An*X(t-n)
Just be enough to build forecast model F, determine the meansigma methods of X (t+1), wherein, A0To AnIt it is coefficient matrix.
Owing to X is a vector the hugest, up to a hundred parameters of a large-scale network will be contained.For letterChanging computing, it is different that the calculating process of F can be divided into several by multiple programming technology in MPI frameworkSubprocess.
I.e.
X=(X1, X2... Xi..., Xm)
The length of each subvector is
And
Xi(t+1)=Fi(X (t), X (t-1), X (t-2) ... X (t-n)),
Xi(t+1)=Ai0*Xi(t)+Ai1*Xi(t-1)+... Ain*Xi(t-n)+Ci,
I=1 ..., m
Ci=Σj≠inAij0*Xj(t)+Σj≠inAij1*Xj(t-1)+...+Σj≠inAijn*Xj(t-n)
The most each FiComputing can independently execute on a virtual machine.At " Publish-subscribe "(Publish-Subscribe) under data distribution strategy framework, XiIt is FiThe input data of module, module itBetween by message passing interface (MPI) utilize by random error parameter CiCarry out data exchange.
Parameter in regression equation can from standard figures bag (such as CRAN-R statistical computation bag) automaticallyCoupling obtains.
Prediction module is collected historical data and is set up data base X(t), X(t-1) ... X(t-P), wherein p isThe empirical parameter of the time relationship of definition X.
Estimate the regression parameter of each module according to historical data, be used for calculating Fi.Forecast model is usedRegression parameter estimation Xi(t+1) meansigma methods.
Subsequently determine whether that module will calculate Xi(t+1) the difference between predictive value and measured value.
If difference is less than sample { Xi(t), Xi(t-1) ... Xi(t-P) standard deviation scope }, it was predicted that mouldType FiThen return a negative acknowledge character (NAK) to decision package.If all module feedback to decision package the most whetherDetermining signal, decision package then allows X(t+1) it is stored in data base, and indicate each forecast model and data baseSynchronize up-to-date sample data and regression parameter, to be ready for use on prediction X(t+2).
If difference is more than sample { Xi(t+1) Xi(t), Xi(t-1) ... Xi(t-P+1) standard deviation model }Enclose, it was predicted that model FiThen return a signal certainly to decision package.Decision package in data base by X(t)It is labeled as anomalous event.Decision package is by instruction this forecast model FiRegression parameter is updated according to parameter database,But use old sample values definition standard deviation, be used for judging exception.
System high-available in this patent passes through data parallel processing (parallel type) model realization.If oneIndividual or multiple data processing module faults, each malfunctioning module will be by raw by multiserver main frame virtual platformThe copy of effect replaces.
The matrix that each data processing module belonging to above-mentioned subset produces, by further standardization, delivers to main determiningPlan unit, is used for differentiating event detection outcome.
Fig. 3 is that multiserver mainframe virtualization envisions framework.(SuSE) Linux OS is arranged on dom U.Each dom U installs an abnormity detecting unit, and by Message Passing Interface (MPI)With the module communication on another dom U.At hardware view, communications protocol takes ICP/IP protocol for takingBusiness device main frame in and server host between communication.The mirror image of the historical data of each virtual machine and facility pipe networkCopy will be stored in Network Attached Storage unit.Network Attached Storage unit i.e. Network storage technologyIn (Network Storage Technologies), its data above can be visited by each accident detection moduleAsk, process for data, it is also possible to when a certain virtual machine or server host fault for Virtual Machine ManagerBy on above virtual machine (vm) migration to existing service device main frame.
Multinuclear is processed the different virtual machine that thread is divided on same server main frame by Fig. 4.Multinuclear processesFirst thread of device is designated as dom0, its connecting virtual machine and the communication of Network Attached Storage unit, andIt is responsible for creating and elimination virtual machine.Remaining calculates resource and supplies the virtual machine of operation exception detecting unit to makeWith.
In conjunction with Fig. 3 and Fig. 4, in the present invention, based on the high-performance abnormality detection service that multiserver main frame is virtualSystem architecture can be divided into three major parts.
[1] physical machine virtualization:
In this construction, physical machine be only install virtual machine server host, it by execution to conveyingThe parallel type abnormality detection of network detection data.Management program or virtual machine manager, such as IBMz/VM,VMware ESX, with XenSource or Novell Xen, will be installed on all virtual machines.Management journeySequence can directly be run on hardware, without specific operating system, and can transport on the hardwareThe multiple virtual machines of row, as shown in Figure 3.
The present invention uses the Xen CPU of acquiescence to distribute policy, and in this case, virtual machine dom0 is designatedFirst thread for each server host (such as Fig. 4) that may be installed on polycaryon processor.Dom0 isFirst virtual machine guided by Xen, it has some privileges, as can be directly accessed hardware, can have bothThe I/O function of all access systems, and (create with other virtual machine interaction being expressed as dom UWith management) etc..
It is virtual that the dom0 that each server host is runs the detection that Heartbeat(sets up on XenThe messaging system that whether good machine running status is), it performs intelligence to all dom U on server hostEnergy fault detect, and process similar with on other server hosts exchange information.Due to Servers-all masterMachine is all connected with mesh network, and the backup manager on each dom0 can being good for interior for group all virtual machinesHealth status Bar becomes inventory.Which backup is migrated node by definition primary fault virtual machine by one inquiry table at,And each backup manager can access this inquiry table.It is distributed more due to the virtual machine in group every timeChange and after performing backup process, this table all can be updated, or system manager the most simply will be hardPart and virtual machine configuration recovery value original state, and keep inquiry table not change.
Virtual machine is virtualized environment, and each virtual machine performs themselves operating system and application journeySequence.In the present invention, Linux is designated as the operation sequence of all virtual machines and physical machine.Each virtualOne abnormity detecting unit is all installed, the example processed as MPI on machine.
Virtual network interface is assigned to each virtual machine.Each interface has single MAC Address and IPAddress.
The present invention only uses TCP/IP communication interface (to have the physical services of certain quantity virtual machine as physical machineDevice main frame) in local data exchange and the interface of inter-node communication.Virtual machine guest dom U and virtual networkDriving Direct Communication, virtual network drives and drives function identical with Ethernet card.It is translated as hardware with by instructionUnlike signal, this driving will interact with dom0 so that connects with the respective rear ends in driving fieldMouth communication.This makes virtual machine on all-network go out as the individual services device main frame having different MAC AddressExisting.Although ICP/IP protocol is not enough to support the data transmission on same server main frame between virtual machine, but phaseThan for the shared drive data transfer protocols that Xensocket Yu Xway provides, by dom0 and domUBetween the computing relay of tcp/ip layer be far smaller than the transformation period of sensing data in the network of rivers.Additionally, due to thisGround Xen management program and the use of MPI code, system stability improves.
It is a trend favourable that polycaryon processor starts to commonly use.The system of the present invention can utilize this trend,Make dom U from dom0 in same server main frame run on different threads, thus can allow themDifferent IPs performs.CPU separates by body plan Xen management program realization.It makes the void in dom0The MPI that plan I/O control protocol and dom U process is carried out parallel, makes reduction cpu resource compete.This can delayWith the delay issue that above-mentioned I/O concentrates MPI process.On all virtual machines all will run based on IP(InternetProtocol, Internet protocol) service, have a following functions:
The file distributing unit that [a] mates from IP subscribes to packet.
Transmission network sensing data subset is inputted abnormity detecting unit by [b], and it is a to NAS to make a copy for.
The prediction module of [c] abnormity detecting unit subscribes to packet from other from the file distributing unit of different IPVirtual machine exchange process data.Prediction module is to go statistically to analyze based on semi-supervised learning frameworkData, thus the statistical distribution prediction of distributed network expecting varialbe state in the case of hypothesis is without exception is provided.In system in the present invention, abnormity detecting unit is compiled into MPI program in identical or different physical machineRun on virtual machine.In the present invention, we use the acquiescence ICP/IP protocol on Xen in different MPI journeysSequence transmits data.Therefore, we need not rebuild data transfer application interface and deacclimatize different serviceInside and outside device main frame, data transmit the agreement with communication control.By the tcp/ip layer between dom0 and dom UComputing relay is far smaller than the transformation period of sensing data in the network of rivers.Additionally, due to local Xen manages programWith the use of MPI code, system stability improves.
[d] combines shown in Fig. 2, and the analysis module of abnormity detecting unit receives actuarial prediction data from prediction module,It potentially includes the distribution of possible range numerical value, variance, and some other statistical indicator.Each time stepResidual error in length must be classified as or outlier consistent with background water quality value.Analyze module according to describedThe regression parameter of the data subset next time that actuarial prediction data estimation obtains from file distributing unit, to calculateThe predictive value of described data subset next time.Data subset and historical data have identical time step next timeLong lower and consistent background water quality value.
[e] combines shown in Fig. 2, and the judge module of abnormity detecting unit is inclined to predictive value and online sensing dataDifference degree judges.Although the absolute value at first unit lower threshold value can change along with water quality index, phaseAcceptable prediction distribution formula network state deviation is fixed to specific standard deviation.Subsequently, one based onThe abnormal accident differentiation of machine learning is used as decision tool with sort module.This judge module can be from visitAsk the historical data being stored in Network Attached Storage unit in data base.
[f] combines shown in Fig. 2, and result is imported on different virtual machine on the main abnormity detecting unit run.This main abnormity detecting unit will analyze the result of all previous concurrent abnormity detecting processes, and determines anomalous eventClassification with in facility pipe network occur position.
[2] use of Network Attached Storage
Each server host has been not required to single disk, and virtual machine epigraph is stored on NAS,It can be accessed by any physical machine.In this case, any virtual machine can run in any physical machineAnd without carrying out backup on local disk again.
[3] monitoring and the control of high availability
REMUS software kit in Xen framework is responsible in Xen management program the General Virtual Machine runOffer high-performance ensures.In a system of the invention, when physical machine or simply certain specific virtual machine generationDuring mistake (whatsoever reason, hardware or software faults), REMUS will be with altofrequency (20-40 inspectionMeasuring point/second) to virtual machine acquisition testing point (checkpoints), and it is copied into another server hostOn to complete dynamic migration.In failover, even if not having fault to have moved different server masterVirtual machine on machine will resume operation from up-to-date test point (checkpoints).All operations of operating systemTime includes that the TCP of activity connects and all can preserve.The process being currently running will be carried out as usual, all files,Network state and disk all will keep integrity, at most TCP storehouse to there will be packet loss, but package also willIt is possibly retransmission.
REMUS is used to be possible to prevent virtual machine that collapse fault occurs.This characteristic contributes to carrying out abnormal inspectionThe parallel computing of the MPI surveyed, because maintenance is all synchronized by all of calculation procedure.
Server exists in pairs with operation/standby both of which under REMUS drives, the service of operational modeDevice will send test point information back-up to standby mode server based on Heartbeat signal in good time.
In the present invention, each server host is simultaneously in operation, two patterns of backup, will be by design oneIndividual inquiry table, is distributed to virtual machine test point (checkpoint) information in certain particular server main frameAnother is on the server host of " backup " pattern, in order to can perform dynamic migration when fault occurs.
Owing to solid state hard disc still possesses high availability in acceptable price, the server master of backup virtual machineMachine can use solid state hard disc to be that test point (checkpoint) provides the most local archive, it is achieved virtual machine poleSpeed (sub-second) is restarted.
Fig. 5 management describes for the method for the high availability server of executed in parallel online abnormity detecting algorithm.This figure illustrates and replicates fault virtual machine or server in the case of 4 server hosts and 16 virtual machinesThe situation of main frame.Each server host runs 4 virtual machines.Dom0 at each existing service device main frameThe backup manager of upper operation will be gone the backup of process fault virtual machine according to inquiry table by body plan.Such as, clothesBusiness device main frame #1, in addition to responsible operation A, B, C, D virtual machine, is also responsible for the test point of E, I, M, FInformation back-up, when certain virtual machine (assuming E) in E, I, M, F breaks down, server hostBackup manager in #1 will enable the backup of corresponding test point, make the virtual machine (E) broken down in serviceResume operation in device main frame #1 (now on server host #1 A, B, C, D, E all in operational mode).After automated back-up process completes, showing if required up optimizer system, system manager can get involved managementDevice manages each server host operating duty, and online by virtual machine (vm) migration to difference physical serversOn main frame.Afterwards, system manager will need in existing service device main frame under the virtual machine of redistribution,(assuming that server host #1 collapses, new inquiry table will is that the inquiry table of renewal control virtual machine backup processServer host #2 will run A, E, F, G, H, back up B, I, N, O, P;Server host #3B, I, J, K, L will be run, back up A, C, D, E, M;Server host #4 will run C, D,M, N, O, P, back up I, J, K, M, N), or simply simply hardware and virtual machine are configured backComplex value original state, and keep inquiry table not change.The design principle of inquiry table is, when any one servicesAfter the collapse of device main frame, all virtual machines that this server host runs will be divided equally to other normal server masterContinue to run with on machine.
Fig. 5 have employed as an embodiment and only comprise 4 server hosts and be separately operable 4 virtual machinesSituation.Should state, it illustrates just for possible embodiments of the present invention, this embodimentAnd it being not used to limit the scope of the claims of the present invention, the present invention should include but not limited to above-mentioned detailed description and toolStyle.The present invention should include all adjustment in the range of core content and amendment, all without departing from institute of the present inventionFor equivalence implement or change, be intended to be limited solely by the scope of the claims of this case.

Claims (8)

Described online pipe network abnormity detecting system based on machine learning farther includes multiple server host,Server host is connected with each other by fully connected topology in LAN, and each server host is equipped with more than oneCore processor, multiple virtual machines that described polycaryon processor is divided on same server host according to thread,Wherein, first thread is designated as virtual machine dom 0, and other thread is divided into virtual machine dom U, described voidPlan machine dom 0 is for accessing the hardware of server host and interacting with virtual machine dom U, describedVirtual machine dom U is used for installing abnormity detecting unit, the virtual machine dom U of the server host of each operationOther server hosts run are provided with corresponding backup.
CN201310581956.0A2013-11-192013-11-19Online pipe network anomaly detection system based on machine learningActiveCN103580960B (en)

Priority Applications (1)

Application NumberPriority DateFiling DateTitle
CN201310581956.0ACN103580960B (en)2013-11-192013-11-19Online pipe network anomaly detection system based on machine learning

Applications Claiming Priority (1)

Application NumberPriority DateFiling DateTitle
CN201310581956.0ACN103580960B (en)2013-11-192013-11-19Online pipe network anomaly detection system based on machine learning

Publications (2)

Publication NumberPublication Date
CN103580960A CN103580960A (en)2014-02-12
CN103580960Btrue CN103580960B (en)2017-01-11

Family

ID=50051937

Family Applications (1)

Application NumberTitlePriority DateFiling Date
CN201310581956.0AActiveCN103580960B (en)2013-11-192013-11-19Online pipe network anomaly detection system based on machine learning

Country Status (1)

CountryLink
CN (1)CN103580960B (en)

Families Citing this family (20)

* Cited by examiner, † Cited by third party
Publication numberPriority datePublication dateAssigneeTitle
CN104852830A (en)*2015-06-012015-08-19广东电网有限责任公司信息中心Service access model based on machine learning and implementation method thereof
CN106649414B (en)*2015-11-042020-01-31阿里巴巴集团控股有限公司Method and equipment for pre-detecting data anomalies of data warehouses
CN105740989B (en)*2016-02-032019-09-27杭州电子科技大学 A method for detecting abnormal events in water supply network based on VARX model
CN106125643A (en)*2016-06-222016-11-16华东师范大学A kind of industry control safety protection method based on machine learning techniques
CN106209843A (en)*2016-07-122016-12-07工业和信息化部电子工业标准化研究院A kind of data flow anomaly towards Modbus agreement analyzes method
CN108023740B (en)*2016-10-312020-06-16腾讯科技(深圳)有限公司Risk prompting method and device for abnormal information in monitoring
CN106775929B (en)*2016-11-252019-11-26中国科学院信息工程研究所A kind of virtual platform safety monitoring method and system
CN107360159B (en)*2017-07-112019-12-03中国科学院信息工程研究所A kind of method and device of the abnormal encryption flow of identification
CN108229537A (en)*2017-12-072018-06-29深圳市宏电技术股份有限公司The singular values standard form method, apparatus and equipment of a kind of precipitation
CN108259482B (en)*2018-01-042019-05-28平安科技(深圳)有限公司Network Abnormal data detection method, device, computer equipment and storage medium
CN110188910B (en)*2018-07-102021-10-22第四范式(北京)技术有限公司Method and system for providing online prediction service by using machine learning model
CN109286526A (en)*2018-10-082019-01-29成都西加云杉科技有限公司A kind of wifi system running policy dynamic adjusting method and device
CN109857611A (en)*2019-01-312019-06-07泰康保险集团股份有限公司Test method for hardware and device, storage medium and electronic equipment based on block chain
CN109981744B (en)*2019-02-282022-03-04东软集团股份有限公司Data distribution method and device, storage medium and electronic equipment
CN109871002B (en)*2019-03-062020-08-25东方证券股份有限公司Concurrent abnormal state identification and positioning system based on tensor label learning
CN110618854B (en)*2019-08-212022-04-26浙江大学 Virtual machine behavior analysis system based on deep learning and memory mirror analysis
CN113641444B (en)*2020-04-272024-06-07南通华信中央空调有限公司Virtual testing method, virtual testing device and related equipment
CN111639430B (en)*2020-05-292024-02-27重庆大学Natural gas pipeline leakage identification system driven by digital twinning
TWI798007B (en)*2022-02-252023-04-01中華電信股份有限公司Anomaly detection system, method and computer readable medium based on system characteristics
CN114623872A (en)*2022-03-082022-06-14内蒙古金原农牧科技有限公司Underground water dynamic monitoring system based on strong magnetic wireless transmission

Citations (5)

* Cited by examiner, † Cited by third party
Publication numberPriority datePublication dateAssigneeTitle
CN101719849A (en)*2009-11-032010-06-02清华大学Pattern clustering-based parallel network flow characteristic detection method
CN101980480A (en)*2010-11-042011-02-23西安电子科技大学 Semi-Supervised Anomaly Intrusion Detection Method
CN102045381A (en)*2010-10-132011-05-04北京博大水务有限公司On-line monitoring system for regenerated water pipe network
CN102635787A (en)*2012-04-162012-08-15中山大学Automatic detection device and detection method for water leakage of water pipeline
CN102970245A (en)*2012-11-212013-03-13北京奇虎科技有限公司Data transmission method and system

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication numberPriority datePublication dateAssigneeTitle
CN101719849A (en)*2009-11-032010-06-02清华大学Pattern clustering-based parallel network flow characteristic detection method
CN102045381A (en)*2010-10-132011-05-04北京博大水务有限公司On-line monitoring system for regenerated water pipe network
CN101980480A (en)*2010-11-042011-02-23西安电子科技大学 Semi-Supervised Anomaly Intrusion Detection Method
CN102635787A (en)*2012-04-162012-08-15中山大学Automatic detection device and detection method for water leakage of water pipeline
CN102970245A (en)*2012-11-212013-03-13北京奇虎科技有限公司Data transmission method and system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
"Wireless water quality monitoring system based on field point technology and kohonen maps";Postloche O. Silva Girao P, Dias Pereira J M.;《Proc of IEEE Sensors》;20021230;第7卷(第5期);全文*
"基于S7-200自来水管网监控系统设计";庄宪骥;《中北大学》;20091019;第40,63,55,32页,图2.1*

Also Published As

Publication numberPublication date
CN103580960A (en)2014-02-12

Similar Documents

PublicationPublication DateTitle
CN103580960B (en)Online pipe network anomaly detection system based on machine learning
CN118916147B (en)Multi-source calculation force data integration and intelligent scheduling system and method
CN110609512B (en)Internet of things platform and Internet of things equipment monitoring method
Ploennigs et al.Adapting semantic sensor networks for smart building diagnosis
US20250298397A1 (en)Cooling system running method and apparatus, device, and storage medium
Ferrari et al.Distributed fault detection and isolation of large-scale discrete-time nonlinear systems: An adaptive approximation approach
CN107085415A (en)Regular composer in process control network
Kavulya et al.Failure diagnosis of complex systems
JP2018180759A (en) System analyzer and system analysis method
CN104412190A (en)Systems and methods for health assessment of a human-machine interface (HMI) device
CN118897809B (en) A method and system for monitoring the testing process of computer network application programs
CN119142750A (en)Intelligent mine ore conveying management system
CN118963327A (en) A mechatronic adaptive fault diagnosis system and method
CN118445106B (en)On-line state monitoring method and health management system for coal machine equipment
CN119494467B (en) A knowledge graph-based method for energy system fault prediction
CN119557134A (en) Fault handling method, device, electronic device and storage medium for cloud computing platform
CN120342896A (en) A remote operation and maintenance control system for smart buildings based on big model and cloud-edge collaborative architecture
JP7062505B2 (en) Equipment management support system
CN119249365A (en) Intelligent multi-sensor fusion fault prediction and diagnosis method and system for depaneling machine
WO2025123876A9 (en)Method and apparatus for determining abnormal processing unit, and non-volatile storage medium
Pahl et al.A quality-driven machine learning governance architecture for self-adaptive edge clouds
EP4369119B1 (en)Industrial automation data staging and transformation
EP4369121A1 (en)Industrial data extraction
Chehida et al.Applied statistical model checking for a sensor behavior analysis
Wu et al.Analysis of data for the carbon dioxide capture domain

Legal Events

DateCodeTitleDescription
C06Publication
PB01Publication
SE01Entry into force of request for substantive examination
SE01Entry into force of request for substantive examination
GR01Patent grant
GR01Patent grant
C56Change in the name or address of the patentee
CP03Change of name, title or address

Address after:528200 Guangdong Province Nanhai District of Foshan city streets Guicheng Shilong Road No. 1 joybon IFC 2 room 1707

Patentee after:Foshan science and Technology Co., Ltd.

Address before:528200 Guangdong city of Foshan province sea road Han day Technology City Building No. 8 901-3

Patentee before:FOSHAN LUOSIXUN ENVIRONMENTAL PROTECTION TECHNOLOGY CO., LTD.


[8]ページ先頭

©2009-2025 Movatter.jp