River channel drainage state real-time identification method based on video imageTechnical Field
The invention relates to the field of intelligent water affairs, in particular to a real-time identifying method of a river drainage state based on video images, which is suitable for real-time video identification of river drainage flow.
Background
With the development of society, especially the progress of scientific technology, the rapid development of social productivity is greatly promoted, and especially the wide-spread application of communication network technology, the development of a plurality of fields from traditional manual statistical analysis to intelligent direction is started.
The river drainage running water identification technology is not mature at present, and can be realized through the video monitoring technology, if abnormal drainage exists at the drainage port, the site situation can be conveniently called and checked, but the disadvantage of video monitoring is that the abnormality can be found only by staring at the people. Therefore, the research of the automatic identification method for the river discharge running water is imperative, so that the accuracy and the efficiency of intelligent identification of the river discharge running water are improved.
In recent years, deep learning (DEEP LEARNING) has made an important and successful breakthrough in the field of artificial intelligence, becomes a new and popular research direction of machine learning, has strong learning and efficient feature expression capability, and has made great success in various fields such as computer vision, image and video analysis, voice recognition, multimedia and the like. The river channel drainage pipeline automatic identification method based on the video images has high identification precision and high video stream data analysis efficiency.
In the prior art, the drainage condition of the river channel drainage is complex, for example, the drainage shape is different and can be round, square, polygonal and the like, the drainage water flow is quite large, sometimes quite small, the measurement by the liquid level meter has quite large error, and sometimes the drainage water flow is quite large in color change and sometimes mixed with the drainage moss into one color, so that the method for measuring the drainage state of the drainage and further controlling the drainage of the river channel drainage sewage, particularly the problem of small amount of toxic sewage, is a problem which needs to be solved urgently.
Disclosure of Invention
Aiming at the defects existing in the prior art, the invention provides a real-time recognition method for the drainage state of the river channel drainage based on video images, which takes a deep learning technology as a core to realize the analysis of the drainage flow video data of the river channel, adopts a deep convolutional neural network to train, adds various scene data into a data set, obtains a judgment model of the drainage flow scene after training, and realizes real-time recognition and gives out abnormal drainage alarm information with time sequence property through the quasi-real-time analysis of video streams.
The purpose of the invention is realized in the following way:
A real-time recognition method for drainage state of river drainage based on video image includes such steps as creating a drainage state depth learning model, collecting drainage and non-drainage state video of river drainage, splitting video into a frame image, calculating to obtain motion vector of each frame image, labeling the motion vector with drainage state label information, creating a drainage state depth learning model, training the drainage state depth learning model, obtaining real-time video stream of river drainage by camera, splitting video stream into a frame image, calculating to obtain motion vector of each frame image, analyzing the motion vector of each frame image by the created drainage state depth learning model to output primary drainage state result, analyzing and filtering to obtain final drainage state information;
The method for establishing the model of river drainage and setting initial conditions is as follows:
establishing a river drainage model
ut+uux=0..................(1)
Describing a drain state of the drain;
t represents time, x represents water level position, u represents velocity wave, x=0 represents state position of water to be discharged from the nozzle but not from the nozzle, when x is less than or equal to 0, the water is all in the pipeline and its velocity is much slower than the velocity after the pipeline, so that it is assumed that u=0, when the water is discharged from the pipeline, its velocity is suddenly increased, the water is smoothly discharged through the drop, and its average velocity is assumed to be v0, so that the initial condition of equation (1) can be defined as
Assuming delta is the minimum drainage rate, when the rate v0 > delta >0, drainage can be determined and subsequent quantitative analysis including flow can be performed;
the solution of equation (1) is in the region { x >0, t > x }, the velocity wave isThe change of the discharge speed with time can be described by a motion vector, and the discharge and drainage state can be obtained by finding the motion vector;
the v calculation method of the motion vector of the image comprises defining the gray level diagram of the motion vector as the difference between two adjacent frames of gray level images (gi,gj) by draining water
gv＝gi-gj,
The motion vector v is one-dimensional expansion of the gray level diagram, the motion vector can be further improved into a weighted average value v-sigma aivi of a plurality of adjacent motion vectors vi, or vi is obtained by adopting an algorithm OptionalFlowFarneback method, and the motion vector is obtained by utilizing the weighted average of v-sigma aivi;
The method for establishing the drainage state deep learning model comprises the following specific steps:
(1) Collecting a large number of drain drainage videos and non-drainage videos;
(2) Calculating the motion vector of each frame of image in the video to form a motion vector gray scale image, wherein the motion vector gray scale image can be approximated according to the difference value of the front frame of image and the rear frame of image or determined by adopting OptionalFlowFarneback algorithm;
(3) Labeling each motion vector gray level diagram with drainage state label information, wherein the label value is the state of the water flow corresponding to the motion vector, and the specific labeling method comprises the steps that the label information is classified into a no-water state, a low-speed water state, a medium-speed water state and a high-speed water state, and the specific flow speed values of the low-speed water state, the medium-speed water state and the high-speed water state are actually adjusted according to specific conditions, wherein the specific flow speed values can be generally 0 m/s, 1 m/s, 2 m/s and 5 m/s;
(4) Establishing a drainage state deep learning model, wherein the drainage state deep learning model is established by training a plurality of convolution graphs and a plurality of full connection graphs, and a convolution layer and a full connection layer of the model are set according to conditions;
(5) Training the model, namely adopting a conventional CNN convolutional neural network training method, wherein the recognition accuracy during convergence reaches more than 99 percent, and the training learning rate is 0.0001.
The invention has the beneficial effects that 1. The invention realizes real-time unmanned monitoring and automatic identification of the drainage state of the river channel drainage port, is beneficial to timely finding out problems of relevant parts, eliminates hidden danger and solves the river channel pollution problem from the source.
2. In order to filter the preliminary result judged by the drainage state deep learning model, the method filters and analyzes the preliminary judgment result of the drainage state deep learning model due to complex drainage environment, poor drainage picture quality and large noise interference, and filters and processes the situation that the abrupt change of the drainage amount is unreasonable according to the situation so as to improve the recognition rate of practical application. For example, when it is determined that water is drained for one frame or a plurality of consecutive frames but a large number of adjacent frames are not drained, the preliminary result is considered to be caused by noise interference of the pictures, and the final result does not report water.
3. The method comprises the steps of judging a drainage state by using a drainage state deep learning model, wherein the drainage state water flow average speed v0 is influenced by factors including illumination, environment, weather change, brightness and day and night interference, the drainage state is difficult to extract in video, the accuracy of directly judging the drainage state of the drainage by using a motion vector calculated by using a video image is not high because of the video image, the drainage state is difficult to directly judge the drainage state by using a motion vector gray level map because of a drainage flow shape, and the drainage state is difficult to directly judge by using the motion vector gray level map.
Drawings
FIG. 1 is a schematic diagram of a real-time identification step of a river drainage state based on video images;
FIG. 2 is a schematic diagram of a drainage state deep learning model building step according to the present invention;
FIG. 3 is an image of a drainage port according to the present invention;
FIG. 4 is a schematic diagram of an analytical solution of the drainage model according to the present invention;
FIG. 5 is a gray scale view of a motion vector of a drainage image according to the present invention;
FIG. 6 conversion of motion vector gray scale map into motion vector
FIG. 7 is a schematic diagram of a deep learning model structure in a drainage state according to the present invention.
Detailed Description
The following detailed description is made with reference to the accompanying drawings and detailed description of the invention:
A real-time recognition method for drainage state of a river drain based on video images comprises the following steps of firstly establishing a drainage state deep learning model, wherein the drainage state deep learning model is established by collecting drainage and non-drainage state videos of the river drain, splitting the videos into one frame of images, calculating to obtain motion vectors of each frame of images, labeling the motion vectors with drainage state label information, establishing the drainage state deep learning model, and training the drainage state deep learning model. And then, acquiring a real-time video stream of the river channel drainage port through a camera, splitting the video stream into a frame of image, calculating a motion vector of each frame of image, analyzing the motion vector of each frame of image by using an established drainage state deep learning model, outputting a preliminary drainage state result, analyzing and filtering the preliminary result, and outputting final drainage state information.
The method for establishing the river drainage model and setting initial conditions comprises the following steps:
establishing a river drainage model
ut+uux=0...............(1)
To describe the drain state of the drain, as shown in fig. 3;
In equation (1), t represents time, x represents water surface position, and u represents velocity wave
X=0 indicates the position of the water in the state where the water will be discharged through the nozzle but not through the nozzle, and when x is 0, the water is all in the pipe at a speed much slower than the speed after the pipe, so that it is assumed that u=0, the speed is suddenly increased after the water is discharged out of the pipe, the water is smoothly discharged through the drop, and the average speed is assumed to be v0, so that the initial condition of equation (1) can be defined as
Assuming delta is the minimum drain rate, when the rate v0 > delta >0, the drain can be determined and subsequent quantitative analysis, such as flow rate, etc., can be performed.
The solution of equation (1) is shown in FIG. 4, where in the region { x >0, t > x }, the velocity wave isIllustrating the change in discharge velocity with time. The change can be described by a motion vector, and the drainage state of the drainage port can be obtained by finding the motion vector. A motion vector (motion vector) is a difference between two frames of images stored in graphic compression, and is a vector describing a change in the spatial position of an object.
The motion vector v of the image is calculated as follows:
Defining a gray scale map (see FIG. 5) of a motion vector by draining the difference between two adjacent frames of gray scale images (gi,gj) with a drain
gv＝gi-gj,
The motion vector v is a one-dimensional expansion of its gray-scale map (see fig. 6). The motion vector may be further refined to a weighted average v= Σaivi of several neighboring motion vectors vi. Or using algorithm OptionalFlowFarneback to find vi and using v= Σaivi to find motion vector. The weighted average may also be an arithmetic average or a gaussian weighted average.
And judging the drainage state by using a drainage state deep learning model. The average velocity v0 of the water flow of the water discharge is difficult to extract in the video, because the video image is interfered by a lot of noise such as illumination, environment, weather change, brightness, day and night, the accuracy of directly judging the water discharge state of the water discharge by using the motion vector calculated by the image is not high, and the gray level diagram of the motion vector is greatly different due to the water discharge shape of the water discharge, so that the water flow state is difficult to be directly judged by the gray level diagram of the motion vector. The method establishes a drainage state deep learning model by using a deep learning method, and judges whether drainage is performed or not by using the model so as to improve the drainage state recognition precision and enhance the robustness.
The drainage state deep learning model building method specifically comprises the following steps:
(1) Collecting a large number of drain drainage videos and non-drainage videos;
(2) And calculating the motion vector of each frame of image in the video, wherein the specific calculation method is shown in the 3 rd point of the specific embodiment mode, and a motion vector gray level diagram is formed. The motion vector gray map may be approximated by the difference between the two images of the previous and subsequent frames, or may be determined by a correlation algorithm such as OptionalFlowFarneback.
(3) Labeling drainage state label information for each motion vector gray level diagram, wherein the label value is the state of the water flow corresponding to the motion vector, and the specific labeling method is as follows: the label information is classified into a no-flow state, a low-speed flow state, a medium-speed flow state and a high-speed flow state, and the flow speed values of the specific low-speed flow state, the medium-speed flow state and the high-speed flow state are actually adjusted according to specific conditions, and generally 0 m/s, 1 m/s, 2 m/s and 5 m/s can be adopted.
(4) And establishing a drainage state deep learning model, wherein the drainage state deep learning model is established by training consisting of a plurality of convolution graphs and a plurality of full connection graphs. The convolution layer and the full connection layer of the model can be set according to the situation, and the patent is not limited. Drainage state deep learning model frame schematic (see fig. 7), which is also the model we employ.
(5) And training a model. By adopting a conventional CNN convolutional neural network training method, the recognition accuracy during convergence reaches more than 99%, and the training learning rate is 0.0001.
The river channel drainage state real-time identification method based on the video image comprises the following steps:
(1) Acquiring a real-time video stream of a river channel outlet through a camera;
(2) Splitting a video stream into a frame image;
(3) Calculating to obtain a motion vector of each frame of image;
(4) Analyzing the motion vector of each frame of image by using the established drainage state deep learning model to output a preliminary drainage state result;
(5) The method is used for filtering and analyzing the preliminary drainage state result of the drainage state deep learning model, and filtering and processing the preliminary judgment result of the drainage state deep learning model according to the situation of unreasonable sudden change of the drainage amount to improve the recognition rate of practical application. For example, when it is determined that water is drained for one frame or a plurality of consecutive frames but a large number of adjacent frames are not drained, the preliminary result is considered to be caused by noise interference of the pictures, and the final result does not report water.
The method realizes real-time unmanned monitoring and automatic identification of the drainage state of the river channel, is beneficial to timely finding out problems of relevant parts, eliminates hidden danger and solves the problem of river channel pollution from the source, judges the drainage state by using a drainage state deep learning model, and judges whether the drainage state is drained or not by using the model to improve the drainage state identification precision and the robustness by using the model, wherein the drainage state is difficult to extract in videos due to the fact that video images are used for calculating motion vectors, and the drainage state is not high in precision due to the fact that the motion vector gray level diagram of the drainage state is large in difference due to the fact that the drainage flow shape is too large.