CN112115928B - Training method and detection method of neural network based on illegal parking vehicle labels - Google Patents

Abstract

The invention discloses a training method and a detection method of a neural network based on illegal parking vehicle labels. The method comprises the following steps: acquiring a first set of street view images containing parked vehicles and a second set of street view images containing vehicles marked with violating parking; inputting the first set into a first convolutional neural network and a first fully-connected layer to obtain a first feature vector; inputting the second set into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector; calculating a mean square error loss function of the first eigenvector and the second eigenvector; classifying the first feature vector with a Softmax function to obtain a Softmax loss function; and updating model parameters based on a weighted sum of the mean square error loss function and the Softmax loss function. In this way, the first feature vector can better identify the features that can represent the vehicle parking violation, so as to improve the accuracy of the vehicle parking violation detection.

Description

Training method and detection method of neural network based on illegal parking vehicle labels

Technical Field

The present application relates to the field of artificial intelligence technology, and more particularly, to a training method for neural network based on illegal parking vehicle labels, a vehicle illegal parking detection method, a training device, a detection device, and an electronic device.

Background

The smart city is a city informatization advanced form which fully applies a new generation of information technology to various industries in the city. Along with the continuous acceleration of urbanization process, a large amount of population pours into the city, and the quantity of private car increases rapidly, leads to urban vehicle quantity to increase rapidly, and urban parking stall is in short supply, leads to the vehicle phenomenon of violating the regulations and is very serious, and urban traffic is chaotic or paralyzed even, causes huge economic loss. Therefore, vehicles must be effectively supervised to ensure the normal operation of urban traffic, and the most direct and effective method for supervising the vehicles is to copy illegal vehicles.

The traditional vehicle illegal parking detection and the plate reading are finished by means of a traffic police patrol ticket-sticking method, the method for manually supervising the vehicle is time-consuming and labor-consuming, the coverage is not wide, and the vehicle is easy to miss detection.

At present, deep learning and neural networks have been widely applied in the fields of computer vision, natural language processing, speech signal processing, and the like. In addition, deep learning and neural networks also exhibit a level close to or even exceeding that of humans in the fields of machine vision, image classification, object detection, semantic segmentation, text translation, and the like.

Deep learning and development of neural networks (particularly development of machine vision technology) provide solutions and schemes for vehicle violation detection.

Disclosure of Invention

The present application is proposed to solve the above-mentioned technical problems. The embodiment of the application provides a training method, a vehicle illegal parking detection method, a training device, a detection device and electronic equipment of a neural network based on illegal parking vehicle labels, wherein the neural network used for vehicle illegal parking detection is trained by taking street view images containing parked vehicles and street view images containing illegal parking vehicles marked as training image sets, so that the characteristics of the illegal parking vehicles can be better extracted by a detection model finally trained, and the classification accuracy is improved.

According to one aspect of the present application, there is provided a method for training a neural network based on violating parking vehicle labels, comprising:

acquiring a first set of street view images each containing a parked vehicle and a second set of street view images containing vehicles marked with violating parking;

inputting the first set of street view images into a first convolutional neural network and a first fully-connected layer to obtain a first feature vector;

inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector, wherein the second convolutional neural network has the same network structure as the first convolutional neural network;

calculating a mean square error loss function of the first eigenvector and the second eigenvector;

classifying the first feature vector with a Softmax function to obtain a Softmax loss function; and

updating parameters of the first convolutional neural network, first fully-connected layer, the second convolutional neural network, and second fully-connected layer based on a weighted sum of the mean-square error loss function and the Softmax loss function.

In the above training method based on neural network of the illegal parking vehicle label, in the second set including the street view images marked with illegal vehicles, the illegal vehicles are marked in each street view image in a candidate frame form;

inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector comprises:

calculating the position coordinates of the center of the candidate frame relative to the street view image in each street view image; and

and inputting the position coordinates as the label value of the street view image and the street view image into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector.

In the above training method based on neural network of illegal parking vehicle labels, in a second set including the street view images marked with illegal parking vehicles, the illegal parking vehicles are marked in each street view image in the form of label information, and the label information includes illegal parking descriptions of the illegal parking vehicles;

inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector comprises:

converting each street view image and the corresponding label information into a mixed feature vector through multi-mode feature vector transformation; and

inputting the hybrid feature vector into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector.

In the training method of the neural network based on the illegal parking vehicle label, the illegal parking description of the illegal parking vehicle comprises the license plate number, the vehicle type, the color, the illegal place and the illegal time of the vehicle.

In the above method for training a neural network based on a parking violation label, updating parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network and the second fully-connected layer based on a weighted sum of the mean square error loss function and the Softmax loss function includes: iteratively updating parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer based on a weighted sum of the mean-square-error-loss function and the Softmax-loss function, wherein, in each iteration, the parameters of the first convolutional neural network and the first fully-connected layer are first fixed and the parameters of the second convolutional neural network and the second fully-connected layer are updated, and then the parameters of the second convolutional neural network and the second fully-connected layer are fixed and the parameters of the first convolutional neural network and the first fully-connected layer are updated.

According to another aspect of the present application, there is provided a method of detecting a vehicle parking violation, comprising:

obtaining a street view image containing a parked vehicle;

inputting the street view image into a first convolutional neural network trained according to the training method of the neural network based on the illegal parking vehicle labels, a first fully connected layer and a Softmax function, wherein the output of the Softmax function is a first probability representing that the parked vehicle belongs to illegal parking and a second probability representing that the parked vehicle does not belong to illegal parking; and

determining whether the parked vehicle violates based on the first probability and the second probability.

According to another aspect of the present application, there is provided a training apparatus based on a neural network of violating parking vehicle labels, including:

a training image set acquisition unit for acquiring a first set of street view images each containing a parked vehicle and a second set of street view images each containing a marked illegal vehicle;

a first feature vector generation unit, configured to input the street view images of the first set obtained by the training image set obtaining unit into a first convolutional neural network and a first fully-connected layer to obtain a first feature vector;

a second feature vector generation unit, configured to input the street view images of the second set obtained by the training image set obtaining unit into a second convolutional neural network and a second fully-connected layer to obtain a second feature map, where the second convolutional neural network and the first convolutional neural network have the same network structure;

a mean square error loss function calculation unit configured to calculate a mean square error loss function of the first eigenvector obtained by the first eigenvector generation unit and the second eigenvector obtained by the second eigenvector generation unit;

a Softmax loss function generating unit configured to classify the first feature vector obtained by the first feature map generating unit by a Softmax function to obtain a Softmax loss function; and

a parameter updating unit, configured to update parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer based on the weighted sum of the mean-square error loss function obtained by the mean-square error loss function calculating unit and the Softmax loss function obtained by the Softmax loss function generating unit.

In the above training apparatus based on a neural network of violating parking vehicle labels, the second feature vector generating unit further includes:

a candidate frame position coordinate calculating subunit, configured to calculate a position coordinate of a center of the candidate frame with respect to the street view image in each street view image; and

and the feature vector calculation subunit is configured to input the position coordinates as a tag value of the street view image together with the street view image into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector.

In the above training apparatus based on a neural network of violating parking vehicle labels, the second feature vector generating unit further includes:

the mixed feature vector generating subunit is used for converting each street view image and the corresponding label information into a mixed feature vector through multi-mode feature vector transformation; and

a feature vector generation subunit for inputting the mixed feature vector into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector

In the training device based on the neural network of the illegal parking vehicle labels, the illegal parking description of the illegal parking vehicle comprises the license plate number, the vehicle type, the color, the illegal place and the illegal time of the vehicle.

In the above training apparatus based on a neural network of violating parking vehicle labels, the parameter updating unit is further configured to: iteratively updating parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer based on a weighted sum of the mean-square-error-loss function and the Softmax-loss function, wherein, in each iteration, the parameters of the first convolutional neural network and the first fully-connected layer are first fixed and the parameters of the second convolutional neural network and the second fully-connected layer are updated, and then the parameters of the second convolutional neural network and the second fully-connected layer are fixed and the parameters of the first convolutional neural network and the first fully-connected layer are updated.

According to still another aspect of the present application, there is provided a vehicle parking violation detection apparatus, including:

the street view image acquisition unit is used for acquiring a street view image containing a parked vehicle;

a classification unit, configured to input the street view image obtained by the street view image obtaining unit into a first convolutional neural network trained by the training device based on the neural network based on the illegal parking vehicle tag as described above, a first fully-connected layer, and a Softmax function, where an output of the Softmax function is a first probability that the parked vehicle belongs to illegal parking and a second probability that the parked vehicle does not belong to illegal parking; and

a determination unit configured to determine whether the parked vehicle is illegal based on the first probability and the second probability obtained by the classification unit.

According to still another aspect of the present application, there is provided an electronic device including: a processor; and a memory in which computer program instructions are stored, which computer program instructions, when executed by the processor, cause the processor to perform the method of training a neural network based on violating parking vehicle signatures as described above or the method of detecting vehicle violations as described above.

According to yet another aspect of the present application, a computer-readable medium is provided, having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the method of training based on a neural network of violating parking vehicle signatures as described above or the method of detecting vehicle violations as described above.

Compared with the prior art, the illegal parking vehicle label-based neural network training method, the illegal parking vehicle detection method, the training device, the detection device and the electronic equipment train the neural network for detecting the illegal parking vehicle by using the street view image containing the parked vehicle and the street view image containing the illegal parking vehicle marked as the training image set, so that the finally trained detection model can reduce the number of low-probability sample points of the feature map for classification in the image space and can better extract the features of the illegal parking vehicle, and the classification accuracy is improved.

Drawings

The above and other objects, features and advantages of the present application will become more apparent by describing in more detail embodiments of the present application with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application. In the drawings, like reference numbers generally represent like parts or steps.

Fig. 1 illustrates an application scenario of a training method based on a neural network of illegal parking vehicle labels and a vehicle illegal parking detection method according to an embodiment of the application.

Fig. 2 illustrates a flowchart of a method for training a neural network based on violating vehicle labels according to an embodiment of the present application.

Fig. 3 illustrates a schematic diagram of a system architecture of a training method based on a neural network of violating parking vehicle labels according to an embodiment of the present application.

FIG. 4 illustrates a flow chart of a method of detecting vehicle parking according to an embodiment of the present application.

Fig. 5 illustrates a block diagram schematic diagram of a training apparatus based on a neural network of violating parking vehicle labels according to an embodiment of the present application.

Fig. 6 illustrates a block diagram of a second feature vector generation unit in a training apparatus using a neural network of violating parking vehicle labels according to an embodiment of the present application.

Fig. 7 illustrates another block diagram of a second feature vector generation unit in the training apparatus of the neural network of the illicit vehicle label according to the embodiment of the present application.

FIG. 8 illustrates a block diagram of a vehicle violation detection device according to an embodiment of the present application.

FIG. 9 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application.

Detailed Description

Hereinafter, example embodiments according to the present application will be described in detail with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present application and not all embodiments of the present application, and that the present application is not limited by the example embodiments described herein.

Overview of a scene

As mentioned above, the traditional vehicle illegal parking detection and the card reading are completed by means of the traffic police patrol ticket, and the method for manually supervising the vehicle is time-consuming and labor-consuming, has a small coverage area and is easy to miss detection. Deep learning and development of neural networks (particularly development of machine vision technology) provide solutions and schemes for vehicle violation detection.

Specifically, when illegal parking detection of vehicles on a road is performed, a large number of street view images including parked vehicles can be obtained. Whether the vehicle is parked or not can be viewed as a binary classification problem for the images. Some existing solutions based on the deep neural network simply perform binary classification through a network structure of the convolutional neural network and the Softmax classifier, are not high in detection accuracy, and cannot be practically applied.

The inventor of the present application finds that the above scheme affects the detection rate in part because the obtained feature map for classification includes a large number of low probability sample points in the image space thereof, because in street view images including parked vehicles, there are many images corresponding to the same street view, and thus a large number of street view images including parked vehicles are similar in their background but differ in the details of the parked vehicles, that is, are generally similar but differ in details. When these street view images are passed through a convolutional neural network as an image set, a large number of low-probability sample points exist in the image space obtained by passing through the convolutional neural network, thereby affecting the accuracy of subsequent classification.

Aiming at the technical problem, the basic idea of the application is that a neural network for detecting vehicle illegal parking is trained by taking street view images containing parked vehicles and street view images containing vehicles marked with illegal parking as training image sets, so that the characteristics of the illegal parking vehicles can be better extracted by a detection model finished by final training, and the classification accuracy is improved.

Based on this, the application provides a training method based on a neural network of the illegal parking vehicle label, which comprises the following steps: acquiring a first set of street view images each containing a parked vehicle and a second set of street view images containing vehicles marked with violating parking; inputting the first set of street view images into a first convolutional neural network and a first fully-connected layer to obtain a first feature vector; inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector, wherein the second convolutional neural network has the same network structure as the first convolutional neural network; calculating a mean square error loss function of the first eigenvector and the second eigenvector; classifying the first feature vector with a Softmax function to obtain a Softmax loss function; and updating parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer based on a weighted sum of the mean-square-error-loss function and the Softmax-loss function.

Specifically, the street view image marked with the illegal vehicle is used as a reference image, the reference image is mapped into an image space through a second convolutional neural network with the same structure, and a second feature vector is obtained through a second full-connection layer and can be used for extracting features capable of representing the illegal vehicle parking. Furthermore, by calculating a mean square error loss function between the first feature vector and the second feature vector and updating the first convolutional neural network in this way, the first feature vector can be better used for extracting features for identifying vehicle parking violation. Meanwhile, the low-probability sample points of the first feature map (which is converted into the first feature vector through the first full-connection layer) in the image space where the first feature map is located are indirectly reduced, so that the classification accuracy is improved; or the low-probability sample point distribution of the first feature map in the image space where the first feature map is located is ignored to a certain extent after passing through the first full-connection layer, so that the classification accuracy is improved.

Accordingly, the purpose of the first feature vector is to classify the vehicle violation, and the second set of images includes the mark of the vehicle violation, so that the first convolutional neural network, the first fully-connected layer, the second convolutional neural network and the second fully-connected need to be trained in conjunction with the mean-square-error-loss function after the Softmax-loss function is obtained by the Softmax classifier with the first feature vector, thereby improving the accuracy of the vehicle violation detection.

Accordingly, according to another aspect of the present application, there is also provided a method for detecting vehicle parking violation, comprising: obtaining a street view image containing a parked vehicle; inputting the street view image into a first convolutional neural network trained according to the training method of the neural network based on the illegal parking vehicle labels, a first fully connected layer and a Softmax function, wherein the output of the Softmax function is a first probability representing that the parked vehicle belongs to illegal parking and a second probability representing that the parked vehicle does not belong to illegal parking; and determining whether the parked vehicle violates based on the first probability and the second probability.

Fig. 1 illustrates an application scenario of a training method based on a neural network of illegal parking vehicle labels and a vehicle illegal parking detection method according to an embodiment of the application.

As shown in fig. 1, in this application scenario, in the training phase, a camera (e.g., C as illustrated in fig. 1) acquires a street view picture containing a parked vehicle and a marked illegal vehicle (e.g., V as illustrated in fig. 1); and inputs it as a training image set into a training server of a neural network for vehicle violation detection (e.g., S illustrated in fig. 1). After the training is completed, in the detection phase, the street view image containing the parked vehicle collected by the camera is input into a server for vehicle illegal parking detection, wherein a trained convolutional neural network model (e.g., S as illustrated in fig. 1) is deployed on the server to output a determination result whether the parked vehicle is illegal parking.

Having described the general principles of the present application, various non-limiting embodiments of the present application will now be described with reference to the accompanying drawings.

Exemplary method

Fig. 2 illustrates a flowchart of a method for training a neural network based on violating vehicle labels according to an embodiment of the present application.

As shown in fig. 2, a method for training a neural network based on violating parking vehicle labels according to an embodiment of the present application includes: s110, acquiring a first set of street view images containing parked vehicles and a second set of street view images containing vehicles marked with illegal parking; s120, inputting the street view images of the first set into a first convolution neural network and a first full-connection layer to obtain a first feature vector; s130, inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector, wherein the second convolutional neural network and the first convolutional neural network have the same network structure; s140, calculating a mean square error loss function of the first eigenvector and the second eigenvector; s150, classifying the first feature vector by a Softmax function to obtain a Softmax loss function; and S160, updating parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network and the second fully-connected layer based on the weighted sum of the mean square error loss function and the Softmax loss function.

Fig. 3 illustrates a schematic diagram of a system architecture of a training method based on a neural network of violating parking vehicle labels according to an embodiment of the present application. In an embodiment of the present application, the system architecture includes: a first convolutional neural network (e.g., CNN1 as illustrated IN fig. 3), a second convolutional neural network (e.g., CNN2 as illustrated IN fig. 3), a first fully-connected layer (e.g., Fcl1 as illustrated IN fig. 3), a second fully-connected layer (e.g., Fcl2 as illustrated IN fig. 3) Softmax function (e.g., S as illustrated IN fig. 3), and a mean-square-error-loss function (e.g., Ml as illustrated IN fig. 3), wherein a first set of street view images (e.g., IN0 as illustrated IN fig. 3) containing parked vehicles is input to the first convolutional neural network for convolution processing images IN the first set to generate a first feature map (e.g., F1 as illustrated IN fig. 3); the first fully-connected layer is configured to translate the first feature map translated feature vector into a first feature vector for classification (e.g., V1 as illustrated in fig. 3), wherein the first feature vector is classified with a Softmax function to obtain a Softmax loss function; a second set (e.g., IN1 as illustrated IN fig. 3) containing the street view images marked with the parked vehicles is input to the second convolutional neural network having the same network structure as the first convolutional neural network, which is used for convolution processing of the images IN the second set to generate a second feature map (e.g., F2 as illustrated IN fig. 3); the second fully-connected layer is used for converting the feature vector converted by the second feature map into a second feature vector for classification (for example, V2 as illustrated in FIG. 3). Accordingly, after obtaining the first feature map and the second feature map, a mean square error loss function of the first feature vector and the second feature vector is calculated. Accordingly, the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer are updated based on a weighted sum of the mean square error loss function and the Softmax loss function.

In step S110, a first set of street view images each containing a parked vehicle and a second set of street view images each containing a marked illegal vehicle are acquired. That is, in the embodiment of the present application, the training image data set includes two types: a street view image set containing parked vehicles and a street view image set containing marked vehicles with violations.

In step S120, the first set of street view images is input into a first convolutional neural network and a first fully-connected layer to obtain a first feature vector. Specifically, the process firstly comprises the steps that the first set of streetscape images are input into the first convolution neural network and are subjected to convolution processing of the first convolution neural network to generate a first feature map; then, the first fully connected layer processes the feature vector converted based on the first feature map to generate a first feature vector for classification, wherein the first feature vector can extract features of the parking vehicle.

It should be understood that in street view images containing parked vehicles, it is possible that multiple images correspond to the same street view. Therefore, a large number of street view images containing parked vehicles are similar in background and different in details of the parked vehicles, that is, generally similar and different in details, and when these street view images are passed through the first convolutional neural network as an image set, a large number of low-probability sample points exist in the image space of the first feature map obtained by passing through the first convolutional neural network. Because of the existence of these low probability sample points, the first feature vector cannot extract well the features representing the vehicle violation.

In step S130, the second set of street view images is input into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector, where the second convolutional neural network has the same network structure as the first convolutional neural network. Specifically, the process firstly includes that the second set of streetscape images are input into the second convolutional neural network and are subjected to convolution processing by the second convolutional neural network to generate a second feature map; then, the second fully connected layer processes the feature vector transformed based on the second feature map to generate a second feature vector for classification.

That is, in the embodiment of the present application, an image including an illegal vehicle is used as a reference image, the reference image is mapped into an image space through a convolutional neural network having the same structure, and a second feature vector is obtained through a full connection layer, and features capable of indicating vehicle illegal parking can be extracted from the second feature vector.

In one example of the present application, in the second set comprising street view images marked with illicit vehicles, the illicit vehicles are marked in each of the street view images in the form of candidate boxes. Accordingly, inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector comprises: calculating the position coordinates of the center of the candidate frame relative to the street view image in each street view image; and inputting the position coordinates as a tag value of the street view image together with the street view image into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector.

In particular, by marking the illegal vehicle in the form of a candidate frame and using the center position coordinates of the candidate frame as the tag value, the spatial position feature of the illegal vehicle relative to the street view image can be extracted from the obtained second feature vector, thereby well extracting the feature capable of representing the illegal vehicle parking.

In another example of the present application, in the second set including the street view images marked with illegal vehicles, the illegal vehicles are marked in each street view image in the form of tag information, and the tag information includes illegal description of the illegal vehicles. Accordingly, inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector comprises: converting each street view image and the corresponding label information into a mixed feature vector through multi-mode feature vector transformation; and inputting the hybrid feature vector into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector.

Particularly, the illegal parking vehicles are marked in the form of label information, the label information and the street view image are converted into the feature vectors and are input into the second convolutional neural network and the second fully connected layer, so that the image features of the whole illegal parking vehicles can be extracted from the obtained second feature vectors, and the features capable of representing the illegal parking of the vehicles can be well extracted.

In the above examples, the description of the violation of parking of the violation vehicle includes, but is not limited to, the license plate, model, color, place of violation, time of violation, etc. of the vehicle.

In step S140, a mean square error loss function of the first eigenvector and the second eigenvector is calculated. Correspondingly, by calculating the mean square error loss function between the first feature vector and the second feature vector and updating the first convolution neural network, the first feature vector can also better extract features for identifying vehicle parking violation, that is, low-probability sample points of the first feature map in the image space where the first feature map is located are indirectly reduced, or the low-probability sample points of the first feature map in the image space where the first feature map is located are distributed and ignored to a certain extent after passing through the first full-connection layer, so that the classification accuracy is improved.

In step S150, the first feature vector is classified with a Softmax function to obtain a Softmax loss function. Here, the purpose of the first feature vector is to perform classification of vehicle violations.

Updating parameters of the first convolutional neural network, first fully-connected layer, the second convolutional neural network, and second fully-connected layer based on a weighted sum of the mean square error loss function and the Softmax loss function in step S160. Preferably, in this embodiment of the present application, updating parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer based on a weighted sum of the mean square error loss function and the Softmax loss function includes: iteratively updating parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer based on a weighted sum of the mean-square-error-loss function and the Softmax-loss function, wherein, in each iteration, the parameters of the first convolutional neural network and the first fully-connected layer are first fixed and the parameters of the second convolutional neural network and the second fully-connected layer are updated, and then the parameters of the second convolutional neural network and the second fully-connected layer are fixed and the parameters of the first convolutional neural network and the first fully-connected layer are updated.

Accordingly, by sequentially updating the second convolutional neural network and the second fully-connected layer and the first convolutional neural network and the first fully-connected layer in each iteration, the situation that parameters are excessively diverged due to the fact that the first convolutional neural network, the first fully-connected layer, the second convolutional neural network and the second fully-connected layer are updated at the same time can be avoided, and convergence of the parameters of the convolutional neural network is facilitated;

in addition, by updating the second convolutional neural network and the second fully-connected layer first and then updating the first convolutional neural network and the first fully-connected layer, the first convolutional neural network and the first fully-connected layer can be updated with more fully extracted features representing vehicle parking violation, so that the detection accuracy is improved. Specifically, the features extracted by the second convolutional neural network can be helpful for identifying whether the vehicle is determined to be illegal, such as lane line positions, road positions, whether a parking space line is drawn, and the like.

In summary, the method for training the neural network based on the illegal parking vehicle label according to the embodiment of the present application is clarified, the neural network for vehicle illegal parking detection is trained by using the street view image containing the parked vehicle and the street view image containing the illegal parking vehicle as the training image set, and the detection model finally trained can reduce the number of low probability sample points in the image space of the feature map for classification, and can better extract the features of the illegal parking vehicle, thereby improving the classification accuracy.

Further, according to another aspect of the present application, a method for detecting vehicle parking violation is also provided. As shown in fig. 4, the method for detecting vehicle parking violation includes: s210, obtaining a street view image containing a parked vehicle; s220, inputting the street view image into a first convolution neural network trained according to the training method of the neural network based on the illegal parking vehicle labels, a first full connection layer and a Softmax function, wherein the output of the Softmax function is a first probability that the parked vehicle belongs to illegal parking and a second probability that the parked vehicle does not belong to illegal parking; and S230, determining whether the parked vehicle violates based on the first probability and the second probability.

Exemplary devices

Fig. 5 illustrates a block diagram schematic diagram of a training apparatus based on a neural network of violating parking vehicle labels according to an embodiment of the present application.

As shown in fig. 5, the training apparatus 500 based on neural network of violating parking vehicle labels according to the embodiment of the present application includes: a training image set acquisition unit 510 for acquiring a first set of street view images each including a parked vehicle and a second set of street view images each including a parked vehicle marked with an illegal vehicle; a first feature vector generating unit 520, configured to input the first set of street view images obtained by the training image set obtaining unit 510 into a first convolutional neural network and a first fully-connected layer to obtain a first feature vector; a second feature vector generating unit 530, configured to input the second set of streetscape images obtained by the training image set obtaining unit 510 into a second convolutional neural network and a second fully-connected layer to obtain a second feature map, where the second convolutional neural network has the same network structure as the first convolutional neural network; a mean square error loss function calculating unit 540, configured to calculate a mean square error loss function of the first eigenvector obtained by the first eigenvector generating unit 520 and the second eigenvector obtained by the second eigenvector generating unit 530; a Softmax loss function generating unit 550, configured to classify the first feature vector obtained by the first feature map generating unit by a Softmax function to obtain a Softmax loss function; and a parameter updating unit 560 configured to update the parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer based on the weighted sum of the mean square error loss function obtained by the mean square error loss function calculating unit 540 and the Softmax loss function obtained by the Softmax loss function generating unit 550.

In one example, as shown in fig. 6, in thetraining apparatus 500 based on a neural network of violating parking vehicle labels, the second featurevector generating unit 530 further includes: a candidate frame position coordinatecalculation subunit 531, configured to calculate position coordinates of the center of the candidate frame with respect to the street view image in each street view image; and a featurevector calculation subunit 532, configured to input the position coordinates as a tag value of the street view image together with the street view image into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector.

In one example, as shown in fig. 7, in thetraining apparatus 500 based on a neural network of violating parking vehicle labels, the second featurevector generating unit 530 further includes: a mixed featurevector generating subunit 533, configured to convert each street view image and the tag information corresponding to the street view image into a mixed feature vector through multi-modal feature vector transformation; and a featurevector generation subunit 534 for inputting the mixed feature vector into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector

In one example, in thetraining apparatus 500 based on neural network of illegal parking vehicle labels, the illegal parking description of the illegal parking vehicle includes the license plate, model, color, place of violation, and time of violation of the vehicle.

In an example, in the above training apparatus based on neural network of violating parking vehicle labels, theparameter updating unit 560 is further configured to: iteratively updating parameters of the first convolutional neural network, the first fully-connected layer, the second convolutional neural network, and the second fully-connected layer based on a weighted sum of the mean-square-error-loss function and the Softmax-loss function, wherein, in each iteration, the parameters of the first convolutional neural network and the first fully-connected layer are first fixed and the parameters of the second convolutional neural network and the second fully-connected layer are updated, and then the parameters of the second convolutional neural network and the second fully-connected layer are fixed and the parameters of the first convolutional neural network and the first fully-connected layer are updated.

According to yet another aspect of the present application, a vehicle parking violation detection apparatus is provided.

FIG. 8 illustrates a block diagram of a vehicle violation detection device according to an embodiment of the present application.

As shown in fig. 8, theapparatus 800 for detecting an illegal vehicle according to the embodiment of the present application includes: a street viewimage obtaining unit 810 for obtaining a street view image including a parked vehicle; aclassification unit 820, configured to input the street view image obtained by the street viewimage obtaining unit 810 into a first convolutional neural network trained by the training apparatus based on the neural network based on the illegal parking vehicle tag as described above, a first fully-connected layer, and a Softmax function, an output of which is a first probability representing that the parked vehicle belongs to illegal parking and a second probability that the parked vehicle does not belong to illegal parking; and a determiningunit 830 for determining whether the parked vehicle is illegal based on the first probability and the second probability obtained by the classifyingunit 820.

Here, it can be understood by those skilled in the art that the specific functions and operations of the respective units and modules in the above-describedtraining apparatus 500 based on the neural network of the illegal parking vehicle tag and thedetection apparatus 800 of the vehicle illegal parking have been described in detail in the above description of the training method based on the neural network of the illegal parking vehicle tag and the detection method of vehicle illegal parking with reference to fig. 1 to 4, and thus, the repeated description thereof will be omitted.

As described above, thetraining apparatus 500 based on neural network of illegal parking vehicle labels and theapparatus 800 for detecting vehicle illegal parking according to the embodiments of the present application can be implemented in various terminal devices, such as a server for vehicle illegal parking detection and the like. In one example, thetraining apparatus 500 based on neural network of violating parking vehicle labels and/or thedetection apparatus 800 of vehicle violations according to the embodiments of the present application may be integrated into a terminal device as a software module and/or a hardware module. For example, thetraining apparatus 500 based on neural network of illegal parking vehicle labels and/or thedetection apparatus 800 of vehicle illegal parking may be a software module in the operating system of the terminal device, or may be an application developed for the terminal device; of course, thetraining device 500 based on the neural network of the illegal parking vehicle labels and/or thedetection device 800 of the vehicle illegal parking can also be one of the hardware modules of the terminal device.

Alternatively, in another example, thetraining apparatus 500 based on the neural network of violating parking vehicle labels and/or thedetection apparatus 800 of violating parking of the vehicle and the terminal device may also be separate devices, and thetraining apparatus 500 based on the neural network of violating parking vehicle labels and/or thedetection apparatus 800 of violating parking of the vehicle may be connected to the terminal device through a wired and/or wireless network and transmit the interactive information according to an agreed data format.

Exemplary electronic device

Next, an electronic apparatus according to an embodiment of the present application is described with reference to fig. 9.

FIG. 9 illustrates a block diagram of an electronic device in accordance with an embodiment of the present application.

As shown in fig. 9, the electronic device 10 includes one ormore processors 11 andmemory 12.

Theprocessor 11 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

Memory 12 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer readable storage medium and executed by theprocessor 11 to implement the above-described methods for training a neural network based on violating parking vehicle labels and methods for detecting vehicle violations of the various embodiments of the present application, and/or other desired functions. Various contents such as a mean square error loss function value, a street view image, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 10 may further include: aninput device 13 and anoutput device 14, which are interconnected by a bus system and/or other form of connection mechanism (not shown).

Theinput device 13 may include, for example, a keyboard, a mouse, and the like.

Theoutput device 14 can output various information including the detection result of the vehicle parking violation and the like to the outside. Theoutput devices 14 may include, for example, a display, speakers, a printer, and a communication network and its connected remote output devices, among others.

Of course, for simplicity, only some of the components of the electronic device 10 relevant to the present application are shown in fig. 9, and components such as buses, input/output interfaces, and the like are omitted. In addition, the electronic device 10 may include any other suitable components depending on the particular application.

Exemplary computer program product and computer-readable storage Medium

In addition to the above-described methods and apparatus, embodiments of the present application may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in the methods for training a neural network based on violating parking vehicle signatures and methods for detecting vehicle violations according to various embodiments of the present application described in the "exemplary methods" section of this specification above.

The computer program product may be written with program code for performing the operations of embodiments of the present application in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present application may also be a computer readable storage medium having stored thereon computer program instructions, which, when executed by a processor, cause the processor to perform the steps of the method for training a neural network based on violating parking vehicle labels and the method for detecting vehicle violations according to various embodiments of the present application described in the above section "exemplary methods" of this specification.

The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The foregoing describes the general principles of the present application in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present application are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present application. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the foregoing disclosure is not intended to be exhaustive or to limit the disclosure to the precise details disclosed.

The block diagrams of devices, apparatuses, systems referred to in this application are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," "comprising," "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

It should also be noted that in the devices, apparatuses, and methods of the present application, the components or steps may be decomposed and/or recombined. These decompositions and/or recombinations are to be considered as equivalents of the present application.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the application to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims (4)

1. A method for training a neural network based on a violated vehicle label is characterized by comprising the following steps:

acquiring a first set of street view images each containing a parked vehicle and a second set of street view images containing vehicles marked with violating parking;

inputting the first set of street view images into a first convolutional neural network and a first fully-connected layer to obtain a first feature vector;

inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector, wherein the second convolutional neural network has the same network structure as the first convolutional neural network;

calculating a mean square error loss function of the first eigenvector and the second eigenvector;

classifying the first feature vector with a Softmax function to obtain a Softmax loss function; and

updating parameters of the first convolutional neural network, first fully-connected layer, the second convolutional neural network, and second fully-connected layer based on a weighted sum of the mean-square error loss function and the Softmax loss function;

wherein, in a second set comprising the street view images marked with illegal vehicles, the illegal vehicles are marked in a candidate frame form in each street view image;

inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector, comprising:

calculating the position of the center of the candidate frame relative to the street view image in each street view image; and

inputting the position coordinates as a tag value of the street view image together with the street view image into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector;

wherein, in a second set comprising the street view images marked with illegal vehicles, the illegal vehicles are marked in each street view image in the form of label information, and the label information comprises illegal description of the illegal vehicles;

inputting the second set of street view images into a second convolutional neural network and a second fully-connected layer to obtain a second feature vector, comprising:

converting each street view image and corresponding label information into a mixed feature vector through a multi-mode feature vector; and

inputting the hybrid feature vector into the second convolutional neural network and the second fully-connected layer to obtain the second feature vector;

wherein updating parameters of the first convolutional neural network, first fully-connected layer, the second convolutional neural network, and second fully-connected layer based on a weighted sum of the mean-square-error-loss function and the Softmax-loss function comprises:

iteratively updating parameters of the first convolutional neural network, first fully-connected layer, the second convolutional neural network, and second fully-connected layer based on a weighted sum of the mean-square-error-loss function and the Softmax-loss function,

in each iteration, firstly, the parameters of the first convolutional neural network and the first fully-connected layer are fixed, the parameters of the second convolutional neural network and the second fully-connected layer are updated, and then the parameters of the second convolutional neural network and the second fully-connected layer are fixed, and the parameters of the first convolutional neural network and the first fully-connected layer are updated.

2. The method for training on the neural network based on the illegal parking vehicle labels as claimed in claim 1, wherein the description of the illegal parking vehicle comprises the license plate, model, color, place of illegal parking, and time of illegal parking of the vehicle.

3. A method for detecting a vehicle parking violation, comprising:

obtaining a street view image containing a parked vehicle;

inputting the street view image into a first convolutional neural network trained by the method for training a neural network based on the labels of the parking vehicles according to any one of claims 1 to 2, a first fully-connected layer and a Softmax function, the output of the Softmax function being a first probability representing that the parked vehicle belongs to the parking violation and a second probability that the parked vehicle does not belong to the parking violation; and

determining whether the parked vehicle violates based on the first probability and the second probability.

4. An electronic device, comprising:

a processor; and

a memory in which computer program instructions are stored, which, when executed by the processor, cause the processor to perform the method of illicit vehicle label-based neural network training according to any one of claims 1-2 or the method of detecting vehicle illicit according to claim 3.

Images