CN109726717A - A vehicle comprehensive information detection system - Google Patents

Abstract

Translated fromChinese

一种车辆综合信息检测系统，可对车辆的车型型号以及车牌信息进行检测识别，具有较高的准确性和较快的检测速度。传统检测方法只能对车辆的车牌信息进行检测，无法得到车辆的整体信息，在实际应用中无法有效的解决套牌车、肇事逃逸等交通问题，同时传统检测方法只能解决较为简单场景下的车牌识别问题，在复杂的实际应用中不具有鲁棒性。而本发明结合了深度学习等理论，能快速、准确地实现车辆定位和车牌识别，可适用于各种环境下的检测对象，同时在车型识别的应用中也表现出了高效性和准确性。将该系统与现有交通系统结合可有效地解决各种交通问题，同时提高整个交通系统的车辆检测效率。

The utility model relates to a vehicle comprehensive information detection system, which can detect and identify the vehicle model type and license plate information, and has higher accuracy and faster detection speed. The traditional detection method can only detect the license plate information of the vehicle, and cannot obtain the overall information of the vehicle. In practical applications, it cannot effectively solve the traffic problems such as fake license plates and hit-and-run. At the same time, the traditional detection method can only solve the problems in relatively simple scenarios The license plate recognition problem is not robust in complex practical applications. The invention combines deep learning and other theories, can quickly and accurately realize vehicle positioning and license plate recognition, can be applied to detection objects in various environments, and also shows high efficiency and accuracy in the application of vehicle recognition. Combining this system with the existing traffic system can effectively solve various traffic problems while improving the vehicle detection efficiency of the entire traffic system.

Description

A kind of vehicle comprehensive information detection system

Technical field

The present invention is based on deep learning theories, design a kind of high-precision vehicle comprehensive information detection system, can be used for intelligenceIn traffic system, detection inquiry is carried out to information of vehicles.

Background technique

China just stepped into automotive society early in 2012, under car ownership continuously and healthily increased situation, to meTraffic, the energy and the ecological environment etc. of state all form huge challenge, such as the generally existing traffic jam in each big city,A series of problems, such as parking difficulty, traffic accident.Intelligent transportation system plays core branch in entire traffic management systemThe effect of column is a kind of accurate, in real time, efficiently and can play the comprehensive of most important effect in broad spectrum and various aspectsClose transport and management system.Intelligent transportation system can effectively reduce traffic jam, save the energy, reduce accident rate andExhaust emissions amount.Therefore, intelligent transportation system has very important facilitation for the development of current China.Natural sceneThe vehicle detection of image and the important component for being intelligent transportation system, while being also current important research content, haveBroad application prospect, such as expressway tol lcollection, social safety, hit-and-run, road occupying violating the regulations and purposive vehiclePrecise search etc..The real-time and accuracy of vehicle detection are the premises that intelligent transportation system plays a role, and license plate is for vapourThe identity card that the Chinese people are equivalent to for vehicle is the same, with unique identification blip facility, by with car plate detectionIn conjunction with vehicle administration office can further excavate automobile and the relevant information of car owner, thus accurately and rapidly identify complicated realityLicense board information in scene is just particularly important.Therefore, license plate is merged again on the basis of vehicle location and vehicle cab recognitionIdentification technology ultimately forms model recognition system and is then capable of handling more complicated situation, than as can quickly solving false-trademark vehicle, setThe problems such as board vehicle, traffic diverging.It is proposed by the present invention based on deep learning theory, computer vision, neural fusion vehicleDetection only needs to be merged with existing traffic video recording, monitoring system, and online high-precision rapid vehicle can be realized and examineIt surveys.Meanwhile the system does not need to destroy road surface, cost is relatively low, it is only necessary to install camera and corresponding in traffic intersectionData acquisition can be run with Transmission system, and the vehicle image data being collected into can be used for other traffic monitorings or big dataThe various aspects such as generaI investigation.

Summary of the invention

The present invention designs a kind of vehicle comprehensive information detection system, which can directly include the photograph of vehicle by detectionPiece or video information carry out the integrated information for quickly, accurately obtaining vehicle.Specific feature is as follows:

1. fast and accurately realizing vehicle location using the vehicle detecting algorithm based on SSD target detection；Using convolutionNeural network structure Inception-AB-Full carries out identification classification to vehicle classification.Above method solves process decision chartIt is no comprising vehicle and carrying out positioning two large problems to vehicle, can to arbitrary size in natural scene image, position it is multipleVehicle target, which is realized, to be accurately positioned, and has good robustness for noises such as illumination variations.

2. being positioned using the license plate locating method based on SSD target detection to license plate, using Radon algorithm to license plateSlant correction is carried out, while being eliminated during detecting license plate because of VLP correction bring noise background and license plate sideFrame realizes the accurate positionin to license board information.It is still direct using SSD object detection method in the identification to license board informationMonolith license plate is identified, this system combines conventional method to identify license plate on this basis, is applicable to different inspectionsSurvey environment.

Vehicle location function, vehicle cab recognition function and Car license recognition function are combined together to form a vehicle by the present inventionComprehensive information detection system.The present invention can be incorporated into existing traffic system, can effectively be reduced because human factor is broughtError message identification, while improving the vehicle detection efficiency of entire traffic system.

Detailed description of the invention

Network structure Fig. 3 vehicle detection schematic diagram figure of Fig. 1 system image process flow diagram 2SSD_VGG4Inception-AB-Net deep learning illustraton of model 5Inception-A structure chart Fig. 6 Inception-B structure chart Fig. 7 license plateIdentification process figure Fig. 8 car plate detection network structure Fig. 9 removes frame and background effect Figure 10 Car license recognition network structure figureLine chart Figure 13 profile is examined before and after the 11 12 vertical projection convolution of algorithm of locating license plate of vehicle flow chart based on morphology and color characteristicMeasure 14 License Plate Character Segmentation effect picture 15Inception-Small-Net unit structure chart Figure 16 vehicle cab recognition system of bitmapSystem software interface

Specific embodiment

To keep the object of the invention, technical solution and advantage more clear, below by Detailed description of the invention, respectively with vehicleThree positioning, vehicle cab recognition, Car license recognition modules are realized to describe various functions of the invention.System main-process stream as shown in Figure 1,It is described as follows:

1 vehicle location

As shown in Fig. 2, the present invention constructs SSD target detection network based on convolutional neural networks VGG-16 structure,It is denoted as SSD_VGG.Fig. 3 is the process that SSD_VGG network carries out identification positioning to vehicle, and when detecting vehicle, network is to photoFeature extraction is carried out, judges from the image with complex background and orients vehicle location.

2 vehicle cab recognitions

In vehicle cab recognition, the present invention, which is used, to be constructed as core using convolutional neural networks Inception-v2 structureInception-AB-Net deep learning model, Fig. 4 are its network structure.Inception-AB-Net is mainly by Inception-A network and Inception-B network are constituted.Convolutional layer part is made of 13 mixed layers, and wherein Inception-A structure is such asShown in Fig. 5, the step-length of all convolution kernels is 1 in Inception-A.Inception-B structure is as shown in fig. 6, its all volumeThe step-length of product core is 2.The design parameter configuration of Inception-AB-Net is as shown in table 1, the first stage of the network modelThe convolution kernel of first convolutional layer is 7 × 7, and step-length 2, its role is to remove the edge background of vehicle image, remaining volumeThe convolution kernel of product mixed layer and pond layer is 3 × 3 or 1 × 1.The convolution kernel size of global pool layer is 7 × 7, step-length 1,One-dimensional feature vector is provided the purpose is to the classifier for next stage.Features above extracting method is able to achieve to automobile imageThe abundant extraction of feature, to realize the Accurate classification to various subclass.

The present invention still uses Inception-A and Inception- on the basis of Inception-AB-Net networkThe network structure of B increases the number of Inception-A and Inception-B convolution kernel in the case where not changing overall structureAmount excavates the potentiality of the structure further to obtain Inception-AB-Full model, and design parameter is configured such as 2 institute of tableShow.Inception-AB-Full model is faster more steady compared to Inception-AB-Net model convergence rate, while possessing moreHigh accuracy rate.

3 Car license recognitions

Fig. 7 is car plate detection process, and the present invention is tied simultaneously using Car license recognition is realized based on SSD object detection methodClose traditional licence plate recognition method.The present invention is made that corresponding Optimal improvements on the basis of conventional method.

3.1 are based on SSD object detection method

3.1.1 License Plate

The present invention equally carries out the building of car plate detection network using SSD_VGG network.Since image to be detected isBy vehicle detection positioning, so generally there was only a license plate or not having license plate.Simultaneously by image to be detected size normalizationIt is 300 × 300, the size of license plate is between 48 × 25-79 × 36 at this time, therefore according to this size range to the SSD of scriptNetwork structure is modified optimization.Fig. 8 be adjusting and optimizing after network structure, be mainly adjusted to former SSD network structure upThe convolutional layers such as Conv9, Conv10 and Conv11 are fallen, because of the default box predicted from the characteristic pattern of these layersIt is general all bigger, it is suitable for detecting larger target, is not suitable for car plate detection task.

3.1.2 VLP correction

VLP correction is carried out using Radon converter technique, this method can not only obtain the horizontal and vertical direction of license plate simultaneouslyTilt angle, and calculating speed is fast.

It is exactly the function of script to be carried out to primary space transformation, for example incite somebody to action for Radon transformation under normal circumstancesOriginally the point in M-N plane was mapped in C-D plane, then in M-N plane and belonged to all the points of same straight line originallyThe same point is both corresponded in C-D plane.So only need to count the levels of accumulation of each point in C-D plane, so that it mayTo judge it in M-N plane with the presence or absence of straight line by this information.For image be exactly in a plane alongDifferent straight lines, which carries out line integral, can be obtained Radon transformation.For its Radon variation such as formula of two dimensional image f (x, y)(1) shown in:

Wherein

Radon (f (x-x', y-y'))=R_f((λ-x'cosθ-y'sinθ),θ) (2)

Formula (2) is the translation of Radon, wherein the transformation for mula of coordinate system x'-y' and coordinate system x-y are as follows:

3.1.3 license plate background and frame are removed

(1) horizontal frame is removed first.For the bianry image of license plate, if the value of a certain pixel of certain a line is 0(or 255), and the value of its next pixel is 255 (or 0), then such case is referred to as primary jump, and due to general completeLicense plate have 7 characters, therefore for theory every a line in characters on license plate region at least exist 14 times jump, and characters on license plate withThe transition times of outer part are less.It is highly height if license plate width is width, then using this information is jumped, from vehicleThe height/2 eminence of board bianry image is scanned upwards, if the transition times of continuous three row are respectively less than the threshold value setThreshold, it is determined that first trip transition times less than threshold row corresponding to picture altitude be coboundary, similarly fromThe lower boundary that license plate image can be obtained is scanned at its height/2 of license plate bianry image downwards.

(2) left and right sides background frame is then removed.Since the object of research is the license plate of blue bottom wrongly written or mispronounced character, it is utilizedColor characteristic carries out the removal of its two sides background.The license plate RGB image for having been removed horizontal frame is converted into HSV image, is connectExtract HSV image in blue component, obtain the binary map of license plate blue bottom plate, then carry out morphological operation form it intoConnected domain then carries out contour detecting, then utilizes license plate area accounting, the rectangular profile of license plate, length-width ratio and license plate wordSymbol necessarily filters out license plate area containing the priori knowledges such as white color component in region, finally to the region filtered out from left to right byColumn are scanned, and count the pixel number that each column pixel value is 255 (whites), if wherein continuous three column are less than the threshold of settingValue threshold_2, it is determined that first white pixel point number is classified as license plate left margin less than threshold_2, similarly fromThe right side is turned left the right margin for being scanned and can obtaining license plate.After removing horizontal frame simultaneously, the height of image is essentially characterHighly, then the image for having completed left and right boundary can be verified using the ratio of standard license plate width and character, it is ensured thatLicense plate area after removing left and right background is complete.

License plate removes frame and the effect of background is as shown in Figure 9.

3.1.4 characters on license plate detects

The licence plate recognition method established based on SSD target detection can directly identify that Figure 10 is it to whole license plateNetwork structure, detection main process is similar with vehicle detection, car plate detection before, is no longer only the inspection of two class objects at this timeIt surveys, method therefor is almost the same when the mark and car plate detection of training data, need to only mark out in every license plate image allSimultaneously fill out respective classes in the position of characters on license plate.

3.2 conventional method

3.2.1 License Plate

The present invention proposes a kind of License Plate based on morphology Yu hsv color feature on the basis of combining conventional methodMethod, Figure 11 are the flow chart of the algorithm.Pretreatment is carried out to image first and is converted grayscale image, the purpose is to reduce dataAmount, while being convenient for subsequent carry out edge detection；Then, the edge detection of vertical direction is carried out using sobel operator, is then incited somebody to actionTo vertical edge image be filtered, binary conversion treatment；Then the morphology closed operation carried out twice in succession to image makesAs soon as license plate area forms a connected domain, subsequent in this way to be known according to priori such as the color of contour detecting and license plate, length-width ratiosKnow and carries out careful screening.

The present invention is using the contour detecting function in the Opencv of computer vision library in the connected domain of binary imageEach profile is detected, while can be calculated the area of each profile and be wrapped up the minimum rectangle frame of the profile.Therefore benefitThe profile detected is screened with prior informations such as the length-width ratios of license plate to obtain the candidate regions of license plate.Candidate regions generally canBetween 1~5, next compared one by one really on original color image by hsv color feature using the coordinate of candidate regionRecognize, finally determines the position of license plate.

3.2.2 VLP correction and removal license plate background and frame

VLP correction and removal license plate background are same as above with frame method.

3.2.3 Character segmentation

Combine sciagraphy and template matching method to license plate word on the basis of being based on connected domain registration number character dividing methodSymbol is split, the specific steps are as follows:

(1) cut-point " " position is searched.As long as find in license plate second and third character cut-point toThe right side is partitioned into 5 characters, is partitioned into 2 characters to the left, to reduce segmentation difficulty.Vertical direction is carried out to license plate image firstProjection, convolutional calculation then is carried out to the projection result, the numerical value in convolution kernel is all 1, the license plate for being Width for widthFor image, the size of convolution kernel is 1*N, and N=Width* (34/440), and wherein parameter 34/440 is according in standard license plateThe interval of second character and third character is come what is determined, and as shown in figure 11, a is original two-value license plate image, and b is vertical projection knotFruit, c are the result after convolution.Take the corresponding abscissa X-left of the minimum value of license plate left side half in figure c be the second character withThe left margin of third character, the then coordinate of decollator " " are as follows: X=X-left+N/2.

(2) after determining cut-point, first two characters on the left of it are split.According to cut-point to the two-value of former license plateFigure is cut, and the bianry image of the first two character is obtained, and morphology is then carried out to it opens operation to avoid two Characters StucksTogether, while second English character being allowed to form connected domain, then utilizes contour detecting, and the minimum square comprising the profileShape frame determines the left margin of second English character.If contour detecting fails, turned left from the right side by column scan using sciagraphy,If the white pixel point number of a certain column is greater than the threshold value threshold_3 of setting for the first time, it is denoted as the right margin of the second character,If the white pixel point for occurring a certain column later is less than the threshold value threshold_3 of setting, it is denoted as the left margin of the second character.Its effect is as shown in figure 13.

(3) segmentation of 5 characters in right side is carried out.Remaining 5 character zones are equally made after carrying out morphology and opening operationWith contour detecting, if there is satisfactory profile, record its right boundary convenient for the sequence of subsequent character, then to image intoRow is cut, and judges whether also to need to be split again according to the width of remaining image, and the process before repeating is to the last totalUntil having 5 character blocks.If there is no satisfactory profile in contour detecting, according to the big of 5 characters of standard vehicle bridge queenSmall and interval, carries out pre-segmentation in proportion, then using sciagraphy according to the company in pre-segmentation line neighborhood (- ω, ω) rangeContinuous zero or minimum are split the correction of line.

Figure 14 is Character segmentation effect.

3.2.4 Recognition of License Plate Characters

The work of character recognition is exactly that the single character picture split is handled and analyzed, and is recognized thereinLicense plate number.Ordinary circumstance, the characters on license plate obtained after segmentation is smaller, therefore the catenet structure used middle before is uncomfortableWith.The present invention is based on the structures of Inception-AB-Net to design Inception-Small-Net network, and table 3 isThe parameter configuration of Inception-Small-Net, the received input data size of the network are 28 × 28, make it easy to handle vehicleBoard character.

The structure of Simple A, Simple B and its base volume product unit is as shown in figure 15 in table 3.Wherein basic convolutionUnit Conv is made of convolutional layer, Norm layers of Batch and Relu active coating；Simple A is mainly used for extracting feature, byThe convolution kernel of one 1 × 1 size and the convolution kernel of 3 × 3 sizes are formed in parallel；Simple B can be described as down-sampling layer,It is composed in parallel by the convolution kernel of 3 × 3 sizes and the Pooling of 3 × 3 sizes.

1 Inception-AB-Net prototype network parameter of table

2 Inception-AB-Full prototype network parameter of table

3 Inception-Small-Net parameter configuration of table

Claims (8)

1. a kind of vehicle comprehensive information detection system possesses three vehicle location, vehicle classification, Car license recognition functions, can rootDetection identification is carried out to the vehicle model and license board information of vehicle according to image or video information, it can be with existing traffic systemIn conjunction with efficiently being managed traffic, it is primarily characterized in that:

A positions the position of vehicle in picture using the vehicle detecting algorithm based on deep learning；

B carries out identification classification to vehicle classification using convolutional neural networks structure I nception-AB-Full；

C positions license plate using the detection method of license plate based on deep learning；

D identifies license plate using the licence plate recognition method based on deep learning；

E proposes a kind of license plate locating method based on morphology Yu hsv color feature；

F is on the basis of being based on connected domain registration number character dividing method, in conjunction with sciagraphy and template matching method to characters on license plateIt is split；

G utilizes the characters on license plate after the segmentation of Inception-Small-Net Network Recognition in traditional detection.

2. according to claim 1 use the vehicle detecting algorithm based on deep learning to position the position of vehicle in pictureIt sets, it is characterised in that:

SSD target detection network is constructed based on convolutional neural networks VGG-16 structure, and feature is carried out to photo with it and is mentionedIt takes, can judge and orient the position of vehicle from the image with complex background.

3. according to claim 1 carry out vehicle classification using convolutional neural networks structure I nception-AB-FullIdentification classification, it is characterised in that:

Inception-AB-Full network is constructed on the basis of convolutional neural networks Inception-AB-Net network,Inception-AB-Full network can more rapidly, the feature of more accurate autonomous learning to a variety of models, so utilizingThe model that Inception-AB-Full network training goes out can efficiently, correctly realize the classification of vehicle model.

4. according to claim 1 position license plate using the detection method of license plate based on deep learning, featureIt is:

Car plate detection network is constructed based on SSD target detection network in claim 2, car plate detection network is in originalThe convolutional layers such as Conv9, Conv10 and Conv11 are eliminated in SSD target detection network structure, because from the feature of these layersThe default box predicted on figure is generally bigger, is suitable for detecting larger target, is not suitable for car plate detection task.

5. according to claim 1 identify license plate using the licence plate recognition method based on deep learning, featureIt is:

Car license recognition is more class object detections, only need to mark out the position of all characters on license plate in every license plate image and fill outUpper respective classes can carry out model training.

6. according to claim 1 propose a kind of license plate locating method based on morphology Yu hsv color feature, featureIt is:

Include the following steps:

Step 1: pre-processing image, is converted into grayscale image；

Step 2: the edge detection of vertical direction is carried out using sobel operator；

Step 3: obtained vertical edge image is filtered, binary conversion treatment；

Step 4: morphology closed operation twice in succession is carried out to image, license plate area is made to form a connected domain；

Step 5: using the contour detecting function in the Opencv of computer vision library to each in the connected domain of binary imageProfile is detected, and can be screened to the profile detected using prior informations such as the length-width ratios of license plate to obtain the time of license plateConstituency；

Step 6: using the coordinate of candidate regions, comparing confirmation by hsv color feature on original color image one by one, last trueDetermine the position of license plate.

7. according to claim 1 on the basis of being based on connected domain registration number character dividing method, in conjunction with sciagraphy andTemplate matching method is split characters on license plate, it is characterised in that:

It comprises the following steps:

Step 1: carrying out the projection of vertical direction to license plate image, then carries out convolutional calculation to the projection result, utilizes calculatingAs a result cut-point " " position is found out；

Step 2: cutting according to binary map of the cut-point to former license plate, obtains the bianry image of the first two character, then rightIts progress morphology opens operation and avoids two Characters Stucks together, while second English character being allowed to form connected domain, thenMinimum rectangle frame using contour detecting, and comprising the profile determines the left margin of second English character, if profile is examinedDendrometry effect, then turned left by column scan using sciagraphy from the right side, if the white pixel point number of a certain column is greater than the threshold of setting for the first timeValue, then be denoted as the right margin of the second character, if the white pixel point for occurring a certain column later is less than the threshold value of setting, is denoted as theThe left margin of two characters；

Step 3: carrying out the segmentation of 5 characters in right side, equally makes after carrying out morphology and opening operation to remaining 5 character zonesWith contour detecting, if there is satisfactory profile, record its right boundary convenient for the sequence of subsequent character, then to image intoRow is cut, and judges whether also to need to be split again according to the width of remaining image, and the process before repeating is to the last totalUntil having 5 character blocks, if there is no satisfactory profile in contour detecting, according to the big of 5 characters of standard vehicle bridge queenSmall and interval, carries out pre-segmentation in proportion, then using sciagraphy according to the company in pre-segmentation line neighborhood (- ω, ω) rangeContinuous zero or minimum are split the correction of line.

8. it is according to claim 1 in traditional detection using Inception-Small-Net Network Recognition segmentation afterCharacters on license plate, it is characterised in that:

Inception-Small-Net network is designed based on convolutional neural networks Inception-AB-Net structure to handle pointCharacters on license plate after cutting.