CN111091131A - Self-adaptive license plate character recognition system and recognition method based on multi-task learning - Google Patents
Self-adaptive license plate character recognition system and recognition method based on multi-task learningDownload PDFInfo
- Publication number
- CN111091131A CN111091131ACN201911305677.5ACN201911305677ACN111091131ACN 111091131 ACN111091131 ACN 111091131ACN 201911305677 ACN201911305677 ACN 201911305677ACN 111091131 ACN111091131 ACN 111091131A
- Authority
- CN
- China
- Prior art keywords
- license plate
- character
- character recognition
- model
- plate character
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
- G06V30/14—Image acquisition
- G06V30/148—Segmentation of character regions
- G06V30/153—Segmentation of character regions using recognition of characters or words
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
- Y02T10/00—Road transport of goods or passengers
- Y02T10/10—Internal combustion engine [ICE] based vehicles
- Y02T10/40—Engine management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Molecular Biology (AREA)
- Computational Linguistics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Biophysics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Evolutionary Biology (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
- Character Discrimination (AREA)
Abstract
Description
Technical Field
The invention relates to the technical field of license plate recognition, in particular to a self-adaptive license plate character recognition system and a recognition method based on multitask learning.
Background
At present, the intelligent license plate recognition technology is widely applied to parking lots, urban roads, expressways and other regional occasions. The intelligent license plate recognition can automatically capture and recognize the license plate number of the vehicle. The intelligent license plate recognition technology is based on computer vision processing, digital image processing and other technologies, processes and analyzes the acquired license plate image to obtain the license plate number of each vehicle, and completes the recognition process.
The existing license plate character recognition method generally comprises the processes of character segmentation, character recognition result processing and the like on the positioning detection result of the vehicle license plate. Wherein, the character segmentation is to cut each character in a plurality of columns or a plurality of rows of character images from the whole image into a single character image; however, the character segmentation process has certain requirements on the definition and the shooting angle of the originally input license plate image, for example, when the license plate is inclined or the characters are stained and adhered, the accuracy of character segmentation is greatly reduced, and the accuracy of license plate character recognition is reduced.
In addition, even though the existing license plate recognition technology can ensure the accuracy of character segmentation, the existing license plate recognition technology cannot ensure the accuracy of recognition of license plates with different colors or different numbering forms due to different license plate colors or different license plate numbering forms used by regions, government departments and organizations of various countries.
Disclosure of Invention
The invention aims to provide a self-adaptive license plate character recognition system and a recognition method based on multitask learning so as to solve the technical problem.
In order to achieve the purpose, the invention adopts the following technical scheme:
the utility model provides a self-adaptation license plate character recognition system based on multitask learning for carry out the automatic identification detection of multitask to the vehicle license plate, this license plate character recognition system includes:
the model training module is used for training and forming a license plate positioning and calibrating model and a license plate character recognition model by taking the license plate sample data set as a training sample;
the vehicle image acquisition module is used for acquiring a vehicle image to be detected;
the license plate detection module is connected with the vehicle image acquisition module and used for taking the vehicle image as the input of the license plate positioning and calibrating model and identifying and outputting a license plate region image in the vehicle image;
the license plate character recognition module is connected with the license plate detection module and used for recognizing license plate characters in the license plate region image based on the license plate character recognition model and finally outputting a license plate character recognition result;
the license plate character recognition module specifically comprises:
the license plate character feature extraction unit is internally provided with a license plate character feature recognition model in advance, and is used for outputting character features corresponding to license plate characters in the license plate region image by taking the license plate region image as the input of the license plate character feature recognition model, and regressing self-adaptive parameters depended by the license plate character feature recognition model when the license plate character features are extracted;
the character sequence classification unit is connected with the license plate character feature extraction unit, a character sequence classification model is preset in the character sequence classification unit and used for taking the output of the license plate character feature extraction unit as the input of the character sequence classification model, and the license plate characters are subjected to character sequence classification in a multitask character sequence classification mode by analyzing the self-adaptive parameters to obtain at least one character sequence classification result;
and the filtering and matching unit is respectively connected with the license plate character feature extraction unit and the character sequence classification unit, and a character filtering and matching model is preset in the filtering and matching unit and is used for taking the character sequence classification result and the self-adaptive parameters as the input of the character filtering and matching model and finally outputting the license plate character recognition result of the vehicle image.
As a preferred scheme of the present invention, the license plate character recognition model at least includes the license plate character feature recognition model, the character sequence classification model, and the character filtering and matching model.
As a preferred scheme of the present invention, the license plate detection module specifically includes:
the vehicle detection unit is used for detecting a vehicle position area image in the vehicle image based on a vehicle detection model trained in advance;
and the license plate positioning and calibrating unit is connected with the vehicle detecting unit and used for taking the vehicle position area image as the input of the license plate positioning and calibrating model and identifying and outputting the license plate area image in the vehicle position area image.
As a preferred scheme of the present invention, the license plate character feature extraction unit in the license plate character recognition module includes:
the license plate character feature extraction subunit is used for taking the license plate region image as the input of the license plate character feature recognition model and outputting the character features corresponding to the license plate characters in the license plate region image;
and the self-adaptive parameter regression subunit is used for calculating the self-adaptive parameters depended by the license plate character feature recognition model when the license plate character features are extracted.
As a preferred scheme of the present invention, the filtering and matching unit in the license plate character recognition module specifically includes:
the license plate type judging subunit is used for judging the license plate type corresponding to the license plate according to the character sequence classification result and based on a pre-trained license plate type judging model;
the character sequence classification result filtering subunit is connected with the license plate type judging subunit and is used for filtering the character sequence classification result which does not conform to the license plate number form corresponding to the license plate type according to the judged license plate type;
and the license plate character matching subunit is connected with the character sequence classification result filtering subunit and is used for performing character feature matching on the license plate characters one by one in the character sequence classification result obtained through filtering and finally outputting the license plate character recognition result of the vehicle image.
The invention also provides a self-adaptive license plate character recognition method based on multitask learning, which is realized by applying the license plate character recognition system and comprises the following steps:
step S1, the license plate character recognition system takes a license plate sample data set as a training sample, and trains to form a license plate positioning and calibration model and a license plate character recognition model;
step S2, the license plate character recognition system collects the vehicle image to be detected;
step S3, the license plate character recognition system takes the vehicle image as the input of the license plate positioning and calibration model, and recognizes and outputs the license plate region image in the vehicle image;
step S4, the license plate character recognition system recognizes the license plate characters in the license plate region image based on the license plate character recognition model, and finally outputs the license plate character recognition result.
As a preferable scheme of the present invention, in step S1, the network architecture of the deep neural network used for training the license plate location and calibration model is any one of FCN, mask-RCNN or WPOD-NET.
As a preferred embodiment of the present invention, in step S1, the network architecture of the deep neural network adopted in the training of the license plate character recognition model is a CTC network.
As a preferable scheme of the present invention, in step S3, the process of recognizing and outputting the license plate region image in the vehicle image by the license plate character recognition system specifically includes the following steps:
step S31, the license plate character recognition system detects a vehicle position area image in the vehicle image based on a pre-trained vehicle detection model;
and step S32, the license plate character recognition system takes the vehicle position area image as the input of the license plate positioning and calibration model, and recognizes and outputs the license plate area image in the vehicle position area image.
As a preferred scheme of the present invention, in step S4, the process of recognizing and outputting the license plate character recognition result by the license plate character recognition system specifically includes the following steps:
step S41, the license plate character recognition system recognizes and outputs the character features corresponding to the license plate characters in the license plate region image based on the license plate character feature recognition model to obtain a sequentially arranged license plate character feature recognition result, and regresses the self-adaptive parameters depended on when recognizing the character features;
step S42, the license plate character recognition system analyzes the self-adaptive parameters obtained in the step S41 and carries out multi-task character sequence recognition and classification on the sequentially arranged license plate character feature recognition results based on the character sequence classification model to obtain at least one character sequence classification result;
and step S43, the license plate character recognition system takes the character sequence classification result and the adaptive parameters as the input of the character filtering and matching model, and finally outputs the license plate character recognition result of the vehicle image.
As a preferred embodiment of the present invention, in step S43, the specific process of the license plate character recognition system performing character filtering and matching on the license plate characters includes the following steps:
step S431, the license plate character recognition system judges the license plate type corresponding to the license plate according to the character sequence classification result and on the basis of a pre-trained license plate type judgment model;
step S432, the license plate character recognition system filters the character sequence classification result which does not conform to the license plate number form of the license plate type according to the license plate type judged in the step S431;
step S433, the license plate character recognition system performs character feature matching on each license plate character in the character sequence classification result retained after filtering in step S432 one by one, and finally outputs the license plate character recognition result for the vehicle image.
The invention has the beneficial effects that:
1. the invention uses the lightweight network model, can simultaneously carry out end-to-end training and recognition on license plate characters of different types, colors and sizes, and has the advantages of high speed, high stability and strong practicability.
2. The invention simultaneously uses the deep convolutional network and the multi-task learning method to automatically learn the image characteristics and the space distribution characteristics of the license plate characters, does not perform manual characteristic selection and design, and improves the accuracy of license plate character recognition.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly described below. It is obvious that the drawings described below are only some embodiments of the invention, and that for a person skilled in the art, other drawings can be derived from them without inventive effort.
Fig. 1 is a schematic structural diagram of an adaptive license plate character recognition system based on multitask learning according to an embodiment of the present invention;
fig. 2 is a schematic diagram of an internal structure of a license plate detection module in the license plate character recognition system according to an embodiment of the present invention;
fig. 3 is a schematic diagram of an internal structure of a license plate character recognition module in the license plate character recognition system according to an embodiment of the present invention;
fig. 4 is a schematic diagram of an internal structure of a license plate character feature extraction unit in the vehicle character recognition module according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of the internal structure of the filtering and matching unit in the vehicle character recognition module according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of an internal structure of a model training module in the license plate character recognition system according to an embodiment of the present invention;
FIG. 7 is a flowchart illustrating the steps of a method for adaptive license plate character recognition based on multitask learning according to an embodiment of the present invention;
FIG. 8 is a flowchart illustrating a method for recognizing the license plate region image in the vehicle image by the license plate character recognition system according to an embodiment of the present invention;
fig. 9 is a diagram illustrating specific steps of a method for recognizing and outputting a recognition result of license plate characters by the license plate character recognition system according to an embodiment of the present invention;
fig. 10 is a diagram illustrating specific steps of a method for filtering and matching license plate characters by a license plate character recognition system according to an embodiment of the present invention:
fig. 11 is a flowchart of a license plate character recognition system training a license plate character recognition model according to an embodiment of the present invention.
Detailed Description
The technical scheme of the invention is further explained by the specific implementation mode in combination with the attached drawings.
Wherein the showings are for the purpose of illustration only and are shown by way of illustration only and not in actual form, and are not to be construed as limiting the present patent; to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of an actual product; it will be understood by those skilled in the art that certain well-known structures in the drawings and descriptions thereof may be omitted.
The same or similar reference numerals in the drawings of the embodiments of the present invention correspond to the same or similar components; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "inner", "outer", etc. are used for indicating the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, it is only for convenience of description and simplification of description, but it is not indicated or implied that the referred device or element must have a specific orientation, be constructed in a specific orientation and be operated, and therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and are not to be construed as limitations of the present patent, and the specific meanings of the terms may be understood by those skilled in the art according to specific situations.
In the description of the present invention, unless otherwise explicitly specified or limited, the term "connected" or the like, if appearing to indicate a connection relationship between the components, is to be understood broadly, for example, as being fixed or detachable or integral; can be mechanically or electrically connected; they may be directly connected or indirectly connected through intervening media, or may be connected through one or more other components or may be in an interactive relationship with one another. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art.
The embodiment of the present invention provides a multitask learning-based adaptive license plate character recognition system, which is used for multitask automatic recognition and detection of a vehicle license plate, and please refer to fig. 1, the license plate character recognition system includes:
themodel training module 1 is used for training and forming a license plate positioning and calibrating model and a license plate character recognition model by taking a license plate sample data set as a training sample;
the vehicleimage acquisition module 2 is used for acquiring a vehicle image to be detected;
the licenseplate detection module 3 is connected with the vehicleimage acquisition module 2 and used for taking the vehicle image as the input of a license plate positioning and calibrating model and identifying and outputting a license plate region image in the vehicle image;
the license platecharacter recognition module 4 is connected with the licenseplate detection module 3 and is used for recognizing license plate characters in the license plate region image based on the license plate character recognition model and finally outputting a license plate character recognition result;
referring to fig. 3, the license platecharacter recognition module 4 specifically includes:
a license plate characterfeature extraction unit 41, wherein a license plate character feature recognition model is preset in the license plate characterfeature extraction unit 41, and is used for inputting a license plate region image as a license plate character feature recognition model, outputting character features corresponding to license plate characters in the license plate region image, and returning self-adaptive parameters depended on when the license plate character features are extracted by the license plate character feature recognition model;
the charactersequence classification unit 42 is connected with the license plate characterfeature extraction unit 41, a character sequence classification model is preset in the charactersequence classification unit 42 and used for taking the output of the license plate character feature extraction unit as the input of the character sequence classification model, and the license plate characters are subjected to character sequence classification in a multitask character sequence classification mode through analyzing self-adaptive parameters to obtain at least one character sequence classification result;
and the filtering and matchingunit 43 is respectively connected with the license plate characterfeature extraction unit 41 and the charactersequence classification unit 42, and a character filtering and matching model is preset in the filtering and matchingunit 43 and is used for inputting the character sequence classification result and the self-adaptive parameter as the character filtering and matching model and finally outputting the license plate character recognition result of the vehicle image.
In the technical scheme, the license plate character recognition model at least comprises a license plate character feature recognition model, a character sequence classification model and a character filtering and matching model. The network architecture of the deep neural network used for training the license plate character recognition model preferably uses a CTC network, which is an existing neural network architecture, and since the process of the method for training the license plate character recognition model is not within the scope of the claims of the present invention, the specific training process thereof will not be described in detail herein. In addition, the network architecture of the deep neural network adopted for training the license plate positioning and calibration model is preferably any one of FCN, mask-RCNN or WPOD-NET, the FCN, mask-RCNN and WPOD-NET are all the existing network architectures, and similarly, the training process is not elaborated herein since the specific method for training the license plate positioning and calibration model is not the scope of the claims of the present invention.
As the image of the vehicle may include some non-recognition object images such as background images in addition to the image of the area where the vehicle is located, referring to fig. 2, the licenseplate detection module 3 preferably includes:
avehicle detection unit 31 for detecting a vehicle position area image in the vehicle image based on a pre-selected trained vehicle detection model;
and the license plate positioning and calibratingunit 32 is connected with thevehicle detecting unit 31 and is used for taking the vehicle position area image as the input of the license plate positioning and calibrating model and identifying and outputting the license plate area image in the vehicle position area image.
In the above technical solution, the neural network structure adopted for training the vehicle detection model is preferably any one of YOLO, SSD, fast-RCNN in the prior art, and the specific training method of the vehicle detection model is not within the scope of the claimed invention, so the specific training process is not described herein.
Referring to fig. 4, the license plate characterfeature extraction unit 41 in the license platecharacter recognition module 4 specifically includes:
the license plate characterfeature extraction subunit 411 is configured to take the license plate region image as an input of a license plate character feature recognition model, and output character features corresponding to license plate characters in the license plate region image;
and the adaptiveparameter regression subunit 412 is configured to calculate adaptive parameters that are relied on by the license plate character feature recognition model when license plate character features are extracted.
Referring to fig. 5, the filtering and matchingunit 43 of the license platecharacter recognition module 4 specifically includes:
the license platetype judging subunit 431 is used for judging to obtain the license plate type corresponding to the license plate according to the character sequence classification result and based on a pre-trained license plate type judging model;
the character sequence classificationresult filtering subunit 432 is connected to the license platetype judging subunit 431, and is configured to filter, according to the judged license plate type, a character sequence classification result that does not conform to the license plate number form corresponding to the license plate type;
and the license platecharacter matching subunit 433 is connected with the character sequence classificationresult filtering subunit 432 and is used for performing character feature matching on the license plate characters in the filtered character sequence classification result one by one and finally outputting a license plate character recognition result of the vehicle image.
In the above technical solution, the detailed process of the license plate character recognition system for recognizing and outputting the license plate character recognition result is as follows:
the license plate character recognition system recognizes and outputs character features corresponding to license plate characters in a license plate region image based on a license plate character feature recognition model to obtain a sequentially arranged license plate character feature recognition result, and regresses self-adaptive parameters depended when the character features are recognized. The license plate characters include various characters used for coding license plates in various regions, government departments and organizations of various countries, including but not limited to Chinese and English letters, numbers and different license plate colors.
The license plate character recognition system carries out multi-task character sequence recognition and classification on the license plate character feature recognition results which are sequentially arranged by analyzing the self-adaptive parameters and based on the character sequence classification model to obtain at least one character sequence classification result. Because the number forms of the license plates in various regions of various countries are not consistent, for example, most countries adopt a one-line license plate number form, and some countries adopt a multi-line number form, the license plate character recognition system firstly carries out character sequence recognition classification on the extracted sequentially arranged license plate character feature recognition results, preliminarily classifies the number form of the license plate, for example, whether the license plate is a one-line license plate or a multi-line license plate, and finally recognizes and classifies to obtain at least one character sequence classification result (for example, the license plate characters of the license plate are in one line). In addition, in order to improve the speed of recognizing and classifying the license plate character sequences, the license plate character recognition system provided by the invention preferably adopts a multitask character sequence classification mode to perform classification recognition on the license plates, namely, a plurality of character sequence classification models are preset in the license plate character recognition system provided by the invention, and the license plates with different numbering forms can be subjected to multitask recognition and classification.
And finally, the license plate character recognition system takes the character sequence classification result and the self-adaptive parameters as the input of the character filtering and matching model, and finally outputs the license plate character recognition result of the vehicle image.
The specific process of the license plate character recognition system for carrying out character filtering and matching on the license plate characters is briefly described as follows:
the license plate character recognition system judges the license plate type corresponding to the license plate according to the character sequence classification result and on the basis of a pre-trained license plate type judgment model; the types of license plates used by different countries, regions or government departments and components are usually inconsistent, for example, the armed organizations of some countries use license plates with white background color, electric vehicles use license plates with green background color, and the like, and a license plate character recognition system firstly recognizes the types of the license plates corresponding to the license plates;
and then, the license plate character recognition system filters out character sequence classification results which do not accord with the license plate number form of the license plate type according to the judged license plate type. For example, the license plate number form of China is that the first character is a Chinese character, the back of the Chinese character is composed of 6 numbers or a combination of 6 English letters and numbers, if the license plate is judged to be a blue-bottom civil license plate, the license plate character recognition system automatically filters out the character sequence classification result which does not accord with the type of the Chinese civil license plate before;
and finally, the license plate character recognition system performs character feature matching on the license plate characters in the character sequence classification result retained after filtering one by one, and finally outputs a license plate character recognition result of the vehicle image.
In the above technical solution, please refer to fig. 6, themodel training module 1 includes an adaptiveparameter training unit 11 and a plurality of characterrecognition training units 12. The input of themodel training module 1 is a license plate sample data set, and the license plate sample data set comprises a vehicle image, a license plate region image and a license plate region image marked with a license plate character label. And the adaptiveparameter training unit 11 is used for training adaptive parameters depended by the license plate characters, and the parameters can be used for guiding the division of the character sequence classification unit in the license plate character recognition process and judging the type of the license plate. The characterrecognition training unit 12 is configured to train and form a license plate character recognition model in a multi-task training manner according to the license plate type specified by the character tag, for example, thecharacter training unit 12 is configured to train and form a license plate positioning and calibration model; thecharacter training unit 12 is used for training and forming a license plate character feature recognition model; acharacter training unit 12 for training and forming a character sequence classification model; acharacter training unit 12 for training and forming character filtering and matching models; acharacter training unit 12 is used to train the formation of a vehicle detection model … …
Referring to fig. 11, the steps of the method for training the license plate character recognition model by the license plate character recognition system provided in this embodiment are briefly described as follows:
and reading training data and a label, wherein the training data refers to a license plate area image, namely a calibrated vehicle image output by license plate detection, or a vehicle image generated in a simulation mode. The label refers to a license plate character sequence and license plate type information (row and column information of license plate characters) corresponding to the meaning of the training data;
then initializing the weight values of each layer of the training network. The initialization may be done using a random number as an initial value, or using pre-training weights on other sample data sets. The training network refers to a lightweight deep neural network for license plate character recognition, such as a neural network of a CNN architecture.
Referring to fig. 11, the training network includes two network branches, namely a first branch training network and a second branch training network. The first branch training network first calculates the license plate character features. And the first branch network calculates and generates a license plate character characteristic diagram in a convolution mode based on the current network weight. And then carrying out character sequence classification and recognition. And generating a multi-task parallel character sequence classification and identification task sub-branch according to the character label and the character characteristic diagram to obtain at least one character sequence classification result.
Then a selective single optimization operation is performed. And only one character sequence classification result in the character sequence classification results output by the character sequence classification and identification sub-branches is a real result, and the branch is selected to carry out single optimization operation. The single optimization operation is to update the model parameter weights through back propagation. Methods that can be used for weight updating include, but are not limited to, any one or combination of SGD, RMSProp, Adam, Nesterov accessed Gradient.
And storing the network weight after the training is updated after the single optimization operation reaches the termination condition. The termination condition here may be the total number of optimizations set, or a loss (loss) value smaller than a preset loss value.
Referring to fig. 11, the second branch network first calculates the adaptive parameters. The self-adaptive parameters are calculated by a regression mode based on the current network weight. The regression calculation of the adaptive parameters is a conventional calculation method, so the specific regression process of the adaptive parameters is not described here. Then, single optimization operation is carried out, and the model parameter weight is updated through back propagation. Methods that can be used for weight updating include, but are not limited to, any one or combination of SGD, RMSProp, Adam, Nesterov acceleratedgradition. And finally, after the single optimization operation reaches a termination condition, storing the network weight after training and updating. The termination condition here may also be a set total number of optimizations, or a loss (loss) value smaller than a preset loss value.
The invention also provides a multitask learning self-adaptive license plate character recognition method, which is realized by applying the license plate character recognition system and refers to fig. 7, and the method comprises the following steps:
step S1, the license plate character recognition system takes a license plate sample data set as a training sample, and trains to form a license plate positioning and calibration model and a license plate character recognition model;
step S2, the license plate character recognition system collects the vehicle image to be detected;
step S3, the license plate character recognition system takes the vehicle image as the input of the license plate positioning and calibration model, and recognizes and outputs the license plate area image in the vehicle image;
and step S4, the license plate character recognition system recognizes license plate characters in the license plate region image based on the license plate character recognition model, and finally outputs a license plate character recognition result.
In step S1, the network architecture of the deep neural network used for training the license plate location and calibration model is preferably any one of FCN, mask-RCNN, and WPOD-NET.
The network architecture of the deep neural network adopted for training the license plate character recognition model is preferably a CTC network.
Referring to fig. 8, in step S3, the process of recognizing the license plate region image in the output vehicle image by the license plate character recognition system specifically includes the following steps:
step S31, the license plate character recognition system detects a vehicle position area image in the vehicle image based on a pre-trained vehicle detection model;
and step S32, the license plate characters are input by the system by taking the vehicle position area image as a license plate positioning and calibration model, and the license plate area image in the vehicle position area image is identified and output.
In step S31, the network architecture of the deep neural network used by the license plate character recognition system to train the vehicle detection model is preferably any one of YOLO, SSD, or fast-RCNN.
Referring to fig. 9, in step S4, the process of recognizing and outputting the license plate character recognition result by the license plate character recognition system specifically includes the following steps:
step S41, the license plate character recognition system recognizes and outputs character features corresponding to license plate characters in the license plate region image based on the license plate character feature recognition model to obtain a sequentially arranged license plate character feature recognition result, and regresses self-adaptive parameters depended on when recognizing the character features;
step S42, the license plate character recognition system carries out multi-task character sequence recognition and classification on the sequentially arranged license plate character feature recognition results by analyzing the self-adaptive parameters obtained in the step S41 and based on a character sequence classification model to obtain at least one character sequence classification result;
and step S43, the license plate character recognition system takes the character sequence classification result and the self-adaptive parameters as the input of the character filtering and matching model, and finally outputs the license plate character recognition result of the vehicle image.
Referring to fig. 10, in step S43, the specific process of the license plate character recognition system performing character filtering and matching on the license plate characters includes the following steps:
step S431, judging the license plate type corresponding to the license plate by the license plate character recognition system according to the character sequence classification result and based on a preselected trained license plate type judgment model;
step S432, the license plate character recognition system filters out character sequence classification results of license plate number forms which do not accord with license plate types according to the license plate types judged in the step S431;
and step S433, the license plate character recognition system performs character feature matching on each license plate character in the character sequence classification result retained after filtering in the step S432 one by one, and finally outputs a license plate character recognition result of the vehicle image.
It should be understood that the above-described embodiments are merely preferred embodiments of the invention and the technical principles applied thereto. It will be understood by those skilled in the art that various modifications, equivalents, changes, and the like can be made to the present invention. However, such variations are within the scope of the invention as long as they do not depart from the spirit of the invention. In addition, certain terms used in the specification and claims of the present application are not limiting, but are used merely for convenience of description.
Claims (11)
1. A self-adaptive license plate character recognition system based on multitask learning is used for carrying out multitask automatic recognition detection on a vehicle license plate, and is characterized by comprising the following steps:
the model training module is used for training and forming a license plate positioning and calibrating model and a license plate character recognition model by taking the license plate sample data set as a training sample;
the vehicle image acquisition module is used for acquiring a vehicle image to be detected;
the license plate detection module is respectively connected with the vehicle image acquisition module and the model training module and is used for taking the vehicle image as the input of the license plate positioning and calibrating model and identifying and outputting a license plate area image in the vehicle image;
the license plate character recognition module is respectively connected with the license plate detection module and the model training module and is used for recognizing license plate characters in the license plate region image based on the license plate character recognition model and finally outputting a license plate character recognition result;
the license plate character recognition module specifically comprises:
the license plate character feature extraction unit is internally provided with a license plate character feature recognition model in advance, and is used for outputting character features corresponding to license plate characters in the license plate region image by taking the license plate region image as the input of the license plate character feature recognition model, and regressing self-adaptive parameters depended by the license plate character feature recognition model when the license plate character features are extracted;
the character sequence classification unit is connected with the license plate character feature extraction unit, a character sequence classification model is preset in the character sequence classification unit and used for taking the output of the license plate character feature extraction unit as the input of the character sequence classification model, and the license plate characters are subjected to character sequence classification in a multitask character sequence classification mode by analyzing the self-adaptive parameters to obtain at least one character sequence classification result;
and the filtering and matching unit is respectively connected with the license plate character feature extraction unit and the character sequence classification unit, and a character filtering and matching model is preset in the filtering and matching unit and is used for taking the character sequence classification result and the self-adaptive parameters as the input of the character filtering and matching model and finally outputting the license plate character recognition result of the vehicle image.
2. The license plate character recognition system of claim 1, wherein the license plate character recognition model comprises at least the license plate character feature recognition model, the character sequence classification model, and the character filtering and matching model.
3. The license plate character recognition system of claim 1, wherein the license plate detection module specifically comprises:
the vehicle detection unit is used for detecting a vehicle position area image in the vehicle image based on a vehicle detection model trained in advance;
and the license plate positioning and calibrating unit is connected with the vehicle detecting unit and used for taking the vehicle position area image as the input of the license plate positioning and calibrating model and identifying and outputting the license plate area image in the vehicle position area image.
4. The license plate character recognition system of claim 1, wherein the license plate character feature extraction unit in the license plate character recognition module comprises:
the license plate character feature extraction subunit is used for taking the license plate region image as the input of the license plate character feature recognition model and outputting the character features corresponding to the license plate characters in the license plate region image;
and the self-adaptive parameter regression subunit is used for calculating the self-adaptive parameters depended by the license plate character feature recognition model when the license plate character features are extracted.
5. The license plate character recognition system of claim 1, wherein the filtering and matching unit in the license plate character recognition module specifically comprises:
the license plate type judging subunit is used for judging the license plate type corresponding to the license plate according to the character sequence classification result and based on a pre-trained license plate type judging model;
the character sequence classification result filtering subunit is connected with the license plate type judging subunit and is used for filtering the character sequence classification result which does not conform to the license plate number form corresponding to the license plate type according to the judged license plate type;
and the license plate character matching subunit is connected with the character sequence classification result filtering subunit and is used for performing character feature matching on the license plate characters one by one in the character sequence classification result obtained through filtering and finally outputting the license plate character recognition result of the vehicle image.
6. A self-adaptive license plate character recognition method based on multitask learning is realized by applying the license plate character recognition system as the right 1-5, and is characterized by comprising the following steps of:
step S1, the license plate character recognition system takes a license plate sample data set as a training sample, and trains to form a license plate positioning and calibration model and a license plate character recognition model;
step S2, the license plate character recognition system collects the vehicle image to be detected;
step S3, the license plate character recognition system takes the vehicle image as the input of the license plate positioning and calibration model, and recognizes and outputs the license plate region image in the vehicle image;
step S4, the license plate character recognition system recognizes the license plate characters in the license plate region image based on the license plate character recognition model, and finally outputs the license plate character recognition result.
7. The license plate character recognition method of claim 6, wherein in step S1, a network architecture of a deep neural network adopted for training the license plate location and calibration model is any one of FCN, mask-RCNN or WPOD-NET.
8. The license plate character recognition method of claim 6, wherein in the step S1, a network architecture of a deep neural network adopted for training the license plate character recognition model is a CTC network.
9. The license plate character recognition method of claim 6, wherein in the step S3, the process of the license plate character recognition system recognizing and outputting the license plate region image in the vehicle image specifically includes the steps of:
step S31, the license plate character recognition system detects a vehicle position area image in the vehicle image based on a pre-trained vehicle detection model;
and step S32, the license plate character recognition system takes the vehicle position area image as the input of the license plate positioning and calibration model, and recognizes and outputs the license plate area image in the vehicle position area image.
10. The license plate character recognition method of claim 6, wherein in the step S4, the process of the license plate character recognition system recognizing and outputting the license plate character recognition result specifically includes the steps of:
step S41, the license plate character recognition system recognizes and outputs the character features corresponding to the license plate characters in the license plate region image based on the license plate character feature recognition model to obtain a sequentially arranged license plate character feature recognition result, and regresses the self-adaptive parameters depended on when recognizing the character features;
step S42, the license plate character recognition system analyzes the self-adaptive parameters obtained in the step S41 and carries out multi-task character sequence recognition and classification on the sequentially arranged license plate character feature recognition results based on the character sequence classification model to obtain at least one character sequence classification result;
and step S43, the license plate character recognition system takes the character sequence classification result and the adaptive parameters as the input of the character filtering and matching model, and finally outputs the license plate character recognition result of the vehicle image.
11. The license plate character recognition method of claim 10, wherein in step S43, the specific process of the license plate character recognition system for performing character filtering and matching on the license plate characters comprises the following steps:
step S431, the license plate character recognition system judges the license plate type corresponding to the license plate according to the character sequence classification result and on the basis of a pre-trained license plate type judgment model;
step S432, the license plate character recognition system filters the character sequence classification result which does not conform to the license plate number form of the license plate type according to the license plate type judged in the step S431;
step S433, the license plate character recognition system performs character feature matching on each license plate character in the character sequence classification result retained after filtering in step S432 one by one, and finally outputs the license plate character recognition result for the vehicle image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911305677.5ACN111091131B (en) | 2019-12-18 | 2019-12-18 | Self-adaptive license plate character recognition system and recognition method based on multitask learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911305677.5ACN111091131B (en) | 2019-12-18 | 2019-12-18 | Self-adaptive license plate character recognition system and recognition method based on multitask learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111091131Atrue CN111091131A (en) | 2020-05-01 |
| CN111091131B CN111091131B (en) | 2023-06-09 |
Family
ID=70396022
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911305677.5AActiveCN111091131B (en) | 2019-12-18 | 2019-12-18 | Self-adaptive license plate character recognition system and recognition method based on multitask learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111091131B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111651960A (en)* | 2020-06-01 | 2020-09-11 | 杭州尚尚签网络科技有限公司 | Optical character joint training and recognition method for moving from contract simplified form to traditional form |
| CN113705577A (en)* | 2021-04-23 | 2021-11-26 | 中山大学 | License plate recognition method based on deep learning |
| US11263485B2 (en)* | 2019-11-14 | 2022-03-01 | Institute For Information Industry | Transfer model training system and method |
| CN115861994A (en)* | 2022-12-29 | 2023-03-28 | 天翼交通科技有限公司 | A license plate detection and recognition method, device, computer equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6553131B1 (en)* | 1999-09-15 | 2003-04-22 | Siemens Corporate Research, Inc. | License plate recognition with an intelligent camera |
| CN108334881A (en)* | 2018-03-12 | 2018-07-27 | 南京云创大数据科技股份有限公司 | A kind of licence plate recognition method based on deep learning |
| CN108388896A (en)* | 2018-02-09 | 2018-08-10 | 杭州雄迈集成电路技术有限公司 | A kind of licence plate recognition method based on dynamic time sequence convolutional neural networks |
| WO2019141741A1 (en)* | 2018-01-19 | 2019-07-25 | Arcus Holding A/S | License plate reader using optical character recognition on plural detected regions |
| CN110097044A (en)* | 2019-05-13 | 2019-08-06 | 苏州大学 | Stage car plate detection recognition methods based on deep learning |
- 2019
- 2019-12-18CNCN201911305677.5Apatent/CN111091131B/enactiveActive
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6553131B1 (en)* | 1999-09-15 | 2003-04-22 | Siemens Corporate Research, Inc. | License plate recognition with an intelligent camera |
| WO2019141741A1 (en)* | 2018-01-19 | 2019-07-25 | Arcus Holding A/S | License plate reader using optical character recognition on plural detected regions |
| CN108388896A (en)* | 2018-02-09 | 2018-08-10 | 杭州雄迈集成电路技术有限公司 | A kind of licence plate recognition method based on dynamic time sequence convolutional neural networks |
| CN108334881A (en)* | 2018-03-12 | 2018-07-27 | 南京云创大数据科技股份有限公司 | A kind of licence plate recognition method based on deep learning |
| CN110097044A (en)* | 2019-05-13 | 2019-08-06 | 苏州大学 | Stage car plate detection recognition methods based on deep learning |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11263485B2 (en)* | 2019-11-14 | 2022-03-01 | Institute For Information Industry | Transfer model training system and method |
| CN111651960A (en)* | 2020-06-01 | 2020-09-11 | 杭州尚尚签网络科技有限公司 | Optical character joint training and recognition method for moving from contract simplified form to traditional form |
| CN111651960B (en)* | 2020-06-01 | 2023-05-30 | 杭州尚尚签网络科技有限公司 | Optical character joint training and recognition method for transferring contract simplified body to complex body |
| CN113705577A (en)* | 2021-04-23 | 2021-11-26 | 中山大学 | License plate recognition method based on deep learning |
| CN115861994A (en)* | 2022-12-29 | 2023-03-28 | 天翼交通科技有限公司 | A license plate detection and recognition method, device, computer equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111091131B (en) | 2023-06-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111091131A (en) | Self-adaptive license plate character recognition system and recognition method based on multi-task learning | |
| CN111062396B (en) | License plate number recognition method and device, electronic equipment and storage medium | |
| CN103473950B (en) | Parking lot parking space monitoring method | |
| CN107545239B (en) | Fake plate detection method based on license plate recognition and vehicle characteristic matching | |
| CN110163069B (en) | Lane line detection method for driving assistance | |
| CN106845487A (en) | A kind of licence plate recognition method end to end | |
| CN110728283A (en) | License plate type identification method and device | |
| Kulkarni et al. | Automatic number plate recognition (anpr) system for indian conditions | |
| CN106709450A (en) | Recognition method and system for fingerprint images | |
| CN106096602A (en) | Chinese license plate recognition method based on convolutional neural network | |
| CN111428550A (en) | Vehicle detection method based on improved YO L Ov3 | |
| CN110807491A (en) | License plate image definition model training method, definition detection method and device | |
| CN111274942A (en) | Traffic cone identification method and device based on cascade network | |
| CN103488974A (en) | Facial expression recognition method and system based on simulated biological vision neural network | |
| CN114782770A (en) | A method and system for license plate detection and license plate recognition based on deep learning | |
| CN110852358A (en) | Vehicle type distinguishing method based on deep learning | |
| CN111967401A (en) | Target detection method, device and storage medium | |
| CN111815609B (en) | Pathological image classification method and system based on context awareness and multi-model fusion | |
| CN112381840B (en) | Method and system for marking vehicle appearance parts in loss assessment video | |
| CN110516691A (en) | A kind of Vehicular exhaust detection method and device | |
| CN112052829B (en) | Pilot behavior monitoring method based on deep learning | |
| CN113627477A (en) | Vehicle multi-attribute identification method and system | |
| CN112949510A (en) | Human detection method based on fast R-CNN thermal infrared image | |
| CN119399615B (en) | Method and related equipment for identifying tunnel lining steel frame based on GAR-Net | |
| CN110956156A (en) | Deep learning-based red light running detection system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right | ||
| TR01 | Transfer of patent right | Effective date of registration:20240614 Address after:Room 611-64, R & D center building, China (Hefei) international intelligent voice Industrial Park, 3333 Xiyou Road, high tech Zone, Hefei, Anhui 230000 Patentee after:AINNOVATION (HEFEI) TECHNOLOGY Co.,Ltd. Country or region after:China Address before:210046 19th floor, building B, Xingzhi Science Park, 6 Xingzhi Road, Nanjing Economic and Technological Development Zone, Jiangsu Province Patentee before:AINNOVATION (NANJING) TECHNOLOGY Co.,Ltd. Country or region before:China |