Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples are not representative of all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as detailed in the accompanying claims.
It should be understood that the terms "comprises" and "comprising," and any variations thereof, in the embodiments of the disclosure are intended to cover a non-exclusive inclusion, such that a product or apparatus that comprises a list of elements is not necessarily limited to those elements expressly listed, but may include other elements not expressly listed or inherent to such product or apparatus.
The term "and/or" in the embodiments of the present disclosure describes an association relationship of association objects, which indicates that three relationships may exist, for example, a and/or B may indicate: a exists alone, A and B exist together, and B exists alone. The character "/" generally indicates that the context-dependent object is an "or" relationship.
The term "plurality" in the embodiments of the present disclosure means two or more, and other adjectives are similar thereto.
The terms "first," "second," "third," and the like in this disclosure are used for distinguishing between similar or similar objects or entities and not necessarily for limiting a particular order or sequence, unless otherwise indicated (Unless otherwise indicated). It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the disclosure are, for example, capable of operation in sequences other than those illustrated or otherwise described herein.
The term "unit/module" as used in this disclosure refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware or/and software code that is capable of performing the function associated with that element.
For the convenience of the reader to understand the present disclosure, at least some of the terms involved in this disclosure are now explained as follows:
artificial intelligence (Artificial Intelligence, AI) technology refers to technology that studies, develops theories, methods, techniques and application systems for simulating, extending and expanding human intelligence.
Deep Learning (DL) is a sub-field in the Machine Learning (ML) field, and is an inherent rule and presentation hierarchy of Learning sample data, and information obtained in these Learning processes greatly helps interpretation of data such as text, images and sounds.
Computer vision refers to a simulation of biological vision using a computer and related equipment to obtain three-dimensional information of a corresponding scene by processing acquired pictures or videos.
Image processing (image processing), which may also be referred to as image processing, refers to a technique of analyzing an image with a computer to achieve a desired result.
Characters refer to class units or symbols, including letters, numbers, operators, punctuation and other symbols, as well as some functional symbols. Characters are a collective term for letters, numbers, symbols in an electronic computer or radio communication, which is the smallest unit of data access in a data structure, and a character is usually represented by 8 binary bits (one byte).
The intelligent transportation system (Intelligent Traffic System, ITS) is also called an intelligent transportation system (Intelligent Transportation System), which is an integrated transportation system that uses advanced scientific technologies (information technology, computer technology, data communication technology, sensor technology, electronic control technology, automatic control theory, operation study, artificial intelligence, etc.) effectively and comprehensively for transportation, service control and vehicle manufacturing, and enhances the connection among vehicles, roads and users, thereby forming an integrated transportation system that ensures safety, improves efficiency, improves environment and saves energy.
The license plate recognition technology refers to a technology capable of extracting and recognizing a stationary or moving license plate (i.e. a license plate) from a complex background, for example, license plate information such as vehicle license plate number, color and the like is obtained through license plate extraction, image preprocessing, feature extraction, license plate character recognition and other technologies.
License plate recognition system (Vehicle License Plate Recognition, VLPR) refers to an application of computer video image recognition technology in vehicle license plate recognition. License plate recognition is widely used in highway vehicle management, and in electronic toll collection (Electronic Toll Collection, ETC) systems, it is also a major means of recognizing vehicle identity in combination with dedicated short range communication (Dedicated Short Range Communication, DSRC) technology.
License plate recognition is an important task in intelligent traffic systems, and has wide application in traffic monitoring, vehicle management and other aspects.
The license plate can be divided into a single-row license plate and a multi-row license plate according to the number of rows occupied by the characters of the license plate. As shown in fig. 1, a single-row license plate refers to a license plate with one row of characters, a multi-row license plate refers to a license plate with multiple rows of characters, and the multi-row license plate comprises a double-row license plate as shown in fig. 1.
Accordingly, the license plate recognition method comprises a method for recognizing a single-row license plate and a method for recognizing a plurality of rows of license plates. In contrast, in the case of applying the method for identifying a single-row license plate to identify a plurality of rows of license plates, character features of different rows of license plates in the plurality of rows of license plates are extruded together to cause mutual interference, so that the identification effect, such as the effectiveness and reliability of identification, is affected.
Compared with the recognition method of a plurality of license plates, the recognition method of the single-row license plates is more mature, so the recognition method of the single-row license plates is not described in the embodiment. In the related art, as shown in fig. 2, three types of license plate recognition methods are mainly used for solving the problem of multi-line license plate recognition, the first type of license plate recognition method is a license plate recognition method based on character detection or segmentation (e.g. "one, character detection or segmentation" as shown in fig. 2), the second type of license plate recognition method is a license plate recognition method based on line segmentation (e.g. "two, line segmentation" as shown in fig. 2), and the third type of license plate recognition method is a license plate recognition method based on non-segmentation character attention (e.g. "three, non-segmentation character attention" as shown in fig. 2).
The first type of method is to use recognition of the license plate to be recognized in the input image shown in fig. 2 as a character detection or segmentation task, for example, a license plate recognition system locates and classifies the position of each character in the license plate to be recognized, and finally, the recognition result is spliced to obtain the license plate number.
The method needs to label characters or pixel levels manually, so that a large amount of manpower and resources are consumed, and the overall performance of identifying the license plate to be identified is easily affected due to the fact that characters are detected or segmented incorrectly. As shown in fig. 2, this type of method cannot identify the penultimate number of the license plate to be identified.
The second type of method can be understood that the license plate recognition system performs horizontal segmentation on a plurality of recognition rows of license plates of the license plates to be recognized in the input image shown in fig. 2 to obtain a plurality of single-row license plates, performs horizontal stitching on the plurality of single-row license plates to obtain new single-row license plates, and then obtains license plates by using a single-row license plate recognition method.
The method needs to consider the direction of the license plate when the license plate recognition system is used for horizontal segmentation, and if the license plate recognition system does not consider the direction of the license plate when the license plate recognition system is used for horizontal segmentation, characters can be mistakenly segmented into two parts when the license plate is inclined, so that recognition is affected. As shown in fig. 2, this type of method cannot accurately identify the first letter and the last number of the license plate to be identified.
The third type of method can be understood that the license plate recognition system does not need to perform character segmentation or line segmentation, but can automatically pay attention to each character of the license plate to be recognized in a two-dimensional space, and the method can be mainly classified into a method based on a convolutional neural network (Convolutional Neural Networks, CNNs) and a method based on a cyclic neural network (Recurrent Neural Networks, RNNs) in an implementation manner, wherein the license plate to be recognized in the image to be recognized is recognized (as "CNNs" and "RNNs" shown in fig. 2). The license plate recognition system can calculate and obtain the space attention map of all characters based on the convolutional neural network, so that character characteristics are enhanced, and the multi-row license plate recognition performance is improved.
However, since the convolutional neural network extracts local features, although the license plate type (as shown in fig. 2) can be identified, the awareness of the license plate type and the character distribution is weak, and attention errors are easily caused, so that the identification performance is affected. The method based on the cyclic neural network can be used for classifying the types of the single-row or multi-row license plates in advance, then inputting the license plate types into the cyclic neural network to improve the type perception capability, and obtaining the space attention diagram of each character through the character sequence relation. However, the cyclic neural network can only perform one step when extracting character features, that is, features extracted in the current time step need to depend on features obtained in the previous time step, and the multi-step operation can increase feature extraction time and affect recognition efficiency.
It should be noted that, the content of the three license plate recognition methods in the related art is only information known to the inventor, and does not represent that the information has entered the public domain before the application date of the present disclosure, or that it may be the prior art of the present disclosure.
In order to avoid at least one of the above technical problems, the present disclosure proposes the technical idea of the inventive effort: by utilizing the correlation theory and method of deep learning, a license plate recognition method based on cascade perception (specifically, cascade perception of a global type perception network and a parallel character perception network) is provided, automatic positioning and recognition of characters in a license plate to be recognized are realized on the premise that characters or line operation is not needed in advance, the overall perception of the license plate type and character distribution of the license plate to be recognized is improved, and the recognition speed is improved, so that rapid and effective multi-line license plate recognition is realized.
The following description of the technical solutions in the embodiments of the present disclosure will be made clearly and completely with reference to the accompanying drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments that a person of ordinary skill in the art would obtain without making any inventive effort are within the scope of protection of this disclosure.
Based on the technical conception, the present disclosure provides a license plate recognition method.
Referring to fig. 3, fig. 3 is a schematic diagram of a license plate recognition method according to an embodiment of the disclosure. As shown in fig. 3, the method includes:
s301: and acquiring an image to be identified of the license plate to be identified.
The execution body of the embodiment may be a license plate recognition device (or a license plate recognition system as described above), the license plate recognition device may be a server (including a local server and a cloud server, where the server may be a cloud control platform, a vehicle-road collaborative management platform, a central subsystem, an edge computing platform, a cloud computing platform, etc.), or may be a road side device, or may be a terminal device, or may be a processor, or may be a chip, etc., and the embodiment is not limited.
By way of example, a road side device, such as a road side sensing device with a computing function, a road side computing device connected to the road side sensing device, in a system architecture for intelligent traffic road collaboration, the road side device comprises a road side sensing device and a road side computing device, the road side sensing device (e.g., a road side camera) is connected to the road side computing device (e.g., a road side computing unit RSCU), the road side computing device is connected to a server, the server can communicate with an autonomous or assisted driving vehicle in various ways; alternatively, the roadside awareness device itself includes a computing function, and the roadside awareness device is directly connected to the server. The above connections may be wired or wireless.
As can be seen from the analysis, the license plate recognition method can be applied to the recognition scene of the single-row license plate and the recognition scene of the multi-row license plate, so that the license plate to be recognized can be the single-row license plate to be recognized and the multi-row license plate to be recognized. And as shown in fig. 1, in the case that the license plate to be identified is a plurality of rows of license plates to be identified, the license plate to be identified may be a double row license plate to be identified.
The mode of acquiring the image to be identified by the license plate identification device is not limited in this embodiment, and for example, the following examples may be adopted:
In one example, the license plate recognition device may be coupled to the image capture device and receive the image to be recognized transmitted by the image capture device.
In another example, the license plate recognition device may provide an image-loading tool by which a user may transmit an image to be recognized to the license plate recognition device.
The image loading tool can be an interface used for being connected with external equipment, such as an interface used for being connected with other storage equipment, and the image to be identified transmitted by the external equipment is obtained through the interface; the image loading tool may also be a display device, for example, the license plate recognition device may input an interface for loading an image function on the display device, through which a user may import an image to be recognized into the license plate recognition device, and the license plate recognition device obtains the imported image to be recognized.
In yet another example, the license plate recognition device is configured with an image acquisition component (e.g., a camera, etc.), which can acquire the image to be recognized, so that the license plate recognition device acquires the image to be recognized.
S302: and sequentially carrying out local feature extraction processing and global type perception processing on the image to be identified to obtain global features of the license plate to be identified, wherein the global features are used for representing the relation between any two pixel points in the image to be identified.
As shown in fig. 4 (fig. 4 is a schematic diagram of a license plate recognition method according to an embodiment of the present disclosure), an exemplary license plate recognition device may first perform local feature extraction on an image to be recognized, and then perform global type sensing on a result obtained by the local feature extraction processing to obtain global features.
Where global and local are relative concepts, local refers to the identification of features between pixels that are relatively close to a range of locations in an image to be identified, such as adjacent pixels, relative to global, and global refers to the identification of features between pixels that are relatively close to and far from a range of locations in an image to be identified, such as any pixels, relative to local.
Therefore, the global feature characterizes a relationship between any at least two pixel points, and the relationship may be a positional relationship, a color relationship, a texture relationship, or the like, which are not listed here.
The global features can also characterize the license plate type of the license plate to be identified, namely the global features can characterize the characteristics of the license plate to be identified from at least two dimensions, one dimension is the dimension of the pixel relationship, and the other dimension is the dimension of the license plate type.
S303: and performing parallel character sensing processing on the image to be recognized according to the global features and the parallel reading sequence (Parallel Reading Order, PRO) to obtain character enhancement features of the license plate to be recognized.
For example, as shown in fig. 4, the license plate recognition device reads characters in the license plate to be recognized (such as license plate number shown in fig. 4) based on the parallel reading sequence, and then performs parallel character sensing by combining all the features and the content read in the parallel reading sequence, so as to obtain character enhancement features.
The parallel reading sequence refers to reading the content (such as characters) in the license plate to be identified based on a parallel reading mode. Parallel character perception refers to determining the character sequence relation of characters in a license plate to be recognized in a parallel mode so as to obtain the corresponding space attention map (namely, character characteristics with enhanced space attention) of each character, and then obtaining character enhancement characteristics.
In this embodiment, the license plate recognition device determines character features with enhanced spatial attention (i.e., character enhancing features) by adopting a manner of cascade sensing of global features and parallel character sensing processing, so that overall sensing of license plate types, relations among pixel points and character distribution of the license plate to be recognized can be realized, cascade sensing is performed in combination with parallel reading sequence, and recognition efficiency can be improved.
That is, compared to the third type license plate recognition method in the above example, the license plate recognition method of the present disclosure further employs the technical features of global type perception processing to determine the pixel features and the type features of the license plate to be recognized from the pixel overall dimension of the image to be recognized, further employs the parallel reading sequence to improve the recognition efficiency in combination with the parallel reading sequence, and further employs the features of cascade perception of global type perception and parallel character perception processing to improve the accuracy and reliability of recognition through the features of cascade perception.
S304: and predicting and obtaining license plate information of the license plate to be recognized according to the character enhancement features.
For example, as shown in fig. 4, after obtaining the character enhancement feature, the license plate recognition device of the license plate recognition device may predict based on the character enhancement feature (e.g., "character prediction" shown in fig. 4), thereby obtaining license plate information. The license plate information includes a license plate number as shown in fig. 4.
Based on the analysis, the disclosure provides a license plate recognition method, which comprises the following steps: obtaining an image to be recognized of a license plate to be recognized, sequentially carrying out local feature extraction processing and global type perception processing on the image to be recognized to obtain global features of the license plate to be recognized, wherein the global features are used for representing the relation between any two pixel points in the image to be recognized, carrying out parallel character perception processing on the image to be recognized according to the global features and the parallel reading sequence to obtain character enhancement features of the license plate to be recognized, predicting license plate information of the license plate to be recognized according to the character enhancement features, and in the embodiment, the license plate recognition device determines the character enhancement features by combining the global features and the parallel reading sequence and determines the technical features of the license plate information according to the character enhancement features, so that on one hand, the defect that a large amount of manpower and resources are consumed due to the first type of method can be avoided, resources are saved, the defect that recognition accuracy is lowered due to character detection or segmentation errors is avoided, and recognition accuracy is improved is facilitated; on the other hand, the defect of low recognition accuracy caused by the fact that characters are divided into two parts by mistake when the license plate is inclined in the second class of methods can be avoided, and the recognition accuracy is improved; on the other hand, the defects of low efficiency and low accuracy caused by the third method can be avoided, and the recognition efficiency and accuracy are improved; in still another aspect, the license plate recognition device determines the pixel characteristics and the type characteristics of the license plate to be recognized from the pixel overall dimension of the image to be recognized by adopting the technical characteristics of global type perception processing, and further adopts the parallel reading sequence to combine the parallel reading sequence to improve the recognition efficiency, and further adopts the characteristics of cascade perception of global type perception and parallel character perception to improve the recognition accuracy and reliability through the characteristics of cascade perception.
In order to make readers more deeply understand the implementation principle of the license plate recognition method of the present disclosure, the license plate recognition method of the present disclosure will now be described in detail with reference to fig. 5. Fig. 5 is a schematic diagram of a license plate recognition method according to another embodiment of the disclosure. As shown in fig. 5, the method includes:
s501: and acquiring an image to be identified of the license plate to be identified.
It should be understood that, in order to avoid the cumbersome statement, the technical features of this embodiment that are the same as those of the above embodiment are not repeated.
For example, regarding the implementation principle of S501, reference may be made to S301, which is not described herein.
S502: and carrying out convolution processing on the image to be identified to extract local features of the image to be identified, wherein the local features are used for representing the relation between adjacent pixel points in the image to be identified.
The license plate recognition device is provided with a license plate recognition model, the license plate recognition model comprises a local feature extraction network, and the local feature extraction network can be used for carrying out local feature extraction processing on an image to be recognized to obtain local features.
The local feature extraction network is a Convolution (Convolition) network, that is, the license plate recognition device may perform local feature extraction by using a Convolution network, so as to obtain a license plate visual feature (i.e., local feature) with rich semantics. The local features may be features of the image to be identified in pixels, colors, edges, strokes, positions, etc.
For example, the width (in pixels), height (in pixels), and number of channels (in pixels) of the license plate image to be recognized are 96×32×3, and the feature size becomes 24×8×256 after passing through an 18-layer convolution network. The convolution network comprises 64 channels, 128 channels and 256 channels, 6 layers are respectively, convolution parameters are kernels (kernel) =3, sampling intervals (stride) =1 during convolution, padding (padding) =1, and maximum value pooling (Max-pooling) with parameters of 2×2 is adopted for double downsampling after the convolution of the layers 6 and 12, so that the local feature size is 1/4 of the size of an image to be identified. In some embodiments, each layer of convolution may be followed by a normalization (BatchNorm) operation and an activation function (ReLU) operation to increase the convergence rate and extract valid local features.
S503: and adding position codes to the pixel points in the local characteristics to obtain position code characteristics.
In some embodiments, in combination with the above example, after obtaining the local feature, the license plate recognition apparatus may convert the local feature from the three-dimensional size 24×8×256 to the two-dimensional size 192×256, where 192 represents the positions of all the pixels and 256 represents the feature of each position, before executing S503.
S504: and performing full connection processing, transposition processing, normalization processing and convolution processing on the position coding features to obtain global features.
In combination with the above example, S503-S504 are one of specific implementations of global type sensing, and global type sensing may be implemented by the license plate recognition device based on the license plate recognition model.
Illustratively, the license plate recognition model includes a global type-aware network for determining global features from the local features, and the global type-aware network is implemented specifically based on a self-attention mechanism.
Since the license plate recognition device refers to Self-Attention calculation when determining global features based on Self-Attention mechanism (SA), and calculates Attention weights of each pixel (or referred to as feature point) and all other pixels in parallel when calculating Self-Attention, so that position information of each pixel is discarded, as shown in fig. 6 (schematic diagram of global type perception model in fig. 6, which is an embodiment of the present disclosure), after the license plate recognition device inputs local features X into the global type perception model, position codes (Positional Encoding, PE) can be added to each pixel to introduce a relative positional relationship between the pixels.
In some embodiments, as can be seen from the combination formula 1, the license plate recognition device may obtain the position coding feature of each pixel point by adding a sine curve value or a cosine curve value to the local feature of each pixel point, and the combination formula 1:
if it isIs sinusoidal, then ∈>If->For cosine curve value, +.>。
Wherein,,representing the position information of the pixel point, < >>The total number of feature dimensions representing each pixel is 256 +.>Represents +.>Or->And characteristic values.
In this embodiment, the license plate recognition device adds a curve value sine or cosine curve value to each pixel point, so that it can ensure that the position coding features of different position information can be obtained through linear transformation, and therefore the position coding features contain the relative position relation of different pixel points.
In some embodiments, the full join process in S504 includes generating first query information, first key information, first value information from the position-coding feature. The transposition process in S504 includes performing a transposition process on the first key information. The normalization processing in S504 includes normalization processing of the first query information and the transposed processed first key information. The convolution processing in S504 includes performing convolution processing on the first value information and the first attention weight obtained by the normalization processing.
The license plate recognition device generates corresponding first query Q information, first key K information and first value V information by using license plate self information characterized by the position coding features, and then multiplies the first query Q information and the first key K information to obtain first attention weight, and multiplies the first attention weight and the first value V information to obtain global features with enhanced self-attention.
Specifically, as shown in fig. 6, the position coding feature obtained by the position coding is input to the fully connected layer of the global type sensing network to perform fully connected processing (such as "fully connected" shown in fig. 6). As shown in fig. 6, the full connection layer includes: linear transformation parameters of first query Q informationLinear transformation parameter of first key K information +.>Linear transformation parameter of the first value V information +.>。
Correspondingly, the full connection layer is based on the position coding feature sumDetermining first key K information (as "key K" shown in fig. 6); the full connection layer is based on the position coding feature and +.>Determining first query Q information (e.g., "query Q" as shown in fig. 6); the full connection layer is based on the position coding feature and +.>The first value V information (the "value V" as shown in fig. 6) is determined.
As shown in fig. 6, the license plate recognition device transposes the "key K" based on the global type sensing network (transpose as shown in fig. 6), then multiplies the "query Q", then normalizes the "key K" based on the normalization function (Softmax as shown in fig. 6), obtains a first attention weight (attention weight as shown in fig. 6), then multiplies the "attention weight" by the "value V", then convolves the "key K" with the "value V" (one-dimensional convolution as shown in fig. 6, and the parameters of the one-dimensional convolution are "one-dimensional convolution":") to obtain a global feature (e.g.," global feature Y "as shown in fig. 6).
In some embodiments, the self-attention mechanismCan be represented by formula 2, formula 2:
wherein,,as can be seen from the description above with respect to fig. 6, < + >>，And->Respectively representing the linear transformation parameters corresponding to the first query Q information, the first key K information and the first value V information respectively, ">The parameter dimension representing the first key K information, softmax represents the normalization function divided byAnd the parameter normalization processing is realized by utilizing the Softmax function.
In some embodiments, the license plate recognition device can use a multi-head self-attention mechanism (multi-head) to obtain more various global features on the basis. Illustratively, the license plate recognition device may divide the self-attention computation into a plurality of heads (heads) by equation 3, and then obtain a final global feature by self-attention stitching (Concat) obtained by each head, equation 3:
wherein,,，indicate->Corresponding linear transformation parameters. In some embodiments, ->Can be 256, and is shared by +.>The head of the device is provided with a plurality of heads,and the linear transformation parameters after feature splicing.
In some embodiments, after multi-head attention, the license plate recognition device may perform additive fusion on the position coding features and the features obtained by multi-head attention in a residual connection manner, in a training stage, to prevent the global type perception network from overfitting and gradient disappearance, and then perform stable training through layer normalization (LayerNorm), and then, the license plate recognition device may implement spatial transformation by using a position-related feed-forward network (Feed Forward Network, FFN), so as to improve the feature extraction capability. The specific implementation manner of the feedforward network may be two-layer one-dimensional convolution, and an activation function (ReLU) is added between the two-layer one-dimensional convolution, as shown in formula 4, formula 4:
Wherein,,and->Linear transformation parameters representing feed forward network, +.>And->Representing the bias parameters of the feed forward network, hidden layer parameters +.>May be 512./>
After passing through the feedforward network, the license plate recognition device performs addition fusion on the characteristics obtained by the multi-head attention and the characteristics obtained by the feedforward network, and in a training stage, the global type perception network is prevented from overfitting and gradient disappearance, and then the license plate recognition device performs stable training through layer normalization (LayerNorm). In the application stage, the license plate recognition device stacks the multi-head attention and the feedforward network for two times with the same parameters, so that the global feature extraction capability is improved, and the final global feature Y is obtained. The global feature Y has the same size as the input local feature X, namely 192 multiplied by 256, but improves the perceptibility of the overall type and character distribution of the license plate.
S505: and generating second key information and second value information according to the global features.
Illustratively, as shown in fig. 7 (fig. 7 is a schematic diagram of the principle of generating character enhancement features according to an embodiment of the present disclosure), the license plate recognition device may determine the second value V information (the "value V" shown in fig. 7) and the second key K information (the "key K" shown in fig. 7) according to the global feature (the "global feature Y" shown in fig. 7), respectively. And as shown in fig. 7, the "key K" may be obtained based on a full connection process (e.g., "full connection" as shown in fig. 7), and the full connection process is specifically based on the linear transformation parameters of the "key K" in the full connection layer (e.g., "as shown in fig. 7"") are determined.
S506: and determining the second inquiry information according to the parallel reading sequence.
In some embodiments, S506 may include the steps of:
a first step of: and acquiring an initial reading sequence vector of the license plate to be identified from the image to be identified based on the parallel reading sequence.
And a second step of: and carrying out embedding processing on the initial reading sequence vector based on a preset embedding function to obtain a target reading sequence vector.
And a third step of: and performing full connection processing on the target reading sequence vector to obtain second query information.
For example, for a license plate to be identified as a double-row license plate as shown in fig. 7, the reading sequence is sequentially increased from left to right. Considering that most license plate characters are within eight bits, the license plate recognition device can obtain an initial reading sequence vector in a mode that the longest character number is maxT=8 and the last complement characters are less than eight bits.
Wherein, the license plate recognition device can read the initial reading sequence vector from the image to be recognized through the parallel reading sequence, and then the license plate recognition device can input the initial reading sequence vector into the embedding function (such as ' embedded ' and ' shown in fig. 7) "") a new reading order vector (i.e., the target reading order vector) is obtained.
In some embodiments, the license plate recognition model further includes a parallel character sensing network, the parallel character sensing network is used for determining character enhancement features through global features and parallel reading sequences, and illustratively, the license plate recognition device performs full-connection processing on the target reading sequence vector based on the parallel character sensing network (as "full-connection" shown in fig. 7, specifically, linear transformation parameters of the second query Q information based on a full-connection layer are as "shown in fig. 7") ""full connection processing), a second query Q information (such as" query Q "shown in fig. 7).
S507: and carrying out fusion processing, nonlinear activation processing, full-connection processing and normalization processing on the second key information and the second query information in sequence to obtain a second attention weight.
Exemplary, the license plate recognition device performs fusion processing (specifically, may be an addition operation as shown in fig. 7) on the second key information (such as "key K" shown in fig. 7) and the second query Q information (such as "query Q" shown in fig. 7) based on the parallel character perception network, and then performs fusionThe result of the process is subjected to a nonlinear activation process, which may be specifically based on a nonlinear activation function (as shown in fig. 7'") and then performing full-connection processing (such as" full-connection "shown in fig. 7) on the result of the nonlinear activation processing, where the full-connection processing may specifically be based on the preset output conversion parameters of the full-connection layer (such as" "shown in fig. 7)>") and then normalizing the result of the full connection processing based on a normalization function (such as" Softmax "shown in fig. 7) to obtain a second attention weight (such as" attention weight "shown in fig. 7).
S508: a character enhancement feature is generated based on the second attention weight and the second value information.
Illustratively, license plate recognition multiplies the second attention weight (the "attention weight" shown in fig. 7) with the second value information (the "value V" shown in fig. 7) to obtain a character enhancement feature (the "character enhancement feature Z" shown in fig. 7).
In some embodiments, character enhancement features may be calculated based on equation 5Formula 5:
wherein,,，representing the second attention weight, softmax being a normalization function for achieving parameter normalization, tanh being a nonlinear activation function for improving the nonlinear expression capacity,For embedding the function +. >，Respectively represent a query parameter, a key parameter and an output conversion parameter, wherein the parameter dimension +.>Can be 256, characteristic position length +.>May be 192.
S509: and predicting and obtaining license plate information of the license plate to be recognized according to the character enhancement features.
For example, regarding the implementation principle of S509, reference may be made to S304, which is not described herein.
In some embodiments, the license plate recognition model further includes a character prediction network for determining license plate information based on the character enhancement features.
In the training stage of training the license plate recognition model by the license plate recognition device, under the condition that the character enhancement features are obtained by the license plate recognition device, the character enhancement features can be converted into output probabilities of predicted characters through a connecting layerAnd performing cross entropy loss on the output probability to obtain a license plate recognition model through iterative optimization. Wherein the cross entropy loss can be represented by formula 6, formula 6:
wherein,,representing the number of categories of all characters, e.g. Arabic numerals and English wordsMother plus blank category, 37 categories, < ->Representing the true category of the character->Representing the character class of the network prediction.
It should be understood that the foregoing examples are merely exemplary, and are not to be construed as limiting the license plate recognition method of the present disclosure in any way as to the possible implementation of the license plate recognition method.
For example, based on the analysis, the license plate recognition device may recognize the image to be recognized of the license plate to be recognized by using a pre-trained license plate recognition model to obtain the license plate number in the license plate to be recognized, and the training of the license plate recognition model may also be the license plate recognition device. However, in other embodiments, the license plate recognition model may be obtained by training by other devices, and the other devices may transmit the license plate recognition model to the license plate recognition device in the case of obtaining the license plate recognition model by training.
Or in some embodiments, the license plate recognition model can be obtained by training by other devices, and in the case that the license plate recognition task of the license plate recognition device is triggered, the license plate recognition device can call the license plate recognition model obtained by training by other devices in a service calling mode so as to recognize the image to be recognized corresponding to the license plate recognition task based on the called license plate recognition model, and license plate information is obtained.
In some embodiments, taking a main body for training the license plate recognition model as a training device, the training device can train to obtain the license plate recognition model by adopting a data set mode.
For example, the training device may train to obtain the license plate recognition model by using the first data set rodol. Wherein the first dataset may comprise 2 ten thousand images, each image may be 1280 pixels wide and 720 pixels high.
The first data set may include two types of license plates of an automobile and a motorcycle, the license plates of the motorcycle may be double-row license plates, the license plates of the automobile may be single-row license plates, and the number of the double-row license plates and the number of the single-row license plates may be the same. The first dataset comprises a training set, a validation set, and a test set, the training set may comprise 8000 images, the validation set may comprise 4000 images, and the test set may comprise 8000 images.
The training device can also adopt a second data set UFPR to train to obtain a license plate recognition model. Wherein the second set of data may include 4500 images, each image may be 1920 pixels wide and 1080 pixels high.
The second data set may include two types of license plates of an automobile and a motorcycle, the license plates of the motorcycle may be double-row license plates, the license plates of the automobile may be single-row license plates, and the number of the single-row license plates is three times that of the double-row license plates. The second data set comprises a training set, a verification set and a test set, the number of images in the training set is 40% of the number of images in the second data set, the number of images in the verification set is 20% of the number of images in the second data set, and the number of images in the test set is 40% of the number of images in the second data set.
The training device may also use the third dataset CCPD for training to obtain a license plate recognition model. Wherein the third dataset comprises two versions of data (CCPDv 1 and CCPDv 2), each image in the third dataset may be 720 pixels wide and 1160 pixels high.
Wherein the CCPDv1 version of the dataset contains about 25 ten thousand images, with the following subsets: base subset (Base), large brightness variation (DB), large distance variation (FN), in-plane rotation (rotation), out-of-plane rotation (Tilt), special Weather (Weather), and various challenging scenarios (Challenge). The CCPDv2 version of the dataset contains about 30 ten thousand images, and the CCPDv2 version of the dataset has no Weather subset but another subset blurs the picture (Blur) relative to the CCPDv1 version of the dataset. The basic subset is used as a training set and a verification set, other subsets are used for testing, and the average value of all the tested subsets is used for measuring the overall recognition performance.
Likewise, the numbers in the above examples are for illustrative purposes only and are not to be construed as limiting the choice of numbers.
For license plate recognition, the entire license plate recognition is considered to be correct only when all the characters in the license plate are recognized correctly. For the first data set and the second data set, the training device can independently test the recognition performance of the double-row license plate and the single-row license plate and calculate the average recognition performance of the two license plates. For both versions of the third dataset, since only a single row of license plates is included, the training device may test only the single row of license plate recognition performance for verification of generalization capability.
For the running speed of the training device, the running frame number per second (Frames Per Second, FPS) can be used for measurement, and the display card used for the experiment is not limited.
In some embodiments, the training device may train 6 ten thousand rounds with an optimizer with momentum parameters set to 0.9 and 0.99, respectively. Training Batch (Batch Size) set to 32, weight Decay (Weight Decay) set toThe Learning Rate (Learning Rate) is set to +.>). In training, two data augmentation modes of color disturbance and random scaling are adopted to improve the robustness of the license plate recognition model.
In some embodiments, to verify the validity of the license plate recognition model, we can perform performance tests of license plate recognition on different data sets based on the verification device, where the test indexes include recognition accuracy and running speed. In order to verify the functions of the global type sensing network and the parallel character sensing network respectively, the global type sensing network and the parallel character sensing network can be removed respectively on the basis of a complete license plate recognition model, and other modules remain unchanged.
As shown in fig. 8 (fig. 8 is a schematic diagram of recognition performance of different license plate recognition methods), after the global type sensing network is removed, the running speed of the verification device is improved, namely 9FPS, but the overall recognition performance, especially the recognition performance of the double-row license plate is greatly reduced, as shown in fig. 9 (fig. 9 is a schematic diagram of comparison of removing the global type sensing network and reserving the global type sensing network). This is because, after the global type sensing network is removed, the local features are difficult to accurately sense the license plate type and character distribution, so that the attention of the subsequent parallel character sensing network is offset and blurred to different degrees, and thus, the license plate recognition error is caused.
After the parallel character sensing network is removed, the license plate recognition model is degenerated into a cyclic neural network, the character prediction of the current time step needs to depend on the prediction result of the previous time step, and the running speed of the verification device is greatly reduced by 30FPS. Meanwhile, because no semantic information exists between license plate characters, the cyclic neural network can forcedly couple the relations before the characters, so that the license plate recognition performance is reduced. That is, the parallel reading order-based method of the present disclosure can decouple semantic relationships between characters, and can improve license plate recognition performance.
Based on the technical conception, the present disclosure further provides a license plate recognition device.
Referring to fig. 10, fig. 10 is a schematic diagram of a license plate recognition device according to an embodiment of the disclosure, and as shown in fig. 10, the license plate recognition device 100 includes:
the acquiring unit 1001 is configured to acquire an image to be identified of a license plate to be identified.
The first processing unit 1002 is configured to sequentially perform local feature extraction processing and global type sensing processing on the image to be identified, so as to obtain global features of the license plate to be identified, where the global features are used to characterize a relationship between any two pixel points in the image to be identified.
The second processing unit 1003 is configured to perform parallel character sensing processing on the image to be identified according to the global feature and the parallel reading sequence, so as to obtain a character enhancement feature of the license plate to be identified.
And the prediction unit 1004 is configured to predict and obtain license plate information of the license plate to be identified according to the character enhancement feature.
In some embodiments, the first processing unit 1002 includes:
the convolution subunit 10021 is configured to perform convolution processing on the image to be identified to extract a local feature of the image to be identified, where the local feature is used to characterize a relationship between adjacent pixel points in the image to be identified.
An adding subunit 10022 is configured to add a position code to the pixel points in the local feature, so as to obtain a position code feature.
The first processing subunit 10023 is configured to perform full connection processing, transposition processing, normalization processing, and convolution processing on the position-coding feature, to obtain the global feature.
In some embodiments, the fully-connected processing includes generating first query information, first key information, first value information from the position-coding feature; the transposition processing includes performing transposition processing on the first key information; the normalization processing comprises normalization processing of the first query information and the transposed first key information; the convolution processing comprises convolution processing of the first value information and the first attention weight obtained by the normalization processing.
In some embodiments, the second processing unit 1003 includes:
a first generating subunit 10031 is configured to generate second key information and second value information according to the global feature.
A determining subunit 10032, configured to determine the second query information according to the parallel reading order.
And a second processing subunit 10033, configured to sequentially perform fusion processing, nonlinear activation processing, full connection processing, and normalization processing on the second key information and the second query information, to obtain a second attention weight.
A second generating subunit 10034 is configured to generate the character enhancement feature according to the second attention weight and the second value information.
In some embodiments, the determining subunit 10032 includes:
and the acquisition module is used for acquiring an initial reading sequence vector of the license plate to be identified from the image to be identified based on the parallel reading sequence.
The first processing module is used for carrying out embedding processing on the initial reading sequence vector based on a preset embedding function to obtain a target reading sequence vector.
And the second processing module is used for carrying out full connection processing on the target reading sequence vector to obtain the second query information.
In some embodiments, the license plate information is obtained by performing recognition processing on an input image to be recognized by a pre-trained license plate recognition model;
the license plate recognition model comprises a local feature extraction network, a global type perception network, a parallel character perception network and a character prediction network;
the local feature extraction network is used for carrying out local feature extraction processing on the image to be identified to obtain local features; the global type perception network is used for determining the global features according to the local features; the parallel character perception model is used for determining the character enhancement features according to the global features and the parallel reading sequence; and the character prediction network is used for determining the license plate information according to the character enhancement features.
According to embodiments of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
According to an embodiment of the present disclosure, the present disclosure also provides a computer program product comprising: a computer program stored in a readable storage medium, from which at least one processor of an electronic device can read, the at least one processor executing the computer program causing the electronic device to perform the solution provided by any one of the embodiments described above.
Fig. 11 illustrates a schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile apparatuses, such as personal digital assistants, cellular telephones, smartphones, wearable devices, and other similar computing apparatuses. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 11, the apparatus 1100 includes a computing unit 1101 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a Random Access Memory (RAM) 1103. In the RAM 1103, various programs and data required for the operation of the device 1100 can also be stored. The computing unit 1101, ROM 1102, and RAM 1103 are connected to each other by a bus 1104. An input/output (I/O) interface 1105 is also connected to bus 1104.
Various components in device 1100 are connected to I/O interface 1105, including: an input unit 1106 such as a keyboard, a mouse, etc.; an output unit 1107 such as various types of displays, speakers, and the like; a storage unit 1108, such as a magnetic disk, optical disk, etc.; and a communication unit 1109 such as a network card, modem, wireless communication transceiver, or the like. The communication unit 1109 allows the device 1100 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.
The computing unit 1101 may be a variety of general purpose and/or special purpose processing components having processing and computing capabilities. Some examples of computing unit 1101 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 1101 performs the respective methods and processes described above, such as a license plate recognition method. For example, in some embodiments, the license plate recognition method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 1108. In some embodiments, some or all of the computer programs may be loaded and/or installed onto device 1100 via ROM 1102 and/or communication unit 1109. When the computer program is loaded into the RAM 1103 and executed by the computing unit 1101, one or more steps of the license plate recognition method described above may be performed. Alternatively, in other embodiments, the computing unit 1101 may be configured to perform the license plate recognition method by any other suitable means (e.g., by means of firmware).
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuit systems, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), systems On Chip (SOCs), complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for carrying out methods of the present disclosure may be written in any combination of one or more programming languages. These program code may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus such that the program code, when executed by the processor or controller, causes the functions/operations specified in the flowchart and/or block diagram to be implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package, partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical hosts and VPS service ("Virtual Private Server" or simply "VPS") are overcome. The server may also be a server of a distributed system or a server that incorporates a blockchain.
It should be noted that, the processes of collecting, storing, using, processing, transmitting, providing, disclosing and the like of the images to be identified in the embodiment all conform to the rules of the related laws and regulations, and do not violate the public order colloquial.
It will be apparent to those skilled in the art that embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, magnetic disk storage, optical storage, and the like) having computer-usable program code embodied therein.
The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These processor-executable instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various modifications and variations can be made to the present disclosure without departing from the spirit or scope of the disclosure. Thus, the present disclosure is intended to include such modifications and alterations insofar as they come within the scope of the appended claims or the equivalents thereof.