CN105809121A - Multi-characteristic synergic traffic sign detection and identification method - Google Patents

Abstract

The invention discloses a multi-characteristic synergic traffic sign detection and identification method which is performed according to the following steps. A color probability model is established for traffic signs with different colors through the images of traffic sign samples and representative colors are determined out of the traffic signs with different colors so as to obtain probability check lists for the representative colors, train and obtain shape classifying devices for traffic signs belonging to different categories and identifying models. For traffic images to be detected, each probability check list for the representative colors is used first to get the probability images of the traffic images, which are then converted to grey scale maps. An MSER algorithm is used to detect the areas in the grey scale maps which change stably and the areas are regarded as potential windows to be picked up that meet the pre-set height-width ratio. The shape classifying devices then determines whether the potential windows to be picked up contain traffic signs or not, and if they do, the identifying models will identify these corresponding shapes. The method can achieve a better detection and identification effect by combining the characteristics of colors and shapes of traffic signs.

Description

Translated fromChinese

多特征协同的交通标志检测与识别方法Multi-feature collaborative traffic sign detection and recognition method

技术领域technical field

本发明属于交通标志检测与识别技术领域，更为具体地讲，涉及一种多特征协同的交通标志检测与识别方法。The invention belongs to the technical field of traffic sign detection and recognition, and more specifically relates to a multi-feature collaborative traffic sign detection and recognition method.

背景技术Background technique

随着经济与技术的发展，智能交通技术得到了大力的发展，交通标志的检测与识别作为智能交通技术的一个重要组成部分，得到了越来越广泛的重视。交通标志的检测与识别一般是根据所拍摄的道路图像，首先对道路图像进行预处理，然后从道路图像中检测出交通标志，最后再进行分类识别。With the development of economy and technology, intelligent transportation technology has been vigorously developed. As an important part of intelligent transportation technology, the detection and recognition of traffic signs has been paid more and more attention. The detection and recognition of traffic signs is generally based on the captured road images. First, the road images are preprocessed, and then traffic signs are detected from the road images, and finally classified and recognized.

交通标志检测的任务是在输入的图像中检测交通标志的位置，应具有低漏检率、低误检率的特点。其中低漏检率是最关键的指标，因为交通标志检测是后续的交通标志识别的基础，前者为后者提供了识别的对象。一旦检测的过程漏掉某个标志，将会直接导致整个交通标志检测与识别系统漏掉这个交通标志。针对交通标志形状规则、颜色鲜明的特点，学者们提出了很多基于形状、基于颜色、基于模板的检测方法。从2013年的德国交通标志检测大赛公布的结果来看，基于模板的方法在准确率方面具有很大的优势，对光照、遮挡等多种不利条件都有很好的适应能力。但是这类方法的运算量普遍较大，难以满足实时性的要求。The task of traffic sign detection is to detect the position of traffic signs in the input image, which should have the characteristics of low missed detection rate and low false detection rate. Among them, low missed detection rate is the most critical indicator, because traffic sign detection is the basis of subsequent traffic sign recognition, and the former provides the recognition object for the latter. Once a certain sign is missed during the detection process, it will directly cause the entire traffic sign detection and recognition system to miss this traffic sign. In view of the regular shape and bright colors of traffic signs, scholars have proposed many shape-based, color-based, and template-based detection methods. Judging from the results published in the German traffic sign detection competition in 2013, the template-based method has great advantages in accuracy, and has good adaptability to various adverse conditions such as lighting and occlusion. However, such methods generally require a large amount of computation, which is difficult to meet the real-time requirements.

对于交通标志的识别而言，是一个多类别的分类问题，因此很多模式识别算法被引入交通标志的识别中来，包括模板匹配、稀疏编码、SupportVectorMachine(SVM)，DeepNeuralNetworks(DNN)，Adaboost算法等等。CNN(ConvolutionalNeuralNetwork,卷积神经网络)相对于其他机器学习算法来说，由于其具有自动提取特征的特性而被广泛关注。但是普通直线型CNN网络在交通标志识别过程中，由于后一层的输入只和前一层的输出有关，对于那些过小且模糊图像的识别能力还是有限制的。For the recognition of traffic signs, it is a multi-category classification problem, so many pattern recognition algorithms are introduced into the recognition of traffic signs, including template matching, sparse coding, SupportVectorMachine (SVM), DeepNeuralNetworks (DNN), Adaboost algorithm, etc. Wait. Compared with other machine learning algorithms, CNN (Convolutional Neural Network, Convolutional Neural Network) has attracted widespread attention due to its feature of automatic feature extraction. However, in the traffic sign recognition process of the ordinary linear CNN network, since the input of the latter layer is only related to the output of the previous layer, the recognition ability of those too small and blurred images is still limited.

发明内容Contents of the invention

本发明的目的在于克服现有技术的不足，提供一种多特征协同的交通标志检测与识别方法，采用交通标志的多种特征协同完成交通标志的检测与识别，降低交通标志的漏检率和误检率，提高交通标志的识别效果。The purpose of the present invention is to overcome the deficiencies of the prior art, provide a multi-feature collaborative traffic sign detection and recognition method, use the multiple features of traffic signs to complete the detection and recognition of traffic signs, reduce the missed detection rate of traffic signs and Increase the false detection rate and improve the recognition effect of traffic signs.

为实现上述发明目的，本发明多特征协同的交通标志检测与识别方法，其特征在于包括：In order to achieve the purpose of the above invention, the multi-feature collaborative traffic sign detection and recognition method of the present invention is characterized in that it includes:

S1：根据交通标志的颜色特点对交通标志进行分类，每个颜色类别分别获取若干张交通标志样本图像；对于每张交通标志样本图像，提取各个像素点的颜色特征，根据颜色特征对该交通标志样本图像的所有像素进行聚类，聚类数量为N+1，N为交通标志的主要颜色数量，将每类交通标志样本图像中各样本图像中对应聚类的像素点合并，得到该颜色类别的N+1个样本集，对每个样本集建立对应的高斯模型；S1: Classify the traffic signs according to the color characteristics of the traffic signs, and obtain several traffic sign sample images for each color category; All the pixels of the sample image are clustered, the number of clusters is N+1, and N is the number of main colors of the traffic sign, and the pixels corresponding to the clusters in each sample image of each type of traffic sign sample image are combined to obtain the color category N+1 sample sets, and establish a corresponding Gaussian model for each sample set;

S2：对于每个颜色类别的交通标志，根据其对应的N+1个的颜色概率模型计算各个R,G,B值属于各颜色的概率p(c_i|x)，x表示像素点R,G,B值，c_i表示颜色，i＝1,2,…,N+1；从该颜色类别的N种主要颜色中选取一种颜色作为代表颜色，记为c_i′，然后对代表颜色的概率进行归一化得到其归一化概率建立各个R,G,B值属于代表颜色的概率查找表；S2: For traffic signs of each color category, calculate the probability p(c_i |x) that each R, G, and B value belongs to each color according to its corresponding N+1 color probability model, x represents the pixel point R, G, B value, c_i represents the color, i=1,2,...,N+1; select a color from the N main colors of the color category as the representative color, denoted as c_i ', and then the representative color The probability is normalized to get its normalized probability Establish a probability lookup table where each R, G, and B value belongs to a representative color;

S3：将交通标志根据形状分为M类，每个形状类别建立一个基于HOG特征的形状分类器，其训练方法为：对于每个形状分类器，获取两类样本图像，一类为对应形状的交通标志样本图像，另一类为其他图像；统一样本图像尺寸，提取每张样本图像的HOG特征，将HOG特征作为形状分类器的输入，是否为所属形状类别的判定值作为形状分类器的输出，训练得到对应形状类别交通标志的形状分类器；S3: Divide the traffic signs into M categories according to their shapes, and establish a shape classifier based on HOG features for each shape category. The training method is as follows: for each shape classifier, two types of sample images are obtained, one is the corresponding shape Traffic sign sample images, the other is other images; unify the size of the sample images, extract the HOG features of each sample image, use the HOG features as the input of the shape classifier, and take the judgment value of whether it belongs to the shape category as the output of the shape classifier , train the shape classifier corresponding to the shape category traffic sign;

S4：按照交通标志的形状类别数M，对每一个形状类别的交通标志分别设置一个识别模型，每个形状类别分别获取若干张交通标志样本图像；先对所有样本图像进行预处理，包括统一图像尺寸和对比度增强；采用每个形状类别预处理后的样本图像对其识别模型进行训练，每次训练完毕后将预处理后的交通标志样本图像随机排序然后分组，设置Q种畸变方式，对每组样本图像在Q种畸变方式中随机选择q种畸变方式按照随机顺序对样本图像进行畸变处理，采用畸变处理后的新样本图像对其识别模型进行训练，直到达到训练结束条件；S4: According to the number M of shape categories of traffic signs, set a recognition model for each shape category of traffic signs, and obtain several traffic sign sample images for each shape category; first preprocess all sample images, including unified images Size and contrast enhancement; the preprocessed sample images of each shape category are used to train its recognition model. After each training, the preprocessed traffic sign sample images are randomly sorted and then grouped, and Q kinds of distortion methods are set. For each A group of sample images randomly selects q kinds of distortion methods among Q kinds of distortion methods to distort the sample images in a random order, and uses the new sample images after the distortion processing to train its recognition model until the training end condition is reached;

S5：遍历待检测的交通图像中的各个像素，根据每个代表颜色的概率查找表，计算每个像素点属于该类颜色的概率，得到待检测交通图像在每个代表颜色下的概率图，然后转换为灰度图；利用MSER算法检测各灰度图中的灰度变化稳定区域，去除稳定区域中高宽比在预设高宽比范围以外的区域，剩余的稳定区域作为候选窗口；S5: traverse each pixel in the traffic image to be detected, calculate the probability of each pixel point belonging to the color according to the probability lookup table of each representative color, and obtain the probability map of the traffic image to be detected under each representative color, Then convert to a grayscale image; use the MSER algorithm to detect the grayscale change stable area in each grayscale image, remove the area with an aspect ratio outside the preset aspect ratio range in the stable area, and use the remaining stable area as a candidate window;

S6：将候选窗口尺寸调整至形状分类器输入尺寸，提取每个候选窗口对应图像块的HOG特征，输入各个交通标志形状分类器，判断该候选窗口是否为交通标志的形状类别，如果是，则说明该候选窗口存在交通标志，交通标志的形状即为对应形状分类器所进行判定的形状，否则不存在交通标志；S6: Adjust the size of the candidate window to the input size of the shape classifier, extract the HOG feature of the image block corresponding to each candidate window, input each traffic sign shape classifier, and judge whether the candidate window is the shape category of the traffic sign, if yes, then Indicates that there is a traffic sign in the candidate window, and the shape of the traffic sign is the shape determined by the corresponding shape classifier, otherwise there is no traffic sign;

S7：对于步骤S6判断结果为存在交通标志的候选窗口，提取对应图像，调整至识别模型的输入图像尺寸，并按照步骤S6判定的交通标志形状，将尺寸调整后的交通标志图像输入对应形状的识别模型，得到识别结果。S7: For the candidate window where the judgment result of step S6 is that there is a traffic sign, extract the corresponding image, adjust it to the input image size of the recognition model, and input the traffic sign image after size adjustment into the corresponding shape according to the traffic sign shape determined in step S6 Identify the model and get the identification result.

本发明多特征协同的交通标志检测与识别方法，先通过交通标志样本图像建立各个颜色类别交通标志的颜色概率模型，选择各个颜色类别交通标志的代表颜色，计算得到各个代表颜色的概率查找表，同时训练得到各个形状类别交通标志的形状分类器以及识别模型，对于待检测交通图像，先使用各个代表颜色的概率查找表得到待检测交通图像的概率图，然后转换为灰度图，利用MSER算法检测各灰度图中的灰度变化稳定区域，将符合预设高宽比范围的区域作为候选窗口，将各个候选窗口采用各个交通标志形状分类器判定是否存在交通标志，对于存在交通标志的候选窗口，采用对应形状的识别模型进行交通标志识别。The multi-feature collaborative traffic sign detection and recognition method of the present invention first establishes the color probability model of the traffic sign of each color category through the traffic sign sample image, selects the representative color of the traffic sign of each color category, and calculates the probability lookup table of each representative color, At the same time, the shape classifier and recognition model of traffic signs of each shape category are trained. For the traffic image to be detected, the probability lookup table of each representative color is used to obtain the probability map of the traffic image to be detected, and then converted into a grayscale image, using the MSER algorithm Detect the gray-scale change stable area in each gray-scale image, and use the area that meets the preset aspect ratio range as the candidate window, and use each traffic sign shape classifier to determine whether there is a traffic sign in each candidate window. For the candidates with traffic signs window, using the corresponding shape recognition model for traffic sign recognition.

本发明在交通标志的检测与识别过程中，采用了颜色和形状特征两种特征的结合来协同完成检测，从而提高了检测效率，提高了交通标志的识别效果。In the detection and recognition process of the traffic sign, the present invention adopts the combination of the color and the shape feature to cooperate to complete the detection, thereby improving the detection efficiency and the recognition effect of the traffic sign.

附图说明Description of drawings

图1是本发明多特征协同的交通标志检测与识别方法的具体实施方式流程图；Fig. 1 is the flow chart of the specific embodiment of the multi-feature collaborative traffic sign detection and recognition method of the present invention;

图2是禁令标志的示例图；Figure 2 is an example of a prohibition sign;

图3是交通标志样本图像聚类结果示例图；Fig. 3 is an example diagram of traffic sign sample image clustering results;

图4是畸变效果示例图；Figure 4 is an example diagram of the distortion effect;

图5是卷积神经网络结构图；Fig. 5 is a convolutional neural network structure diagram;

图6是第一卷积层的卷积核及卷积结果；Fig. 6 is the convolution kernel and the convolution result of the first convolutional layer;

图7是第二卷积层的卷积核及卷积结果；Fig. 7 is the convolution kernel and the convolution result of the second convolutional layer;

图8是待检测交通图像；Figure 8 is a traffic image to be detected;

图9是待检测交通图像根据红色概率查找表得到的概率图；Fig. 9 is the probability map that the traffic image to be detected obtains according to the red probability lookup table;

图10是图9所示概率图对应的灰度图；Fig. 10 is the grayscale map corresponding to the probability map shown in Fig. 9;

图11是MESE算法检测得到的灰度变化稳定区域结果图；Fig. 11 is the result map of the stable area of gray scale change detected by the MESE algorithm;

图12是待检测交通图像的候选窗口图；Fig. 12 is a candidate window diagram of traffic images to be detected;

图13是待检测交通图像的交通标志检测结果图；Fig. 13 is a traffic sign detection result diagram of a traffic image to be detected;

图14是直线型卷积神经网络(网络1)与多列型卷积神经网络(网络2)的误识别率对比图。Fig. 14 is a comparison chart of the misrecognition rate of the linear convolutional neural network (network 1) and the multi-column convolutional neural network (network 2).

具体实施方式detailed description

下面结合附图对本发明的具体实施方式进行描述，以便本领域的技术人员更好地理解本发明。需要特别提醒注意的是，在以下的描述中，当已知功能和设计的详细描述也许会淡化本发明的主要内容时，这些描述在这里将被忽略。Specific embodiments of the present invention will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

实施例Example

图1是本发明多特征协同的交通标志检测与识别方法的具体实施方式流程图。如图1所示，本发明多特征协同的交通标志检测与识别方法包括以下步骤：Fig. 1 is a flow chart of a specific embodiment of the multi-feature collaborative traffic sign detection and recognition method of the present invention. As shown in Figure 1, the multi-feature collaborative traffic sign detection and recognition method of the present invention includes the following steps:

S101：建立颜色概率模型：S101: Establish a color probability model:

根据交通标志的颜色特点对交通标志进行分类，每一类分别获取若干张交通标志样本图像。对于每张交通标志样本图像，提取各个像素点的颜色特征，根据颜色特征对该交通标志样本图像的所有像素进行聚类，聚类数量为N+1，N为交通标志的主要颜色数量，也就是说交通标志的各个主要颜色分别为一类，剩下一类为背景色，聚类方法可以根据需要进行选择。将每类交通标志样本图像中各样本图像中对应聚类的像素点合并，得到该分类的N+1个样本集，对每个样本集建立对应的高斯模型，并且计算每个颜色类别下各样本集的先验概率。The traffic signs are classified according to their color characteristics, and several traffic sign sample images are obtained for each category. For each traffic sign sample image, the color feature of each pixel is extracted, and all pixels of the traffic sign sample image are clustered according to the color feature. The number of clusters is N+1, and N is the number of main colors of the traffic sign. That is to say, the main colors of the traffic signs are divided into one category, and the remaining category is the background color, and the clustering method can be selected according to the needs. Merge the corresponding clustered pixel points in each sample image of each type of traffic sign sample image to obtain N+1 sample sets of this classification, establish a corresponding Gaussian model for each sample set, and calculate each color category under each color category. The prior probability of the sample set.

对于交通标志图像来说，其颜色主要由N+1中颜色构成(前N种为构成交通标志的主要颜色，第N+1种为背景)，例如除开背景色之后，警告标志的主要颜色为黄色和黑色，禁令标志的主要颜色为红色、黑色和白色，指示标志的主要颜色为蓝色和白色。图2是禁令标志的示例图。图2中的禁令标志均为限速标志。因此交通标志的颜色特点还是较为明显的。For the traffic sign image, its color is mainly composed of N+1 colors (the first N is the main color of the traffic sign, and the N+1 is the background). For example, after removing the background color, the main color of the warning sign is Yellow and black, the main colors of the prohibition signs are red, black and white, and the main colors of the instruction signs are blue and white. Figure 2 is an example diagram of a prohibition sign. The prohibition signs in Figure 2 are all speed limit signs. Therefore, the color characteristics of traffic signs are more obvious.

在建立颜色概率模型时，可以根据需要选择颜色特征。由于自然场景下的交通标志受到不同条件(背光、强烈光照等)的光照影响，为了使颜色概率模型更为准确，本实施例中提出了一种采用颜色不变量(colorinvariance)作为颜色特征。When building a color probability model, color features can be selected as needed. Since traffic signs in natural scenes are affected by different lighting conditions (backlight, strong lighting, etc.), in order to make the color probability model more accurate, this embodiment proposes a color invariance (color invariance) as a color feature.

根据Geosebmek等人建立的高斯光谱模型，彩色图像各像素值(R,G,B)和高斯彩色模型的参数(E,E_λ,E_λλ)的关系为：According to the Gaussian spectral model established by Geosebmek et al., the relationship between each pixel value (R, G, B) of the color image and the parameters (E, E_λ , E_λλ ) of the Gaussian color model is:

$\left[\begin{array}{c}\mathrm{<span><span>E</span>E.</span>}\\ {\mathrm{<span><span>E</span>E.</span>}}_{\mathrm{<span><span>\&lambda;</span>\&lambda;</span>}}\\ {\mathrm{<span><span>E</span>E.</span>}}_{\mathrm{<span><span>\&lambda;</span>\&lambda;</span>}\mathrm{<span><span>\&lambda;</span>\&lambda;</span>}}\end{array}\right]<span><span>=</span>=</span>\mathrm{<span><span>H</span>h</span>}\left[\begin{array}{c}\mathrm{<span><span>R</span>R</span>}\\ \mathrm{<span><span>G</span>G</span>}\\ \mathrm{<span><span>B</span>B</span>}\end{array}\right]$

其中H为3×3的系数矩阵，近似为where H is a 3×3 coefficient matrix, approximately

然后计算得到C_λ和C_λλ是色彩衡量，与视角、光照强度、表面朝向、照明方向无关。因此本实施例中优选色彩衡量C_λ和C_λλ作为各个像素的颜色特征。Then calculate C_λ and C_λλ are color measurements independent of viewing angle, light intensity, surface orientation, and lighting direction. Therefore, in this embodiment, the color metrics C_λ and C_λλ are preferably used as the color features of each pixel.

每个颜色类别下各样本集的先验概率的计算公式为：The formula for calculating the prior probability of each sample set under each color category is:

$\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>)</span>)</span><span><span>=</span>=</span>\frac{{\mathrm{<span><span>n</span>no</span>}}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}}}{\mathrm{<span><span>n</span>no</span>}}$

其中，c_i表示第i个样本集，i＝1,2,…,N+1，表示属于c_i的像素点个数，n表示颜色类别下所有样本像素点的个数。Among them, c_i represents the i-th sample set, i=1,2,...,N+1, Indicates the number of pixels belonging to c_i , and n indicates the number of all sample pixels under the color category.

图3是交通标志样本图像聚类结果示例图。如图3所示，该交通标志样本图像为禁令标志，其主要颜色有三种，因此令聚类数量为4，采用kmeans聚类方法进行聚类。可见，该交通标志样本图像中的像素点被很好地分为了4个类。对所有的红色禁令交通标志样本采用同样的方式进行聚类，将对应聚类的像素点进行合并，得到4种颜色的样本集。然后对每个样本集建立高斯模型。通过对样本集元素的分析，为了提高模型的鲁棒性，本实施例中对红色、黑色、背景样本集分别建立混合高斯模型，对白色样本集建立单高斯模型。Figure 3 is an example diagram of the clustering results of traffic sign sample images. As shown in Figure 3, the traffic sign sample image is a prohibition sign with three main colors, so the number of clusters is set to 4, and the kmeans clustering method is used for clustering. It can be seen that the pixels in the traffic sign sample image are well divided into 4 classes. All red prohibition traffic sign samples are clustered in the same way, and the corresponding clustered pixels are merged to obtain a sample set of 4 colors. Then build a Gaussian model for each sample set. Through the analysis of the elements of the sample set, in order to improve the robustness of the model, in this embodiment, a mixed Gaussian model is established for the red, black, and background sample sets, and a single Gaussian model is established for the white sample set.

S102：计算交通标志颜色概率查找表：S102: Calculating the traffic sign color probability lookup table:

对于每个颜色类别的交通标志，根据其对应的N+1个的颜色概率模型计算各个R,G,B值属于各颜色的概率p(c_i|x)，x表示像素点R,G,B值，c_i表示颜色，i＝1,2,…,N+1。由于各个颜色类别的交通标志，可以选择出一个代表颜色，即可以将该颜色类别与另一个颜色类别完全区分的颜色，例如警告标志的代表颜色为黄色，禁令标志的代表颜色为红色，指示标志的代表颜色为蓝色，因此从其N种主要颜色中选取一种颜色作为代表颜色，然后对代表颜色进行归一化得到其归一化概率，建立各个R,G,B值属于代表颜色的概率查找表。For traffic signs of each color category, calculate the probability p(c_i |x) that each R, G, and B value belongs to each color according to its corresponding N+1 color probability model, x represents the pixel point R, G, B value, c_i represents color, i=1,2,...,N+1. Due to the traffic signs of each color category, a representative color can be selected, that is, the color that can completely distinguish the color category from another color category, for example, the representative color of warning signs is yellow, the representative color of prohibition signs is red, and the representative color of indicator signs The representative color is blue, so one color is selected from its N main colors as the representative color, and then the representative color is normalized to obtain its normalized probability, and each R, G, B value belongs to the representative color. Probability lookup table.

对于像素点x＝(r,g,b)，其属于颜色c_i的概率为：For a pixel point x=(r,g,b), the probability that it belongs to color c_i is:

$\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>|</span>|</span>\mathrm{<span><span>x</span>x</span>}<span><span>)</span>)</span><span><span>=</span>=</span>\frac{\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>|</span>|</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>)</span>)</span>\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>)</span>)</span>}{\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>)</span>)</span>}$

在一张具体的图片中，p(x)是确定的，所以可以认为：In a specific picture, p(x) is determined, so it can be considered that:

p(c_i|x)≈p(x|c_i)p(c_i)p(c_i |x)≈p(x|c_i )p(c_i )

又由上一步得到的颜色概率模型，可以直接得到p(x|c_i)：From the color probability model obtained in the previous step, p(x|_ci ) can be obtained directly:

$\mathrm{<span><span>N</span>N</span>}<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>;</span>;</span>{\mathrm{<span><span>u</span>u</span>}}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}<span><span>,</span>,</span>{<span><span>\&Sigma;</span>\&Sigma;</span>}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}<span><span>)</span>)</span><span><span>=</span>=</span>\frac{\mathrm{<span><span>1</span>1</span>}}{\sqrt{\mathrm{<span><span>2</span>2</span>}\mathrm{<span><span>\&pi;</span>\&pi;</span>}{<span><span>\&Sigma;</span>\&Sigma;</span>}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}}}{\mathrm{<span><span>exp</span>exp</span>}}^{<span><span>-</span>-</span>\frac{\mathrm{<span><span>1</span>1</span>}}{\mathrm{<span><span>2</span>2</span>}}<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>-</span>-</span>{\mathrm{<span><span>u</span>u</span>}}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}<span><span>)</span>)</span>{{<span><span>\&Sigma;</span>\&Sigma;</span>}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}}^{<span><span>-</span>-</span>\mathrm{<span><span>1</span>1</span>}}{<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>-</span>-</span>{\mathrm{<span><span>u</span>u</span>}}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}<span><span>)</span>)</span>}^{\mathrm{<span><span>T</span>T</span>}}}$

$\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>|</span>|</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>)</span>)</span><span><span>=</span>=</span>\underset{\mathrm{<span><span>j</span>j</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>K</span>K</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>\&lambda;</span>\&lambda;</span>}}_{\mathrm{<span><span>k</span>k</span>}}\mathrm{<span><span>N</span>N</span>}<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>,</span>,</span>{\mathrm{<span><span>u</span>u</span>}}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}<span><span>,</span>,</span>{<span><span>\&Sigma;</span>\&Sigma;</span>}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}<span><span>)</span>)</span>$

式中，表示颜色c_i的高斯模型中第j个高斯模型的均值，表示颜色c_i高斯模型中第j个高斯模型的协方差，K表示颜色c_i的高斯模型数量。In the formula, Indicates the mean value of the jth Gaussian model in the Gaussian model of color c_i , Indicates the covariance of the jth Gaussian model in the Gaussian model of color c_i , and K indicates the number of Gaussian models of color c_i .

由此可以确定p(c_i|x)。From this p(_ci |x) can be determined.

$\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>|</span>|</span>\mathrm{<span><span>x</span>x</span>}<span><span>)</span>)</span><span><span>=</span>=</span>\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>|</span>|</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>)</span>)</span>\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>)</span>)</span><span><span>=</span>=</span>\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}<span><span>)</span>)</span>\underset{\mathrm{<span><span>j</span>j</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>K</span>K</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}{\mathrm{<span><span>\&lambda;</span>\&lambda;</span>}}_{\mathrm{<span><span>k</span>k</span>}}\mathrm{<span><span>N</span>N</span>}<span><span>(</span>(</span>\mathrm{<span><span>x</span>x</span>}<span><span>,</span>,</span>{\mathrm{<span><span>u</span>u</span>}}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}<span><span>,</span>,</span>{<span><span>\&Sigma;</span>\&Sigma;</span>}_{{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}}<span><span>)</span>)</span>$

记代表颜色为c_i′，其归一化后的概率为：Denote the representative color as c_i ′, and its normalized probability is:

$\stackrel{<span><span>~</span>~</span>}{\mathrm{<span><span>p</span>p</span>}}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}^{<span><span>\&prime;</span>\&prime;</span>}<span><span>|</span>|</span>\mathrm{<span><span>x</span>x</span>}<span><span>)</span>)</span><span><span>=</span>=</span>\frac{\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{\mathrm{<span><span>i</span>i</span>}}^{<span><span>\&prime;</span>\&prime;</span>}<span><span>|</span>|</span>\mathrm{<span><span>x</span>x</span>}<span><span>)</span>)</span>}{\underset{{\mathrm{<span><span>i</span>i</span>}}^{<span><span>\&prime;</span>\&prime;</span>}<span><span>=</span>=</span>\mathrm{<span><span>1</span>1</span>}}{\overset{\mathrm{<span><span>N</span>N</span>}<span><span>+</span>+</span>\mathrm{<span><span>1</span>1</span>}}{<span><span>\&Sigma;</span>\&Sigma;</span>}}\mathrm{<span><span>p</span>p</span>}<span><span>(</span>(</span>{\mathrm{<span><span>c</span>c</span>}}_{{\mathrm{<span><span>i</span>i</span>}}^{<span><span>\&prime;</span>\&prime;</span>}}<span><span>|</span>|</span>\mathrm{<span><span>x</span>x</span>}<span><span>)</span>)</span>}$

由此，就可以计算出每一个像素点属于代表颜色c_i′的概率。为加快检测速度，本发明根据颜色概率模型，计算每个颜色类别的代表颜色的离线概率查找表。代表颜色就是可以将该颜色类别与其他颜色类别区分开的颜色。以红色禁令交通标志为例，针对其红色特征比较明显的特点，计算红色概率查找表，含有256^3个元素，这样对于任意一个像素点，都可以根据其R,G,B值直接查表得到其属于红色的概率。Thus, the probability that each pixel belongs to the representative color c_i ′ can be calculated. In order to speed up the detection speed, the present invention calculates the off-line probability lookup table of the representative color of each color category according to the color probability model. A representative color is one that distinguishes the color class from other color classes. Taking the red prohibition traffic sign as an example, according to its obvious red feature, calculate the red probability lookup table, which contains 256^3 elements, so that for any pixel point, it can be directly looked up according to its R, G, B values Get the probability that it belongs to red.

S103：训练交通标志形状分类器：S103: Training traffic sign shape classifier:

将交通标志根据形状分为M类，每个形状类别建立一个基于HOG((HistogramofOrientedGradient，方向梯度直方图)特征的形状分类器，其训练方法为：对于每个形状分类器，采集两类样本图像，一类为对应形状的交通标志样本图像，另一类为其他图像，包括其他形状的交通标志样本图像或其他非交通标志的样本图像。统一样本图像尺寸，本实施例中将所有的样本图像归一化到20*20的大小，提取每张样本图像的HOG特征，将HOG特征作为形状分类器的输入，是否为所属类别的判定值作为形状分类器的输出，训练得到对应形状类别交通标志的形状分类器。本实施例中将交通标志按照形状分为圆形和三角形两类，所采用的分类器为SVM(SupportVectorMachine，支持向量机)分类器。The traffic signs are divided into M categories according to their shapes, and a shape classifier based on HOG ((Histogram of Oriented Gradient, histogram of oriented gradient) feature is established for each shape category. The training method is as follows: for each shape classifier, two types of sample images are collected , one class is a traffic sign sample image of the corresponding shape, and the other is other images, including traffic sign sample images of other shapes or other non-traffic sign sample images.Unify the sample image size, and in this embodiment, all sample images Normalize to the size of 20*20, extract the HOG feature of each sample image, use the HOG feature as the input of the shape classifier, and take the judgment value of whether it belongs to the category as the output of the shape classifier, and train to obtain the traffic sign corresponding to the shape category The shape classifier of. In the present embodiment, traffic sign is divided into two classes of circle and triangle according to shape, and the classifier adopted is SVM (SupportVectorMachine, support vector machine) classifier.

S104：训练交通标志识别模型：S104: Train the traffic sign recognition model:

为了完成交通标志的识别，需要预先训练好识别模型。为了提高识别模型的识别准确率，本发明按照交通标志的形状类别数M，对每一个形状类别的交通标志分别设置一个识别模型，每个形状类别分别获取若干张交通标志样本图像。先对所有样本图像进行预处理，包括统一图像尺寸和对比度增强。统一图像尺寸是为了避免样本尺寸大小对识别模型的训练效果产生影响。本实施例中采用限制对比度自适应直方图均衡(CLAHE)算法进行对比度增强。限制对比度自适应直方图均衡算法相对比传统的对比度自适应直方图均衡(AHE)增强图像，在增强过程中不会过度放大噪声，可以达到较为理想的处理效果。In order to complete the recognition of traffic signs, the recognition model needs to be trained in advance. In order to improve the recognition accuracy of the recognition model, according to the number M of traffic sign shape categories, the present invention sets a recognition model for each shape category of traffic signs, and obtains several traffic sign sample images for each shape category. All sample images are first preprocessed, including uniform image size and contrast enhancement. The purpose of unifying the image size is to avoid the impact of the sample size on the training effect of the recognition model. In this embodiment, a Contrast Constrained Adaptive Histogram Equalization (CLAHE) algorithm is used for contrast enhancement. Compared with the traditional contrast adaptive histogram equalization (AHE) enhanced image, the limited contrast adaptive histogram equalization algorithm will not over-amplify the noise during the enhancement process, and can achieve a more ideal processing effect.

此外，为了保证识别模型的训练效果并增加识别的鲁棒性，对于每个形状类别，每次训练完毕后将预处理后的交通标志样本图像随机排序然后分组，设置Q种畸变方式，对每组样本图像在Q种畸变方式中随机选择q种畸变方式按照随机顺序对样本图像进行畸变处理，采用畸变处理后的新样本图像对其识别模型进行训练，直到达到训练结束条件。训练结束条件即为预先设置的识别模型所要达到的要求，一般是输出误差，如果满足要求即训练结束。In addition, in order to ensure the training effect of the recognition model and increase the robustness of recognition, for each shape category, after each training, the preprocessed traffic sign sample images are randomly sorted and grouped, and Q kinds of distortion methods are set. A group of sample images randomly selects q kinds of distortion methods among Q kinds of distortion methods to distort the sample images in a random order, and uses the new sample images after distortion processing to train its recognition model until the training end condition is reached. The training end condition is the requirement to be met by the pre-set recognition model, generally the output error, if the requirement is met, the training ends.

本实施例中畸变方式包括添加白噪声、平移、仿射变换、旋转四种。图4是畸变效果示例图。在畸变时，可以随机选择几种畸变方式，以随机顺序进行畸变，例如选择平移和旋转两种方式，先对样本图像进行旋转，然后再进行平衡。通过畸变处理，可以大量增加样本数量与类型，可以使得最后训练得到的交通标志识别模型更加准确，从而提升交通标志识别率。Distortion methods in this embodiment include adding white noise, translation, affine transformation, and rotation. Figure 4 is an example diagram of the distortion effect. When distorting, several distortion methods can be randomly selected and distorted in a random order, for example, two methods of translation and rotation are selected, and the sample image is first rotated and then balanced. Through distortion processing, the number and types of samples can be greatly increased, which can make the final trained traffic sign recognition model more accurate, thereby improving the traffic sign recognition rate.

现有的识别模型有多种，本发明中采用卷积神经网络。为了提高交通标志的识别率，本实施例中重新设计了一种卷积神经网络。图5是卷积神经网络结构图。如图5所示，本实施例中的卷积神经网络的第一层为输入层，作为第一阶段--低层特征提取的输入，包括第一卷积层、第一最大池化层和第一局部归一化层，然后是第二阶段高层特征提取：第二卷积层、第二最大池化层和第二局部归一化层；再将第二局部归一化层的输出结果输入到第一局部卷积层和第二局部卷积层组成的卷积网络中；在全连接层，本发明不仅将第二局部卷积层作为全连接层的输入来训练交通标志识别模型，还与更前面层的输出(即第一局部卷积层)相结合，第一局部卷积层和第二局部卷积层具有不同层次的特征表达，所以将这两者相结合，同时输入给全连接层，作为网络最终的特征描述；最后，为了获得待识别对象属于某类别的概率数据，使用SoftMax回归作为输出层。There are many kinds of existing recognition models, and the convolutional neural network is adopted in the present invention. In order to improve the recognition rate of traffic signs, a convolutional neural network is redesigned in this embodiment. Figure 5 is a structural diagram of a convolutional neural network. As shown in Figure 5, the first layer of the convolutional neural network in this embodiment is the input layer, as the first stage - the input of low-level feature extraction, including the first convolutional layer, the first maximum pooling layer and the second A local normalization layer, followed by the second stage of high-level feature extraction: the second convolutional layer, the second maximum pooling layer, and the second local normalization layer; then the output of the second local normalization layer is input In the convolutional network composed of the first local convolutional layer and the second local convolutional layer; in the fully connected layer, the present invention not only uses the second local convolutional layer as the input of the fully connected layer to train the traffic sign recognition model, but also Combined with the output of the previous layer (that is, the first local convolutional layer), the first local convolutional layer and the second local convolutional layer have different levels of feature expression, so the two are combined and input to the full The connection layer is used as the final feature description of the network; finally, in order to obtain the probability data that the object to be identified belongs to a certain category, SoftMax regression is used as the output layer.

卷积神经网络的工作过程如下：The working process of convolutional neural network is as follows:

1)第一卷积层：1) The first convolutional layer:

第一卷积层对输入图像进行卷积。本实施例中交通标志样本图像的尺寸都统一为32×32，将3通道彩色交通标志图像作为输入图像，输入训练数据的维度为3072维。第一卷积层采用64个大小为5×5的不同卷积核进行卷积，得到卷积层，其输出即为特征图，输出至第一最大池化层。图6是第一卷积层的卷积核及卷积结果。如图6所示，本实施例中第一卷积层得到的特征图总共64个，大小为28×28，本实施例中采用的计算公式为：The first convolutional layer performs convolution on the input image. In this embodiment, the size of the traffic sign sample images is uniformly 32×32, and the 3-channel color traffic sign image is used as the input image, and the dimension of the input training data is 3072 dimensions. The first convolution layer uses 64 different convolution kernels with a size of 5×5 to perform convolution to obtain a convolution layer, and its output is the feature map, which is output to the first maximum pooling layer. Fig. 6 is the convolution kernel and convolution result of the first convolutional layer. As shown in Figure 6, there are a total of 64 feature maps obtained by the first convolutional layer in this embodiment, with a size of 28×28. The calculation formula used in this embodiment is:

${\mathrm{<span><span>y</span>the\; y</span>}}_{\mathrm{<span><span>2</span>2</span>}}^{\mathrm{<span><span>r</span>r</span>}}<span><span>=</span>=</span>\mathrm{<span><span>f</span>f</span>}<span><span>(</span>(</span><span><span>\&Sigma;</span>\&Sigma;</span>{\mathrm{<span><span>y</span>the\; y</span>}}_{\mathrm{<span><span>1</span>1</span>}}<span><span>*</span>*</span>{\mathrm{<span><span>k</span>k</span>}}_{\mathrm{<span><span>i</span>i</span>}\mathrm{<span><span>j</span>j</span>}}^{\mathrm{<span><span>r</span>r</span>}}<span><span>+</span>+</span>{\mathrm{<span><span>b</span>b</span>}}_{\mathrm{<span><span>j</span>j</span>}}^{\mathrm{<span><span>r</span>r</span>}}<span><span>)</span>)</span>$

其中，y₁为输入的交通标志样本图像，为经卷积后得到的第r张特征图，为第r个卷积核，为相应的偏置，与是需要训练的参数，f(·)为sigmoid激活函数。Among them,_y1 is the input traffic sign sample image, is the rth feature map obtained after convolution, is the rth convolution kernel, For the corresponding bias, and is the parameter that needs to be trained, and f(·) is the sigmoid activation function.

2)第一最大池化层2) The first maximum pooling layer

第一最大池化层对步骤1)所得64张特征图进行最大池化。本实施例中池化方式选择最大池化，本实施例中池化核大小为2×2，得到的池化层大小为14×14，计算公式如下：The first maximum pooling layer performs maximum pooling on the 64 feature maps obtained in step 1). In this embodiment, the pooling method selects maximum pooling. In this embodiment, the pooling kernel size is 2×2, and the obtained pooling layer size is 14×14. The calculation formula is as follows:

${\mathrm{<span><span>y</span>the\; y</span>}}_{\mathrm{<span><span>3</span>3</span>}}^{\mathrm{<span><span>r</span>r</span>}}<span><span>=</span>=</span>\mathrm{<span><span>f</span>f</span>}<span><span>(</span>(</span>{\mathrm{<span><span>\&beta;</span>\&beta;</span>}}_{\mathrm{<span><span>2</span>2</span>}}^{\mathrm{<span><span>r</span>r</span>}}\mathrm{<span><span>d</span>d</span>}\mathrm{<span><span>o</span>o</span>}\mathrm{<span><span>w</span>w</span>}\mathrm{<span><span>n</span>no</span>}<span><span>(</span>(</span>{\mathrm{<span><span>y</span>the\; y</span>}}_{\mathrm{<span><span>2</span>2</span>}}^{\mathrm{<span><span>r</span>r</span>}}<span><span>)</span>)</span><span><span>+</span>+</span>{\mathrm{<span><span>b</span>b</span>}}_{\mathrm{<span><span>2</span>2</span>}}^{\mathrm{<span><span>r</span>r</span>}}<span><span>)</span>)</span>$

其中，是池化后所得图像，为卷积层1后所得的第r个特征图，down(·)为一个下采样函数，和分别为乘性偏置和加性偏置，是需要训练的参数。in, is the image obtained after pooling, is the rth feature map obtained after convolutional layer 1, down( ) is a downsampling function, and Respectively, the multiplicative bias and the additive bias are parameters that need to be trained.

3)第一局部归一化层3) The first local normalization layer

第一局部归一化层将2)得到的池化层进行局部归一化。第一局部响应归一化层完成一种“临近抑制”操作，对局部输入区域进行归一化。归一化公式如下：The first local normalization layer performs local normalization on the pooling layer obtained in 2). The first local response normalization layer performs a kind of "proximity suppression" operation to normalize the local input region. The normalization formula is as follows:

${\mathrm{<span><span>y</span>the\; y</span>}}_{\mathrm{<span><span>4</span>4</span>}}<span><span>=</span>=</span>{<span><span>(</span>(</span>\mathrm{<span><span>1</span>1</span>}<span><span>+</span>+</span><span><span>(</span>(</span>\mathrm{<span><span>\&alpha;</span>\&alpha;</span>}<span><span>/</span>/</span>\mathrm{<span><span>\&lambda;</span>\&lambda;</span>}<span><span>)</span>)</span><span><span>\&Sigma;</span>\&Sigma;</span>{\mathrm{<span><span>x</span>x</span>}}_{\mathrm{<span><span>i</span>i</span>}}^{\mathrm{<span><span>2</span>2</span>}}<span><span>)</span>)</span>}^{\mathrm{<span><span>\&beta;</span>\&beta;</span>}}$

其中，y₄为归一化后所得图像，λ为局部尺寸大小，α为归一化缩放因子，β为指数项，本实施例中α和β分别选1和5。Wherein, y₄ is the image obtained after normalization, λ is the local size, α is the normalized scaling factor, and β is the exponent item. In this embodiment, α and β are selected as 1 and 5 respectively.

4)第二阶段高层特征提取。4) The second stage of high-level feature extraction.

第二阶段高层所涉及到的第二卷积层、第二局部归一化层、第二最大池化层的公式如前面所述，只是此时不再是如第一卷积层以输入图像为基础而是在第一阶段输出的基础上进行。图7是第二卷积层的卷积核及卷积结果。The formulas of the second convolutional layer, the second local normalization layer, and the second maximum pooling layer involved in the second stage of the high-level layer are as described above, but at this time it is no longer the input image as the first convolutional layer It is based on the output of the first stage. Fig. 7 is the convolution kernel and convolution result of the second convolution layer.

5)第一局部卷积层和第二局部卷积层5) The first local convolution layer and the second local convolution layer

第一局部卷积层接收池化层进行局部卷积，将得到的局部卷积层分别发送给第二局部卷积层和全连接层；第二局部卷积层接收局部卷积层继续进行局部卷积，将得到的局部卷积层输出至会连接层。通过第一局部卷积层和第二局部卷积层进一步提取特征。The first local convolution layer receives the pooling layer for local convolution, and sends the obtained local convolution layer to the second local convolution layer and the fully connected layer respectively; the second local convolution layer receives the local convolution layer and continues to perform local convolution. Convolution, output the resulting local convolutional layer to the concatenated layer. Features are further extracted through the first local convolutional layer and the second local convolutional layer.

6)全连接层6) Fully connected layer

在多列型卷积神经网络中，全连接层的输入既包括了第二局部卷积层的输出，也包括了第一局部卷积层的输出。全连接层融合第一局部卷积层和第二局部卷积层的局部卷积层，将融合结果输出至Softmax层。通过这种方式，全连接层输入的信息是多层次的，既包括了该更高层次的抽象特征，也包括了较低层的特征输入，获得了更好的交通标志识别效果。In a multi-column convolutional neural network, the input of the fully connected layer includes both the output of the second local convolutional layer and the output of the first local convolutional layer. The fully connected layer fuses the first local convolution layer and the local convolution layer of the second local convolution layer, and outputs the fusion result to the Softmax layer. In this way, the input information of the fully connected layer is multi-level, including both the higher-level abstract features and the lower-level feature inputs, and better traffic sign recognition results are obtained.

6)Softmax层。6) Softmax layer.

使用卷积、将采样方式获取的数据，通过全连接层可以获得一个卷积神经网络下的特征描述，然后通过Softmax层将全连接层输出的融合结果(即网络提取的特征)采用SoftMax回归进行分类。输出大小根据训练时确定的子类数目决定，比如三角形识别模型就有15种类型，那么该层的最大输出就为15。Using the data obtained by convolution and sampling, the feature description under a convolutional neural network can be obtained through the fully connected layer, and then the fusion result output by the fully connected layer (that is, the features extracted by the network) is used for SoftMax regression through the Softmax layer. Classification. The output size is determined according to the number of subclasses determined during training. For example, there are 15 types of triangle recognition models, so the maximum output of this layer is 15.

以上四个步骤完成了交通标志检测与识别的准备工作，接下来由进入实际的检测和识别。The above four steps complete the preparatory work for traffic sign detection and recognition, and then enter the actual detection and recognition.

S105：交通图像预处理：S105: traffic image preprocessing:

遍历待检测的交通图像中的各个像素，根据每个颜色类别的代表颜色的概率查找表，计算每个像素点属于该类颜色的概率，得到待检测交通图像在每个代表颜色下的概率图，然后转换为灰度图。也就是说，待检测的交通图像会得到多副灰度图，每个代表颜色对应一副。图8是待检测交通图像。图9是待检测交通图像根据红色概率查找表得到的概率图。图10是图9所示概率图对应的灰度图。根据图8至图10可以看出，图中红色的区域概率较高，其在灰度图中对应的区域较亮，其他区域概率较低，其在灰度图中对应的区域较暗。Traverse each pixel in the traffic image to be detected, calculate the probability of each pixel belonging to the color according to the probability lookup table of the representative color of each color category, and obtain the probability map of the traffic image to be detected under each representative color , and then converted to grayscale. That is to say, the traffic image to be detected will get multiple grayscale images, one for each representative color. Fig. 8 is a traffic image to be detected. FIG. 9 is a probability map obtained from a traffic image to be detected according to a red probability lookup table. FIG. 10 is a grayscale image corresponding to the probability map shown in FIG. 9 . It can be seen from Figures 8 to 10 that the red area in the figure has a higher probability, and its corresponding area in the grayscale image is brighter, while other areas have a lower probability, and its corresponding area in the grayscale image is darker.

利用MSER(MaximallyStableExtremalRegions，最大稳定极值区域)算法检测各个灰度图中的灰度变化稳定区域。经过反复实验，交通标志所在的区域在灰度图中一定是灰度变化稳定的区域。因此在得到灰度变化稳定区域之后，根据交通标志的形状特点设置交通标志的高宽比范围，去除稳定区域中高宽比在所设置的高宽比范围以外的区域，剩余的稳定区域作为候选窗口。本实施例中，交通标志的高宽比范围为[0.6,1.4]。图11是MESE算法检测得到的灰度变化稳定区域结果图。图12是待检测交通图像的候选窗口图。采用以上方法来得到候选窗口，相比传统的滑动窗口检测，可以极大地减少候选窗口数量，加快检测速度。The MSER (MaximallyStableExtremalRegions, maximum stable extremal region) algorithm is used to detect the stable region of grayscale change in each grayscale image. After repeated experiments, the area where the traffic sign is located must be an area with stable grayscale changes in the grayscale image. Therefore, after obtaining the gray-scale change stable area, set the aspect ratio range of the traffic sign according to the shape characteristics of the traffic sign, remove the area whose aspect ratio is outside the set aspect ratio range in the stable area, and use the remaining stable area as a candidate window . In this embodiment, the aspect ratio range of the traffic sign is [0.6, 1.4]. Fig. 11 is the result map of the stable area of gray scale change detected by the MESE algorithm. Fig. 12 is a candidate window diagram of traffic images to be detected. Using the above method to obtain candidate windows, compared with traditional sliding window detection, can greatly reduce the number of candidate windows and speed up detection.

S106：交通标志检测：S106: Traffic sign detection:

将候选窗口尺寸调整至形状分类器输入尺寸，提取每个候选窗口对应图像块的HOG特征，输入步骤S103得到的各个交通标志形状分类器，判断该候选窗口是否为交通标志的形状类别，如果是，则说明该候选窗口存在交通标志，否则不存在。可见，采用这种方式可以同时得到该交通标志的形状。图13是待检测交通图像的交通标志检测结果图。Adjust the size of the candidate window to the input size of the shape classifier, extract the HOG feature of the corresponding image block of each candidate window, input each traffic sign shape classifier obtained in step S103, and judge whether the candidate window is the shape category of the traffic sign, if so , it means that there is a traffic sign in the candidate window, otherwise it does not exist. It can be seen that the shape of the traffic sign can be obtained at the same time in this way. Fig. 13 is a diagram of traffic sign detection results of traffic images to be detected.

S107：交通标志识别：S107: Traffic sign recognition:

将步骤S106提取的交通标志图像调整至识别模型的输入图像尺寸，并按照步骤S106判定的交通标志形状，将尺寸调整后的交通标志图像输入识别模型，得到识别结果。The traffic sign image extracted in step S106 is adjusted to the input image size of the recognition model, and according to the shape of the traffic sign determined in step S106, the size-adjusted traffic sign image is input into the recognition model to obtain a recognition result.

本实施例中识别模型采用多列式卷积神经网络。图14是直线型卷积神经网络(网络1)与多列型卷积神经网络(网络2)的误识别率对比图。如图14所示，多列型卷积神经网络的收敛性更快，要达到同样的误识别率，所需训练样本较少，从而降低识别模型的训练复杂度。In this embodiment, the recognition model adopts a multi-column convolutional neural network. Fig. 14 is a comparison chart of the misrecognition rate of the linear convolutional neural network (network 1) and the multi-column convolutional neural network (network 2). As shown in Figure 14, the convergence of the multi-column convolutional neural network is faster, and to achieve the same misrecognition rate, fewer training samples are required, thereby reducing the training complexity of the recognition model.

从以上步骤可以看出，本发明在检测阶段，利用交通标志的颜色特征建立颜色概率模型进行粗筛选，利用交通标志的形状特点搜索灰度变化稳定区域进行细筛选；在识别阶段，利用交通标志的HOG特征和图像特点进行识别。可见本发明结合了交通标志的颜色和形状特征，采用特征协同的方式来进行交通标志的检测与识别，从而得到更好的检测与识别效果。As can be seen from the above steps, in the detection stage, the present invention utilizes the color features of traffic signs to establish a color probability model for rough screening, and utilizes the shape features of traffic signs to search for areas with stable gray scale changes for fine screening; The HOG features and image features are identified. It can be seen that the present invention combines the color and shape features of the traffic signs, and uses feature synergy to detect and recognize traffic signs, thereby obtaining better detection and recognition effects.

尽管上面对本发明说明性的具体实施方式进行了描述，以便于本技术领域的技术人员理解本发明，但应该清楚，本发明不限于具体实施方式的范围，对本技术领域的普通技术人员来讲，只要各种变化在所附的权利要求限定和确定的本发明的精神和范围内，这些变化是显而易见的，一切利用本发明构思的发明创造均在保护之列。Although the illustrative specific embodiments of the present invention have been described above, so that those skilled in the art can understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, As long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are included in the protection list.

Claims (6)

1. A multi-feature cooperative traffic sign detection and identification method is characterized by comprising the following steps:

s1: classifying the traffic signs according to the color characteristics of the traffic signs, and respectively acquiring a plurality of traffic sign sample images for each color category; for each traffic sign sample image, extracting the color characteristics of each pixel point, clustering all the pixels of the traffic sign sample image according to the color characteristics, wherein the clustering number is N +1, N is the main color number of the traffic sign, merging the pixel points corresponding to clustering in each sample image in each type of traffic sign sample image to obtain N +1 sample sets of the color category, and establishing a corresponding Gaussian model for each sample set;

s2: for each color type of traffic sign, calculating the probability p (c) that each R, G and B value belongs to each color according to the corresponding N +1 color probability models_i| x), x represents the R, G, B value of the pixel point, c_iRepresents color, i ═ 1,2, …, N + 1; selecting one color from the N main colors of the color category as a representative color, and recording the color as c'_iThen, the probability of the representative color is normalized to obtain the normalized probabilityEstablishing a probability lookup table of each R, G and B value belonging to the representative color;

s3: the traffic signs are divided into M types according to shapes, each shape type is established with a shape classifier based on HOG characteristics, and the training method comprises the following steps: for each shape classifier, two types of sample images are obtained, one type is a traffic sign sample image with a corresponding shape, and the other type is other images; unifying the sizes of the sample images, extracting HOG characteristics of each sample image, taking the HOG characteristics as the input of a shape classifier, taking a judgment value of whether the HOG characteristics are of the shape class as the output of the shape classifier, and training to obtain the shape classifier of the traffic sign corresponding to the shape class;

s4: according to the number M of the shape types of the traffic signs, respectively setting an identification model for the traffic signs of each shape type, and respectively acquiring a plurality of traffic sign sample images for each shape type; preprocessing all sample images, including unifying image size and contrast enhancement; training the identification model of each preprocessed shape sample image by adopting each preprocessed shape type, randomly sequencing and grouping the preprocessed traffic sign sample images after each training is finished, setting Q distortion modes, randomly selecting Q distortion modes from the Q distortion modes for each group of sample images, carrying out distortion processing on the sample images according to the random sequence, and training the identification model of each preprocessed shape sample image by adopting a new distorted sample image until the training finishing condition is reached;

s5: traversing each pixel in the traffic image to be detected, calculating the probability of each pixel point belonging to the color according to the probability lookup table of each representative color to obtain a probability map of the traffic image to be detected under each representative color, and converting the probability map into a gray map; detecting a gray change stable region in each gray image by using an MSER algorithm, removing a region of the stable region, the aspect ratio of which is outside a preset aspect ratio range, and taking the remaining stable region as a candidate window;

s6: adjusting the size of the candidate window to the input size of a shape classifier, extracting HOG characteristics of image blocks corresponding to each candidate window, inputting each traffic sign shape classifier, judging whether the candidate window is the shape class of the traffic sign, if so, indicating that the traffic sign exists in the candidate window, and the shape of the traffic sign is the shape judged by the corresponding shape classifier, otherwise, not judging that the traffic sign exists;

s7: for the candidate window in which the traffic sign exists as a result of the determination in step S6, the corresponding image is extracted, the input image size of the recognition model is adjusted, and the traffic sign image after size adjustment is input to the recognition model of the corresponding shape in accordance with the traffic sign shape determined in step S6, so that the recognition result is obtained.

2. The method for detecting and identifying traffic signs according to claim 1, wherein the color characteristics in step S1 are measured by a color scale C_λAnd C_λλThe calculation method comprises the following steps:

recording a certain pixel value of the traffic sign sample image as (R, G, B), and calculating by adopting a Gaussian spectrum model to obtain corresponding parameters (E, E) of the pixel_λ,E_λλ) The calculation formula is as follows:

$\left[\begin{array}{c}E\\ E_{\mathrm{\&lambda;}}\\ E_{\mathrm{\&lambda;}\mathrm{\&lambda;}}\end{array}\right]=H\left[\begin{array}{c}R\\ G\\ B\end{array}\right]$

where H is a 3 x 3 coefficient matrix;

the color metric C corresponding to the pixel_λAnd C_λλThe calculation formula of (2) is as follows:

$C_{\mathrm{\&lambda;}}=\frac{E_{\mathrm{\&lambda;}}}{E},C_{\mathrm{\&lambda;}\mathrm{\&lambda;}}=\frac{E_{\mathrm{\&lambda;}\mathrm{\&lambda;}}}{E}.$

3. the method for detecting and identifying traffic signs according to claim 1, wherein the classifier in the step S3 is an SVM classifier.

4. The method for detecting and identifying traffic signs according to claim 1, wherein the contrast enhancement in step S4 employs a limited contrast adaptive histogram equalization algorithm.

5. The method for detecting, detecting and identifying traffic signs according to claim 1, wherein the identification model in step S4 adopts a multi-column convolutional neural network, which includes a first convolutional layer, a first max-pooling layer, a first local normalization layer, a second convolutional layer, a second local normalization layer, a second max-pooling layer, a first local convolutional layer, a second local convolutional layer, a fully-connected layer, and a Softmax layer, wherein:

the first convolution layer receives the image and carries out convolution, and the obtained convolution layer is output to a first maximum pooling layer; the first maximum pooling layer pools the convolutional layer, and the obtained pooled layer is output to the first local normalization layer; the first local normalization layer performs local normalization on the pooling layer, and outputs the obtained local response normalization layer to the second convolution layer; the second convolution layer receives the local response normalization layer for convolution, and outputs the obtained convolution layer to the second maximum pooling layer; the second largest pooling layer pools the convolutional layer, and the obtained pooled layer is output to the first local convolutional layer; the first local convolution layer receives the pooling layer to perform local convolution, and the obtained local convolution layers are respectively sent to the second local convolution layer and the full-connection layer; the second local convolution layer receives the local convolution layer and continues to carry out local convolution, and the obtained local convolution layer is output to the connection layer; the full-connection layer fuses the local convolution layers of the first local convolution layer and the second local convolution layer and outputs a fusion result to the Softmax layer; and the Softmax layer classifies the fusion result by adopting Softmax regression and outputs a classification result.

6. The method for detecting and identifying traffic signs according to claim 1, wherein the aspect ratio range in step S5 is [0.6,1.4 ].