CN114092389A - A surface defect detection method for glass panels based on small sample learning - Google Patents

Abstract

Translated fromChinese

本发明公开了一种基于小样本学习的玻璃面板表面缺陷检测方法，包括以下步骤：采集少量有缺陷的玻璃面板图像并标注边界框和缺陷类别；对少量玻璃面板缺陷图像进行预处理及数量扩展；构建用于对玻璃面板缺陷图像进行识别和定位的网络；使用训练好的缺陷检测模型对玻璃面板图像进行缺陷检测，输出缺陷边框和类别。该方法使用数据增强、迁移学习、L2正则化缓解小样本问题，对标注框进行随机抖动增加边框的多样性，以提高模型鲁棒性，使用全局ROI提取层为候选区域特征引入背景信息，自适应改变损失中各个样本的权重以提高模型性能。该方法适合只有少量玻璃面板图像的表面缺陷检测任务，检测精度高。

The invention discloses a glass panel surface defect detection method based on small sample learning. ; Build a network for identifying and locating glass panel defect images; use the trained defect detection model to detect defects on glass panel images, and output defect borders and categories. This method uses data enhancement, transfer learning, and L2 regularization to alleviate the small sample problem, randomly jitters the annotation frame to increase the diversity of the frame to improve the robustness of the model, and uses a global ROI extraction layer to introduce background information for candidate region features. Adaptation changes the weights of individual samples in the loss to improve model performance. This method is suitable for surface defect detection tasks with only a few glass panel images, and the detection accuracy is high.

Description

Glass panel surface defect detection method based on small sample learning

Technical Field

The invention relates to the technical field of glass panel detection, in particular to a glass panel surface defect detection method based on small sample learning.

Background

Glass panels are in great demand in industries such as computers, communication and consumer electronics, and along with the continuous increase of market demand, the requirements on the quality of the glass panels are higher and higher. The traditional manual inspection method needs a large number of trained workers, often consumes a large amount of manpower, is low in efficiency, and due to personal subjectivity, standards may have large differences. With the development of optical and computer technology, many automated optical inspection solutions have been proposed for surface defect inspection tasks. The non-contact detection method can fundamentally improve the detection precision and efficiency, provide guidance for production and reduce the manual burden.

Traditional machine vision methods are often designed for several specific defects and are not versatile. Compared with the traditional machine vision method, the deep convolutional neural network has the advantages that the strong ability of learning advanced features is demonstrated, and the precision and the efficiency of image classification and detection are improved. Currently, in the field of target detection, there are two-stage detectors represented by fast R-CNN and single-stage detectors represented by SSD and YOLO, wherein fast R-CNN is widely used for defect detection of industrial products due to its high detection accuracy and high speed. However, the current direct application of fast R-CNN to the surface defect detection of glass panels still has several difficulties:

(1) most current defect detection methods are based on data-driven methods to train relevant detection models, and it is very difficult to collect defect sample data in the actual glass panel industry field, because most glass panels are qualified. Therefore, when the fast R-CNN is applied to the surface defect detection of the glass panel, the problem of insufficient data needs to be considered.

(2) In actual conditions, the boundaries of partial defects of the glass panel are fuzzy, the defects such as scratches and the like are intermittent, the defects are difficult to mark by using a very clear rule, and the problem of inaccurate marking exists.

(3) Defects such as pinholes and the like of an actual glass panel are usually small, and most of the current defect detection methods are not high in detection precision of small targets and are easy to miss detection and the like.

Disclosure of Invention

In order to solve the problems, the invention provides a glass panel surface defect detection method based on small sample learning, which realizes the efficient and accurate detection of the glass panel surface defects under the conditions that only a small number of glass panel defect sample images are available, the labeling is not accurate enough, and the small targets are more, and comprises the following specific steps:

s1: collecting a small number of defective glass panel images, and labeling a boundary frame and defect types;

s2: preprocessing the glass panel images and expanding the number of the glass panel images to construct a glass panel surface defect detection data set;

s3: constructing a defect detection network for identifying and positioning the glass panel image, wherein the defect detection network comprises a main feature extraction network, an improved RPN (resilient packet network), an ROI posing layer, a global ROI extraction layer, a frame regression network and a classification network, wherein the main feature extraction network is composed of ResNet101 and a feature pyramid network;

the system comprises a backbone feature extraction network, an improved RPN (resilient packet network) network and an ROI posing layer, wherein the backbone feature extraction network is used for extracting global features of an image, the improved RPN network and the ROI posing layer are used for extracting candidate region features of the image, the global ROI extraction layer is used for fusing the global features and the candidate region features and updating the candidate region features, and a frame regression network and a classification network are used for generating a positioning boundary frame and a defect type according to the updated candidate region features;

s4: training the constructed defect detection network by using a glass panel surface defect detection data set to obtain a trained defect detection model;

s5: and performing defect detection on the glass panel image by using the trained defect detection model, and outputting a defect positioning boundary frame and the defect type to which the defect positioning boundary frame belongs.

Further, the backbone feature extraction network comprises a ResNet101 and a feature pyramid network;

the ResNet101 comprises a layer of convolution layer and four residual blocks from bottom to top which are sequentially connected; and the output of each residual block is sequentially connected with the corresponding layer of the feature pyramid network from top to bottom, and the output results of each layer of the feature pyramid network are subjected to L2 regularization processing to form a multi-scale feature map.

The improved RPN network working mode is as follows: taking a multi-scale feature map output by the feature pyramid network as the input of an RPN network, generating p anchor frames in the feature map of each scale, obtaining the length and width dimensions of the anchor frames through clustering, and clustering marking frames in a training set into p × q classes by using a k-means method, wherein q is the number of different scales of the multi-scale feature map; and generating p × q anchor frames according to each clustering center, sequencing according to the area of the anchor frames, and enabling the feature map of each scale to correspond to the p anchor frames. And automatically selecting a positive sample and a negative sample by adopting a self-adaptive training sample selection method, and carrying out classification and frame regression training. And performing frame regression and scoring on the anchor frame through an RPN (resilient packet network) to obtain an initial candidate region, and filtering through NMS (network management system) to obtain a final candidate region.

Inputting the candidate region generated by the improved RPN network into an ROI posing layer, extracting the characteristics of the candidate region from the characteristic diagram with the corresponding scale output by the characteristic pyramid network, and unifying the size; taking a multi-scale feature map output by the feature pyramid network as a global feature, performing attention mechanism-based processing on the candidate regional feature and the averaged pooled global feature, and taking the candidate regional feature as a mask to generate a background feature; and dynamically fusing the background features and the candidate region features to obtain updated candidate region features.

And sending the updated candidate region characteristics into a frame regression network and a classification network, wherein the total loss of the defect detection network comprises classification loss and frame loss, the classification loss uses a cross entropy function, and the frame loss uses a smoothL1 function.

Further, the positive sample and the negative sample are obtained according to a self-adaptive training sample selection method, firstly, the sample with the IOU lower than a threshold value is filtered according to the IOU statistical characteristics of the anchor frame and the marking frame, then whether the center of the anchor frame falls into the marking frame is judged, if yes, the positive sample is obtained, and if not, the negative sample is obtained.

Under the condition that only a small number of defective pictures of the glass panel can be obtained, the contrast of the pictures is improved by using the contrast-limiting self-adaptive histogram equalization, the noise is added to simulate the pictures without accurate focusing, the image blocks are randomly adopted to realize the expansion of the small sample images according to the intersection of the labeling frames and the image blocks, and the problem of the small samples is solved by using data enhancement, transfer learning and L2 regularization; random dithering is carried out on the labeling frame to increase the diversity of the frame and enhance the robustness of the model under the condition of inaccurate labeling; the characteristic pyramid is used for fusing the bottom layer structure information and the high-level semantic information to generate multi-scale characteristics, so that the detection accuracy of small target defects such as glass panel pinholes is improved; the candidate region features are fused with the features of the whole image, background information is introduced into the candidate region features, and classification and regression of a frame which is smaller than a labeled frame are facilitated; and considering the influence of each sample on the mAP, the weight of various samples in the loss is changed, and the performance index of the model is improved on the whole.

Drawings

FIG. 1 is a flowchart of a method for detecting surface defects of a glass panel based on a small sample learning.

FIG. 2 is a schematic diagram of a glass panel surface defect detection network model.

Fig. 3 is a schematic structural diagram of a global ROI extraction layer.

FIG. 4 is a graph of experimental results of the method provided by the present invention.

FIG. 5 is a partial schematic view showing the result of detecting surface defects of a glass panel, wherein (a) is a bubble defect; (b) is a tin ash defect; (c) a pinhole defect; (d) is a scratch defect.

Detailed Description

The invention will be further elucidated and described with reference to the drawings and the detailed description. The technical features of the embodiments of the present invention can be combined correspondingly without mutual conflict.

The overall flow chart of the glass panel surface defect detection method based on small sample learning disclosed by the invention is shown in fig. 1, and the specific implementation process is as follows:

(1) a small number of defective glass panel images are collected. And labeling each picture by using labeling software labelImg to generate an xml file, wherein the xml file comprises a defect boundary frame and defect types, and the defect types comprise bubbles, tin ash, pinholes and scratches.

(2) And preprocessing the glass panel images and expanding the number of the glass panel images to construct a glass panel surface defect detection data set.

The method comprises the following specific steps:

a. preprocessing includes limiting contrast adaptive histogram equalization, adding noise, etc.

By using the contrast-limited self-adaptive histogram equalization method, the upper threshold of contrast limitation is 4, the grid size of histogram equalization is selected to be 8, and the contrast of the image can be enhanced by using the method because the acquired glass panel image is dark.

The specific process of the contrast-limiting adaptive histogram equalization method comprises the following steps: firstly, converting an image from an RGB color mode to an LAB color mode, then dividing the image into 8 × 8 rectangular blocks with equal size, then counting the histogram distribution of a luminance channel (L channel) of each rectangular block, and if a threshold value is exceeded in each histogram

The gray level exceeding part is cut and evenly distributed to each gray level, then each histogram is equalized, the brightness value of each rectangular block after the center point is equalized is calculated, then each pixel point is subjected to bilinear difference by using the center point of the adjacent rectangular block to obtain the brightness value of the pixel point, and finally the image after the difference is converted into an RGB color mode from an LAB color mode.

Random addition of gaussian or salt and pepper noise simulates a picture without accurate focus.

b. The number expansion is achieved by random sampling.

And randomly cutting each glass panel image with the defects to obtain a plurality of image blocks, wherein each image block comprises one or more bounding boxes with defect types.

In this embodiment, when an image of a defective glass panel is cut, a threshold set {0.1,0.3,0.5,0.7,0.9} of the intersection ratio between an image block and a bounding box is set, and multiple rounds of cutting are performed in sequence from small to large according to the threshold, and it is ensured that the intersection ratio between the bounding box with the defective category and the image block in the obtained image block is greater than the threshold in each round of cutting. For example, the threshold value is set to 0.1, the extended data set is randomly sampled in the acquisition area of the defective glass panel image, after one round of sampling is finished, the threshold value is set to 0.3, and the above process is repeated.

After the cutting is finished, random dithering is carried out on the marking frame to increase the diversity of the marking frame so as to improve the adaptability and robustness of the model, the center point of the marking frame is unchanged, and the scaling proportion of the length and the width is uniformly sampled between 0.9 and 1.1; and transforming the marking frame while enhancing the data, and finally manually checking to ensure that the image and the marking frame generated by enhancing the data are in accordance with the reality and meet the construction principle of the data set.

(3) And constructing a glass panel surface defect detection network model for identifying and positioning the glass panel defect image. As shown in fig. 2, the defect detection network model includes a backbone feature extraction network composed of ResNet101 and a feature pyramid network, an improved RPN network, a ROI posing layer, a global ROI extraction layer, a bounding box regression network, and a classification network.

The following describes the structure of each sub-network included in the glass panel surface defect detection network model.

(3.1) backbone feature extraction network

The main feature extraction network is composed of a ResNet101 and a feature pyramid network, in the embodiment, the ResNet101 adopts a weight parameter pre-trained by a classification task through a large-scale data set ImageNet, and overfitting of a depth model on small data is relieved by means of transfer learning.

The ResNet101 comprises a convolution layer and 4 residual blocks, and feature pyramid networks are used for fusing feature outputs of the 4 residual blocks of the ResNet101, namely, bottom layer structure information and high-level semantic information are combined, feature expression is enhanced, and small target detection performance is improved. The output of the characteristic pyramid network consists of 5 layers of characteristic graphs, and the characteristic graphs of each branch of the characteristic pyramid network are subjected to L2 regularization processing to form a multi-scale characteristic graph.

(3.2) improved RPN network

And inputting the multi-scale features output by the feature pyramid network into the improved RPN network for extracting the candidate region. Specifically, a multi-scale feature map output by a feature pyramid network is used as input of an RPN, 3 anchor frames are generated in the feature map of each scale, the length and width dimensions of the anchor frames are obtained through clustering, and a group of prior values more suitable for a data set is automatically generated by using a k-means method. The IOU of the anchor frame and the label frame is improved, the detection precision of the model is improved, therefore, for improving the IOU of the anchor frame and the label frame, the IOU value is adopted as the judgment standard of the cluster, and the distance measurement formula is as follows:

d(box,centroid)＝1-IOU(box,centroid)

the IOU (box) represents the intersection ratio of the marking box and the cluster center box.

Automatically selecting a positive sample and a negative sample by adopting a self-adaptive training sample selection method, and carrying out classification and frame regression training; performing frame regression and scoring on the anchor frame through an RPN (resilient packet network) to obtain an initial candidate region; the candidate regions filtered by NMS are sent to an ROI posing layer for feature extraction, and the feature sizes are uniformly set to be 7 x 7.

In this embodiment, the labeling boxes in the training set are grouped into 15 types; and generating 15 anchor frames according to each clustering center. The large-resolution feature map contains finer structural features and is suitable for detecting small targets, the small-resolution feature map has a larger receptive field and is suitable for detecting large targets, therefore, the generated anchor frames are sorted according to areas, the anchor frames with small areas serve as prior frames of the large-resolution feature map, and 3 anchor frames are distributed to each feature map.

In a common method for selecting positive and negative samples, the IOU of an anchor frame and the IOU of a marking frame are calculated, and the IOU is compared with a set threshold value to select the positive and negative samples, so that the defect that the anchor frame is a positive sample or a negative sample is obvious, and the method is very limited by the design of the length and the width of the anchor frame and the selection of the threshold value. The invention provides a self-adaptive training sample selection method, which is characterized in that a part of anchor frames with smaller IOU are filtered according to the IOU statistical characteristics of the anchor frames and a marking frame, and then positive and negative samples are judged by judging whether the centers of the anchor frames fall into the marking frame, so that the defect is obviously overcome.

(3.3) ROI pooling layer

And inputting the candidate region generated by the improved RPN network into an ROI posing layer, extracting the characteristics of the candidate region from the characteristic diagram of the corresponding scale output by the characteristic pyramid network, and unifying the size to be 7 multiplied by 7.

(3.4) Global ROI extraction layer

The global ROI extraction layer is used for extracting the features of the whole image, fusing the global features and the candidate region features by using a residual error network, introducing background information into the candidate region features, and facilitating classification and regression of a frame which is smaller than a labeling frame.

In this embodiment, the global ROI extraction layer adopts a residual structure, and a side branch of the residual structure is favorable for avoiding a network degradation phenomenon, and the specific structure is as shown in fig. 3, where the global feature changes the resolution to 7 × 7 by adaptive average pooling, and then uses the candidate region feature X _ pro as a mask based on an attention mechanism to further optimize and generate the background feature X _ bg, and considering that simply adding the positions of the two features means that the two features are equally important, but the background feature can only play an auxiliary role, a dynamic fusion strategy is used to fuse the candidate region feature X _ pro and the background feature X _ bg, and w _ pro and w _ bg are both used as learnable parameters, and finally an updated candidate region feature is output through a convolution layer.

(3.5) bounding Box regression network and Classification network

And sending the updated candidate region characteristics into a frame regression network and a classification network to realize the detection of the surface defects of the glass panel.

And detecting the total loss of the network, wherein the classification loss and the border loss comprise classification loss and border loss, the classification loss uses a cross entropy function, the border loss uses a smoothL1 function, and the total loss realizes importance weighting of the prediction boxes from the aspect that each prediction box influences the average accuracy average (mAP).

The prediction box importance weighting specifically includes: the classification weight is adjusted to take into account the influence of each sample on mAP, the larger the IOU for positive samples, and labeled boxes, the classification weight w_iThe larger; for negative samples, the greater the score predicted as a positive sample (i.e., the probability of being classified as a defect), the classification weight w_jThe larger; and adjusting the frame regression weight, wherein for the positive sample, the larger the score predicted to be a certain type of defect is, the frame regression weight c_iThe larger.

The loss function is as follows:

L＝λL_cls+μL_loc

wherein λ and μ are parameters for balancing classification loss and bounding box loss, and L is total loss;

the classification loss is as follows:

wherein L is_clsIs the classification loss, cross _ entropy (. -) is the cross entropy function, n is the number of positive samples, m is the number of negative samples, s_iIs the prediction score, s, of the ith positive sample_jIs the prediction score of the jth negative sample,

for the true defect class of the ith positive sample,

is the category of the jth negative example (i.e., background category), w_iAnd w_jIs weight, beta and gamma are hyperparameters, r_iOrder number of ith positive sample, r_jThe ranking sequence number of the jth negative sample, wherein the positive samples are ranked according to the IOU of the sum label box, the larger the IOU is, the smaller the sequence number is, the negative samples are ranked according to the probability of being predicted as the defect category, the larger the probability is, the smaller the sequence number is, n_maxThe total number of the classes to which the samples belong;

the frame loss is as follows:

wherein L is_locFor bezel loss, smoothL1(.) is a smoothL1 function, n is the number of positive samples, d_iFor the prediction bounding box offset of the ith positive sample,

as deviation of the annotation box and the prediction bounding box, c_iAre weights, b and k are hyperparameters, p_iAnd the prediction score of the real defect category corresponding to the prediction frame of the ith positive sample. Thus, for positive samples, the greater the score predicted to be a defect of a certain type, the greater the weight in the loss.

Model training is carried out by utilizing a glass panel surface defect detection data set, the initial learning rate is 0.004, the change of the learning rate adopts the periodic learning rate and cosine annealing, the training period is 50 rounds, the momentum is set to be 0.9, and the weight attenuation is set to be 0.0001.

In practical application, the trained defect detection model is used for detecting the defects of the glass panel image and outputting a defect frame and a defect category. As shown in fig. 5, the detection results of four defects are shown. Because the detection picture is large, a local picture is displayed for convenient display.

In the embodiment, the defect detection result is evaluated by using mAP, Precision (Precision) and Recall (Recall), and the higher mAP indicates that the defect detection effect of the method on the glass panel is better. The defect detection result of the algorithm of the embodiment is shown in FIG. 4, the comparison between the algorithm of the embodiment and the conventional Faster R-CNN is shown in Table 1, and it can be seen from Table 1 that the method provided by the embodiment has obvious advantages in terms of accuracy or recall compared with the conventional Faster R-CNN.

TABLE 1 detection results of conventional Faster R-CNN and the algorithm of this example on a glass panel defect data set

Detecting a network	mAP(％)	Precision(％)	Recall(％)
				Conventional fast R-CNN	72.8	73.1	70.1
The method of the invention	85.1	82.2	82.8

In conclusion, under the condition that only a small number of defective pictures of the glass panel can be obtained, the contrast of the pictures is improved by using the contrast-limiting self-adaptive histogram equalization, the noise is added to simulate the pictures without accurate focusing, the expansion of the small sample images is realized according to the intersection of the marking frame and the image blocks compared with the randomly sampled image blocks, and the problem of the small samples is relieved by using data enhancement, transfer learning and L2 regularization; random dithering is carried out on the labeling frame to increase the diversity of the frame and enhance the robustness of the model under the condition of inaccurate labeling; the characteristic pyramid is used for fusing the bottom layer structure information and the high-level semantic information to generate multi-scale characteristics, so that the detection accuracy of small target defects such as glass panel pinholes is improved; the candidate region features are fused with the whole image features, background information is introduced into the candidate region features, and classification and regression of a frame which is smaller than a real frame are facilitated; and considering the influence of each sample on the mAP, the weight of various samples in the loss is changed, and the performance index of the model is improved on the whole.

Claims (10)

Translated fromChinese

1.一种基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，包括以下步骤：1. a glass panel surface defect detection method based on small sample learning, is characterized in that, comprises the following steps:S1：采集少量有缺陷的玻璃面板图像，并标注边界框和缺陷类别；S1: Collect a small number of defective glass panel images, and annotate bounding boxes and defect categories;S2：对玻璃面板图像进行预处理及数量扩展，构建玻璃面板表面缺陷检测数据集；S2: Preprocess and expand the number of glass panel images to construct a glass panel surface defect detection data set;S3：构建用于对玻璃面板图像进行识别和定位的缺陷检测网络，所述的缺陷检测网络包含由ResNet101和特征金字塔网络构成的主干特征提取网络、改进的RPN网络、ROIpooling层、全局ROI提取层、边框回归网络和分类网络；S3: Build a defect detection network for identifying and locating glass panel images. The defect detection network includes a backbone feature extraction network composed of ResNet101 and a feature pyramid network, an improved RPN network, a ROIpooling layer, and a global ROI extraction layer. , bounding box regression network and classification network;所述的主干特征提取网络用于提取图像的全局特征，改进的RPN网络和ROI pooling层用于提取图像的候选区域特征，全局ROI提取层用于融合全局特征和候选区域特征，对候选区域特征进行更新，边框回归网络和分类网络用于根据更新后的候选区域特征生成定位边界框和缺陷类别；The backbone feature extraction network is used to extract the global features of the image, the improved RPN network and the ROI pooling layer are used to extract the candidate region features of the image, and the global ROI extraction layer is used to fuse the global features and the candidate region features. After updating, the bounding box regression network and the classification network are used to generate and locate bounding boxes and defect categories according to the updated candidate region features;S4：使用玻璃面板表面缺陷检测数据集对构建的缺陷检测网络进行训练，得到训练好的缺陷检测模型；S4: Use the glass panel surface defect detection data set to train the constructed defect detection network to obtain a trained defect detection model;S5：使用训练好的缺陷检测模型对玻璃面板图像进行缺陷检测，输出缺陷定位边界框和所属的缺陷类别。S5: Use the trained defect detection model to detect defects on the glass panel image, and output the defect localization bounding box and the defect category it belongs to.2.如权利要求1所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，所述S2具体为：2. The glass panel surface defect detection method based on small sample learning according to claim 1, wherein the S2 is specifically:a.预处理：包括限制对比度自适应直方图均衡化和添加噪声；a. Preprocessing: including limiting contrast adaptive histogram equalization and adding noise;所述的对比度自适应直方图均衡化为：使用限制对比度自适应直方图均衡化方法，将玻璃面板图像从RGB颜色模式转换为LAB颜色模式，将图像分割为大小相等的8×8矩形块，统计每个矩形块亮度通道的直方图分布，对每个直方图中超过了阈值的灰度级部分裁剪并平均分配到各个阈值内的灰度级上，接着对每个直方图均衡化，计算每个矩形块中心点均衡化后的亮度值，对每个像素点使用邻近的矩形块的中心点进行双线性差值，得到该像素点亮度值，将差值后的图像从LAB颜色模式转换为RGB颜色模式；The contrast adaptive histogram equalization is as follows: using the limited contrast adaptive histogram equalization method, the glass panel image is converted from the RGB color mode to the LAB color mode, and the image is divided into 8×8 rectangular blocks of equal size, The histogram distribution of the luminance channel of each rectangular block is counted, and the gray level part of each histogram that exceeds the threshold is cropped and evenly distributed to the gray levels within each threshold, and then each histogram is equalized and calculated. The brightness value of the center point of each rectangular block after equalization, the center point of the adjacent rectangular block is used to perform bilinear difference value for each pixel point, and the brightness value of the pixel point is obtained, and the image after the difference value is changed from the LAB color mode. Convert to RGB color mode;所述的噪声为随机添加高斯或椒盐噪声；The noise is randomly added Gaussian or salt and pepper noise;b.数量扩展：对每一张带有缺陷的玻璃面板图像随机剪裁得到若干图像块，每一个图像块中包括一个或多个带有缺陷类别的边界框。b. Quantity expansion: randomly crop each defective glass panel image to obtain several image blocks, each image block includes one or more bounding boxes with defect categories.3.如权利要求2所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，对带有缺陷的玻璃面板图像进行剪裁时，设置图像块和边界框交并比的阈值集合为{0.1,0.3,0.5,0.7,0.9}，按照阈值从小到大的顺序依次进行多轮剪裁，每一轮剪裁中保证得到的图像块中带有缺陷类别的边界框和图像块的交并比大于阈值。3. The glass panel surface defect detection method based on small sample learning according to claim 2, characterized in that, when the image of the glass panel with defects is cropped, the threshold set of the intersection ratio of the image block and the bounding box is set as: {0.1, 0.3, 0.5, 0.7, 0.9}, perform multiple rounds of clipping in order of threshold value from small to large, each round of clipping guarantees the obtained image block with the defect category in the bounding box and the intersection ratio of the image block greater than the threshold.4.如权利要求2或3所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，对剪裁得到的图像块中标注的边界框尺寸进行随机缩放，缩放时边界框的中心点位置不变，边界框的长宽缩放比例在[0.9,1.1]之间。4. The method for detecting surface defects of glass panels based on small sample learning according to claim 2 or 3, wherein the size of the bounding box marked in the clipped image block is randomly scaled, and the center point of the bounding box is scaled when scaling. The position is unchanged, and the length and width of the bounding box is scaled between [0.9, 1.1].5.如权利要求1所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，所述的主干特征提取网络包括ResNet101和特征金字塔网络；5. The glass panel surface defect detection method based on small sample learning as claimed in claim 1, wherein the backbone feature extraction network comprises ResNet101 and a feature pyramid network;所述的ResNet101包括一层卷积层和自下而上连接的四个残差块；每一个残差块的输出与自上而下的特征金字塔网络的对应层依次连接，将特征金字塔网络各层输出结果进行L2正则化处理，得到多尺度特征图。The described ResNet101 includes a convolutional layer and four residual blocks connected from bottom to top; the output of each residual block is connected to the corresponding layer of the top-down feature pyramid network in turn, and the feature pyramid network is connected to each layer. The layer output results are L2 regularized to obtain multi-scale feature maps.6.如权利要求1或5所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，所述的改进的RPN网络工作方式为：将特征金字塔网络输出的多尺度特征图作为RPN网络的输入，每个尺度的特征图中生成p种锚框，所述锚框的长和宽尺寸通过聚类得到；采用自适应训练样本选择方法，自动选择正样本和负样本，进行分类和边框回归训练；所述的锚框经过RPN网络进行边框回归和打分，得到初始的候选区域，经过NMS过滤后，得到最终的候选区域。6. The glass panel surface defect detection method based on small sample learning according to claim 1 or 5, wherein the improved RPN network works as follows: the multi-scale feature map output by the feature pyramid network is used as the RPN For the input of the network, p types of anchor boxes are generated in the feature map of each scale, and the length and width of the anchor boxes are obtained by clustering; the adaptive training sample selection method is used to automatically select positive samples and negative samples for classification and classification. Frame regression training; the anchor frame is subjected to frame regression and scoring through the RPN network to obtain the initial candidate area, and after NMS filtering, the final candidate area is obtained.7.如权利要求6所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，利用k-means方法，将训练集中的标注框聚成p*q类，其中q为多尺度特征图的不同尺度数量；根据各个聚类中心共生成p*q个锚框，按照锚框面积排序，每种尺度的特征图对应p个锚框。7 . The glass panel surface defect detection method based on small sample learning according to claim 6 , wherein, using the k-means method, the annotation frames in the training set are grouped into p*q classes, wherein q is a multi-scale feature. 8 . The number of different scales of the graph; a total of p*q anchor boxes are generated according to each clustering center, sorted according to the anchor box area, and the feature map of each scale corresponds to p anchor boxes.8.如权利要求6所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，将改进的RPN网络生成的候选区域输入到ROIpooling层，从特征金字塔网络输出的对应尺度的特征图中提取候选区域特征，并统一尺寸；将特征金字塔网络输出的多尺度特征图作为全局特征，将候选区域特征与平均池化后的全局特征进行基于注意力机制的处理，以候选区域特征作为掩膜，生成背景特征；再将背景特征与候选区域特征动态融合，得到更新后的候选区域特征。8. The glass panel surface defect detection method based on small sample learning as claimed in claim 6, wherein the candidate region generated by the improved RPN network is input to the ROIpooling layer, and the feature map of the corresponding scale output from the feature pyramid network Extract the features of the candidate region from , and unify the size; take the multi-scale feature map output by the feature pyramid network as the global feature, process the feature of the candidate region and the global feature after average pooling based on the attention mechanism, and use the feature of the candidate region as the mask Then, the background features and the candidate region features are dynamically fused to obtain the updated candidate region features.9.如权利要求6所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，所述的正样本和负样本根据自适应训练样本选择方法得到，先根据锚框与标注框的IOU统计特征，过滤IOU低于阈值的样本，然后判定锚框的中心是否落入标注框中，若是，则为正样本，若否，则为负样本。9. The method for detecting surface defects of glass panels based on small sample learning according to claim 6, wherein the positive samples and negative samples are obtained according to the adaptive training sample selection method, and firstly according to the difference between the anchor frame and the label frame. IOU statistical features, filter the samples whose IOU is lower than the threshold, and then determine whether the center of the anchor box falls into the label box, if so, it is a positive sample, if not, it is a negative sample.10.如权利要求1所述的基于小样本学习的玻璃面板表面缺陷检测方法，其特征在于，将更新后的候选区域特征送入边框回归网络和分类网络，缺陷检测网络的总损失包括分类损失和边框损失，分类损失使用交叉熵函数，边框损失使用smoothL1函数，具体为：10. The glass panel surface defect detection method based on small sample learning according to claim 1, wherein the updated candidate region features are sent to the frame regression network and the classification network, and the total loss of the defect detection network includes the classification loss And the border loss, the classification loss uses the cross entropy function, and the border loss uses the smoothL1 function, specifically:L＝λL_cls+μL_locL=λL_cls + μL_loc其中，λ和μ是用于平衡分类损失和边框损失的参数，L为总损失；where λ and μ are the parameters used to balance the classification loss and the bounding box loss, and L is the total loss;所述的分类损失为：The described classification loss is:

其中，L_cls是分类损失，cross_entropy(.)为交叉熵函数，n为正样本的个数，m为负样本的个数，s_i为第i个正样本的预测分数，s_j为第j个负样本的预测分数，

为第i个正样本的真实缺陷类别，

为第j个负样本的类别(即背景类别)，w_i和w_j为权重，β和γ为超参数，r_i为第i个正样本的排序序号，r_j为第j个负样本的排序序号，n_max为样本所属类别的总个数；where L_cls is the classification loss, cross_entropy(.) is the cross entropy function, n is the number of positive samples, m is the number of negative samples, s_i is the prediction score of the ith positive sample, and s_j is the jth The predicted scores of the negative samples,

is the true defect category of the i-th positive sample,

is the category of the jth negative sample (that is, the background category), w_i and w_j are the weights, β and γ are hyperparameters, ri is the sorting number of the ith positive sample, and r_j is the_jth negative sample. Sort number, n_max is the total number of categories to which the samples belong;所述的边框损失为：The bounding box loss is:

其中，L_loc为边框损失，smoothL1(.)为smoothL1函数，n为正样本的个数，d_i为第i个正样本的预测框偏移量，

为标注框和预测边界框的偏差，c_i为权重，b和k为超参数，p_i为第i个正样本的预测框对应的真实缺陷类别的预测分数。Among them, L_loc is the frame loss, smoothL1(.) is the smoothL1 function, n is the number of positive samples, d_i is the predicted frame offset of the ith positive sample,

is the deviation between the labeled box and the predicted bounding box, c_i is the weight, b and k are hyperparameters, and p_i is the predicted score of the true defect category corresponding to the predicted box of the ith positive sample.

Images