Detailed Description
The embodiment of the application provides an intelligent detection method for an automobile decorative plate processing process. The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
For ease of understanding, a specific flow of an embodiment of the present application is described below with reference to fig. 1, and an embodiment of an intelligent detection method for a processing procedure of an automobile decorative board according to an embodiment of the present application includes:
S101, performing high-resolution image acquisition and preprocessing on an automobile decorative plate to obtain standardized image data;
it will be appreciated that the execution body of the present application may be an intelligent detection system for a car decorative board processing process, and may also be a terminal or a server, which is not limited herein. The embodiment of the application is described by taking a server as an execution main body as an example.
Specifically, a special image acquisition system is built, and the system consists of a high-resolution industrial camera, an LED light source array with specific wavelength and an accurate positioning device. The industrial camera adopts 1920 x 1080 pixel high resolution model to ensure capturing fine texture and tiny structural characteristics of the surface of the automobile decorative board, meanwhile, in order to ensure uniform illumination of the surface of the decorative board, a light source system adopts an annular LED light source with the color temperature stabilized at 5500 K+/-100K, and the light source can provide uniform and stable white light irradiation, so that shadows or reflections caused by uneven light rays are avoided to influence the image quality. The accurate positioning device is matched with the automatic triggering mechanism through the conveyor belt, so that when the decorative plate moves to the appointed shooting position, the image acquisition system automatically captures the front view of the decorative plate, and the full-automatic image acquisition process is realized. The original decorative plate image is subjected to size standardization processing, all acquired original images are adjusted to be uniform 1280 multiplied by 720 pixel sizes, consistency of image data is guaranteed, and the problem of characteristic mismatch caused by difference of image resolution is avoided in the model training process, so that a standardized size image is obtained. And carrying out histogram equalization processing on the standardized size image so as to enhance the contrast of the image. The histogram equalization is an image enhancement method for improving details of bright and dark areas of an image by stretching gray value distribution of the image, so that surface textures of a decorative plate are more clear and prominent, and a contrast enhancement image is obtained. And carrying out non-local mean filtering on the contrast enhanced image, and carrying out weighted average on similar pixels by calculating the similarity between each pixel in the image and surrounding pixel blocks, so that Gaussian noise and spiced salt noise in the image are effectively removed, and meanwhile, the edge and detail characteristics of the image are maintained, and a denoising image is obtained. An adaptive thresholding operation is performed on the denoised image to generate a binary mask map of the trim panel. The self-adaptive threshold segmentation is a method for dynamically calculating the binarization threshold of each pixel in an image, and the decorative plate area is accurately separated from the background by analyzing the brightness characteristic of a local area and effectively aiming at the condition of uneven illumination or complex background in the image. By this step, a binary mask pattern including only the shape of the decorative plate is obtained. And performing background removal operation on the denoising image by using the binary mask map to obtain a clean decorative plate image. In the background removal process, the non-decorative panel region in the original image is filled with black, so that only the pixel data of the decorative panel body remains in the output image. And performing image enhancement processing on the clean decorative plate image to obtain standardized image data. The image enhancement processing comprises brightness, contrast and color saturation adjustment of the image, sharpening operation based on a specific filter, improving the visual effect of the image and strengthening the expression of the surface characteristics of the decorative plate.
Step S102, performing region segmentation operation on standardized image data to obtain a target region mask map of a decorative plate, and extracting multi-mode features from the target region mask map of the decorative plate to obtain a feature vector set;
Specifically, the standardized image data is input into a U-Net network for feature extraction so as to realize the construction of a multi-scale feature map. The U-Net network structure is designed into a symmetrical structure of a four-layer encoder and a four-layer decoder, wherein the encoder part adopts ResNet modules with hole convolution to realize multiscale and high efficiency of feature extraction, and the decoder part gradually restores the space dimension of a feature map in a transposed convolution mode, so that the image segmentation precision is ensured to be improved while the feature richness is maintained. In the encoder, the introduction of the cavity convolution remarkably enlarges the receptive field, so that the network can capture the detail features and global context information in the decorative plate image, and the ResNet module effectively avoids the problem of gradient disappearance in the deep network through the design of residual connection, so that the key region features of the decorative plate can be stably extracted from the model in a complex background. And performing network training based on the multi-scale feature map to obtain a region segmentation model. In the training process, a supervised learning method with a weighted boundary loss function is used, so that the model gives higher punishment weight to the boundary region when predicting the target region of the decorative plate, and the segmentation precision of the model at the complex boundary is improved. For example, at these key locations of arc and place board junction, push assembly installation face, locked groove periphery, fixture block and draw-in groove meshing area and foam adhesion area, the model can accurately generate regional mask diagram, provides space location basis for follow-up feature extraction. After the region segmentation model is trained, standardized image data is input into the model, and the model outputs a mask map of the target region of the decorative plate, wherein different target regions are identified by different mask categories. And calculating morphological characteristics and texture characteristics based on the target area mask map of the decorative plate, and constructing a primary characteristic set. The method of extracting morphological features includes calculating the area, perimeter, circularity, aspect ratio, and minimum circumscribed rectangular dimensions of each target area, which features help identify potential dimensional deviations or shape anomalies in, for example, the cartridge-to-cartridge engagement area. Meanwhile, the texture of the target area is analyzed by using a local binary pattern operator, and statistical characteristics such as energy, contrast, correlation, entropy and the like of pixel gray distribution in the area are obtained, and the texture characteristics can effectively reflect whether defects such as scratches, material cracks or poor welding exist on the surface of the decorative plate. On the basis of primary feature extraction, edge features and color features are extracted based on a decorative board target area mask map to construct a medium-level feature set. In the aspect of edge feature extraction, edge detection is carried out through an improved Canny operator, and edge density, directionality and continuity features are calculated and are helpful for fine structure identification of a pushing component mounting surface and a locking groove periphery. In terms of color feature extraction, by converting the target region into the HSV color space, the average, variance, skewness and kurtosis of hue, saturation and brightness within the region are calculated, respectively, color features are able to distinguish decorative panels of different materials and identify, for example, color irregularities or pollution phenomena present in the foam adhesive region. Depth features are extracted from a trim panel target area mask map to obtain a target feature vector. Introducing a pre-trained EFFICIENTNET-B0 network, inputting the image of the decorative board mask region into the network, extracting high-dimensional depth features through a convolutional neural network, and obtaining 1280-dimensional feature vectors. And reducing the dimension of the high-dimensional feature vector to 128 dimensions by a principal component analysis method, so that redundant information is removed, and the feature dimension with a key effect on the defect identification of the decorative plate is reserved. And performing feature selection on the primary feature set, the intermediate feature set and the target feature vector to obtain a feature vector set. In the feature selection process, the contribution degree of each feature to defect identification is calculated through a random forest algorithm, and the features with the contribution degree of 80% are reserved through analyzing the importance scores of the features, so that low-correlation or noise features are removed, and the precision and efficiency of feature sets are optimized.
Step S103, inputting the feature vector set into a transfer learning network based on ResNet-v1-50 for training to obtain a defect classification model, and calculating a candidate defect area mask through the defect classification model;
Specifically, a ResNet-v 1-50-based decoration plate defect classification network is constructed, and the design of the network comprises a feature extraction convolution layer, a global average pooling layer, two full-connection layers and a Softmax classification layer. In the network structure, the feature extraction convolution layer reserves the infrastructure of ResNet-v1-50 and comprises a plurality of layers of residual error modules, the modules effectively avoid the gradient vanishing problem in the depth network by introducing residual error connection, and the multi-scale features in the decorative board image are extracted through convolution operation. The introduction of the global average pooling layer effectively replaces the traditional full-connection layer, converts the feature map into a low-dimension feature vector, reduces the number of model parameters, and effectively avoids the phenomenon of overfitting. And in the full-connection layer part, the design is of a two-layer structure, the feature is mapped into a two-classification output result (good product/defect) through a Softmax classification layer through 1024-256-dimensional dimension reduction operation, and the model is ensured to rapidly output the quality judgment result of the decorative plate in the reasoning stage. And carrying out regional extraction on the standardized image data, and dividing the original image data set into a first image data set, a second image data set and a third image data set so as to realize targeted analysis on images with different scales. Wherein the first map data set contains images of the complete trim panel for assessing the quality status of the complete trim panel, such images maintaining a large field of view, facilitating the model capturing defect features present in the trim panel on a global scale. The second image data set focuses on a partial image of the joint of the arc-shaped plate and the placement plate, and the region is a key part of the decorative plate, which is easy to deform or deviate from installation, so that the image size of the data set is smaller, and the model is subjected to fine detection on a detail scale. And the third graph data set focuses on the pushing component area, which is an important object of mechanical clamping quality evaluation, and by providing a partial image with higher resolution, the model can identify the problems of fine cracks, abrasion or poor assembly and the like on the surface of the pushing component. And respectively corresponding the characteristic vector set with the first graph data set, the second graph data set and the third graph data set to obtain a complete data set for model training. In the data preparation process, the image data and the feature vectors are paired to provide multi-mode input information for the model, so that the accuracy and the robustness of the model in a defect identification task are effectively improved. In the training process, the first stage training is executed, the parameters of the convolution blocks in the feature extraction convolution layer are locked, the parameters of only two fully connected layers are optimized, and the model is quickly adapted to the data distribution of the decorative plate by freezing the feature extraction part at the front end of the model, so that the model does not need to consume too much time on the relearning of the bottom layer features. In this way, the model uses the generic features already learned on the ImageNet dataset in the pre-trained ResNet-v1-50 network to adjust only the final classification to fit the particular pattern of the trim panel image, resulting in a classification model that is primarily adapted to the trim panel data distribution. in this stage, a larger learning rate (e.g., 0.001) is selected to accelerate the model convergence rate, and at the same time, monitor the performance on the verification set during the training process, so as to ensure that the model improves the classification accuracy on the premise of fitting. And performing second-stage training on the classification model which is preliminarily adapted to the data distribution of the decorative plate, unlocking all network layer parameters, and performing whole network fine tuning on the whole model. Through the training of the whole network, the model not only adjusts the parameters of the classification layer, but also updates the parameters of the characteristic extraction convolution layer through a back propagation mechanism, so that the model learns more specific visual characteristics from the decorative plate image, and particularly aims at the recognition capability of the surface texture, edge detail and local morphological characteristics of the decorative plate. In this stage, a smaller learning rate (e.g., 0.0001) is used to ensure finer and more stable adjustment of model parameters, while combining learning rate decay strategies with regularization techniques (e.g., weight decay and Dropout) to prevent the problem of model overfitting under small or unbalanced sample conditions. In the training process, by setting a training termination condition (for example, the accuracy of the verification set is continuously improved by five periods or the maximum training period number is reached), the model is ensured to automatically stop training when the optimal performance is reached, and an optimized classification model is obtained. And respectively reasoning the test parts of the first graph data set, the second graph data set and the third graph data set based on the optimized classification model, and obtaining a defect classification result through model prediction. In the reasoning stage, the model rapidly judges whether the decorative plate has defects according to the feature vector and the image data of the input image, and outputs a specific classification result. In order to achieve accurate localization of candidate defect regions, gradient weighted class activation maps are calculated based on an optimized classification model, a thermodynamic diagram is generated, and image regions focused by the model when making defect classification decisions are displayed. And (3) calculating the gradient of the final layer convolution feature map by calculating the score of the model output category, and generating a thermodynamic diagram after weighting and fusing the gradient information and the feature map, so that the area which is considered to be most likely to have defects by the model is highlighted on the image. In order to improve the accuracy of the candidate defect region mask, an image processing operation such as setting an adaptive threshold value, performing binarization processing on the thermodynamic diagram, and removing a noise region by morphological operation to generate the candidate defect region mask.
In this embodiment, a target score function is defined according to the defect class in the defect classification model, and the target score function represents the output score of a specific defect class predicted by the model, for example, in the Softmax classification layer, and the score is expressed as the activation value of the specific class. The objective score function is expressed asWherein c represents the target defect class. By defining the objective scoring function, the explicit model specifically focuses on which feature regions in the input image when a certain class of defects is identified. And calculating the gradient value of the target score function on the characteristic diagram of the last layer of convolution layer in the model to obtain target gradient data. In the model back propagation process, the objective score function is compared with the characteristic diagram of the convolution layerCalculating partial derivatives, expressed mathematically asWhereinRepresenting the first of the feature mapAnd a plurality of channels. And (3) quantifying the sensitivity of the model to the classification result at a specific position by calculating the influence degree of each position pixel in the feature map on the target classification score. And performing global average pooling operation on the target gradient data in the space dimension to calculate importance weights of the feature map channels. The importance of a channel to a target defect class is represented by averaging the feature map gradients across the width and height dimensions such that only one weight value is retained for each channel. The feature map channel importance weights are multiplied with the corresponding feature maps and summed over the channel dimension to generate an initial thermodynamic diagram. And generating a multi-scale thermodynamic diagram based on the middle layer characteristic diagram of the defect classification model, and carrying out weighted fusion on the initial thermodynamic diagram and the multi-scale thermodynamic diagram according to a preset weight coefficient to obtain an optimized thermodynamic diagram. In addition to the feature maps of the last convolution layer, the feature maps of the middle layers in the model (e.g., res3d_branch2c, res4f_branch2c, etc. layers in ResNet) are typically selected, the Grad-CAM thermodynamic diagrams of these middle layer feature maps are calculated separately, and each thermodynamic diagram is assigned a different weight coefficient, e.g., [0.2,0.3, 0.5]. The weighting fusion strategy can integrate the advantages of different scale features, so that the optimized thermodynamic diagram can capture the whole macroscopic decorative plate features and identify microscopic surface details, thereby improving the positioning capability of the model on multiple scales, and particularly showing stronger identification precision in complex structures such as the joint of an arc plate and a placing plate or the meshing area of a clamping block and a clamping groove. Morphological open and close operations are performed on the optimized thermodynamic diagram to refine the candidate defect region mask. Morphological open operations (using small circular structural elements, e.g., structural elements with radius 3) are used to eliminate noise points in thermodynamic diagrams, e.g., to eliminate isolated high-brightness pixels with smaller areas, and to avoid interference of these noise points with defective region detection results. The morphological closing operation (using a circular structural element with the radius of 5) enables the mask of the defect area to be more coherent and smooth by filling small holes in the thermodynamic diagram, and particularly, the morphological integrity of the candidate area can be remarkably improved in the places where the boundaries of the mounting surface of the pushing component or the foam adhesive attachment area are complex. Geometric features such as the area, the perimeter, the circularity and the like of each connected region are calculated in morphological operation, and false detection regions are effectively filtered by setting a threshold (for example, removing regions with the area smaller than 30 pixels or the circularity larger than 0.9), so that the output candidate defect region mask only comprises regions which are considered to be most likely to have defects by a model.
Step S104, performing multi-scale hierarchical analysis on the candidate defect area mask to obtain a defect risk heat map, and performing clamping quality analysis on the connection area of the clamping block and the clamping groove according to the defect risk heat map to obtain a clamping quality evaluation result.
Specifically, multi-scale hierarchical analysis is performed based on the candidate defect region mask to obtain detection results of different precision and granularity. The multi-scale hierarchical analysis adopts a three-level progressive method, and the mask of the candidate defect region is decomposed into a first-scale hierarchical region, a second-scale hierarchical region and a third-scale hierarchical region, so that the defect region can be accurately identified and positioned in the whole-to-local and coarse-grained-to-fine process. The first scale level region mainly retains the whole image information of the decorative plate, has a larger visual field range, and provides a rough global detection visual angle, so that obvious defect characteristics are rapidly identified on the whole structure of the decorative plate. when coarse-granularity defect detection is carried out on the first-scale hierarchical region, the constructed defect classification model is utilized to rapidly scan the whole decorative plate image, and a preliminary defect judgment result is generated by analyzing the highlight region in the candidate defect region mask and combining the classification result output by the model. The defect categories present in the image are determined by Softmax classification output of the model and the classification results are mapped back to specific locations in the image to generate a preliminary thermodynamic diagram. According to the preliminary defect judging result, the analysis range is reduced to a second scale level region, the analysis at this stage focuses on a local image of the joint of the arc-shaped plate and the placing plate, and the model is enabled to execute more accurate defect positioning operation on a specific key region on a finer scale through the input of image data with higher resolution. In the process, a Grad-CAM technology is applied to generate a refined thermodynamic diagram of a local area, and a morphological analysis method is combined to remove the false detection area by calculating geometric features such as area, perimeter, shape complexity and the like of the area, so that the accuracy and reliability of a positioning result are ensured. And carrying out defect positioning on the third-scale hierarchical region according to the defect positioning result of the joint of the arc-shaped plate and the placing plate, and accurately calculating specific position data of the target defect. The model uses a high-resolution small image data set to infer, and the specific position coordinates of the target defect are obtained by analyzing smaller details (such as micro cracks or assembly errors on the surface of the pushing component) in the image. Dividing the automobile decorative plate into a plurality of grids, and mapping the preliminary defect judging result, the defect positioning result of the connecting part of the arc-shaped plate and the placing plate and the target defect position data into the grids to count the historical defect thermodynamic diagram accumulated value in each grid so as to obtain the defect frequency distribution data of different areas of the decorative plate. The grid division adopts a grid mode with fixed size, for example, a decorative plate image is divided according to 10 multiplied by 10 grids, each grid represents a specific space area, and the probability distribution of defects of the decorative plate in different areas is quantified by counting the accumulated situation of defect thermal values in each grid. And carrying out standardized processing on the defect frequency distribution data and classifying risk levels to generate a defect risk heat map. By normalizing the cumulative thermal value of each grid to a range of 0 to 1 and marking the thermal value as extremely low risk between 0 and 0.2, low risk between 0.2 and 0.4, medium risk between 0.4 and 0.6, high risk between 0.6 and 0.8, and extremely high risk between 0.8 and 1.0 according to preset risk classification criteria, quantitative risk assessment results are provided, and the quality status of each region of the decorative plate is intuitively displayed through visual codes of different colors (e.g. blue for low risk and red for high risk). And determining a high risk area according to the defect risk heat map, and carrying out clamping quality analysis on the connection area of the clamping block and the clamping groove to obtain a clamping quality assessment result. In this process, a grid showing a high risk (usually red area) in the thermodynamic diagram is focused, and whether the problem of poor engagement exists is evaluated by analyzing the connection state of the clamping block and the clamping groove in the areas. The specific analysis method comprises the steps of calculating texture features, edge features and illumination reflection modes of the surface of the connection area, and identifying the problems of clamping deviation, connection looseness or assembly errors and the like through comparison with features of good samples. Based on the method, a support vector machine or other classification algorithms are adopted to divide the clamping quality into four grades of good, general and bad, and a quantitative clamping quality assessment score is output.
In this embodiment, a high risk region with a risk value greater than a preset target value is screened out according to the defect risk heat map. A risk threshold is set, e.g. 0.8, and a region is marked as a high risk region when its risk value exceeds this threshold. The plastic foam plastic decorative plate comprises a push block, a rotating groove, an arc plate, a cavity structure corner, a foam plastic attaching area and other areas which are easy to generate defects due to mechanical movement, stress concentration or material characteristics in the decorative plate structure. And extracting the connection area of the clamping block and the clamping groove in the high risk area to obtain the surface image of the connection area. The image extraction process combines the candidate defect area mask generated in the earlier stage and the coordinate data of the high risk area, and cuts out the images in the areas by locating the specific positions of the clamping blocks and the clamping grooves in the decorative plate image, so that the extracted images only contain the effective information of the surfaces of the clamping parts, and the interference of surrounding backgrounds is avoided. and performing multi-feature extraction operation on the surface image of the connection region to construct a surface morphology feature set. In the feature extraction process, the texture features of the surface are calculated, for example, the gray pattern distribution of the pixels of the surface is extracted through a Local Binary Pattern (LBP) operator, and the statistical features such as energy, contrast, correlation, entropy value and the like are obtained, and can effectively reflect the material consistency and texture regularity of the clamping blocks and the surfaces of the clamping grooves. And extracting surface edge characteristics, calculating edge density, directionality and continuity by adopting an improved Canny operator, and further deducing the assembly precision of a clamping block and a clamping groove and the stability of a clamping state by analyzing the rule degree of edge lines of the characteristic identification clamping area. In terms of color feature extraction, an image is converted into an HSV color space, and the average value, variance, skewness and kurtosis of hue, saturation and brightness are calculated respectively, wherein the color features play an important role in identifying the illumination reflection characteristics of the surface of a connecting area and judging whether the clamping state causes uneven surface color due to mechanical deformation. In order to more comprehensively analyze the clamping quality, depth features of the region are extracted through a deep learning model, and the high-dimensional features effectively capture tiny morphological changes of the surface, particularly potential quality problems in the connecting region, which are difficult to detect through a traditional image feature extraction method. And constructing a mapping relation model of the clamping state and the surface morphology, and correlating the surface characteristics of the connecting area with the actual clamping quality state so as to realize quantitative analysis of the clamping quality. When the mapping relation model is constructed, a large number of sample data of known clamping states are analyzed, and the distribution rule of the surface morphology features under different clamping states is extracted. For example, in the complete engagement state, the connection area of the clamping block and the clamping groove presents regular texture distribution, the edge lines are clear and continuous, the illumination reflection mode is uniform, and no obvious deformation trace exists. In the case of insufficient engagement or poor assembly, the surface of the connection region is slightly deformed due to the influence of mechanical stress or loosening of the structure, so that the illumination reflection mode is changed, for example, an irregular highlight or shadow region is formed. The texture features of the surface may also deviate from normal ranges, such as edge line breaks, disordered texture distribution, or fine cracks. labeling and modeling the surface features and the clamping state, constructing a multi-dimensional mapping relation model, and mapping the input surface feature set into a specific clamping quality state through model prediction. And carrying out clamping quality analysis based on the surface morphology feature set and the mapping relation model to obtain a clamping quality assessment result. And inputting the surface characteristics of the connection area in the high risk area into a mapping relation model, and obtaining a prediction result of the clamping state through model reasoning. The engagement quality is classified into a plurality of classes, such as excellent, good, general, bad, etc., by using a classification algorithm such as a support vector machine, a random forest, a deep neural network, etc. In the evaluation process, a classification result of the engagement quality is output, and a quantitative score is provided, for example, the degree of the engagement quality is expressed in the form of a percentage. in the analysis result, the good engagement state is judged to be a poor engagement state when the score is greater than 90 minutes, the good state is between 80 and 90 minutes, the general state is between 70 and 80 minutes, and the score is lower than 70 minutes.
According to the embodiment of the application, the multi-mode feature extraction and fusion technology is used for effectively capturing multi-dimensional information on the surface of the decorative plate and comprehensively reflecting the feature expression of different types of defects, the problem of insufficient defect samples of the decorative plate is solved based on a ResNet-v1-50 migration learning strategy, the classification precision of the joint of an arc plate and a placement plate is remarkably improved, the gradient weighting type activation mapping technology is combined with multi-level feature fusion, the accurate positioning of a defect area is realized, the multi-scale analysis framework is used for progressively analyzing from a large image to a small image, the detection resolution is gradually improved along with the refinement of analysis scale, so that the defect detection is more accurate, the clamping state between decorative plate assemblies is creatively evaluated based on the surface morphology feature, the clamping quality assessment can be carried out without disassembly, a feasible scheme is provided for the online detection of a production line, and the data support is provided for the optimization of the production process by constructing the risk levels of different areas of the defect risk heat image quantification.
In a specific embodiment, the process of executing step S101 may specifically include the following steps:
acquiring an image of an automobile decorative plate by using an industrial camera to obtain an original decorative plate image, and carrying out size standardization processing on the original decorative plate image to obtain a standardized size image;
performing histogram equalization on the standardized size image to obtain a contrast enhanced image, and performing non-local mean filtering on the contrast enhanced image to obtain a denoising image;
Performing self-adaptive threshold segmentation on the denoising image to obtain a decorative plate binary mask image, and performing background removal operation on the denoising image by using the decorative plate binary mask image to obtain a pure decorative plate image;
and performing image enhancement processing on the clean decorative plate image to obtain standardized image data.
Specifically, the image acquisition system is constructed to ensure that the original image acquired has sufficient resolution and quality. The image acquisition system comprises a high-resolution industrial camera, an LED light source array with specific wavelength and a precise positioning device. The industrial camera adopts 1920×1080 pixel camera to ensure the richness of image details. In the aspect of light source selection, an annular LED light source with the color temperature fixed at 5500 K+/-100K is adopted, and the light source can provide a uniform and stable illumination environment and avoid image quality fluctuation caused by light source change. Through the mode of automatic triggering, when the automobile decorative board passes through the conveyer belt and reaches the position of predetermineeing, the front view of decorative board is caught in the system to realize automatic image acquisition in the high-efficient operation of production line. And carrying out size standardization processing on the original image to obtain a standardized size image with uniform specification. All images are adjusted to 1280 multiplied by 720 pixel sizes, and image proportion differences caused by different decorative plate sizes and camera distance changes are eliminated through size normalization, so that the stability of an image processing algorithm is improved. In the process of realizing size standardization, a bilinear interpolation algorithm is adopted, and the image with the target size is generated by carrying out weighted calculation on the pixels of the original image, and meanwhile, the details and the edge definition in the image are kept as unaffected as possible. And carrying out histogram equalization processing on the standardized size image to obtain a contrast enhancement image. Histogram equalization is an image enhancement technique based on pixel gray value distribution, which linearizes the cumulative distribution function of the image gray values, so that the brightness distribution of the image is more uniform. In mathematical terms, histogram equalization is achieved by the following formula:
Wherein,Representing the pixel value after the enhancement,Is the number of gray levels of the image, typically 256,AndThe width and height of the image respectively,Is a gray value ofIs used for the number of pixels of a display device,Calculating that the pixel value is not greater thanIs used for the number of accumulated pixels. A non-local mean filtering operation is performed on the contrast enhanced image to obtain a denoised image. The non-local mean filtering is used for carrying out weighted average on similar pixels by calculating the similarity between each pixel and all pixels in the image, so that Gaussian noise and spiced salt noise are effectively removed while the edges and details of the image are reserved. The calculation formula of the non-local mean filtering is as follows:
Wherein,Representing pixelsThe new value after the filtering is used for the filtering,Is a pixelIs used for the search window of (a),Is a pixelIs used for the gray-scale value of (c),Is a pixelAndThe calculation of the weight coefficient is based on the similarity of the pixel neighborhood, and is specifically defined as:
Wherein,AndRespectively pixelsAndIs used for the block of the neighborhood of (c),The distance of the euro type is expressed,Is a filtration intensity control parameter. The noise is effectively restrained through the image after non-local mean value filtering, and the definition of the edge of the decorative plate can be kept. An adaptive thresholding is performed on the denoised image to generate a binary mask map of the trim panel. The self-adaptive threshold segmentation method dynamically calculates the segmentation threshold of each pixel by analyzing the brightness change of the local area of the image, and is suitable for the scene with uneven illumination. An image is divided into a plurality of tiles and a pixel gray level histogram for each tile is calculated to determine an optimal segmentation threshold for the region. In the generated binary mask pattern, white pixels represent the decorative plate area, and black pixels represent the background area, and the decorative plate is effectively separated from the complex background by the mask pattern. And performing background removal operation on the denoising image by using the binary mask image of the decorative plate to obtain a clean decorative plate image. In the background removal process, the background part pixel value in the mask map is set to zero, and only the pixel data of the decorative plate part is reserved, so that only the effective information of the decorative plate is left in the processed image. Image enhancement processing is performed on the clear trim panel image to obtain normalized image data. In the image enhancement operation, a contrast-limited adaptive histogram equalization method is used to enhance the local contrast of the image and avoid the phenomenon of excessive enhancement that occurs in conventional histogram equalization. And enhance the edge characteristics and brightness balance of the image through sharpening filtering and gamma correction techniques. For example, small scratches or imperfections are made more pronounced by sharpening filtering to enhance the decorative panel surface detail, while gamma correction enables images to remain well visible under different brightness conditions by adjusting the non-linear mapping of pixel values.
In a specific embodiment, the process of executing step S102 may specifically include the following steps:
inputting standardized image data into a U-Net network for feature extraction to obtain a multi-scale feature map, wherein the U-Net network comprises four layers of encoders and four layers of decoders, each layer of encoder uses ResNet modules with cavity convolution to extract features, and each layer of decoder restores the space dimension of the feature map through transposed convolution;
Performing network training based on the multi-scale feature map to obtain a region segmentation model;
Dividing the standardized image data by using a region division model to obtain a mask map of a target region of the decorative plate, wherein the target region of the decorative plate comprises a joint of an arc-shaped plate and a placing plate, a pushing component mounting surface, a locking groove periphery, a clamping block and clamping groove meshing region and a foam adhesive attachment region;
calculating morphological characteristics and texture characteristics based on the target area mask map of the decorative plate to obtain a primary characteristic set, and extracting edge characteristics and color characteristics according to the target area mask map of the decorative plate to obtain a medium-grade characteristic set;
and extracting depth features from the target area mask map of the decorative plate to obtain a target feature vector, and performing feature selection on the primary feature set, the intermediate feature set and the target feature vector to obtain a feature vector set.
In particular, an improved U-Net network architecture is constructed that includes four layers of encoders and four layers of decoders, each layer of encoders using ResNet modules with hole convolutions to enhance feature extraction capabilities, while the decoder portion progressively restores the spatial dimensions of the feature map by transposed convolutions, thereby preserving the spatial details of the image while maintaining high semantic information. The introduction of the convolution with the cavity enables the receptive field to be exponentially expanded under the condition of not increasing the calculated amount by inserting the cavity into the convolution kernel, which is particularly important in the segmentation model, because the multiscale characteristics of complex structures such as an arc plate, a pushing assembly, a clamping groove and the like in the automobile decorative plate can be captured. In the feature extraction process of the U-Net network, standardized image data are processed through four layers of encoders, each layer of encoder is composed of ResNet modules with hole convolution, and the gradient vanishing problem in the depth network is relieved through residual connection. The calculation formula of the convolution with the cavity is as follows:
Wherein,Representing the output characteristic diagram in coordinatesThe pixel value at which it is located,Is the weight of the convolution kernel,Is the pixel value of the input sign graph,Is half the size of the convolution kernel,Is the void fraction, m represents the first index variable of the convolution kernel and n represents the second index variable of the convolution kernel. The existence of the void ratio enables the convolution kernel to extract features in a larger receptive field, for example, when r=2, the convolution kernel skips one pixel to perform convolution operation, so that the model obtains wider image information without losing detail. In ResNet modules, by introducing cavity convolution, the model can extract large-scale contour information of the surface of the decorative plate and capture small-scale fine features such as the periphery of a locking groove, a meshing area of a clamping block and the clamping groove and the like. After the encoder processing is completed, the multi-scale feature map is passed to a decoder section, which restores the spatial resolution of the feature map by transpose convolution, making the output mask map consistent with the original image size. And each layer of decoder fuses the corresponding encoder feature map into the decoding process through jump connection while recovering the space dimension, so as to realize multi-scale fusion of the features. Through supervised learning training on the U-Net network, a mask map of a target area of the decorative board is output in a semantic segmentation mode, and the mask map accurately marks the joint of the arc-shaped board and the placing board. The key parts of the pushing component mounting surface, the periphery of the locking groove, the meshing area of the clamping block and the clamping groove, the foam adhesive attachment area and the like. Morphological features and texture features are calculated based on the trim panel target region mask map to construct a primary feature set. In morphological feature calculation, features such as area, perimeter, aspect ratio, circularity and the like of a region are extracted by analyzing connected regions in a mask map. For example, the calculation formula of the circularity is:
Wherein,The degree of circularity is indicated as such,Is the area of the region and,Is the area perimeter. When (when)The closer to 1, the closer to circular the shape of the area is to be described, which is particularly effective in detecting the area where the foam adheres, since the foam is generally distributed in a circular or regular geometric shape on the surface of the trim panel. For extraction of texture features, a local binary pattern operator is adopted, and binary patterns are generated by comparing gray values of pixels and surrounding pixels, and the patterns can effectively reflect texture information of a region, such as abrasion or irregular texture change on a push component mounting surface by calculating entropy identification of textures. And after the primary feature set is finished, extracting edge features and color features according to the target area mask map of the decorative plate to obtain a medium-level feature set. In the process of extracting the edge features, an improved Canny operator is applied to perform edge detection operation on the target region, and the structural features of the region are evaluated by calculating indexes such as edge density, directionality, continuity of edge length and the like. For example, the entropy of the edge direction distribution is calculated by the following formula:
Wherein,The entropy of the direction is indicated,Is the first in the edge direction histogramThe probability distribution of the individual directions is determined,Is the total number of directions. When the directional entropy is lower, the edge lines are more regular. Extraction of color features statistical information, including mean, variance, skewness, and kurtosis, of hue, saturation, and brightness of pixels within a region is calculated by converting an image into an HSV color space. The multidimensional color characteristics can identify the color consistency of the surface materials of the decorative plate, and judge whether the problems such as adhesive residues or color pollution exist at the joint of the arc-shaped plate and the placing plate or not through a color change mode. Depth features are extracted from a trim panel target area mask map to obtain a target feature vector. In the depth feature extraction process, a pre-trained EFFICIENTNET-B0 model is used, the mask region image is input into a deep learning network, and 1280-dimensional high-dimensional feature vectors are extracted through a convolutional neural network. These feature vectors contain spatial and texture information of the image and incorporate abstract semantic features learned by the depth model on a large-scale dataset. For example, by an activation pattern in the depth feature, small defects in a complex scene, such as micro-cracks around the lock groove or assembly errors in the snap-in area, are identified. Feature selection is performed on the primary feature set, the intermediate feature set, and the target feature vector to obtain an optimized feature vector set. In the characteristic selection process, the contribution degree of each characteristic to the defect identification of the decorative plate is calculated by adopting a random forest algorithm, redundant characteristics with lower contribution degree are removed by analyzing the importance score of the characteristic, and only the characteristic with the most distinguishing capability is reserved. For example, the importance of each feature in the classification task is quantitatively evaluated by using a Gini coefficient or information gain method, so that the feature vector set has comprehensiveness and high efficiency.
In a specific embodiment, the process of executing step S103 may specifically include the following steps:
Constructing a decorative plate defect classification network based on ResNet-v1-50, wherein the decorative plate defect classification network comprises a feature extraction convolution layer, a global average pooling layer, two full-connection layers and a Softmax classification layer;
carrying out regional extraction on the standardized image data to obtain a first image data set, a second image data set and a third image data set, wherein the first image data set comprises a complete decorative plate image, the second image data set is focused on a region image of the joint of the arc-shaped plate and the placement plate, and the third image data set is focused on a region image of the pushing component;
Respectively corresponding the feature vector set with the first graph data set, the second graph data set and the third graph data set to obtain model training data;
Performing a first-stage training on the decorative plate defect classification network, extracting convolutional block parameters in the convolutional layer by locking features, and performing parameter optimization on only two fully-connected layers to obtain a classification model which is preliminarily adapted to decorative plate data distribution;
performing second-stage training on the classification model which is preliminarily adapted to the data distribution of the decorative plate, unlocking all network layer parameters, and performing whole network fine tuning to obtain an optimized classification model;
respectively reasoning the test parts of the first graph data set, the second graph data set and the third graph data set based on the optimized classification model to obtain a defect classification model;
and calculating gradient weighted class activation mapping according to the defect classification model to obtain a candidate defect area mask.
Specifically, an efficient classification network architecture is designed, and a deep learning model capable of accurately identifying defects of an automobile decorative plate is constructed by taking ResNet-v1-50 as a basis for feature extraction and introducing a feature extraction convolution layer, a global average pooling layer, two full-connection layers and a Softmax classification layer. In the network structure, the part of the ResNet-v1-50 characteristic extraction convolution layer is composed of 50 convolution layers with residual connection, and each residual module realizes the addition operation of front and back characteristic diagrams through jump connection, thereby effectively solving the gradient vanishing problem in the deep network. In the feature extraction process, multi-scale features in the decorative plate image are extracted through convolution operation, and the perceptibility of the model to high semantic information in the image is gradually improved through feature image stacking and downsampling operation. The global averaging pooling layer is used to compress the feature map from a spatial dimension (e.g., 7×7×2048) to a one-dimensional vector (e.g., 2048), which reduces the number of parameters of the model and improves the robustness of the model to overfitting. In the full-connection layer design, the model adopts a two-layer structure, the 2048-dimensional feature vector is reduced to 1024 dimensions through linear transformation, then is further reduced to 256 dimensions, and the probability distribution (good products/defects) of two classifications is output through a Softmax classification layer. The calculation formula of the Softmax classification layer is as follows:
Wherein,Representing an input imageBelonging to the target defect classIs a function of the probability of (1),Is the class of full connection layer outputThe corresponding activation value is used to determine,Is the total category number (2 in this task, indicating good and defect), e is the index parameter in the sum formula. The real activation value of the model output is converted to a probability distribution between 0 and 1 by Softmax operation. After the classification network is built, the normalized image data is extracted in regions by segmenting the original dataset into a first graph dataset. The second graph data set and the third graph data set realize multi-scale image analysis. The first image data set contains images of the complete decorative plate for coarse granularity assessment of the whole quality, the second image data set focuses on the joint of the arc-shaped plate and the placement plate, the region is a key part with stress concentration and easy assembly error generation, the third image data set focuses on the pushing component region, and the existing fine defects such as abrasion or poor clamping of the component surface are captured with higher resolution. In the regional image data sets, the feature vector sets are in one-to-one correspondence with each data set, and the model is subjected to targeted training according to the data characteristics of images of different scales, so that high defect detection precision is maintained on the global scale and the local scale. After data preparation is completed, performing first-stage training on the decorative plate defect classification network, locking the parameters of the convolution blocks in the feature extraction convolution layer, and performing parameter optimization on only two fully connected layers. The method only adjusts the parameters of the newly added full connection layer by freezing the weights of the first four convolution modules in ResNet-v1-50, so that the model is quickly adapted to the distribution characteristics of the decorative plate image data on the basis of keeping the original feature extraction capability. In the first stage training, the model adopts a cross entropy loss function, and the calculation formula is as follows:
Wherein,The loss value is indicated as such,Is the one-time thermal encoding of the actual tag,Is a model predictive categoryQ is the number of samples. The loss function guides the updating direction of model parameters by measuring the distance between the model predictive probability distribution and the actual labels, so that the full-connection layer gradually learns the characteristic mode related to the defects in the decorative plate image. In specific training, a random small-batch gradient descent algorithm is adopted, rapid convergence is realized in an early training stage of the model by setting a larger initial learning rate (such as 0.001), and parameters are updated at a more stable speed when the model approaches an optimal solution through a learning rate attenuation strategy. After the first-stage training is completed and a classification model which is preliminarily adapted to the data distribution of the decorative plate is obtained, the second-stage training is carried out, all network layer parameters are unlocked at the moment, and full-network fine tuning is carried out on the whole model so as to obtain an optimized classification model. In this stage, the model no longer adjusts the weights of the full connection layer only, but allows all convolution layer parameters to participate in updating through a smaller learning rate (e.g., 0.0001), so that the model feature extraction part can focus more on detail features in the decorative plate image, such as micro cracks at the edge of the arc plate or assembly deviation of the fixture block and the clamping groove meshing area. To prevent overfitting, the model introduces regularization techniques (e.g., dropout and weight decay) into the training process, making the model more focused on global features rather than relying on a single feature by randomly discarding some neurons in the fully connected layer (e.g., dropout ratio of 0.5). The training termination condition of the model is set to verify that the set accuracy automatically stops training when there is no boost for five consecutive cycles to avoid over-fitting of the model on the training data. After the optimized classification model is obtained, the test parts of the first graph data set, the second graph data set and the third graph data set are respectively inferred based on the model, whether the decorative board image has defects is judged through classification output of the model, and a specific defect type prediction result is output. In the reasoning stage, finer defect positioning information is obtained by analyzing classification score distribution of the model on each pixel point. Furthermore, to enable visualization and accurate localization of defect regions, a gradient weighted class activation map (Grad-CAM) is calculated based on the optimized classification model to generate a mask of candidate defect regions. In Grad-CAM calculation, an objective score function is definedA score representing the model predictive image belonging to the defect class is then calculated against the model's final layer of convolution layer feature mapIs a gradient of (2):
Wherein,Is a feature map channelIs used for the weight of the (c),Is the spatial size of the feature map,Representing object score versus feature map at coordinatesIs used for the gradient of (a),To the coordinates of the imageIs a convolution feature map of (1). A two-dimensional thermodynamic diagram is generated by multiplying and summing the weights and the feature map pixel by pixel, and negative values are zeroed by a ReLU activation function to highlight the defect regions of most interest to the model, resulting in candidate defect region masks.
In one embodiment, the performing step calculates a gradient weighted class activation map according to the defect classification model, and the obtaining the candidate defect region mask may specifically include the steps of:
Defining a target score function according to the defect category in the defect classification model, and calculating a gradient value of the target score function on a convolution kernel corresponding to a convolution layer feature map in the defect classification model to obtain target gradient data;
performing global average pooling operation on the target gradient data in the space dimension to obtain importance weights of the feature map channels;
Multiplying the importance weights of the feature map channels with the corresponding feature maps and summing the feature map channel importance weights in the channel dimension to obtain an initial thermodynamic diagram;
Generating a multi-scale thermodynamic diagram based on an intermediate layer characteristic diagram of the defect classification model, and carrying out weighted fusion on the initial thermodynamic diagram and the multi-scale thermodynamic diagram according to a preset weight coefficient to obtain an optimized thermodynamic diagram;
and performing morphological open operation and closed operation on the optimized thermodynamic diagram to obtain candidate defect area masks.
In particular, an objective score function is defined in the defect classification model for a particular defect class, which function is used to quantify the confidence that the model belongs to a particular defect class in the predicted image. Objective scoring functionDefined as the category of defects with the target in the Softmax classification layerCorresponding output values. For example, for a classification model (good/defective), if the model predictive image is defective, the probability isThe objective score function is expressed as:
Wherein,Is the target defect category output by the model at the full connection layerIs used for the activation value of (a),Is the total number of classification categories, b is the index parameter in the sum formula. By selecting a particular one(E.g., a "defect" category) focusing only on this category's response when calculating the gradient. After defining a target score function, calculating a gradient value of the score function on a convolution kernel corresponding to a convolution layer feature map in the model to obtain target gradient data. In the gradient calculation process, selecting a characteristic diagram of a convolution layer of the last layer of the modelAs an object of gradient calculation, the mathematical expression of the gradient is:
Wherein,Representing objective scoring functionFor characteristic diagramIn positionA gradient of pixel values. By calculating this gradient, the contribution of each pixel in the feature map to the model predicted defect class is quantified, with a larger gradient value indicating that the pixel plays a more critical role in classification decisions. This process is implemented in combination with a back-propagation algorithm, which transfers the error from the output layer to the target convolution layer by the chain law, thus calculating the gradient values for all pixels. After the target gradient data are obtained, a global average pooling operation is performed on the gradient data in the spatial dimension to calculate the importance weights of the feature map channels. In convolutional neural networks, the feature map has multiple channels (e.g., of sizeWhereinAndThe height and width of the feature map respectively,Is the number of channels), global averaging pooling reduces the gradient value of each channel to a scalar weight by averaging the feature map in the spatial dimensionThe calculation formula is as follows:
In the course of this formula (ii) the formula,Representing feature map channelsIs added to the importance of the weight of (a),Is the value of the gradient and,AndThe height and width of the feature map are respectively, i, j are respectively the index numbers of the corresponding summation formulas. Through global averaging pooling operation, the weight of each channel is proportional to the importance degree of the channel to the target class prediction, and the method can automatically learn the channel characteristics focused by the model in the decision process. The importance weights of the feature map channels are multiplied pixel-by-pixel with the corresponding feature map and summed over the channel dimensions to generate an initial thermodynamic diagram. Initial thermodynamic diagramThe calculation formula of (2) is as follows:
Wherein,Representing thermodynamic diagrams in positionThe value at which the value is to be calculated,Is a channelIs used for the weight of the (c),Is a characteristic diagramIn positionAt pixel values, the ReLU activation function is used to filter out values less than 0, K is the number of channels. Ensuring that the thermodynamic diagram retains only the regions that contribute positively to the target class. In the initial thermodynamic diagram generated, the highlight region represents the portion of the image in the model prediction that contributes more to the target defect class, by which the focused region of the model is visually displayed on the image, such as the location of potential defects in the area of the panel where the arcuate panel joins the placement panel or the pushing assembly is displayed in the trim panel image. In order to improve the accuracy of the thermodynamic diagram, a multi-scale thermodynamic diagram is generated based on the middle layer characteristic diagram of the defect classification model, and the initial thermodynamic diagram and the multi-scale thermodynamic diagram are subjected to weighted fusion according to a preset weight coefficient to obtain an optimized thermodynamic diagram. Feature maps of model middle layers (e.g., the res3d_brach2c, res4f_brach2c layers of ResNet) are selected, and Grad-CAM thermodynamic diagrams of the feature maps are calculated, respectively, which represent regions of interest of the model on different scale features. For example, in detecting the engagement of a clip with a slot of an automotive trim panel, the shallower features focus on local edge features, while the deeper features focus on overall texture and morphology patterns. By setting weight coefficients(E.g., [0.2,0.3,0.5 ]), the thermodynamic diagrams of different layers are weighted and fused, and the calculation formula is as follows:
Wherein,Is to optimize thermodynamic diagram positionThe pixel value at which it is located,Is the firstThe weighting coefficients of the layer thermodynamic diagram,Is the firstThe values of the respective thermodynamic diagrams at the corresponding locations,Is the number of feature maps involved in the fusion. By the multi-scale weighted fusion method, information of the model on different feature scales is effectively integrated, so that the generated optimized thermodynamic diagram not only maintains global information, but also has high sensitivity to detail areas. And performing morphological open operation and closed operation on the optimized thermodynamic diagram to obtain candidate defect area masks. In the image processing process, the open operation is used for eliminating small noise points, the isolated high-point points in the thermodynamic diagram are removed through the operation of corrosion and then expansion, and the closed operation fills small holes in the thermodynamic diagram through the operation of corrosion and then expansion, so that the form of a defect area is more complete and coherent.
In a specific embodiment, the process of executing step S104 may specifically include the following steps:
Performing multi-scale hierarchical analysis based on the candidate defect region mask to obtain a first-scale hierarchical region, a second-scale hierarchical region and a third-scale hierarchical region;
Coarse-grain defect detection is carried out on the first scale level region, and a preliminary defect judgment result is obtained;
Performing refined analysis on the second scale level region based on the preliminary defect judgment result to obtain a defect positioning result of the joint of the arc-shaped plate and the placing plate;
Performing defect positioning on the third-scale hierarchical region according to a defect positioning result of the joint of the arc-shaped plate and the placing plate to obtain target defect position data;
Dividing an automobile decorative plate into a plurality of grids, and counting a preliminary defect judgment result, a defect positioning result of the joint of the arc plate and the placing plate and a historical defect thermodynamic diagram accumulated value of target defect position data in each grid to obtain defect frequency distribution data;
Carrying out standardized treatment on the defect frequency distribution data and classifying risk levels to obtain a defect risk heat map;
and determining a high risk area according to the defect risk heat map, and carrying out clamping quality analysis on the connection area of the clamping block and the clamping groove in the high risk area to obtain a clamping quality assessment result.
Specifically, multi-scale hierarchical analysis is performed on the candidate defect region mask, and a progressive analysis method from whole to part is realized by decomposing an image into regions of different scales. Dividing the decorative board image into a first scale level region, a second scale level region and a third scale level region, wherein the three levels respectively correspond to different detection precision and target regions. For example, the first scale level region covers the whole decorative board image and is used for coarse-grained global analysis so as to rapidly screen out a significant defect region of the decorative board, the second scale level region focuses on the joint of the arc-shaped board and the placement board, the region is a part with concentrated mechanical stress and easy material deformation or assembly errors, the third scale level region further reduces the analysis range and focuses on the joint region of the pushing component and the clamping block and the clamping groove, and the system recognizes fine defects on the surface of the joint region, such as slight surface scratches, micro cracks at the joint or material abrasion marks through a higher-resolution image input model. And carrying out coarse-grain defect detection on the first scale level region to obtain a preliminary defect judgment result. And (3) rapidly scanning the whole decoration plate image through a defect classification model, and generating a preliminary defect thermodynamic diagram by using a statistical analysis method based on pixel activation values in combination with the highlight areas of the candidate defect area masks. In the mathematical model, the calculation formula of the preliminary defect thermodynamic diagram is expressed as follows:
Wherein,Representing the preliminary thermodynamic diagram in positionThe pixel value at which it is located,Is the number of defect masks involved in the analysis,Is the firstLocation of defect masksThe binary value at (1 indicates the presence of a defect, 0 indicates no defect),Is the probability value of the model predictive corresponding defect. By this method, the defective areas are identified in a global scope, and particularly in the complex texture background of the whole decorative plate, the defective high-risk areas can be rapidly positioned. And based on the preliminary defect judgment result, performing more refined analysis on the second scale level region to obtain a specific defect positioning result of the joint of the arc-shaped plate and the placement plate. In the process of fine analysis, a model adopts a multi-scale feature fusion method, a preliminary thermodynamic diagram is combined with a multi-scale feature diagram, a noise region is filtered through setting a higher detection threshold, and only pixels with high confidence of the model on defect types are reserved. For example, by introducing morphological analysis techniques, geometric features (e.g., area, perimeter, form factor, etc.) of the defect region are calculated to accurately delineate the defect profile at the junction of the arcuate plate and the placement plate. The calculation formula adopted in the analysis of the geometric features is similar to the above-mentioned connected regions in the analysis mask map, and will not be described here again. For example, when detecting the connection of the arcuate plate of the trim panel to the placement plate, the model can accurately locate the cracking or spalling of the material due to stress concentrations during assembly by analyzing these shape features. Based on the defect positioning result of the connection part of the arc-shaped plate and the placing plate, the analysis range is further reduced, the key points are transferred to the third-scale hierarchical region, and the refined defect positioning analysis is performed on the pushing assembly and the connection region of the clamping block and the clamping groove, so that target defect position data are obtained. At this stage, the image is analyzed at the pixel level by high resolution image data in combination with a deep learning model. For example, by the activation pattern in the feature map of the convolutional neural network, the model identifies the existence of micro-cracks, metal fatigue traces, or material aging phenomena in the connection region. Meanwhile, by introducing an edge detection operator (such as a Canny operator), the continuity and linearity of the edge of the connecting area are calculated, and whether the mechanical clamping state of the clamping block and the clamping groove is good or not can be judged in an auxiliary mode, for example, whether the clamping block is deformed or the clamping groove is worn or not is judged. The multi-level progressive analysis method ensures that the model can keep higher detection precision on both global and local scales, and particularly shows excellent defect recognition capability in a complex decorative plate structure. After the target defect position data are obtained, dividing the automobile decorative plate into a plurality of grids, and obtaining defect frequency distribution data by counting historical defect thermodynamic diagram accumulated values in each grid. For example, by dividing the trim panel image intoRecording the occurrence frequency of defects of each grid in different time periods, and calculating the accumulated value of the historical thermodynamic diagram:
Wherein,Is a grid locationThe frequency of the cumulative defects at that point,Is a period of time that is a period of time,Is the firstPhase thermodynamic diagram in positionA value at. Through the step, probability distribution of defects in different areas of the decorative plate is effectively quantified. And carrying out standardized processing on the defect frequency distribution data, and dividing risk grades to generate a defect risk heat map. In the normalization process, each grid is assigned a different color coding, e.g. blue represents low risk, red represents high risk, by normalizing the cumulative value to a range of 0 to 1 and according to a preset risk level criterion (e.g. 0 to 0.2 low risk, 0.2 to 0.4 medium risk, 0.4 to 0.6 high risk, 0.6 to 0.8 extremely high risk, 0.8 to 1.0 highest risk). After the high risk areas are determined, the connection areas of the clamping blocks and the clamping grooves in the areas are subjected to clamping quality analysis, and the quality of the clamping quality is accurately evaluated by analyzing the texture characteristics, the edge characteristics and the illumination reflection characteristics of the surfaces of the connection areas. For example, the extracted surface features are input into a clamping state classification model through a support vector machine algorithm, clamping quality is classified into four grades of good, general and bad, and an analysis result is compared with historical defect data by combining a high risk area in a risk heat map to judge whether an abnormal process problem exists in the current production process.
In a specific embodiment, the executing step determines a high risk area according to the defect risk heat map, and performs a clamping quality analysis on a connection area between the clamping block and the clamping groove in the high risk area, so as to obtain a clamping quality evaluation result, which may specifically include the following steps:
Screening a high risk area with a risk value larger than a preset target value according to the defect risk heat map, wherein the high risk area comprises the joint of the pushing block and the rotating groove, the edge of the joint of the arc plate and the placing plate, the corner of the cavity structure and the edge of the foam adhesive area;
extracting a clamping block and clamping groove connection area in a high risk area to obtain a connection area surface image, and extracting multiple features of the connection area surface image to obtain a surface morphology feature set;
Constructing a mapping relation model of a clamping state and a surface morphology, wherein the mapping relation model describes the characteristics of regular texture distribution and edge lines of the surface in a complete meshing state and the characteristics of irregular illumination reflection modes caused by deformation of the surface in a clamping insufficient state;
and carrying out clamping quality analysis based on the surface morphology feature set and the mapping relation model to obtain a clamping quality assessment result.
Specifically, a high risk area with a risk value larger than a preset target value is screened from the defect risk heat map. The defect risk heat map is generated by dividing the trim panel image into a plurality of grids based on the historical defect frequency distribution data, and counting the frequency distribution of occurrence of defects in each grid. In the risk screening stage, a risk threshold value is setFor exampleRisk valueAreas above the threshold are marked as high risk areas, which are expressed mathematically as:
Wherein,Representing the high risk mask map in positionIn the state, 1 represents a high risk region, and 0 represents a safety region. Risk valueThe high risk area is more likely to occur through the accumulated defect frequency data after standardized processing, such as the complex structure, mechanical stress concentration and easily worn parts of materials at the connection part of the pushing block and the rotating groove, the connection part edge of the arc plate and the placing plate, the corner of the cavity structure and the edge of the foam adhesive area. And precisely screening out a high-risk area needing to be focused on from the whole decorative plate image through a binary mask map based on the risk value. After the high risk areas are determined, image extraction is carried out on the connection areas of the clamping blocks and the clamping grooves in the areas so as to obtain the surface images of the connection areas. In the image extraction process, the high risk mask image and the original image are multiplied at the pixel level, only the pixel values in the high risk area are reserved, and the other parts are set to be zero, so that the connection area of the clamping block and the clamping groove is separated from the complex decorative plate background. The method can ensure that only the high-risk area is calculated in the subsequent analysis, effectively reduces the consumption of calculation resources and avoids the noise interference of the irrelevant area. After obtaining the surface images of the connected regions, the images are subjected to multi-feature extraction to construct a surface morphology feature set. During feature extraction, the surface geometry, texture features, edge features, and illumination reflectance properties are of interest. For example, by calculating the shape characteristics of the surface, the geometric characteristics of the connection area are effectively quantified, which is helpful for identifying whether the fixture block is deformed, whether the connection area is mechanically worn, and the like. In texture feature extraction, gray scale distribution features of surface pixels, such as energy, contrast, correlation and entropy values, are calculated by a local binary pattern method, which can reflect the material consistency of the surface, and potential defects in the connection region, such as material microcracks due to mechanical stress or surface wear due to long-term friction, are identified by a texture change pattern. The improved Cannv operator is used for carrying out edge detection on the image of the connecting area, calculating the density, the directionality and the continuity of the edge, and being helpful for evaluating the mechanical engagement state of the clamping block and the clamping groove, for example, in the complete engagement state, the edge line of the connecting area is represented as a regular parallel line or a closed boundary, and in the case of insufficient clamping or poor assembly, the edge characteristic is represented as a discontinuous, irregular or broken line characteristic. After the construction of the surface morphology feature set is completed, a mapping relation model between the clamping state and the surface morphology is established, and the model correlates the surface feature with the actual clamping quality state to realize quantitative quality assessment. In the mapping relation model, a mathematical model is established by collecting a large amount of sample data with known clamping states and using a classification algorithm such as a support vector machine or a random forest and the like. For example, in the fully engaged state, the edge density of the connection regionAnd texture energyIs generally higher, and the bias of the illumination reflection modeNear zero indicates that the illumination is uniformly distributed without significant highlighting or shadowing areas. Mathematical representation of model by classification decision functionExpressed as:
Wherein,Is a function of the prediction of the engagement state,、、Is the characteristic weight of the model and,Is the value of the offset and,Is the input surface morphology feature vector. By training a model, the prediction function is madeThe quality level can be outputted in different engagement states, for example, excellent, good, general, bad, etc. In the model reasoning process, the score of the clamping quality is calculated by inputting the surface morphology feature in the high risk area into the mapping relation model, and the clamping quality is classified into different grades according to the range of the score. For example, whenWhen the engagement state is good,The time is good, and the time is good,In general, if the value is lower than 0.4, the engagement state is determined to be poor. After the clamping quality evaluation result is obtained, the high risk area data in the risk heat map are combined, and whether the quality state in the current production process is stable or not is judged by analyzing the frequency distribution of the historical defects. For example, in certain grids of connection areas, if in a high risk state for a long period of time, adjustments to production equipment, process parameters or raw materials are required to reduce potential quality risks. And by comparing the current clamping quality evaluation result with the historical data, the potential problems existing in the production line, such as whether the specific equipment has consistent clamping deviation or whether the specific equipment has periodical assembly errors in the specific working procedure, are identified.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, systems and units may refer to the corresponding processes in the foregoing method embodiments, which are not repeated herein.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or partly in the form of a software product, or all or part of the technical solution, which is stored in a storage medium and includes several instructions for causing an automotive trim panel manufacturing process intelligent detection device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. The storage medium includes a U disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
While the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those skilled in the art that the foregoing embodiments may be modified or equivalents may be substituted for some of the features thereof, and that the modifications or substitutions do not depart from the spirit and scope of the embodiments of the invention.