CN107705334B - A kind of camera abnormality detection method and device - Google Patents

Abstract

Translated fromChinese

本发明公开一种摄像机异常检测方法及装置，通过计算机视觉方法实时检测摄像机是否发生异常，并在检测发生异常时进行报警。方法包括：根据预置的异常图像集和正常图像集对神经网络进行训练，得到用于确定图像中各像素所属类型的语义分割模型，类型的种类包含异常类型和正常类型；将摄像机拍摄的图像输入所述语义分割模型中，得到所述图像对应的特征信息；根据所述图像对应的特征信息判断所述摄像机是否异常，并在确定异常时进行报警。

The invention discloses a camera abnormality detection method and device. The computer vision method is used to detect whether the camera is abnormal in real time, and alarm when the abnormality is detected. The method includes: training a neural network according to a preset abnormal image set and a normal image set, to obtain a semantic segmentation model for determining the type of each pixel in the image, and the type of the type includes abnormal type and normal type; Inputting the semantic segmentation model to obtain feature information corresponding to the image; judging whether the camera is abnormal according to the feature information corresponding to the image, and alarming when the abnormality is determined.

Description

Translated fromChinese

一种摄像机异常检测方法及装置A kind of camera abnormality detection method and device

技术领域technical field

本发明涉及计算机视觉领域，特别涉及一种摄像机异常检测方法及装置。The invention relates to the field of computer vision, in particular to a method and device for detecting abnormality of a camera.

背景技术Background technique

在基于视觉的无人驾驶车辆、机器人等应用场景中，通常由装载在自动驾驶车辆、高级辅助驾驶车辆、机器人等上的摄像机拍摄周边的环境，再由决策控制单元根据摄像机拍摄的图像进行决策并控制自动驾驶车辆、高级辅助驾驶车辆、机器人等。因此，摄像头拍摄的图像质量对决策控制单元进行决策起着至关重要的作用。In vision-based unmanned vehicles, robots and other application scenarios, the surrounding environment is usually captured by cameras mounted on autonomous vehicles, advanced assisted driving vehicles, robots, etc., and then the decision control unit makes decisions based on the images captured by the cameras. And control autonomous vehicles, advanced assisted driving vehicles, robots, and more. Therefore, the image quality captured by the camera plays a crucial role in the decision-making control unit.

然而，在实际应用中，摄像机由于出现一些异常情况导致拍摄的图像质量较差，从而会影响决策控制单元作出决策的准确性，例如：摄像机镜头上有虫子尸体、粘稠液体污损、雨滴、镜头的曝光装置出现问题导致图像过曝光或欠曝光等异常情况。如图1A、图1B、图1C、图1D、图1E和图1F所示，分别为摄像机拍摄得到的异常图像，依次为包含不透明污损图像、包含雨滴图像、包含透明污损图像、包含昆虫液体图像、过曝光图像和欠曝光图像。因此，如何能够实时的检测出摄像机异常并进行报警则成为亟待解决的技术问题。However, in practical applications, the quality of the images captured by the camera is poor due to some abnormal conditions, which will affect the accuracy of the decision-making control unit, for example: there are bug corpses on the camera lens, viscous liquid contamination, raindrops, There is a problem with the exposure device of the lens, resulting in abnormal situations such as image overexposure or underexposure. As shown in Figure 1A, Figure 1B, Figure 1C, Figure 1D, Figure 1E, and Figure 1F, the abnormal images captured by the cameras are, respectively, including opaque stained images, raindrop images, transparent stained images, and insects. Liquid images, overexposed images, and underexposed images. Therefore, how to detect the abnormality of the camera in real time and issue an alarm has become an urgent technical problem to be solved.

发明内容SUMMARY OF THE INVENTION

鉴于上述问题，本发明提供一种摄像机异常检测方法及装置，通过计算机视觉方法实时检测摄像机是否发生异常，并在检测发生异常时进行报警。In view of the above problems, the present invention provides a camera abnormality detection method and device, which can detect whether the camera is abnormal in real time through a computer vision method, and alarm when the abnormality is detected.

本发明实施例，一方面，提供一种摄像机异常检测方法，该方法包括：An embodiment of the present invention, on the one hand, provides a camera anomaly detection method, the method comprising:

根据预置的异常图像集和正常图像集对神经网络进行训练，得到用于确定图像中各像素所属类型的语义分割模型，类型的种类包含异常类型和正常类型；The neural network is trained according to the preset abnormal image set and normal image set, and a semantic segmentation model for determining the type of each pixel in the image is obtained, and the type of the type includes abnormal type and normal type;

将摄像机拍摄的图像输入所述语义分割模型中，得到所述图像对应的特征信息；Input the image captured by the camera into the semantic segmentation model to obtain feature information corresponding to the image;

根据所述图像对应的特征信息判断所述摄像机是否异常，并在确定异常时进行报警。Whether the camera is abnormal is determined according to the feature information corresponding to the image, and an alarm is issued when the abnormality is determined.

本发明实施例，还提供一种摄像机异常检测装置，包括：An embodiment of the present invention further provides a camera anomaly detection device, including:

训练单元，用于根据预置的异常图像集和正常图像集对神经网络进行训练，得到用于确定图像中各像素所属类型的语义分割模型，类型的种类包含异常类型和正常类型；The training unit is used to train the neural network according to the preset abnormal image set and the normal image set, and obtain a semantic segmentation model for determining the type of each pixel in the image, and the types of the types include abnormal types and normal types;

信息获取单元，用于将摄像机拍摄的图像输入所述语义分割模型中，得到所述图像对应的特征信息；an information acquisition unit, configured to input the image captured by the camera into the semantic segmentation model to obtain feature information corresponding to the image;

异常检测单元，用于根据所述图像对应的特征信息判断所述摄像机是否异常，并在确定异常时进行报警。An abnormality detection unit, configured to judge whether the camera is abnormal according to the feature information corresponding to the image, and to give an alarm when the abnormality is determined.

本发明实施例，本发明实施例，一方面，能够实现实时检测摄像机是否异常以及在异常时进行报警，以便操作人员及时获知摄像机发生异常，并对摄像机进行及时处理；另一方面，通过异常图像集和正常图像集对神经网络进行训练，能够提高训练得到的语义分割模型的健壮性和准确性。The embodiment of the present invention, the embodiment of the present invention, on the one hand, can realize real-time detection of whether the camera is abnormal and alarm when it is abnormal, so that the operator can know the abnormality of the camera in time, and deal with the camera in time; on the other hand, through the abnormal image The neural network is trained on the set and normal image set, which can improve the robustness and accuracy of the trained semantic segmentation model.

附图说明Description of drawings

附图用来提供对本发明的进一步理解，并且构成说明书的一部分，与本发明的实施例一起用于解释本发明，并不构成对本发明的限制。The accompanying drawings are used to provide a further understanding of the present invention, and constitute a part of the specification, and are used to explain the present invention together with the embodiments of the present invention, and do not constitute a limitation to the present invention.

图1A～图1F为摄像机拍摄的异常图像；1A to 1F are abnormal images captured by the camera;

图2为本发明实施例中摄像异常检测方法的流程图；2 is a flowchart of a method for detecting abnormality in imaging in an embodiment of the present invention;

图3为本发明实施例中训练神经网络得到语义分割模型的示意图之一；3 is one of the schematic diagrams of training a neural network to obtain a semantic segmentation model according to an embodiment of the present invention;

图4为本发明实施例中训练神经网络得到语义分割模型的示意图之二；4 is the second schematic diagram of training a neural network to obtain a semantic segmentation model according to an embodiment of the present invention;

图5为本发明实施例中根据连续多帧图像对应的特征信息确定摄像机是否异常的示意图之一；FIG. 5 is one of the schematic diagrams of determining whether a camera is abnormal according to feature information corresponding to multiple consecutive frames of images in an embodiment of the present invention;

图6A～图6C为图5所示方法的一个具体实例的过程示意图；6A to 6C are process schematic diagrams of a specific example of the method shown in FIG. 5;

图7A～图7C为图5所示方法的另一个具体实例的过程示意图；7A to 7C are process schematic diagrams of another specific example of the method shown in FIG. 5;

图8为本发明实施例中根据连续多帧图像对应的特征信息确定摄像机是否异常的示意图之二；FIG. 8 is the second schematic diagram of determining whether a camera is abnormal according to feature information corresponding to multiple consecutive frames of images in an embodiment of the present invention;

图9为本发明实施例中根据一帧图像对应的特征信息判断摄像机是否异常的示意图之一；9 is one of the schematic diagrams of judging whether a camera is abnormal according to feature information corresponding to a frame of image in an embodiment of the present invention;

图10为本发明实施例中根据一帧图像对应的特征信息判断摄像机是否异常的示意图之二；10 is the second schematic diagram of judging whether a camera is abnormal according to feature information corresponding to a frame of image in an embodiment of the present invention;

图11为本发明实施例中摄像异常检测装置的结构示意图。FIG. 11 is a schematic structural diagram of a device for detecting abnormality in imaging according to an embodiment of the present invention.

具体实施方式Detailed ways

为了使本技术领域的人员更好地理解本发明中的技术方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都应当属于本发明保护的范围。In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described The embodiments are only some of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

本发明实施例提供的摄像机异常检测方法主要适用于自动驾驶车辆、高级辅助驾驶车辆、机器人等人工智能设备上。The camera anomaly detection method provided by the embodiment of the present invention is mainly applicable to artificial intelligence devices such as automatic driving vehicles, advanced assisted driving vehicles, and robots.

实施例一Example 1

参见图2，为本发明实施例中摄像机异常检测方法的流程图，该方法包括：Referring to FIG. 2, it is a flowchart of a method for detecting abnormality of a camera in an embodiment of the present invention, and the method includes:

步骤201、根据预置的异常图像集和正常图像集对神经网络进行训练，得到用于确定图像中各像素所属类型的语义分割模型，类型的种类包含异常类型和正常类型。Step 201: Train the neural network according to the preset abnormal image set and the normal image set to obtain a semantic segmentation model for determining the type of each pixel in the image, and the types of the types include abnormal types and normal types.

第一示例，所述异常图像集包括透明污损图像和不透明污损图像，所述正常类型为无污损，所述异常类型为有污损。In a first example, the abnormal image set includes transparent defaced images and opaque defaced images, the normal type is no defacement, and the abnormal type is defaced.

第二示例，所述异常图像集包括透明污损图像和不透明污损图像，所述正常类型为无污损，所述异常类型包括透明污损和不透明污损。In a second example, the abnormal image set includes transparent defaced images and opaque defaced images, the normal type is no defacement, and the abnormal type includes transparent defacement and opaque defacement.

第三示例，所述异常图像集包括过曝光图像和欠曝光图像，所述异常类型包括过曝光、欠曝光，所述正常类型为正常曝光。In a third example, the abnormal image set includes over-exposure images and under-exposure images, the abnormal types include over-exposure and under-exposure, and the normal type is normal exposure.

第四示例，所述异常图像集包括透明污损图像、不透明污损图像、曝光图像和欠曝光图像，所述正常类型包括正常曝光和无污损，所述异常类型包括有污损、无污损、过曝光和欠曝光。In a fourth example, the abnormal image set includes transparent stained images, opaque stained images, exposed images, and underexposed images, the normal types include normal exposure and no stains, and the abnormal types include stained, non-stained damage, overexposure and underexposure.

第五示例，所述异常图像集包括透明污损图像、不透明污损图像、曝光图像和欠曝光图像，所述正常类型包括正常曝光和无污损，所述异常类型包括透明污损、不透明污损、无污损、过曝光和欠曝光。In a fifth example, the abnormal image set includes transparent stained images, opaque stained images, exposed images, and underexposed images, the normal types include normal exposure and no stains, and the abnormal types include transparent stains, opaque stains damage, smear-free, overexposure and underexposure.

本发明实施例中，异常图像集、正常图像集、异常类型和正常类型，并不仅限于第一示例～第五示例，在此不再穷举。In this embodiment of the present invention, the abnormal image set, the normal image set, the abnormal type, and the normal type are not limited to the first example to the fifth example, and will not be exhaustive here.

本发明实施例中，异常图像集可预先通过镜头包含不同污损物的同类型摄像机(即后续被判断是否异常的摄像机同类型)分别采集图像得到。正常图像集可预先通过镜头上无污损的同类型摄像机采集图像得到。本申请不做详细描述。In this embodiment of the present invention, the abnormal image set may be obtained by separately collecting images from cameras of the same type whose lenses contain different contaminants (ie, cameras of the same type that are subsequently judged to be abnormal). The normal image set can be obtained in advance by collecting images from the same type of camera with no contamination on the lens. This application does not describe in detail.

用于训练语义分割模型的神经网络可以是具有语义分割功能的任意一种神经网络模型，本申请不做严格限定。The neural network used for training the semantic segmentation model can be any neural network model with a semantic segmentation function, which is not strictly limited in this application.

步骤202、将摄像机拍摄的图像输入所述语义分割模型中，得到所述图像对应的特征信息。Step 202: Input the image captured by the camera into the semantic segmentation model to obtain feature information corresponding to the image.

步骤203、根据所述图像对应的特征信息判断所述摄像机是否异常，并在确定异常时进行报警。Step 203: Determine whether the camera is abnormal according to the feature information corresponding to the image, and issue an alarm when the abnormality is determined.

在一个实施例中，前述步骤201中，具体可通过图3所示的流程实现，包括：In one embodiment, theforegoing step 201 can be specifically implemented through the process shown in FIG. 3 , including:

步骤201A、采用异常图像集对所述神经网络进行训练，得到初级语义分割模型。Step 201A, using the abnormal image set to train the neural network to obtain a primary semantic segmentation model.

步骤201B、将正常图像集中的正常图像输入所述初级语义分割模型中，输出正常图像对应的语义分割结果，并将语义分割结果有误的正常图像加入所述异常图像集中，以得到新的异常图像集。Step 201B, input the normal images in the normal image set into the primary semantic segmentation model, output the corresponding semantic segmentation results of the normal images, and add the normal images with wrong semantic segmentation results to the abnormal image set to obtain new abnormal images image set.

由于正常图像与异常图像相比，其采集难度更低，为了提高语义分割模型的训练效率，本发明实施例仅将通过初级语义分割模型后输出的检测有误的正常图像添加到异常图像集中。Since normal images are more difficult to collect than abnormal images, in order to improve the training efficiency of the semantic segmentation model, the embodiment of the present invention only adds normal images with incorrect detection output after passing through the primary semantic segmentation model to the abnormal image set.

步骤201C、采用新的异常图像集对所述初级语义分割模型进行训练，得到语义分割模型。Step 201C, using a new abnormal image set to train the primary semantic segmentation model to obtain a semantic segmentation model.

本发明实施例中，在通过异常图像集训练得到初始语义分割模型之后，还通过正常图像集对初始语义分割模型进行测试，将错误检测的图像(即正样本)挑选出来并为该图像的每个像素打上正常类型的标签，加入到异常图像集中；并基于新的异常图像集对初始语义分割模型进行训练得到语义分割模型，这样做的目的是，希望通过添加错误检测的正样本，提高语义分割模型的健壮性和降低语义分割模型的错误检测率。In the embodiment of the present invention, after the initial semantic segmentation model is obtained through the training of the abnormal image set, the initial semantic segmentation model is also tested through the normal image set, and the wrongly detected image (that is, the positive sample) is selected and used for each image of the image. Each pixel is labeled with a normal type and added to the abnormal image set; and the initial semantic segmentation model is trained based on the new abnormal image set to obtain a semantic segmentation model. The purpose of this is to add false positive samples to improve semantics. Robustness of segmentation models and reducing false detection rates of semantic segmentation models.

优选地，为进一步提高训练得到的语义分割模型的健壮性和准确性，本领域技术人员还可以基于图3所示流程的启示，得到如图4所示的替代方案来实现步骤201，方法包括：Preferably, in order to further improve the robustness and accuracy of the semantic segmentation model obtained by training, those skilled in the art can also obtain an alternative solution as shown in FIG. 4 based on the enlightenment of the process shown in FIG. 3 to implementstep 201. The method includes: :

步骤201A’、采用异常图像集对所述神经网络进行训练，得到初级语义分割模型。Step 201A', using the abnormal image set to train the neural network to obtain a primary semantic segmentation model.

步骤201B’、将正常图像集中的正常图像输入初级语义分割模型中，输出正常图像对应的语义分割结果。Step 201B', input the normal images in the normal image set into the primary semantic segmentation model, and output the semantic segmentation results corresponding to the normal images.

步骤201C’、确定语义分割错误的正常图像的数量，并根据所述数量判断是否满足预置的停止迭代条件，若否则执行步骤201D’，若是则执行步骤201F’。Step 201C', determine the number of normal images with wrong semantic segmentation, and judge whether the preset stop iteration condition is satisfied according to the number, if otherwise, performstep 201D', if so, performstep 201F'.

本发明实施例中，停止迭代条件可以为所述数量小于预置的数量阈值，还可以是所述数量与输入初始语义分割模型的正常图像总数量的比值小于预置的比例阈值。本申请不作严格限定，本领域技术人员可以根据实际需求进行灵活设置。In this embodiment of the present invention, the stop iteration condition may be that the number is less than a preset number threshold, or the ratio of the number to the total number of normal images input to the initial semantic segmentation model may be less than a preset proportional threshold. This application is not strictly limited, and those skilled in the art can flexibly set according to actual needs.

步骤201D’、将语义分割错误的正常图像添加到所述异常图像集中得到新的异常图像集，并基于新的异常图像集对初始语义分割模型进行训练，得到中级语义分割模型。Step 201D', adding the normal image with wrong semantic segmentation to the abnormal image set to obtain a new abnormal image set, and training the initial semantic segmentation model based on the new abnormal image set to obtain an intermediate semantic segmentation model.

步骤201E’、将所述中级语义分割模型作为初始语义分割模型，执行所述步骤201B’。Step 201E', take the intermediate semantic segmentation model as the initial semantic segmentation model, and execute thestep 201B'.

步骤201F’、将初级语义分割模型确定为语义分割模型。Step 201F', determine the primary semantic segmentation model as the semantic segmentation model.

本发明实施例中，前述步骤202～步骤203的具体实现可通过但不仅限于以下任意一种方式(方式1～方式2)实现：In this embodiment of the present invention, the specific implementation of the foregoingsteps 202 to 203 may be implemented in, but not limited to, any one of the following manners (mode 1 to mode 2):

方式1way 1

步骤202、将摄像机拍摄的连续多帧图像输入语义分割模型中，得到每帧图像对应的特征信息；Step 202: Input the continuous multi-frame images captured by the camera into the semantic segmentation model to obtain feature information corresponding to each frame of image;

步骤203、根据所述多帧图像对应的特征信息判断摄像机是否异常，并在确定异常时报警。Step 203: Determine whether the camera is abnormal according to the feature information corresponding to the multi-frame images, and alarm when the abnormality is determined.

在实际应用中，若摄像机发生异常则会导致在一段时间内连续拍摄的图像均会存在异常，因此该方式1结合连续多帧图像对应的特征信息判断摄像机是否异常更准确。In practical applications, if the camera is abnormal, the images captured continuously for a period of time will be abnormal. Therefore,method 1 is more accurate to judge whether the camera is abnormal by combining the feature information corresponding to the continuous multiple frames of images.

在一个实施例中，所述特征信息为所述语义分割模型softmax层输出的各像素所属类型；即将图像输入语义分割模型之后，将该语义分割模型的softmax层输出各像素所属类型作为特征信息。方式1中，所述步骤203中，根据所述多帧图像对应的特征信息判断所述摄像机是否异常具体实现可如下：将多帧图像中位于同一位置的多个像素拟化成一个虚拟像素，并根据所述多个像素所属类型确定所述虚拟像素所属类型，将所述虚拟像素作为语义分割结果图中相应位置的一个像素，依此得到语义分割模型；在判断所述语义分割结果图中包含所属类型为异常类型的像素的数量大于等于预置的数量阈值时，确定所述摄像机异常。所述数量阈值根据经验值确定，例如可根据引起异常类型的污损物确定，例如可通过实验，采用不同尺寸的污损物对摄像头镜头造成不同尺寸的污损区域，判断摄像机在不同尺寸的污损区域下拍摄得到的图像对决策控制单元作出决策的影响程度，根据影响程度达到预置影响程度的最小尺寸的无损区域确定数量阈值。如图5所示、连续N帧图像P1、P2、…、PN输入语义分割模型之后，从语义分割模型的softmax层输出对应的N个语义分割结果G1、G2、…、GN，根据语义分割结果G1、G2、…、GN得到语义分割结果图。以下以两个具体实例进行描述。In one embodiment, the feature information is the type of each pixel output by the softmax layer of the semantic segmentation model; that is, after the image is input into the semantic segmentation model, the softmax layer of the semantic segmentation model outputs the type of each pixel as the feature information. Inmode 1, instep 203, determining whether the camera is abnormal according to the feature information corresponding to the multi-frame images can be implemented as follows: a plurality of pixels located in the same position in the multi-frame images are simulated into a virtual pixel, and The type to which the virtual pixel belongs is determined according to the type of the plurality of pixels, and the virtual pixel is used as a pixel in the corresponding position in the semantic segmentation result graph, and a semantic segmentation model is obtained accordingly; When the number of pixels belonging to an abnormal type is greater than or equal to a preset number threshold, it is determined that the camera is abnormal. The quantity threshold is determined according to empirical values, for example, it can be determined according to the type of contamination that causes abnormality. For example, through experiments, contamination of different sizes can be used to cause contamination areas of different sizes to the camera lens, and it can be judged that the camera is in different sizes. The degree of influence of the images captured in the contaminated area on the decision-making control unit, and the number threshold is determined according to the non-destructive area whose degree of influence reaches the smallest size of the preset degree of influence. As shown in Figure 5, after consecutive N frames of images P1, P2, ..., PN are input into the semantic segmentation model, the corresponding N semantic segmentation results G1, G2, ..., GN are output from the softmax layer of the semantic segmentation model. According to the semantic segmentation results G1, G2, ..., GN get the semantic segmentation result graph. Two specific examples are described below.

实例1、假设异常图像集包括透明污损图像和不透明污损图像，所述正常类型为无污损，所述异常类型为有污损，语义分割模型对输入图像中的每个像素打类别标签，例如，像素所属类型为无污损则对应类别标签为0，像素所述类型为有污损则对应类别标签为1。以摄像头连续拍摄的4帧尺寸为4*4的图像为例，该4帧图像分别用P1、P2、P3和P4表示，如图6A所示，将该P1、P2、P3和P4中的像素按照从左到右、从上到下的顺序依次编号为x1、x2、…、x16。将P1、P2、P3和P4分别输入到语义分割模型中得到如图6B所示的语义分割结果，则将图P1、P2、P3和P4中的相同位置(即编号相同)的像素虚拟成一个像素，得到虚拟像素x1’、x2’、…、x16’；针对每个虚拟像素，则根据虚拟化该虚拟像素的4个像素所属类型确定所述虚拟像素所属类型。例如，方式1、统计该4个像素中所属类型为0的个数N1与所属类型为1的个数N2，将N1与N2中取值大于预置数值(该数值一般设置为N的1/2，以连续N帧图像为10帧为例，则该10帧图像中10个x1像素中为0的个数超过5时则确定相应的虚拟像素所属类型为0)的数量对应的类型作为虚拟像素所属类型.方式2，将N1与N2取值大的类型作为该虚拟像素所述类型，以像素x1’为例，该像素x1’对应的四个像素所属类型分别为0001，因此该像素x1’所属类型为0(即无污损)。依此得到如图6C所示的语义分割结果图。Example 1. Suppose the abnormal image set includes transparent defaced images and opaque defaced images, the normal type is no defacement, the abnormal type is defaced, and the semantic segmentation model classifies each pixel in the input image. , for example, if the type to which the pixel belongs is non-stained, the corresponding category label is 0, and if the type of the pixel is defaced, the corresponding category label is 1. Taking the 4-frame images with the size of 4*4 continuously captured by the camera as an example, the 4-frame images are represented by P1, P2, P3 and P4 respectively. As shown in Figure 6A, the pixels in P1, P2, P3 and P4 are They are numbered x1, x2, ..., x16 in order from left to right and top to bottom. Input P1, P2, P3 and P4 into the semantic segmentation model respectively to obtain the semantic segmentation result as shown in Figure 6B, then virtualize the pixels in the same position (ie the same number) in Figures P1, P2, P3 and P4 into one For each virtual pixel, the type to which the virtual pixel belongs is determined according to the type of the four pixels that virtualize the virtual pixel. For example, inmethod 1, count the number N1 of the 4 pixels whose type is 0 and the number N2 whose type is 1, and set the value of N1 and N2 to be greater than the preset value (this value is generally set to 1/ 2. Taking 10 consecutive N frames of images as an example, when the number of 0 in the 10 x1 pixels in the 10-frame image exceeds 5, it is determined that the corresponding virtual pixel belongs to the type of 0) and the corresponding type is used as the virtual pixel. The type to which the pixel belongs. Inmethod 2, the type with a large value of N1 and N2 is used as the type of the virtual pixel. Taking the pixel x1' as an example, the types of the four pixels corresponding to the pixel x1' are 0001 respectively, so the pixel x1 ' is of type 0 (ie no defacement). Accordingly, the semantic segmentation result graph as shown in FIG. 6C is obtained.

实例2、假设异常图像集包括透明污损图像、不透明污损图像、欠曝光图像和过曝光图像，所述正常类型为无污损、正常曝光，所述异常类型为有污损、过曝光和欠曝光，语义分割模型对输入图像中的每个像素打类别标签，例如，所属类型为无污损则对应类别标签为0，所属类型为有污损则对应类别标签为1，所属类型为过曝光则对应的类别标签为2，所属类型为欠曝光则对应的类别标签为4。以摄像头连续拍摄的6帧尺寸为4*4的图像为例，该6帧图像分别用P1、P2、P3、P4、P5、P6表示，如图7A所示，将该P1、P2、P3、P4、P5、P6中的像素按照从左到右、从上到下的顺序依次编号为x1、x2、…、x16。将P1、P2、P3、P4、P5、P6分别输入到语义分割模型中得到如图7B所示的语义分割结果，则将图P1、P2、P3、P4、P5、P6中的相同位置(即编号相同)的像素虚拟成一个像素，得到虚拟像素x1’、x2’、…、x16’；针对每个虚拟像素，则根据虚拟化该虚拟像素的6个像素所属类型确定所述虚拟像素所属类型，例如，统计该6个像素中所属类型为0的个数N1、所属类型为1的个数N2、所属类型为2的个数N3、所属类型为3的个数N4，将N1、N2、N3、N4中取值大的类型作为该虚拟像素所述类型，以像素x1’为例，该像素x1’对应的6个像素所属类型分别为0、1、1、1、2、3，因此该像素x1’所属类型为1(即有污损)，依此得到如图7C所示的语义分割结果图。Example 2. Suppose the abnormal image set includes transparent defaced images, opaque defaced images, underexposed images and overexposed images, the normal types are no defacement, normal exposure, and the anomaly types are defaced, overexposed and Under-exposure, the semantic segmentation model labels each pixel in the input image. For example, if the type is non-contaminated, the corresponding category label is 0; if the type is contamination, the corresponding category label is 1, and the type is too much. The corresponding category label is 2 for exposure, and the corresponding category label for underexposure is 4. Take the 6-frame images with the size of 4*4 continuously shot by the camera as an example, the 6-frame images are represented by P1, P2, P3, P4, P5, and P6 respectively. As shown in Figure 7A, the P1, P2, P3, The pixels in P4, P5, and P6 are sequentially numbered as x1, x2, . . . , x16 in the order from left to right and top to bottom. Input P1, P2, P3, P4, P5, and P6 into the semantic segmentation model respectively to obtain the semantic segmentation results shown in Figure 7B. Pixels with the same number) are virtualized into one pixel to obtain virtual pixels x1', x2', . , for example, count the number N1 of the 6 pixels whose type is 0, the number N2 whose type is 1, the number N3 whose type is 2, and the number N4 whose type is 3, and N1, N2, The type with the larger value in N3 and N4 is used as the type of the virtual pixel. Taking the pixel x1' as an example, the types of the six pixels corresponding to the pixel x1' are 0, 1, 1, 1, 2, and 3, respectively. Therefore, The type to which the pixel x1' belongs is 1 (that is, it has contamination), and the semantic segmentation result graph shown in FIG. 7C is obtained accordingly.

在另一个实施例中，所述特征信息为所述语义分割模型中softmax层的前一层神经网络卷积层输出的特征图；即将图像输入语义分割模型之后，将语义分割模型的softmax层的前一层神经网络卷积层输出的特征图作为特征信息。方式1中，所述步骤203中根据所述多帧图像对应的特征信息判断所述摄像机是否异常具体实现可如下：将所述多帧图像对应的特征图进行合并后输入到预置的第一后处理模型中，得到所述摄像机是否异常的结果。本发明实施例中，可以是通过异常图像集和正常图像集训练得到语义分割模型之后，根据样本图像(样本图像可以是所述异常图像集中的图像，也可以不是所述异常图像集中的图像，本申请不做严格限定)和训练好的语义分割模型对神经网络模型进行训练得到第一后处理模型，即在训练过程中语义分割模型保持不变。或者，还可以是通过异常图像集和正常图像集对用于训练语义分割模型的神经网络模型和用于训练第一后处理模型的神经网络模型进行联合训练，得到语义分割模型和第一后处理模型。训练方式可采用现有的神经网络训练方式，例如梯度下降方式，本申请不做严格限定。其中用于训练后处理模型的神经网络模型可以为全连接层网络或者卷积层网络，本申请不做严格限定。具体可如图8所示，多帧图像P1、P2、…、PN输入语义分割模型之后，从语义分割模型的softmax层的前一层神经网络卷积层输出对应的N个特征图T1、T2、…、TN，将特征图T1、T2、…、TN按照预置的顺序合并成一个三维空间的特征信息并输入到第一后处理模型中，得到摄像机是否正常的结果信息。当然，本领域技术人员也可以将语义分割模型中softmax层之前的任意一个神经网络卷积层输出的特征图作为本发明实施例中的特征信息，并按照如图8所示的方式得到语义分割结果图，本申请不做严格限定。In another embodiment, the feature information is the feature map output by the convolutional layer of the neural network of the previous layer of the softmax layer in the semantic segmentation model; that is, after the image is input into the semantic segmentation model, the softmax layer of the semantic segmentation model is The feature map output by the previous layer of neural network convolution layer is used as feature information. InMode 1, thestep 203 determines whether the camera is abnormal according to the feature information corresponding to the multi-frame images. The specific implementation can be as follows: Merge the feature maps corresponding to the multi-frame images and input them into a preset first image. In the post-processing model, the result of whether the camera is abnormal or not is obtained. In the embodiment of the present invention, after the semantic segmentation model is obtained by training the abnormal image set and the normal image set, according to the sample image (the sample image may be an image in the abnormal image set, or may not be an image in the abnormal image set, The first post-processing model is obtained by training the neural network model with the trained semantic segmentation model, that is, the semantic segmentation model remains unchanged during the training process. Alternatively, it is also possible to jointly train the neural network model used for training the semantic segmentation model and the neural network model used for training the first post-processing model through the abnormal image set and the normal image set to obtain the semantic segmentation model and the first post-processing model. Model. The training method may adopt the existing neural network training method, such as the gradient descent method, which is not strictly limited in this application. The neural network model used for training the post-processing model may be a fully connected layer network or a convolution layer network, which is not strictly limited in this application. Specifically, as shown in Figure 8, after the multi-frame images P1, P2, . , ..., TN, the feature maps T1, T2, ..., TN are combined into a three-dimensional space feature information in a preset order and input into the first post-processing model to obtain the result information of whether the camera is normal. Of course, those skilled in the art can also use the feature map output by any neural network convolution layer before the softmax layer in the semantic segmentation model as the feature information in the embodiment of the present invention, and obtain the semantic segmentation as shown in FIG. 8 . The results are not strictly limited in this application.

所述预置的顺序可以是多帧图像被采集的时间先后顺序。所述预置的顺序与前期训练后处理模型所采用的顺序相同。The preset sequence may be a chronological sequence in which multiple frames of images are collected. The preset sequence is the same as the sequence adopted by the pre-training and post-processing models.

方式2way 2

步骤202、将摄像机拍摄的一张图像输入语义分割模型中，得到该一张图像分别对应的特征信息；Step 202: Input an image captured by the camera into the semantic segmentation model, and obtain feature information corresponding to the image respectively;

步骤203、根据所述一张图像对应的特征信息判断所述摄像机是否正常。Step 203: Determine whether the camera is normal according to the feature information corresponding to the one image.

方式2可实现每拍摄一帧图像即进行一次摄像机异常判断，无需等待连续多帧图像完成拍摄后再进行一次摄像机异常判断，与方式1相比该方式2实时性更强。本领域技术人员可以根据实际的需求选择方式1还是方式2，本申请不做严格限定。Method 2 can realize the camera abnormality judgment once every frame of image is captured, and does not need to wait for the completion of shooting of multiple consecutive frames before performing a camera abnormality judgment. Compared withmethod 1,method 2 is more real-time. Those skilled in the art can selectMode 1 orMode 2 according to actual needs, which is not strictly limited in this application.

在一个实施例中，所述特征信息为所述语义分割模型softmax层输出的各像素所属类型；即将图像输入语义分割模型之后，将该语义分割模型的softmax层输出各像素所属类型作为特征信息。方式2中，所述步骤203中，根据所述一张图像对应的特征信息判断所述摄像机是否异常，具体包括：判断所述一张图像中所属类型为异常类型的像素的数量是否大于等于预置数量阈值，若是则确定所述摄像机异常，若否则确定所述摄像机正常。所述数量阈值根据经验值确定，例如可根据引起异常类型的污损物确定。例如可通过实验，采用不同尺寸的污损物对摄像头镜头造成不同尺寸的污损区域，判断摄像机在不同尺寸的污损区域下拍摄得到的图像对决策控制单元作出决策的影响程度，根据影响程度达到预置影响程度的最小尺寸的无损区域确定数量阈值。如图9所示，图像P输入语义分割模型之后，从语义分割模型的softmax层输出图像P对应的语义分割结果G，根据语义分割结果G得到摄像机是否正常的结果信息。In one embodiment, the feature information is the type of each pixel output by the softmax layer of the semantic segmentation model; that is, after the image is input into the semantic segmentation model, the softmax layer of the semantic segmentation model outputs the type of each pixel as the feature information. Inmode 2, instep 203, determining whether the camera is abnormal according to the feature information corresponding to the one image specifically includes: determining whether the number of pixels of the abnormal type in the one image is greater than or equal to a predetermined number. The number threshold is set, if it is, it is determined that the camera is abnormal; otherwise, it is determined that the camera is normal. The quantity threshold is determined based on empirical values, for example, based on the type of fouling that causes the abnormality. For example, through experiments, contaminants of different sizes can be used to cause contamination areas of different sizes to the camera lens, and the degree of influence of the images captured by the camera in the contamination areas of different sizes on the decision-making control unit can be judged. According to the degree of influence The minimum size of lossless area that reaches a preset level of influence determines the threshold for the number of areas. As shown in Figure 9, after the image P is input into the semantic segmentation model, the semantic segmentation result G corresponding to the image P is output from the softmax layer of the semantic segmentation model, and the result information of whether the camera is normal is obtained according to the semantic segmentation result G.

在另一个实施例中，所述特征信息为所述语义分割模型中softmax层的前一层神经网络卷积层输出的特征图；即将图像输入语义分割模型之后，将语义分割模型的softmax层的前一层神经网络卷积层输出的特征图作为特征信息。方式2中，所述步骤203中根据所述一张图像对应的特征信息判断所述摄像机是否异常，具体包括：将所述一张图像对应的特征图输入到预置的第二后处理模型中，得到所述摄像机是否异常的结果。本发明实施例中，可以是通过异常图像集和正常图像集训练得到语义分割模型之后，根据样本图像(样本图像可以是所述异常图像集中的图像，也可以不是所述异常图像集中的图像，本申请不做严格限定)和训练好的语义分割模型对神经网络模型进行训练得到第二后处理模型，即在训练过程中语义分割模型保持不变。或者，还可以是通过异常图像集和正常图像集对用于训练语义分割模型的神经网络模型和用于训练第二后处理模型的神经网络模型进行联合训练，得到语义分割模型和第二后处理模型。训练方式可采用现有的神经网络训练方式，例如梯度下降方式，本申请不做严格限定。其中用于训练后处理模型的神经网络模型可以为全连接层网络或者卷积层网络，本申请不做严格限定。具体可如图10所示，图像P输入语义分割模型之后，从语义分割模型的softmax层的前一层神经网络卷积层输出对应的特征图T，将特征图T输入到第二后处理模型中，得到摄像机是否正常的结果信息。当然，本领域技术人员也可以将语义分割模型中softmax层之前的任意一个神经网络卷积层输出的特征图作为本发明实施例中的特征信息，本申请不做严格限定。In another embodiment, the feature information is the feature map output by the convolutional layer of the neural network of the previous layer of the softmax layer in the semantic segmentation model; that is, after the image is input into the semantic segmentation model, the softmax layer of the semantic segmentation model is The feature map output by the previous layer of neural network convolution layer is used as feature information. InMode 2, instep 203, determining whether the camera is abnormal according to the feature information corresponding to the one image specifically includes: inputting the feature map corresponding to the one image into a preset second post-processing model to get the result of whether the camera is abnormal. In the embodiment of the present invention, after the semantic segmentation model is obtained by training the abnormal image set and the normal image set, according to the sample image (the sample image may be an image in the abnormal image set, or may not be an image in the abnormal image set, The second post-processing model is obtained by training the neural network model with the trained semantic segmentation model, that is, the semantic segmentation model remains unchanged during the training process. Alternatively, it is also possible to jointly train the neural network model for training the semantic segmentation model and the neural network model for training the second post-processing model through the abnormal image set and the normal image set to obtain the semantic segmentation model and the second post-processing model. Model. The training method may adopt the existing neural network training method, such as the gradient descent method, which is not strictly limited in this application. The neural network model used for training the post-processing model may be a fully connected layer network or a convolution layer network, which is not strictly limited in this application. Specifically, as shown in Figure 10, after the image P is input into the semantic segmentation model, the corresponding feature map T is output from the previous neural network convolution layer of the softmax layer of the semantic segmentation model, and the feature map T is input into the second post-processing model. , get the result information of whether the camera is normal or not. Of course, those skilled in the art can also use the feature map output by any neural network convolution layer before the softmax layer in the semantic segmentation model as the feature information in the embodiment of the present invention, which is not strictly limited in this application.

本发明实施例一提供的摄像机异常检测方法，一方面，能够实现实时检测摄像机是否异常以及在异常时进行报警，以便操作人员及时获知摄像机发生异常，并对摄像机进行及时处理；另一方面，通过异常图像集和正常图像集对神经网络进行训练，能够提高训练得到的语义分割模型的健壮性和准确性。The camera abnormality detection method provided by the first embodiment of the present invention can, on the one hand, realize real-time detection of whether the camera is abnormal and alarm when it is abnormal, so that the operator can know the abnormality of the camera in time and deal with the camera in time; Training the neural network on the abnormal image set and the normal image set can improve the robustness and accuracy of the trained semantic segmentation model.

实施例二Embodiment 2

基于前述实施例一提供的一种摄像机异常检测方法的相同构思，本发明实施例二提供一种摄像机异常检测装置，该装置的结构如图11所示，包括训练单元11、信息获取单元12、异常检测单元13，其中：Based on the same concept of the camera abnormality detection method provided in the first embodiment, the second embodiment of the present invention provides a camera abnormality detection device. The structure of the device is shown in FIG. 11 , including atraining unit 11, an information acquisition unit 12,Anomaly detection unit 13, wherein:

训练单元11，用于根据预置的异常图像集和正常图像集对神经网络进行训练，得到用于确定图像中各像素所属类型的语义分割模型，类型的种类包含异常类型和正常类型。Thetraining unit 11 is used to train the neural network according to the preset abnormal image set and the normal image set to obtain a semantic segmentation model for determining the type of each pixel in the image, and the types of the types include abnormal types and normal types.

异常图像集、正常图像集、异常类型和正常类型的具体信息可如前实施例一中的第一示例～第五示例，在此不再赘述。The specific information of the abnormal image set, the normal image set, the abnormal type, and the normal type can be the same as the first example to the fifth example in theprevious embodiment 1, and details are not repeated here.

信息获取单元12，用于将摄像机拍摄的图像输入所述语义分割模型中，得到所述图像对应的特征信息。The information acquisition unit 12 is configured to input the image captured by the camera into the semantic segmentation model to obtain feature information corresponding to the image.

异常检测单元13，用于根据所述图像对应的特征信息判断所述摄像机是否异常，并在确定异常时进行报警。Theabnormality detection unit 13 is configured to determine whether the camera is abnormal according to the feature information corresponding to the image, and to issue an alarm when the abnormality is determined.

前述信息获取单元12和异常检测单元13具体实现可通过但不仅限于以下两种方式：The aforementioned information acquisition unit 12 andabnormality detection unit 13 can be specifically implemented in but not limited to the following two ways:

方式1way 1

所述信息获取单元12，具体用于：将摄像机拍摄的连续多帧图像输入所述语义分割模型中，得到每帧图像对应的特征信息；The information acquisition unit 12 is specifically configured to: input the continuous multi-frame images captured by the camera into the semantic segmentation model to obtain feature information corresponding to each frame of image;

所述异常检测单元13具体用于：根据所述多帧图像对应的特征信息判断所述摄像机是否异常，并在确定异常时进行报警。Theabnormality detection unit 13 is specifically configured to: determine whether the camera is abnormal according to the feature information corresponding to the multi-frame images, and issue an alarm when the abnormality is determined.

在一个示例中，所述特征信息为所述语义分割模型中softmax层输出的各像素所属类型；异常检测单元13根据所述多帧图像对应的特征信息判断所述摄像机是否异常，具体包括：将多帧图像中位于同一位置的多个像素拟化成一个虚拟像素，并根据所述多个像素所属类型确定所述虚拟像素所属类型，将所述虚拟像素作为语义分割结果图中相应位置的一个像素，依此得到语义分割结果图；以及，在判断所述语义分割结果图中包含所属类型为异常类型的像素的数量大于等于预置的数量阈值时，确定所述摄像机异常。In an example, the feature information is the type of each pixel output by the softmax layer in the semantic segmentation model; theabnormality detection unit 13 judges whether the camera is abnormal according to the feature information corresponding to the multi-frame images, specifically including: Multiple pixels located at the same position in the multi-frame image are simulated into a virtual pixel, and the type of the virtual pixel is determined according to the type of the multiple pixels, and the virtual pixel is used as a pixel in the corresponding position in the semantic segmentation result graph , and obtain a semantic segmentation result map accordingly; and when it is judged that the number of pixels belonging to an abnormal type in the semantic segmentation result map is greater than or equal to a preset number threshold, it is determined that the camera is abnormal.

在另一个示例中，所述特征信息为所述语义分割模型中softmax层的前一层神经网络卷积层输出的特征图；异常检测单元13根据所述多帧图像对应的特征信息判断所述摄像机是否异常，具体包括：将所述多帧图像对应的特征图进行合并后输入到预置的第一后处理模型中，得到所述摄像机是否异常的结果；其中，所述第一后处理模型为预先采用样本图像和所述语义分割模型对神经网络模型进行训练得到。In another example, the feature information is a feature map output by a neural network convolution layer of the previous layer of the softmax layer in the semantic segmentation model; theabnormality detection unit 13 judges the Whether the camera is abnormal, specifically includes: merging the feature maps corresponding to the multi-frame images and inputting them into a preset first post-processing model to obtain a result of whether the camera is abnormal; wherein, the first post-processing model It is obtained by training the neural network model with sample images and the semantic segmentation model in advance.

方式2way 2

所述信息获取单元12，具体用于：将摄像机拍摄的一张图像输入所述语义分割模型中，得到所述一张图像对应的特征信息；The information acquisition unit 12 is specifically configured to: input an image captured by a camera into the semantic segmentation model to obtain feature information corresponding to the image;

所述异常检测单元13，具体用于：根据所述一张图像对应的特征信息判断所述摄像机是否异常，并在确定异常时进行报警。Theabnormality detection unit 13 is specifically configured to: determine whether the camera is abnormal according to the feature information corresponding to the one image, and issue an alarm when the abnormality is determined.

在一个示例中，所述特征信息为所述语义分割模型中softmax层输出的各像素所属类型；异常检测单元13根据所述一张图像对应的特征信息判断所述摄像机是否异常，具体包括：判断所述一张图像中所属类型为异常类型的像素的数量是否大于等于预置数量阈值，若是则确定所述摄像机异常，若否则确定所述摄像机正常。In an example, the feature information is the type of each pixel output by the softmax layer in the semantic segmentation model; theabnormality detection unit 13 judges whether the camera is abnormal according to the feature information corresponding to the one image, specifically including: judging Whether the number of pixels belonging to an abnormal type in the one image is greater than or equal to a preset number threshold, if so, it is determined that the camera is abnormal; otherwise, it is determined that the camera is normal.

在另一个示例中，所述特征信息为所述语义分割模型中softmax层的前一层神经网络卷积层输出的特征图；异常检测单元13根据所述一张图像对应的特征信息判断所述摄像机是否异常，具体包括：将所述一张图像对应的特征图输入到预置的第二后处理模型中，得到所述摄像机是否异常的结果；其中，所述第二后处理模型为预先采用样本图像和所述语义分割模型对神经网络模型进行训练得到。In another example, the feature information is a feature map output by a neural network convolution layer of the previous layer of the softmax layer in the semantic segmentation model; theanomaly detection unit 13 judges the Whether the camera is abnormal specifically includes: inputting the feature map corresponding to the one image into a preset second post-processing model to obtain a result of whether the camera is abnormal; wherein, the second post-processing model is a pre-applied The sample image and the semantic segmentation model are obtained by training the neural network model.

所述训练单元11具体用于：采用异常图像集对所述神经网络进行训练，得到初级语义分割模型；将正常图像集中的正常图像输入所述初级语义分割模型中，输出正常图像对应的语义分割结果，并将语义分割结果有误的正常图像加入所述异常图像集中，以得到新的异常图像集；采用新的异常图像集对所述初级语义分割模型进行训练，得到语义分割模型。Thetraining unit 11 is specifically used for: using the abnormal image set to train the neural network to obtain a primary semantic segmentation model; inputting the normal image in the normal image set into the primary semantic segmentation model, and outputting the semantic segmentation corresponding to the normal image As a result, normal images with wrong semantic segmentation results are added to the abnormal image set to obtain a new abnormal image set; the primary semantic segmentation model is trained by using the new abnormal image set to obtain a semantic segmentation model.

前述训练单元11具体实现还可以通过如图4所示的流程，在此不再赘述。The specific implementation of the foregoingtraining unit 11 may also be performed through the process shown in FIG. 4 , which will not be repeated here.

本发明技术方案，带来以下技术效果：The technical scheme of the present invention brings the following technical effects:

技术效果1、在通过预置的异常图像集训练神经网络模型得到初始语义分割模型之后，还通过正常图像集对初始语义分割模型进行测试，将检测错误的正常图像(即正样本)加入到异常图像集中得到新的异常图像集，再基于该新的异常图像集对初始语义分割模型进行训练。本发明实施例，自动将正常图像集中检测有误的正常图像添加到异常图像集中，由于检测有误的正常图像具有反映初始语义分割模型检测错误的典型性，因此检测有误的正常图像能够形成很好的训练正样本，将训练正样本添加到异常图像集中不仅能够自动扩充异常图像集，降低人力成本，而且基于新的异常图像集训练得到的语义分割模型具有更强的健壮性和更低的错误检测率。Technical effect 1. After training the neural network model through the preset abnormal image set to obtain the initial semantic segmentation model, the initial semantic segmentation model is also tested through the normal image set, and the normal images (ie positive samples) that have detected errors are added to the abnormal image. A new abnormal image set is obtained from the image set, and then the initial semantic segmentation model is trained based on the new abnormal image set. In this embodiment of the present invention, the normal images that are incorrectly detected in the normal image set are automatically added to the abnormal image set. Since the normal images that are incorrectly detected have the typicality of reflecting the detection errors of the initial semantic segmentation model, the normal images that are incorrectly detected can form Good training positive samples, adding training positive samples to the abnormal image set can not only automatically expand the abnormal image set and reduce labor costs, but also the semantic segmentation model trained based on the new abnormal image set has stronger robustness and lower cost. false detection rate.

技术效果2、在实际应用中若摄像头发生异常则会导致在一段时间内连续拍摄的图像均会存在异常，因此结合连续多帧图像对应的特征信息判断摄像机是否异常更准确。Technical effect 2. In practical application, if the camera is abnormal, the images captured continuously for a period of time will be abnormal. Therefore, it is more accurate to judge whether the camera is abnormal or not based on the feature information corresponding to the continuous multi-frame images.

以上是本发明的核心思想，为了使本技术领域的人员更好地理解本发明实施例中的技术方案，并使本发明实施例的上述目的、特征和优点能够更加明显易懂，下面结合附图对本发明实施例中技术方案作进一步详细的说明。The above is the core idea of the present invention. In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, and to make the above-mentioned objects, features and advantages of the embodiments of the present invention more obvious and understandable, the following is combined with the following appendixes. The drawings further describe the technical solutions in the embodiments of the present invention in detail.

显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包含这些改动和变型在内。It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention. Thus, provided that these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims (18)

1. A camera abnormality detection method characterized by comprising:

training a neural network according to a preset abnormal image set and a preset normal image set to obtain a semantic segmentation model for determining the type of each pixel in an image, wherein the type comprises an abnormal type and a normal type;

inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image;

and judging whether the camera is abnormal according to the characteristic information corresponding to the image, and giving an alarm when the abnormality is determined.

2. The method according to claim 1, wherein the image captured by the camera is input into the semantic segmentation model to obtain feature information corresponding to the image, specifically: inputting continuous multi-frame images shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to each frame of image;

judging whether the camera is abnormal according to the characteristic information corresponding to the image, specifically: and judging whether the camera is abnormal or not according to the characteristic information corresponding to the multi-frame image.

3. The method according to claim 2, wherein the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs;

judging whether the camera is abnormal according to the characteristic information corresponding to the multi-frame images, specifically comprising:

simulating a plurality of pixels at the same position in a multi-frame image into a virtual pixel, determining the type of the virtual pixel according to the type of the plurality of pixels, and taking the virtual pixel as a pixel at a corresponding position in a semantic segmentation result graph so as to obtain the semantic segmentation result graph;

and when the semantic segmentation result graph contains pixels with abnormal types, the number of the pixels is judged to be larger than or equal to a preset number threshold value, and the camera is determined to be abnormal.

4. The method according to claim 2, wherein the feature information is a feature map output by a neural network convolution layer previous to a softmax layer in the semantic segmentation model;

judging whether the camera is abnormal according to the characteristic information corresponding to the multi-frame images, specifically comprising:

merging the characteristic graphs corresponding to the multiple frames of images, and inputting the merged characteristic graphs into a preset first post-processing model to obtain a result of whether the camera is abnormal; the first post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.

5. The method according to claim 1, wherein the image captured by the camera is input into the semantic segmentation model to obtain feature information corresponding to the image, specifically: inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image;

judging whether the camera is abnormal according to the characteristic information corresponding to the image, specifically: and judging whether the camera is abnormal or not according to the characteristic information corresponding to the image.

6. The method according to claim 5, wherein the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs;

judging whether the camera is abnormal according to the characteristic information corresponding to the image, which specifically comprises the following steps:

and judging whether the number of the pixels of which the types are abnormal in the image is greater than or equal to a preset number threshold, if so, determining that the camera is abnormal, and otherwise, determining that the camera is normal.

7. The method according to claim 5, wherein the feature information is a feature map output by a neural network convolution layer previous to a softmax layer in the semantic segmentation model;

judging whether the camera is abnormal according to the characteristic information corresponding to the image, which specifically comprises the following steps:

inputting the characteristic diagram corresponding to the image into a preset second post-processing model to obtain a result of whether the camera is abnormal or not; and the second post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.

8. The method according to any one of claims 1 to 7, wherein the training of the neural network is performed according to a preset abnormal image set and a normal image set to obtain a semantic segmentation model for determining the abnormal type of each pixel in the image, and specifically comprises:

training the neural network by adopting an abnormal image set to obtain a primary semantic segmentation model;

inputting normal images in a normal image set into the primary semantic segmentation model, outputting semantic segmentation results corresponding to the normal images, and adding normal images with wrong semantic segmentation results into the abnormal image set to obtain a new abnormal image set;

and training the primary semantic segmentation model by adopting a new abnormal image set to obtain a semantic segmentation model.

9. The method of any of claims 1 to 7, wherein the set of abnormal images comprises transparent defacement images and opaque defacement images, the normal type is no defacement, the abnormal type is defacement, or the abnormal type comprises transparent defacement and opaque defacement;

or the abnormal image set comprises an overexposed image and an underexposed image, the abnormal type comprises overexposure and underexposure, and the normal type is normal exposure;

or the abnormal image set comprises a transparent stained image, an opaque stained image, an exposed image and an underexposed image, the normal type comprises normal exposure and no stain, and the abnormal type comprises stain, no stain, overexposure and underexposure; alternatively, the types of anomalies include clear smudge, opaque smudge, no smudge, overexposure, and underexposure.

10. A camera abnormality detection device characterized by comprising:

the training unit is used for training the neural network according to a preset abnormal image set and a preset normal image set to obtain a semantic segmentation model for determining the type of each pixel in the image, wherein the type comprises an abnormal type and a normal type;

the information acquisition unit is used for inputting the image shot by the camera into the semantic segmentation model to obtain the characteristic information corresponding to the image;

and the abnormality detection unit is used for judging whether the camera is abnormal according to the characteristic information corresponding to the image and giving an alarm when the abnormality is determined.

11. The apparatus according to claim 10, wherein the information obtaining unit is specifically configured to: inputting continuous multi-frame images shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to each frame of image;

the abnormality detection unit is specifically configured to: and judging whether the camera is abnormal or not according to the characteristic information corresponding to the multi-frame images, and giving an alarm when the abnormality is determined.

12. The apparatus according to claim 11, wherein the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs;

the abnormality detection unit judges whether the camera is abnormal according to the feature information corresponding to the multi-frame image, and specifically includes:

simulating a plurality of pixels at the same position in a multi-frame image into a virtual pixel, determining the type of the virtual pixel according to the type of the plurality of pixels, and taking the virtual pixel as a pixel at a corresponding position in a semantic segmentation result graph so as to obtain the semantic segmentation result graph;

and when the semantic segmentation result graph contains pixels with abnormal types, the number of the pixels is judged to be larger than or equal to a preset number threshold value, and the camera is determined to be abnormal.

13. The apparatus according to claim 11, wherein the feature information is a feature map output by a neural network convolution layer previous to a softmax layer in the semantic segmentation model;

the abnormality detection unit judges whether the camera is abnormal according to the feature information corresponding to the multi-frame image, and specifically includes:

merging the characteristic graphs corresponding to the multiple frames of images, and inputting the merged characteristic graphs into a preset first post-processing model to obtain a result of whether the camera is abnormal; the first post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.

14. The apparatus according to claim 10, wherein the information obtaining unit is specifically configured to: inputting an image shot by a camera into the semantic segmentation model to obtain characteristic information corresponding to the image;

the abnormality detection unit is specifically configured to: and judging whether the camera is abnormal according to the characteristic information corresponding to the image, and giving an alarm when the abnormality is determined.

15. The apparatus according to claim 14, wherein the feature information is a type to which each pixel output by a softmax layer in the semantic segmentation model belongs;

the abnormality detection unit judges whether the camera is abnormal according to the feature information corresponding to the image, and specifically includes:

and judging whether the number of the pixels of which the types are abnormal in the image is greater than or equal to a preset number threshold, if so, determining that the camera is abnormal, and otherwise, determining that the camera is normal.

16. The apparatus according to claim 14, wherein the feature information is a feature map output by a neural network convolution layer previous to a softmax layer in the semantic segmentation model;

the abnormality detection unit judges whether the camera is abnormal according to the feature information corresponding to the image, and specifically includes:

inputting the characteristic diagram corresponding to the image into a preset second post-processing model to obtain a result of whether the camera is abnormal or not; and the second post-processing model is obtained by adopting a sample image and the semantic segmentation model to train a neural network model in advance.

17. The device according to any one of claims 10 to 16, wherein the training unit is specifically configured to:

training the neural network by adopting an abnormal image set to obtain a primary semantic segmentation model;

inputting normal images in a normal image set into the primary semantic segmentation model, outputting semantic segmentation results corresponding to the normal images, and adding normal images with wrong semantic segmentation results into the abnormal image set to obtain a new abnormal image set;

and training the primary semantic segmentation model by adopting a new abnormal image set to obtain a semantic segmentation model.

18. The apparatus according to any one of claims 10 to 16, wherein the abnormal image set comprises a transparent defacement image and an opaque defacement image, the normal type is no defacement, the abnormal type is defacement, or the abnormal type comprises a transparent defacement and an opaque defacement;

or the abnormal image set comprises an overexposed image and an underexposed image, the abnormal type comprises overexposure and underexposure, and the normal type is normal exposure;

or the abnormal image set comprises a transparent stained image, an opaque stained image, an exposed image and an underexposed image, the normal type comprises normal exposure and no stain, and the abnormal type comprises stain, no stain, overexposure and underexposure; alternatively, the types of anomalies include clear smudge, opaque smudge, no smudge, overexposure, and underexposure.

Images