WO2021184628A1

Movatterモバイル変換

Info

Publication number: WO2021184628A1
Application number: PCT/CN2020/104673
Authority: WO
Inventors: 张嘉
Original assignee: Ping An International Smart City Technology Co Ltd
Current assignee: Ping An International Smart City Technology Co Ltd
Priority date: 2020-03-18
Filing date: 2020-07-25
Publication date: 2021-09-23
Anticipated expiration: 2022-09-18
Also published as: CN111476107A; CN111476107B

Abstract

Disclosed is an image processing method, which can be implemented in artificial intelligence. The method comprises: determining a first image set, wherein the first image set comprises a plurality of first images, and the plurality of first images are images of a target vehicle that are captured within the same time period in the same scenario; determining one or more second image sets from the first image set, wherein each second image set of the one or more second image sets comprises a plurality of second images, and each second image set records one pending violative driving process of the target vehicle; and marking a reference position of a traffic sign in at least one second image set of the one or more second image sets, and respectively identifying a change in position of the target vehicle in the one or more second image sets to determine whether pending violative driving respectively corresponding to the one or more second image sets is a violation. By means of the embodiments of the present application, the efficiency of test data pre-processing can be improved, and an algorithm can be effectively evaluated.

Description

Translated fromChinese

一种图像处理方法及装置Image processing method and device

本申请要求于2020年03月18日提交中国专利局、申请号为202010190279.X，发明名称为“一种图像处理方法及装置”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。This application claims the priority of a Chinese patent application filed with the Chinese Patent Office on March 18, 2020, the application number is 202010190279.X, and the invention title is "an image processing method and device", the entire content of which is incorporated herein by reference Applying.

技术领域Technical field

本申请实施例涉及人工智能技术领域，尤其涉及一种图像处理方法及装置。The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an image processing method and device.

背景技术Background technique

在目前的交通管理系统中，通常运用图片识别算法对摄像头拍摄的多张图片进行识别，以判断图中相应车辆的驾驶行为是否违规或者违法。那么，从众多的图片识别算法选择合适的识别算法，对交通违法行为进行准确的识别和判断，起到至关重要的作用。在算法进行选择时，需要依据统一的算法精度标准，有效评估各个算法的精度和准确性，从而判断该算法与相应场景的识别匹配度。In the current traffic management system, image recognition algorithms are usually used to recognize multiple pictures taken by the camera to determine whether the driving behavior of the corresponding vehicle in the picture is illegal or illegal. Then, choosing a suitable recognition algorithm from a large number of image recognition algorithms, and accurately identifying and judging traffic violations, plays a vital role. When the algorithm is selected, it is necessary to effectively evaluate the accuracy and accuracy of each algorithm according to a unified algorithm accuracy standard, so as to judge the recognition matching degree of the algorithm and the corresponding scene.

发明人意识到，在对特定识别算法的评估过程中，首先在测试前需要对大量的测试数据(例如图片数据)进行清洗、筛选和分类等等操作，该步骤需要花费一定的时间；然后需要对经过前述预处理的数据进行裁剪后，再进行标注相关信息，但是手工操作耗时较长且可能出现失误，造成对图片中参考物的标注不准确等问题。The inventor realizes that in the process of evaluating a specific recognition algorithm, a large amount of test data (such as image data) needs to be cleaned, filtered, and classified before testing. This step takes a certain amount of time; then After the aforementioned pre-processed data is cropped, the relevant information is then labeled, but the manual operation takes a long time and errors may occur, causing problems such as inaccurate labeling of reference objects in the picture.

因此，如何提高测试前的数据预处理效率，从而有效评估图片识别算法，是需要解决的问题。Therefore, how to improve the efficiency of data preprocessing before the test, so as to effectively evaluate the image recognition algorithm, is a problem that needs to be solved.

发明内容Summary of the invention

本申请实施例提供一种图像处理方法及装置，可以提高测试前的数据预处理效率，从而有效评估图片识别算法。The embodiments of the present application provide an image processing method and device, which can improve the efficiency of data preprocessing before testing, thereby effectively evaluating the image recognition algorithm.

第一方面，本申请实施例提供了一种图像处理方法，该方法可包括：In the first aspect, an embodiment of the present application provides an image processing method, which may include:

确定第一图像集合，所述第一图像集合包括多张第一图像，所述多张第一图像为同一场景下相同时间段内针对目标车辆的拍摄图像；Determining a first image set, where the first image set includes a plurality of first images, and the plurality of first images are images taken for the target vehicle in the same scene and the same time period;

从所述第一图像集合中确定一个或多个第二图像集合，所述一个或多个第二图像集合中每一个第二图像集合包括多张第二图像，所述每一个第二图像集合记录了所述目标车辆的一次待定违规行驶过程；One or more second image sets are determined from the first image set, each second image set in the one or more second image sets includes a plurality of second images, and each second image set Recorded a pending illegal driving process of the target vehicle;

标注所述一个或多个第二图像集合中至少一个第二图像集合中的交通标识的参考位置，并分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，以判断所述一个或多个第二图像集合分别对应的待定违规行驶是否违规。Mark the reference position of the traffic sign in the at least one second image set in the one or more second image sets, and respectively identify the position change of the target vehicle in the one or more second image sets to determine Whether the pending illegal driving respectively corresponding to the one or more second image sets is illegal.

第二方面，本申请实施例提供了一种图像处理装置，该装置可包括：In the second aspect, an embodiment of the present application provides an image processing device, which may include:

确定单元，用于确定第一图像集合，所述第一图像集合包括多张第一图像，所述多张第一图像为同一场景下相同时间段内针对目标车辆的拍摄图像；A determining unit, configured to determine a first image set, where the first image set includes a plurality of first images, and the plurality of first images are images taken for the target vehicle in the same scene in the same time period;

筛选单元，用于从所述第一图像集合中确定一个或多个第二图像集合，所述一个或多个第二图像集合中每一个第二图像集合包括多张第二图像，所述每一个第二图像集合记录了所述目标车辆的一次待定违规行驶过程；The screening unit is configured to determine one or more second image sets from the first image set, and each second image set in the one or more second image sets includes multiple second images, and each A second image set records a pending illegal driving process of the target vehicle;

标注单元，用于标注所述一个或多个第二图像集合中至少一个第二图像集合中的交通标识的参考位置，并分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，以判断所述一个或多个第二图像集合分别对应的待定违规行驶是否违规。The labeling unit is used to label the reference position of the traffic sign in the at least one second image set in the one or more second image sets, and to respectively identify the target vehicle in the one or more second image sets The position changes to determine whether the pending illegal driving respectively corresponding to the one or more second image sets is illegal.

第三方面，本申请实施例提供了一种图像处理设备，包括处理器和存储器，所述处理器和所述存储器相互连接，其中，所述存储器用于存储计算机程序，所述计算机程序包括程序指令，所述处理器被配置用于调用所述程序指令，执行以下步骤：In a third aspect, an embodiment of the present application provides an image processing device, including a processor and a memory, the processor and the memory are connected to each other, wherein the memory is used to store a computer program, and the computer program includes a program Instructions, the processor is configured to call the program instructions to perform the following steps:

第四方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序被处理器执行时，用于实现以下步骤：In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the following steps:

第五方面，本申请实施例提供了一种计算机程序，该计算机程序包括指令，当该计算机程序被计算机执行时，使得计算机可以执行第一方面中任意一种图像处理方法的部分或全部步骤。In the fifth aspect, the embodiments of the present application provide a computer program, which includes instructions, when the computer program is executed by a computer, the computer can execute part or all of the steps of any image processing method in the first aspect.

实施本申请实施例，从图片数据中根据图像尺寸和图像分辨率筛选出图像尺寸和图像分辨率一致的多张图片；其中，在特定场景下的同一台摄像机(即卡口)拍摄的图片尺寸是一样的，以及在特定时刻的分辨率也几乎相同。那么，筛选出的多张第一图片可以认为是同一台摄像机拍摄的。根据第二图片的拍摄规则(例如，摄像机在违法驾驶发生前拍摄一张第二图片；在违法驾驶过程中拍摄一张第二图片；在违法驾驶过程结束后拍摄一张第二图片)，从多张第一图片中筛选出一组或多组第二图片。每一组第二图片完整记录了一次违规驾驶过程。从任一组第二图片中选择一张图片，对图片中与该违法行驶过程相关的参考物进行标注。进一步具体地地，根据已经标注的图片对剩余的图片进行批量标注。或者一次性根据标注规则对所有图片包含的参考物进行标注。具体地，通过标注图片中与违法标准相关的参考物位置，例如红绿灯位置，停止线位置，白实线位置等，自动生成相关标注信息，并根据图片分辨率的关系，通过封装好的自适应算法，将标注信息自适应匹配到上所有图片，从而完成批量标注的操作。通过程序进行测试数据的预处理，本申请能够极大的缩短测试数据的准备和预处理时间，提高测试效率。通过批量读取标注信息的脚本，批量将测试数据传入被测算法，简化测试人员手工操作步骤，避免出错。通过脚本的自动标注和统计方法，自动生成标准的测试报告，记录测试结果，生成便于分析的算法混淆矩阵，并将测试结果按照需求分类，便于算法团队分析被测图片结果，提高算法精度，优化产品识别精度。To implement the embodiment of this application, multiple pictures with the same image size and image resolution are filtered from the picture data according to the image size and resolution; among them, the size of the pictures taken by the same camera (ie bayonet) in a specific scene It is the same, and the resolution at a particular moment is almost the same. Then, the multiple first pictures selected can be considered to be taken by the same camera. According to the shooting rules of the second picture (for example, the camera takes a second picture before the illegal driving occurs; the second picture is taken during the illegal driving; the second picture is taken after the illegal driving is over), from One or more sets of second pictures are filtered out of the plurality of first pictures. Each set of second pictures completely records the process of driving a violation. Select a picture from any group of second pictures, and mark the reference objects in the picture related to the illegal driving process. More specifically, the remaining pictures are batch-labeled according to the already-labeled pictures. Or label all the reference objects contained in the picture according to the labeling rules at one time. Specifically, by labeling the position of the reference object related to the illegal standard in the picture, such as the position of the traffic light, the position of the stop line, the position of the white solid line, etc., the relevant labeling information is automatically generated, and according to the relationship of the picture resolution, the packaged adaptive The algorithm adaptively matches the labeling information to all the pictures above to complete the batch labeling operation. By preprocessing the test data through the program, this application can greatly shorten the preparation and preprocessing time of the test data, and improve the test efficiency. Through the scripts that read the marked information in batches, the test data is sent to the tested algorithm in batches, which simplifies the manual operation steps of testers and avoids errors. Automatically generate standard test reports, record test results, generate an easy-to-analyze algorithm confusion matrix, and categorize the test results according to requirements through the automatic annotation and statistical methods of the script, which is convenient for the algorithm team to analyze the results of the tested pictures, improve the accuracy of the algorithm, and optimize Product recognition accuracy.

附图说明Description of the drawings

图1是本申请实施例提供的一种图像处理的系统架构示意图；FIG. 1 is a schematic diagram of an image processing system architecture provided by an embodiment of the present application;

图2是本申请实施例提供的一种图像处理方法所应用场景的示意图；Fig. 2 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application;

图3是本申请实施例提供的一种图像处理方法的流程示意图；FIG. 3 is a schematic flowchart of an image processing method provided by an embodiment of the present application;

图4是本申请实施例提供的一种图像处理装置的结构示意图；FIG. 4 is a schematic structural diagram of an image processing device provided by an embodiment of the present application;

图5是本申请实施例提供的一种图像处理设备的结构示意图。Fig. 5 is a schematic structural diagram of an image processing device provided by an embodiment of the present application.

具体实施方式Detailed ways

本申请实施例提供一种图像处理方法及装置，可以快速有效地识别待识别图片中文本载体的文本含义。The embodiments of the present application provide an image processing method and device, which can quickly and effectively identify the text meaning of the text carrier in the picture to be recognized.

首先，对本申请实施例中的部分用语进行解释说明，以便于本领域技术人员理解。First, some terms in the embodiments of the present application are explained to facilitate the understanding of those skilled in the art.

(1)治安卡口是道路交通治安卡口监控系统的简称，是指依托道路上特定场所，如收费站、交通或治安检查站等卡口点，对所有通过该卡口点的机动车辆进行拍摄、记录与处理的一种道路交通现场监测系统。(1) The security checkpoint is the abbreviation of the road traffic security checkpoint monitoring system, which refers to the checkpoint of all motor vehicles passing through the checkpoint at a specific place on the road, such as toll stations, traffic or public security checkpoints. A road traffic on-site monitoring system for shooting, recording and processing.

(2)图像的传统识别流程分为四个步骤：图像采集→图像预处理→特征提取→图像识别。图像识别可能是以图像的主要特征为基础的。每个图像都有它的特征,如字母A有个尖，P有个圈、而Y的中心有个锐角等。(2) The traditional image recognition process is divided into four steps: image acquisition→image preprocessing→feature extraction→image recognition. Image recognition may be based on the main features of the image. Each image has its characteristics, such as the letter A has a sharp point, P has a circle, and the center of Y has an acute angle.

(3)查准率(Precision)(即精度)是衡量某一检索系统的信号噪声比的一种指标，即检出的相关文献与检出的全部文献的百分比。普遍表示为：查准率＝(检索出的相关信息量/检索出的信息总量)x100％。(3) Precision (ie precision) is an index that measures the signal-to-noise ratio of a certain retrieval system, that is, the percentage of the detected related documents to the entire detected documents. It is generally expressed as: precision rate = (relevant information retrieved/total information retrieved) x 100%.

(4)查全率(Recall Ratio)是指从数据库内检出的相关的信息量与总量的比率。查全率绝对值可以根据数据库内容、数量来估算。(4) Recall Ratio (Recall Ratio) refers to the ratio of the amount of relevant information detected from the database to the total amount. The absolute value of recall can be estimated based on the content and quantity of the database.

(5)F1分数(F1Score)，是统计学中用来衡量二分类模型精确度的一种指标。它同时兼顾了分类模型的精确率和召回率。F1分数可以看作是模型精确率和召回率的一种调和平均，它的最大值是1，最小值是0。(5) F1 score (F1Score) is an index used to measure the accuracy of a two-class model in statistics. It takes into account both the accuracy rate and recall rate of the classification model. F1 score can be regarded as a harmonic average of model accuracy and recall. Its maximum value is 1 and its minimum value is 0.

(6)交通信号可以分为交通信号灯、交通标志、交通标线等。(6) Traffic signals can be divided into traffic lights, traffic signs, and traffic markings.

(7)超文本传输协议请求(http请求)是指从客户端到服务器端的请求消息。包括：消息首行中，对资源的请求方法、资源的标识符及使用的协议。(7) Hypertext Transfer Protocol request (http request) refers to the request message from the client to the server. Including: In the first line of the message, the request method for the resource, the identifier of the resource, and the protocol used.

(8)Python是一种跨平台的计算机程序设计语言。是一种面向对象的动态类型语言，最初被设计用于编写自动化脚本(shell)，随着版本的不断更新和语言新功能的添加，越多被用于独立的、大型项目的开发。(8) Python is a cross-platform computer programming language. It is an object-oriented dynamically typed language. It was originally designed to write automated scripts (shell). With the continuous update of the version and the addition of new language features, the more it is used for the development of independent and large-scale projects.

下面先对本申请实施例所基于的其中一种系统架构进行描述，本申请实施例提出的图像处理方法可以应用于该系统架构。请参见图1，图1是本申请实施例提供的一种图像处理的系统架构示意图，如图1所示，包含了终端和服务器；该终端需要具备拍摄图片和通信(或者联网)功能；其中，拍摄图片的功能可以对特定区域或者场景下的车辆、行人或者目标物体进行拍摄；而通信(联网)功能可以将拍摄的图片发送给服务器，便于服务器进一步处理。本申请实施例中提及的终端可为摄像机、手机、平板电脑、笔记本电脑、掌上电脑、移动互联网设备或其他移动终端；其中，The following first describes one of the system architectures on which the embodiments of the present application are based, and the image processing method proposed in the embodiments of the present application can be applied to this system architecture. Please refer to Figure 1. Figure 1 is a schematic diagram of an image processing system architecture provided by an embodiment of the present application. , The function of taking pictures can take pictures of vehicles, pedestrians or target objects in a specific area or scene; and the communication (networking) function can send the taken pictures to the server for further processing. The terminal mentioned in the embodiment of this application may be a camera, a mobile phone, a tablet computer, a notebook computer, a palmtop computer, a mobile Internet device or other mobile terminals; among them,

终端，可以是计算机网络中处于网络最外围的设备，也可以用于信息(例如图像数据)的输入等。也可以称为系统、用户单元、用户站、移动站、移动台、远方站、远程终端、移动设备、用户终端、移动终端、无线通信设备、用户代理、用户装置、可安装插件的服务设备或用户设备(user equipment，UE)。例如，终端可以是蜂窝电话、移动电话、无绳电话、智能手表、可穿戴设备(wearable device)、平板设备、会话启动协议(session initiation protocol，SIP)电话、无线本地环路(wireless local loop，WLL)站、个人数字助手(personal digital assistant，PDA)、具备无线通信功能的手持设备、计算设备、车载通信模块、智能电表或连接到无线调制解调器的其它具备拍摄、通信等基本功能的处理设备。在图1中所示的系统框架中，终端可以按照一定的上传周期或者实时地将图像上传至服务器。本申请实施例对终端的数量不做限定，图中终端1(即卡口1)、终端2(卡口2)、……、终端N(卡口N)为示例性的数量，不代表服务器或者服务器组具体连接的终端数量。A terminal can be a device at the periphery of a computer network, or it can be used to input information (such as image data). It can also be called a system, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, mobile terminal, wireless communication device, user agent, user device, service device with plug-in installation or User equipment (user equipment, UE). For example, the terminal may be a cellular phone, a mobile phone, a cordless phone, a smart watch, a wearable device (wearable device), a tablet device, a session initiation protocol (SIP) phone, a wireless local loop (wireless local loop, WLL). ) Stations, personal digital assistants (PDAs), handheld devices with wireless communication functions, computing devices, in-vehicle communication modules, smart meters or other processing devices connected to wireless modems with basic functions such as shooting and communication. In the system framework shown in Figure 1, the terminal can upload images to the server according to a certain upload cycle or in real time. The embodiments of this application do not limit the number of terminals. In the figure, terminal 1 (ie bayonet 1), terminal 2 (bayonet 2), ..., terminal N (bayonet N) are exemplary numbers, and do not represent servers. Or the number of terminals specifically connected to the server group.

服务器，或者服务器组，是提供计算服务的设备。由于服务器需要响应服务请求，并进行处理，因此一般来说服务器应具备承担服务并且保障服务的能力。服务器的构成包括处理器、硬盘、内存、系统总线等，和通用的计算机架构类似，但是由于需要提供高可靠的服务，因此在处理能力、稳定性、可靠性、安全性、可扩展性、可管理性等方面要求较高。在网络环境下，根据服务器提供的服务类型不同，分为文件服务器、数据库服务器、应用程序服务器、WEB服务器等。在图中所示的服务器按照目标算法对图片数据进行算法识别；具体地，收集各个卡口的图像数据，然后根据图片对应的卡口名称以及图片大小(由图像尺寸和图像分辨率决定)，通过封装好的方法，将不同卡口的图片按照图片大小分别保存对应卡口文件夹下；根据每个卡口图片中，图片的不同排列方式，自动进行图片的切割(可以理解的是，一个摄像头可以拍摄多张图片而获得图像数据流包括了多张图片)，并从中选择一张作为标注图片，用于提取标注信息；通过标注图片中对应的违法标准，例如红绿灯位置，停止线位置，白实线位置等，自动生成相关标注信息，并根据图片分辨率的关系，通过封装好的自适应算法，将标注信息自适应匹配到上所有图片，从而完成批量标注的操作。A server, or server group, is a device that provides computing services. Since the server needs to respond to service requests and process them, in general, the server should have the ability to undertake the service and guarantee the service. The structure of the server includes processor, hard disk, memory, system bus, etc., which is similar to the general computer architecture. However, due to the need to provide highly reliable services, it is Requirements for management and other aspects are relatively high. In the network environment, according to the different types of services provided by the server, it is divided into file server, database server, application server, WEB server, etc. The server shown in the figure performs algorithmic recognition on the picture data according to the target algorithm; specifically, collects the image data of each bayonet, and then according to the bayonet name and picture size (determined by the image size and image resolution) corresponding to the picture, Through the encapsulation method, the pictures of different bayonet are saved in the corresponding bayonet folder according to the picture size; according to the different arrangement of the pictures in each bayonet picture, the picture is automatically cut (understandably, one The camera can take multiple pictures to obtain the image data stream including multiple pictures), and select one of them as annotated pictures for extracting annotation information; by annotating the corresponding illegal standards in the pictures, such as the position of the traffic light, the position of the stop line, The position of the white solid line, etc., automatically generates relevant annotation information, and according to the relationship of the picture resolution, through the encapsulated adaptive algorithm, the annotation information is adaptively matched to all the pictures, so as to complete the operation of batch annotation.

接着，按照不同文件夹(对应多个摄像头拍摄的图片集合)，读取标注好的文件，使用脚本批量测试文件夹内的所有图片，统计需要的结果。Then, according to different folders (corresponding to the collection of pictures taken by multiple cameras), read the marked files, use the script to batch test all the pictures in the folder, and count the required results.

最后，在测试结束后，自动将测试结果与预期结果进行匹配，统计下述公式需要的参数，从而得出该目标算法与应用场景的匹配程度。Finally, after the test is over, the test result is automatically matched with the expected result, and the parameters required by the following formula are counted to obtain the degree of matching between the target algorithm and the application scenario.

可以理解的是，图1所示的内容只是本申请实施例中的一种示例性的实施方式。本申请实施例中的系统架构可以包括但不仅限于以上系统架构。It can be understood that the content shown in FIG. 1 is only an exemplary implementation in the embodiments of the present application. The system architecture in the embodiment of the present application may include but is not limited to the above system architecture.

为了便于理解本申请实施例，以下示例性列举本申请中图像处理方法所应用的场景，可以包括车辆违规经过信号灯的场景：In order to facilitate the understanding of the embodiments of the present application, the following exemplarily enumerate the application scenarios of the image processing method in the present application, which may include a scenario where a vehicle passes a traffic light in violation of regulations:

请参见图2，图2是本申请实施例提供的一种图像处理方法所应用场景的示意图，该应用场景中包括终端(图2中以终端为摄像机为例)和服务器，而终端和服务器之间则可以通过网络等无线方式进行连接。如图2所示，当斑马线前的红绿灯显示行人可以通行的情况下，行人在预设的时间段内穿越斑马线达到对面。此时禁止车辆通过信号灯，必须停止在距离斑马线一定的距离内。假设车辆在此时发生了闯红灯的情况，即当信号灯禁止车辆继续前行的前提下车辆继续驶入斑马线；那么设置在路边的摄像头或者安装在红绿灯横杆上的抓拍设备会记录下发生该违规行为的一张或多张图片。可选地，当车辆驶离了当前的违规现场后，摄像机可以拍摄一张或多张违规结束后的现场图片。例如，在违规驾驶行为发生后拍摄的图片可以用于记录违规现场的现状；该现场可能包括车辆对现场或者行人造成的损坏，便于后续进行事故等级评定。可选地，当车辆出现在摄像机的拍摄范围内时，摄像机自动拍摄一张或多张图片。进一步可选地，摄像机按照一定的拍摄周期对拍摄范围内的各个角度或者场景进行拍摄。Please refer to Figure 2. Figure 2 is a schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application. The application scenario includes a terminal (the terminal is a camera as an example in Figure 2) and a server. The time can be connected via wireless means such as the network. As shown in Figure 2, when the traffic lights in front of the zebra crossing indicate that pedestrians can pass, the pedestrians cross the zebra crossing to the opposite side within a preset time period. At this time, vehicles are prohibited from passing the signal lights and must stop within a certain distance from the zebra crossing. Assuming that the vehicle is running a red light at this time, that is, when the traffic light prohibits the vehicle from moving forward, the vehicle continues to drive into the zebra crossing; then the camera installed on the roadside or the capture device installed on the traffic light crossbar will record the occurrence of this One or more pictures of the violation. Optionally, when the vehicle leaves the current violation scene, the camera can take one or more scene pictures after the violation ends. For example, the pictures taken after the illegal driving behavior occurs can be used to record the current situation of the illegal scene; the scene may include the damage caused by the vehicle to the scene or pedestrians, which is convenient for subsequent accident level assessment. Optionally, when the vehicle appears within the shooting range of the camera, the camera automatically takes one or more pictures. Further optionally, the camera shoots various angles or scenes within the shooting range according to a certain shooting period.

可以理解的是，图2中的应用场景的只是本申请实施例中的几种示例性的实施方式，本申请实施例中的应用场景包括但不仅限于以上应用场景。It can be understood that the application scenarios in FIG. 2 are only a few exemplary implementations in the embodiments of the present application, and the application scenarios in the embodiments of the present application include but are not limited to the above application scenarios.

下面结合上述系统架构和本申请实施例中提供的图像处理方法的实施例，对本申请实施例中提出的技术问题进行具体分析和解决。In the following, in combination with the foregoing system architecture and the embodiments of the image processing method provided in the embodiments of the present application, the technical problems proposed in the embodiments of the present application will be specifically analyzed and resolved.

请参见图3，图3是本申请实施例提供的一种图像处理方法的流程示意图，图像处理方法可以应用于图像处理系统(包括上述架构)；本申请实施例的图像处理方法，具体可以应用于交通违规识别的场景。下面将结合图3，以服务器为执行主体为例进行描述，该方法可以包括以下步骤S301-步骤S305；其中，可选的步骤可以包括步骤S304和步骤S305。Please refer to FIG. 3, which is a schematic flowchart of an image processing method provided by an embodiment of the present application. The image processing method can be applied to an image processing system (including the foregoing architecture); the image processing method of the embodiment of the present application can be specifically applied It is used in the scene of traffic violation identification. The following will describe with reference to FIG. 3, taking the server as the execution subject as an example. The method may include the following steps S301 to S305; wherein, the optional steps may include step S304 and step S305.

步骤S301：确定第一图像集合。Step S301: Determine the first image set.

具体地，服务器在接收到多个摄像机(卡口)发送的图像数据后，先对图像数据进行分类，确定出某一个场景下预设时间段内的拍摄图像数据。所述第一图像集合包括多张第一图像，所述多张第一图像为同一场景下相同时间段内针对目标车辆的拍摄图像。例如，首先分析图像数据，根据图像标识(例如图像对应的卡口编号或者拍摄某一场景下的摄像机的名称，本申请实施例以卡口为例进行说明)，将图像保存至对应的卡口的文件夹下。然后对每个卡口文件夹下的图片进行整理；例如，根据图片的分辨率和预设的分类规则，对文件夹下的图片进行分类。不同分辨率的图片，对应标注的标准信息的位置不同。例如，白天拍摄的图片和夜晚拍摄的图片分辨率不同，那么同一个摄像头拍摄的图片中，比如红绿灯的位置可能有一定程度的偏差。为了便于“违法识别算法”进行判断，提前对图片的分辨率进行预处理。例如，同一个卡口(即摄像机拍摄的多张图像)可以存储于两类文件夹中，如白天的拍摄图像和夜晚的拍摄图像。Specifically, after receiving the image data sent by multiple cameras (bayonet), the server first classifies the image data, and determines the captured image data within a preset time period in a certain scene. The first image set includes a plurality of first images, and the plurality of first images are captured images of the target vehicle in the same time period in the same scene. For example, first analyze the image data, and save the image to the corresponding bayonet according to the image identification (such as the bayonet number corresponding to the image or the name of the camera under which a certain scene was shot, the embodiment of this application takes the bayonet as an example). Under the folder. Then organize the pictures in each bayonet folder; for example, classify the pictures in the folder according to the resolution of the pictures and preset classification rules. Images of different resolutions have different positions of the standard information corresponding to the annotations. For example, if the resolution of the pictures taken during the day and the pictures taken at night are different, the positions of traffic lights, for example, may have a certain degree of deviation in the pictures taken by the same camera. In order to facilitate the judgment of the "illegal recognition algorithm", the resolution of the picture is preprocessed in advance. For example, the same bayonet (ie, multiple images taken by a camera) can be stored in two types of folders, such as images taken during the day and images taken at night.

可选地，将图片保存至对应文件夹下，以及对文件夹下的图片进行整理，这2个步骤可以同步完成。可以理解的是，卡口可以指的是在道路交通治安卡口的摄像头；比如，在道路上特定场所如收费站、交通或治安检查站等，对所有通过该点的机动车辆进行拍摄、记录。每一个摄像头都会有各自的摄像头编号；并且每个摄像头抓拍出来的图片类型是固定的，例如，图片的格式、图像分辨率、图片像素等。图片类型的内容还可以包括该摄像头的拍摄场景。例如，摄像头A安装在高速路口，那么A拍摄的图片就只会是出入该高速路口车辆的相关图片；再例如，摄像头B安装在十字路口，那么B拍摄的图片就是不同时间段内这个路段的机动车、非机动车以及行人等行驶情况或者抓拍违规行驶的情况。Optionally, save the pictures in the corresponding folder, and organize the pictures in the folder, these two steps can be completed synchronously. It is understandable that a bayonet can refer to a camera at a road traffic security bayonet; for example, in a specific place on the road, such as a toll station, traffic or security checkpoint, etc., all motor vehicles passing through the point are photographed and recorded . Each camera will have its own camera number; and the type of pictures captured by each camera is fixed, for example, the format of the picture, the resolution of the picture, and the picture pixel. The content of the picture type may also include the shooting scene of the camera. For example, if camera A is installed at a high-speed intersection, the pictures taken by A will only be related pictures of vehicles entering and leaving the high-speed intersection; for another example, if camera B is installed at an intersection, then the pictures taken by B are of this road section in different time periods Motor vehicles, non-motor vehicles and pedestrians, etc., or capture illegal driving situations.

可选地，每个摄像头拍摄的图片携带有该摄像头的编号信息，用于指示出是哪个摄像头拍摄的图片。Optionally, the picture taken by each camera carries the number information of the camera, which is used to indicate which camera took the picture.

可选地，第一图像集合中的第一图像可以是图像尺寸和图像分辨率相同的图像；在该同一场景下，可以是不同摄像机器(保持图像尺寸和图像分辨率一致，便于后续服务器将该特征作为筛选的标准筛选该场景下拍摄的一系列图片)从不同角度进行拍摄的。Optionally, the first image in the first image set can be an image with the same image size and image resolution; in the same scene, it can be a different camera (keep the image size and image resolution consistent, so that subsequent servers can This feature is used as a screening criterion to screen a series of pictures taken in the scene) taken from different angles.

所述多张第一图像中每一张第一图像包括指示所述第一图像的标识；所述确定第一图像集合，包括：根据所述第一图像的标识确定所述第一图像集合。具体地，所述第一图像的标识可以包括摄像机的编号、拍摄场景名称、与摄像机对应的违规行为的名称。Each first image in the plurality of first images includes an identifier indicating the first image; the determining the first image set includes: determining the first image set according to the identifier of the first image. Specifically, the identification of the first image may include the number of the camera, the name of the shooting scene, and the name of the illegal behavior corresponding to the camera.

在一种可能的实现方式中，获取多张第三图片，所述多张第三图片中每一张第三图片的图像尺寸以及图像分辨率相同或者不同；从所述多张第三图片中，确定图像尺寸和图像分辨率相同的所述多张第一图片。其中，所述第三图片可以包括所有摄像机拍摄的图像。在确定第一图像集合之前，获取到每个摄像机的拍摄图像。In a possible implementation manner, a plurality of third pictures are acquired, and the image size and image resolution of each third picture in the plurality of third pictures are the same or different; from the plurality of third pictures , Determining the plurality of first pictures with the same image size and image resolution. Wherein, the third picture may include images taken by all cameras. Before determining the first image set, the captured images of each camera are acquired.

可选地，将图片按照不同的拍摄机器(卡口、摄像头)拍摄的面积进行分类；或者按照图片面积和图片分辨率，对多张图片进行分成N类，其中，N类中的每一类对应预设的图片面积和图片分辨率；每一类包括多组图片；每一组图片包括多张图片；每一组图片对应一次违法行为。Optionally, the pictures are classified according to the area taken by different shooting machines (bayonet, camera); or according to the picture area and picture resolution, multiple pictures are divided into N categories, where each of the N categories Corresponding to the preset picture area and picture resolution; each category includes multiple sets of pictures; each set of pictures includes multiple pictures; each set of pictures corresponds to an illegal act.

步骤S302：从所述第一图像集合中确定一个或多个第二图像集合。Step S302: Determine one or more second image sets from the first image set.

具体地，服务器从第一图像集合中筛选出与疑似违规行驶的车辆的相关图片，将每一次疑似违规行为涉及的所有图片作为第二图像集合，从而确定出一个或多个第二图像集合。即，在第一图像集合中包含了一辆或者多辆目标车辆的一次或者多次待定违规驾驶行为的图像。可选地，一个卡口的摄像设备只初步识别并拍摄一种类型的待定违规驾驶行为。Specifically, the server filters out pictures related to vehicles suspected of driving in violation from the first image set, and uses all pictures involved in each suspected violation as the second image set, thereby determining one or more second image sets. That is, the first image set contains images of one or more undetermined driving violations of one or more target vehicles. Optionally, a camera device with a bayonet only initially recognizes and photographs one type of pending illegal driving behavior.

可选地，根据图片的拍摄规则从第一图像集合的子集中确定一个或多个第二图像集合。例如，在初步识别车辆A发生了违规行为(如闯红灯)时，进行了拍摄。在车辆A进入视野时且红灯亮起时进行拍摄；在车辆A闯过红灯继续前进时进行拍摄。本申请实施例对拍摄的图像数量不作限定。在一种可能的实现方式中，所述第二图片的拍摄规则为在所述一次违规行驶过程发生前拍摄一张或多张所述第二图片；在所述一次违规行驶过程发生时拍摄一张或多张所述第二图片；在所述一次违规行驶过程发生后拍摄一张或多张所述第二图片。Optionally, one or more second image sets are determined from a subset of the first image set according to the shooting rules of the pictures. For example, when it is preliminarily identified that vehicle A has violated regulations (such as running a red light), a photograph is taken. Shoot when vehicle A enters the field of view and the red light is on; shoot when vehicle A passes the red light and continues to move forward. The embodiments of the present application do not limit the number of captured images. In a possible implementation manner, the shooting rule of the second picture is that one or more second pictures are taken before the one driving violation occurs; one or more second pictures are taken when the one driving violation occurs. One or more of the second pictures; one or more of the second pictures are taken after the occurrence of the one driving violation.

例如，获取在一段时间内每个卡口拍摄的图片数据流；卡口对于拍摄的图片数据会做相应的标记，特别地，对于违法行为，卡口在拍摄后对得到图片数据会有做出区分；例如，对于未检测到违法行为的情况下，拍摄的图片数据的标识为A；在对违法行为进行拍摄的情况下，得到的图片数据的标识为B；当违法行为结束后进行拍摄的情况下，得到的图片数据的标识为C。可选地，标识为A或者标识为B或者标识为C的图片至少为一张。For example, to obtain the data stream of pictures taken by each bayonet within a period of time; the bayonet will mark the captured picture data accordingly. In particular, for illegal activities, the bayonet will make a response to the image data obtained after shooting. Distinguish; For example, in the case of no illegal act detected, the image data captured is marked as A; in the case of shooting illegal acts, the obtained image data is marked as B; when the illegal act is over, the image is taken In this case, the identifier of the obtained picture data is C. Optionally, there is at least one picture identified as A or B or C.

再例如，在前述的三类图片均为一张的情况下，卡口的图片排列方式为一行三列；在前述三类图片均为2张的情况下，卡口的图片排列方式为二行三列。For another example, in the case where the aforementioned three types of pictures are all one, the picture arrangement of the bayonet is one row and three columns; when the aforementioned three types of pictures are all two, the picture arrangement of the bayonet is two rows Three columns.

可以理解的是，本申请实施例对标识的数量不作限定，如ABC等。还可以对整个违法行为(包含违法前、违法时以及违法后进行细分)。在细分的情况下，卡口得到的图片排列方式可以为M行N列，即对违法行为的全过程做了N个阶段的划分，每个阶段有M张图片(M，N都为大于0的整数)。根据每个卡口的图片排列方式(如，一行三列的图片，共三张)对应的图片数据标识，对图片数据流进行切割，获得多组图片(一组图片为一个阶段的图片集合)；其中，每一组图片中包含了至少一张与车辆相关的图片。对图片进行切割，主要取决于对应卡口拍摄出来的图片的排列方式，不再赘述。It can be understood that the embodiment of the present application does not limit the number of identifiers, such as ABC. You can also subdivide the entire illegal behavior (including before, during, and after the violation). In the case of subdivision, the picture arrangement method obtained by the bayonet can be M rows and N columns, that is, the whole process of illegal acts is divided into N stages, and each stage has M pictures (M, N are all greater than An integer of 0). According to the picture data identification corresponding to the picture arrangement method of each bayonet (for example, three pictures in one row and three columns), the picture data stream is cut to obtain multiple sets of pictures (a set of pictures is a set of pictures in one stage) ; Among them, each group of pictures contains at least one picture related to the vehicle. Cutting the picture mainly depends on the arrangement of the pictures taken by the corresponding bayonet, so I won’t go into details here.

可选地，在一组图片(即图像集合)包含一张图片的情况下，将该组图片(即某个阶段拍摄的一张图片)从数据流中识别并切割出来。可选地，在一组图片包含多张图片的情况下，先将该组图片切割出来，再对从该组图片中切割出多张图片。进一步可选地，对图片按照分辨率进行分类，再对同一类分辨率的图片数据完成切割后，将在该卡口的图片行列中，依次填入该分辨率的若干图片；例如，在十字路口，该卡口为该路口的一个摄像头，专用于抓拍车辆闯红灯。为了能够识别车辆的行为是闯红灯的行为，输入算法的图片排列模式为一行三列的图片组(例如在白天，光强为1000lx等情况下拍摄的违法行为的相关图片的集合)。根据一行三列的图片排列方式，在一行三列中依次填入三张分辨率相同(即某分辨率)的图片，如，图片A为车辆在停止线内的图片，图片B为车辆覆盖了停止线的图片，图片C为车辆超过停止线的图片。后续算法根据这三张图片进行判断。可以理解的是，只根据图A和图B是无法准确判断车辆进行了闯红灯行为。Optionally, in a case where a group of pictures (that is, an image collection) includes a picture, the group of pictures (that is, a picture taken at a certain stage) is identified and cut out from the data stream. Optionally, in a case where a group of pictures includes multiple pictures, the group of pictures is first cut out, and then the group of pictures is cut out to cut out multiple pictures. Further optionally, the pictures are classified according to the resolution, and after the picture data of the same type of resolution is cut, several pictures of the resolution are sequentially filled in the picture row of the bayonet; for example, in the crossroads The bayonet is a camera at the intersection, which is dedicated to capturing vehicles running red lights. In order to be able to recognize that the behavior of the vehicle is the behavior of running a red light, the picture arrangement mode of the input algorithm is a group of pictures in one row and three columns (for example, a collection of related pictures of illegal behaviors taken in the daytime with a light intensity of 1000lx). According to the arrangement of the pictures in one row and three columns, fill in three pictures with the same resolution (ie a certain resolution) in one row and three columns. The picture of the stop line, picture C is the picture of the vehicle passing the stop line. The subsequent algorithm judges based on these three pictures. It is understandable that it is impossible to accurately determine that the vehicle has made a red light running behavior based on only Figure A and Figure B.

步骤S303：标注所述一个或多个第二图像集合中至少一个第二图像集合中的交通标识的参考位置，并分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，以判断所述一个或多个第二图像集合分别对应的待定违规行驶是否违规。Step S303: Annotate the reference position of the traffic sign in the at least one second image set in the one or more second image sets, and respectively identify the position change of the target vehicle in the one or more second image sets , To determine whether the pending illegal driving respectively corresponding to the one or more second image sets is illegal.

具体地，服务器对第二图像集合中每一个集合包含的第二图像中交通标识进行标注，例如，对信号灯、白实线、转向标志等等进行标注。可选地，不同违规判断场景下，针对不同违规行驶设置不同的交通标识的参考位置。例如，在闯红灯的识别场景下，红绿灯、斑马线等内容是重要的必须标记的交通标识之一。Specifically, the server labels the traffic signs in the second images included in each of the second image sets, for example, labels signal lights, solid white lines, turn signs, and so on. Optionally, in different violation judgment scenarios, different reference positions of traffic signs are set for different violation driving. For example, in the recognition scene of running a red light, traffic lights, zebra crossings, etc. are one of the important traffic signs that must be marked.

在一种可能的实现方式中，所述标注所述一个或多个第二图像集合中至少一个第二图像集合中的交通标识的参考位置，并分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，以判断所述一个或多个第二图像集合分别对应的待定违规行驶是否违规，包括：标注所述一个或多个第二图像集合中目标第二图像集合中交通标识的参考位置；根据所述目标第二图像集合中交通标识的参考位置和位置标注算法，标注所述一个或多个第二图像集合中除所述目标第二图像集合外的第二图像集合中交通标识的参考位置。例如，对某卡口的首张图片(即选择的一张需要人工标注的图片)，进行人工标注“违法标准信息”，标注图片中的“违法标注信息”，例如，红绿灯位置，停止线位置，白实线位置等。前述这些位置，是算法判断车辆违法的参考位置之一。In a possible implementation manner, the mark the reference position of the traffic sign in the at least one second image set in the one or more second image sets, and respectively identify the one or more second image sets The change in the position of the target vehicle in the second image set to determine whether the pending illegal driving respectively corresponding to the one or more second image sets violates the rules, including: marking the target second image set in the one or more second image sets The reference position of the traffic sign; according to the reference position of the traffic sign in the target second image set and the location labeling algorithm, label the second image in the one or more second image sets except the target second image set The reference position of the traffic sign in the collection. For example, for the first picture of a certain bayonet (that is, a selected picture that needs to be manually marked), manually mark the "violation standard information" and mark the "violation mark information" in the picture, for example, the position of the traffic light, the position of the stop line , The position of the white solid line, etc. The aforementioned locations are one of the reference locations for the algorithm to determine the vehicle's illegality.

例如，服务器对某卡口的首张图片(即选择的一张需要人工标注的图片)，进行人工标注“违法标准信息”，标注图片中的“违法标注信息”，例如，红绿灯位置，停止线位置，白实线位置等。前述这些位置，是算法判断车辆违法的参考位置之一。再例如，交通识别算法针对这些标注位置、该场景的交通规则和预设的判断规则，判断输入的图片组中的车辆行为是否违法。进一步地，从前述的一行三列的三张图片中，选择一张图片作为标注图片，在完成对该图片的标注后，提取图中的标注信息。可以理解的是，选择的标注图片中，能够涵盖这一类图片(如闯红灯的图片组)基本特征值。For example, the server manually annotates the first picture of a certain bayonet (that is, a selected picture that needs to be manually marked), and annotates the "infringement standard information" in the picture, such as the position of the traffic light, and the stop line. Position, white solid line position, etc. The aforementioned locations are one of the reference locations for the algorithm to determine the vehicle's illegality. For another example, the traffic recognition algorithm determines whether the vehicle behavior in the input picture group is illegal according to the marked locations, the traffic rules of the scene, and the preset judgment rules. Further, from the aforementioned three pictures in one row and three columns, one picture is selected as the annotated picture, and after the annotation of the picture is completed, the annotation information in the figure is extracted. It is understandable that the selected annotated pictures can cover the basic feature values of this type of picture (for example, a picture group running a red light).

标注完一组图片中的一个图片后，可以对同组的其他图片进行标注。具体地，对于某个卡口的某分辨率的图片，将标注图片上的标注信息通过自适应算法匹配到上所有相同分辨率图片，从而完成批量标注的操作。可选地，通过传入自适应算法的图片排列格式，单张含有标注信息的标注图片以及对应的图片分辨率参数，可以对相同卡口的其他相同分辨率的图片进行批量地增加标注信息。After marking one picture in a group of pictures, you can mark other pictures in the same group. Specifically, for a picture with a certain resolution of a certain bayonet, the label information on the label picture is matched to all the pictures with the same resolution through an adaptive algorithm, thereby completing the operation of batch labeling. Optionally, by passing in the picture arrangement format of the adaptive algorithm, a single annotated picture containing annotated information and corresponding picture resolution parameters, annotated information can be added in batches to other pictures of the same resolution of the same bayonet.

当完成一个卡口的一组图片的特征提取后，后续传入该卡口的其他组图片，可以进行相同的操作，如分类、切割、标注等图片预处理，从而完成批量标注的操作。其中，标注图片的标注信息和图片分辨率是自适应算法的参数。After the feature extraction of a set of pictures of a bayonet is completed, the other sets of pictures that are subsequently transferred to the bayonet can perform the same operations, such as image preprocessing such as classification, cutting, and annotation, to complete the operation of batch annotation. Among them, the annotation information of the annotated picture and the picture resolution are parameters of the adaptive algorithm.

在一种可能的实现方式中，所述分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，以判断所述一个或多个第二图像集合分别对应的待定违规行驶是否违规，包括：通过交通识别算法分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，获得所述一个或多个第二图像集合分别对应的待定违规行驶的判断结果；所述判断结果包括违规驾驶和非违规驾驶。In a possible implementation manner, the position change of the target vehicle in the one or more second image sets is identified respectively to determine the pending illegal driving corresponding to the one or more second image sets respectively Whether to violate the regulations, including: respectively identifying the position change of the target vehicle in the one or more second image sets through a traffic recognition algorithm, and obtaining the judgment results of pending illegal driving respectively corresponding to the one or more second image sets ; The judgment result includes illegal driving and non-violating driving.

例如，在不同文件夹中，读取标注好的文件(即标准好的图片)，放入HTTP请求中。然后，将批量生成的标注信息和被测图片，通过发送该请求，发送给被测算法。使用python脚本对全部文件夹的所有图片进行自动化测试，统计结果。具体地，使用python脚本(包含了待评估的用于违法行为识别的算法)，批量测试某个文件夹内的所有图片。在测试过程中，一般需要监控目标算法的接口，对传入图片、反馈的结果以及预期的结果进行分析，确定该算法的准确度。可以理解的是，图片识别算法基本上都是提供一个图片传入的接口给服务应用者，并通过该接口返回对应的结果，所以需要通过监控该接口的传入图片与对应结果。For example, in different folders, read the marked files (ie, standard pictures) and put them in the HTTP request. Then, the batch generated annotation information and tested pictures are sent to the tested algorithm by sending the request. Use python scripts to perform automated tests on all pictures in all folders and count the results. Specifically, a python script (including the algorithm for identifying illegal behaviors to be evaluated) is used to batch test all pictures in a certain folder. In the test process, it is generally necessary to monitor the interface of the target algorithm, analyze the incoming pictures, the feedback results, and the expected results to determine the accuracy of the algorithm. It is understandable that the image recognition algorithm basically provides an interface for incoming pictures to the service application, and returns corresponding results through the interface, so it is necessary to monitor the incoming pictures and corresponding results of the interface.

可选地，可以根据图片对应的卡口名称，图片大小，通过封装好的方法，将不同卡口的图片按照图片大小分别保存对应卡口文件夹下；根据每个卡口图片中，图片的不同排列方式，自动进行图片的切割(一个摄像头可以拍摄多个图片)，并从中选择一张作为标注图片，用于提取标注信息。其中提取的标注信息可以用于对同一类型的图像进行标注。Optionally, according to the name of the bayonet corresponding to the picture and the size of the picture, the pictures of different bayonets can be saved in the corresponding bayonet folder according to the picture size through the encapsulation method; according to the picture of each bayonet, the picture Different arrangement methods, automatic cutting of pictures (a camera can take multiple pictures), and select one of them as annotated pictures for extracting annotation information. The extracted annotation information can be used to annotate images of the same type.

步骤S304：将所述一个或多个第二图像集合分别对应的待定违规行驶的判断结果和所述一个或多个第二图像集合分别对应的参考结果进行对比。Step S304: Compare the determination results of the pending illegal driving corresponding to the one or more second image sets with the reference results respectively corresponding to the one or more second image sets.

具体地，服务器将交通识别算法针对第二图像集合中每一个集合对应的待定违规行驶给出的结果，与真实的判断结果进行比较。其中，真实的判断结果可以是根据人工判断或者以某一个准确率最高的识别算法根据相同方法或计算方式得出的结论。例如，将测试结果(即预测结果)与预期结果(即下表中对应的真实情况)进行匹配，其中，预期的结果是针对传入算法接口的数据(即图片)进行人为分析和确定，图片中涉及的行为是否违法。该预期的结果可以直接从第三方数据库(如道路交通监控系统等等)中获得。Specifically, the server compares the result of the traffic recognition algorithm for the pending illegal driving corresponding to each set in the second image set with the real judgment result. Among them, the true judgment result may be a conclusion drawn based on manual judgment or a certain recognition algorithm with the highest accuracy according to the same method or calculation method. For example, match the test result (that is, the predicted result) with the expected result (that is, the corresponding real situation in the following table), where the expected result is the artificial analysis and determination of the data (that is, the picture) passed into the algorithm interface, the picture Whether the behavior involved in is illegal. The expected result can be obtained directly from a third-party database (such as a road traffic monitoring system, etc.).

步骤S305：评估所述交通识别算法的准确度。Step S305: Evaluate the accuracy of the traffic recognition algorithm.

具体地，服务器通过一种或多种算法评估指标对交通识别算法进行比较。按照前述的步骤对图像处理，但是每一次识别都更换不同的测试的算法；每一种完成识别测试的算法会得到结果报告，以评估该算法与当前识别场景的匹配程度。Specifically, the server compares the traffic recognition algorithms through one or more algorithm evaluation indicators. The image is processed according to the aforementioned steps, but a different test algorithm is replaced for each recognition; each algorithm that completes the recognition test will get a result report to evaluate the matching degree of the algorithm with the current recognition scene.

在一种可能的实现方式中，所述评估所述交通识别算法的准确度，包括：通过查准率、查全率以及平衡F分数评估所述交通识别算法的准确度。例如，统计下述公式需要的参数，从而得出交通识别算法的精度。请参见表1，表1列出了交通识别算法的测试结果与真实结果的对比表；In a possible implementation manner, the evaluating the accuracy of the traffic recognition algorithm includes: evaluating the accuracy of the traffic recognition algorithm through a precision rate, a recall rate, and a balanced F score. For example, calculate the parameters required by the following formula to obtain the accuracy of the traffic recognition algorithm. Please refer to Table 1. Table 1 lists the comparison table between the test results of the traffic recognition algorithm and the real results;

表1Table 1

那么，查准率(Precision)在被算法预测为正的样本中，真正为正的样本比例为：Then, among the samples predicted to be positive by the algorithm, the proportion of samples that are truly positive is:

其中，P表示查准率的概率，TP为测试结果为均为真正例的数量，FP为测试结果(即交通识别算法对图像给出的结果)为假正例(即真实情况不违规，而待测算法评估违规的例子)的数量。正例表示判断为违规，而反例表示判断不违规。Among them, P represents the probability of precision, TP is the number of real cases in the test result, and FP is the test result (that is, the result given by the traffic recognition algorithm on the image) is a false positive case (that is, the real situation does not violate the rules, and The number of examples of violations evaluated by the algorithm to be tested). A positive example means that it is judged to be a violation, while a negative example means that it is judged not to be a violation.

查全率(Recall)在所有正的样本上，被算法预测为正的样本比例为：The recall rate (Recall) on all positive samples, the proportion of samples predicted to be positive by the algorithm is:

其中，FN为测试算法判断结果为违规且真实情况为不违规的情况。TN为测试算法判断结果为违规且真实情况为不违规的情况。Among them, FN is a situation where the test algorithm judges that the result is a violation and the real situation is not a violation. TN is a situation where the test algorithm judges that the result is a violation and the real situation is not a violation.

平衡F分数又称为F1-Score，被定义为Precision和Recall的调和平均数。如下：The balanced F score is also called F1-Score, which is defined as the harmonic average of Precision and Recall. as follows:

使用封装好的分发脚本，将测试结果标注在图片上，并将不同测试结果的对应图片分类管理；例如，违法行为的图片放在一个文件夹中，非违法行为的图片放在另一个文件夹。可选地，对于违法行为的图片还可以细分违法行为的类型，按照违法行为的类型对同一文件夹下的图片再分文件夹存放，便于后续查看。Use the packaged distribution script to mark the test results on the pictures, and categorize and manage the corresponding pictures of different test results; for example, pictures of illegal acts are placed in one folder, and pictures of non-illegal acts are placed in another folder . Optionally, for pictures of illegal acts, the types of illegal acts can be further subdivided, and the pictures in the same folder are stored in folders according to the types of illegal acts, so as to facilitate subsequent viewing.

可选地，同时将相关记录转换成测试报告，记录算法比例以及失败原因。Optionally, the relevant records are converted into test reports at the same time, and the algorithm ratio and the reason for the failure are recorded.

可选地，测试结果可以包含查准率、查全率以及平衡F分数，同时还可以包含一些其他的结果，比如算法的处理单张图片的平均时间、最大时间以及最小时间等等。Optionally, the test result may include precision, recall, and balanced F score, and may also include some other results, such as the average time, maximum time, and minimum time of the algorithm to process a single image.

实施本申请实施例，主要通过预处理待测图片数据和分析算法的结果，提高了算法评估效率。根据卡口的图片排列方式，将接收到的图片数据，切割成固定排列方式的图片。先对某一张图片进行标注，然后依据该被标注的图片批量标注剩余的相同分辨率的图片。(同理，对其他分辨率的图片也进行相同操作)。将被标注好的图片输入待测算法，得到结果；并从查全率、查准率等指标来评估算法的匹配度。The implementation of the embodiments of the present application mainly improves the efficiency of algorithm evaluation by preprocessing the image data to be tested and analyzing the results of the algorithm. According to the picture arrangement of the bayonet, the received picture data is cut into pictures with a fixed arrangement. First annotate a certain picture, and then annotate the remaining pictures of the same resolution in batches based on the annotated picture. (Similarly, do the same for pictures of other resolutions). Input the marked picture into the algorithm to be tested, and get the result; and evaluate the matching degree of the algorithm from indexes such as recall rate and precision rate.

上述详细阐述了本申请实施例的方法，下面提供了本申请实施例的相关装置。The foregoing describes the method of the embodiment of the present application in detail, and the relevant device of the embodiment of the present application is provided below.

请参见图4，图4是本申请实施例提供的一种图像处理装置的结构示意图，图像处理装置40可以包括确定单元401、筛选单元402、标注单元403和评估单元404。其中，可选的单元包括评估单元404。Please refer to FIG. 4, which is a schematic structural diagram of an image processing apparatus provided by an embodiment of the present application. Theimage processing apparatus 40 may include a determiningunit 401, ascreening unit 402, alabeling unit 403, and anevaluation unit 404. Among them, the optional unit includes anevaluation unit 404.

确定单元401，用于确定第一图像集合，所述第一图像集合包括多张第一图像，所述多张第一图像为同一场景下相同时间段内针对目标车辆的拍摄图像；The determiningunit 401 is configured to determine a first image set, where the first image set includes a plurality of first images, and the plurality of first images are captured images of a target vehicle in the same scene and the same time period;

筛选单元402，用于从所述第一图像集合中确定一个或多个第二图像集合，所述一个或多个第二图像集合中每一个第二图像集合包括多张第二图像，所述每一个第二图像集合记录了所述目标车辆的一次待定违规行驶过程；Thescreening unit 402 is configured to determine one or more second image sets from the first image set, and each second image set in the one or more second image sets includes multiple second images, and Each second image set records a pending illegal driving process of the target vehicle;

标注单元403，用于标注所述一个或多个第二图像集合中至少一个第二图像集合中的交通标识的参考位置，并分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，以判断所述一个或多个第二图像集合分别对应的待定违规行驶是否违规。Thelabeling unit 403 is configured to label the reference position of the traffic sign in the at least one second image set in the one or more second image sets, and respectively identify the target vehicle in the one or more second image sets To determine whether the pending illegal driving corresponding to the one or more second image sets is illegal.

在一种可能的实现方式中，所述多张第一图像中每一张第一图像包括指示所述第一图像的标识；所述确定单元401，具体用于根据所述第一图像的标识确定所述第一图像集合。In a possible implementation manner, each of the plurality of first images includes an identifier indicating the first image; the determiningunit 401 is specifically configured to perform according to the identifier of the first image Determine the first image set.

在一种可能的实现方式中，所述多张第一图像中每一张第一图像的图像尺寸和图像分辨率相同。In a possible implementation manner, the image size and image resolution of each first image in the plurality of first images are the same.

在一种可能的实现方式中，所述标注单元403，具体用于：In a possible implementation manner, thelabeling unit 403 is specifically configured to:

标注所述一个或多个第二图像集合中目标第二图像集合中交通标识的参考位置；根据所述目标第二图像集合中交通标识的参考位置和位置标注算法，标注所述一个或多个第二图像集合中除所述目标第二图像集合外的第二图像集合中交通标识的参考位置。Label the reference position of the traffic sign in the target second image set in the one or more second image sets; label the one or more traffic signs according to the reference position of the traffic sign in the target second image set and the location labeling algorithm The reference position of the traffic sign in the second image set except the target second image set in the second image set.

在一种可能的实现方式中，所述标注单元403，具体用于：通过交通识别算法分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，获得所述一个或多个第二图像集合分别对应的待定违规行驶的判断结果；所述判断结果包括违规驾驶和非违规驾驶。In a possible implementation manner, thelabeling unit 403 is specifically configured to: separately recognize the position change of the target vehicle in the one or more second image sets through a traffic recognition algorithm, and obtain the one or more The two second image sets respectively correspond to pending judgment results of illegal driving; the judgment results include illegal driving and non-violating driving.

在一种可能的实现方式中，所述装置还包括评估单元404，用于：In a possible implementation manner, the device further includes anevaluation unit 404, configured to:

将所述一个或多个第二图像集合分别对应的待定违规行驶的判断结果和所述一个或多个第二图像集合分别对应的参考结果进行对比；评估所述交通识别算法的准确度。Comparing the judgment result of the pending illegal driving respectively corresponding to the one or more second image sets with the reference results respectively corresponding to the one or more second image sets; evaluating the accuracy of the traffic recognition algorithm.

在一种可能的实现方式中，所述评估单元404，具体用于：In a possible implementation manner, theevaluation unit 404 is specifically configured to:

通过查准率、查全率以及平衡F分数评估所述交通识别算法的准确度。The accuracy of the traffic recognition algorithm is evaluated by the precision rate, the recall rate and the balanced F score.

需要说明的是，本申请的装置实施例中所描述的图像处理装置40的各功能单元的功能，可参见上述图2所述的方法实施例中图像处理方法的相关描述，此处不再赘述。It should be noted that, for the functions of each functional unit of theimage processing device 40 described in the device embodiment of the present application, please refer to the relevant description of the image processing method in the method embodiment described in FIG. 2, which will not be repeated here. .

本申请实施例提供了一种图像处理设备50，请参见图5，图5是本申请实施例提供的一种图像处理设备的结构示意图，如图5所示，图像处理装置60能以图5的结构实现，图像处理设备50可以包括至少一个存储部件501、至少一个处理部件502、至少一个通信部件503。此外，该设备还可以包括天线、电源等通用部件，在此不再详述。An embodiment of the present application provides animage processing device 50. Please refer to FIG. 5. FIG. 5 is a schematic structural diagram of an image processing device provided in an embodiment of the present application. As shown in FIG. The structure of theimage processing device 50 may include at least onestorage component 501, at least oneprocessing component 502, and at least onecommunication component 503. In addition, the device may also include general components such as an antenna and a power supply, which are not described in detail here.

存储部件501可以包括一个或多个存储单元，每个单元可以包括一个或多个存储器，存储部件可用于存储程序和各种数据，并能在通用设备50运行过程中高速、自动地完成程序或数据的存取。可以采用具有两种稳定状态的物理器件来存储信息，所述两种稳定状态分别表示为“0”和“1”。前述存储部件501，可以是只读存储器(read-only memory，ROM)或可存储静态信息和指令的其他类型的静态存储设备，随机存取存储器(random access memory，RAM)或者可存储信息和指令的其他类型的动态存储设备，也可以是电可擦可编程只读存储器(Electrically Erasable Programmable Read-Only Memory，EEPROM)、只读光盘(Compact Disc Read-Only Memory，CD-ROM)或其他光盘存储、光碟存储(可以包括压缩光碟、激光碟、光碟、数字通用光碟、蓝光光碟等)、磁盘存储介质或者其他磁存储设备、或者能够用于携带或存储具有指令或数据结构形式的期望的程序代码并能够由计算机存取的任何其他介质，但不限于此。存储器可以是独立存在，通过总线与处理器相连接。存储器也可以和处理器集成在一起。Thestorage component 501 may include one or more storage units, and each unit may include one or more memories. The storage component can be used to store programs and various data, and can complete programs or programs at high speed and automatically during the operation of the general-purpose device 50. Data access. A physical device with two stable states can be used to store information, and the two stable states are represented as "0" and "1", respectively. Theaforementioned storage component 501 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), or can store information and instructions Other types of dynamic storage devices can also be Electrically Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory, CD-ROM or other optical disk storage , CD storage (can include compressed CDs, laser disks, CDs, digital versatile CDs, Blu-ray CDs, etc.), disk storage media or other magnetic storage devices, or can be used to carry or store desired program codes in the form of instructions or data structures And any other media that can be accessed by the computer, but not limited to this. The memory can exist independently and is connected to the processor through a bus. The memory can also be integrated with the processor.

处理部件502，也可以称为处理器，处理单元，处理单板，处理模块、处理装置等。处理部件可以是中央处理器(central processing unit，CPU)，网络处理器(network processor，NP)或者CPU和NP的组合，也可以是微处理器，特定应用集成电路(application-specific integrated circuit，ASIC)，或一个或多个用于控制以上方案程序执行的集成电路。Theprocessing component 502 may also be referred to as a processor, a processing unit, a processing board, a processing module, a processing device, and so on. The processing component can be a central processing unit (CPU), a network processor (NP), or a combination of CPU and NP, or a microprocessor, application-specific integrated circuit (ASIC) ), or one or more integrated circuits used to control the execution of the program above.

通信部件503，也可以称为收发机，或收发器等，可以是用于与其他设备或通信网络通信，其中可以包括用来进行无线、有线或其他通信方式的单元。Thecommunication component 503, which may also be referred to as a transceiver, or a transceiver, may be used to communicate with other devices or a communication network, and may include a unit used for wireless, wired, or other communication methods.

当图像处理设备50为图1或图2所述服务器时，所述处理部件502用于调用所述存储部件501的数据执行如下操作：确定第一图像集合，所述第一图像集合包括多张第一图像，所述多张第一图像为同一场景下相同时间段内针对目标车辆的拍摄图像；从所述第一图像集合中确定一个或多个第二图像集合，所述一个或多个第二图像集合中每一个第二图像集合包括多张第二图像，所述每一个第二图像集合记录了所述目标车辆的一次待定违规行驶过程；标注所述一个或多个第二图像集合中至少一个第二图像集合中的交通标识的参考位置，并分别识别所述一个或多个第二图像集合中所述目标车辆的位置变化，以判断所述一个或多个第二图像集合分别对应的待定违规行驶是否违规。When theimage processing device 50 is the server described in FIG. 1 or FIG. 2, theprocessing component 502 is configured to call the data of thestorage component 501 to perform the following operations: determine a first image set, and the first image set includes multiple images The first image, the multiple first images are images taken for the target vehicle in the same time period in the same scene; one or more second image sets are determined from the first image set, the one or more Each second image set in the second image set includes a plurality of second images, and each second image set records a pending driving process of the target vehicle; annotates the one or more second image sets At least one of the reference positions of the traffic signs in the second image set, and respectively identify the position change of the target vehicle in the one or more second image sets, to determine that the one or more second image sets are respectively Whether the corresponding pending illegal driving is illegal.

本申请实施例还提供一种计算机存储介质，其中，该计算机存储介质可存储有程序，该程序执行时包括上述方法实施例中记载的任意一种的部分或全部步骤。所述计算机可读存储介质可以是非易失性，也可以是易失性的。An embodiment of the present application further provides a computer storage medium, wherein the computer storage medium may store a program, and the program includes part or all of the steps of any one of the above method embodiments when executed. The computer-readable storage medium may be non-volatile or volatile.

本申请实施例还提供一种计算机程序，该计算机程序包括指令，当该计算机程序被计算机执行时，使得计算机可以执行任意一种图像处理方法的部分或全部步骤。在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。The embodiments of the present application also provide a computer program, the computer program including instructions, when the computer program is executed by a computer, the computer can execute part or all of the steps of any image processing method. In the above-mentioned embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in an embodiment, reference may be made to related descriptions of other embodiments.

需要说明的是，对于前述的各方法实施例，为了简单描述，故将其都表述为一系列的动作组合，但是本领域技术人员应该知悉，本申请并不受所描述的动作顺序的限制，因为依据本申请，某些步骤可能可以采用其他顺序或者同时进行。其次，本领域技术人员也应该知悉，说明书中所描述的实施例均属于优选实施例，所涉及的动作和模块并不一定是本申请所必须的。It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the described sequence of actions. Because according to this application, some steps may be performed in other order or at the same time. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

在本申请所提供的几个实施例中，应该理解到，所揭露的装置，可通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如上述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性或其它的形式。In the several embodiments provided in this application, it should be understood that the disclosed device may be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above-mentioned units is only a logical function division, and there may be other divisions in actual implementation, for example, multiple units or components can be combined or integrated. To another system, or some features can be ignored, or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some interfaces, devices or units, and may be in electrical or other forms.

在本申请实施例中，所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本申请实施例方案的目的。In the embodiments of the present application, the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed. To multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments of the present application.

另外，在本申请实施例各个实施例中的各功能组件可以集成在一个组件也可以是各个组件单独物理存在，也可以是两个或两个以上组件集成在一个组件中。上述集成的组件既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。In addition, the functional components in the various embodiments of the embodiments of the present application may be integrated into one component, or each component may exist alone physically, or two or more components may be integrated into one component. The above-mentioned integrated components can be implemented in the form of hardware or software functional units.

所述集成的组件如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分，或者该技术方案的全部或部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本申请实施例各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器(ROM，Read-Only Memory)、随机存取存储器(RAM，Random Access Memory)、磁碟或者光盘等各种可以存储程序代码的介质。If the integrated component is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application are essentially or the part that contributes to the existing technology, or all or part of the technical solutions can be embodied in the form of software products, and the computer software products are stored in a The storage medium includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical disks and other media that can store program codes. .

以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。The above are only specific implementations of this application, but the protection scope of this application is not limited to this. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in this application. Should be covered within the scope of protection of this application. Therefore, the protection scope of this application should be subject to the protection scope of the claims.