技术领域Technical Field
本申请属于互联网技术领域，尤其涉及一种图像处理方法、装置、电子设备及存储介质。The present application belongs to the field of Internet technology, and in particular, relates to an image processing method, device, electronic device and storage medium.
背景技术Background Art
随着自动驾驶算法架构的发展，鸟瞰图(Bird'sEyeView，BEV)算法和Transformer结合的自动驾驶推理模型已开始在自动驾驶领域得到广泛使用，该架构的自动驾驶推理模型可有效提升感知精确度，利于后续规划自动驾驶算法的实施。With the development of autonomous driving algorithm architecture, the autonomous driving reasoning model combining the Bird's Eye View (BEV) algorithm and Transformer has begun to be widely used in the field of autonomous driving. The autonomous driving reasoning model of this architecture can effectively improve the perception accuracy and facilitate the subsequent planning and implementation of the autonomous driving algorithm.
目前，随着自动驾驶技术的普及，用户对自动驾驶的实时地图构建能力有更高的要求。在实现本申请过程中，发明人发现现有技术中至少存在如下问题：静态对象(如车道线和路沿)的3d真值获取比较难。At present, with the popularization of autonomous driving technology, users have higher requirements for the real-time map building capabilities of autonomous driving. In the process of implementing this application, the inventor found that there are at least the following problems in the prior art: it is difficult to obtain the 3D true value of static objects (such as lane lines and curbs).
发明内容Summary of the invention
本申请实施例提供一种图像处理方法、装置、设备及存储介质，能够解决路面的静态对象真值构建的效率和成本问题。The embodiments of the present application provide an image processing method, apparatus, device and storage medium, which can solve the efficiency and cost issues of constructing the true value of static objects on the road surface.
第一方面，本申请实施例提供一种图像处理方法，该方法包括：In a first aspect, an embodiment of the present application provides an image processing method, the method comprising:
获取移动装置在移动过程中的样本数据，所述样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；Acquire sample data of the mobile device during movement, the sample data including: first point cloud data collected by the sensor, positioning data of the sensor, and image data collected by the camera;
根据所述定位数据对所述第一点云数据进行处理，得到路面点云数据；Processing the first point cloud data according to the positioning data to obtain road surface point cloud data;
根据所述定位数据对所述路面点云数据进行重建处理，得到反射率图像；Reconstructing the road surface point cloud data according to the positioning data to obtain a reflectivity image;
根据所述路面点云数据和所述图像数据，生成纹理图像；Generate a texture image according to the road surface point cloud data and the image data;
根据所述定位数据对所述反射率图像和所述纹理图像中的静态对象进行标注，得到训练样本数据。Static objects in the reflectivity image and the texture image are annotated according to the positioning data to obtain training sample data.
第二方面，本申请实施例提供一种图像处理装置，图像处理装置包括：In a second aspect, an embodiment of the present application provides an image processing device, the image processing device comprising:
获取模块，用于获取移动装置在移动过程中的样本数据，所述样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；An acquisition module, used to acquire sample data of the mobile device during movement, wherein the sample data includes: first point cloud data acquired by the sensor, positioning data of the sensor, and image data acquired by the camera;
处理模块，用于根据所述定位数据对所述第一点云数据进行处理，得到路面点云数据；A processing module, used for processing the first point cloud data according to the positioning data to obtain road surface point cloud data;
所述处理模块，还用于根据所述定位数据对所述路面点云数据进行重建处理，得到反射率图像；The processing module is further used to reconstruct the road surface point cloud data according to the positioning data to obtain a reflectivity image;
生成模块，用于根据所述路面点云数据和所述图像数据，生成纹理图像；A generating module, used for generating a texture image according to the road surface point cloud data and the image data;
标注模块，用于根据所述定位数据对所述反射率图像和所述纹理图像中的静态对象进行标注，得到训练样本数据。The labeling module is used to label the static objects in the reflectivity image and the texture image according to the positioning data to obtain training sample data.
第三方面，本申请实施例提供了一种电子设备，该设备包括：处理器以及存储有计算机程序指令的存储器；处理器执行计算机程序指令时，实现如第一方面或者第一方面的任一可能实现方式中的方法。In a third aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the method in the first aspect or any possible implementation of the first aspect.
第四方面，本申请实施例提供了一种可读存储介质，该计算机可读存储介质上存储有计算机程序指令，计算机程序指令被处理器执行时实现如第一方面或者第一方面的任一可能实现方式中的方法。In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the method in the first aspect or any possible implementation of the first aspect is implemented.
本申请实施例中，通过获取移动装置在移动过程中的样本数据，样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；由于采集得到的第一点云数据包括实际场景中的多种对象的点云数据，因此需要对根据定位数据对第一点云数据进行处理，得到路面点云数据。然后，根据定位数据对路面点云数据进行重建处理，得到反射率图像。接着，为了提升训练样本数据在各个场景下的适用性，根据路面点云数据和图像数据生成纹理图像，纹理图像在各种极端天气场景中能够清晰地体现场景特征，提升场景适用性，最后，对反射率图像和纹理图像中的静态对象进行标注，得到训练样本数据。由此，训练样本数据能够用于精进推理模型的推理准确度。In an embodiment of the present application, by acquiring sample data of a mobile device during movement, the sample data includes: first point cloud data collected by a sensor, positioning data of the sensor, and image data collected by a camera; since the first point cloud data collected includes point cloud data of multiple objects in the actual scene, it is necessary to process the first point cloud data according to the positioning data to obtain road surface point cloud data. Then, the road surface point cloud data is reconstructed according to the positioning data to obtain a reflectivity image. Next, in order to improve the applicability of the training sample data in various scenarios, a texture image is generated based on the road surface point cloud data and image data. The texture image can clearly reflect the scene features in various extreme weather scenarios and improve the applicability of the scene. Finally, the static objects in the reflectivity image and the texture image are annotated to obtain training sample data. Therefore, the training sample data can be used to improve the reasoning accuracy of the reasoning model.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更清楚地说明本申请实施例的技术方案，下面将对本申请实施例中所需要使用的附图作简单的介绍，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
图1是本申请实施例提供的一种图像处理方法的流程图；FIG1 is a flow chart of an image processing method provided by an embodiment of the present application;
图2是本申请实施例提供的一种图像处理方法的示意图；FIG2 is a schematic diagram of an image processing method provided in an embodiment of the present application;
图3是本申请实施例提供的另一种图像处理方法的流程图；FIG3 is a flow chart of another image processing method provided by an embodiment of the present application;
图4是本申请实施例提供的一种图像处理装置的结构示意图；FIG4 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application;
图5是本申请实施例提供的一种电子设备的硬件结构示意图。FIG5 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.
具体实施方式DETAILED DESCRIPTION
下面将详细描述本申请的各个方面的特征和示例性实施例，为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及具体实施例，对本申请进行进一步详细描述。应理解，此处所描述的具体实施例仅被配置为解释本申请，并不被配置为限定本申请。对于本领域技术人员来说，本申请可以在不需要这些具体细节中的一些细节的情况下实施。下面对实施例的描述仅仅是为了通过示出本申请的示例来提供对本申请更好的理解。The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present application and are not configured to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括……”限定的要素，并不排除在包括要素的过程、方法、物品或者设备中还存在另外的相同要素。It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
下面对本申请涉及到的技术术语进行简要介绍。The following is a brief introduction to the technical terms involved in this application.
鸟瞰图(Bird'sEyeView，BEV)，该算法旨在将多传感器收集的图像信息投射至统一3D空间，再输入至单一大模型进行整体推理。相较于传统的摄像头图像，BEV提供了一个更贴近实际物理世界的统一空间，为后续的多传感器融合以及规划控制模块开发提供了更大的便利和更多的可能。Bird's Eye View (BEV), this algorithm aims to project the image information collected by multiple sensors into a unified 3D space, and then input it into a single large model for overall reasoning. Compared with traditional camera images, BEV provides a unified space that is closer to the actual physical world, providing greater convenience and more possibilities for the subsequent multi-sensor fusion and planning control module development.
具体来说，BEV感知的优势在于：统一了多模态数据处理维度，将多个摄像头或雷达数据转换至3D视角，再做目标检测与分割等任务，从而降低感知误差，并为下游预测和规划控制模块提供更丰富的输出；实现时序信息融合，BEV下的3D视角相较于2D信息可有效减少尺度和遮挡问题，有效提高自动驾驶安全性；感知和预测可在统一3D空间中实施，通过神经网络直接完成端到端优化，可有效降低传统感知任务中感知与预测串行的误差累积。Specifically, the advantages of BEV perception are: unifying the multimodal data processing dimensions, converting multiple camera or radar data into 3D perspectives, and then performing tasks such as target detection and segmentation, thereby reducing perception errors and providing richer outputs for downstream prediction and planning control modules; realizing time series information fusion, the 3D perspective under BEV can effectively reduce scale and occlusion problems compared to 2D information, and effectively improve the safety of autonomous driving; perception and prediction can be implemented in a unified 3D space, and end-to-end optimization can be directly completed through neural networks, which can effectively reduce the error accumulation of perception and prediction serially in traditional perception tasks.
Transformer的注意力机制可帮助实现2D图像数据至3D的BEV空间的转化。随着技术的发展开始进军图像视觉领域，目前已成功涉足分类、检测和分割三大图像问题。据汽车之心微信公众号介绍，传统CNN模型的原理是通过卷积层构造广义过滤器，从而对图像中的元素进行不断地筛选压缩。The attention mechanism of Transformer can help realize the transformation of 2D image data into 3D BEV space. With the development of technology, it has begun to enter the field of image vision, and has successfully involved in three major image problems: classification, detection and segmentation. According to the WeChat public account of Auto Heart, the principle of the traditional CNN model is to construct a generalized filter through the convolution layer, so as to continuously screen and compress the elements in the image.
Transformer的网络结构在嫁接2D图像和3D空间时借鉴了人脑的注意力(Attention)机制，在处理大量信息时能够只选择处理关键信息，以提升神经网络的效率，因此Transformer的饱和区间很大，更适宜于大规模数据训练的需求。在自动驾驶领域，Transformer相比于传统CNN，具备更强的序列建模能力和全局信息感知能力，目前已广泛用于视觉2D图像数据至3D空间的转化。The Transformer network structure draws on the attention mechanism of the human brain when grafting 2D images and 3D space. When processing a large amount of information, it can only select key information to process to improve the efficiency of the neural network. Therefore, the saturation range of the Transformer is large, which is more suitable for the needs of large-scale data training. In the field of autonomous driving, compared with traditional CNN, the Transformer has stronger sequence modeling capabilities and global information perception capabilities, and is currently widely used in the conversion of visual 2D image data to 3D space.
惯性测量单元(Inertial Measurement Unit，IMU)，主要用于测量自身位姿，位姿包含位置和姿态。惯性导航系统由陀螺仪、加速度计等惯性传感器和导航解算系统集成而成。陀螺仪和加速度计是系统的核心器件，陀螺仪测量物体的角速度，加速度计测量物体的加速度。典型的惯导产品包含3组陀螺仪和加速度计，分别测量三个自由度的角速度和加速度，通过积分即可获得物体在三维空间的运动速度和轨迹。Inertial Measurement Unit (IMU) is mainly used to measure its own posture, which includes position and attitude. The inertial navigation system is integrated with inertial sensors such as gyroscopes and accelerometers and navigation solution systems. Gyroscopes and accelerometers are the core components of the system. Gyroscopes measure the angular velocity of objects, and accelerometers measure the acceleration of objects. Typical inertial navigation products contain 3 sets of gyroscopes and accelerometers, which measure the angular velocity and acceleration of three degrees of freedom respectively. The speed and trajectory of the object in three-dimensional space can be obtained by integration.
实时动态(Real Time Kinematic，RTK)载波相位差分技术，是实时处理两个测量站载波相位观测量的差分方法，将基准站采集的载波相位发给用户接收机，进行求差解算坐标。这是一种新的常用的卫星定位测量方法，以前的静态、快速静态、动态测量都需要事后进行解算才能获得厘米级的精度，而RTK是能够在野外实时得到厘米级定位精度的测量方法，它采用了载波相位动态实时差分方法，是GPS应用的重大里程碑，它的出现为工程放样、地形测图，各种控制测量带来了新的测量原理和方法，极大地提高了作业效率。Real Time Kinematic (RTK) carrier phase differential technology is a differential method for real-time processing of carrier phase observations of two measuring stations. The carrier phase collected by the base station is sent to the user receiver to calculate the coordinates. This is a new and commonly used satellite positioning measurement method. The previous static, fast static, and dynamic measurements all required post-calculation to obtain centimeter-level accuracy. RTK is a measurement method that can obtain centimeter-level positioning accuracy in real time in the field. It uses a carrier phase dynamic real-time differential method and is a major milestone in GPS applications. Its appearance has brought new measurement principles and methods to engineering layout, topographic mapping, and various control measurements, greatly improving work efficiency.
图1是本申请实施例提供的一种图像处理方法的流程图。FIG. 1 is a flow chart of an image processing method provided in an embodiment of the present application.
如图1所示，该图像处理方法可以包括步骤110-步骤150，该方法应用于图像处理装置，具体如下所示：As shown in FIG. 1 , the image processing method may include steps 110 to 150. The method is applied to an image processing device, and is specifically as follows:
步骤110，获取移动装置在移动过程中的样本数据，样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；Step 110, acquiring sample data of the mobile device during movement, the sample data including: first point cloud data collected by the sensor, positioning data of the sensor, and image data collected by the camera;
移动装置可以为车辆、飞行器或机器人等，可以移动的装置。The mobile device may be a vehicle, an aircraft, a robot, or the like, which is a mobile device.
第一点云数据可以为：激光雷达点云数据。The first point cloud data may be: laser radar point cloud data.
激光雷达点云数据，是由三维激光雷达设备扫描得到的空间点的数据集，每一个点都包含了三维坐标信息，也是X、Y、Z三个元素，有的还包含颜色信息、反射值、回波次数信息等。激光雷达点云数据，由激光扫描系统向周围发射激光信号，然后收集反射的激光信号得来的，再通过外业数据采集、组合导航、点云解算，便可以计算出这些点的准确空间信息。LiDAR point cloud data is a data set of spatial points scanned by a 3D LiDAR device. Each point contains 3D coordinate information, which is also the three elements of X, Y, and Z. Some also contain color information, reflection value, echo number information, etc. LiDAR point cloud data is obtained by the laser scanning system emitting laser signals to the surroundings and then collecting the reflected laser signals. Through field data collection, combined navigation, and point cloud solution, the accurate spatial information of these points can be calculated.
定位数据可以包括：IMU数据、RTK数据，全球定位系统(Global PositioningSystem，GPS)数据，和全球卫星导航系统(Global Navigation Satellite System，GNSS)信息。Positioning data may include: IMU data, RTK data, Global Positioning System (GPS) data, and Global Navigation Satellite System (GNSS) information.
定位数据可以由移动装置中设置的传感器采集得到。The positioning data can be collected by sensors provided in the mobile device.
图像数据可以由具有环视功能的摄像头采集得到，例如，摄像头由四个鱼眼相机组成，每一个鱼眼相机的感受视场角都超过180度；通过将四个视场拼接从而形成了一个360度的感知视野。Image data can be collected by a camera with a surround view function. For example, the camera consists of four fisheye cameras, each of which has a field of view of more than 180 degrees; a 360-degree perception field of view is formed by stitching the four fields of view together.
步骤120，根据定位数据对第一点云数据进行处理，得到路面点云数据；Step 120, processing the first point cloud data according to the positioning data to obtain road surface point cloud data;
在自动驾驶中，激光雷达等传感器收集到的环境信息通常以点云的形式表示。点云是由大量离散的三维坐标点组成的数据集，每个点代表空间中的一个位置。In autonomous driving, environmental information collected by sensors such as lidar is usually represented in the form of point clouds. Point clouds are data sets consisting of a large number of discrete three-dimensional coordinate points, each of which represents a position in space.
由于采集得到的第一点云数据包括实际场景中的多种对象的点云数据，例如，车辆、行人和指示牌等。因此需要对根据定位数据对第一点云数据进行处理，得到路面点云数据。如图2所示，根据定位数据对第一点云数据进行处理，得到路面点云数据，路面点云数据用于后续生成反射率图像和纹理图像。Since the first point cloud data collected includes point cloud data of various objects in the actual scene, such as vehicles, pedestrians, and signboards, etc., it is necessary to process the first point cloud data according to the positioning data to obtain road surface point cloud data. As shown in FIG2 , the first point cloud data is processed according to the positioning data to obtain road surface point cloud data, and the road surface point cloud data is used to subsequently generate a reflectivity image and a texture image.
步骤130，根据定位数据对路面点云数据进行重建处理，得到反射率图像；Step 130, reconstructing the road surface point cloud data according to the positioning data to obtain a reflectivity image;
其中，由于原始采集的第一点云数据中包括反射率，使用定位数据对路面点云数据关联的反射率进行路面元素的重建，得到反射率图像。步骤140，根据路面点云数据和图像数据，生成纹理图像；Wherein, since the originally collected first point cloud data includes reflectivity, the reflectivity associated with the road surface point cloud data is used to reconstruct the road surface elements using the positioning data to obtain a reflectivity image. Step 140, generating a texture image based on the road surface point cloud data and the image data;
对于雨天、雾天和扬尘天气情况，根据路面点云数据和图像数据，生成的纹理图像，能够清晰地体现路面元素。For rainy, foggy and dusty weather conditions, the texture images generated based on the road point cloud data and image data can clearly reflect the road elements.
步骤150，根据定位数据对反射率图像和纹理图像中的静态对象进行标注，得到训练样本数据。Step 150 , annotating static objects in the reflectivity image and the texture image according to the positioning data to obtain training sample data.
其中，根据所述定位数据对反射率图像和纹理图像中的静态对象进行标注，具体可以标注下述中的至少一项：车道线、地面标志和人行横道。The static objects in the reflectivity image and the texture image are annotated according to the positioning data, and specifically at least one of the following may be annotated: lane lines, ground signs, and crosswalks.
根据所述定位数据对反射率图像和纹理图像中的静态对象进行标注，能够提升路面重建后的特征，提升训练样本数据的有效性。Labeling static objects in the reflectivity image and the texture image according to the positioning data can improve the features of the reconstructed road surface and improve the effectiveness of the training sample data.
本申请实施例中，通过获取移动装置在移动过程中的样本数据，样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；由于采集得到的第一点云数据包括实际场景中的多种对象的点云数据，因此需要对根据定位数据对第一点云数据进行处理，得到路面点云数据。然后，根据定位数据对路面点云数据进行重建处理，得到反射率图像。接着，为了提升训练样本数据在各个场景下的适用性，根据路面点云数据和图像数据生成纹理图像，纹理图像在各种极端天气场景中能够清晰地体现场景特征，提升场景适用性，最后，对反射率图像和纹理图像中的静态对象进行标注，得到训练样本数据。由此，训练样本数据能够用于精进推理模型的推理准确度。In an embodiment of the present application, by acquiring sample data of a mobile device during movement, the sample data includes: first point cloud data collected by a sensor, positioning data of the sensor, and image data collected by a camera; since the first point cloud data collected includes point cloud data of multiple objects in the actual scene, it is necessary to process the first point cloud data according to the positioning data to obtain road surface point cloud data. Then, the road surface point cloud data is reconstructed according to the positioning data to obtain a reflectivity image. Next, in order to improve the applicability of the training sample data in various scenarios, a texture image is generated based on the road surface point cloud data and image data. The texture image can clearly reflect the scene features in various extreme weather scenarios and improve the applicability of the scene. Finally, the static objects in the reflectivity image and the texture image are annotated to obtain training sample data. Therefore, the training sample data can be used to improve the reasoning accuracy of the reasoning model.
下面，对步骤110-步骤150的内容分别进行描述：The contents of step 110 to step 150 are described below respectively:
涉及步骤110。This involves step 110 .
获取移动装置在移动过程中的样本数据，样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；Acquire sample data of the mobile device during movement, the sample data including: first point cloud data collected by the sensor, positioning data of the sensor, and image data collected by the camera;
其中，移动装置设置有传感器和摄像头。The mobile device is provided with a sensor and a camera.
传感器包括激光雷达、毫米波雷达、深度相机或3D扫描仪等，它们可以从现实世界中获取物体和环境的几何、形状和比例信息等现实环境信息。Sensors include lidar, millimeter-wave radar, depth cameras or 3D scanners, which can obtain real-world environmental information such as the geometry, shape and scale information of objects and environments from the real world.
定位数据可以包括下述中的至少一项：IMU数据、RTK数据、GPS数据和GNSS数据。The positioning data may include at least one of the following: IMU data, RTK data, GPS data, and GNSS data.
图像数据可以由具有环视功能的摄像头采集得到。The image data can be collected by a camera with a surround view function.
一种可能的实施例中，步骤110，具体可以包括以下步骤：In a possible embodiment, step 110 may specifically include the following steps:
确定移动装置所在的场景；Determine the scene where the mobile device is located;
在场景为预设场景的情况下，获取移动装置在多次移动过程中采集的多个样本数据。When the scene is a preset scene, a plurality of sample data collected by the mobile device during multiple movement processes are obtained.
可以通过三维点云分割技术确定移动装置所在的场景，具体地，通过三维点云分割技术将道路环境中的点云分为不同的类别，如道路表面、建筑物、树木等。根据点云对应的类别，确定移动装置所处的场景。The scene where the mobile device is located can be determined by the 3D point cloud segmentation technology. Specifically, the point cloud in the road environment is divided into different categories, such as road surface, buildings, trees, etc. The scene where the mobile device is located is determined according to the category corresponding to the point cloud.
移动装置所在的场景大致可以分为下述两种：The scenarios in which mobile devices are located can be roughly divided into the following two types:
简单场景：高速和城区快速路。Simple scenario: highways and urban expressways.
复杂场景：城区的路口等。Complex scenes: intersections in urban areas, etc.
其中，本申请实施例中涉及到的预设场景为复杂场景。Among them, the preset scene involved in the embodiments of the present application is a complex scene.
对于不同的移动装置所在的场景，可以采取不同的数据采集方式和数据处理方式。Different data collection methods and data processing methods may be adopted for different scenarios where mobile devices are located.
一方面，对于简单场景，可以采集一次样本数据，根据一次采集到的样本数据重建完整的路面点云数据。On the one hand, for simple scenes, sample data can be collected once, and the complete road surface point cloud data can be reconstructed based on the sample data collected once.
另一方面，对于复杂场景，需要移动装置执行多次移动过程，以获取移动装置在多次移动过程中采集的多个样本数据，然后通过聚合不同时间段采集的数据来获得完整的路面点云数据。On the other hand, for complex scenes, the mobile device needs to perform multiple movement processes to obtain multiple sample data collected by the mobile device during the multiple movement processes, and then obtain complete road surface point cloud data by aggregating data collected in different time periods.
由此，在场景为预设场景的情况下，能够灵活地根据移动装置所处的场景，获取移动装置在多次移动过程中采集的多个样本数据，由此，能够更加完善地获取样本数据。Therefore, when the scene is a preset scene, multiple sample data collected by the mobile device during multiple movements can be flexibly obtained according to the scene in which the mobile device is located, thereby enabling more complete sample data to be obtained.
涉及步骤120。This involves step 120 .
根据定位数据对第一点云数据进行处理，得到路面点云数据；Processing the first point cloud data according to the positioning data to obtain road surface point cloud data;
具体可以对第一点云数据进行点云分割和障碍物检测，从第一点云数据中提取路面对应的点云数据，并进行拼接得到路面点云数据；Specifically, point cloud segmentation and obstacle detection may be performed on the first point cloud data, point cloud data corresponding to the road surface may be extracted from the first point cloud data, and the road surface point cloud data may be obtained by splicing;
其中，点云分割旨在将点云中的点按照其所属的物体或区域进行分类和分割。通过将点云中的点分配给不同的类别，可以实现对环境中不同物体的识别和理解。Point cloud segmentation aims to classify and segment the points in the point cloud according to the objects or regions they belong to. By assigning points in the point cloud to different categories, different objects in the environment can be recognized and understood.
障碍物检测是指检测和识别道路上的障碍物，如其他车辆、行人、自行车等。这为车辆规划路径、避免碰撞和决策提供了重要的信息。Obstacle detection refers to the detection and identification of obstacles on the road, such as other vehicles, pedestrians, bicycles, etc. This provides important information for vehicle path planning, collision avoidance, and decision making.
其中，步骤120中，具体可以包括以下步骤：Wherein, step 120 may specifically include the following steps:
一种可能的实施例中，步骤120，具体可以包括以下步骤：In a possible embodiment, step 120 may specifically include the following steps:
从第一点云数据中去除动态对象对应的点云数据，得到第二点云数据；removing the point cloud data corresponding to the dynamic object from the first point cloud data to obtain second point cloud data;
根据定位数据对第二点云数据进行处理，得到路面点云数据。The second point cloud data is processed according to the positioning data to obtain road surface point cloud data.
其中，上述涉及到的从第一点云数据中去除动态对象对应的点云数据，得到第二点云数据的步骤之前，还可以包括下述步骤：Before the above-mentioned step of removing the point cloud data corresponding to the dynamic object from the first point cloud data to obtain the second point cloud data, the following steps may also be included:
对第一点云数据进行单帧点云目标检测，识别动态对象；其中，动态对象包括下述中的至少一项：车辆、行人和单车。由此，以便于可以从第一点云数据中去除动态对象对应的点云数据，得到第二点云数据。Single-frame point cloud target detection is performed on the first point cloud data to identify dynamic objects, wherein the dynamic objects include at least one of the following: vehicles, pedestrians, and bicycles. Thus, the point cloud data corresponding to the dynamic objects can be removed from the first point cloud data to obtain the second point cloud data.
其中，根据定位数据对第二点云数据进行处理，得到路面点云数据的步骤中，具体可以包括以下步骤：The step of processing the second point cloud data according to the positioning data to obtain the road surface point cloud data may specifically include the following steps:
对第二点云数据进行地面分割，提取第三点云数据；Perform ground segmentation on the second point cloud data to extract third point cloud data;
根据定位数据对第三点云数据进行拼接，得到第四点云数据；splicing the third point cloud data according to the positioning data to obtain fourth point cloud data;
对第四点云数据进行点云降采样处理，得到路面点云数据。The fourth point cloud data is subjected to point cloud downsampling processing to obtain road surface point cloud data.
由于地面点云数据是由大量点构成的数据集，点云数据通常非常庞大，导致计算和存储的负担很重。点云降采样技术通过减少点的数量，从而降低了数据的复杂性，提高了数据处理效率。Since ground point cloud data is a data set composed of a large number of points, point cloud data is usually very large, resulting in a heavy burden on calculation and storage. Point cloud downsampling technology reduces the complexity of data and improves data processing efficiency by reducing the number of points.
由此，通过对原始采集的第一点云数据进行去除动态对象处理，得到不包括动态对象的第二点云数据，以及根据定位数据对不包括动态对象的第二点云数据进行拼接处理，能够快速准确地基于第一点云数据生成路面点云数据，便于后续对路面点云数据进行重建处理。Therefore, by removing dynamic objects from the originally collected first point cloud data to obtain second point cloud data that does not include dynamic objects, and splicing the second point cloud data that does not include dynamic objects according to the positioning data, it is possible to quickly and accurately generate road surface point cloud data based on the first point cloud data, thereby facilitating subsequent reconstruction of the road surface point cloud data.
涉及步骤130。This involves step 130 .
根据定位数据对路面点云数据进行重建处理，得到反射率图像；Reconstruct the road surface point cloud data according to the positioning data to obtain a reflectivity image;
第一点云数据中包括反射率，使用定位数据对路面点云数据关联的反射率进行路面元素的重建，得到反射率图像。The first point cloud data includes reflectivity, and the reflectivity associated with the road surface point cloud data is used to reconstruct road surface elements to obtain a reflectivity image.
一种可能的实施例中，第一点云数据包括反射率，步骤130，具体可以包括以下步骤：In a possible embodiment, the first point cloud data includes reflectivity, and step 130 may specifically include the following steps:
从第一点云数据中确定路面点云数据关联的反射率；Determining reflectivity associated with the road surface point cloud data from the first point cloud data;
根据路面点云数据关联的反射率对路面点云数据进行重建处理，得到反射率图像。The road surface point cloud data is reconstructed according to the reflectivity associated with the road surface point cloud data to obtain a reflectivity image.
3D传感器通常以3D点云的形式保存每个点的多维度信息，包括三维坐标、反射率、尺寸等。其中，第一点云数据包括反射率。The 3D sensor usually stores the multi-dimensional information of each point in the form of a 3D point cloud, including three-dimensional coordinates, reflectivity, size, etc. Among them, the first point cloud data includes reflectivity.
第一点云数据的数据结构为：[x，y，z，Intensity]。“Intensity”为反射率，反射率的数值范围为：[0，255]，通过规则的方法将反射率值和图像的RGB值进行一一对应。The data structure of the first point cloud data is: [x, y, z, Intensity]. “Intensity” is the reflectivity, and the value range of the reflectivity is: [0, 255]. The reflectivity value and the RGB value of the image are matched one by one through a rule method.
对路面点云数据进行重建处理，首先可以根据点云数据的某个特征进行上色。由于不同的对象在反射率和高度等信息上会有区别，可以基于反射率或高度，将不同的点渲染为不同的颜色，以进行区分。To reconstruct the road point cloud data, you can first color it according to a certain feature of the point cloud data. Since different objects have different information such as reflectivity and height, different points can be rendered in different colors based on reflectivity or height to distinguish them.
具体包括：Specifically include:
(1)车道线的反射率大多集中在0.1-50这个区间，(2)将小于或者大于该区间的点云反射率设置为0.1或者50。(3)计算重建后地面点云的均值，认为该均值为出去车道线等反射率高的部分的地面点云的反射率阈值，通过计算，大约在2-3之间。(4)将小于地面点云阈值的点云，按照反射率的变化赋值到RGB值为[0,0,0]-[50,50,50]的区间，(5)大于地面阈值的点云，按照反射率增长变化赋值到[51,51,51]-[255,255,255]的区间，以加大车道线和地面点云的区分度。(1) The reflectivity of lane lines is mostly concentrated in the range of 0.1-50. (2) The reflectivity of point clouds that are less than or greater than this range is set to 0.1 or 50. (3) The mean of the reconstructed ground point cloud is calculated, and the mean is considered to be the reflectivity threshold of the ground point cloud excluding the lane lines and other parts with high reflectivity. After calculation, it is approximately between 2 and 3. (4) Point clouds that are less than the ground point cloud threshold are assigned to the RGB value range of [0,0,0]-[50,50,50] according to the change of reflectivity. (5) Point clouds that are greater than the ground threshold are assigned to the range of [51,51,51]-[255,255,255] according to the increase of reflectivity to increase the distinction between lane lines and ground point clouds.
由于车道线和交通标志的反射率通常和周边环境有很大的区别，因此可以车道线或者交通标志的标注任务可以通过反射率映射来实现。Since the reflectivity of lane lines and traffic signs is usually very different from the surrounding environment, the labeling task of lane lines or traffic signs can be achieved through reflectivity mapping.
由此，根据路面点云数据关联的反射率对路面点云数据进行重建处理，得到的反射率图像，能够体现车道线和路沿等静态对象和周边环境的区别。Therefore, the road surface point cloud data is reconstructed according to the reflectivity associated with the road surface point cloud data, and the obtained reflectivity image can reflect the difference between static objects such as lane lines and curbs and the surrounding environment.
一种可能的实施例中，在场景为预设场景的情况下，步骤130，具体可以包括以下步骤：In a possible embodiment, when the scene is a preset scene, step 130 may specifically include the following steps:
确定任意两次移动过程对应的路面点云数据之间的转换矩阵；Determine the transformation matrix between the road surface point cloud data corresponding to any two movement processes;
根据转换矩阵对多次移动过程对应的路面点云数据进行拼接，得到路面点云数据；The road surface point cloud data corresponding to the multiple movement processes are spliced according to the transformation matrix to obtain the road surface point cloud data;
根据路面点云数据关联的反射率对路面点云数据进行重建处理，得到反射率图像。The road surface point cloud data is reconstructed according to the reflectivity associated with the road surface point cloud data to obtain a reflectivity image.
由于在场景为预设场景的情况下，移动装置存在多次移动过程，上述步骤110相应地获取了移动装置在多次移动过程中采集的多个样本数据。Since the mobile device has multiple movement processes when the scene is a preset scene, the above step 110 correspondingly obtains multiple sample data collected by the mobile device during the multiple movement processes.
多个样本数据分别经过步骤120的处理，得到多个路面点云数据。The plurality of sample data are processed in step 120 respectively to obtain a plurality of road surface point cloud data.
为了对多次移动过程中采集的多个路面点云数据进行重建处理，需要对路面点云数据进行校正。In order to reconstruct multiple road surface point cloud data collected during multiple movements, the road surface point cloud data needs to be corrected.
具体地，可以对任意两次移动过程对应的路面点云数据进行配准，确定任意两次移动过程对应的路面点云数据之间的转换矩阵。转换矩阵用于表征任意两次移动过程对应的路面点云数据之间的相对位置关系。Specifically, the road surface point cloud data corresponding to any two movement processes can be registered to determine the transformation matrix between the road surface point cloud data corresponding to any two movement processes. The transformation matrix is used to characterize the relative position relationship between the road surface point cloud data corresponding to any two movement processes.
然后，根据转换矩阵对多次移动过程对应的路面点云数据进行拼接，得到预设位置区域内的路面点云数据。Then, the road surface point cloud data corresponding to the multiple movement processes are spliced according to the transformation matrix to obtain the road surface point cloud data in the preset position area.
由于第一点云数据中包括反射率，可以从第一点云数据关联的反射率中确定出路面点云数据关联的反射率，根据路面点云数据关联的反射率对路面点云数据进行重建处理，得到反射率图像。Since the first point cloud data includes reflectivity, the reflectivity associated with the road surface point cloud data can be determined from the reflectivity associated with the first point cloud data, and the road surface point cloud data is reconstructed according to the reflectivity associated with the road surface point cloud data to obtain a reflectivity image.
由此，对于城区的路口等复杂场景，需要移动装置执行多次移动过程，以获取移动装置在多次移动过程中采集的多个样本数据，然后对多次移动过程对应的路面点云数据进行重建处理，得到反射率图像，由此，本申请实施例对于复杂场景，通过多次采集的路面点云数据能够生成更加完善的反射率图像。Therefore, for complex scenes such as intersections in urban areas, the mobile device is required to perform multiple movement processes to obtain multiple sample data collected by the mobile device during the multiple movement processes, and then reconstruct the road point cloud data corresponding to the multiple movement processes to obtain a reflectivity image. Therefore, for complex scenes, the embodiment of the present application can generate a more complete reflectivity image through multiple collected road point cloud data.
其中，上述涉及到的确定任意两次移动过程对应的路面点云数据之间的转换矩阵的步骤之前，还可以包括以下步骤：Before the step of determining the conversion matrix between the road surface point cloud data corresponding to any two movement processes, the following steps may also be included:
获取多次移动过程中采集得到的参照物位置信息；Obtaining reference object position information collected during multiple movements;
对多次采集得到的参照物位置信息进行聚类，得到聚合后的参照物位置信息；Clustering the reference object position information collected multiple times to obtain aggregated reference object position information;
根据聚合后的参照物位置信息，确定路口位置信息；Determine the intersection location information according to the aggregated reference object location information;
根据路口位置信息，从多次移动过程对应的路面点云数据中识别出每次采集得到的路面点云数据。According to the intersection location information, the road surface point cloud data collected each time is identified from the road surface point cloud data corresponding to the multiple movement processes.
其中，参照物可以包括下述中的至少一项：红绿灯、路标和广告牌等静态参照物。The reference objects may include at least one of the following: static reference objects such as traffic lights, road signs and billboards.
下面以参照物位置信息为红绿灯位置信息为例进行说明：The following description is made by taking the reference object position information as the traffic light position information as an example:
通过识别红绿灯的位置，使用全局的融合定位将红绿灯位置转到全局坐标系，以得到多次移动过程中采集得到的红绿灯位置信息。By identifying the position of the traffic light, the global fusion positioning is used to transfer the traffic light position to the global coordinate system to obtain the traffic light position information collected during multiple movements.
由于多次检测的红绿灯位置有很多的重合，而且路口有很多红绿灯，因此需要对多次采集得到的红绿灯位置信息进行聚类，得到聚合后的红绿灯位置信息。Since there are many overlaps in the positions of traffic lights detected multiple times and there are many traffic lights at intersections, it is necessary to cluster the traffic light position information collected multiple times to obtain aggregated traffic light position information.
根据聚合后的红绿灯位置信息，确定路口中心位置；Determine the center position of the intersection based on the aggregated traffic light position information;
根据路口中心位置确定路口位置信息。The intersection location information is determined according to the center position of the intersection.
其中，路口位置信息，可以为基于路口中心位置确定的位置范围。例如，以路口中心位置为中心点，选择300米×300米的位置范围。The intersection location information may be a location range determined based on the center location of the intersection, for example, taking the center location of the intersection as the center point and selecting a location range of 300 meters×300 meters.
根据路口位置信息，从多次移动过程对应的路面点云数据中识别出每次采集得到的路面点云数据，相当于从多次移动过程得到的路面点云数据进行分离，得到多个单次的路面点云数据。According to the intersection location information, the road surface point cloud data collected each time is identified from the road surface point cloud data corresponding to the multiple movement processes, which is equivalent to separating the road surface point cloud data obtained from the multiple movement processes to obtain multiple single road surface point cloud data.
其中，根据路口位置信息，从多次移动过程对应的路面点云数据中识别出每次采集得到的路面点云数据，具体可以包括：Among them, according to the intersection location information, identifying the road surface point cloud data collected each time from the road surface point cloud data corresponding to the multiple movement processes may specifically include:
根据移动装置的定位数据，选择出经过路口位置信息内的所有路径的路面点云数据，然后确定这些路面点云数据对应的路径的起始点时间，分别确定多个单次的路面点云数据。According to the positioning data of the mobile device, the road surface point cloud data of all paths passing through the intersection location information are selected, and then the starting point time of the path corresponding to these road surface point cloud data is determined, and multiple single road surface point cloud data are determined respectively.
由此，根据路口位置信息，能够从多次移动过程对应的路面点云数据中识别出每次采集得到的路面点云数据，便于确定任意两个单次的移动过程对应的路面点云数据之间的转换矩阵。Therefore, according to the intersection location information, the road surface point cloud data collected each time can be identified from the road surface point cloud data corresponding to multiple movement processes, so as to facilitate the determination of the conversion matrix between the road surface point cloud data corresponding to any two single movement processes.
涉及步骤140。This involves step 140 .
根据路面点云数据和图像数据，生成纹理图像；Generate texture images based on road surface point cloud data and image data;
对于极端的天气。例如，雨天、雾天和扬尘天气。纹理图像能够准确体现静态对象，因此对路面点云数据和图像数据进行融合来生成路面元素的纹理图。For extreme weather conditions, such as rainy, foggy, and dusty weather, texture images can accurately reflect static objects, so the road point cloud data and image data are fused to generate a texture map of the road elements.
一种可能的实施例中，步骤140，具体可以包括以下步骤：In a possible embodiment, step 140 may specifically include the following steps:
步骤310，将路面点云数据投影至多个摄像头的坐标系对应的图像数据中，得到与路面点云数据对应的第一像素值；Step 310, projecting the road surface point cloud data into image data corresponding to the coordinate systems of multiple cameras to obtain a first pixel value corresponding to the road surface point cloud data;
步骤320，根据定位数据对第一像素值进行重建，得到纹理图像。Step 320: reconstruct the first pixel value according to the positioning data to obtain a texture image.
涉及步骤310，具体可以将同一帧的路面点云数据，分段进行地面拟合。使用分段拟合的结果对地面点云进行插值，形成稠密的地面点云数据。分段拟合，是不同空间区间的地面点云，比如从-50米到50米这个范围内，分成了十段。然后将稠密的地面点云数据投影至多个摄像头的坐标系对应的图像数据中，得到与路面点云数据对应的第一像素值。In step 310, the road point cloud data of the same frame can be segmented for ground fitting. The ground point cloud is interpolated using the result of segmented fitting to form dense ground point cloud data. Segmented fitting is ground point clouds in different spatial intervals, such as from -50 meters to 50 meters, divided into ten segments. Then the dense ground point cloud data is projected into the image data corresponding to the coordinate system of multiple cameras to obtain the first pixel value corresponding to the road point cloud data.
具体可以通过传感器和摄像头的内外参数把路面点云数据投影到摄像头所在的平面，那么就可以获取该路面点云数据对应的第一像素值。Specifically, the road surface point cloud data can be projected onto the plane where the camera is located through the internal and external parameters of the sensor and the camera, so that the first pixel value corresponding to the road surface point cloud data can be obtained.
涉及步骤320，根据定位数据对第一像素值进行拼接和渲染处理，得到纹理图像。In step 320, the first pixel value is spliced and rendered according to the positioning data to obtain a texture image.
一种可能的实施例中，步骤310，包括：In a possible embodiment, step 310 includes:
在路面点云数据对应多个坐标系的情况下，确定路面点云数据在每个坐标系下图像数据对应的第二像素值；In the case where the road surface point cloud data corresponds to multiple coordinate systems, determining a second pixel value corresponding to the image data of the road surface point cloud data in each coordinate system;
根据每个坐标系下图像数据对应的第二像素值，与路面点云数据对应的第一像素值。According to the second pixel value corresponding to the image data in each coordinate system, and the first pixel value corresponding to the road point cloud data.
对于同一个点的地面点云数据，如果可以投影到不同的坐标系上，确定路面点云数据在每个坐标系下图像数据对应的第二像素值。For the ground point cloud data of the same point, if it can be projected onto different coordinate systems, the second pixel value corresponding to the image data of the road surface point cloud data in each coordinate system is determined.
其中，可以根据摄像头的坐标系范围，来确定同一个点的地面点云数据是否可以投影到不同的坐标系上。Among them, it can be determined whether the ground point cloud data of the same point can be projected into different coordinate systems according to the coordinate system range of the camera.
根据每个坐标系下图像数据对应的第二像素值，与路面点云数据对应的第一像素值，具体可以包括以下步骤：According to the second pixel value corresponding to the image data in each coordinate system and the first pixel value corresponding to the road point cloud data, the following steps may be specifically included:
对于同一个点的地面点云数据，对路面点云数据在每个坐标系下图像数据对应的第二像素值进行取均值处理，得到第一像素值。For the ground point cloud data of the same point, the second pixel value corresponding to the image data of the road surface point cloud data in each coordinate system is averaged to obtain the first pixel value.
由此，可以对多个摄像头坐标系采集的图像数据进行综合考量，根据路面点云数据在每个坐标系下图像数据对应的第二像素值，确定路面点云数据对应的第一像素值，由此，能够提升路面点云数据在每个坐标系下对应的准确度。Therefore, the image data collected by multiple camera coordinate systems can be comprehensively considered, and the first pixel value corresponding to the road point cloud data can be determined according to the second pixel value corresponding to the image data in each coordinate system of the road point cloud data. Therefore, the accuracy of the road point cloud data in each coordinate system can be improved.
一种可能的实施例中，步骤320，包括：In a possible embodiment, step 320 includes:
从第一像素值中去除动态对象的像素值，得到第三像素值；Subtracting the pixel value of the dynamic object from the first pixel value to obtain a third pixel value;
根据定位数据对第三像素值进行重建，得到纹理图像。The third pixel value is reconstructed according to the positioning data to obtain a texture image.
由于有些地面点投影到图像上不是图像上的路面点，因此，需要确定动态对象。Since some ground points projected onto the image are not road surface points on the image, dynamic objects need to be identified.
具体可以通过3D检测框识别动态对象，其中，3D检测框可以是一个外部模块，具体可以使用bevfusion的模型来进行检测。Specifically, the dynamic object can be identified through a 3D detection frame, wherein the 3D detection frame can be an external module, and specifically, the bevfusion model can be used for detection.
从第一像素值中去除动态对象的像素值，能够去除获取动态对象在不同图像中的像素值，得到第三像素值。由此，根据定位数据对第三像素值进行重建得到的纹理图像中，不会包括动态对象对应的像素值，便于后续对纹理图像中的静态对象进行标注。By removing the pixel value of the dynamic object from the first pixel value, the pixel value of the dynamic object in different images can be removed to obtain a third pixel value. Therefore, the texture image obtained by reconstructing the third pixel value according to the positioning data will not include the pixel value corresponding to the dynamic object, which is convenient for subsequent marking of the static object in the texture image.
涉及步骤150。This involves step 150 .
根据所述定位数据对反射率图像和纹理图像中的静态对象进行标注，得到训练样本数据。Static objects in the reflectivity image and the texture image are annotated according to the positioning data to obtain training sample data.
根据所述定位数据对反射率图像和纹理图像中的静态元素进行自动标注，可以减少人工标注的工作量，提升训练样本数据的获取效率。Automatically labeling static elements in the reflectivity image and the texture image according to the positioning data can reduce the workload of manual labeling and improve the efficiency of obtaining training sample data.
一种可能的实施例中，步骤150，包括：In a possible embodiment, step 150 includes:
根据定位数据，将反射率图像和纹理图像划分为多个子图像，多个子图像包括：基于反射率图像生成的多个第一子图像和基于纹理图像生成的多个第二子图像；According to the positioning data, the reflectivity image and the texture image are divided into a plurality of sub-images, wherein the plurality of sub-images include: a plurality of first sub-images generated based on the reflectivity image and a plurality of second sub-images generated based on the texture image;
分别标注第一子图像和第二子图像中的静态对象位置信息；Respectively marking the position information of the static objects in the first sub-image and the second sub-image;
根据定位数据将各个子图像中的静态对象位置信息进行合并，得到目标静态元素位置信息；The static object position information in each sub-image is merged according to the positioning data to obtain the target static element position information;
在反射率图像和纹理图像上标注目标静态元素位置信息，得到训练样本数据。The position information of the target static elements is marked on the reflectivity image and the texture image to obtain the training sample data.
根据定位数据，将反射率图像和纹理图像划分为多个子图像，每个子图像的尺寸可以为1000×1000。According to the positioning data, the reflectivity image and the texture image are divided into a plurality of sub-images, and the size of each sub-image may be 1000×1000.
然后通过一个神经网络作为编码器来得到子图像的特征图，再通过解码器分别标注第一子图像和第二子图像中的静态对象位置信息，来得到实例级的静态元素位置信息。Then, a neural network is used as an encoder to obtain the feature map of the sub-image, and then a decoder is used to annotate the static object position information in the first sub-image and the second sub-image respectively to obtain the instance-level static element position information.
静态对象包括下述中的至少一项：包括车道线、路沿，交通标志、停止线和人行道；Static objects include at least one of the following: lane lines, curbs, traffic signs, stop lines, and sidewalks;
也就是说，对于每个切分得到的子图像进行模型推理，得到单个子图像的静态元素位置信息，然后根据定位数据对静态对象位置信息进行拼接，目标静态元素位置信息。That is to say, model inference is performed on each sub-image obtained by segmentation to obtain the static element position information of a single sub-image, and then the static object position information is spliced according to the positioning data to obtain the target static element position information.
最后，在反射率图像和纹理图像上标注目标静态元素位置信息，得到训练样本数据。Finally, the position information of the target static elements is marked on the reflectivity image and the texture image to obtain the training sample data.
可选地，在反射率图像和纹理图像上标注目标静态元素位置信息，得到训练样本数据之前，还可以包括：人工修正目标静态元素位置信息。Optionally, before marking the target static element position information on the reflectivity image and the texture image to obtain the training sample data, the method may further include: manually correcting the target static element position information.
本申请实施例中，通过获取移动装置在移动过程中的样本数据，样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；由于采集得到的第一点云数据包括实际场景中的多种对象的点云数据，因此需要对根据定位数据对第一点云数据进行处理，得到路面点云数据。然后，根据定位数据对路面点云数据进行重建处理，得到反射率图像。接着，为了提升训练样本数据在各个场景下的适用性，根据路面点云数据和图像数据生成纹理图像，纹理图像在各种极端天气场景中能够清晰地体现场景特征，提升场景适用性，最后，对反射率图像和纹理图像中的静态对象进行标注，得到训练样本数据。由此，训练样本数据能够用于精进推理模型的推理准确度。In an embodiment of the present application, by acquiring sample data of a mobile device during movement, the sample data includes: first point cloud data collected by a sensor, positioning data of the sensor, and image data collected by a camera; since the first point cloud data collected includes point cloud data of multiple objects in the actual scene, it is necessary to process the first point cloud data according to the positioning data to obtain road surface point cloud data. Then, the road surface point cloud data is reconstructed according to the positioning data to obtain a reflectivity image. Next, in order to improve the applicability of the training sample data in various scenarios, a texture image is generated based on the road surface point cloud data and image data. The texture image can clearly reflect the scene features in various extreme weather scenarios and improve the applicability of the scene. Finally, the static objects in the reflectivity image and the texture image are annotated to obtain training sample data. Therefore, the training sample data can be used to improve the reasoning accuracy of the reasoning model.
基于上述图1所示的图像处理方法，本申请实施例还提供一种图像处理装置，如图2所示，该图像处理装置400可以包括：Based on the image processing method shown in FIG. 1 , the embodiment of the present application further provides an image processing device. As shown in FIG. 2 , the image processing device 400 may include:
获取模块410，用于获取移动装置在移动过程中的样本数据，所述样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；An acquisition module 410 is used to acquire sample data of the mobile device during movement, wherein the sample data includes: first point cloud data acquired by the sensor, positioning data of the sensor, and image data acquired by the camera;
处理模块420，用于根据所述定位数据对所述第一点云数据进行处理，得到路面点云数据；A processing module 420, configured to process the first point cloud data according to the positioning data to obtain road surface point cloud data;
所述处理模块420，还用于根据所述定位数据对所述路面点云数据进行重建处理，得到反射率图像；The processing module 420 is further used to reconstruct the road surface point cloud data according to the positioning data to obtain a reflectivity image;
生成模块430，用于根据所述路面点云数据和所述图像数据，生成纹理图像；A generating module 430, configured to generate a texture image according to the road surface point cloud data and the image data;
标注模块440，用于根据所述定位数据对所述反射率图像和所述纹理图像中的静态对象进行标注，得到训练样本数据。The labeling module 440 is used to label the static objects in the reflectivity image and the texture image according to the positioning data to obtain training sample data.
在一种可能的实施例中，处理模块420，具体用于：In a possible embodiment, the processing module 420 is specifically configured to:
从所述第一点云数据中去除动态对象对应的点云数据，得到第二点云数据；removing point cloud data corresponding to the dynamic object from the first point cloud data to obtain second point cloud data;
根据所述定位数据对所述第二点云数据进行处理，得到所述路面点云数据。The second point cloud data is processed according to the positioning data to obtain the road surface point cloud data.
在一种可能的实施例中，所述第一点云数据包括反射率，处理模块420，具体用于：In a possible embodiment, the first point cloud data includes reflectivity, and the processing module 420 is specifically used to:
从所述第一点云数据中确定所述路面点云数据关联的反射率；Determining a reflectivity associated with the road surface point cloud data from the first point cloud data;
根据所述路面点云数据关联的反射率对路面点云数据进行重建处理，得到所述反射率图像。The road surface point cloud data is reconstructed according to the reflectivity associated with the road surface point cloud data to obtain the reflectivity image.
在一种可能的实施例中，获取模块410，具体用于：In a possible embodiment, the acquisition module 410 is specifically configured to:
确定所述移动装置所在的场景；Determining a scene where the mobile device is located;
在所述场景为预设场景的情况下，获取所述移动装置在多次移动过程中采集的多个所述样本数据。When the scene is a preset scene, a plurality of sample data collected by the mobile device during multiple movements are obtained.
在一种可能的实施例中，处理模块420，具体用于：In a possible embodiment, the processing module 420 is specifically configured to:
确定任意两次所述移动过程对应的路面点云数据之间的转换矩阵；Determine a transformation matrix between road surface point cloud data corresponding to any two movement processes;
根据所述转换矩阵对多次移动过程对应的路面点云数据进行拼接，得到所述路面点云数据；splicing the road surface point cloud data corresponding to the multiple movement processes according to the transformation matrix to obtain the road surface point cloud data;
根据所述路面点云数据关联的反射率对路面点云数据进行重建处理，得到所述反射率图像。The road surface point cloud data is reconstructed according to the reflectivity associated with the road surface point cloud data to obtain the reflectivity image.
在一种可能的实施例中，获取模块410，还用于获取多次所述移动过程中采集得到的参照物位置信息；In a possible embodiment, the acquisition module 410 is further used to acquire the reference object position information collected during the multiple movement processes;
该图像处理装置400可以包括：The image processing device 400 may include:
聚类模块，用于对多次采集得到的参照物位置信息进行聚类，得到聚合后的参照物位置信息；A clustering module, used for clustering the reference object position information obtained by multiple collections to obtain aggregated reference object position information;
确定模块，用于根据聚合后的参照物位置信息，确定路口位置信息；A determination module, used to determine the intersection position information according to the aggregated reference object position information;
识别模块，用于根据路口位置信息，从所述多次移动过程对应的所述路面点云数据中识别出每次采集得到的路面点云数据。The identification module is used to identify the road surface point cloud data collected each time from the road surface point cloud data corresponding to the multiple movement processes according to the intersection position information.
在一种可能的实施例中，生成模块430，具体用于：In a possible embodiment, the generating module 430 is specifically configured to:
将所述路面点云数据投影至多个所述摄像头的坐标系对应的图像数据中，得到与路面点云数据对应的第一像素值；Projecting the road surface point cloud data onto image data corresponding to the coordinate systems of the plurality of cameras to obtain a first pixel value corresponding to the road surface point cloud data;
根据所述定位数据对所述第一像素值进行重建，得到所述纹理图像。The first pixel value is reconstructed according to the positioning data to obtain the texture image.
在一种可能的实施例中，生成模块430，具体用于：In a possible embodiment, the generating module 430 is specifically configured to:
在所述路面点云数据对应多个坐标系的情况下，确定路面点云数据在每个所述坐标系下对应的第二像素值；In a case where the road surface point cloud data corresponds to a plurality of coordinate systems, determining a second pixel value corresponding to the road surface point cloud data in each of the coordinate systems;
根据每个所述坐标系下对应的所述第二像素值，与路面点云数据对应的第一像素值。According to the second pixel value corresponding to each of the coordinate systems, the first pixel value corresponding to the road point cloud data.
在一种可能的实施例中，生成模块430，具体用于：In a possible embodiment, the generating module 430 is specifically configured to:
从所述第一像素值中去除动态对象的像素值，得到第三像素值；Subtract the pixel value of the dynamic object from the first pixel value to obtain a third pixel value;
根据所述定位数据对所述第三像素值进行重建，得到所述纹理图像。The third pixel value is reconstructed according to the positioning data to obtain the texture image.
在一种可能的实施例中，标注模块440，具体用于：In a possible embodiment, the marking module 440 is specifically configured to:
根据所述定位数据，将所述反射率图像和所述纹理图像划分为多个子图像，所述多个子图像包括：基于所述反射率图像生成的多个第一子图像和基于所述纹理图像生成的多个第二子图像；According to the positioning data, the reflectivity image and the texture image are divided into a plurality of sub-images, wherein the plurality of sub-images include: a plurality of first sub-images generated based on the reflectivity image and a plurality of second sub-images generated based on the texture image;
分别标注所述第一子图像和所述第二子图像中的静态对象位置信息；Respectively marking the position information of static objects in the first sub-image and the second sub-image;
根据所述定位数据将各个所述子图像中的静态对象位置信息进行合并，得到目标静态元素位置信息；Merging the static object position information in each of the sub-images according to the positioning data to obtain the target static element position information;
在所述反射率图像和所述纹理图像上标注所述目标静态元素位置信息，得到所述训练样本数据。The target static element position information is marked on the reflectivity image and the texture image to obtain the training sample data.
本申请实施例中，通过获取移动装置在移动过程中的样本数据，样本数据包括：传感器采集的第一点云数据、传感器的定位数据和摄像头采集的图像数据；由于采集得到的第一点云数据包括实际场景中的多种对象的点云数据，因此需要对根据定位数据对第一点云数据进行处理，得到路面点云数据。然后，根据定位数据对路面点云数据进行重建处理，得到反射率图像。接着，为了提升训练样本数据在各个场景下的适用性，根据路面点云数据和图像数据生成纹理图像，纹理图像在各种极端天气场景中能够清晰地体现场景特征，提升场景适用性，最后，对反射率图像和纹理图像进行自动标注，得到训练样本数据。由此，训练样本数据能够用于精进推理模型的推理准确度。In an embodiment of the present application, by acquiring sample data of a mobile device during movement, the sample data includes: first point cloud data collected by a sensor, positioning data of the sensor, and image data collected by a camera; since the first point cloud data collected includes point cloud data of multiple objects in the actual scene, it is necessary to process the first point cloud data according to the positioning data to obtain road surface point cloud data. Then, the road surface point cloud data is reconstructed according to the positioning data to obtain a reflectivity image. Next, in order to improve the applicability of the training sample data in various scenarios, a texture image is generated based on the road surface point cloud data and image data. The texture image can clearly reflect the scene features in various extreme weather scenarios and improve the applicability of the scene. Finally, the reflectivity image and the texture image are automatically annotated to obtain the training sample data. Therefore, the training sample data can be used to improve the reasoning accuracy of the reasoning model.
图5示出了本申请实施例提供的一种电子设备的硬件结构示意图。FIG5 shows a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.
在电子设备可以包括处理器501以及存储有计算机程序指令的存储器502。The electronic device may include a processor 501 and a memory 502 storing computer program instructions.
具体地，上述处理器501可以包括中央处理器(CPU)，或者特定集成电路(Application Specific Integrated Circuit，ASIC)，或者可以被配置成实施本申请实施例的一个或多个集成电路。Specifically, the processor 501 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.
存储器502可以包括用于数据或指令的大容量存储器。举例来说而非限制，存储器502可包括硬盘驱动器(Hard Disk Drive，HDD)、软盘驱动器、闪存、光盘、磁光盘、磁带或通用串行总线(Universal Serial Bus，USB)驱动器或者两个或更多个以上这些的组合。在合适的情况下，存储器502可包括可移除或不可移除(或固定)的介质。在合适的情况下，存储器502可在综合网关容灾设备的内部或外部。在特定实施例中，存储器502是非易失性固态存储器。在特定实施例中，存储器502包括只读存储器(ROM)。在合适的情况下，该ROM可以是掩模编程的ROM、可编程ROM(PROM)、可擦除PROM(EPROM)、电可擦除PROM(EEPROM)、电可改写ROM(EAROM)或闪存或者两个或更多个以上这些的组合。The memory 502 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 502 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 502 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 502 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 502 is a non-volatile solid-state memory. In a specific embodiment, the memory 502 includes a read-only memory (ROM). In appropriate cases, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM) or a flash memory or a combination of two or more of these.
处理器501通过读取并执行存储器502中存储的计算机程序指令，以实现图所示实施例中的任意一种图像处理方法。The processor 501 reads and executes computer program instructions stored in the memory 502 to implement any one of the image processing methods in the embodiments shown in the figures.
在一个示例中，电子设备还可包括通信接口505和总线510。其中，如图5所示，处理器501、存储器502、通信接口505通过总线510连接并完成相互间的通信。In one example, the electronic device may further include a communication interface 505 and a bus 510. As shown in Fig. 5, the processor 501, the memory 502, and the communication interface 505 are connected via the bus 510 and communicate with each other.
通信接口505，主要用于实现本申请实施例中各模块、装置、单元和/或设备之间的通信。The communication interface 505 is mainly used to implement communication between various modules, devices, units and/or equipment in the embodiments of the present application.
总线510包括硬件、软件或两者，将电子设备的部件彼此耦接在一起。举例来说而非限制，总线可包括加速图形端口(AGP)或其他图形总线、增强工业标准架构(EISA)总线、前端总线(FSB)、超传输(HT)互连、工业标准架构(ISA)总线、无限带宽互连、低引脚数(LPC)总线、存储器总线、微信道架构(MCA)总线、外围组件互连(PCI)总线、PCI-Express(PCI-X)总线、串行高级技术附件(SATA)总线、视频电子标准协会局部(VLB)总线或其他合适的总线或者两个或更多个以上这些的组合。在合适的情况下，总线510可包括一个或多个总线。尽管本申请实施例描述和示出了特定的总线，但本申请考虑任何合适的总线或互连。Bus 510 includes hardware, software or both, and the parts of electronic equipment are coupled to each other. For example, but not limitation, bus may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, infinite bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral component interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 510 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the application considers any suitable bus or interconnection.
该电子设备可以执行本申请实施例中的图像处理方法，从而实现结合图2描述的图像处理方法。The electronic device can execute the image processing method in the embodiment of the present application, thereby realizing the image processing method described in combination with FIG. 2 .
另外，结合上述实施例中的图像处理方法，本申请实施例可提供一种计算机可读存储介质来实现。该计算机可读存储介质上存储有计算机程序指令；该计算机程序指令被处理器执行时实现图1的图像处理方法。In addition, in combination with the image processing method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, the image processing method of FIG. 1 is implemented.
需要明确的是，本申请并不局限于上文所描述并在图中示出的特定配置和处理。为了简明起见，这里省略了对已知方法的详细描述。在上述实施例中，描述和示出了若干具体的步骤作为示例。但是，本申请的方法过程并不限于所描述和示出的具体步骤，本领域的技术人员可以在领会本申请的精神后，作出各种改变、修改和添加，或者改变步骤之间的顺序。It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
以上所述的结构框图中所示的功能块可以实现为硬件、软件、固件或者它们的组合。当以硬件方式实现时，其可以例如是电子电路、专用集成电路(ASIC)、适当的固件、插件、功能卡等等。当以软件方式实现时，本申请的元素是被用于执行所需任务的程序或者代码段。程序或者代码段可以存储在机器可读介质中，或者通过载波中携带的数据信号在传输介质或者通信链路上传送。“机器可读介质”可以包括能够存储或传输信息的任何介质。机器可读介质的例子包括电子电路、半导体存储器设备、ROM、闪存、可擦除ROM(EROM)、软盘、CD-ROM、光盘、硬盘、光纤介质、射频(RF)链路，等等。代码段可以经由诸如因特网、内联网等的计算机网络被下载。The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
还需要说明的是，本申请中提及的示例性实施例，基于一系列的步骤或者装置描述一些方法或系统。但是，本申请不局限于上述步骤的顺序，也就是说，可以按照实施例中提及的顺序执行步骤，也可以不同于实施例中的顺序，或者若干步骤同时执行。It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
以上所述，仅为本申请的具体实施方式，所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的系统、模块和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。应理解，本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到各种等效的修改或替换，这些修改或替换都应涵盖在本申请的保护范围之内。The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410583744.4ACN118628860A (en) | 2024-05-11 | 2024-05-11 | Image processing method, device, electronic device and storage medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410583744.4ACN118628860A (en) | 2024-05-11 | 2024-05-11 | Image processing method, device, electronic device and storage medium |
| Publication Number | Publication Date |
|---|---|
| CN118628860Atrue CN118628860A (en) | 2024-09-10 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410583744.4APendingCN118628860A (en) | 2024-05-11 | 2024-05-11 | Image processing method, device, electronic device and storage medium |
| Country | Link |
|---|---|
| CN (1) | CN118628860A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119413181A (en)* | 2025-01-02 | 2025-02-11 | 苏州天硕导航科技有限责任公司 | Visual layout algorithm and GNSS integrated navigation system based on image matching |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119413181A (en)* | 2025-01-02 | 2025-02-11 | 苏州天硕导航科技有限责任公司 | Visual layout algorithm and GNSS integrated navigation system based on image matching |
| Publication | Publication Date | Title |
|---|---|---|
| CN114862901B (en) | A road-side multi-source sensor fusion target perception method and system for open-pit mines | |
| CN112740225B (en) | A kind of pavement element determination method and device | |
| US11482014B2 (en) | 3D auto-labeling with structural and physical constraints | |
| CN111542860B (en) | Sign and lane creation for high definition maps of autonomous vehicles | |
| CN114842438A (en) | Terrain detection method, system and readable storage medium for autonomous driving vehicle | |
| CN111353453B (en) | Obstacle detection method and device for vehicle | |
| CN112465970B (en) | Navigation map construction method, device, system, electronic device and storage medium | |
| WO2023123837A1 (en) | Map generation method and apparatus, electronic device, and storage medium | |
| Zhou et al. | Developing and testing robust autonomy: The university of sydney campus data set | |
| WO2020133415A1 (en) | Systems and methods for constructing a high-definition map based on landmarks | |
| CN117576652B (en) | Road object identification method and device, storage medium and electronic equipment | |
| CN116817891A (en) | Real-time multi-mode sensing high-precision map construction method | |
| CN111008660A (en) | Method, device, system, storage medium and electronic device for generating semantic map | |
| CN114926485B (en) | Image depth annotation method, device, equipment and storage medium | |
| CN115564865A (en) | Construction method and system of crowdsourcing high-precision map, electronic equipment and vehicle | |
| CN115235478B (en) | Intelligent automobile positioning method and system based on visual label and laser SLAM | |
| CN116309943B (en) | Parking lot semantic map road network construction method and device and electronic equipment | |
| CN114550116A (en) | Object identification method and device | |
| CN113724387A (en) | Laser and camera fused map construction method | |
| CN117611762B (en) | Multi-level map construction method, system and electronic equipment | |
| CN114419180A (en) | Method, device and electronic device for reconstructing high-precision map | |
| CN113988197A (en) | Multi-camera and multi-laser radar based combined calibration and target fusion detection method | |
| CN118628860A (en) | Image processing method, device, electronic device and storage medium | |
| EP4239288B1 (en) | Methods for processing a map, and vehicle | |
| CN113781639B (en) | Quick construction method for digital model of large-scene road infrastructure |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |