CN109889799B

Movatterモバイル変換

Info

Publication number: CN109889799B
Application number: CN201711279413.8A
Authority: CN
Inventors: 葛晨阳; 邓作为; 侯晓琦
Original assignee: Xian Jiaotong University
Current assignee: Xian Jiaotong University
Priority date: 2017-12-06
Filing date: 2017-12-06
Publication date: 2020-08-25
Anticipated expiration: 2037-12-06
Also published as: CN109889799A

Abstract

The invention provides a monocular structured light depth perception method and a monocular structured light depth perception device based on an RGBIR camera, which are characterized in that an infrared laser speckle coding projector is adopted to project an infrared speckle image, or an infrared pan-light source is utilized to irradiate the infrared image, then the RGBIR camera is used for collecting RGBIR RAW data according to frames and separating the RGBIR RAW data, image signal processing is carried out on a separated RGB image, depth perception decoding is carried out on the infrared speckle image to obtain a depth value or a 3D point cloud, infrared image enhancement processing is carried out on the infrared image, and finally, the depth image, the RGB image, the infrared image, the RGBD combined data or XYZRGB point cloud data can be selected to be output.

Description

Translated fromChinese

基于RGBIR摄像头的单目结构光深度感知方法及装置Monocular structured light depth perception method and device based on RGBIR camera

技术领域technical field

本发明属于计算机视觉、人机交互和集成电路技术领域，具体涉及一种基于RGBIR摄像头的单目结构光深度感知方法及装置。The invention belongs to the technical fields of computer vision, human-computer interaction and integrated circuits, and in particular relates to a monocular structured light depth perception method and device based on an RGBIR camera.

背景技术Background technique

当前虚拟世界正无限接近于现实世界，人机交互技术将成为连接虚拟与现实的“桥梁”。随着计算机性能的不断提升，人们也在不断地探索更加简单便捷的方式与机器交流。其中视觉感官则是人类观察、认识、体验的最直接也是最重要的手段。At present, the virtual world is infinitely close to the real world, and human-computer interaction technology will become a "bridge" connecting virtual and reality. With the continuous improvement of computer performance, people are constantly exploring simpler and more convenient ways to communicate with machines. Among them, the visual sense is the most direct and important means of human observation, cognition and experience.

基于结构光主动视觉的深度感知技术可以获取较为准确的深度信息，且不受环境光的影响、算法和硬件实现相对简单。但目前的结构光深度感知装置中RGB摄像头和IR摄像头是分离的，RGB摄像头可以获得纹理信息，IR摄像头可以获得红外图像和深度信息，两者需要通过同步和RGBD精确配准才能建立一对一的对应关系，参见专利CN204481940U，双目摄像头拍照移动终端，就是利用一RGB彩色摄像头，一IR红外摄像头的双摄像头配置进行深度信息获取。但由于装配精度、标定精度和使用过程的影响，往往很难实现远近不同距离的RGB图像素与深度图或3D点云能一一对应。The depth perception technology based on structured light active vision can obtain more accurate depth information, and is not affected by ambient light, and the algorithm and hardware implementation are relatively simple. However, in the current structured light depth sensing device, the RGB camera and the IR camera are separated. The RGB camera can obtain texture information, and the IR camera can obtain infrared image and depth information. The two need to be synchronized and RGBD accurate registration to establish one-to-one. For the corresponding relationship, see patent CN204481940U. The mobile terminal for taking pictures with binocular cameras uses a dual-camera configuration of an RGB color camera and an IR infrared camera to obtain depth information. However, due to the influence of assembly accuracy, calibration accuracy and use process, it is often difficult to achieve one-to-one correspondence between RGB image pixels at different distances and depth maps or 3D point clouds.

发明内容SUMMARY OF THE INVENTION

鉴于此，本发明提供了一种基于RGBIR摄像头的单目结构光深度感知方法及装置，采用红外激光散斑编码投射器投射红外散斑图像，或用红外泛光源照射出红外图像，用单一的RGBIR摄像头采集RGBIR图，深度感知解码模块对输入的RGBIR RAW数据进行分离，包括RGB图、IR图(红外散斑图或红外图)，其中对RGB图进行传统图像信号处理(Image SignalProcessing，ISP)，对红外散斑图进行深度感知解码得到深度值或3D点云，对红外图则进行红外图像增强处理，最后可选择输出深度图、RGB图、红外图、RGBD联合数据或XYZRGB点云数据。In view of this, the present invention provides a monocular structured light depth perception method and device based on an RGBIR camera, using an infrared laser speckle encoder projector to project an infrared speckle image, or using an infrared flood light source to illuminate an infrared image, and using a single The RGBIR camera collects RGBIR images, and the depth perception decoding module separates the input RGBIR RAW data, including RGB images, IR images (infrared speckle images or infrared images), and traditional image signal processing (Image Signal Processing, ISP) is performed on the RGB images. , perform depth perception decoding on the infrared speckle image to obtain the depth value or 3D point cloud, perform infrared image enhancement processing on the infrared image, and finally select the output depth map, RGB image, infrared image, RGBD combined data or XYZRGB point cloud data.

为实现以上发明目的，本发明提供了一种基于RGBIR摄像头的单目结构光深度感知方法，具体地包括以下步骤：In order to achieve the above purpose of the invention, the present invention provides a monocular structured light depth perception method based on an RGBIR camera, which specifically includes the following steps:

步骤1：RGBIR图像投射，利用红外激光散斑编码投射器投射红外散斑图像或利用红外泛光源照射出红外图像到目标物体或投射空间上；Step 1: RGBIR image projection, use the infrared laser speckle encoder projector to project the infrared speckle image or use the infrared flood light source to irradiate the infrared image to the target object or projection space;

步骤2：RGBIR图像采集，利用单一RGBIR摄像头将步骤1发出的红外散斑图像或者红外图像，和可见光RGB图像一起采集进来，获得RGBIR RAW数据；Step 2: RGBIR image acquisition, using a single RGBIR camera to collect the infrared speckle image or infrared image sent instep 1 together with the visible light RGB image to obtain RGBIR RAW data;

步骤3：RGBIR图像分离，将步骤2采集到的RGBIR RAW数据进行图像分离，形成RGB图像，红外散斑图像或红外图像，为后面各自的图像处理做好准备；Step 3: RGBIR image separation, separate the RGBIR RAW data collected instep 2 to form an RGB image, an infrared speckle image or an infrared image, and prepare for the subsequent image processing;

步骤4：分离后图像处理，将分离出来的RGB图像进行常规信号处理ISP，分离出来的红外图像进行红外增强处理，分离出来的红外散斑图则进行深度感知解码，其处理过程包括：预处理，图像旋转，块匹配，深度计算，后处理，三维点云生成；Step 4: Image processing after separation, the separated RGB image is subjected to conventional signal processing ISP, the separated infrared image is subjected to infrared enhancement processing, and the separated infrared speckle image is subjected to depth perception decoding. The processing process includes: preprocessing , image rotation, block matching, depth calculation, post-processing, 3D point cloud generation;

步骤5：数据选择输出，将经过图像信号处理后的RGB图像、红外图像、深度图或点云进行选择性的输出，可选择输出深度图，RGB图像，红外图像，RGBD数据或者XYZRGB点云数据。Step 5: Data selection output, selectively output RGB image, infrared image, depth map or point cloud after image signal processing, you can choose to output depth map, RGB image, infrared image, RGBD data or XYZRGB point cloud data .

此外，本发明还提供了一种基于RGBIR摄像头的单目结构光深度感知装置，具体包括：RGBIR图像产生模块，RGBIR图像采集模块，RGBIR图像分离模块，RGBIR图像处理模块，和选择输出模块：In addition, the present invention also provides a monocular structured light depth perception device based on RGBIR camera, which specifically includes: RGBIR image generation module, RGBIR image acquisition module, RGBIR image separation module, RGBIR image processing module, and selection output module:

RGBIR图像产生模块，用于通过红外激光散斑编码投射器投射出红外散斑图像到目标物体或投射空间，或利用红外泛光源照射出红外图像；The RGBIR image generation module is used to project the infrared speckle image to the target object or the projection space through the infrared laser speckle encoder projector, or use the infrared flood light source to illuminate the infrared image;

RGBIR图像采集模块，用于利用单一RGBIR摄像头将红外激光散斑编码投射器投射的所述红外散斑图像或红外泛光源照射出的所述红外图像，和可见光RGB图像一起采集进来，获得RGBIR RAW数据；RGBIR图像分离模块，用于将采集到的RGBIR RAW数据进行图像分离，形成RGB图像，红外散斑图像或红外图像，分离后的RGB图像、红外散斑图像或红外图像可进行存储或分别送给后续模块进行处理；The RGBIR image acquisition module is used to use a single RGBIR camera to collect the infrared speckle image projected by the infrared laser speckle coding projector or the infrared image irradiated by the infrared flood light source together with the visible light RGB image to obtain RGBIR RAW Data; RGBIR image separation module, which is used to separate the collected RGBIR RAW data to form RGB images, infrared speckle images or infrared images. The separated RGB images, infrared speckle images or infrared images can be stored or separately Send it to subsequent modules for processing;

RGBIR图像处理模块，用于对RGB图像和红外散斑图像，或者红外图像分别处理；RGBIR image processing module, used to process RGB image and infrared speckle image, or infrared image separately;

选择输出模块，用于将经过处理后的RGB图像、红外图像、深度图或点云进行选择性的输出，可选择输出深度图，RGB图像，红外图像，RGBD数据或者XYZRGB点云数据。Select the output module to selectively output the processed RGB image, infrared image, depth map or point cloud. You can choose to output the depth map, RGB image, infrared image, RGBD data or XYZRGB point cloud data.

本发明采用了单目RGBIR摄像头统一采集图像，并对图像分离，最后对分离后的RGB图像和IR图像(红外散斑图像或红外图像)分别处理再融合配准，相比其他现有技术中的双目摄像头结构光深度感知装置，本发明的单目结构光深度感知装置可以减少装配精度、标定精度和使用过程的影响，并且可以实现远近不同距离的RGB图像素与深度图或3D点云一一对应。The invention adopts a monocular RGBIR camera to collect images uniformly, separates the images, and finally processes the separated RGB images and IR images (infrared speckle images or infrared images) respectively, and then fuses and registers. The binocular camera structured light depth perception device, the monocular structured light depth perception device of the present invention can reduce the influence of assembly accuracy, calibration accuracy and use process, and can realize RGB image pixels and depth maps or 3D point clouds at different distances. One-to-one correspondence.

此外，本发明对分离后的红外图像采用了改进的红外图像增强方法，可以对红外图像的噪声进行去除，而且很好的保护图像的边缘，轮廓等细节。在对分离后红外散斑图像进行处理的时候，本发明还会对图像进行旋转操作，从而使图像方便的读取和储存，解决实际使用中摄像头因为人们的需要而进行旋转时，数据按图像的短边进行读取，会破坏图像的数据结构的问题。上述的多个有益效果将通过以下实施例的阐述而得到具体的体现。In addition, the present invention adopts an improved infrared image enhancement method for the separated infrared image, which can remove the noise of the infrared image, and can well protect the edge, outline and other details of the image. When processing the separated infrared speckle image, the present invention will also rotate the image, so that the image can be easily read and stored, and solve the problem that when the camera rotates due to people's needs in actual use, the data is based on the image. The problem of reading the short side of the image will destroy the data structure of the image. The above-mentioned beneficial effects will be embodied in detail through the description of the following embodiments.

附图说明Description of drawings

图1是基于RGBIR摄像头的单目结构光深度感知装置结构框架图；Figure 1 is a structural frame diagram of a monocular structured light depth perception device based on an RGBIR camera;

图2是基于RGBIR摄像头的单目结构光深度感知方法流程图；Fig. 2 is the flow chart of the monocular structured light depth perception method based on RGBIR camera;

图3是RGB图像的图像信号处理ISP流程图；Fig. 3 is the image signal processing ISP flow chart of RGB image;

图4是红外散斑图的深度感知解码框架图；Fig. 4 is the frame diagram of depth perception decoding of infrared speckle image;

图5是新型鱼眼校正模块框架图；Figure 5 is a frame diagram of a new type of fisheye correction module;

图6(a)和图6(b)是图像旋转参考图。6(a) and 6(b) are image rotation reference diagrams.

具体实施方式Detailed ways

下面结合附图1-6对本发明进行进一步的详细说明。The present invention will be further described in detail below in conjunction with accompanying drawings 1-6.

图1是一种基于RGBIR摄像头的单目结构光深度感知装置的结构框架图，该深度感知装置包括RGBIR图像产生模块，RGBIR图像采集模块，RGBIR图像分离模块，RGBIR图像处理模块，和选择输出模块。1 is a structural frame diagram of a monocular structured light depth sensing device based on an RGBIR camera. The depth sensing device includes an RGBIR image generation module, an RGBIR image acquisition module, an RGBIR image separation module, an RGBIR image processing module, and a selection output module. .

RGBIR图像产生模块是利用红外激光散斑编码投射器投射红外散斑图像或利用红外泛光源照射出红外图像，并利用RGBIR摄像头投射出可见光RGB图像。The RGBIR image generation module uses the infrared laser speckle encoder projector to project the infrared speckle image or uses the infrared flood light source to illuminate the infrared image, and uses the RGBIR camera to project the visible light RGB image.

RGBIR图像采集模块是利用单一RGBIR摄像头将红外激光散斑编码投射器投射的红外散斑图像或红外泛光源照射出的红外图像，和可见光RGB图像一起采集进来，获得RGBIR RAW数据。所述的RGBIR摄像头、红外激光散斑编码投射器或红外泛光源处于同一基线上排列，水平排列或垂直排列。The RGBIR image acquisition module uses a single RGBIR camera to collect the infrared speckle image projected by the infrared laser speckle encoder projector or the infrared image irradiated by the infrared flood light source, together with the visible light RGB image, to obtain RGBIR RAW data. The RGBIR cameras, infrared laser speckle coding projectors or infrared flood light sources are arranged on the same baseline, either horizontally or vertically.

RGBIR图像分离模块是将采集到的RGBIR RAW数据进行图像分离，具体地，可以利用滤光片等现有装置将RGBIR RAW数据分离，形成RGB图像，红外散斑图像或红外图像，分离后的RGB图像、红外散斑图像或红外图像可进行存储并/或分别送给后续模块进行处理。The RGBIR image separation module is to separate the collected RGBIR RAW data. Specifically, existing devices such as filters can be used to separate the RGBIR RAW data to form RGB images, infrared speckle images or infrared images, and the separated RGB Images, IR speckle images or IR images can be stored and/or sent to subsequent modules respectively for processing.

RGBIR图像处理模块是将RGBIR图像分离模块分离出的RGB图像，红外散斑图像/或红外图像分开进行图像处理。其中，对RGB图像进行传统的图像信号处理ISP处理，对红外图像进行红外图像增强处理，对红外散斑图像的处理则包括：预处理，图像旋转，块匹配，深度计算，后处理，三维点云生成等处理操作。红外图像增强处理的具体方法和对红外散斑图像的处理具体操作的见后详述。The RGBIR image processing module separates the RGB images, infrared speckle images and/or infrared images from the RGBIR image separation module for image processing. Among them, the traditional image signal processing ISP processing is performed on the RGB image, the infrared image enhancement processing is performed on the infrared image, and the processing on the infrared speckle image includes: preprocessing, image rotation, block matching, depth calculation, post-processing, 3D point processing Processing operations such as cloud generation. The specific method of infrared image enhancement processing and the specific operation of infrared speckle image processing are described in detail later.

选择输出模块是将经过ISP处理的RGB图像，经过红外图像增强处理的红外图像，以及经过预处理，图像旋转，块匹配，深度计算，后处理，三维点云生成的红外散斑图像进行选择输出。RGB图像经过ISP处理后能够得到效果良好的RGB彩色图，与此同时红外散斑图像通过预处理，图像旋转，块匹配之后再通过结构光单目计算公式得到图像深度值D，结合深度值D和内外相机标定参数三维重建得到XYZ数据。而RGB图像可以用来对点云配准或者进行选择输出。选择输出模块根据不同的需求可选择输出深度图、RGB图像、红外图像、RGBD数据或XYZRGB数据。The selection output module is to select and output the RGB image processed by ISP, the infrared image processed by infrared image enhancement, and the infrared speckle image generated by preprocessing, image rotation, block matching, depth calculation, post processing, and 3D point cloud. . After the RGB image is processed by ISP, a good RGB color image can be obtained. At the same time, the infrared speckle image is processed by preprocessing, image rotation, and block matching, and then the image depth value D is obtained through the structured light monocular calculation formula. Combined with the depth value D The XYZ data is obtained by 3D reconstruction with the calibration parameters of the internal and external cameras. The RGB image can be used for point cloud registration or selection output. The output module can choose to output depth map, RGB image, infrared image, RGBD data or XYZRGB data according to different needs.

图2是本发明的基于RGBIR摄像头的单目结构光深度感知方法的流程图，为了更加详细清楚的解释该方法，下面将结合图2-6来描述该方法。FIG. 2 is a flowchart of a monocular structured light depth perception method based on an RGBIR camera of the present invention. In order to explain the method in detail and clearly, the method will be described below with reference to FIGS. 2-6 .

步骤1：接通电源让电源驱动正常工作，使红外激光散斑编码投射器发出红光。进一步地，红外激光散斑编码投射器射出准直后的激光束，通过光学衍射元件(DOE)进行散射，进而得到所需的红外散斑图。在另一个实施例中，也可以利用红外泛光源照射出红外图像。同时驱动RGBIR摄像头，使RGBIR摄像头投射出RGB彩色图。Step 1: Turn on the power and let the power drive work normally, so that the infrared laser speckle encoder projector emits red light. Further, the infrared laser speckle encoder projector emits a collimated laser beam, which is scattered by an optical diffractive element (DOE) to obtain the required infrared speckle pattern. In another embodiment, an infrared flood light source can also be used to illuminate an infrared image. At the same time, the RGBIR camera is driven to make the RGBIR camera project an RGB color image.

步骤2：利用RGBIR摄像头将红外激光散斑编码投射器投射的红外散斑图像或红外泛光源照射出的红外图像，和由RGBIR摄像头投射出的RGB彩色图按帧采集，获得RGBIR RAW数据。Step 2: Use the RGBIR camera to capture the infrared speckle image projected by the infrared laser speckle coding projector or the infrared image irradiated by the infrared flood light source, and the RGB color image projected by the RGBIR camera by frame to obtain RGBIR RAW data.

步骤3：将经过步骤2采集到的RGBIR RAW数据进行图像分离，对分离后的红外散斑图像或红外图像和RGB彩色图进行分开读取和储存。RGB彩色图，红外散斑图像或红外图像进行分离后，为后面的图像处理做好准备。Step 3: Image separation of the RGBIR RAW data collected instep 2, and separate reading and storage of the separated infrared speckle image or infrared image and RGB color map. RGB color map, IR speckle image or IR image are separated and ready for subsequent image processing.

步骤4：将经过图像分离得到的RGB彩色图和红外散斑图像或红外图像分别进行图像处理。Step 4: Perform image processing on the RGB color map and the infrared speckle image or the infrared image obtained by the image separation respectively.

(1)对RGB彩色图进行图像信号处理ISP处理。具体的处理方式如图3所示。通过对前端图像传感器输出的信号做后期处理，对图像进行线性纠正、噪声去除、坏点去除的作用，从而得到效果良好的RGB彩色图。(1) Image signal processing ISP processing is performed on the RGB color image. The specific processing method is shown in Figure 3. By post-processing the signal output by the front-end image sensor, the image is linearly corrected, noise removed, and dead pixels removed, so as to obtain a good RGB color image.

(2)对红外散斑图像进行如下步骤的图像处理操作，见图4所示：(2) Perform the following image processing operations on the infrared speckle image, as shown in Figure 4:

预处理。本发明采用鱼眼校正的方法把输入的畸变，图像进行校正。如图5所示，利用DDR、QPI Flash的外部存储功能，存储畸变图像坐标与理想图像坐标的位移差值，然后与实时图像坐标同步运算，达到鱼眼畸变校正的目的，很好的满足实时性处理要求。将经过鱼眼校正的图像采用平均值法将彩色图像转换为灰度图，然后利用阈值转化算法将灰度图转化为黑白图，接着使用梯度锐化法使图像更加清晰便于分析。preprocessing. The invention uses the fisheye correction method to correct the input distortion and image. As shown in Figure 5, the external storage function of DDR and QPI Flash is used to store the displacement difference between the coordinates of the distorted image and the ideal image, and then synchronously operate with the real-time image coordinates to achieve the purpose of fisheye distortion correction, which is very good for real-time Sexual processing requirements. The fisheye-corrected image is converted into a grayscale image by the average method, and then the grayscale image is converted into a black-and-white image by the threshold conversion algorithm, and then the gradient sharpening method is used to make the image clearer and easier to analyze.

图像旋转。现有技术中对图像进行读取是按行进行读取的，也就是按图像的长边进行读取，而实际操作中摄像头经常会随着人们的需要而进行旋转，此时图像如果不进行旋转就会按图像的短边进行读取，从而破坏了图像的数据结构。图像旋转是为了更方便的进行图像的读取和储存。本发明中当摄像头进行旋转的时候，此时图像传感器会给图像处理模块一个信号，从而在图像处理的时候进行图像旋转，否则不旋转。图6(a)和图6(b)是图像旋转参考图，现有的图像信息的读取方式为从第一行1，2，3，4开始读取和存储，如图6(a)所示。图6(b)是图6(a)经过顺时针旋转90度后图像的读取方式，此时图像从第一行9，5，1开始读取和储存。Image rotation. In the prior art, the image is read by line, that is, the long side of the image is read. In actual operation, the camera often rotates according to people's needs. At this time, if the image is not read. Rotation breaks the data structure of the image by reading on the short side of the image. Image rotation is for easier image reading and storage. In the present invention, when the camera rotates, the image sensor will give a signal to the image processing module at this time, so as to rotate the image during image processing, otherwise it will not rotate. Fig. 6(a) and Fig. 6(b) are reference images of image rotation. The existing way of reading image information is to read and store from thefirst row 1, 2, 3, and 4, as shown in Fig. 6(a) shown. Figure 6(b) shows the image reading method after rotating 90 degrees clockwise in Figure 6(a). At this time, the image is read and stored from thefirst row 9, 5, and 1.

块匹配。输入红外散斑图像与已固化的参考散斑图像进行块匹配运动估计，通过块匹配相似度比较计算，按图像块方式得到输入红外散斑图像中的每个像素点与参考散斑图像中的最优匹配块中心点之间的偏移量，即运动估计矢量。其中，所述的参考散斑图像是作为参考比较基准，事先投射到与红外散斑编码图案投射器的光轴垂直的平面上采集并经相同的预处理得到，其离投射器的垂直参考距离d是已知的。block match. The input infrared speckle image and the solidified reference speckle image are subjected to block matching motion estimation. By comparing and calculating the similarity of block matching, each pixel in the input infrared speckle image and the reference speckle image are obtained according to the image block method. The offset between the center points of the optimal matching block, that is, the motion estimation vector. Wherein, the reference speckle image is used as a reference and comparison benchmark. It is projected on a plane perpendicular to the optical axis of the infrared speckle coding pattern projector to be collected and obtained through the same preprocessing, and its vertical reference distance from the projector is obtained. d is known.

深度计算。根据结构光单目深度计算公式获得红外散斑图像每个像素点对应的深度值信息，计算公式如下：

其中f是摄像机的焦距，s是基线，d是参考距离，Δx是偏移量，μ为像素大小，dis为深度值。Depth calculation. According to the monocular depth calculation formula of structured light, the depth value information corresponding to each pixel of the infrared speckle image is obtained. The calculation formula is as follows:

where f is the focal length of the camera, s is the baseline, d is the reference distance, Δx is the offset, μ is the pixel size, and dis is the depth value.

后处理。对经过深度计算得到的深度图像进行后处理。对深度图像进行去噪，平滑，和空洞修补。去噪平滑：减少误匹配、边缘空洞带来的噪声，可采用时空3D去噪或采用滑动窗去噪平滑、插值的方式。空洞修补：采用双边滤波器进行空洞修补，作为一种非线性滤波器，能将空间距离考虑进去，不但能去除噪声，还能保持图像的边缘，同时，它对缺失的深度值也能进行填充。post-processing. Post-process the depth image obtained by depth calculation. Denoise, smooth, and inpaint depth images. Denoising and smoothing: To reduce the noise caused by mismatching and edge holes, spatiotemporal 3D denoising or sliding window denoising and smoothing and interpolation can be used. Hole repair: Use bilateral filter for hole repair. As a nonlinear filter, it can take into account the spatial distance, which can not only remove noise, but also maintain the edge of the image. At the same time, it can also fill in the missing depth values. .

点云生成。三维点云生成是将深度图的像素坐标转换到空间坐标。结合P_d＝dis·In_d^-1·p_d其中p_d＝[i_d，j_d，1]^T表示深度图像素点的齐次坐标，In_i表示深度感知器DepthSensor的内参矩阵，dis表示每个点到摄像头的距离值。Point cloud generation. 3D point cloud generation is to convert the pixel coordinates of the depth map to spatial coordinates. Combined with P_d =dis · In_d^-1 · p_d where p_d =[_{id , j d}_, 1]^T represents the homogeneous coordinates of the depth map pixel, In_i represents the internal parameter matrix of the depth sensor DepthSensor, and dis represents The distance value of each point to the camera.

(3)对红外图像进行红外图像增强处理。(3) Infrared image enhancement processing is performed on the infrared image.

如果采用传统的红外图像增强技术，不能很好的对红外图像的噪声进行去除，而且无法很好的保护图像的边缘，轮廓等细节，不能很好实现对红外图像增强的效果。本发明除了可以应用传统红外图像增强处理外，也提供了一种改进的直方图均衡算法，对比起现有技术的图像增强技术，可以使图像的对比度得到很大的提升，并且比较好的对目标，背景，噪声等信号进行调节，突出了目标的灰度级，提升了目标灰度空间，保护了大量细节，使红外图得到很好的增强效果。If the traditional infrared image enhancement technology is used, the noise of the infrared image cannot be removed well, and the edge, outline and other details of the image cannot be well protected, and the effect of infrared image enhancement cannot be well achieved. In addition to applying traditional infrared image enhancement processing, the present invention also provides an improved histogram equalization algorithm. Compared with the prior art image enhancement technology, the contrast of the image can be greatly improved, and the image contrast is better. The target, background, noise and other signals are adjusted to highlight the gray level of the target, improve the gray space of the target, protect a large number of details, and make the infrared image have a good enhancement effect.

具体步骤如下：Specific steps are as follows:

步骤s1：分析加权直方图，剔除零单元，剩下的单元保持前后位置关系不变，构成一个集合{N(i)|1≤i≤K}，K为非零单元个数。Step s1: Analyze the weighted histogram, remove zero units, and keep the positional relationship between the remaining units unchanged to form a set {N(i)|1≤i≤K}, where K is the number of non-zero units.

步骤s2：求出集合的极大值和最大值。具体地，首先判断M(i-1)＜M(i)并且M(i)＞M(i+1)时，(2≤i≤K-1)，得出极大值，接着再分析M(1)与M(K)。Step s2: Find the maximum and maximum values of the set. Specifically, when it is first judged that M(i-1)<M(i) and M(i)>M(i+1), (2≤i≤K-1), the maximum value is obtained, and then M is analyzed again (1) and M(K).

步骤s3：把上一步骤s2中的值构成另一个集合，{N(i)|1≤i≤L}，L为该集合极大值的个数。Step s3: The values in the previous step s2 are formed into another set, {N(i)|1≤i≤L}, where L is the number of maximum values of the set.

步骤s4：接着对步骤s3中得到的集合{N(i)|1≤i≤L}再进行求极大值和最大值，得到一个新的集合{Z(j)|1≤j≤S}，S为这个新集合的个数，并且S＜L。Step s4: Then, perform the maximum and maximum values on the set {N(i)|1≤i≤L} obtained in step s3 to obtain a new set {Z(j)|1≤j≤S} , S is the number of this new set, and S<L.

步骤s5：求出步骤s4得到的集合{Z(j)|1≤j≤S}的中值

平台阈值

Step s5: Find the median of the set {Z(j)|1≤j≤S} obtained in step s4

platform threshold

步骤s6：计算出新加权直方图：Step s6: Calculate a new weighted histogram:

其中P_t(k)是新加权直方图，P_r(k)是原来的直方图。图像灰度值在区间[0，255]，0表示黑色，255表示白色，n_k表示数字图像的灰度级k出现的次数，其中，where P_t (k) is the new weighted histogram and P_r (k) is the original histogram. The gray value of the image is in the interval [0, 255], 0 represents black, 255 represents white, n_k represents the number of occurrences of gray level k of the digital image, where,

P_r(k)＝n_k k＝0，1，2……255P_r (k)=n_k k=0, 1, 2...255

步骤s7：计算累积直方图并得出灰度映射。Step s7: Calculate the cumulative histogram and obtain a grayscale map.

累积直方图计算公式为：The cumulative histogram calculation formula is:

其中F_t(k)是累积直方图。where F_t (k) is the cumulative histogram.

灰度映射计算公式为：

R_t(k)是原有k灰度级对应的新灰度级。The grayscale mapping formula is:

R_t (k) is the new gray level corresponding to the original k gray level.

通过对直方图均衡化算法的改进，可以使图像的对比度得到很大的提升，并且比较好的对目标，背景，噪声等信号进行调节，突出了目标的灰度级，提升了目标灰度空间，保护了大量细节，使红外图像得到很好的增强效果。By improving the histogram equalization algorithm, the contrast of the image can be greatly improved, and the target, background, noise and other signals can be better adjusted, the gray level of the target is highlighted, and the target gray space is improved. , protects a lot of details and makes infrared images well enhanced.

步骤5：将经过图像处理后的RGB图像，红外散斑图像或红外图像进行选择性的输出，可选择输出深度图，RGB图像，红外图像，RGBD数据或者XYZRGB数据。Step 5: Selectively output the image-processed RGB image, infrared speckle image or infrared image, and optionally output depth map, RGB image, infrared image, RGBD data or XYZRGB data.

本发明不仅支持不同种激光源产生的结构光模式，如红外、可见光、紫外线、不可见光等，也可使用不同形状发光元件的投射方案，如圆点状、块状、十字状、条纹状等的。因而在不脱离本发明的精神和范围内的修改和完善，均应包含在上述的权利要求范围内。The present invention not only supports structured light modes generated by different laser sources, such as infrared, visible light, ultraviolet light, invisible light, etc., but also can use projection schemes of light-emitting elements of different shapes, such as dots, blocks, crosses, stripes, etc. of. Therefore, modifications and improvements without departing from the spirit and scope of the present invention should be included in the scope of the above-mentioned claims.

Claims

1. The utility model provides a monocular configuration light depth perception device based on RGBIR camera which characterized in that:

the depth perception device includes:

the RGBIR image generation module is used for projecting an infrared speckle image through an infrared laser speckle coding projector or irradiating the infrared image by utilizing an infrared flood light source and projecting a visible light RGB image through an RGBIR camera;

the RGBIR image acquisition module is used for acquiring the infrared speckle image or the infrared image and the visible light RGB image together through a single RGBIR camera to obtain RGBIR RAW data;

the RGBIR image separation module is used for carrying out image separation on the collected RGBIR RAW data to obtain a separated RGB image, an infrared speckle image or an infrared image;

the RGBIR image processing module is used for processing the RGB image, the infrared speckle image or the infrared image separated by the RGBIR image separation module separately;

the selective output module is used for selectively outputting the RGB images, the infrared speckle images or the infrared images which are processed by the RGBIR image processing module.

2. A monocular structured light depth perception device as described in claim 1, wherein: the image processing of the infrared speckle image comprises preprocessing, image rotation, block matching, depth calculation, post-processing and three-dimensional point cloud generation.

3. A monocular structured light depth perception device as described in claim 1, wherein:

and the selection output of the selection output module further selects and outputs the depth map, the RGBD data or the XYZRGB data.

4. A monocular structured light depth perception device as described in claim 2, wherein:

the image rotation is specifically that when the camera rotates, the image sensor transmits a signal to the rgbiir image processing module, so that the image rotates during image processing, otherwise, the image does not rotate.

5. A monocular structured light depth perception device as described in claim 1, wherein:

the image processing performed on the infrared image is an infrared image enhancement operation, and specifically comprises the following steps:

analyzing the weighted histogram, eliminating zero units, keeping the front-back position relation of the remaining units unchanged, and forming a set { N (i) |1 is more than or equal to i and less than or equal to K }, wherein K is the number of nonzero units;

determining the maximum value and the maximum value of the set N (i);

forming another set by the values in the previous step, { N (i) |1 ≦ i ≦ L }, wherein L is the number of the maximum values of the set;

then carrying out maximum value and maximum value calculation on the set { N (i) |1 ≦ i ≦ L } to obtain a new set { Z (j) |1 ≦ j ≦ S }, wherein S is the number of the new set, and S is less than L;

find the median of the set { Z (j) |1 ≦ j ≦ S ≦

Plateau threshold

Calculating a new weighted histogram:

wherein P is_t(k) Is a new histogram, P_r(k) Is the original histogram, and the number of times of gray level k of the digital image is n_k

P_r(k)＝n_kk＝0，1，2……255

Calculating an accumulated histogram and obtaining gray mapping;

the cumulative histogram calculation formula is:

the gray scale mapping calculation formula is as follows:

rt (k) is a new gray level corresponding to the original k gray level.

6. A monocular structured light depth perception method based on an RGBIR camera is characterized in that:

the monocular structured light depth perception method comprises the following steps:

step 1: the power supply is switched on to drive the power supply to work normally, so that the infrared laser speckle coding projector projects an infrared speckle image or an infrared floodlight source is utilized to irradiate the infrared image, and the RGBIR camera is driven to project an RGB color image;

step 2: collecting the infrared speckle image or the infrared image and the RGB color image by an RGBIR camera according to frames to obtain RGBIR RAW data;

and step 3: performing image separation on the RGBIR RAW data acquired in the step 2;

and 4, step 4: respectively carrying out image processing on the RGB color image, the infrared speckle image or the infrared image obtained by image separation;

and 5: and selectively outputting the RGB color image, the infrared speckle image or the infrared image after image processing.

7. The monocular structured light depth perception method of claim 6, wherein:

and (5) selectively outputting, namely further selecting an output depth map, RGBD data or XYZRGB data.

8. The monocular structured light depth perception method of claim 6, wherein:

the image processing of the infrared speckle image in the step 4 comprises preprocessing, image rotation, block matching, depth calculation, post-processing and three-dimensional point cloud generation.

9. The monocular structured light depth perception method of claim 8, wherein:

the image rotation is specifically that when the camera rotates, the image sensor transmits a signal to the image processing module, so that the image rotates during image processing, otherwise, the image does not rotate.

10. The monocular structured light depth perception method of claim 6, wherein:

the image processing on the infrared image in the step 4 is an infrared image enhancement operation, and specifically comprises the following steps:

step s 1: analyzing the weighted histogram, eliminating zero units, keeping the front-back position relation of the remaining units unchanged, and forming a set { N (i) |1 is more than or equal to i and less than or equal to K }, wherein K is the number of nonzero units;

step s 2: solving the maximum value and the maximum value of the set;

step s 3: forming another set of the values in the previous step s2, { N (i) |1 ≦ i ≦ L }, where L is the number of maxima of the set;

step s 4: then, the maximum value and the maximum value are calculated for { N (i) |1 ≦ i ≦ L } to obtain a new set { Z (j) |1 ≦ j ≦ S }, S is the number of the new set, and S is less than L;

step s 5: find the median of the set { Z (j) |1 ≦ j ≦ S ≦

Plateau threshold

Step s 6: calculating a new weighted histogram:

wherein P is_t(k) Is a new histogram, P_r(k) Is the original histogram, n_kRepresenting the number of occurrences of a gray level k, where P is_r(k)＝n_kk＝0，1，2……255

Step s 7: calculating an accumulated histogram and obtaining gray mapping;

the cumulative histogram calculation formula is:

the gray scale mapping calculation formula is as follows:

R_t(k) is a new gray level corresponding to the original k gray level.