CN101902657A

Movatterモバイル変換

Info

Publication number: CN101902657A
Application number: CN 201010228696
Authority: CN
Inventors: 席明; 薛玖飞; 王梁昊; 李东晓; 张明
Original assignee: Zhejiang University ZJU
Current assignee: Wanwei Display Technology Shenzhen Co ltd
Priority date: 2010-07-16
Filing date: 2010-07-16
Publication date: 2010-12-01
Anticipated expiration: 2030-07-16
Also published as: CN101902657B

Abstract

本发明公开了一种基于深度图分层的虚拟多视点图像的生成方法，包括如下步骤：(1)对待处理的深度图像进行预处理；(2)对预处理后的深度图进行分层，得到分层深度图像；(3)选定相机阵列聚焦的深度层，确定前景层和背景层；(4)对待处理的二维图像进行分层，得到分层二维图像；(5)计算各个深度层对应二维图像层对应的视差值；(6)对分层二维图像进行扩展，得到扩展后的分层二维图像；(7)通过加权水平平移算法得到各个虚拟视点位置的虚拟二维图像。本发明不需要虚拟多视点相机阵列模型的参数，可以快速有效的生成多视点自由立体显示系统所需的虚拟多视点图像，并对输入深度图像具有较好的容错能力。The invention discloses a method for generating a virtual multi-viewpoint image based on depth map layering, comprising the following steps: (1) preprocessing the depth image to be processed; (2) layering the preprocessed depth map, Obtain a layered depth image; (3) select the depth layer focused by the camera array, and determine the foreground layer and the background layer; (4) layer the two-dimensional image to be processed to obtain a layered two-dimensional image; (5) calculate each The depth layer corresponds to the parallax value corresponding to the two-dimensional image layer; (6) expand the layered two-dimensional image to obtain the expanded layered two-dimensional image; (7) obtain the virtual position of each virtual viewpoint through a weighted horizontal translation algorithm 2D image. The invention does not need the parameters of the virtual multi-viewpoint camera array model, can quickly and effectively generate virtual multi-viewpoint images required by a multi-viewpoint autostereoscopic display system, and has better fault-tolerant capability for input depth images.

Description

Translated fromChinese

一种基于深度图分层的虚拟多视点图像的生成方法A method for generating virtual multi-viewpoint images based on depth map layering

技术领域technical field

本发明涉及多视点自由立体显示系统中虚拟多视点图像的生成方法，尤其涉及一种基于深度图分层的虚拟多视点图像的生成方法。The invention relates to a method for generating virtual multi-viewpoint images in a multi-viewpoint autostereoscopic display system, in particular to a method for generating virtual multi-viewpoint images based on depth map layers.

背景技术Background technique

多视点自由立体显示技术是一种将同一场景的多个不同视点的图像同时呈献给观众，观众不需要佩戴眼镜即可在多个位置看到立体图像的技术。该技术需要多幅同一场景不同视点的二维图像。Multi-viewpoint autostereoscopic display technology is a technology that simultaneously presents images of multiple different viewpoints of the same scene to the audience, and the audience can see stereoscopic images in multiple positions without wearing glasses. The technique requires multiple 2D images of the same scene from different viewpoints.

为了获取多幅同一场景不同视点的二维图像，现在主要有两种解决办法。In order to obtain multiple 2D images of the same scene from different viewpoints, there are currently two main solutions.

一种办法是在拍摄时采用多个摄相机组成的相机阵列同时进行拍摄，然后将多幅二维图像数据同时传输到显示端进行立体合成显示。这种方法摄像机阵列成本高昂，而且多幅二维图像数据的传输需要占用大量的带宽资源，无法在现有网络上传输。另一种方法是在拍摄时采用双摄像机拍摄，经过立体匹配求取深度图的处理后，传输一路二维图像加上一路深度图像，然后在显示端重构出多幅虚拟视点的二维图像，最后进行立体合成显示。这种方法拍摄成本较低，视频传输增加的带宽很少，可以直接应用现有网络进行传输，而且可以根据自由立体显示器的需要生成不同数目的虚拟视点二维图像，灵活性更强。因此，对基于二维图像和深度图像生成多幅虚拟视点图像这一技术的研究具有更加现实的意义，这一技术的研究不仅可以为立体显示提供丰富的素材，而且能大大节省内容制作的成本。One method is to use a camera array composed of multiple cameras to shoot simultaneously, and then simultaneously transmit multiple pieces of two-dimensional image data to the display terminal for stereoscopic composite display. The cost of the camera array in this method is high, and the transmission of multiple two-dimensional image data needs to occupy a large amount of bandwidth resources, which cannot be transmitted on the existing network. Another method is to use dual cameras to shoot, after stereo matching to obtain the depth map, transmit one 2D image plus one depth image, and then reconstruct multiple 2D images of virtual viewpoints on the display side , and finally perform stereocompositing display. This method has low shooting cost, little bandwidth increase in video transmission, can be directly applied to the existing network for transmission, and can generate two-dimensional images of different numbers of virtual viewpoints according to the needs of the autostereoscopic display, which is more flexible. Therefore, the research on the technology of generating multiple virtual viewpoint images based on two-dimensional images and depth images has more practical significance. The research on this technology can not only provide rich materials for stereoscopic display, but also greatly save the cost of content production .

基于二维图像和深度图像生成多幅虚拟视点图像技术现在采用的主流算法是DIBR算法(Depth-Image-Based Rendering)，其主要原理是根据输入的深度图像和相机参数将输入的二维图像中的每一个像素点映射到三维空间中的一点，然后再根据虚拟相机阵列模型的参数将这些空间中的点映射到虚拟相机成像平面上，得到虚拟视点的二维图像。这种方法需要做复杂的映射操作，运算量大，耗时久，对深度数据的准确性要求很高，而且相机阵列模型的参数不易获得。Based on two-dimensional images and depth images to generate multiple virtual viewpoint images, the mainstream algorithm currently used is the DIBR algorithm (Depth-Image-Based Rendering). Each pixel of the camera is mapped to a point in the three-dimensional space, and then these points in the space are mapped to the imaging plane of the virtual camera according to the parameters of the virtual camera array model to obtain a two-dimensional image of the virtual viewpoint. This method requires complex mapping operations, which requires a large amount of calculation, takes a long time, and requires high accuracy of depth data, and the parameters of the camera array model are not easy to obtain.

发明内容Contents of the invention

本发明的目的是克服现有技术的缺陷和不足，提出了一种基于深度图分层的虚拟多视点图像的生成方法。The purpose of the present invention is to overcome the defects and deficiencies of the prior art, and propose a method for generating virtual multi-viewpoint images based on depth map layers.

基于深度图分层的虚拟多视点图像的生成方法包括以下步骤：The generation method of the virtual multi-viewpoint image based on depth map layering comprises the following steps:

(1)对待处理的参考视点深度图像进行中值滤波操作，并在水平方向上对深度值小的物体进行轮廓扩展操作，得到滤波扩展后的深度图像；(1) Carry out a median filter operation on the depth image of the reference viewpoint to be processed, and perform a contour extension operation on objects with small depth values in the horizontal direction to obtain a depth image after filtering;

(2)根据设定的分层层数，对滤波扩展后的深度图像进行分层操作，得到分层深度图像；(2) According to the set number of layered layers, perform a layered operation on the depth image after filtering and expansion to obtain a layered depth image;

(3)在分层深度图像中选定一个深度层作为相机阵列的聚焦层，则深度值比聚焦层深度值大的为背景层，深度值比聚焦层深度值小的为前景层；(3) Select a depth layer in the layered depth image as the focus layer of the camera array, then the background layer whose depth value is larger than the focus layer depth value, and the foreground layer whose depth value is smaller than the focus layer depth value;

(4)对待处理的参考视点二维图像，根据二维图像相对应的分层深度图像，对二维图像进行分层操作，得到分层二维图像；(4) For the reference viewpoint 2D image to be processed, perform layering operations on the 2D image according to the layered depth image corresponding to the 2D image to obtain a layered 2D image;

(5)根据设定的参考视点前景层和其相邻视点前景层之间的最大视差值，计算虚拟相机阵列相邻视点前景层与背景层之间的最大视差值，并结合各个深度层到聚焦层的距离计算各个深度层对应二维图像层对应的视差值；(5) According to the maximum disparity value between the set reference viewpoint foreground layer and its adjacent viewpoint foreground layer, calculate the maximum disparity value between the virtual camera array adjacent viewpoint foreground layer and background layer, and combine each depth Calculate the disparity value corresponding to the two-dimensional image layer corresponding to each depth layer from the distance from the layer to the focus layer;

(6)根据虚拟视点位置到参考视点位置的相对距离和相机阵列相邻视点前景层与背景层之间的最大视差值，对分层二维图像进行扩展操作，得到扩展后的分层二维图像；(6) According to the relative distance from the virtual viewpoint position to the reference viewpoint position and the maximum disparity value between the foreground layer and the background layer of adjacent viewpoints of the camera array, the layered two-dimensional image is extended to obtain the extended layered two-dimensional image. dimensional image;

(7)根据虚拟视点位置到参考视点位置的相对距离和各个二维图像层对应的视差值，通过加权水平平移算法得到各个虚拟视点位置的虚拟二维图像。(7) According to the relative distance from the virtual viewpoint position to the reference viewpoint position and the disparity value corresponding to each two-dimensional image layer, the virtual two-dimensional image of each virtual viewpoint position is obtained through a weighted horizontal translation algorithm.

所述的对待处理的参考视点深度图像进行中值滤波操作，并在水平方向上对深度值较小的物体进行扩展操作，得到滤波扩展后的深度图像步骤为：The step of performing a median filtering operation on the depth image of the reference viewpoint to be processed, and performing an expansion operation on objects with smaller depth values in the horizontal direction, to obtain the depth image after filtering and expansion is as follows:

(a)对于参考视点深度图像I_Depth(x，y)进行5×5窗口大小的中值滤波，得到平滑的深度图I′_Depth(x，y)，x表示图像像素点水平方向上的坐标，y表示图像像素点垂直方向上的坐标，x的取值为x＝0，1，2，...，H-1，y的取值为y＝0，1，2，...，V-1，H表示图像的水平分辨率，V表示图像的垂直分辨率；(a) For the reference viewpoint depth image I_Depth (x, y), perform a median filter with a window size of 5×5 to obtain a smooth depth map I'_{Depth (} x, y), where x represents the coordinates of the image pixel in the horizontal direction , y represents the coordinates in the vertical direction of the image pixel point, the value of x is x=0, 1, 2, ..., H-1, the value of y is y = 0, 1, 2, ..., V-1, H represents the horizontal resolution of the image, and V represents the vertical resolution of the image;

(b)对于步骤(a)得到的平滑深度图I′_Depth(x，y)沿水平方向进行扫描，计算水平方向上相邻两个像素点之间深度差值的绝对值，并将该绝对值与一个预先设定的阈值th进行比较，确定深度值较小的物体水平方向轮廓扩展的方向dir(x，y)，-1表示水平向左扩展，1表示水平向右扩展，0表示不做扩展，表示公式如下：(b) Scan the smooth depth map I′_Depth (x, y) obtained in step (a) along the horizontal direction, calculate the absolute value of the depth difference between two adjacent pixels in the horizontal direction, and convert the absolute value The value is compared with a preset threshold value th to determine the direction dir(x, y) of the horizontal outline extension of the object with a smaller depth value, -1 means horizontal expansion to the left, 1 means horizontal expansion to the right, and 0 means no To expand, the expression formula is as follows:

$dir dir ((x x,, y the y)) = = \{\begin{matrix} - - 11 & | | {I I}_{Depth Depth}^{' '} ((x x,, y the y)) - - {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) | | > > thand more than {I I}_{Depth Depth}^{' '} ((x x,, y the y)) > > {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) \\ 11 & | | {I I}_{Depth Depth}^{' '} ((x x,, y the y)) - - {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) | | > > thand more than {I I}_{Depth Depth}^{' '} ((x x,, y the y)) < < {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) & ;; \\ 00 & otherwise otherwise \end{matrix}$

(c)将每一个像素点的赋值标志位flag(x，y)初始化为零，表示公式如下：(c) Initialize the assignment flag flag (x, y) of each pixel to zero, and the expression formula is as follows:

$flag flag ((x x,, y the y)) = = 00,, &ForAll; &ForAll; x x = = 0,1,2 0,1,2,, . . . . . .,, H h - - 11 &ForAll; &ForAll; y the y = = 0,1,2 0,1,2,, . . . . . .,, V V - - 11$

其中表示任意给定；in means any given;

(d)根据步骤(b)得到的每个像素的扩展方向dir(x，y)对步骤(a)得到的平滑深度图I′_Depth(x，y)中待扩展的像素点进行赋值操作，得到深度图I″_Depth(x，y)，并记录对应赋值标志位flag(x，y)，表示公式如下：(d) According to the extension direction dir(x, y) of each pixel obtained in step (b), assign values to the pixels to be extended in the smooth depth map I′_Depth (x, y) obtained in step (a), Get the depth map I″_Depth (x, y), and record the corresponding assignment flag bit flag (x, y), the expression formula is as follows:

${I I}_{Depth Depth}^{' '' '} ((x x + + dir dir ((x x,, y the y)) \cdot \cdot i i,, y the y)) = = \{\begin{matrix} {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) & dir dir ((x x,, y the y)) = = - - 11 \\ {I I}_{Depth Depth}^{' '} ((x x,, y the y)) & dir dir ((x x,, y the y)) = = 11 \end{matrix},, i i = = 0,1,2 0,1,2,, . . . . . .,, K K - - 11$

flag(x+dir(x，y)·i，y)＝1|dir(x，y)|＝1，i＝0，1，2，...，K-1flag(x+dir(x, y)·i, y)=1|dir(x, y)|=1, i=0, 1, 2, ..., K-1

其中，K表示扩展像素的个数；Among them, K represents the number of expanded pixels;

(e)对步骤(d)得到的深度图I″_Depth(x，y)中对应赋值标志位flag(x，y)＝0的像素点进行赋值操作，得到扩展深度图

表示公式如下：(e) carry out the assignment operation to the pixel corresponding to the assignment flag flag (x, y)=0 in the depth map I″_Depth (x, y) that step (d) obtains, obtain the extended depth map

The expression formula is as follows:

${I I}_{Depth Depth}^{E E.} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Depth Depth}^{' '} ((x x,, y the y)) & flag flag ((x x,, y the y)) = = 00 \\ {I I}_{Depth Depth}^{' '' '} ((x x,, y the y)) & flag flag ((x x,, y the y)) = = 11 \end{matrix} . .$

所述的根据设定的分层层数，对滤波扩展后的深度图像进行分层操作，得到分层深度图像步骤为：According to the set number of layered layers, the layered operation is performed on the filtered and expanded depth image to obtain the layered depth image. The steps are:

(f)根据设定的分层层数N，N为大于等于1的整数，计算步骤(e)得到的扩展深度图

中的每一个像素点对应深度层的索引值Index(x，y)，表示公式如下：(f) According to the set number of layers N, where N is an integer greater than or equal to 1, calculate the extended depth map obtained in step (e)

Each pixel in corresponds to the index value Index(x, y) of the depth layer, and the expression formula is as follows:

其中，

表示取小于等于·的最大整数的操作；in,

Indicates the operation of taking the largest integer less than or equal to ;

(g)根据步骤(f)得到的像素点深度层索引值Index(x，y)，得到每一层的分层深度图

分层深度图

的索引值index越小表示该深度层越靠近观看者，索引值index越大表示该深度层越远离观看者，表示公式如下：(g) According to the pixel depth layer index value Index(x, y) obtained in step (f), obtain the layered depth map of each layer

layered depth map

The smaller the index value of , the closer the depth layer is to the viewer, and the larger the index value is, the farther the depth layer is from the viewer. The expression formula is as follows:

${I I}_{Depth Depth}^{index index} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Depth Depth}^{E E.} ((x x,, y the y)) & Index Index ((x x,, y the y)) = = index index \\ 255255 & otherwise otherwise \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11 . .$

所述的在分层深度图像中选定一个深度层作为相机阵列的聚焦层，则深度值比聚焦层深度值大的为背景层，深度值比聚焦层深度值小的为前景层步骤为：In the described layered depth image, a depth layer is selected as the focus layer of the camera array, then the depth value is larger than the focus layer depth value as the background layer, and the depth value is smaller than the focus layer depth value as the foreground layer. The steps are:

(h)根据设定的相机阵列聚焦层focus，确定前景图层和背景图层，索引值index＞focus的深度层和聚焦层相比，距离观看者远，为背景层，索引值index＜focus的深度层和聚焦层相比，距离观看者近，为前景图层。(h) Determine the foreground layer and background layer according to the set camera array focus layer focus, the depth layer with index value index>focus is farther away from the viewer than the focus layer, and is the background layer, and the index value index<focus Compared with the focus layer, the depth layer is closer to the viewer and is the foreground layer.

所述的对待处理的参考视点二维图像，根据二维图像相对应的分层深度图像，对二维图像进行分层操作，得到分层二维图像步骤为：For the reference viewpoint 2D image to be processed, according to the layered depth image corresponding to the 2D image, the layered operation is performed on the 2D image to obtain the layered 2D image. The steps are as follows:

(i)根据步骤(f)得到的像素点深度层索引值Index(x，y)，对于参考视点二维图像I_Color(x，y)进行分层操作，得到每一层的分层二维图像

同时记录每一层二维图像

的赋值标志位

分层二维图像

的索引值index越小表示该二维图像层越靠近观看者，索引值index越大表示该二维图像层越远离观看者，表示公式如下：(i) According to the pixel depth layer index value Index(x, y) obtained in step (f), perform layering operation on the reference viewpoint two-dimensional image I_Color (x, y) to obtain the layered two-dimensional image of each layer image

Simultaneously record each layer of 2D image

The assignment flag

layered 2D image

The smaller the index value index of , the closer the 2D image layer is to the viewer, and the larger the index value index is, the farther the 2D image layer is from the viewer. The expression formula is as follows:

${I I}_{Color color}^{index index} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Color color} ((x x,, y the y)) & Index Index ((x x,, y the y)) = = index index \\ 00 & otherwise otherwise \end{matrix},index, index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11$

${F f}_{Color color}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 11 & Index Index ((x x,, y the y)) = = index index \\ 00 & otherwise otherwise \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11 . .$

所述的根据设定的参考视点前景层和其相邻视点前景层之间的最大视差值，计算虚拟相机阵列相邻视点前景层与背景层之间的最大视差值，并结合各个深度层到聚焦层的距离计算各个深度层对应二维图像层对应的视差值步骤为：According to the maximum parallax value between the set reference viewpoint foreground layer and its adjacent viewpoint foreground layer, calculate the maximum parallax value between the adjacent viewpoint foreground layer and the background layer of the virtual camera array, and combine each depth The steps to calculate the disparity value corresponding to each depth layer corresponding to the two-dimensional image layer are:

(j)根据设定的参考视点前景层和其相邻视点前景层之间的最大视差值MaxD，MaxD为大于等于1的整数，计算相机阵列相邻视点前景层与背景层之间的最大视差值MaxR，表示公式如下：(j) According to the maximum parallax value MaxD between the set reference viewpoint foreground layer and its adjacent viewpoint foreground layer, MaxD is an integer greater than or equal to 1, calculate the maximum disparity between the adjacent viewpoint foreground layer and the background layer of the camera array Parallax value MaxR, the expression formula is as follows:

MaxR＝(MaxD/focus)·(N-1)；MaxR=(MaxD/focus)·(N-1);

(k)根据步骤(j)得到的最大视差值MaxR，结合各个深度层到聚焦层的距离计算各个深度层对应二维图像层对应的视差值Dis^index，表示公式如下：(k) According to the maximum parallax value MaxR obtained in step (j), combine the distance from each depth layer to the focus layer to calculate the parallax value Dis^index corresponding to each depth layer corresponding to the two-dimensional image layer, and the expression formula is as follows:

所述的根据虚拟视点位置到参考视点位置的相对距离和相机阵列相邻视点前景层与背景层之间的最大视差值，对分层二维图像进行扩展操作，得到扩展后的分层二维图像步骤为：According to the relative distance from the virtual viewpoint position to the reference viewpoint position and the maximum parallax value between the foreground layer and the background layer of adjacent viewpoints of the camera array, the layered two-dimensional image is extended to obtain the extended layered two-dimensional image. Dimensional image steps are:

(1)设定虚拟平行多视点阵列中虚拟视点个数为M，M为大于等于2的整数，以参考视点作为虚拟平行多视点阵列的中心视点，即虚拟平行多视点阵列最中间位置的视点，令参考视点的索引值为零，则在水平方向上按从右向左的顺序，各个虚拟视点的索引值sub_i的取值依次为：(1) Set the number of virtual viewpoints in the virtual parallel multi-viewpoint array as M, M is an integer greater than or equal to 2, and use the reference viewpoint as the central viewpoint of the virtual parallel multi-viewpoint array, that is, the viewpoint at the middlemost position of the virtual parallel multi-viewpoint array , let the index value of the reference viewpoint be zero, then in the order from right to left in the horizontal direction, the values of the index value sub_i of each virtual viewpoint are as follows:

(m)根据步骤(1)得到的各个虚拟视点的索引值sub_i计算虚拟视点位置到参考视点位置的相对距离D_{sub_i}＝Sub_i-0＝sub_i；(m) Calculate the relative distance D_{sub_i} = Sub_i -0 = sub_i from the virtual view point position to the reference view point position according to the index value sub_i of each virtual view point obtained in step (1);

(n)对步骤(i)得到的每一层二维图像的赋值标志位

沿水平方向进行扫描，计算相邻两个赋值标志位差的绝对值，将该绝对值和0进行比较，判断二维图像层

水平扩展的方向

-1表示水平向左扩展，1表示水平向右扩展，0表示不做扩展，表示公式如下：(n) for each layer of two-dimensional image obtained in step (i) The assignment flag

Scan along the horizontal direction, calculate the absolute value of the bit difference between two adjacent assignment marks, compare the absolute value with 0, and judge the two-dimensional image layer

Direction of horizontal expansion

-1 means to expand horizontally to the left, 1 means to expand horizontally to the right, and 0 means not to expand. The expression formula is as follows:

${dir dir}_{Color color}^{index index} ((x x,, y the y)) = = \{\begin{matrix} - - 11,, & | | {F f}_{Color color}^{index index} ((x x,, y the y)) - - {F f}_{Color color}^{index index} ((x x + + 11,, y the y)) | | > > 00 and and {F f}_{Color color}^{index index} ((x x,, y the y)) < < {F f}_{Color color}^{index index} ((x x + + 11,, y the y)) \\ 11,, & | | {F f}_{Color color}^{index index} ((x x,, y the y)) - - {F f}_{Color color}^{index index} ((x x + + 11,, y the y)) | | > > 00 and and {F f}_{Color color}^{index index} ((x x,, y the y)) > > {F f}_{Color color}^{index index} ((x x + + 11,, y the y)) \\ 00,, & otherwise otherwise \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11;;$

(o)将每一层扩展二维图像的赋值标志位

初始化为零，表示公式如下：(o) Extend the assignment flag of each layer of two-dimensional image

Initialized to zero, the representation formula is as follows:

${F f}_{EC EC}^{index index} ((x x,, y the y)) = = 00,, &ForAll; &ForAll; x x = = 0,1,2 0,1,2,, . . . . . .,, H h - - 11 &ForAll; &ForAll; y the y = = 0,1,2 0,1,2,, . . . . . .,, V V - - 11;;$

(p)根据步骤(n)得到的水平扩展方向

对步骤(i)得到的每一层二维图像

待扩展的像素点进行赋值操作，得到二维图像层

同时记录赋值标志位

表示公式如下：(p) Horizontal expansion direction obtained according to step (n)

For each layer of two-dimensional image obtained in step (i)

The pixels to be expanded are assigned to obtain a two-dimensional image layer

Simultaneous record assignment flag bit

The expression formula is as follows:

${I I}_{EC EC}^{index index} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot &Center Dot; j j,, y the y)) = = \{\begin{matrix} {I I}_{Color color} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot &Center Dot; j j,, y the y)) & {dir dir}_{Color color}^{index index} ((x x,, y the y)) = = - - 11 \\ {I I}_{Color color} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot &Center Dot; j j,, y the y)) & {dir dir}_{Color color}^{index index} ((x x,, y the y)) = = 11 \end{matrix},, j j = = 0,1,2 0,1,2,, . . . . . .,, L L - - 11$

${F f}_{EC EC}^{index index} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot &Center Dot; j j,, y the y)) = = 11 | | {dir dir}_{Color color}^{index index} ((x x,, y the y)) | | = = 11,, j j = = 0,1,2 0,1,2,, . . . . . .,, L L - - 11$

其中，L表示二维图像层

扩展像素点的个数，L＝D_sub·MaxR；Among them, L represents the two-dimensional image layer

The number of expanded pixels, L=D_sub MaxR;

(q)对步骤(p)得到的二维图像层

对应赋值标志位

的像素点进行赋值操作，得到最终的扩展二维图像层

并记录最终的二维图像层赋值标志位表示公式如下：(q) for the two-dimensional image layer obtained in step (p)

Corresponding assignment flag bit

The pixel points are assigned to obtain the final extended two-dimensional image layer

And record the final two-dimensional image layer assignment flag bit The expression formula is as follows:

${I I}_{ExColor ExColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Color color}^{index index} ((x x,, y the y)) & {F f}_{EC EC}^{index index} ((x x,, y the y)) = = 00 \\ {I I}_{EC EC}^{index index} ((x x,, y the y)) & {F f}_{EC EC}^{index index} ((x x,, y the y)) = = 11 \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11$

${F f}_{ExColor ExColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 00 & {F f}_{EC EC}^{index index} ((x x,, y the y)) = = 00 and and {F f}_{Color color}^{index index} ((x x,, y the y)) = = 00 \\ 11 & {F f}_{Color color}^{index index} ((x x,, y the y)) = = 11 \\ 22 & {F f}_{EC EC}^{index index} ((x x,, y the y)) = = 11 and and {F f}_{Color color}^{index index} ((x x,, y the y)) = = 00 \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11 . .$

所述的根据虚拟视点位置到参考视点位置的相对距离和各个二维图像层对应的视差值，通过加权水平平移算法得到各个虚拟视点位置的虚拟二维图像步骤为：According to the relative distance from the virtual viewpoint position to the reference viewpoint position and the parallax value corresponding to each two-dimensional image layer, the step of obtaining the virtual two-dimensional image of each virtual viewpoint position through a weighted horizontal translation algorithm is as follows:

(r)假设要生成M个虚拟视点位置中第i个位置的图像，按照步骤(m)得到的虚拟视点位置到参考视点位置的相对距离D_{sub_i}和步骤(k)得到的各个二维图像层对应的视差值Dis^index，对步骤(q)得到的扩展二维图像层

和赋值标志位

进行加权水平平移操作，得到平移后的二维图像层和平移后的赋值标志位

表示公式如下：(r) Assuming that the image of the i-th position among the M virtual viewpoint positions is to be generated, the relative distance D_{sub_i} from the virtual viewpoint position to the reference viewpoint position obtained in step (m) and each two-dimensional image layer obtained in step (k) Corresponding disparity value Dis^index , for the extended two-dimensional image layer obtained in step (q)

and assignment flags

Perform a weighted horizontal translation operation to obtain the translated two-dimensional image layer and shifted assignment flags

The expression formula is as follows:

${I I}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 00 & ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) < < 00 or or ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) &GreaterEqual; &Greater Equal; H h \\ {I I}_{ExColor ExColor}^{index index} ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i},, y the y)) & 00 \leq \leq ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) < < H h \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11$

${F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 00 & ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) < < 00 or or ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) &GreaterEqual; &Greater Equal; H h \\ {F f}_{ExColor ExColor}^{index index} ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i},, y the y)) & 00 \leq \leq ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) < < H h \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11$

其中，H表示图像的水平分辨率；Among them, H represents the horizontal resolution of the image;

(s)初始化虚拟视点位置i二维图像的赋值标志位

为零，表示公式如下；(s) Initialize the assignment flag bit of the virtual viewpoint position i two-dimensional image

is zero, the expression formula is as follows;

${F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00,, &ForAll; &ForAll; x x = = 0,1,2 0,1,2,, . . . . . .,, H h - - 11 &ForAll; &ForAll; y the y = = 0,1,2 0,1,2,, . . . . . .,, V V - - 11;;$

(t)将步骤(r)中得到的平移后的二维图像层按照先放背景后放前景，由远及近的顺序，即index从大到小的顺序，一层一层地叠加起来，每叠加一层二维图像层，更新一次虚拟视点位置i二维图像的赋值标志位叠加操作结束后根据最终的赋值标志位

得到虚拟视点位置i的二维图像

表示公式如下：(t) the translated two-dimensional image layer obtained in step (r) Put the background first and then the foreground, in the order from far to near, that is, in the order of index from large to small, superimpose layer by layer, and update the virtual viewpoint position i two-dimensional image every time a layer of two-dimensional image is superimposed The assignment flag After the superposition operation ends, according to the final assignment flag

Get the two-dimensional image of the virtual viewpoint position i

The expression formula is as follows:

${F f}_{vir vir}^{i i} ((x x,, y the y)) = = \{\begin{matrix} 00 & {F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00 and and {F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = 00 \\ index index + + 11 & {F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = 11 \\ index index + + 11 + + N N & {F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00 and and {F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = 22 \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11$

${I I}_{vir vir}^{i i} ((x x,, y the y)) = = \{\begin{matrix} 00 & {F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00 \\ {I I}_{HSColor HSColor}^{{F f}_{vir vir}^{i i} ((x x,, y the y)) - - 11} ((x x,, y the y)) & {F f}_{vir vir}^{i i} ((x x,, y the y)) > > 00 and and {F f}_{vir vir}^{i i} ((x x,, y the y)) - - N N \leq \leq 00 \\ {I I}_{HSColor HSColor}^{{F f}_{vir vir}^{i i} ((x x,, y the y)) - - 11 - - N N} ((x x,, y the y)) & {F f}_{vir vir}^{i i} ((x x,, y the y)) - - N N > > 00 \end{matrix} index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11;;$

(u)对M个虚拟视点位置依次进行步骤(r)至步骤(t)的操作，即可得到M幅虚拟视点位置的二维图像。(u) Step (r) to step (t) are sequentially performed on the M virtual viewpoint positions to obtain M two-dimensional images of the virtual viewpoint positions.

本发明适用于对已有一路二维图像序列和一路深度图像序列的数据产生多个虚拟视点，以往采用的DIBR算法运算量大，耗时久，对深度图像的质量要求很高，而且相机阵列模型的参数不易获得。本方法不需要已知或者标定出虚拟多视点相机阵列模型的参数，可以快速有效的生成多视点自由立体显示系统所需的虚拟多视点图像，此外，为了提高对输入深度数据的容错能力，对深度图进行了分层处理，降低了对输入深度图图像质量的要求，扩大了本方法的适用范围。The present invention is suitable for generating multiple virtual viewpoints on the data of an existing two-dimensional image sequence and a depth image sequence. The DIBR algorithm used in the past has a large amount of calculation, takes a long time, and has high requirements on the quality of the depth image, and the camera array The parameters of the model are not easy to obtain. This method does not need to know or calibrate the parameters of the virtual multi-view camera array model, and can quickly and effectively generate the virtual multi-view images required by the multi-view autostereoscopic display system. In addition, in order to improve the fault tolerance of the input depth data, the The depth map is processed in layers, which reduces the requirements on the image quality of the input depth map and expands the scope of application of the method.

附图说明Description of drawings

图1是基于深度分层生成虚拟多视点图像的流程框图；Fig. 1 is a flowchart of generating a virtual multi-viewpoint image based on depth layering;

图2(a)是Racket二维图像测试序列的视频截图；Figure 2(a) is a video screenshot of the Racket two-dimensional image test sequence;

图2(b)是Racket深度图像测试序列的视频截图；Figure 2(b) is a video screenshot of the Racket depth image test sequence;

图3是图2(a)二维图像视频截图对应的预处理后的深度图像；Fig. 3 is the preprocessed depth image corresponding to the screenshot of the two-dimensional image video in Fig. 2(a);

图4(a)是图3预处理后深度图像经过分层处理后的第0层深度图像；Fig. 4(a) is the 0th layer depth image after the depth image is preprocessed in Fig. 3 after layered processing;

图4(b)是图2(b)深度图像视频截图经过分层处理后的第0层二维图像；Fig. 4(b) is the 0th layer two-dimensional image after the layered processing of the depth image video screenshot of Fig. 2(b);

图5是图4(b)第0层二维图像对应的第0层扩展后的二维图像；Fig. 5 is the expanded two-dimensional image of the 0th layer corresponding to the 0th layer two-dimensional image of Fig. 4 (b);

图6是最终生成的虚拟九视点二维图像。Figure 6 is the final generated virtual nine-viewpoint two-dimensional image.

具体实施方式Detailed ways

基于深度图分层的虚拟多视点图像的生成方法包括以下步骤(整体流程图如图1所示)：The generation method of the virtual multi-viewpoint image based on depth map layering comprises the following steps (the overall flow chart is as shown in Figure 1):

所述的对待处理的参考视点深度图像进行中值滤波操作，并在水平方向上对深度值小的物体进行扩展操作，得到滤波扩展后的深度图像步骤为：The steps of performing a median filtering operation on the reference viewpoint depth image to be processed, and performing an expansion operation on objects with small depth values in the horizontal direction, to obtain a filtered and expanded depth image are as follows:

(a)对于参考视点深度图像I_Depth(x，y)进行5×5窗口大小的中值滤波，得到平滑的深度图I′_Depth(x，y)，x表示图像像素点水平方向上的坐标，y表示图像像素点垂直方向上的坐标，x的取值为x＝0，1，2，...，H-1，y的取值为y＝0，1，2，...，V-1，H表示图像的水平分辨率，V表示图像的垂直分辨率；(a) For the reference viewpoint depth image I_Depth (x, y), perform a median filter with a window size of 5×5 to obtain a smooth depth map I'_Depth (x, y), where x represents the coordinates of the image pixel in the horizontal direction , y represents the coordinates in the vertical direction of the image pixel point, the value of x is x=0, 1, 2, ..., H-1, the value of y is y = 0, 1, 2, ..., V-1, H represents the horizontal resolution of the image, and V represents the vertical resolution of the image;

中值滤波有两个目的：一个目的是去除参考视点深度图像中孤立的像素点，这些孤立点大部分是深度值错误的像素点，这样可以去除一部分错误；另一个目的是使输入的深度图更为平滑，有助于提高最后生成的虚拟视点二维图像的质量。图像的中值滤波一般采用一个含有奇数个点的滑动窗口，用窗口中各点灰度值的中值代替窗口中点的灰度值，本专利中采用5×5大小的窗口。Median filtering has two purposes: one purpose is to remove isolated pixels in the depth image of the reference viewpoint, most of these isolated points are pixels with wrong depth values, so that part of the error can be removed; the other purpose is to make the input depth map It is smoother and helps to improve the quality of the final generated 2D image of the virtual viewpoint. The median filter of the image generally adopts a sliding window containing an odd number of points, and replaces the gray value of the middle point of the window with the median value of the gray value of each point in the window. In this patent, a window with a size of 5×5 is used.

$dir dir ((x x,, y the y)) = = \{\begin{matrix} - - 11 & | | {I I}_{Depth Depth}^{' '} ((x x,, y the y)) - - {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) | | > > thand more than {I I}_{Depth Depth}^{' '} ((x x,, y the y)) > > {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) \\ 11 & | | {I I}_{Depth Depth}^{' '} ((x x,, y the y)) - - {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) | | > > thand more than {I I}_{Depth Depth}^{' '} ((x x,, y the y)) < < {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) \\ 00 & otherwise otherwise \end{matrix};;$

(c)将每一个像素点的赋值标志位flag(x，y)初始化为零，表示公式如下：(c) Initialize the assignment flag (x, y) of each pixel to zero, and the expression formula is as follows:

其中

表示任意给定；in

means any given;

${I I}_{Depth Depth}^{' '' '} ((x x + + dir dir ((x x,, y the y)) \cdot \cdot i i,, y the y)) = = \{\begin{matrix} {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) & dir dir ((x x,, y the y)) = = - - 11 \\ {I I}_{Depth Depth}^{' '} ((x x,, y the y)) & dir dir ((x x,, y the y)) = = 11 \end{matrix},, i i = = 0,1,2 0,1,2 . . . . . .,, K K - - 11$

The expression formula is as follows:

由于深度图像中物体边缘轮廓不够准确，传统的DIBR(Depth-Image-BasedRendering)算法生成虚拟视点图像时，经常会发生前景物体被割裂在空洞区域两边的现象，对深度图像中前景物体轮廓进行扩展操作可以有效地避免该现象。Due to the inaccuracy of the object edge outline in the depth image, when the traditional DIBR (Depth-Image-BasedRendering) algorithm generates a virtual viewpoint image, the phenomenon that the foreground object is often split on both sides of the hollow area, the foreground object outline in the depth image is extended. operation can effectively avoid this phenomenon.

所述的根据设定的分层层数，对滤波扩展后的深度图像进行分层操作，得到分层深度图像步骤为：According to the set number of layered layers, the layered operation is performed on the depth image after filtering and expansion, and the steps of obtaining the layered depth image are:

其中，

表示取小于等于·的最大整数的操作；in,

Indicates the operation of taking the largest integer less than or equal to ;

分层深度图

layered depth map

${I I}_{Depth Depth}^{index index} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Depth Depth}^{E E.} ((x x,, y the y)) & Index Index ((x x,, y the y)) = = index index \\ 255255 & otherwise otherwise \end{matrix} index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11 . .$

(h)根据设定的相机阵列聚焦层focus，确定前景图层和背景图层，索引值index＞focus的深度层和聚焦层相比，距离观看者远，为背景层，索引值index＜focus的深度层和聚焦层相比，距离观看者近，为前景图层。(h) Determine the foreground layer and background layer according to the set camera array focus layer focus, the depth layer with index value index>focus is farther away from the viewer than the focus layer, and is the background layer, index value index<focus Compared with the focus layer, the depth layer is closer to the viewer and is the foreground layer.

(i)根据步骤(f)得到的像素点深度层索引值Index(x，y)，对于参考视点二维图像I_Color(x，y)进行分层操作，得到每一层的分层二维图像同时记录每一层二维图像的赋值标志位

分层二维图像的索引值index越小表示该二维图像层越靠近观看者，索引值index越大表示该二维图像层越远离观看者，表示公式如下：(i) According to the pixel depth layer index value Index(x, y) obtained in step (f), perform layering operation on the reference viewpoint two-dimensional image I_Color (x, y) to obtain the layered two-dimensional image of each layer image Simultaneously record each layer of 2D image The assignment flag

layered 2D image The smaller the index value index of , the closer the 2D image layer is to the viewer, and the larger the index value index is, the farther the 2D image layer is from the viewer. The expression formula is as follows:

${I I}_{Color color}^{index index} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Color color} ((x x,, y the y)) & Index Index ((x x,, y the y)) = = index index \\ 00 & otherwise otherwise \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11$

所述的根据设定的参考视点前景层和其相邻视点前景层之间的最大视差值，计算虚拟相机阵列相邻视点前景层与背景层之间的最大视差值，并结合各个深度层到聚焦层的距离计算各个深度层对应二维图像层对应的视差值步骤为：According to the maximum parallax value between the set reference viewpoint foreground layer and its adjacent viewpoint foreground layer, calculate the maximum parallax value between the adjacent viewpoint foreground layer and the background layer of the virtual camera array, and combine each depth The steps to calculate the disparity value corresponding to the two-dimensional image layer corresponding to each depth layer from the distance from the layer to the focus layer are:

MaxR＝(MaxD/focus)·(N-1)；MaxR=(MaxD/focus)·(N-1);

(m)根据步骤(1)得到的各个虚拟视点的索引值sub_i计算虚拟视点位置到参考视点位置的相对距离D_{sub_i}＝sub_i-0＝sub_i；(m) Calculate the relative distance D_{sub_i} =sub_i -0=sub_i from the virtual viewpoint position to the reference viewpoint position according to the index value sub_i of each virtual viewpoint obtained in step (1);

(n)对步骤(i)得到的每一层二维图像

的赋值标志位

水平扩展的方向

-1表示水平向左扩展，1表示水平向右扩展，0表示不做扩展，表示公式如下：(n) for each layer of two-dimensional image obtained in step (i)

The assignment flag

Direction of horizontal expansion

(o)将每一层扩展二维图像的赋值标志位

Initialized to zero, the representation formula is as follows:

(p)根据步骤(n)得到的水平扩展方向

对步骤(i)得到的每一层二维图像待扩展的像素点进行赋值操作，得到二维图像层

同时记录赋值标志位表示公式如下：(p) Horizontal expansion direction obtained according to step (n)

For each layer of two-dimensional image obtained in step (i) The pixels to be expanded are assigned to obtain a two-dimensional image layer

Simultaneous record assignment flag bit The expression formula is as follows:

${I I}_{EC EC}^{index index} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot &Center Dot; j j,, y the y)) = = \{\begin{matrix} {I I}_{Color color} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot \cdot j j,, y the y)) & {dir dir}_{Color color}^{index index} ((x x,, y the y)) = = - - 11 \\ {I I}_{Color color} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot &Center Dot; j j,, y the y)) & {dir dir}_{Color color}^{index index} ((x x,, y the y)) = = 11 \end{matrix} j j = = 0,1,2 0,1,2,, . . . . . .,, L L - - 11$

其中，L表示二维图像层

The number of expanded pixels, L=D_sub MaxR;

每一层二维图像对应不同的视差值，在利用加权水平平移算法进行虚拟视点二维图像绘制时，会在层与层之间产生没有值像素点形成的空洞区域，对视觉效果产生极大地影响。对分层二维图像进行扩展的目的就是利用扩展的像素部分填补平移产生的空洞区域，提高生成的虚拟视点图像的质量。Each layer of two-dimensional images corresponds to a different parallax value. When using the weighted horizontal translation algorithm to draw a virtual viewpoint two-dimensional image, there will be a hollow area formed by pixels with no value between layers, which will greatly affect the visual effect. earth influence. The purpose of extending the layered two-dimensional image is to use the expanded pixel part to fill the hole area generated by the translation and improve the quality of the generated virtual viewpoint image.

(q)对步骤(p)得到的二维图像层

对应赋值标志位

的像素点进行赋值操作，得到最终的扩展二维图像层

并记录最终的二维图像层赋值标志位

表示公式如下：(q) for the two-dimensional image layer obtained in step (p)

Corresponding assignment flag bit

And record the final two-dimensional image layer assignment flag bit

The expression formula is as follows:

和赋值标志位

进行加权水平平移操作，得到平移后的二维图像层

和平移后的赋值标志位

and assignment flags

Perform a weighted horizontal translation operation to obtain the translated two-dimensional image layer

and shifted assignment flags

The expression formula is as follows:

${I I}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 00 & ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) < < 00 or or ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) &GreaterEqual; &Greater Equal; H h \\ {I I}_{ExColor ExColor}^{index index} ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i},, y the y)) & 00 \leq \leq ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) < < H h \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11$

${F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 00 & ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) < < 00 or or ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) &GreaterEqual; &Greater Equal; H h \\ {F f}_{ExColor ExColor}^{index index} ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i},, y the y)) & 00 \leq \leq ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) < < H h \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11$

(s)初始化虚拟视点位置i二维图像的赋值标志位

为零，表示公式如下：(s) Initialize the assignment flag bit of the virtual viewpoint position i two-dimensional image

is zero, the formula is as follows:

${F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00,, &ForAll; &ForAll; x x = = 0,1,2 0,1,2,, . . . . . .,, H h - - 11,, &ForAll; &ForAll; y the y = = 0,1,2 0,1,2,, . . . . . .,, V V - - 11;;$

(t)将步骤(r)中得到的平移后的二维图像层

按照先放背景后放前景，由远及近的顺序，即index从大到小的顺序，一层一层地叠加起来，每叠加一层二维图像层，更新一次虚拟视点位置i二维图像的赋值标志位

叠加操作结束后根据最终的赋值标志位

得到虚拟视点位置i的二维图像表示公式如下：(t) the translated two-dimensional image layer obtained in step (r)

Put the background first and then the foreground, in the order from far to near, that is, in the order of index from large to small, superimpose layer by layer, and update the virtual viewpoint position i two-dimensional image every time a layer of two-dimensional image is superimposed The assignment flag

After the superposition operation ends, according to the final assignment flag

Get the two-dimensional image of the virtual viewpoint position i The expression formula is as follows:

${I I}_{vir vir}^{i i} ((x x,, y the y)) = = \{\begin{matrix} 00 & {F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00 \\ {I I}_{HSColor HSColor}^{{F f}_{vir vir}^{i i} ((x x,, y the y)) - - 11} ((x x,, y the y)) & {F f}_{vir vir}^{i i} ((x x,, y the y)) > > 00 and and {F f}_{vir vir}^{i i} ((x x,, y the y)) - - N N \leq \leq 00 \\ {I I}_{HSColor HSColor}^{{F f}_{vir vir}^{i i} ((x x,, y the y)) - - 11 - - N N} ((x x,, y the y)) & {F f}_{vir vir}^{i i} ((x x,, y the y)) - - N N > > 00 \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11;;$

在对平移后的二维图像层进行叠加操作时按照先放背景后放前景，由远及近的顺序进行叠加，有效地保证了场景中前景物体和背景物体之间正确的遮挡关系。When superimposing the translated two-dimensional image layer, put the background first and then the foreground, and superimpose in the order from far to near, effectively ensuring the correct occlusion relationship between the foreground object and the background object in the scene.

实施例：Example:

(1)将图像分辨率为640×360的Racket二维图像测试码流和分辨率为640×360的Racket深度图像测试码流作为待生成多视点虚拟图像的视频文件。图2(a)即为Racket二维图像测试码流的截图，图2(b)即为Racket深度图像测试码流的截图。(1) The Racket two-dimensional image test stream with an image resolution of 640×360 and the Racket depth image test stream with a resolution of 640×360 are used as video files to generate multi-viewpoint virtual images. Figure 2(a) is a screenshot of the Racket two-dimensional image test stream, and Figure 2(b) is a screenshot of the Racket depth image test stream.

(2)对输入深度图像进行中值滤波和边缘扩展，得到预处理后的深度图像。图3即为Racket深度图像预处理后的深度图像。(2) Perform median filtering and edge extension on the input depth image to obtain the preprocessed depth image. Figure 3 is the depth image after Racket depth image preprocessing.

(3)设定分层数N＝17，对预处理后的深度图像进行分层处理，得到分层后的深度图像。图4(a)即为第0层分层深度图像。(3) Set the layer number N=17, perform layer processing on the preprocessed depth image, and obtain the layered depth image. Figure 4(a) is the layer-0 layered depth image.

(4)选定第12层为聚焦深度层，则第0-11层为前景层，第13-16层为背景层。(4) Select the 12th layer as the focus depth layer, then the 0-11th layer is the foreground layer, and the 13th-16th layer is the background layer.

(5)根据分层深度图像，对Racket二维图像进行分层处理，得到分层后的二维图像。图4(b)即为第0层分层二维图像。(5) According to the layered depth image, the Racket two-dimensional image is layered to obtain a layered two-dimensional image. Figure 4(b) is the layered 2D image of layer 0.

(6)设定参考视点前景层和其相邻视点前景层之间的最大视差值为12，计算虚拟相机阵列相邻视点前景层与背景层之间的最大视差值，并结合各个深度层到聚焦层的距离计算各个深度层对应二维图像层对应的视差值。(6) Set the maximum disparity value between the reference viewpoint foreground layer and its adjacent viewpoint foreground layer to 12, calculate the maximum disparity value between the virtual camera array adjacent viewpoint foreground layer and background layer, and combine each depth The distance from the layer to the focus layer is used to calculate the disparity value corresponding to the two-dimensional image layer corresponding to each depth layer.

(7)根据虚拟视点位置到参考视点位置的相对距离和相机阵列相邻视点前景层与背景层之间的最大视差值，对分层二维图像进行扩展操作，得到扩展后的分层二维图像。图5即为第0层扩展二维图像。(7) According to the relative distance from the virtual viewpoint position to the reference viewpoint position and the maximum parallax value between the foreground layer and the background layer of adjacent viewpoints of the camera array, the layered two-dimensional image is extended to obtain the extended layered two-dimensional image. dimensional image. Figure 5 is the extended two-dimensional image of the 0th layer.

(8)根据虚拟视点位置到参考视点位置的相对距离和各个二维图像层对应的视差值，通过加权水平平移算法得到各个虚拟视点位置的虚拟二维图像。图6即为最终生成的虚拟九视点二维图像。(8) According to the relative distance from the virtual viewpoint position to the reference viewpoint position and the disparity value corresponding to each two-dimensional image layer, the virtual two-dimensional image of each virtual viewpoint position is obtained through a weighted horizontal translation algorithm. Figure 6 is the final generated virtual nine-viewpoint two-dimensional image.

Claims

Translated fromChinese

1.一种基于深度图分层的虚拟多视点图像的生成方法，其特征在于包括如下步骤：1. A method for generating virtual multi-viewpoint images based on depth map layering, characterized in that it comprises the steps:

2.根据权利要求1所述的一种基于深度图分层的虚拟多视点图像的生成方法，其特征在于所述的对待处理的参考视点深度图像进行中值滤波操作，并在水平方向上对深度值小的物体进行扩展操作，得到滤波扩展后的深度图像步骤为：2. A method for generating a virtual multi-viewpoint image based on depth map layering according to claim 1, wherein the reference viewpoint depth image to be processed is subjected to a median filtering operation, and horizontally The object with a small depth value is expanded, and the steps to obtain the depth image after filtering and expansion are as follows:

dir dir ((x x,, y the y)) = = \{\begin{matrix} - - 11 & | | {I I}_{Depth Depth}^{' '} ((x x,, y the y)) - - {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) | | > > thand more than {I I}_{Depth Depth}^{' '} ((x x,, y the y)) > > {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) \\ 11 & | | {I I}_{Depth Depth}^{' '} ((x x,, y the y)) - - {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) | | > > thand more than {I I}_{Depth Depth}^{' '} ((x x,, y the y)) < < {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) \\ 00 & otherwise otherwise \end{matrix};;

flag flag ((x x,, y the y)) = = 00,, &ForAll; &ForAll; x x = = 0,1,2 0,1,2,, . . . . . .,, H h - - 11 &ForAll; &ForAll; y the y = = 0,1,2 0,1,2,, . . . . . .,, V V - - 11

其中

表示任意给定；in

means any given;(d)根据步骤(b)得到的每个像素的扩展方向dir(x，y)对步骤(a)得到的平滑深度图I′_Depth(x，y)中待扩展的像素点进行赋值操作，得到深度图I″_Depth(x，y)，并记录对应赋值标志位flag(x，y)，表示公式如下：(d) According to the extension direction dir(x, y) of each pixel obtained in step (b), assign values to the pixels to be extended in the smooth depth map I′_Depth (x, y) obtained in step (a), Get the depth map I″_Depth (x, y), and record the corresponding assignment flag bit flag (x, y), the expression formula is as follows:

{I I}_{Depth Depth}^{' '' '} ((x x + + dir dir ((x x,, y the y)) \cdot &Center Dot; i i,, y the y)) = = \{\begin{matrix} {I I}_{Depth Depth}^{' '} ((x x + + 11,, y the y)) & dir dir ((x x,, y the y)) = = - - 11 \\ {I I}_{Depth Depth}^{' '} ((x x,, y the y)) & dir dir ((x x,, y the y)) = = 11 \end{matrix},, i i = = 0,1,2 0,1,2,, . . . . . .,, K K - - 11

The expression formula is as follows:

{I I}_{Depth Depth}^{E E.} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Depth Depth}^{' '} ((x x,, y the y)) & flag flag ((x x,, y the y)) = = 00 \\ {I I}_{Depth Depth}^{' '' '} ((x x,, y the y)) & flag flag ((x x,, y the y)) = = 11 \end{matrix} . .

3.根据权利要求1所述的一种基于深度图分层的虚拟多视点图像的生成方法，其特征在于所述的根据设定的分层层数，对滤波扩展后的深度图像进行分层操作，得到分层深度图像步骤为：3. A method for generating a virtual multi-viewpoint image based on depth map layering according to claim 1, wherein the depth image after filtering and expansion is layered according to the layered number of layers set. Operation, the steps to obtain the layered depth image are:

其中，

表示取小于等于·的最大整数的操作；in,

Indicates the operation of taking the largest integer less than or equal to ;

分层深度图的索引值index越小表示该深度层越靠近观看者，索引值index越大表示该深度层越远离观看者，表示公式如下：(g) According to the pixel depth layer index value Index(x, y) obtained in step (f), obtain the layered depth map of each layer

layered depth map The smaller the index value of , the closer the depth layer is to the viewer, and the larger the index value is, the farther the depth layer is from the viewer. The expression formula is as follows:

{I I}_{Depth Depth}^{index index} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Depth Depth}^{E E.} ((x x,, y the y)) & Index Index ((x x,, y the y)) = = index index \\ 255255 & otherwise otherwise \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11 . .

4.根据权利要求1所述的一种基于深度图分层的虚拟多视点图像的生成方法，其特征在于所述的在分层深度图像中选定一个深度层作为相机阵列的聚焦层，则深度值比聚焦层深度值大的为背景层，深度值比聚焦层深度值小的为前景层步骤为：4. the generation method of a kind of virtual multi-viewpoint image based on depth map layering according to claim 1, it is characterized in that described in layered depth image, select a depth layer as the focusing layer of camera array, then The depth value greater than the depth value of the focus layer is the background layer, and the depth value smaller than the depth value of the focus layer is the foreground layer. The steps are:

5.根据权利要求1所述的一种基于深度图分层的虚拟多视点图像的生成方法，其特征在于所述的对待处理的参考视点二维图像，根据二维图像相对应的分层深度图像，对二维图像进行分层操作，得到分层二维图像步骤为：5. A method for generating a virtual multi-viewpoint image based on depth map layering according to claim 1, wherein the reference viewpoint two-dimensional image to be processed is based on the layered depth corresponding to the two-dimensional image Image, perform layered operation on the two-dimensional image, and the steps to obtain the layered two-dimensional image are:

同时记录每一层二维图像

的赋值标志位

分层二维图像

Simultaneously record each layer of 2D image

The assignment flag

layered 2D image

{I I}_{Color color}^{index index} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Color color} ((x x,, y the y)) & Index Index ((x x,, y the y)) = = index index \\ 00 & otherwise otherwise \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11

{F f}_{Color color}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 11 & Index Index ((x x,, y the y)) = = index index \\ 00 & otherwise otherwise \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11 . .

6.根据权利要求1所述的一种基于深度图分层的虚拟多视点图像的生成方法，其特征在于所述的根据设定的参考视点前景层和其相邻视点前景层之间的最大视差值，计算虚拟相机阵列相邻视点前景层与背景层之间的最大视差值，并结合各个深度层到聚焦层的距离计算各个深度层对应二维图像层对应的视差值步骤为：6. A method for generating virtual multi-viewpoint images based on depth map layers according to claim 1, characterized in that according to the set reference viewpoint foreground layer and its adjacent viewpoint foreground layer, the maximum Disparity value, calculate the maximum disparity value between the foreground layer and the background layer of the adjacent viewpoint of the virtual camera array, and combine the distance from each depth layer to the focus layer to calculate the disparity value corresponding to the two-dimensional image layer corresponding to each depth layer. The steps are: :

MaxR＝(MaxD/focus)·(N-1)；MaxR=(MaxD/focus)·(N-1);

7.根据权利要求1所述的一种基于深度图分层的虚拟多视点图像的生成方法，其特征在于所述的根据虚拟视点位置到参考视点位置的相对距离和相机阵列相邻视点前景层与背景层之间的最大视差值，对分层二维图像进行扩展操作，得到扩展后的分层二维图像步骤为：7. The generation method of a virtual multi-viewpoint image based on depth map layering according to claim 1, wherein the relative distance from the virtual viewpoint position to the reference viewpoint position and the adjacent viewpoint foreground layer of the camera array are characterized in that The maximum parallax value between the background layer and the layered two-dimensional image is extended, and the steps to obtain the extended layered two-dimensional image are:

(m)根据步骤(1)得到的各个虚拟视点的索引值sub_i计算虚拟视点位置到参考视点位置的相对距离D_{sub_i′}＝sub_i-0＝sub_i；(m) calculate the relative distance D_{sub_i '} =sub_i -0=sub_i from the virtual viewpoint position to the reference viewpoint position according to the index value sub_i of each virtual viewpoint obtained in step (1);

(n)对步骤(i)得到的每一层二维图像

的赋值标志位

水平扩展的方向-1表示水平向左扩展，1表示水平向右扩展，0表示不做扩展，表示公式如下：(n) for each layer of two-dimensional image obtained in step (i)

The assignment flag

Direction of horizontal expansion -1 means to expand horizontally to the left, 1 means to expand horizontally to the right, and 0 means not to expand. The expression formula is as follows:

(o)将每一层扩展二维图像的赋值标志位初始化为零，表示公式如下：(o) Extend the assignment flag of each layer of two-dimensional image Initialized to zero, the representation formula is as follows:

{F f}_{EC EC}^{index index} ((x x,, y the y)) = = 00,, &ForAll; &ForAll; x x = = 0,1,2 0,1,2,, . . . . . .,, H h - - 11 &ForAll; &ForAll; y the y = = 0,1,2 0,1,2,, . . . . . .,, V V - - 11;;

(p)根据步骤(n)得到的水平扩展方向

对步骤(i)得到的每一层二维图像

待扩展的像素点进行赋值操作，得到二维图像层

同时记录赋值标志位

For each layer of two-dimensional image obtained in step (i)

The pixels to be expanded are assigned to obtain a two-dimensional image layer

Simultaneous record assignment flag bit

The expression formula is as follows:

{I I}_{EC EC}^{index index} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot \cdot j j,, y the y)) = = \{\begin{matrix} {I I}_{Color color} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot &Center Dot; j j,, y the y)) & {dir dir}_{Color color}^{index index} ((x x,, y the y)) = = - - 11 \\ {I I}_{Color color} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot \cdot j j,, y the y)) & {dir dir}_{Color color}^{index index} ((x x,, y the y)) = = 11 \end{matrix},, j j = = 0,1,2 0,1,2,, . . . . . .,, L L - - 11

{F f}_{EC EC}^{index index} ((x x + + {dir dir}_{Color color}^{index index} ((x x,, y the y)) \cdot \cdot j j,, y the y)) = = 11 | | {dir dir}_{Color color}^{index index} ((x x,, y the y)) | | = = 11,, j j = = 0,1,2 0,1,2,, . . . . . .,, L L - - 11

其中，L表示二维图像层扩展像素点的个数，L＝D_sub·MaxR；Among them, L represents the two-dimensional image layer The number of expanded pixels, L=D_sub MaxR;

(q)对步骤(p)得到的二维图像层

对应赋值标志位

的像素点进行赋值操作，得到最终的扩展二维图像层并记录最终的二维图像层赋值标志位

Corresponding assignment flag bit

The pixel points are assigned to obtain the final extended two-dimensional image layer And record the final two-dimensional image layer assignment flag bit

The expression formula is as follows:

{I I}_{ExColor ExColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} {I I}_{Color color}^{index index} ((x x,, y the y)) & {F f}_{EC EC}^{index index} ((x x,, y the y)) = = 00 \\ {I I}_{EC EC}^{index index} ((x x,, y the y)) & {F f}_{EC EC}^{index index} ((x x,, y the y)) = = 11 \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11

{F f}_{ExColor ExColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 00 & {F f}_{EC EC}^{index index} ((x x,, y the y)) = = 00 and and {F f}_{Color color}^{index index} ((x x,, y the y)) = = 00 \\ 11 & {F f}_{Color color}^{index index} ((x x,, y the y)) = = 11 \\ 22 & {F f}_{EC EC}^{index index} ((x x,, y the y)) = = 11 and and {F f}_{Color color}^{index index} ((x x,, y the y)) = = 00 \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11 . .

8.根据权利要求1所述的一种基于深度图分层的虚拟多视点图像的生成方法，其特征在于所述的根据虚拟视点位置到参考视点位置的相对距离和各个二维图像层对应的视差值，通过加权水平平移算法得到各个虚拟视点位置的虚拟二维图像步骤为：8. A method for generating virtual multi-viewpoint images based on depth map layers according to claim 1, characterized in that the relative distance from the virtual viewpoint position to the reference viewpoint position and the corresponding two-dimensional image layers are The parallax value, the steps to obtain the virtual two-dimensional image of each virtual viewpoint position through the weighted horizontal translation algorithm are as follows:

(r)假设要生成M个虚拟视点位置中第i个位置的图像，按照步骤(m)得到的虚拟视点位置到参考视点位置的相对距离D_{sub_i}和步骤(k)得到的各个二维图像层对应的视差值Disindex，对步骤(q)得到的扩展二维图像层

和赋值标志位

进行加权水平平移操作，得到平移后的二维图像层

和平移后的赋值标志位表示公式如下：(r) Assuming that the image of the i-th position among the M virtual viewpoint positions is to be generated, the relative distance D_{sub_i} from the virtual viewpoint position to the reference viewpoint position obtained in step (m) and each two-dimensional image layer obtained in step (k) Corresponding disparity value Disindex, for the extended two-dimensional image layer obtained in step (q)

and assignment flags

and shifted assignment flags The expression formula is as follows:

{I I}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 00 & ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) < < 00 or or ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) &GreaterEqual; &Greater Equal; H h \\ {I I}_{ExColor ExColor}^{index index} ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i},, y the y)) & 00 \leq \leq ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) < < H h \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11

{F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = \{\begin{matrix} 00 & ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) < < 00 or or ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i})) &GreaterEqual; &Greater Equal; H h \\ {F f}_{ExColor ExColor}^{index index} ((x x + + {Dis dis}^{index index} \cdot \cdot {D D.}_{sub sub__i i},, y the y)) & 00 \leq \leq ((x x + + {Dis dis}^{index index} \cdot &Center Dot; {D D.}_{sub sub__i i})) < < H h \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11

(s)初始化虚拟视点位置i二维图像的赋值标志位为零，表示公式如下；(s) Initialize the assignment flag bit of the virtual viewpoint position i two-dimensional image is zero, the expression formula is as follows;

{F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00,, &ForAll; &ForAll; x x = = 0,1,2 0,1,2,, . . . . . .,, H h - - 11 &ForAll; &ForAll; y the y = = 0,1,2 0,1,2,, . . . . . .,, V V - - 11;;

(t)将步骤(r)中得到的平移后的二维图像层

叠加操作结束后根据最终的赋值标志位

得到虚拟视点位置i的二维图像

表示公式如下：(t) the translated two-dimensional image layer obtained in step (r)

After the superposition operation ends, according to the final assignment flag

Get the two-dimensional image of the virtual viewpoint position i

The expression formula is as follows:

{F f}_{vir vir}^{i i} ((x x,, y the y)) = = \{\begin{matrix} 00 & {F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00 and and {F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = 00 \\ index index + + 11 & {F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = 11 \\ index index + + 11 + + N N & {F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00 and and {F f}_{HSColor HSColor}^{index index} ((x x,, y the y)) = = 22 \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11

{I I}_{vir vir}^{i i} ((x x,, y the y)) = = \{\begin{matrix} 00 & {F f}_{vir vir}^{i i} ((x x,, y the y)) = = 00 \\ {I I}_{HSColor HSColor}^{{F f}_{vir vir}^{i i} ((x x,, y the y)) - - 11} ((x x,, y the y)) & {F f}_{vir vir}^{i i} ((x x,, y the y)) > > 00 and and {F f}_{vir vir}^{i i} ((x x,, y the y)) - - N N \leq \leq 00 \\ {I I}_{HSColor HSColor}^{{F f}_{vir vir}^{i i} ((x x,, y the y)) - - 11 - - N N} ((x x,, y the y)) & {F f}_{vir vir}^{i i} ((x x,, y the y)) - - N N > > 00 \end{matrix},, index index = = 0,1,2 0,1,2,, . . . . . .,, N N - - 11;;