技术领域technical field
本发明涉及图像的三维显示优化技术领域，具体涉及一种基于光场采集的光栅棱柱三维显示渲染方法。The invention relates to the technical field of three-dimensional display optimization of images, in particular to a grating prism three-dimensional display rendering method based on light field collection.
背景技术Background technique
三维显示技术因其更为贴近人类实际感知而具有广阔的前景。光场三维显示系统的研究目的就是探索如何将光场进行还原，亦即如何控制每一根光线的出射方向和角度。Three-dimensional display technology has broad prospects because it is closer to the actual perception of human beings. The research purpose of the light field 3D display system is to explore how to restore the light field, that is, how to control the outgoing direction and angle of each light ray.
光栅棱柱显示设备包括LCD显示屏(Liquid Crystal Display，液晶显示器)、透明有机玻璃和柱镜光栅板。通过在LCD显示屏和柱镜光栅板中间加装透明有机玻璃，使得两者的距离为柱镜光栅单元的焦距。N个视点的像素会被全部放大投射到观察者眼前，只要观察者的目光落在任意两个相邻视点，就可以分别观看到对应视点的图像进而形成立体视觉。高质量的虚拟视点合成与显示正是目前3D技术的关键和难点，真三维显示目前存在的问题：The grating prism display device includes an LCD display (Liquid Crystal Display, liquid crystal display), transparent plexiglass and a lenticular grating plate. By installing transparent organic glass between the LCD display screen and the lenticular lens grating plate, the distance between the two is the focal length of the lenticular lens grating unit. The pixels of N viewpoints will be enlarged and projected to the eyes of the observer. As long as the observer's gaze falls on any two adjacent viewpoints, the images of the corresponding viewpoints can be viewed respectively to form stereoscopic vision. High-quality virtual viewpoint synthesis and display is the key and difficulty of current 3D technology. The current problems of true 3D display:
1、光栅棱柱显示仍存在相邻视点之间串扰(crosstalk)的问题；1. The grating prism display still has the problem of crosstalk between adjacent viewpoints;
2、光栅棱柱显示视点个数较少，影响3D显示的最终效果。2. The number of viewpoints displayed by the grating prism is small, which affects the final effect of 3D display.
其中，串扰是由于相邻视点光线不完全分离传输至观看者的双眼所导致的，观看者的左眼会接收到部分本应该传输至右眼的光线，同样地，右眼也会接收到部分本应该传输至左眼的光线。串扰现象已经成为影响高质量光栅棱柱三维显示技术的一个很大的因素，目前的解决方案一般可以归纳成两种：一个是基于设备的方法，例如用视差屏障技术或时控的方式；另一种是基于子像素(每个像素由RGB三原色组成，每个像素上的每种颜色叫一个“子像素”)值修正的方式，例如计算串扰因子的方式。虽然这些方法在一定程度上解决了视点串扰问题，但是硬件成本的开销较大，或者计算量较大。Among them, crosstalk is caused by the incomplete separation of light rays from adjacent viewpoints to the eyes of the viewer. The left eye of the viewer will receive part of the light that should have been transmitted to the right eye. Similarly, the right eye will also receive part of the light. The light that should have been transmitted to the left eye. The phenomenon of crosstalk has become a major factor affecting high-quality grating prism 3D display technology. The current solutions can generally be summarized into two types: one is based on equipment, such as parallax barrier technology or time control; the other One is based on sub-pixel (each pixel is composed of RGB three primary colors, and each color on each pixel is called a "sub-pixel") value correction method, such as the method of calculating the crosstalk factor. Although these methods solve the problem of viewpoint crosstalk to a certain extent, the overhead of hardware costs is relatively large, or the amount of calculation is relatively large.
发明内容Contents of the invention
为了解决现有技术中的上述问题，本发明提出了一种基于光场采集的光栅棱柱三维显示渲染方法，在不增加硬件成本的前提下，优化效果更佳，并且能够渲染出超多视角的三维显示(super multiview display)。In order to solve the above-mentioned problems in the prior art, the present invention proposes a grating prism 3D display rendering method based on light field acquisition. Without increasing the hardware cost, the optimization effect is better, and it can render super multi-view images. Three-dimensional display (super multiview display).
本发明提出一种基于光场采集的光栅棱柱三维显示渲染方法，利用所生成的目标合成索引图对三维图像进行渲染；The present invention proposes a grating prism three-dimensional display and rendering method based on light field acquisition, and uses the generated target composite index map to render the three-dimensional image;
所述目标合成索引图，与当前显示三维图像的显示器规格相匹配；The target synthetic index map matches the specifications of the display currently displaying the three-dimensional image;
所述目标合成索引图由以下标定方法生成，具体包括：The target synthetic index map is generated by the following calibration methods, specifically including:
步骤S1，利用多视点图像渲染方法，分别渲染出各预设视点的初始合成索引图；Step S1, using a multi-viewpoint image rendering method to render the initial synthetic index map of each preset viewpoint;
步骤S2，相机移动到每一个预设视点，在LCD面板上依次显示所述各预设视点的初始合成索引图，并用相机捕获图像；将相机在各预设视点上捕获到的图像，进行预处理，包括：反畸变、感兴趣区域提取、2D投影变化、滤波操作；将预处理后的图像通过非线性映射得到对应的各预设视点的光场图像；Step S2, the camera moves to each preset viewpoint, and the initial composite index images of the preset viewpoints are sequentially displayed on the LCD panel, and the image is captured by the camera; the images captured by the camera on each preset viewpoint are previewed Processing, including: anti-distortion, region of interest extraction, 2D projection change, filtering operation; the preprocessed image is obtained through nonlinear mapping to obtain the corresponding light field image of each preset viewpoint;
步骤S3，合并步骤S2中得到的各预设视点上的光场图像，得到实际光场；根据捕获时的相机标定关系，计算出实际相机位置上的理想光场；Step S3, merging the light field images at each preset viewpoint obtained in step S2 to obtain the actual light field; according to the camera calibration relationship at the time of capture, calculate the ideal light field at the actual camera position;
步骤S4，建立从LCD面板上显示的初始合成索引图到观看平面的映射法则，记录到查找表中；Step S4, establishing a mapping rule from the initial synthetic index map displayed on the LCD panel to the viewing plane, and recording it in the look-up table;
步骤S5，通过最小化||L-wL～||2求得优化矩阵w；利用所述优化矩阵w，对所述实际光场L～进行优化，得到wL～；其中，L为理想光场；Step S5, obtain the optimization matrix w by minimizing ||L-wL～ ||2 ; use the optimization matrix w to optimize the actual light field L～ to obtain wL～ ; where L is the ideal light field ;
步骤S6，根据优化后的所述实际光场，利用所述查找表，反向搜索得到目标合成索引图。Step S6 , according to the optimized actual light field, use the lookup table to perform a reverse search to obtain a target composite index map.
优选地，步骤S1中对每个视角图的渲染方法为：Preferably, the rendering method for each perspective image in step S1 is:
将第i个视角图置为白色，其余视角图置为黑色，利用多视点图像渲染方法，渲染出第i个预设视点的初始合成索引图；其中，i＝1,2,3,...,N，N为预设视点的数量。Set the i-th perspective map to white, and set the rest of the perspective maps to black, and use the multi-viewpoint image rendering method to render the initial synthetic index map of the i-th preset viewpoint; where, i=1,2,3,... .,N, where N is the number of preset viewpoints.
优选地，步骤S2具体为：Preferably, step S2 is specifically:
步骤S21，令m＝1；其中，m为预设视点的序号；Step S21, let m=1; wherein, m is the serial number of the preset viewpoint;
步骤S22，移动相机到第m个预设视点，依次显示第1个、第2个、…、第N个视点对应的初始合成索引图，并用相机捕获每一幅初始合成索引图的图像；Step S22, moving the camera to the m-th preset viewpoint, sequentially displaying the initial synthetic index images corresponding to the first, second, ..., N-th viewpoints, and using the camera to capture the images of each initial composite index image;
步骤S23，m＝m+1；若m≤N，则转至步骤S22；否则，转至步骤S24；其中，N为预设视点的数量；Step S23, m=m+1; if m≤N, go to step S22; otherwise, go to step S24; wherein, N is the number of preset viewpoints;
步骤S24，对捕获的图像进行预处理，包括：反畸变、感兴趣区域提取、2D投影变化、滤波操作；Step S24, preprocessing the captured image, including: anti-distortion, region of interest extraction, 2D projection change, filtering operation;
步骤S25，对预处理后的图像进行非线性映射，得到对应的各预设视点的光场图像。In step S25, non-linear mapping is performed on the preprocessed image to obtain light field images of corresponding preset viewpoints.
优选地，步骤S4中所述映射法则为光线从LCD面板上的各子像素点到观看平面的一一映射。Preferably, the mapping rule in step S4 is a one-to-one mapping of light from each sub-pixel on the LCD panel to the viewing plane.
优选地，所述感兴趣区域提取的方法为：对捕获图像进行反畸变处理后，提取图像上的3D显示屏所在区域作为感兴趣区域；Preferably, the method for extracting the region of interest is: after performing anti-distortion processing on the captured image, extract the region where the 3D display screen on the image is located as the region of interest;
所述2D投影变化的方法：提取所述感兴趣区域的4个顶点以及对应视点反畸变后的光场图像的4个顶点，计算得到单应矩阵；通过所述单应矩阵对捕获图像进行2D投影变化。The method for changing the 2D projection: extracting the 4 vertices of the region of interest and the 4 vertices of the light field image after de-distorting the corresponding viewpoint, and calculating the homography matrix; performing 2D processing on the captured image through the homography matrix. Projection changes.
优选地，步骤S2中，用相机捕获图像时，将标定板放置在3D显示屏和相机之间。Preferably, in step S2, when using the camera to capture images, the calibration plate is placed between the 3D display screen and the camera.
优选地，步骤S2中所述非线性映射为：Preferably, the nonlinear mapping described in step S2 is:
X＝Mr_sp-1(Vsub)；X = Mr_sp-1 (Vsub );
用于从预处理后的图像子像素值Vsub恢复出对应光线的亮度值X。It is used to restore the brightness value X of the corresponding light from the preprocessed image sub-pixel value Vsub .
优选地，步骤S3中，理想光场根据一个无串扰残留的理想的预设视点数量的光亮分布图得到。Preferably, in step S3, the ideal light field is obtained according to an ideal light distribution map with a preset number of viewpoints without crosstalk remaining.
优选地，所述利用所生成的目标合成索引图对三维图像进行渲染，具体为：Preferably, rendering the three-dimensional image by using the generated target composite index map is specifically:
其中，IIm为第m个视点上的目标合成索引图，Viewm为通过OpenGL(Open GraphicsLibrary，开放图形库)或3d Max(3D Studio Max，常简称为3d Max或3ds Max，是Discreet公司开发的基于PC系统的三维动画渲染和制作软件)渲染出的三维模型的第m个视角图，IIdisplay为最终显示在3D显示器上的合成图。Among them, IIm is the target synthesis index map on the mth viewpoint, and Viewm is the index map developed by Discreet through OpenGL (Open Graphics Library) or 3d Max (3D Studio Max, often referred to as 3d Max or 3ds Max). The m-th view of the 3D model rendered by the 3D animation rendering and production software based on the PC system, IIdisplay is the composite image finally displayed on the 3D display.
本发明提出的基于光场采集的光栅棱柱三维显示标定方法，在LCD面板上均匀地预设多个视点，用多视点渲染方法，分别渲染出各预设视点的初始合成索引图；将相机移动到每个视点，并依次显示上述初始合成索引图，进行图像的光场采集；将各视点上得到的光场合并成为一个实际的光场，利用采集图像时的相机标定关系，计算得到理想光场；建立从LCD面板上显示的初始合成索引图到观看平面的映射法则，记录到查找表中；再对所述实际光场进行优化，使之最大化地接近所述理想光场；根据优化后的所述实际光场，利用所述查找表，反向搜索得到目标合成索引图。通过上述标定方法，我们得到了针对当前显示设备的目标合成索引图。The grating prism three-dimensional display calibration method based on light field acquisition proposed by the present invention uniformly presets multiple viewpoints on the LCD panel, and uses the multi-viewpoint rendering method to render the initial synthetic index map of each preset viewpoint respectively; move the camera Go to each viewpoint, and display the above-mentioned initial synthetic index map in sequence, and collect the light field of the image; combine the light fields obtained at each viewpoint into an actual light field, and use the camera calibration relationship when collecting the image to calculate the ideal light field field; establish the mapping rule from the initial synthetic index map displayed on the LCD panel to the viewing plane, and record it in the look-up table; then optimize the actual light field to maximize it close to the ideal light field; according to the optimization After the actual light field, use the look-up table to perform a reverse search to obtain a target composite index map. Through the above calibration method, we get the target synthetic index map for the current display device.
本发明提出的基于光场采集的光栅棱柱三维显示渲染方法，对将要显示在3D显示设备上的三维图像，利用事先生成的与该3D显示设备相匹配的所述目标合成索引图进行渲染，就能够有效减轻串扰，与传统的优化方式相比，无需增加硬件设备，优化效果更佳，并且可以渲染出超多视角的三维显示(super multiview display)。The grating prism three-dimensional display rendering method based on light field acquisition proposed by the present invention renders the three-dimensional image to be displayed on the 3D display device by using the pre-generated target composite index map that matches the 3D display device to render the three-dimensional image. It can effectively reduce crosstalk, and compared with the traditional optimization method, it does not need to increase hardware devices, and the optimization effect is better, and it can render a super multiview three-dimensional display (super multiview display).
附图说明Description of drawings
图1为本实施例中，光线最终落于观看平面上时经过的路径示意图；FIG. 1 is a schematic diagram of the path that the light passes through when it finally falls on the viewing plane in this embodiment;
图2为本实施例中，基于光场采集的光栅棱柱三维显示标定方法流程示意图；FIG. 2 is a schematic flow chart of a grating prism three-dimensional display calibration method based on light field acquisition in this embodiment;
图3为本实施例中，生成实际光场并建立映射法则的示意图。FIG. 3 is a schematic diagram of generating an actual light field and establishing a mapping rule in this embodiment.
具体实施方式detailed description
下面参照附图来描述本发明的优选实施方式。本领域技术人员应当理解的是，这些实施方式仅仅用于解释本发明的技术原理，并非旨在限制本发明的保护范围。Preferred embodiments of the present invention are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present invention, and are not intended to limit the protection scope of the present invention.
以光学的角度看待光栅棱柱三维显示，例如一个硬件参数为9视点的光栅棱柱三维显示器，在裸眼观看时，若两眼目光正好落于两个相邻的视点区，两眼将会分别接受到来自两个不同视点的光线，在大脑中形成三维感知，产生立体视觉。当人的目光落于非视点区域，会接受到部分来自一个视点的光线，部分来自相邻视点的光线。这个视点串扰现象，在视点区域也存在，在非视点区域更为明显。串扰现象已经成为影响高质量光栅棱柱三维显示技术的一个很大的因素。Look at the three-dimensional grating prism display from an optical point of view. For example, a three-dimensional grating prism display with hardware parameters of 9 viewpoints, when viewing with the naked eye, if the eyes of both eyes fall on two adjacent viewpoint areas, the two eyes will respectively receive Light from two different viewpoints creates a three-dimensional perception in the brain, resulting in stereoscopic vision. When people's eyes fall on the non-viewpoint area, they will receive part of the light from one viewpoint and part of the light from the adjacent viewpoint. This viewpoint crosstalk phenomenon also exists in the viewpoint area, and is more obvious in the non-viewpoint area. The phenomenon of crosstalk has become a big factor affecting high-quality grating prism three-dimensional display technology.
人通过眼睛接受光的信息，通过大脑处理后得到对周边环境的感知。周围环境的任何可见的物体都是通过不断地自身发光或者通过光源照射后反射出光线，进入人的眼睛后得以被人看见。这些光源和物体发出的光线在空间中形成了一个光线的集合，称为光场。理论上，如果一个系统能够根据某个物体的光场完全还原，则人将无法通过眼睛区分出是该系统重构的物体光场，还是物体本身。光场三维显示系统的研究目的就是探索如何将光场进行还原，亦即如何控制每一条光线的出射方向和角度。能够用于产生光线和控制光线方向和信息的光学器件有许多种，如激光器、显示屏、透镜等，光场三维显示的研究在于如何用现有器件实现光线的方向和信息控制，或是制作出新的器件。People receive light information through the eyes, and get the perception of the surrounding environment after being processed by the brain. Any visible object in the surrounding environment can be seen by people after entering the human eye by continuously emitting light or reflecting light after being irradiated by a light source. The light emitted by these light sources and objects forms a collection of light rays in space, called a light field. Theoretically, if a system can completely restore the light field of an object, people will not be able to distinguish the light field of the object reconstructed by the system from the object itself. The research purpose of the light field 3D display system is to explore how to restore the light field, that is, how to control the outgoing direction and angle of each light. There are many kinds of optical devices that can be used to generate light and control the direction and information of light, such as lasers, display screens, lenses, etc. The research on three-dimensional display of light field is how to use existing devices to control the direction and information of light, or to make new device.
在真实世界中，一个物体发出的光场是连续的，而在实际计算中我们只能用离散的光场来近似模拟连续的光场，即呈现在人眼前的是N张光场图片。我们回到之前所说的光栅棱柱显示，我们的目的是在距离硬件参数为9视点的显示器的最佳观看位置上为观看者在视点区域上渲染出N个视点图像，当人的目光落于该视点上时能够接受到全部来自该视点的光线，就好像这幅视点图放在人眼前一样。视点个数N的取值范围为：In the real world, the light field emitted by an object is continuous, but in actual calculations, we can only use discrete light fields to approximate the continuous light field, that is, N light field pictures are presented to people. Let’s go back to the grating prism display mentioned earlier. Our purpose is to render N viewpoint images for the viewer on the viewpoint area at the optimal viewing position of the display with a distance of 9 viewpoints from the hardware parameters. When people’s eyes fall on When it is on this viewpoint, it can receive all the light from this viewpoint, just as if this viewpoint map is placed in front of people's eyes. The value range of the number of viewpoints N is:
其中，Z+表示正整数，即N为9的正整数倍，并且开平方后仍为正整数，如：9、36、81、…。在我们的实验中，N＝9或36，当N＝36，就叫做超多视角三维显示(super multiviewdisplay)，如图1所示，光线从LCD面板上的子像素出发，经过光栅板上的光栅单元，最终在最佳观看平面上结束。Among them, Z+ represents a positive integer, that is, N is a positive integer multiple of 9, and it is still a positive integer after the square root, such as: 9, 36, 81, .... In our experiment, N=9 or 36. When N=36, it is called a super multiview display (super multiview display). As shown in Figure 1, the light starts from the sub-pixels on the LCD panel and passes through the Raster elements that end up on the best viewing plane.
本发明提出一种基于光场采集的光栅棱柱三维显示标定方法，在最佳观看位置上我们接收到的是来自N个视点图像的光线的集合，它们从LCD面板上的H×W×3个子像素(H和W分别为LCD面板的高度和宽度，每个像素分为红、绿、蓝3个子像素)发射出来，经过透镜后，分散于各个视点上。我们需要分配的是LCD面板上的H×W×3个子像素的值。我们分别用II～、II、L～、L来代表初始合成索引图、目标合成索引图、相机捕获得到的实际光场以及理想光场。我们的目的就是计算出实际光场L～，并优化使其最大化地与理想光场接近。The present invention proposes a grating prism three-dimensional display calibration method based on light field collection. At the best viewing position, what we receive is a collection of light rays from N viewpoint images, which are obtained from the H×W×3 sub-images on the LCD panel. The pixels (H and W are the height and width of the LCD panel respectively, and each pixel is divided into three sub-pixels of red, green, and blue) are emitted, and after passing through the lens, they are scattered at various viewpoints. What we need to assign is the value of the H×W×3 subpixels on the LCD panel. We use II～ , II, L～ , L to represent the initial synthetic index map, the target synthetic index map, the actual light field captured by the camera, and the ideal light field. Our purpose is to calculate the actual light field L∼ and optimize it to be as close as possible to the ideal light field.
本实施例中，我们取N＝36，首先在9视点硬件设备的最佳观看平面上确定好9个视点的位置，并将相邻两视点平分为N/9份，于是得到36个预设的视点。显示系统的硬件参数如表1所示：In this embodiment, we take N=36, first determine the positions of 9 viewpoints on the best viewing plane of the 9-viewpoint hardware device, and divide the adjacent two viewpoints into N/9 parts, thus obtaining 36 presets point of view. The hardware parameters of the display system are shown in Table 1:
表1Table 1
在每个预设视点上，利用飞利浦研究实验室提出的多视点渲染方法，生成对应的初始合成索引图；将相机移动到每个预设视点，并依次显示上述初始合成索引图，进行图像的光场采集；将各视点上得到的光场合并成为一个实际的光场，利用采集图像时的相机标定关系，计算得到理想光场；建立从LCD面板上显示的初始合成索引图到观看平面的映射法则，记录到查找表中；再对所述实际光场进行优化，使之最大化地接近所述理想光场；根据优化后的所述实际光场，利用所述查找表，反向搜索得到目标合成索引图。At each preset viewpoint, use the multi-viewpoint rendering method proposed by Philips Research Labs to generate the corresponding initial composite index map; move the camera to each preset viewpoint, and display the above initial composite index map in turn to perform image processing Light field acquisition; combine the light fields obtained at each viewpoint into an actual light field, and use the camera calibration relationship when collecting images to calculate the ideal light field; establish the relationship from the initial synthetic index map displayed on the LCD panel to the viewing plane The mapping rule is recorded in the lookup table; and then the actual light field is optimized to maximize the approach to the ideal light field; according to the optimized actual light field, using the lookup table, reverse search Obtain the target synthetic index map.
本发明提出一种基于光场采集的光栅棱柱三维显示渲染方法，就是利用上述标定方法得到当前三维LCD显示器所对应的目标合成索引图，然后利用所述目标合成索引图，对将要显示在同样规格的LCD显示器上的三维图像进行渲染。The present invention proposes a grating prism three-dimensional display rendering method based on light field acquisition, which is to use the above-mentioned calibration method to obtain the target composite index map corresponding to the current three-dimensional LCD display, and then use the target composite index map to perform The 3D image is rendered on the LCD monitor.
本实施例中，如图2所示，所述目标合成索引图由以下标定方法生成，具体包括：In this embodiment, as shown in Figure 2, the target composite index map is generated by the following calibration method, specifically including:
步骤S1，将第i个视角图置为白色，其余视角图置为黑色，利用飞利浦研究实验室提出的多视点渲染方法，渲染出第i个预设视点的初始合成索引图IIi～；按此方法分别渲染出各预设视点的初始合成索引图IIi～(i＝1,2,3,...,N)；组成一个集合：N为预设视点的数量(本实施例中为36)；Step S1, set the i-th perspective image to white, and set the other perspective images to black, use the multi-viewpoint rendering method proposed by Philips Research Laboratory to render the initial composite index image IIi～ of the i-th preset viewpoint; press This method respectively renders the initial synthetic index map IIi～ (i=1,2,3,...,N) of each preset viewpoint; forming a set: N is the number of preset viewpoints (36 in this embodiment);
步骤S2，相机移动到每一个预设视点，在LCD面板上依次显示所述各预设视点的初始合成索引图IIi～(i＝1,2,3,...,N)，并用相机捕获图像；将相机在各预设视点上捕获到的图像，进行预处理，包括：反畸变、感兴趣区域提取、2D投影变化、滤波操作；将预处理后的图像通过非线性映射得到对应的各预设视点的光场图像；Step S2, the camera moves to each preset viewpoint, and the initial composite index graphs IIito (i=1,2,3,...,N) of the preset viewpoints are sequentially displayed on the LCD panel, and the camera Capture images; preprocess the images captured by the camera at each preset viewpoint, including: anti-distortion, region of interest extraction, 2D projection change, and filtering operations; obtain the corresponding preprocessed images through nonlinear mapping Light field images of each preset viewpoint;
用相机捕获图像时，将相机标定板放置在3D显示器和相机之间，在捕获的过程中共移动相机N-1次；When capturing images with a camera, place the camera calibration plate between the 3D display and the camera, and move the camera N-1 times during the capture process;
步骤S3，合并步骤S2中得到的各预设视点上的光场图像，得到实际光场L～；根据捕获时的相机标定关系，计算出实际相机位置上的理想光场L；Step S3, merging the light field images at each preset viewpoint obtained in step S2 to obtain the actual light field L~ ; according to the camera calibration relationship at the time of capture, calculate the ideal light field L at the actual camera position;
实际光场L～被集合为：其中，均表示集合；The actual light field L~ is aggregated as: in, Both represent collections;
由于预先设定的相机视角位置与实际移动的位置是有偏差的，我们通过相机标定得到实际获取光场图像时的相机位置。当相机水平移动获取每个视点的光场图片时，标定板被放置在3D显示器和相机之间；在实际相机位置的理想光场L可以根据一个无串扰残留的理想的N视点的光亮分布图得到；两眼在任意位置能同时接受到两个(并且只有两个)不同的视角图；Since there is a deviation between the preset camera viewing angle position and the actual moving position, we obtain the camera position when the light field image is actually acquired through camera calibration. The calibration plate is placed between the 3D display and the camera when the camera is moved horizontally to acquire light field pictures for each viewpoint; the ideal light field L at the actual camera position can be based on an ideal N viewpoint light distribution map without crosstalk residue Obtained; two eyes can receive two (and only two) different viewing angles at any position at the same time;
步骤S4，建立从LCD面板上显示的初始合成索引图到观看平面的映射法则，记录到查找表中；所述映射法则，为光线从LCD面板上的各子像素点到观看平面的一一映射；Step S4, establishing a mapping rule from the initial synthetic index map displayed on the LCD panel to the viewing plane, and recording it in the look-up table; the mapping rule is a one-to-one mapping of light from each sub-pixel point on the LCD panel to the viewing plane ;
步骤S5，通过最小化||L-wL～||2求得优化矩阵w；利用所述优化矩阵w，对所述实际光场L～进行优化，得到wL～；其中，L为理想光场；Step S5, obtain the optimization matrix w by minimizing ||L-wL～ ||2 ; use the optimization matrix w to optimize the actual light field L～ to obtain wL～ ; where L is the ideal light field ;
步骤S6，根据优化后的所述实际光场，利用所述查找表，反向搜索得到目标合成索引图其中，IIm第m个视点的目标合成索引图IIm，表示N个视点的目标合成索引图的集合。Step S6, according to the optimized actual light field, use the look-up table to perform a reverse search to obtain the target composite index map Among them, the target synthesis index map IIm of the mth viewpoint of IIm , Represents a collection of target synthetic index maps for N viewpoints.
本实施例中，步骤S2具体为：In this embodiment, step S2 is specifically:
步骤S21，令m＝1；其中，m为预设视点的序号；Step S21, let m=1; wherein, m is the serial number of the preset viewpoint;
步骤S22，移动相机到第m个预设视点，依次显示第1个、第2个、…、第N个视点对应的初始合成索引图IIi～(i＝1,2,...,N)，并用相机捕获每一幅初始合成索引图的图像；Step S22, move the camera to the m-th preset viewpoint, and sequentially display the initial synthetic index map IIi～ (i=1,2,...,N) corresponding to the first, second, ..., N-th viewpoint ), and each image of the initial synthetic index map is captured by the camera;
步骤S23，m＝m+1；若m≤N，则转至步骤S22；否则，转至步骤S24；其中，N为预设视点的数量；Step S23, m=m+1; if m≤N, go to step S22; otherwise, go to step S24; wherein, N is the number of preset viewpoints;
步骤S24，对捕获的图像进行预处理，包括：反畸变、感兴趣区域提取、2D投影变化、滤波操作；预处理后得到图像其中，m为视点的序号，m＝1,2,...,N；n为在第m个视点上捕获的图像的序号，n＝1,2,...,N；Step S24, preprocessing the captured image, including: anti-distortion, region of interest extraction, 2D projection change, filtering operation; image obtained after preprocessing Among them, m is the serial number of the viewpoint, m=1,2,...,N; n is the serial number of the image captured on the mth viewpoint, n=1,2,...,N;
步骤S25，对预处理后的图像进行非线性映射，得到对应的各预设视点的光场图像Step S25, to the preprocessed image Perform nonlinear mapping to obtain light field images of corresponding preset viewpoints
本实施例中，优化矩阵w的计算方法如下：In this embodiment, the calculation method of the optimization matrix w is as follows:
如图3所示，由于对于一个固定的显示设备来说，光栅面板和LCD面板的物理位置是不变的，所以显示II1～时，从LCD面板上射出的H×W×3条光线到N个捕获视点的光路，和显示II2～时，从LCD面板上射出的H×W×3条光线的光路是一样的。所有的光路都可以通过反向映射方法计算得到。我们用一个查找表LUT1来保存从LCD面板上的子像素点到观看平面的映射关系，如公式(1)所示：As shown in Figure 3, since the physical positions of the grating panel and the LCD panel are unchanged for a fixed display device, when displaying II1~ , the H×W×3 rays emitted from the LCD panel arrive at The optical paths of the N capturing viewpoints are the same as the optical paths of H×W×3 rays emitted from the LCD panel when II2~ is displayed. All light paths can be calculated by the reverse mapping method. We use a lookup table LUT1 to save the mapping relationship from the sub-pixels on the LCD panel to the viewing plane, as shown in formula (1):
L～＝LUT1(II～) (1)L～ ＝LUT1 (II～ ) (1)
在我们的算法中，实际光场L～被表示成一个维度为(H×W×3，N)的二维矩阵。相应地，查找表从LUT1变成了LUT2。对于矩阵的每一列来说，记录的是第m个视角点的光场，记录的是捕获后计算得到的光场图片。In our algorithm, the actual light field L∼ is represented as a two-dimensional matrix with dimension (H×W×3, N). Accordingly, the lookup table changes from LUT1 to LUT2 . For each column of the matrix For example, what is recorded is the light field at the mth viewpoint, What is recorded is a picture of the light field computed after capture.
我们希望找到一个优化矩阵w使得min||L-wL～||2，引入一个高斯白噪向量v，其方差为σ2，寻找一个w使得似然方程P(L|w)最大化。lm和分别表示的是L和L～的第m列。由于lm是相互独立的，所以似然方程如公式(2)所示：We hope to find an optimized matrix w such that min||L-wL～ ||2 , introduce a Gaussian white noise vector v with variance σ2 , and find a w that maximizes the likelihood equation P(L|w). lm and represent the mth column of L and L~ , respectively. Since lm is independent of each other, the likelihood equation is shown in formula (2):
又由于所以公式(2)可以由公式(3)所示的方法计算出来：And because of So formula (2) can be calculated by the method shown in formula (3):
其中，代表第m个视点的光场，我们从100次相机光场捕获中估计一个的概率分布。用函数来近似的概率分布β可以通过参数估计的方法得到，于是公式(3)可以改写为公式(4)：in, Representing the light field of the mth viewpoint, we estimate one from 100 camera light field captures probability distribution. use function to approximate The probability distribution of β can be obtained by parameter estimation, so formula (3) can be rewritten as formula (4):
其中，c和c′代表的是两个不同的常数。于是整个问题变成求公式(5)所示的最小值：Among them, c and c' represent two different constants. The whole problem then becomes to find the minimum value shown in formula (5):
我们用迭代的方法来解决公式(5)，如公式(6)所示：We use an iterative method to solve Equation (5), as shown in Equation (6):
一旦优化矩阵w被计算出来，优化后的合成索引图II可以通过反向搜索的方法获得，如公式(7)所示：Once the optimization matrix w is calculated, the optimized synthetic index map II can be obtained by the reverse search method, as shown in formula (7):
本实施例中，所述感兴趣区域提取的方法为：对捕获图像进行反畸变处理后，提取图像上的3D显示屏所在区域作为感兴趣区域；In this embodiment, the method for extracting the region of interest is: after performing anti-distortion processing on the captured image, extract the region where the 3D display screen on the image is located as the region of interest;
所述2D投影变化的方法：提取所述感兴趣区域的4个顶点以及对应视点反畸变后的光场图像(正规矩形)的4个顶点：(0,0)、计算得到单应矩阵；通过所述单应矩阵对捕获图像进行2D投影变化。The method for changing the 2D projection: extracting the 4 vertices of the region of interest and the 4 vertices of the light field image (regular rectangle) after dedistorting the corresponding viewpoint: (0, 0), A homography matrix is obtained through calculation; 2D projection changes are performed on the captured image through the homography matrix.
本实施例中，步骤S2中所述非线性映射，如公式(8)所示：In this embodiment, the nonlinear mapping described in step S2 is as shown in formula (8):
X＝Mr_sp-1(Vsub) (8)X＝Mr_sp-1 (Vsub ) (8)
用于从预处理后的图像子像素值Vsub恢复出对应光线的亮度值X。It is used to restore the brightness value X of the corresponding light from the preprocessed image sub-pixel value Vsub .
本实施例中，非线性映射函数Mr_sp的拟合方法如下：In this embodiment, the fitting method of the nonlinear mapping function Mr_sp is as follows:
在该部分中我们将介绍从光线亮度值(光场图像)到相机捕获图像子像素值Mr_sp映射函数的恢复算法，即公式(9)：In this section we will introduce the value of light intensity (light field image ) to the camera to capture the image The restoration algorithm of the sub-pixel value Mr_sp mapping function, that is, the formula (9):
我们对同一个场景进行同曝光度的相机采集(光线的变化可以忽略)。在采集过后，我们得到每张捕获图像的子像素数值zi。我们可以对每一条原始光线亮度值li写出非线性方程如公式(10)所示：We capture the same scene with the camera at the same exposure (the change in light can be ignored). After acquisition, we get the sub-pixel value zi of each captured image. We can write a nonlinear equation for each original light luminance value li as shown in formula (10):
zi＝Mr_sp(li) (10)zi =Mr_sp (li ) (10)
假设Mr_sp是光滑的，i为从0开始的所有子像素的序号，zi和li是已知的，未知的是方程Mr_sp。我们希望能够估计一个Mr_sp函数，使其最大化地满足公式(10)。因此，最小化如公式(11)所示的二次目标函数(非线性方程的参数估计，找到一个Mr_sp，使得E最小)：Assuming that Mr_sp is smooth, i is the serial number of all sub-pixels starting from 0, zi and li are known, and the unknown is the equation Mr_sp . We hope to be able to estimate a Mr_sp function such that it satisfies formula (10) maximally. Therefore, to minimize the quadratic objective function shown in formula (11) (parameter estimation of nonlinear equations, find a Mr_sp that minimizes E):
其中，N代表的是子像素的个数，α是相对于数据拟合项对平滑项进行的加权。我们用一个三参函数t(x)＝a+bxc去近似刻画Mr_sp。参数a、b、c的具体数值根据相机采集时的光照环境、相机参数(如色相、饱和度等有关)，用非线性最小二乘法估计得到，在本实施例中，a＝0.13，b＝1.24，c＝0.92。Among them, N represents the number of sub-pixels, and α is the weighting of the smoothing item relative to the data fitting item. We use a three-parameter function t(x)=a+bxc to describe Mr_sp approximately. The specific numerical values of parameters a, b, c are estimated according to the lighting environment and camera parameters (such as hue, saturation, etc.) when the camera is collected, and are estimated by the non-linear least squares method. In this embodiment, a=0.13, b= 1.24, c=0.92.
由于在取值范围中部的像素值更可靠，并且Mr_sp在饱和值附近变为单一的。我们增加一个加权函数k(z)，k(z)在像素值范围的两头衰减为0，其中zmin＝0，zmax＝255，如公式(12)所示：Since the pixel values in the middle of the value range are more reliable, and Mr_sp becomes unitary around the saturation value. We add a weighting function k(z), k(z) decays to 0 at both ends of the pixel value range, where zmin =0, zmax =255, as shown in formula (12):
将其加入到(11)中可以得到公式(13)：Adding this to (11) yields formula (13):
其中，γ1和γ2为权重系数。计算出满足公式(13)的Mr_sp函数，即为我们所需要的非线性映射函数。Among them, γ1 and γ2 are weight coefficients. Calculate the Mr_sp function that satisfies the formula (13), which is the nonlinear mapping function we need.
本实施例中，所述利用所生成的目标合成索引图对三维图像进行渲染，如公式(14)所示：In this embodiment, the three-dimensional image is rendered by using the generated target composite index map, as shown in formula (14):
其中，IIm为第m个视点上的目标合成索引图，Viewm为通过OpenGL或3d Max渲染出的三维模型的第m个视角图，IIdisplay为最终显示在3D显示器上的合成图。Wherein, IIm is the composite index image of the target on the mth viewpoint, Viewm is the mth view image of the 3D model rendered by OpenGL or 3d Max, and IIdisplay is the composite image finally displayed on the 3D display.
本领域技术人员应该能够意识到，结合本文中所公开的实施例描述的各示例的方法步骤，能够以电子硬件、计算机软件或者二者的结合来实现，为了清楚地说明电子硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各示例的组成及步骤。这些功能究竟以电子硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。本领域技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本发明的范围。Those skilled in the art should be able to realize that the method steps described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the possibility of electronic hardware and software For interchangeability, in the above description, the composition and steps of each example have been generally described according to their functions. Whether these functions are performed by electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may implement the described functionality using different methods for each particular application, but such implementation should not be considered as exceeding the scope of the present invention.
至此，已经结合附图所示的优选实施方式描述了本发明的技术方案，但是，本领域技术人员容易理解的是，本发明的保护范围显然不局限于这些具体实施方式。在不偏离本发明的原理的前提下，本领域技术人员可以对相关技术特征作出等同的更改或替换，这些更改或替换之后的技术方案都将落入本发明的保护范围之内。So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings, but those skilled in the art will easily understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to relevant technical features, and the technical solutions after these changes or substitutions will all fall within the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710329750.7ACN106991715A (en) | 2017-05-11 | 2017-05-11 | Grating prism Three-dimensional Display rendering intent based on optical field acquisition |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710329750.7ACN106991715A (en) | 2017-05-11 | 2017-05-11 | Grating prism Three-dimensional Display rendering intent based on optical field acquisition |
| Publication Number | Publication Date |
|---|---|
| CN106991715Atrue CN106991715A (en) | 2017-07-28 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710329750.7APendingCN106991715A (en) | 2017-05-11 | 2017-05-11 | Grating prism Three-dimensional Display rendering intent based on optical field acquisition |
| Country | Link |
|---|---|
| CN (1) | CN106991715A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107563088A (en)* | 2017-09-14 | 2018-01-09 | 北京邮电大学 | A kind of light field display device emulation mode based on Ray Tracing Algorithm |
| CN108495117A (en)* | 2018-03-29 | 2018-09-04 | 香港光云科技有限公司 | Holographic image multi-view processing conversion and display method and equipment |
| CN112750156A (en)* | 2020-03-16 | 2021-05-04 | 奕目(上海)科技有限公司 | Light field imaging system, processing method and device |
| CN113808271A (en)* | 2020-06-17 | 2021-12-17 | 广东博智林机器人有限公司 | Rendering method and device of three-dimensional model, electronic equipment and storage medium |
| CN119200252A (en)* | 2024-10-11 | 2024-12-27 | 深圳清华大学研究院 | Method for correcting multi-display light field fusion conflict |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101800906A (en)* | 2009-02-11 | 2010-08-11 | 中国科学院自动化研究所 | True three-dimensional imaging display screen |
| CN101982741A (en)* | 2010-09-08 | 2011-03-02 | 北京航空航天大学 | Underwater light field sampling and simulating method |
| US8044994B2 (en)* | 2006-04-04 | 2011-10-25 | Mitsubishi Electric Research Laboratories, Inc. | Method and system for decoding and displaying 3D light fields |
| CN103345771A (en)* | 2013-06-28 | 2013-10-09 | 中国科学技术大学 | Efficient image rendering method based on modeling |
| CN106257995A (en)* | 2016-07-25 | 2016-12-28 | 深圳大学 | A kind of light field three-D imaging method and system thereof |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8044994B2 (en)* | 2006-04-04 | 2011-10-25 | Mitsubishi Electric Research Laboratories, Inc. | Method and system for decoding and displaying 3D light fields |
| CN101800906A (en)* | 2009-02-11 | 2010-08-11 | 中国科学院自动化研究所 | True three-dimensional imaging display screen |
| CN101982741A (en)* | 2010-09-08 | 2011-03-02 | 北京航空航天大学 | Underwater light field sampling and simulating method |
| CN103345771A (en)* | 2013-06-28 | 2013-10-09 | 中国科学技术大学 | Efficient image rendering method based on modeling |
| CN106257995A (en)* | 2016-07-25 | 2016-12-28 | 深圳大学 | A kind of light field three-D imaging method and system thereof |
| Title |
|---|
| PEI R ET AL.: "A Real-Time Depth Map Refinement and Disparity Ranges Expansion System(DRDE) for Multiview Rendering", 《THE 33D COMPUTER GRAPHICS INTERNATIONAL》* |
| PEI R,ET AL: "Three-dimensional lenticular display synthetic image rendering based on light field acquisition", 《 JOURNAL OF THE SOCIETY FOR INFORMATION DISPLAY》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107563088A (en)* | 2017-09-14 | 2018-01-09 | 北京邮电大学 | A kind of light field display device emulation mode based on Ray Tracing Algorithm |
| CN107563088B (en)* | 2017-09-14 | 2019-10-01 | 北京邮电大学 | A kind of light field display device simulating method based on Ray Tracing Algorithm |
| CN108495117A (en)* | 2018-03-29 | 2018-09-04 | 香港光云科技有限公司 | Holographic image multi-view processing conversion and display method and equipment |
| CN112750156A (en)* | 2020-03-16 | 2021-05-04 | 奕目(上海)科技有限公司 | Light field imaging system, processing method and device |
| CN112750156B (en)* | 2020-03-16 | 2022-09-09 | 奕目(上海)科技有限公司 | Light field imaging system, processing method and device |
| CN113808271A (en)* | 2020-06-17 | 2021-12-17 | 广东博智林机器人有限公司 | Rendering method and device of three-dimensional model, electronic equipment and storage medium |
| CN119200252A (en)* | 2024-10-11 | 2024-12-27 | 深圳清华大学研究院 | Method for correcting multi-display light field fusion conflict |
| Publication | Publication Date | Title |
|---|---|---|
| EP2786583B1 (en) | Image processing apparatus and method for subpixel rendering | |
| JP5036132B2 (en) | Critical alignment of parallax images for autostereoscopic display | |
| US8189035B2 (en) | Method and apparatus for rendering virtual see-through scenes on single or tiled displays | |
| CN105611278B (en) | The image processing method and system and display equipment of anti-bore hole 3D viewings spinning sensation | |
| CN108513123B (en) | Image array generation method for integrated imaging light field display | |
| CN106991715A (en) | Grating prism Three-dimensional Display rendering intent based on optical field acquisition | |
| US10229528B2 (en) | Method for visualizing three-dimensional images on a 3D display device and 3D display device | |
| US20180184066A1 (en) | Light field retargeting for multi-panel display | |
| CN102360489B (en) | Method and device for realizing conversion from two-dimensional image to three-dimensional image | |
| TW201243763A (en) | Method for 3D video content generation | |
| CN105763865A (en) | Naked eye 3D augmented reality method and device based on transparent liquid crystals | |
| US11172190B2 (en) | Stereo weaving for head-tracked autostereoscopic displays | |
| US20080238930A1 (en) | Texture processing apparatus, method and program | |
| CN101729920A (en) | Method for displaying stereoscopic video with free visual angles | |
| US20120212477A1 (en) | Fast Haze Removal and Three Dimensional Depth Calculation | |
| CN108573521A (en) | Real-time interactive glasses-free 3D display method based on CUDA parallel computing framework | |
| CN115482323A (en) | Stereoscopic video parallax control and editing method based on nerve radiation field | |
| CN116708746A (en) | Naked eye 3D-based intelligent display processing method | |
| US10277879B2 (en) | 3D system including rendering with eye displacement | |
| US9897806B2 (en) | Generation of three-dimensional imagery to supplement existing content | |
| CN108833893A (en) | A 3D Image Correction Method Based on Light Field Display | |
| CN112866676B (en) | A naked-eye 3D display algorithm based on single-pixel multi-view reconstruction | |
| Seitner et al. | Trifocal system for high-quality inter-camera mapping and virtual view synthesis | |
| US10148933B2 (en) | 3D system including rendering with shifted compensation | |
| KR101567002B1 (en) | Computer graphics based stereo floting integral imaging creation system |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | Application publication date:20170728 | |
| WD01 | Invention patent application deemed withdrawn after publication |