CN114820980A - Three-dimensional reconstruction method and device, electronic equipment and readable storage medium - Google Patents
Three-dimensional reconstruction method and device, electronic equipment and readable storage mediumDownload PDFInfo
- Publication number
- CN114820980A CN114820980ACN202210439570.5ACN202210439570ACN114820980ACN 114820980 ACN114820980 ACN 114820980ACN 202210439570 ACN202210439570 ACN 202210439570ACN 114820980 ACN114820980 ACN 114820980A
- Authority
- CN
- China
- Prior art keywords
- patch
- texture information
- mesh model
- dimensional
- dimensional reconstruction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
- G06T17/205—Re-meshing
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/04—Texture mapping
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Geometry (AREA)
- Software Systems (AREA)
- Image Generation (AREA)
- Processing Or Creating Images (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本申请属于图像数据处理技术领域，具体涉及一种三维重建方法、三维重建装置、电子设备和可读存储介质。The present application belongs to the technical field of image data processing, and in particular relates to a three-dimensional reconstruction method, a three-dimensional reconstruction device, an electronic device and a readable storage medium.
背景技术Background technique
现有技术中，对目标场景进行三维网格模型是展示场景最直观的方法。在对场景进行数据采集的过程中，由于图像采集设备的限制(离散有限固定采集点)，往往会由于遮挡和自遮挡的问题，以及有限采集点的个数限制，导致采集的图像数据的不完备，无法构建完整的三维网格模型。In the prior art, performing a three-dimensional mesh model on the target scene is the most intuitive method for displaying the scene. In the process of data collection of the scene, due to the limitation of the image collection equipment (discrete limited fixed collection points), often due to the problems of occlusion and self-occlusion, as well as the limitation of the number of limited collection points, the collected image data is not accurate. Complete, unable to build a complete 3D mesh model.
发明内容SUMMARY OF THE INVENTION
本申请实施例的目的是提供一种三维重建方法、三维重建装置、电子设备和可读存储介质，在采集的图像数据(纹理信息)不完备的情况下，通过构建虚拟点位，估算缺失的纹理信息，将估算的纹理信息填入三维网格模型中，从而有效的补全纹理信息，保证模型构建的完整性。The purpose of the embodiments of the present application is to provide a three-dimensional reconstruction method, a three-dimensional reconstruction device, an electronic device, and a readable storage medium. In the case where the collected image data (texture information) is incomplete, by constructing virtual points, it is possible to estimate the missing image data (texture information). Texture information, fill the estimated texture information into the 3D mesh model, so as to effectively complement the texture information and ensure the integrity of the model construction.
第一方面，本申请实施例提供了一种三维重建方法，三维重建方法包括：为三维网格模型中的多个面片匹配纹理信息，在三维网格模型中的第一面片未匹配到对应的纹理信息的情况下，确定第一虚拟点位；基于第一虚拟点位渲染三维网格模型的第一全景图，并确定第一全景图中的第一区域，第一区域与第一面片对应；根据第一全景图和第一区域确定目标纹理信息，将目标纹理信息赋值至第一面片。In a first aspect, an embodiment of the present application provides a three-dimensional reconstruction method. The three-dimensional reconstruction method includes: matching texture information for multiple patches in the three-dimensional mesh model, and the first patch in the three-dimensional mesh model is not matched to In the case of the corresponding texture information, determine the first virtual point; render the first panorama of the three-dimensional mesh model based on the first virtual point, and determine the first area in the first panorama, the first area and the first The patch corresponds; the target texture information is determined according to the first panorama image and the first area, and the target texture information is assigned to the first patch.
第二方面，本申请实施例提供了一种三维重建装置，三维重建装置包括：处理单元，用于为三维网格模型中的多个面片匹配纹理信息，在三维网格模型中的第一面片未匹配到对应的纹理信息的情况下，确定第一虚拟点位；处理单元，还用于基于第一虚拟点位渲染三维网格模型的第一全景图，并确定第一全景图中的第一区域，第一区域与第一面片对应；处理单元，还用于根据第一全景图和第一区域确定目标纹理信息，将目标纹理信息赋值至第一面片。In a second aspect, an embodiment of the present application provides a three-dimensional reconstruction device, where the three-dimensional reconstruction device includes: a processing unit configured to match texture information for a plurality of patches in a three-dimensional mesh model. In the case where the patch does not match the corresponding texture information, determine the first virtual point; the processing unit is further configured to render the first panorama of the 3D mesh model based on the first virtual point, and determine the first panorama The first area of the , the first area corresponds to the first patch; the processing unit is further configured to determine target texture information according to the first panorama and the first area, and assign the target texture information to the first patch.
第三方面，本申请实施例提供了一种电子设备，包括存储器，处理器及存储在存储器上并可在处理器上运行的程序或指令，程序或指令被处理器执行时实现如第一方面的三维重建方法的步骤。In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the first aspect is implemented The steps of the 3D reconstruction method.
第四方面，本申请实施例提供了一种可读存储介质，该可读存储介质上存储程序或指令，该程序或指令被处理器执行时实现如第一方面的三维重建方法的步骤。In a fourth aspect, an embodiment of the present application provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or instruction is executed by a processor, the steps of the three-dimensional reconstruction method of the first aspect are implemented.
第五方面，本申请实施例提供了一种芯片，该芯片包括处理器和通信接口，该通信接口和该处理器耦合，该处理器用于运行程序或指令，实现如第一方面的三维重建方法的步骤。In a fifth aspect, an embodiment of the present application provides a chip, the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement the three-dimensional reconstruction method of the first aspect A step of.
第六方面，本申请实施例提供一种计算机程序产品，该程序产品被存储在存储介质中，该程序产品被至少一个处理器执行以实现如第一方面的三维重建方法。In a sixth aspect, embodiments of the present application provide a computer program product, where the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the three-dimensional reconstruction method according to the first aspect.
本申请实施例中，在对目标场景进行三维网格模型的重建过程中，对模型中缺少纹理信息的第一面片，构建第一虚拟点位，根据在第一虚拟点位获取的第一全景图和第一全景图中与缺少纹理信息的面片对应的第一区域，重新估算第一面片的目标纹理信息，将估算的目标纹理信息填入三维网格模型，进而补全三维网格模型中缺少的纹理信息，从而有效的保证三维网格模型的完整性，提升了目标场景三维网格模型的展示效果。In the embodiment of the present application, in the process of reconstructing the 3D mesh model of the target scene, a first virtual point is constructed for the first patch lacking texture information in the model. In the panorama and the first area in the first panorama corresponding to the patch lacking texture information, re-estimate the target texture information of the first patch, fill the estimated target texture information into the 3D mesh model, and then complete the 3D mesh The texture information missing in the grid model can effectively ensure the integrity of the 3D grid model and improve the display effect of the 3D grid model of the target scene.
附图说明Description of drawings
图1示出了本申请实施例提供的三维重建方法的流程示意图之一；FIG. 1 shows one of the schematic flowcharts of the three-dimensional reconstruction method provided by the embodiment of the present application;
图2示出了本申请实施例提供的三维重建方法的流程示意图之二；FIG. 2 shows the second schematic flowchart of the three-dimensional reconstruction method provided by the embodiment of the present application;
图3示出了本申请实施例提供的三维重建方法的示意图之一；FIG. 3 shows one of the schematic diagrams of the three-dimensional reconstruction method provided by the embodiment of the present application;
图4示出了本申请实施例提供的三维重建方法的示意图之二；FIG. 4 shows the second schematic diagram of the three-dimensional reconstruction method provided by the embodiment of the present application;
图5示出了本申请实施例提供的三维重建方法的示意图之三；FIG. 5 shows the third schematic diagram of the three-dimensional reconstruction method provided by the embodiment of the present application;
图6示出了本申请实施例提供的三维重建方法的示意图之四；FIG. 6 shows the fourth schematic diagram of the three-dimensional reconstruction method provided by the embodiment of the present application;
图7示出了本申请实施例提供的三维重建方法的示意图之五；FIG. 7 shows the fifth schematic diagram of the three-dimensional reconstruction method provided by the embodiment of the present application;
图8示出了本申请实施例提供的三维重建方法的流程示意图之三；FIG. 8 shows the third schematic flowchart of the three-dimensional reconstruction method provided by the embodiment of the present application;
图9示出了本申请实施例提供的三维重建装置的结构框图；FIG. 9 shows a structural block diagram of a three-dimensional reconstruction apparatus provided by an embodiment of the present application;
图10示出了本申请实施例提供的电子设备的结构框图；FIG. 10 shows a structural block diagram of an electronic device provided by an embodiment of the present application;
图11示出了本申请实施例的一种电子设备的硬件结构示意图。FIG. 11 shows a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present application.
具体实施方式Detailed ways
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员获得的所有其他实施例，都属于本申请保护的范围。The technical solutions in the embodiments of the present application will be clearly described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art fall within the protection scope of this application.
本申请的说明书和权利要求书中的术语“第一”、“第二”等是用于区别类似的对象，而不用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施，且“第一”、“第二”等所区分的对象通常为一类，并不限定对象的个数，例如第一对象可以是一个，也可以是多个。此外，说明书以及权利要求中“和/或”表示所连接对象的至少其中之一，字符“/”，一般表示前后关联对象是一种“或”的关系。The terms "first", "second" and the like in the description and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the present application can be practiced in sequences other than those illustrated or described herein, and distinguish between "first", "second", etc. The objects are usually of one type, and the number of objects is not limited. For example, the first object may be one or more than one. In addition, "and/or" in the description and claims indicates at least one of the connected objects, and the character "/" generally indicates that the associated objects are in an "or" relationship.
为了便于理解，下面对本申请涉及的部分技术术语进行简要介绍：For ease of understanding, some technical terms involved in this application are briefly introduced below:
三维网格模型：三维网格模型是物体的多边形表示，通常用计算机或者其它视频设备进行显示。显示的物体可以是现实世界的实体，也可以是虚构的物体。任何物理自然界存在的东西都可以用三维模型表示。本申请实施例中，物体的三维模型用于指示目标场景(如：房屋)的三维结构和尺寸信息。三维模型的数据存储形式有多种，例如以三维点云、网格或体元等形式表示，具体此处不做限定。3D mesh model: A 3D mesh model is a polygonal representation of an object, usually displayed on a computer or other video device. The displayed objects can be real-world entities or fictitious objects. Anything that exists in physical nature can be represented by a 3D model. In this embodiment of the present application, the three-dimensional model of the object is used to indicate the three-dimensional structure and size information of the target scene (eg, house). There are various data storage forms for the 3D model, for example, it is represented in the form of a 3D point cloud, a mesh, or a voxel, which is not specifically limited here.
面片：面片是指三维网格模型中最小的平面构成单元。通常在渲染中，需要将空间中的模型划分成无数个微小的平面。这些平面又被称为面片，它们可以是任意多边形，常用的是三角形和四边形。这些面片各条边的交点则是各个面片的顶点。面片可以是根据模型的材质或颜色等信息随机划分的。Patch: Patch refers to the smallest planar constituent unit in a 3D mesh model. Usually in rendering, the model in space needs to be divided into countless tiny planes. These planes, also known as patches, can be arbitrary polygons, but triangles and quadrilaterals are commonly used. The intersections of the edges of these patches are the vertices of each patch. Patches can be randomly divided according to information such as the material or color of the model.
全景图：广义的全景图是指广角图，即视角较大的图像。全景图可以通过不同的投影方式实现，常见的包括：等角投影、等矩形投影(equirectangular projection)、正交投影和等积投影等，具体此处不做限定。Panorama: A generalized panorama refers to a wide-angle image, that is, an image with a larger viewing angle. The panorama can be realized by different projection methods, and common ones include: equiangular projection, equirectangular projection, orthogonal projection, and equal-area projection, etc., which are not specifically limited here.
下面结合附图1至图11，通过具体的实施例及其应用场景对本申请实施例提供的三维重建方法、三维重建装置、电子设备和可读存储介质进行详细地说明。The three-dimensional reconstruction method, the three-dimensional reconstruction apparatus, the electronic device, and the readable storage medium provided by the embodiments of the present application are described in detail below with reference to FIGS. 1 to 11 through specific embodiments and application scenarios.
在本申请实施例提供了一种三维重建方法，图1示出了本申请实施例提供的三维重建方法的流程示意图之一，如图1所示，三维重建方法包括：An embodiment of the present application provides a three-dimensional reconstruction method. FIG. 1 shows one of the schematic flowcharts of the three-dimensional reconstruction method provided by the embodiment of the present application. As shown in FIG. 1 , the three-dimensional reconstruction method includes:
步骤102：为三维网格模型中的多个面片匹配纹理信息，在三维网格模型中的第一面片未匹配到对应的纹理信息的情况下，确定第一虚拟点位。Step 102: Match texture information for multiple patches in the three-dimensional mesh model, and determine a first virtual point in the case that the first patch in the three-dimensional mesh model does not match the corresponding texture information.
具体而言，三维网格模型中具有多个面片，在三维网格模型构建完成后，需要对三维网格模型进行纹理映射，在纹理映射的过程中，需要为三维网格模型中的每一个面片匹配纹理信息。Specifically, there are multiple patches in the 3D mesh model. After the 3D mesh model is constructed, texture mapping needs to be performed on the 3D mesh model. A patch matches texture information.
具体地，可以通过纹理估计算法，为三维网格模型中的多个面片匹配纹理信息。Specifically, texture information can be matched for multiple patches in the three-dimensional mesh model through a texture estimation algorithm.
进一步地，多个面片中的第一面片可以为三维网格模型中缺少纹理信息的面片。通常情况下，上述第一面片的数量为多个，多个第一面片缺少纹理信息可能会导致纹理映射后的三维网格模型表达不完整。因此，需要对缺失纹理信息的第一面片进行填充赋值，以使得三维网格模型更加完整。Further, the first patch of the multiple patches may be a patch lacking texture information in the three-dimensional mesh model. Usually, the number of the above-mentioned first patches is multiple, and the lack of texture information of multiple first patches may result in incomplete expression of the 3D mesh model after texture mapping. Therefore, it is necessary to fill and assign the first patch with missing texture information to make the 3D mesh model more complete.
进一步地，第一面片可以为三维点集合P(x’,y’,z’)，集合内点均无法正确估计出纹理信息。Further, the first patch may be a three-dimensional point set P(x', y', z'), and the texture information cannot be correctly estimated for the points in the set.
进一步地，由于第一面片缺失纹理信息，可以通过每个第一面片的空间参数信息，为每个第一面片确定出一个虚拟点位，上述第一虚拟点位为三维网格模型中的视点，通过该第一虚拟点位的坐标位置，可以较好地观测到三维网格模型中的大部分面片，其中包括与该第一虚拟点位对应的缺失纹理信息的第一面片。Further, since the first patch lacks texture information, a virtual point can be determined for each first patch through the spatial parameter information of each first patch, and the above-mentioned first virtual point is a three-dimensional grid model. Through the coordinate position of the first virtual point, most of the patches in the 3D mesh model can be better observed, including the first surface with missing texture information corresponding to the first virtual point. piece.
可以理解，如果上述虚拟点位中，存在一个虚拟点位能够观测到缺失纹理信息的全部第一面片，则可以将该虚拟点位确定为第一虚拟点位，也即该一个第一虚拟点位就可以观测到所有缺失纹理的面片。如果上述虚拟点位中，没有一个虚拟点位能够观察到所有缺失纹理的全部第一面片，则需要确定出多个第一虚拟点位，以保证所有缺失纹理的第一面片都能够被观测到。It can be understood that if there is a virtual point in the above virtual point that can observe all the first patches with missing texture information, the virtual point can be determined as the first virtual point, that is, the first virtual point. All patches with missing textures can be observed with a single click. If none of the above virtual points can observe all the first patches of all missing textures, multiple first virtual points need to be determined to ensure that all the first patches of missing textures can be viewed observed.
需要说明的是，无论缺失纹理信息的第一面片对应的是一个或者多个第一虚拟点位，仅影响后续获取第一全景图的次数及估算缺失纹理信息的次数，本申请实施例中仅就多个缺失纹理的第一面片对应一个第一虚拟点位的情况进行说明，对通过多个第一虚拟点位估算缺失纹理信息的方式不再赘述。It should be noted that, regardless of whether the first patch with missing texture information corresponds to one or more first virtual points, it only affects the number of subsequent acquisitions of the first panorama and the estimated number of times of missing texture information. Only the case where the first patches of multiple missing textures correspond to one first virtual point will be described, and the method of estimating missing texture information by using the multiple first virtual points will not be described again.
步骤104：基于第一虚拟点位渲染三维网格模型的第一全景图，并确定第一全景图中的第一区域；Step 104: Rendering a first panorama of the three-dimensional mesh model based on the first virtual point, and determining a first area in the first panorama;
其中，第一区域与第一面片对应；Wherein, the first area corresponds to the first patch;
需要说明的是，第一全景图包括三维网格模型对应的360度全景图像，全景图可以分为多个区域，第一全景图中的多个区域和三维网格模型中的多个面片具有对应关系。即第一全景图中的第一区域与三维网格模型中的第一面片存在对应关系。It should be noted that the first panorama image includes a 360-degree panoramic image corresponding to the 3D mesh model, and the panorama image can be divided into multiple areas, multiple areas in the first panoramic image and multiple patches in the 3D mesh model. have a corresponding relationship. That is, there is a corresponding relationship between the first area in the first panorama and the first patch in the 3D mesh model.
进一步地，对上述第一全景图的尺寸进行限定，具体包括：第一全景图宽度，即纬度，范围[0,2×Pai]，第一全景图高度，即经度，范围[-Pai/2,Pai/2]，其中，Pai＝3.1415926。Further, the size of the first panorama image is limited, specifically including: the width of the first panorama image, that is, the latitude, the range [0,2×Pai], the height of the first panoramic image, that is, the longitude, the range [-Pai/2 , Pai/2], where Pai=3.1415926.
进一步地，在上述第一全景图中计算第一区域的公式为：Further, the formula for calculating the first area in the above-mentioned first panorama is:
PY＝arcsin(P_z’/R)；PY =arcsin(P_z'/R);
Px＝arcsin(P_x’/R×cos(PY))；Px =arcsin(P_x'/R×cos(PY ));
其中，R为第一全景图绘制时候的半径，(P_x’,P_y’,P_z’)为第一面片的每一个点的坐标值，(Px,Py)为第一区域内一个点的坐标值。Among them, R is the radius when the first panorama is drawn, (P_x', P_y', P_z') is the coordinate value of each point of the first patch, (Px, Py) is the coordinate of a point in the first area value.
可以理解，基于第一虚拟点位获取上述三维网格模型的第一全景图，能够更准确地体现第一面片的位置信息，也能够确定缺失纹理信息的第一面片对目标场景图像造成的空洞情况。It can be understood that obtaining the first panorama of the above-mentioned three-dimensional mesh model based on the first virtual point can more accurately reflect the position information of the first patch, and can also determine the effect of the first patch missing texture information on the target scene image. empty situation.
步骤106：根据第一全景图和第一区域确定目标纹理信息，将目标纹理信息赋值至第一面片。Step 106: Determine target texture information according to the first panorama image and the first area, and assign the target texture information to the first patch.
具体地，上述目标纹理信息中包括深度数据和彩色数据。Specifically, the above-mentioned target texture information includes depth data and color data.
可以理解，通过第一虚拟点位渲染得到的上述第一全景图，以及从第一全景图中多个区域中确定出的第一区域，经过解析和估算，可以确定出目标纹理信息，并使用目标纹理信息为三维网格模型中的第一面片进行赋值，以填补纹理信息缺失造成的空洞区域，从而使三维网格模型在显示上更加完整美观。示例性地，如图6所示，在目标场景的三维网格模型中，图像中“床”的部分区域(黑色区域)缺少纹理信息，将缺少纹理信息的区域对应的面片设为第一面片(目标面片)。基于第一面片构建第一虚拟点位，在第一虚拟点位的坐标位置，对目标场景的三维网格模型重新渲染出360度的第一全景图，第一全景图中包括图6中的“床”，在该第一全景图内的黑色区域，即为与第一面片对应的第一区域。It can be understood that the above-mentioned first panorama image obtained by rendering through the first virtual point, and the first region determined from multiple regions in the first panorama map, through analysis and estimation, can determine the target texture information, and use The target texture information is assigned to the first patch in the 3D mesh model to fill in the hollow area caused by the lack of texture information, so that the 3D mesh model is more complete and beautiful in display. Exemplarily, as shown in Figure 6, in the 3D mesh model of the target scene, the partial area (black area) of the "bed" in the image lacks texture information, and the patch corresponding to the area lacking texture information is set as the first area. patch (target patch). A first virtual point is constructed based on the first patch, and at the coordinate position of the first virtual point, a 360-degree first panorama is re-rendered to the 3D mesh model of the target scene. The first panorama includes the one shown in FIG. 6 . , the black area in the first panorama is the first area corresponding to the first patch.
进一步示例性地，在重新渲染的第一全景图中确定第一面片对应的第一区域(黑色区域)，根据上述第一全景图和第一区域等信息，估算出第一面片的目标纹理信息，将估算的目标纹理信息重新填入目标场景的三维网格模型的第一面片中。将目标纹理信息填充到图6中的黑色空洞区域，填充效果如图6所示，对纹理信息缺失的区域填充目标纹理信息，使三维网格模型在显示上更加完整美观。Further exemplarily, the first area (black area) corresponding to the first patch is determined in the re-rendered first panorama, and the target of the first patch is estimated according to the above-mentioned information such as the first panorama and the first area. Texture information, refilling the estimated target texture information into the first patch of the 3D mesh model of the target scene. Fill the target texture information into the black hole area in Figure 6, the filling effect is shown in Figure 6, fill the target texture information to the area where the texture information is missing, so that the 3D mesh model is more complete and beautiful in display.
本申请实施例提供的三维重建方法通过对缺少纹理信息的第一面片构建第一虚拟点位，并重新渲染第一全景图，估算出对应的目标纹理信息填入三维网格模型的第一面片中。通过本申请提供的三维重建方法能够有效补充三维网格模型中缺少的纹理信息，保证了目标场景的三维网格模型中纹理信息的完整性。The three-dimensional reconstruction method provided by the embodiment of the present application constructs a first virtual point for the first patch lacking texture information, and re-renders the first panorama, and estimates that the corresponding target texture information is filled into the first surface of the three-dimensional mesh model. in the dough. The 3D reconstruction method provided by the present application can effectively supplement the texture information missing in the 3D mesh model, and ensure the integrity of the texture information in the 3D mesh model of the target scene.
在本申请的一些实施例中，上述确定第一虚拟点位的步骤102，具体包括：In some embodiments of the present application, the
步骤102a:根据第一面片的空间参数，确定出第一面片对应的第一虚拟点位。Step 102a: According to the spatial parameters of the first patch, determine the first virtual point corresponding to the first patch.
本申请实施例中，确定第一面片在三维网格模型内部的空间参数，空间参数包括第一面片在三维网格模型中的三维坐标参数。In this embodiment of the present application, the spatial parameters of the first patch inside the three-dimensional mesh model are determined, and the spatial parameters include three-dimensional coordinate parameters of the first patch in the three-dimensional mesh model.
具体地，根据第一面片在三维网格模型中的空间参数，计算出第一虚拟点位在三维网格模型中的空间坐标参数。Specifically, according to the spatial parameters of the first patch in the three-dimensional grid model, the spatial coordinate parameters of the first virtual point in the three-dimensional grid model are calculated.
进一步地，可以先确定第一面片的中心点在三维网格模型中的三维坐标值，根据该中心点的三维坐标值，计算出第一面片对应的第一虚拟点位的空间坐标参数。本申请实施例提供的三维重建方法根据第一面片的空间参数，确定出第一面片对应的虚拟点位，保证了第一虚拟点位定位的准确性，进而保证了在第一虚拟点位渲染得到第一全景图的显示效果。Further, the three-dimensional coordinate value of the center point of the first patch in the three-dimensional grid model can be determined first, and the spatial coordinate parameter of the first virtual point corresponding to the first patch can be calculated according to the three-dimensional coordinate value of the center point. . The three-dimensional reconstruction method provided by the embodiment of the present application determines the virtual point corresponding to the first patch according to the spatial parameters of the first patch, which ensures the accuracy of the positioning of the first virtual point, and further ensures that the first virtual point is located at the first virtual point. Bit rendering to obtain the display effect of the first panorama.
在本申请的一些实施例中，上述步骤102a，具体还可以包括：In some embodiments of the present application, the above step 102a may further include:
步骤1021:根据第一面片的法向量，确定第一虚拟点位；Step 1021: according to the normal vector of the first patch, determine the first virtual point;
其中，第一面片的中心至第一虚拟点位的向量以及第一面片的法向量的夹角小于90°。Wherein, the included angle between the center of the first patch and the vector of the first virtual point and the normal vector of the first patch is less than 90°.
在上述实施例中，可以根据第一面片在三维网格模型中的三维参数，确定第一面片的法向量，进而根据第一面片的法向量，计算出第一虚拟点位的空间坐标参数。In the above embodiment, the normal vector of the first patch can be determined according to the 3D parameters of the first patch in the 3D mesh model, and then the space of the first virtual point can be calculated according to the normal vector of the first patch Coordinate parameters.
进一步的，由第一面片的中心点坐标和第一虚拟点位坐标确定的向量与第一面片的法向量的夹角小于90°。Further, the included angle between the vector determined by the coordinates of the center point of the first patch and the coordinates of the first virtual point and the normal vector of the first patch is less than 90°.
可以理解，限定第一面片的中心点坐标和第一虚拟点位坐标确定的向量与第一面片的法向量的夹角，可以使得第一虚拟点位能够在更准确的视角渲染第一全景图。It can be understood that defining the angle between the vector determined by the coordinates of the center point of the first patch and the coordinates of the first virtual point and the normal vector of the first patch can enable the first virtual point to render the first patch at a more accurate perspective. Panorama.
具体地，上述第一虚拟点位的位置点并不唯一，第一虚拟点位可以位于第一面片的中心点法向量的延长线上，也可以在第一面片上其他点的法向量的延长线上，只要第一面片的中心至第一虚拟点位的向量以及第一面片的法向量的夹角小于90°的范围内的位置，都可以作为第一虚拟点位。Specifically, the position of the first virtual point is not unique. The first virtual point may be located on the extension line of the normal vector of the center point of the first patch, or may be on the normal vector of other points on the first patch. On the extension line, as long as the angle between the center of the first patch and the vector of the first virtual point and the angle between the normal vector of the first patch is less than 90°, it can be used as the first virtual point.
进一步地，第一虚拟点位的坐标计算公式为下：Further, the coordinate calculation formula of the first virtual point is as follows:
Vx＝D×avg(N_x’)+P_x’；Vx =D×avg(N_x')+P_x';
Vy＝D×avg(N_y’)+P_y’；Vy =D×avg(N_y')+P_y';
Vz＝D×avg(N_z’)+P_z’；Vz =D×avg(N_z')+P_z';
其中，(Vx,Vy,Vz)为第一虚拟点位的坐标值，(P_x’,P_y’,P_z’)为第一面片中心点的坐标值，(N_x’,N_y’,N_z’)为第一面片的法向量上一个点的坐标值，D为第一虚拟点位和第一面片中心点的空间距离。Among them, (Vx, Vy, Vz ) is the coordinate value of the first virtual point, (P_x', P_y', P_z') is the coordinate value of the center point of the first patch, (N_x', N_y', N_z' ) is the coordinate value of a point on the normal vector of the first patch, and D is the spatial distance between the first virtual point and the center point of the first patch.
进一步地，第一虚拟点位可以为上述范围内对应第一面片的数量最多的虚拟点位。Further, the first virtual point may be the virtual point corresponding to the largest number of first patches within the above range.
本申请实施例提供的三维重建方法，在目标场景的三维网格模型中，寻找到纹理缺失的第一面片，确定目标面片的法向量，根据目标面片的法向量，计算出目标面片对应的第一虚拟点位的坐标值，进而确定出第一面片对应的虚拟点位，以保证虚拟点位空间参数的准确性。In the 3D reconstruction method provided by the embodiment of the present application, in the 3D mesh model of the target scene, the first patch with missing texture is found, the normal vector of the target patch is determined, and the target surface is calculated according to the normal vector of the target patch The coordinate value of the first virtual point corresponding to the patch is determined, and the virtual point corresponding to the first patch is further determined, so as to ensure the accuracy of the spatial parameters of the virtual point.
在本申请的一些实施例中，图2示出了本申请实施例提供的三维重建方法的流程示意图之二，如图2所示，上述三维重建方法具体包括：In some embodiments of the present application, FIG. 2 shows the second schematic flowchart of the three-dimensional reconstruction method provided by the embodiment of the present application. As shown in FIG. 2 , the above-mentioned three-dimensional reconstruction method specifically includes:
步骤202：为三维网格模型中的多个面片匹配纹理信息，在三维网格模型中的第一面片未匹配到对应的纹理信息的情况下，确定第一虚拟点位；Step 202: Match texture information for a plurality of patches in the three-dimensional mesh model, and determine a first virtual point when the first patch in the three-dimensional mesh model does not match the corresponding texture information;
步骤204：基于第一虚拟点位渲染三维网格模型的第一全景图，并确定第一全景图中的第一区域，第一区域与第一面片对应；Step 204: Rendering a first panorama of the three-dimensional mesh model based on the first virtual point, and determining a first area in the first panorama, where the first area corresponds to the first patch;
步骤206，将第一区域渲染为目标颜色，在目标颜色对应的区域生成掩膜图像；
步骤208，根据第一全景图的渲染结果和掩膜图像，确定目标纹理信息；
步骤210，将目标纹理信息赋值至第一面片。
具体来说，确定第一全景图中的第一区域，该第一区域与第一面片对应，上述第一区域可以对应多个第一面片。Specifically, a first area in the first panoramic image is determined, the first area corresponds to a first patch, and the first area may correspond to a plurality of first patches.
进一步地，将第一区域的颜色渲染为目标颜色，例如纯黑色、纯绿色或者其他纯色。Further, the color of the first area is rendered as a target color, such as pure black, pure green or other pure colors.
进一步地，上述掩膜图像可以限定目标纹理信息的轮廓形状信息。Further, the above-mentioned mask image may define the outline shape information of the target texture information.
示例性地，如图3所示，本申请提供的三维重建方法，可以在360度的第一全景图中将缺少纹理信息的第一区域渲染为黑色，根据图3中“门”对应的黑色区域，生成图3对应的掩膜图像，掩膜图像如图4所示，掩膜图像可以确定目标纹理信息的边缘轮廓，估算出目标纹理信息后，将目标纹理信息填充到图3中黑色区域对应的第一面片上，使“门”缺失纹理被补全，效果如图5所示，从而使三维网格模型的内容更加完整。Exemplarily, as shown in FIG. 3 , the three-dimensional reconstruction method provided by the present application can render the first area lacking texture information as black in the first 360-degree panorama, according to the black corresponding to the “door” in FIG. 3 . area, generate the mask image corresponding to Figure 3, the mask image is shown in Figure 4, the mask image can determine the edge contour of the target texture information, after estimating the target texture information, fill the target texture information into the black area in Figure 3 On the corresponding first patch, the missing texture of the "door" is completed, and the effect is shown in Figure 5, so that the content of the 3D mesh model is more complete.
本申请实施例提供的三维重建方法通过将第一区域的颜色渲染为目标颜色，并在渲染为目标颜色的区域生成掩膜图像，便于估算出纹理缺失区域的纹理信息，保证了目标纹理信息的准确性。The three-dimensional reconstruction method provided by the embodiment of the present application renders the color of the first area as the target color, and generates a mask image in the area rendered as the target color, so as to facilitate the estimation of the texture information of the texture-missing area, and ensure the target texture information. accuracy.
在本申请的一些实施例中，本申请实施例提供的三维重建方法中的步骤204，具体可以包括：In some embodiments of the present application,
步骤204a：将第一全景图的渲染结果和掩膜图像的参数信息输入至第一预测模型，以通过第一预测模型输出目标纹理信息。Step 204a: Input the rendering result of the first panoramic image and the parameter information of the mask image into the first prediction model, so as to output the target texture information through the first prediction model.
在上述实施例中，将第一全景图的渲染结果和掩膜图像等参数信息输入至第一预测模型，经过第一预测模型的运算后，获得目标纹理信息。In the above embodiment, parameter information such as the rendering result of the first panoramic image and the mask image is input into the first prediction model, and after the operation of the first prediction model, the target texture information is obtained.
进一步地，掩膜图像包括轮廓形状信息，可以确定目标纹理信息的边缘轮廓。Further, the mask image includes contour shape information, and the edge contour of the target texture information can be determined.
具体地，第一预测模型可以为深度学习模型，示例性地，该深度学习模型可以为encoder-decoder架构的模型，将全景图的渲染结果和掩膜图像输入上述第一预测模型，该第一预测模型经过运算后输出纹理信息，该纹理信息可以为适合于为第一面片进行赋值的目标纹理信息。Specifically, the first prediction model may be a deep learning model. Exemplarily, the deep learning model may be a model of an encoder-decoder architecture, and the rendering result of the panorama and the mask image are input into the first prediction model. The prediction model outputs texture information after operation, and the texture information may be target texture information suitable for assigning value to the first patch.
示例性地，根据图6的据出上生成对应掩膜图像，掩膜图像可以确定目标纹理信息的边缘轮廓，将第一全景图和掩膜图像作为输入参数，输入到第一预测模型，第一预测模型经过运算输出目标纹理信息。目标纹理信息的轮廓信息符合图6中的黑色区域，将目标纹理信息填充到黑色区域对应的第一面片上，使“床”边的缺失纹理被补全，效果如图7所示，从而使三维网格模型的内容更加完整。Exemplarily, a corresponding mask image is generated according to the data shown in FIG. 6 , the mask image can determine the edge contour of the target texture information, and the first panorama image and the mask image are used as input parameters to be input into the first prediction model, and the first A prediction model outputs target texture information through operations. The contour information of the target texture information conforms to the black area in Figure 6, and the target texture information is filled on the first patch corresponding to the black area, so that the missing texture on the "bed" side is completed. The effect is shown in Figure 7, so that the The content of the 3D mesh model is more complete.
本申请实施例提供的三维重建方法通过使用预测模型确定目标纹理信息，保证估算的目标纹理信息的准确性，进而确保三维网格模型的显示完整性，丰富，丰富三维网格模型的信息参数。The 3D reconstruction method provided by the embodiment of the present application determines the target texture information by using the prediction model, so as to ensure the accuracy of the estimated target texture information, thereby ensuring the display integrity of the 3D mesh model, and enriching and enriching the information parameters of the 3D mesh model.
在本申请的一些实施例中，在本申请提供的三维重建方法中的步骤102之前，该三维重建方法还包括：In some embodiments of the present application, before
步骤101：获取拍摄场景的多张深度图和与多张深度图对应的多张彩色图，并根据多张深度图和多张彩色图构建三维网格模型。Step 101: Acquire multiple depth maps of the shooting scene and multiple color maps corresponding to the multiple depth maps, and construct a three-dimensional mesh model according to the multiple depth maps and multiple color maps.
本申请实施例中，获取拍摄场景的多张深度图和与多张深度图对应的多张彩色图，根据获取到的多张深度图和多张彩色图构建三维网格模型。在一种可能的实施方式中，构建目标场景的三维网格模型需要多张目标场景的彩色二维图像和深度图像，获取在目标场景的内拍摄的深度图和与彩色图后，对上述多张深度图和多张彩色图进行RGBD多点位数据的点位估计，根据获得的点位生成目标场景的三维点云，进而根据三维点云数据生成目标场景的网格，并估算纹理信息，最后构建目标场景三维网格模型。In the embodiment of the present application, multiple depth maps of the shooting scene and multiple color maps corresponding to the multiple depth maps are acquired, and a three-dimensional grid model is constructed according to the acquired multiple depth maps and multiple color maps. In a possible implementation, constructing a three-dimensional mesh model of the target scene requires multiple color two-dimensional images and depth images of the target scene. A depth map and multiple color maps are used for point estimation of RGBD multi-point data, and a 3D point cloud of the target scene is generated according to the obtained points, and then a grid of the target scene is generated according to the 3D point cloud data, and texture information is estimated. Finally, the 3D mesh model of the target scene is constructed.
根据这些深度图和彩色二维图构建目标场景的三维网格模型。A 3D mesh model of the target scene is constructed from these depth maps and color 2D maps.
本申请实施例提供的三维重建方法根据多张深度图和多张彩色图构建三维网格模型，保证了三维网格模型的准确性和完整性。The three-dimensional reconstruction method provided by the embodiments of the present application constructs a three-dimensional mesh model according to multiple depth maps and multiple color maps, which ensures the accuracy and integrity of the three-dimensional mesh model.
在本申请的一些实施例中，图8示出了本申请实施例提供的三维重建方法的流程示意图之三，如图8所示，上述三维重建方法具体包括：In some embodiments of the present application, FIG. 8 shows a third schematic flowchart of a three-dimensional reconstruction method provided by an embodiment of the present application. As shown in FIG. 8 , the above-mentioned three-dimensional reconstruction method specifically includes:
步骤802：获取三维网格模型中的多个面片；Step 802: Acquire multiple patches in the 3D mesh model;
步骤804：通过纹理估计算法确定多个面片中任一个面片匹配对应的纹理信息；Step 804: Determine the texture information corresponding to the matching of any one of the multiple patches by a texture estimation algorithm;
步骤806：基于第一虚拟点位渲染三维网格模型的第一全景图，并确定第一全景图中的第一区域，第一区域与第一面片对应；Step 806: Render the first panorama of the three-dimensional mesh model based on the first virtual point, and determine a first area in the first panorama, where the first area corresponds to the first patch;
步骤808：根据第一全景图和第一区域确定目标纹理信息，将目标纹理信息赋值至第一面片。Step 808: Determine target texture information according to the first panorama image and the first area, and assign the target texture information to the first patch.
本申请实施例中，三维网格模型内部包括多个面片，通过纹理估计算法，能够计算出每一个面片的纹理信息。In the embodiment of the present application, the three-dimensional mesh model includes a plurality of patches, and the texture information of each patch can be calculated through a texture estimation algorithm.
在一种可能的实施方式中，纹理估计算法可以为马尔可夫随机数算法。马尔可夫随机数算法需要设定数据项和平滑项，数据项的计算公式为：In a possible implementation, the texture estimation algorithm may be a Markov random number algorithm. The Markov random number algorithm needs to set a data item and a smoothing item. The calculation formula of the data item is:
Data(F,v)＝S/|dt-d|；Data(F,v)=S/|dt-d|;
其中，v为第一面片，S为第一面片投影到二维视图上的面积，dt为第一面片投影到二维视图上的深度参数，d为第一面片在对应的深度图中的深度参数。Among them, v is the first patch, S is the area of the first patch projected onto the 2D view, dt is the depth parameter of the first patch projected onto the 2D view, and d is the corresponding depth of the first patch The depth parameter in the figure.
平滑项的计算公式为：The calculation formula of the smoothing term is:
其中，Ik为第一面片的二位语义，Ip为第一面片相邻面片的二位语义，若相邻两个面片的语义相同，平滑项设为1，反之，平滑项设为0。Among them, Ik is the two-bit semantics of the first patch, Ip is the two-bit semantics of the adjacent patches of the first patch, if the semantics of the two adjacent patches are the same, the smooth item is set to 1, otherwise, the smooth item is set to 0.
马尔可夫随机场的计算式为：The formula for the Markov random field is:
根据数据项和平滑项，计算出第一面片的纹理信息E(v)。According to the data item and the smoothing item, the texture information E(v) of the first patch is calculated.
本申请实施例提供的三维重建方法通过纹理估计算法，计算出三维网格模型中每一个面片的纹理信息，进而能够确认缺少纹理信息的第一面片，并对第一面片进行纹理信息的填充，保证三维网格模型显示的完整性。The 3D reconstruction method provided by the embodiment of the present application calculates the texture information of each facet in the 3D mesh model through a texture estimation algorithm, so as to confirm the first facet lacking texture information, and perform texture information on the first facet Filling to ensure the integrity of the 3D mesh model display.
在本申请的一些实施例中提供了一种三维重建装置，图9示出了本申请实施例提供的三维重建装置的结构框图，如图9所示，三维重建装置900包括：Some embodiments of the present application provide a three-dimensional reconstruction apparatus. FIG. 9 shows a structural block diagram of the three-dimensional reconstruction apparatus provided by the embodiments of the present application. As shown in FIG. 9 , the three-
处理单元902，用于为三维网格模型中的多个面片匹配纹理信息，在三维网格模型中的第一面片未匹配到对应的纹理信息的情况下，确定第一虚拟点位；The
处理单元902，还用于基于第一虚拟点位渲染三维网格模型的第一全景图，并确定第一全景图中的第一区域，第一区域与第一面片对应；The
处理单元902，还用于根据第一全景图和第一区域确定目标纹理信息，将目标纹理信息赋值至第一面片。The
具体来说，构建目标场景的三维网格模型的构建的过程中，三维网格模型内部包括多个面片，处理单元902确认三维网格模型内部的所有面片后，对每个面片进行纹理信息确认，将缺少纹理信息的面片设为第一面片，并基于第一面片的坐标参数构建第一虚拟点位；处理单元902在第一虚拟点位的位置对三维网格模型重新渲染出第一全景图，并在第一全景图中确定出第一面片对应的第一区域；处理单元902根据第一全景图和第一区域估算出第一面片的目标纹理信息，将目标纹理信息重新赋值给第一面片。Specifically, in the process of constructing the 3D mesh model of the target scene, the 3D mesh model includes multiple patches. After confirming all the patches in the 3D mesh model, the
本申请实施例提供的三维重建装置通过处理单元对缺少纹理信息的面片构建虚拟点位，并重新渲染全景图，估算出对应的纹理信息填入三维网格模型中。有效补充了三维网格模型中缺少的纹理信息，保证了目标场景的三维网格模型中纹理信息的完整性。The three-dimensional reconstruction device provided by the embodiment of the present application uses a processing unit to construct virtual points for patches lacking texture information, re-renders the panorama, and estimates the corresponding texture information to fill in the three-dimensional mesh model. It effectively supplements the missing texture information in the 3D mesh model and ensures the integrity of the texture information in the 3D mesh model of the target scene.
在本申请的一些实施例中，处理单元902还用于根据第一面片的空间参数，确定出第一面片对应的第一虚拟点位。In some embodiments of the present application, the
本申请实施例提供的三维重建装置通过处理单元根据第一面片的空间参数，确定出第一面片对应的第一虚拟点位，保证虚拟点位的准确性。The three-dimensional reconstruction device provided by the embodiment of the present application determines, through the processing unit, the first virtual point corresponding to the first patch according to the spatial parameters of the first patch, so as to ensure the accuracy of the virtual point.
在本申请的一些实施例中，处理单元902还用于根据第一面片的法向量，确定第一虚拟点位；其中，第一面片的中心至第一虚拟点位的向量以及第一面片的法向量的夹角小于90°。In some embodiments of the present application, the
本申请实施例提供的三维重建装置通过处理单元根据第一面片的法向量，确定出第一面片对应的虚拟点位，保证虚拟点位空间参数的准确性，进而保证在第一虚拟点位渲染得到第一全景图的显示效果。The three-dimensional reconstruction device provided in the embodiment of the present application determines the virtual point corresponding to the first patch according to the normal vector of the first patch by the processing unit, so as to ensure the accuracy of the spatial parameters of the virtual point, and further ensure that the first virtual point is located at the first virtual point. Bit rendering to obtain the display effect of the first panorama.
在本申请的一些实施例中，处理单元902还用于将第一区域渲染为目标颜色，对目标颜色对应的区域生成掩膜图像；处理单元902还用于根据第一全景图的渲染结果和掩膜图像，确定目标纹理信息。In some embodiments of the present application, the
本申请实施例提供的三维重建装置通过处理单元将第一区域的颜色渲染为目标颜色，生成掩膜图像，估算出纹理缺失区域的纹理信息，保证了目标纹理信息的准确性。The three-dimensional reconstruction device provided by the embodiment of the present application renders the color of the first region as the target color through the processing unit, generates a mask image, and estimates the texture information of the texture-missing region, thereby ensuring the accuracy of the target texture information.
在本申请的一些实施例中，处理单元902还用于将第一全景图的渲染结果和掩膜图像的参数信息输入至第一预测模型，以通过第一预测模型输出目标纹理信息。In some embodiments of the present application, the
本申请实施例提供的三维重建装置通过处理单元使用预测模型，确定纹理信息，保证估算的目标纹理信息的准确性，进而确保三维网格模型的显示完整性，丰富三维网格模型的信息参数。The three-dimensional reconstruction device provided by the embodiment of the present application uses the prediction model to determine the texture information through the processing unit to ensure the accuracy of the estimated target texture information, thereby ensuring the display integrity of the three-dimensional mesh model and enriching the information parameters of the three-dimensional mesh model.
在本申请的一些实施例中，处理单元902还用于获取拍摄场景的多张深度图和与多张深度图对应的多张彩色图，并根据多张深度图和多张彩色图构建三维网格模型。In some embodiments of the present application, the
本申请实施例提供的三维重建装置通过处理单元根据多张深度图和多张彩色图构建三维网格模型，保证了三维网格模型的准确性和完整性。The three-dimensional reconstruction device provided by the embodiment of the present application constructs a three-dimensional grid model according to a plurality of depth maps and a plurality of color maps by a processing unit, which ensures the accuracy and integrity of the three-dimensional grid model.
在本申请的一些实施例中，三维重建装置900，获取单元，用于获取三维网格模型中的多个面片；In some embodiments of the present application, the three-
处理单元902，还用于通过纹理估计算法确定多个面片中任一个面片匹配对应的纹理信息。The
本申请实施例提供的三维重建装置通过处理单元使用纹理估计算法，计算出三维网格模型中每一个面片的纹理信息，确认缺少纹理信息的目标面片。The three-dimensional reconstruction device provided by the embodiment of the present application uses a texture estimation algorithm by a processing unit to calculate the texture information of each patch in the three-dimensional mesh model, and confirms the target patch lacking texture information.
本申请实施例中的三维重建装置可以是电子设备，也可以是电子设备中的部件，例如集成电路或芯片。该电子设备可以是终端，也可以为除终端之外的其他设备。示例性的，电子设备可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备、移动上网装置(Mobile Internet Device，MID)、增强现实(augmented reality，AR)/虚拟现实(virtualreality，VR)设备、机器人、可穿戴设备、超级移动个人计算机(ultra-mobile personalcomputer，UMPC)、上网本或者个人数字助理(personal digital assistant，PDA)等，还可以为服务器、网络附属存储器(Network Attached Storage，NAS)、个人计算机(personalcomputer，PC)、电视机(television，TV)、柜员机或者自助机等，本申请实施例不作具体限定。The three-dimensional reconstruction apparatus in this embodiment of the present application may be an electronic device, or may be a component in the electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be other devices other than the terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a notebook computer, a PDA, a vehicle electronic device, a Mobile Internet Device (MID), an augmented reality (AR)/virtual reality (VR) Devices, robots, wearable devices, ultra-mobile personal computers (UMPCs), netbooks or personal digital assistants (PDAs), etc., and can also be servers, network attached storages (NAS) , a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which are not specifically limited in the embodiments of the present application.
本申请实施例中的三维重建装置可以为具有操作系统的装置。该操作系统可以为安卓(Android)操作系统，可以为iOS操作系统，还可以为其他可能的操作系统，本申请实施例不作具体限定。The three-dimensional reconstruction device in this embodiment of the present application may be a device with an operating system. The operating system may be an Android (Android) operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.
本申请实施例提供的三维重建装置能够实现上述方法实施例实现的各个过程，为避免重复，这里不再赘述。The three-dimensional reconstruction apparatus provided in the embodiment of the present application can implement each process implemented by the foregoing method embodiment, and to avoid repetition, details are not repeated here.
可选地，本申请实施例还提供一种电子设备，图10示出了根据本申请实施例的电子设备的结构框图，如图10所示，电子设备1000包括处理器1002和存储器1004，存储器1004上存储有可在处理器1002上运行的程序或指令，该程序或指令被处理器1002执行时实现上述方法实施例的各个步骤，且能达到相同的技术效果，为避免重复，这里不再赘述。Optionally, an embodiment of the present application further provides an electronic device. FIG. 10 shows a structural block diagram of an electronic device according to an embodiment of the present application. As shown in FIG. 10 , the
需要说明的是，本申请实施例中的电子设备包括上述的移动电子设备和非移动电子设备。It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
图11为实现本申请实施例的一种电子设备的硬件结构示意图。FIG. 11 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.
该电子设备1100包括但不限于：射频单元1101、网络模块1102、音频输出单元1103、输入单元1104、传感器1105、显示单元1106、用户输入单元1107、接口单元1108、存储器1109以及处理器1110等部件。The
本领域技术人员可以理解，电子设备1100还可以包括给各个部件供电的电源(比如电池)，电源可以通过电源管理系统与处理器1110逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。图11中示出的电子设备结构并不构成对电子设备的限定，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置，在此不再赘述。Those skilled in the art can understand that the
其中，处理器1110，用于为三维网格模型中的多个面片匹配纹理信息，在三维网格模型中的第一面片未匹配到对应的纹理信息的情况下，确定第一虚拟点位；The
处理器1110，用于基于第一虚拟点位渲染三维网格模型的第一全景图，并确定第一全景图中的第一区域，第一区域与第一面片对应；a
处理器1110，用于根据第一全景图和第一区域确定目标纹理信息，将目标纹理信息赋值至第一面片。The
本申请实施例提供的电子设备对缺少纹理信息的面片构建虚拟点位，并重新渲染全景图，估算出对应的纹理信息填入三维网格模型中。有效补充了三维网格模型中缺少的纹理信息，保证了目标场景的三维网格模型中纹理信息的完整性。The electronic device provided by the embodiments of the present application constructs virtual points for patches lacking texture information, re-renders the panorama, and estimates the corresponding texture information to fill in the three-dimensional mesh model. It effectively supplements the missing texture information in the 3D mesh model and ensures the integrity of the texture information in the 3D mesh model of the target scene.
进一步地，处理器1110，用于根据第一面片的空间参数，确定出第一面片对应的第一虚拟点位。Further, the
本申请实施例提供的电子设备根据第一面片的空间参数，确定出第一面片对应的虚拟点位，保证虚拟点位的准确性。The electronic device provided by the embodiment of the present application determines the virtual point corresponding to the first patch according to the spatial parameters of the first patch, so as to ensure the accuracy of the virtual point.
进一步地，处理器1110，用于根据第一面片的法向量，确定第一虚拟点位；Further, the
其中，第一面片的中心至第一虚拟点位的向量以及第一面片的法向量的夹角小于90°。Wherein, the included angle between the center of the first patch and the vector of the first virtual point and the normal vector of the first patch is less than 90°.
本申请实施例提供的电子设备根据第一面片的法向量，确定出第一面片对应的虚拟点位。保证虚拟点位空间参数的准确性。The electronic device provided by the embodiment of the present application determines the virtual point corresponding to the first patch according to the normal vector of the first patch. Ensure the accuracy of virtual point space parameters.
进一步地，处理器1110，用于将第一区域渲染为目标颜色，对目标颜色对应的区域生成掩膜图像；根据第一全景图的渲染结果和掩膜图像，确定目标纹理信息。Further, the
本申请实施例提供的电子设备将第一区域的颜色渲染为目标颜色，生成掩膜图像，估算出纹理缺失区域的纹理信息。The electronic device provided by the embodiment of the present application renders the color of the first area as the target color, generates a mask image, and estimates the texture information of the texture-missing area.
进一步地，处理器1110，用于将第一全景图的渲染结果和掩膜图像的参数信息输入至第一预测模型，以通过第一预测模型输出目标纹理信息。Further, the
本申请实施例提供的电子设备使用预测模型，确定纹理信息。保证估算纹理图像的准确性。The electronic device provided by the embodiment of the present application uses a prediction model to determine texture information. Guarantees the accuracy of estimating texture images.
进一步地，处理器1110，用于获取拍摄场景的多张深度图和与多张深度图对应的多张彩色图，并根据多张深度图和多张彩色图构建三维网格模型。Further, the
本申请实施例提供的电子设备根据多张深度图和多张彩色图构建三维网格模型，保证了三维网格模型的准确性和完整性。The electronic device provided by the embodiment of the present application constructs a three-dimensional grid model according to a plurality of depth maps and a plurality of color maps, which ensures the accuracy and integrity of the three-dimensional grid model.
进一步地，处理器1110，用于确定三维网格模型中的不可见区域的边界所对应的第三面片，将第四面片的纹理颜色填充至第三面片，第四面片为与第三面片相邻的并位于不可见区域边界外的面片；用于获取三维网格模型中的多个面片；用于通过纹理估计算法确定多个面片中任一个面片匹配对应的纹理信息。Further, the
本申请实施例提供的电子设备使用纹理估计算法，计算出三维网格模型中每一个面片的纹理信息，确认缺少纹理信息的目标面片。The electronic device provided by the embodiment of the present application uses a texture estimation algorithm to calculate the texture information of each patch in the three-dimensional mesh model, and confirms the target patch lacking texture information.
应理解的是，本申请实施例中，输入单元1104可以包括图形处理器(GraphicsProcessing Unit，GPU)11041和麦克风11042，图形处理器11041对在视频捕获模式或图像捕获模式中由图像捕获装置(如摄像头)获得的静态图片或视频的图像数据进行处理。显示单元1106可包括显示面板11061，可以采用液晶显示器、有机发光二极管等形式来配置显示面板11061。用户输入单元1107包括触控面板11071以及其他输入设备11072中的至少一种。触控面板11071，也称为触摸屏。触控面板11071可包括触摸检测装置和触摸控制器两个部分。其他输入设备11072可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、操作杆，在此不再赘述。It should be understood that, in this embodiment of the present application, the
存储器1109可用于存储软件程序以及各种数据。存储器1109可主要包括存储程序或指令的第一存储区和存储数据的第二存储区，其中，第一存储区可存储操作系统、至少一个功能所需的应用程序或指令(比如声音播放功能、图像播放功能等)等。此外，存储器1109可以包括易失性存储器或非易失性存储器，或者，存储器1109可以包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM，DDRSDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(Synch link DRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DRRAM)。本申请实施例中的存储器1109包括但不限于这些和任意其它适合类型的存储器。The
处理器1110可包括一个或多个处理单元；可选的，处理器1110集成应用处理器和调制解调处理器，其中，应用处理器主要处理涉及操作系统、用户界面和应用程序等的操作，调制解调处理器主要处理无线通信信号，如基带处理器。可以理解的是，上述调制解调处理器也可以不集成到处理器1110中。The
本申请实施例还提供一种可读存储介质，可读存储介质上存储有程序或指令，该程序或指令被处理器执行时实现上述三维重建方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。The embodiments of the present application further provide a readable storage medium, where a program or an instruction is stored on the readable storage medium. When the program or instruction is executed by a processor, each process of the foregoing three-dimensional reconstruction method embodiment can be implemented, and the same technology can be achieved. The effect, in order to avoid repetition, is not repeated here.
其中，处理器为上述实施例中的电子设备中的处理器。可读存储介质，包括计算机可读存储介质，如计算机只读存储器ROM、随机存取存储器RAM、磁碟或者光盘等。The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as computer read-only memory ROM, random access memory RAM, magnetic disk or optical disk, and the like.
本申请实施例另提供了一种芯片，芯片包括处理器和通信接口，通信接口和处理器耦合，处理器用于运行程序或指令，实现上述三维重建方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。An embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled with the processor, and the processor is used for running a program or an instruction to implement each process of the above three-dimensional reconstruction method embodiment, and can achieve the same The technical effect, in order to avoid repetition, will not be repeated here.
应理解，本申请实施例提到的芯片还可以称为系统级芯片、系统芯片、芯片系统或片上系统芯片等。It should be understood that the chip mentioned in the embodiments of the present application may also be referred to as a system-on-chip, a system-on-chip, a system-on-a-chip, or a system-on-a-chip, or the like.
本申请实施例提供一种计算机程序产品，该程序产品被存储在存储介质中，该程序产品被至少一个处理器执行以实现如上述三维重建方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。The embodiments of the present application provide a computer program product, the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the various processes in the foregoing three-dimensional reconstruction method embodiments, and can achieve the same technical effect , in order to avoid repetition, it will not be repeated here.
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。此外，需要指出的是，本申请实施方式中的方法和装置的范围不限按示出或讨论的顺序来执行功能，还可包括根据所涉及的功能按基本同时的方式或按相反的顺序来执行功能，例如，可以按不同于所描述的次序来执行所描述的方法，并且还可以添加、省去、或组合各种步骤。另外，参照某些示例所描述的特征可在其他示例中被组合。It should be noted that, herein, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements, It also includes other elements not expressly listed or inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in the reverse order depending on the functions involved. To perform functions, for example, the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to some examples may be combined in other examples.
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以计算机软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端(可以是手机，计算机，服务器，或者网络设备等)执行本申请各个实施例的方法。From the description of the above embodiments, those skilled in the art can clearly understand that the method of the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solutions of the present application can be embodied in the form of computer software products that are essentially or contribute to the prior art, and the computer software products are stored in a storage medium (such as ROM/RAM, magnetic disk , CD-ROM), including several instructions to enable a terminal (which may be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of the various embodiments of the present application.
上面结合附图对本申请的实施例进行了描述，但是本申请并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本申请的启示下，在不脱离本申请宗旨和权利要求所保护的范围情况下，还可做出很多形式，均属于本申请的保护之内。The embodiments of the present application have been described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the inspiration of this application, without departing from the scope of protection of the purpose of this application and the claims, many forms can be made, which all fall within the protection of this application.
Claims (10)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210439570.5ACN114820980B (en) | 2022-04-25 | 2022-04-25 | Three-dimensional reconstruction method, device, electronic device and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210439570.5ACN114820980B (en) | 2022-04-25 | 2022-04-25 | Three-dimensional reconstruction method, device, electronic device and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114820980Atrue CN114820980A (en) | 2022-07-29 |
| CN114820980B CN114820980B (en) | 2025-06-27 |
Family
ID=82508163
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210439570.5AActiveCN114820980B (en) | 2022-04-25 | 2022-04-25 | Three-dimensional reconstruction method, device, electronic device and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114820980B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116934984A (en)* | 2023-09-19 | 2023-10-24 | 成都中轨轨道设备有限公司 | Intelligent terminal and method for constructing virtual panoramic scene space |
| CN117496069A (en)* | 2023-11-24 | 2024-02-02 | 北京百度网讯科技有限公司 | Signed distance field generation method and device, three-dimensional model generation method and device |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2018049A2 (en)* | 2007-07-18 | 2009-01-21 | Samsung Electronics Co., Ltd. | Method of assembling a panoramic image, method of providing a virtual 3D projection of a panoramic image and camera therefor |
| US20090244062A1 (en)* | 2008-03-31 | 2009-10-01 | Microsoft | Using photo collections for three dimensional modeling |
| CN103077509A (en)* | 2013-01-23 | 2013-05-01 | 天津大学 | Method for synthesizing continuous and smooth panoramic video in real time by using discrete cubic panoramas |
| CN110473294A (en)* | 2018-05-11 | 2019-11-19 | 杭州海康威视数字技术股份有限公司 | A kind of texture mapping method based on threedimensional model, device and equipment |
| CN111325693A (en)* | 2020-02-24 | 2020-06-23 | 西安交通大学 | Large-scale panoramic viewpoint synthesis method based on single-viewpoint RGB-D image |
| CN112802206A (en)* | 2021-02-07 | 2021-05-14 | 北京字节跳动网络技术有限公司 | Roaming view generation method, device, equipment and storage medium |
| CN113313818A (en)* | 2021-06-07 | 2021-08-27 | 聚好看科技股份有限公司 | Three-dimensional reconstruction method, device and system |
- 2022
- 2022-04-25CNCN202210439570.5Apatent/CN114820980B/enactiveActive
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2018049A2 (en)* | 2007-07-18 | 2009-01-21 | Samsung Electronics Co., Ltd. | Method of assembling a panoramic image, method of providing a virtual 3D projection of a panoramic image and camera therefor |
| US20090244062A1 (en)* | 2008-03-31 | 2009-10-01 | Microsoft | Using photo collections for three dimensional modeling |
| CN103077509A (en)* | 2013-01-23 | 2013-05-01 | 天津大学 | Method for synthesizing continuous and smooth panoramic video in real time by using discrete cubic panoramas |
| CN110473294A (en)* | 2018-05-11 | 2019-11-19 | 杭州海康威视数字技术股份有限公司 | A kind of texture mapping method based on threedimensional model, device and equipment |
| CN111325693A (en)* | 2020-02-24 | 2020-06-23 | 西安交通大学 | Large-scale panoramic viewpoint synthesis method based on single-viewpoint RGB-D image |
| CN112802206A (en)* | 2021-02-07 | 2021-05-14 | 北京字节跳动网络技术有限公司 | Roaming view generation method, device, equipment and storage medium |
| CN113313818A (en)* | 2021-06-07 | 2021-08-27 | 聚好看科技股份有限公司 | Three-dimensional reconstruction method, device and system |
Non-Patent Citations (1)
| Title |
|---|
| 刘乾坤: ""基于多张全景图的虚拟视点合成系统"", 《中国优秀硕士学位论文全文数据库》, 15 April 2019 (2019-04-15)* |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116934984A (en)* | 2023-09-19 | 2023-10-24 | 成都中轨轨道设备有限公司 | Intelligent terminal and method for constructing virtual panoramic scene space |
| CN116934984B (en)* | 2023-09-19 | 2023-12-08 | 成都中轨轨道设备有限公司 | Intelligent terminal and method for constructing virtual panoramic scene space |
| CN117496069A (en)* | 2023-11-24 | 2024-02-02 | 北京百度网讯科技有限公司 | Signed distance field generation method and device, three-dimensional model generation method and device |
| CN117496069B (en)* | 2023-11-24 | 2025-01-21 | 北京百度网讯科技有限公司 | Signed distance field generation method and device, three-dimensional model generation method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114820980B (en) | 2025-06-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11842438B2 (en) | Method and terminal device for determining occluded area of virtual object | |
| JP7337104B2 (en) | Model animation multi-plane interaction method, apparatus, device and storage medium by augmented reality | |
| US9779508B2 (en) | Real-time three-dimensional reconstruction of a scene from a single camera | |
| US20170186219A1 (en) | Method for 360-degree panoramic display, display module and mobile terminal | |
| CN113240769B (en) | Spatial link relation identification method and device and storage medium | |
| WO2017113731A1 (en) | 360-degree panoramic displaying method and displaying module, and mobile terminal | |
| JP7701932B2 (en) | Efficient localization based on multiple feature types | |
| CN110378947B (en) | 3D model reconstruction method and device and electronic equipment | |
| CN114782647A (en) | Model reconstruction method, device, equipment and storage medium | |
| CN112766027A (en) | Image processing method, device, equipment and storage medium | |
| CN114820980B (en) | Three-dimensional reconstruction method, device, electronic device and readable storage medium | |
| CN110363860B (en) | 3D model reconstruction method and device and electronic equipment | |
| JP2023527438A (en) | Geometry Recognition Augmented Reality Effect Using Real-time Depth Map | |
| CN114782646A (en) | House model modeling method and device, electronic equipment and readable storage medium | |
| CN111161398A (en) | Image generation method, device, equipment and storage medium | |
| CN114821055B (en) | Method, device, readable storage medium and electronic device for constructing house model | |
| CN113129362B (en) | Method and device for acquiring three-dimensional coordinate data | |
| Hartl et al. | Rapid reconstruction of small objects on mobile phones | |
| CN110378948B (en) | 3D model reconstruction method and device and electronic equipment | |
| WO2018151612A1 (en) | Texture mapping system and method | |
| CN112348965A (en) | Imaging method, imaging device, electronic equipment and readable storage medium | |
| Hall et al. | Networked and multimodal 3d modeling of cities for collaborative virtual environments | |
| CN115690365A (en) | Method, device, device and medium for managing three-dimensional models | |
| CN114820968A (en) | Three-dimensional visualization method and device, robot, electronic device and storage medium | |
| CN114299253A (en) | Model interior rendering method and device, electronic equipment and medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |