技术领域Technical Field
本发明涉及计算机视觉技术领域，更具体的说，它涉及一种基于先验驱动的多视图神经隐式表面重建方法。The present invention relates to the field of computer vision technology, and more specifically, to a multi-view neural implicit surface reconstruction method based on prior-driven.
背景技术Background technique
多视图神经隐式表面重建方法旨在带相机姿态的多视角图像中通过体渲染和表面渲染技术重建出场景的表面模型。由于该方法能够以无监督的方式学习出场景的三维表达，近年来，基于神经隐式学习的多视图表面重建方法得到了快速发展。虽然神经隐式表面重建方法在简单场景上展现出了令人印象深刻的结果，但由于其主要依赖于颜色重建损失进行优化，几何场的重建是颜色场重建的附属产品，即几何场存在欠约束优化的问题，这导致基于神经隐式学习的多视图表面重建方法难以处理真实世界的复杂场景。因此，如何加强对于几何场的优化约束，使其能够重建出复杂场景下的高质量三维表达是有待解决的关键问题。The multi-view neural implicit surface reconstruction method aims to reconstruct the surface model of the scene in multi-view images with camera poses through volume rendering and surface rendering techniques. Since this method can learn the three-dimensional expression of the scene in an unsupervised manner, the multi-view surface reconstruction method based on neural implicit learning has developed rapidly in recent years. Although the neural implicit surface reconstruction method has shown impressive results on simple scenes, it mainly relies on color reconstruction loss for optimization, and the reconstruction of the geometric field is a by-product of the color field reconstruction, that is, the geometric field has the problem of under-constrained optimization, which makes it difficult for the multi-view surface reconstruction method based on neural implicit learning to handle complex scenes in the real world. Therefore, how to strengthen the optimization constraints on the geometric field so that it can reconstruct high-quality three-dimensional expressions in complex scenes is a key issue to be solved.
发明内容Summary of the invention
本发明克服了现有技术的不足，提供了解决现有方法难以恢复出复杂场景下的表面模型的技术问题的一种基于点云引导的多视图神经隐式表面重建方法。The present invention overcomes the shortcomings of the prior art and provides a multi-view neural implicit surface reconstruction method based on point cloud guidance to solve the technical problem that the prior methods are difficult to restore surface models in complex scenes.
本发明的技术方案如下：The technical solution of the present invention is as follows:
一种基于先验驱动的多视图神经隐式表面重建方法，包括如下步骤：A multi-view neural implicit surface reconstruction method based on prior-driven, comprising the following steps:
S1先验提取步骤：给定一个场景的K幅图像及各图像对应的相机参数，从K幅场景图像中依次选择一幅图像为参考图像，与其最邻近的N-1幅图像为邻域图像，并进行先验提取；S1 prior extraction step: given K images of a scene and the camera parameters corresponding to each image, select one image from the K scene images as the reference image, and the N-1 images closest to it as the neighborhood images, and perform prior extraction;
首先，将每幅参考图像和其对应的邻域图像，输入到预训练好的多视图立体视觉网络中，得到该参考图像对应的深度图DMVS、邻域图像对应的可见性图Vm，以及参考图像和邻域图像对应的图像特征Fr和Fm；该过程将一直重复，直到所有图像都处理完，之后根据光度一致性和几何一致性滤除每张图像深度图中不可靠的深度部分；First, each reference image and its corresponding neighborhood image are input into the pre-trained multi-view stereo vision network to obtain the depth map DMVS corresponding to the reference image, the visibility map Vm corresponding to the neighborhood image, and the image features Fr and Fm corresponding to the reference image and the neighborhood image; this process will be repeated until all images are processed, and then the unreliable depth part of each image depth map is filtered out according to photometric consistency and geometric consistency;
S2图像训练步骤：对于一个场景的K幅图像，从中随机选择一幅图像作为目标图像进行训练；S2 image training step: for K images of a scene, randomly select one image as the target image for training;
首先，在选择的图像中随机选择Np个像素x，根据选择的图像对应的相机参数，假设射线r从相机中心o沿着其视角方向v穿过像素x进行投射；之后，采用误差有界采样和深度先验辅助采样在射线r上采样P个深度值进而得到射线上的P个采样点；First, randomly select Np pixels x in the selected image. According to the camera parameters corresponding to the selected image, assume that ray r is projected from the camera center o along its viewing direction v through pixel x. Then, error-bounded sampling and depth prior-assisted sampling are used to sample P depth values on ray r. Then P sampling points on the ray are obtained;
S3编码步骤：构建多尺度哈希编码并将S2图像训练步骤中采样得到的三维点进行编码；S3 encoding step: construct a multi-scale hash code and encode the three-dimensional points sampled in the S2 image training step;
S4 SDF处理步骤：对于射线上采样的三维点和其对应的哈希编码特征h(p)，首先通过基于傅里叶级数的位置编码对三维点/>进行编码得到位置编码特征γ(p)；之后，将位置编码特征γ(p)和哈希编码特征h(p)进行结合并输入到由两层多层感知机MLP组成的几何网络fθ来进行SDF场学习，并输出学习到的SDFs值s和通道数为256的SDF特征；S4 SDF processing step: for the 3D points sampled on the ray And its corresponding hash code feature h(p), firstly, the three-dimensional point is encoded by Fourier series based position coding/> The position coding feature γ(p) is obtained by encoding; then, the position coding feature γ(p) and the hash coding feature h(p) are combined and input into a geometric network fθ composed of two layers of multi-layer perceptrons MLP to learn the SDF field, and the learned SDFs value s and the SDF feature with 256 channels are output;
S5颜色处理步骤：将射线r的视角方向v、几何网络学习到的SDF特征和三维点的法向/>输入到由两层多层感知机MLP组成的颜色网络来对颜色场进行建模，并输出学习到的射线上r采样点/>对应的颜色值/>S5 color processing step: the viewing direction v of ray r, the SDF features learned by the geometric network and the 3D point Normal/> Input to the color network composed of two layers of multi-layer perceptron MLP to model the color field and output the learned r sampling points on the ray/> The corresponding color value />
S6渲染步骤：将S4 SDF处理步骤中学习到的SDFs值s转换为体密度，并基于体渲染将S5颜色处理步骤中得到的颜色值和法向/>进行渲染从而得到射线r的颜色/>法向和深度/>S6 rendering step: convert the SDFs value s learned in S4 SDF processing step into volume density, and render the color value obtained in S5 color processing step based on volume and normal/> Rendering is performed to obtain the color of ray r/> Normal and depth/>
S7重要点渲染步骤：在通过上述步骤得到当前射线r的SDFs值s和渲染深度值后；构建样本集/>在样本集/>的基础上执行步骤S3至S6的操作，得到重要性渲染的颜色/>S7 important point rendering step: After the above steps, the SDFs value s and rendering depth value of the current ray r are obtained After; build sample set /> In the sample set/> Based on this, the operations of steps S3 to S6 are performed to obtain the color of importance rendering/>
S8监督步骤：采用可见性感知的特征一致性损失Lvfc、法向一致性损失Lnormal、几何偏置损失Lbias、平滑性损失Lsmooth、Eikonal损失Leikonal和颜色损失Lrgb来进行监督步骤S1至S7的操作；S8 supervision step: use visibility-aware feature consistency loss Lvfc , normal consistency loss Lnormal , geometric bias loss Lbias , smoothness loss Lsmooth , Eikonal loss Leikonal and color loss Lrgb to perform the operations of supervision steps S1 to S7;
S9表面模型获取步骤：在训练过程中不断重复步骤S2至S7的操作，直至达到预设的迭代次数；训练完成后，采用Marching Cube算法从几何网络学习到的SDF场的零水平级中提取场景的表面模型。S9 surface model acquisition step: during the training process, the operations of steps S2 to S7 are continuously repeated until a preset number of iterations is reached; after the training is completed, the Marching Cube algorithm is used to extract the surface model of the scene from the zero level of the SDF field learned by the geometric network.
进一步的，S2图像训练具体包括如下步骤：Furthermore, S2 image training specifically includes the following steps:
S201：对于一个场景的K幅图像，从中随机选择一幅图像作为目标图像进行训练，并在该图像中随机选择Np个像素x，根据该图像对应的相机参数，假设射线r从相机中心o沿着其视角方向v穿过像素x进行投射；S201: For K images of a scene, randomly select one image as the target image for training, and randomly select Np pixels x in the image. According to the camera parameters corresponding to the image, assume that the ray r is projected from the camera center o along its viewing direction v through the pixel x;
S202：采用误差有界采样在射线r上采样PE个采样点得到样本集S202: Use bounded error sampling to sample PE sampling points on ray r to obtain a sample set
S203：采用深度先验辅助采样在射线r上采样PMVS个采样点得到样本集S203: Use depth prior assisted sampling to sample PMVS sampling points on ray r to obtain a sample set
S204：将基于误差有界采样得到的样本集和深度先验辅助采样得到的样本集结合得到最终的样本集，并将样本集中的采样值/>转换为射线上的三维点tn和tf分别表示射线的近边界和远边界。S204: The sample set obtained based on bounded error sampling And the sample set obtained by deep prior assisted sampling Combine to get the final sample set, and put the sample values in the sample set/> Convert to a 3D point on the ray tn and tf represent the near and far boundaries of the ray respectively.
进一步的，步骤S202中具体处理如下：Furthermore, the specific processing in step S202 is as follows:
首先对误差阈值ε和可学习参数β分别随机初始化一个大于0的值，在[tn,tf]中进行均匀采样得到初始样本即/>并初始化β+使其满足β+>β，从而能够使误差界限/>满足条件/>First, the error threshold ε and the learnable parameter β are randomly initialized to a value greater than 0, and the initial sample is obtained by uniform sampling in [tn ,tf ] That is/> And initialize β+ to satisfy β+ >β, so that the error bound /> Satisfy the conditions/>
之后为了在保持的同时减少β+，对/>进行上采样得到候选样本集/>若/>被充分上采样后满足/>则将β+向β方向减小；该采样算法在/>的情况下会持续迭代，根据中值定理，存在β*∈(β,β+)，使得/>并使用最多10次迭代的等分方法来寻找β*，并更新β+的值；上述上采样、迭代过程将循环进行直到达到循环的最大次数或/>Then in order to maintain while reducing β+ , to/> Up-sample to obtain candidate sample set/> If/> After being fully upsampled, it satisfies/> Then reduce β+ to β; the sampling algorithm is in/> In the case of continuous iteration, according to the mean value theorem, there exists β* ∈(β,β+ ) such that/> And use the equal division method with a maximum of 10 iterations to find β* and update the value of β+ ; the above upsampling and iterative process will be repeated until the maximum number of cycles is reached or/>
接着，使用最终得到的和β来估计当前的不透明度/>即Next, use the resulting and β to estimate the current opacity/> Right now
其中δi＝ti+1-ti，σ表示体密度，其通过体密度转换函数σβ(s)将符号距离函数值s转换为体密度，即：Wherein δi =ti+1 -ti , σ represents the volume density, which is converted from the signed distance function value s to the volume density by the volume density conversion function σβ (s), that is:
最后，通过逆变换采样来得到最终的样本集/>Finally, the inverse transform is used to sample To get the final sample set/>
进一步的，步骤S203中具体处理如下：Furthermore, the specific processing in step S203 is as follows:
首先基于步骤S1中得到的可靠深度图DMVS提取出目标图像中的像素x对应的深度值dMVS，并将其投影到3D空间中，得到pMVS；First, based on the reliable depth map DMVS obtained in step S1, the depth value dMVS corresponding to the pixel x in the target image is extracted, and it is projected into the 3D space to obtain pMVS ;
之后，基于如下公式将3D点pMVS转换为在射线r到相机中心o的距离tMVS：After that, the 3D point pMVS is converted to the distance tMVS from the ray r to the camera center o based on the following formula:
接着，在中均匀地PMVS个样本得到本集/>其中表示第i个和第i+1个样本之间的假设间隔，Pt是用于控制假设间隔Ht的预定义标量；对于没有可靠深度值dMVS的像素x，则从步骤S202中产生的候选样本集/>中随机选取PMVS个样本来构建样本集/>Then, in Evenly distribute PMVS samples in this set to obtain/> in represents the hypothesis interval between the i-th and i+1-th samples,Pt is a predefined scalar used to control the hypothesis intervalHt ; for a pixel x without a reliable depth value dMVS , the candidate sample set generated in step S202 is Randomly select PMVS samples to construct the sample set/>
进一步的，S3编码步骤具体处理如下：Furthermore, the S3 encoding step is specifically processed as follows:
首先，在空间中构建一个带有可学习参数θ的多分辨率特征网格其分为L个层级，每个层级含有T个特征向量，即每个层级的哈希表大小为T，而每个特征向量的维度为F；然后，对于每个待编码三维点/>通过插值来得到每个层级的哈希网格上的编码特征/>并将它们连接在一起形成哈希编码特征向量h(p)。First, a multi-resolution feature grid with learnable parameters θ is constructed in space It is divided into L levels, each level contains T feature vectors, that is, the hash table size of each level is T, and the dimension of each feature vector is F; then, for each 3D point to be encoded/> The encoded features on the hash grid at each level are obtained by interpolation/> And concatenate them together to form the hash-coded feature vector h(p).
进一步的，S7重要点渲染步骤具体处理如下：Furthermore, the S7 important point rendering step is specifically processed as follows:
S701：基于如下公式，并根据学习到的SDFs值s定位得到位于SDF场的零水平级的点的深度/>S701: Based on the following formula, locate the point at the zero level of the SDF field according to the learned SDFs value s Depth/>
S702：在[tn,tf]内均匀采样Q个点，这Q个点的深度和以及/>共同构成了一个新的样本集/>S702: uniformly sample Q points in [tn , tf ], and the depth and and/> Together they form a new sample set/>
S703：在样本集的基础上执行步骤S3至S6的操作，得到重要性渲染的颜色S703: In the sample set Based on this, the operations of steps S3 to S6 are performed to obtain the color of importance rendering.
7.根据权利要求6所述的一种基于先验驱动的多视图神经隐式表面重建方法，其特征在于，S8监督步骤具体处理包括如下：7. According to the multi-view neural implicit surface reconstruction method based on prior drive in claim 6, it is characterized in that the specific processing of the S8 supervision step includes the following:
S801：计算可见性感知的特征一致性损失；首先，基于S701得到的可以得到点的位置；然后，基于当前目标视图Ir中像素x对应的3D点/>通过平面诱导单应性Hm得到第m个邻域视图Im中对应的像素xm：S801: Calculate the visibility-aware feature consistency loss; first, based on S701 Can get points The position of; then, based on the 3D point corresponding to pixel x in the current target view Ir /> The corresponding pixel xm in the mth neighborhood view Im is obtained through the plane-induced homography Hm :
其中，Kr、Rr、tr和Km、Rm、tm分别表示参考视图和第m个邻域视图的相机内参、旋转和平移；Where Kr , Rr , tr and Km , Rm , tm represent the camera intrinsic parameters, rotation and translation of the reference view and the mth neighboring view respectively;
接着，基于从步骤S1中从预训练的多视图立体视觉网络提取的参考视图和邻域视图的图像特征，提取像素x的图像特征和其对应像素xm的图像特征；为了增强表示能力，提取以像素x为中心的大小为Q×Q块qr对应的图像特征Fr和qr对应的块qm的相应特征Fm；Fr和Fm的光度一致性用余弦相似度来衡量：Next, based on the image features of the reference view and the neighborhood view extracted from the pre-trained multi-view stereo vision network in step S1, the image features of pixel x and its corresponding pixelxm are extracted; in order to enhance the representation capability, the image featureFr corresponding to the blockqr ofsize Q×Q centered on pixel x and the corresponding featureFm of the blockqm corresponding to qr are extracted; the photometric consistency ofFr andFm is measured by cosine similarity:
最后，为了处理可见性的问题，进一步引入了步骤S1中从预训练的多视图立体视觉网络提取邻域视图的可见性图，并得到如下式所示的可见性感知的特征一致性损失Lvfc：Finally, in order to deal with the visibility problem, the visibility map of the neighborhood view extracted from the pre-trained multi-view stereo vision network in step S1 is further introduced, and the visibility-aware feature consistency loss Lvfc is obtained as shown in the following formula:
其中，Np和M分别表示小批量中像素的数量和邻域视图的数量，Nq表示块qr中的像素数量；WhereNp and M represent the number of pixels and the number of neighborhood views in a mini-batch, respectively, andNq represents the number of pixels in a blockqr ;
S802：基于如下公式计算法向一致性损失Lnormal：S802: Calculate the normal consistency loss Lnormal based on the following formula:
其中，表示由步骤S6中得到的射线r的法向，/>表示从预训练的Ominidata模型中提取的单目法向；in, represents the normal direction of the ray r obtained in step S6, /> represents the monocular normal extracted from the pre-trained Ominidata model;
S803：基于如下公式计算几何偏置损失Lbias：S803: Calculate the geometric bias loss Lbias based on the following formula:
其中，表示通过步骤S801计算得到的学习到的SDF的零水平集点的位置，S表示小批量中的点/>的几何，|S|表示/>的个数，/>表示/>的绝对值；in, represents the position of the zero level set point of the learned SDF calculated in step S801, and S represents the point in the mini-batch/> The geometry of |S| represents/> The number of, /> Indicates/> The absolute value of
S804：基于如下公式计算平滑性损失Lsmooth：S804: Calculate the smoothness loss Lsmooth based on the following formula:
其中，n(p)表示通过步骤S5得到的法向，ε表示值很小的常数，X是均匀采样点与近表面点的集合；Wherein, n(p) represents the normal direction obtained by step S5, ε represents a very small constant, and X is a set of uniform sampling points and near-surface points;
S805：基于如下公式计算Eikonal损失Leikonal：S805: Calculate the Eikonal lossLeikonal based on the following formula:
S806：基于如下公式计算颜色损失Lrgb和S806: Calculate the color loss Lrgb and
其中，和/>分别表示通过步骤S6和步骤S7计算得到的颜色值，C(r)表示真值颜色值，R表示小批量中射线的集合。in, and/> They represent the color values calculated by step S6 and step S7 respectively, C(r) represents the true color value, and R represents the set of rays in the small batch.
本发明相比现有技术优点在于：Compared with the prior art, the present invention has the following advantages:
本发明为了解决几何网络在优化过程中的欠约束问题，提出了可见性感知的特征一致性损失，可显著提高重建表面模型的几何一致性。此外，提出了深度先验辅助采样策略，有助于定位表面上点的位置。为了缓解由体渲染带来的有偏表面渲染问题，提出了内部先验引导重要性渲染，可提高重建表面模型的保真度。In order to solve the under-constraint problem of geometric networks in the optimization process, the present invention proposes a visibility-aware feature consistency loss, which can significantly improve the geometric consistency of the reconstructed surface model. In addition, a depth prior-assisted sampling strategy is proposed to help locate the position of points on the surface. In order to alleviate the biased surface rendering problem caused by volume rendering, an internal prior-guided importance rendering is proposed to improve the fidelity of the reconstructed surface model.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1为本发明的整体架构示意图。FIG1 is a schematic diagram of the overall structure of the present invention.
具体实施方式Detailed ways
下面详细描述本发明的实施方式，其中自始至终相同或类似的标号表示相同或类似的元件或类似功能的元件。下面通过参考附图描述的实施方式是示例性的，仅用于解释本发明而不能作为对本发明的限制。The embodiments of the present invention are described in detail below, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and cannot be used as a limitation of the present invention.
本技术领域技术人员可以理解的是，除非另外定义，这里使用的所有术语(包括技术术语和科技术语)具有与本发明所属领域中的普通技术人员的一般理解相同的意义。还应该理解的是，诸如通用字典中定义的那些术语应该被理解为具有与现有技术的上下文中的意义一致的意义，并且除非像这里一样的定义，不会用理想化或过于正式的含义来解释。It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the field to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.
各实施方式中提到的有关于步骤的标号，仅仅是为了描述的方便，而没有实质上先后顺序的联系。各具体实施方式中的不同步骤，可以进行不同先后顺序的组合，实现本发明的发明目的。The step numbers mentioned in each embodiment are only for the convenience of description and have no substantial sequence relationship. Different steps in each specific embodiment can be combined in different sequences to achieve the purpose of the invention.
下面结合附图和具体实施方式对本发明进一步说明。The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.
如图1所示，一种基于先验驱动的多视图神经隐式表面重建方法，包括如下步骤：As shown in FIG1 , a multi-view neural implicit surface reconstruction method based on prior-driven includes the following steps:
S1先验提取步骤：给定一个场景的K幅图像及各图像对应的相机参数，从K幅场景图像中依次选择一幅图像为参考图像，与其最邻近的N-1幅图像为邻域图像，并进行先验提取；S1 prior extraction step: given K images of a scene and the camera parameters corresponding to each image, select one image from the K scene images as the reference image, and the N-1 images closest to it as the neighborhood images, and perform prior extraction;
首先，将每幅参考图像和其对应的邻域图像，输入到预训练好的多视图立体视觉网络中，得到该参考图像对应的深度图DMVS、邻域图像对应的可见性图Vm，以及参考图像和邻域图像对应的图像特征Fr和Fm；该过程将一直重复，直到所有图像都处理完即得到所有图像的深度图，之后根据光度一致性和几何一致性滤除每张图像深度图中不可靠的深度部分，只留下可靠的深度；First, each reference image and its corresponding neighborhood image are input into the pre-trained multi-view stereo vision network to obtain the depth map DMVS corresponding to the reference image, the visibility map Vm corresponding to the neighborhood image, and the image features Fr and Fm corresponding to the reference image and the neighborhood image; this process will be repeated until all images are processed and the depth maps of all images are obtained. After that, the unreliable depth part of each image depth map is filtered out according to the photometric consistency and geometric consistency, leaving only the reliable depth;
其中，预训练好的多视图立体视觉网络采用了CasMVSNet的级联架构。构造代价体时，采用了PVSNet中的可见性感知代价体方式，在训练时采用不确定性感知的损失进行监督，从而以无监督的方式得到每张深度图对应的不确定性。更具体地说，在通过多视图图像特征网络提取出参考图像和邻域图像对应的深度特征Fr和后，在级联架构的第1个阶段，在整个场景深度范围R1内均匀采样L1个深度假设值。然后，通过可微单应变换，在每个深度假设下，将第m幅邻域视图的深度特征/>投影变换至参考视图下，再利用组相关度量来构建两视图代价体Mm。接着，对于第m个两视图代价体Mm，采用浅层3D CNN来估计可见性图Vm，并基于每幅领域视图的可见性图Vm，对所有的两视图代价体进行加权求和，得到最终聚合的代价体M。最后，利用三维卷积神经网络对上一步得到的代价体M进行正则化，得到具有两个通道的正则化后的代价体，并通过Softmax对代价体的两个通道分别进行操作，得到深度概率体和不确定性概率体，并基于该深度概率体，采用soft-argmax得到深度图对不确定性概率体进熵操作得到不确定性图/>在级联架构的第2阶段和第3阶段，根据上一阶段估计的深度图/>确定该阶段的深度假设采样范围Rs，并在该深度范围内进行均匀采样Ls个深度假设值。然后，重复上述所述的代价体构造与聚合、代价体正则化、深度图和不确定性回归步骤，直到得到第3阶段的深度图DMVS。在多视图立体视觉网络的训练过程中，每个阶段得到的深度图都采用下述不确定感知损失函数进行监督：The pre-trained multi-view stereo vision network uses the cascade architecture of CasMVSNet. When constructing the cost body, the visibility-aware cost body method in PVSNet is adopted, and the uncertainty-aware loss is used for supervision during training, so as to obtain the uncertainty corresponding to each depth map in an unsupervised manner. More specifically, the depth features F r and Fr corresponding to the reference image and the neighborhood image are extracted through the multi-view image feature network. Finally, in the first stage of the cascade architecture, L1 depth hypothesis values are uniformly sampled over the entire scene depth range R1. Then, through a differentiable homography, the depth features of the mth neighborhood view are transformed under each depth hypothesis. Project the transformation to the reference view, and then use the group correlation metric to construct the two-view cost volume Mm . Next, for the mth two-view cost volume Mm , a shallow 3D CNN is used to estimate the visibility map Vm , and based on the visibility map Vm of each domain view, all two-view cost volumes are weighted summed to obtain the final aggregated cost volume M. Finally, the cost volume M obtained in the previous step is regularized using a three-dimensional convolutional neural network to obtain a regularized cost volume with two channels, and the two channels of the cost volume are operated separately by Softmax to obtain a depth probability volume and an uncertainty probability volume, and based on the depth probability volume, soft-argmax is used to obtain the depth map Perform entropy operation on the uncertainty probability body to obtain the uncertainty graph/> In the second and third stages of the cascade architecture, the depth map estimated in the previous stage is used to Determine the depth hypothesis sampling range Rs of this stage, and uniformly sample Ls depth hypothesis values within this depth range. Then, repeat the cost volume construction and aggregation, cost volume regularization, depth map and uncertainty regression steps described above until the depth map DMVS of the third stage is obtained. During the training process of the multi-view stereo vision network, the depth map obtained at each stage is supervised using the following uncertainty perception loss function:
其中，和/>分别表示第s个阶段的真值深度和预测得到的深度，in, and/> Represent the true depth and predicted depth of the sth stage respectively,
光度一致性基于多视图立体视觉网络预测深度图过程产生的确定性图对预测的深度图进行过滤，3个阶段的过滤阈值均为0.6，即对于预测的深度图DMVS的某个像素在三个阶段确定性图中对应的确定性值均大于0.6，则该深度被认为是可靠的，否则会被认为是不可靠的。几何一致性计算重投影误差来对预测的深度过滤，在至少5个视图下的重投影坐标误差小于1和重投影深度误差小于0.01的深度被认为是可靠的。Photometric consistency is a deterministic graph generated by the process of predicting depth maps based on multi-view stereo vision networks. The predicted depth map is filtered, and the filtering threshold of the three stages is 0.6. That is, if the corresponding certainty value of a certain pixel in the predicted depth map DMVS in the three-stage certainty map is greater than 0.6, the depth is considered reliable, otherwise it is considered unreliable. The geometric consistency calculates the reprojection error to filter the predicted depth. The depth with a reprojection coordinate error less than 1 and a reprojection depth error less than 0.01 in at least 5 views is considered reliable.
S2图像训练步骤：对于一个场景的K幅图像，从中随机选择一幅图像作为目标图像进行训练；S2 image training step: for K images of a scene, randomly select one image as the target image for training;
首先，在选择的图像中随机选择Np个像素x，根据选择的图像对应的相机参数，假设射线r从相机中心o沿着其视角方向v穿过像素x进行投射；之后，采用误差有界采样和深度先验辅助采样在射线r上采样P个深度值进而得到射线上的P个采样点；First, randomly select Np pixels x in the selected image. According to the camera parameters corresponding to the selected image, assume that ray r is projected from the camera center o along its viewing direction v through pixel x. Then, error-bounded sampling and depth prior-assisted sampling are used to sample P depth values on ray r. Then P sampling points on the ray are obtained;
S2图像训练具体包括如下步骤：S2 image training specifically includes the following steps:
S201：对于一个场景的K幅图像，从中随机选择一幅图像作为目标图像进行训练，并在该图像中随机选择Np个像素x，根据该图像对应的相机参数，假设射线r从相机中心o沿着其视角方向v穿过像素x进行投射；S201: For K images of a scene, one image is randomly selected as the target image for training, and Np pixels x are randomly selected in the image. According to the camera parameters corresponding to the image, it is assumed that the ray r is projected from the camera center o along its viewing direction v through the pixel x;
S202：采用误差有界采样在射线r上采样PE个采样点得到样本集具体处理如下：S202: Use bounded error sampling to sample PE sampling points on ray r to obtain a sample set The specific processing is as follows:
首先对误差阈值ε和可学习参数β分别随机初始化一个大于0的值，在[tn,tf]中进行均匀采样得到初始样本即/>并初始化β+使其满足β+>β，从而能够使误差界限/>满足条件/>First, the error threshold ε and the learnable parameter β are randomly initialized to a value greater than 0, and the initial sample is obtained by uniform sampling in [tn ,tf ] That is/> And initialize β+ to satisfy β+ >β, so that the error bound /> Satisfy the conditions/>
之后为了在保持的同时减少β+，对/>进行上采样得到候选样本集/>若/>被充分上采样后满足/>则将β+向β方向减小；该采样算法在/>的情况下会持续迭代，根据中值定理，存在β*∈(β,β+)，使得/>并使用最多10次迭代的等分方法来寻找β*，并更新β+的值；上述上采样、迭代过程将循环进行直到达到循环的最大次数或/>Then in order to maintain while reducing β+ , to/> Up-sample to obtain candidate sample set/> If/> After being fully upsampled, it satisfies/> Then reduce β+ to β; the sampling algorithm is in/> In the case of continuous iteration, according to the mean value theorem, there exists β* ∈(β,β+ ) such that/> And use the equal division method with a maximum of 10 iterations to find β* and update the value of β+ ; the above upsampling and iterative process will be repeated until the maximum number of cycles is reached or/>
接着，使用最终得到的和β来估计当前的不透明度/>即Next, use the resulting and β to estimate the current opacity/> Right now
其中δi＝ti+1-ti，σ表示体密度，其通过体密度转换函数σβ(s)将符号距离函数(Signed Distance Functions,SDFs)值s转换为体密度，即：Where δi =ti+1 -ti , σ represents the volume density, which is converted from the signed distance function (SDFs) value s to the volume density through the volume density conversion function σβ (s), that is:
最后，通过逆变换采样来得到最终的样本集/>Finally, the inverse transform is used to sample To get the final sample set/>
S203：采用深度先验辅助采样在射线r上采样PMVS个采样点得到样本集具体处理如下：S203: Use depth prior assisted sampling to sample PMVS sampling points on ray r to obtain a sample set The specific processing is as follows:
首先基于步骤S1中得到的可靠深度图DMVS提取出目标图像中的像素x对应的深度值dMVS，并将其投影到3D空间中，得到pMVS；First, based on the reliable depth map DMVS obtained in step S1, the depth value dMVS corresponding to the pixel x in the target image is extracted, and it is projected into the 3D space to obtain pMVS ;
之后，基于如下公式将3D点pMVS转换为在射线r到相机中心o的距离tMVS：After that, the 3D point pMVS is converted to the distance tMVS from the ray r to the camera center o based on the following formula:
接着，在中均匀地PMVS个样本得到本集/>其中表示第i个和第i+1个样本之间的假设间隔，Pt是用于控制假设间隔Ht的预定义标量；对于没有可靠深度值dMVS的像素x，则从步骤S202中产生的候选样本集/>中随机选取PMVS个样本来构建样本集/>Then, in Evenly distribute PMVS samples in this set to obtain/> in represents the hypothesis interval between the i-th and i+1-th samples,Pt is a predefined scalar used to control the hypothesis intervalHt ; for a pixel x without a reliable depth value dMVS , the candidate sample set generated in step S202 is Randomly select PMVS samples to construct the sample set/>
S204：将基于误差有界采样得到的样本集和深度先验辅助采样得到的样本集结合得到最终的样本集，并将样本集中的采样值/>转换为射线上的三维点tn和tf分别表示射线的近边界和远边界。S204: The sample set obtained based on bounded error sampling And the sample set obtained by deep prior assisted sampling Combine to get the final sample set, and put the sample values in the sample set/> Convert to a 3D point on the ray tn and tf represent the near and far boundaries of the ray respectively.
S3编码步骤：构建多尺度哈希编码并将S2图像训练步骤中采样得到的三维点进行编码；具体处理如下：S3 encoding step: construct a multi-scale hash code and encode the three-dimensional points sampled in the S2 image training step; the specific process is as follows:
首先，在空间中构建一个带有可学习参数θ的多分辨率特征网格其分为L个层级，每个层级含有T个特征向量，即每个层级的哈希表大小为T，而每个特征向量的维度为F；每个层级的特征网格是独立的，特征向量被存储在网格的顶点上。为了结合不同频率的特征，网格的分辨率是在几何空间中的采样，第l层级的哈希网格分辨率Rl可以由如下公式定义，其中Rmax为网格的最高分辨率，Rmin为网格的最低分辨率。First, a multi-resolution feature grid with learnable parameters θ is constructed in space It is divided into L levels, each level contains T feature vectors, that is, the hash table size of each level is T, and the dimension of each feature vector is F; the feature grid of each level is independent, and the feature vectors are stored on the vertices of the grid. In order to combine features of different frequencies, the resolution of the grid is sampled in the geometric space. The hash grid resolution Rl of the lth level can be defined by the following formula, where Rmax is the highest resolution of the grid and Rmin is the lowest resolution of the grid.
然后，对于每个待编码三维点通过插值来得到每个层级的哈希网格上的编码特征/>并将它们连接在一起形成哈希编码特征向量h(p)。h(p)具体计算公式如下：Then, for each 3D point to be encoded The encoded features on the hash grid at each level are obtained by interpolation/> And connect them together to form a hash code feature vector h(p). The specific calculation formula of h(p) is as follows:
S4 SDF处理步骤：对于射线上采样的三维点和其对应的哈希编码特征h(p)，首先通过基于傅里叶级数的位置编码对三维点/>进行编码得到位置编码特征γ(p)；γ(p)具体计算公式如下：S4 SDF processing step: for the 3D points sampled on the ray And its corresponding hash code feature h(p), firstly, the three-dimensional point is encoded by Fourier series based position coding/> Encoding is performed to obtain the position encoding feature γ(p); the specific calculation formula of γ(p) is as follows:
γ(p)＝{sin(20πp),cos(20πp),…,sin(2L-1πp),cos(2L-1πp)}γ(p)＝{sin(20 πp),cos(20 πp),…,sin(2L-1 πp),cos(2L-1 πp)}
之后，将位置编码特征γ(p)和哈希编码特征h(p)进行结合并输入到由两层多层感知机(Multi-Layer Perceptron,MLP)组成的几何网络fθ来进行SDF场学习，并输出学习到的SDFs值s和通道数为256的SDF特征；Afterwards, the position encoding feature γ(p) and the hash encoding feature h(p) are combined and input into a geometric network fθ consisting of two layers of multi-layer perceptrons (MLP) to learn the SDF field, and the learned SDFs value s and the SDF feature with 256 channels are output;
S5颜色处理步骤：将射线r的视角方向v、几何网络学习到的SDF特征和三维点的法向/>输入到由两层多层感知机MLP组成的颜色网络来对颜色场进行建模，并输出学习到的射线上r采样点/>对应的颜色值/>S5 color processing step: the viewing direction v of ray r, the SDF features learned by the geometric network and the 3D point Normal/> Input to the color network composed of two layers of multi-layer perceptron MLP to model the color field and output the learned r sampling points on the ray/> The corresponding color value />
具体的处理如下：The specific processing is as follows:
S501：基于γ(p)的计算，通过基于傅里叶级数的位置编码对视角方向v进行编码得到位置编码特征γ(v)。S501: Based on the calculation of γ(p), the viewing direction v is encoded by Fourier series-based position coding to obtain a position coding feature γ(v).
S502：根据如下公式计算三维点的法向/>S502: Calculate the three-dimensional point according to the following formula Normal/>
S503：将编码特征γ(v)、几何网络学习到的SDF特征和三维点的法向/>输入到由两层MLP组成的颜色网络来对颜色场进行建模，并输出学习到的射线上r采样点/>对应的颜色值/>S503: The encoded feature γ(v), the SDF feature learned by the geometric network and the 3D point Normal/> Input to the color network composed of two layers of MLP to model the color field and output r sampling points on the learned ray/> The corresponding color value />
S6渲染步骤：将S4 SDF处理步骤中学习到的SDFs值s转换为体密度，并基于体渲染将S5颜色处理步骤中得到的颜色值和法向/>进行渲染从而得到射线r的颜色/>法向和深度/>具体公式如下：S6 rendering step: convert the SDFs value s learned in S4 SDF processing step into volume density, and render the color value obtained in S5 color processing step based on volume and normal/> Rendering is performed to obtain the color of ray r/> Normal and depth/> The specific formula is as follows:
其中和/>分别表示射线r上第i个点的累积透射率和α值。in and/> They represent the cumulative transmittance and α value of the i-th point on the ray r respectively.
S7重要点渲染步骤：在通过上述步骤得到当前射线r的SDFs值s和渲染深度值后；构建样本集/>在样本集/>的基础上执行步骤S3至S6的操作，得到重要性渲染的颜色/>具体处理如下：S7 important point rendering step: After the above steps, the SDFs value s and rendering depth value of the current ray r are obtained After; build sample set /> In the sample set/> Based on this, the operations of steps S3 to S6 are performed to obtain the color of importance rendering/> The specific processing is as follows:
S701：基于如下公式，并根据学习到的SDFs值s定位得到位于SDF场的零水平级的点的深度/>(即位于表面上的点/>的深度)：S701: Based on the following formula, locate the point at the zero level of the SDF field according to the learned SDFs value s Depth/> (i.e. points on the surface/> depth):
点和深度为/>的点/>被认为能够指示表面上的点的位置的重要点，其被用于进行重要性渲染。point and depth is/> Points/> Important points that are considered to indicate the location of a point on the surface are used for importance rendering.
S702：在[tn,tf]内均匀采样Q个点，这Q个点的深度和以及/>共同构成了一个新的样本集/>S702: uniformly sample Q points in [tn , tf ], and the depth and and/> Together they form a new sample set/>
S703：在样本集的基础上执行步骤S3至S6的操作，得到重要性渲染的颜色S703: In the sample set Based on this, the operations of steps S3 to S6 are performed to obtain the color of importance rendering.
S8监督步骤：采用可见性感知的特征一致性损失Lvfc、法向一致性损失Lnormal、几何偏置损失Lbias、平滑性损失Lsmooth、Eikonal损失Leikonal和颜色损失Lrgb来进行监督步骤S1至S7的操作；具体处理包括如下：S8 supervision step: use visibility-aware feature consistency loss Lvfc , normal consistency loss Lnormal , geometric bias loss Lbias , smoothness loss Lsmooth , Eikonal loss Leikonal and color loss Lrgb to perform the operations of supervision steps S1 to S7; the specific processing includes the following:
S801：计算可见性感知的特征一致性损失；首先，基于S701得到的可以得到点的位置；然后，基于当前目标(参考)视图Ir中像素x对应的3D点/>通过平面诱导单应性Hm得到第m个邻域视图Im中对应的像素xm：S801: Calculate the visibility-aware feature consistency loss; first, based on S701 Can get points Then, based on the 3D point corresponding to pixel x in the current target (reference) view Ir /> The corresponding pixel xm in the mth neighborhood view Im is obtained through the plane-induced homography Hm :
其中，Kr、Rr、tr和Km、Rm、tm分别表示参考视图和第m个邻域视图的相机内参、旋转和平移；Where Kr , Rr , tr and Km , Rm , tm represent the camera intrinsic parameters, rotation and translation of the reference view and the mth neighboring view respectively;
接着，基于从步骤S1中从预训练的多视图立体视觉网络提取的参考视图和邻域视图的图像特征，提取像素x的图像特征和其对应像素xm的图像特征；为了增强表示能力，提取以像素x为中心的大小为Q×Q块qr对应的图像特征Fr和qr对应的块qm的相应特征Fm；Fr和Fm的光度一致性用余弦相似度来衡量：Next, based on the image features of the reference view and the neighborhood view extracted from the pre-trained multi-view stereo vision network in step S1, the image features of pixel x and its corresponding pixelxm are extracted; in order to enhance the representation capability, the image featureFr corresponding to the blockqr ofsize Q×Q centered on pixel x and the corresponding featureFm of the blockqm corresponding to qr are extracted; the photometric consistency ofFr andFm is measured by cosine similarity:
最后，为了处理可见性的问题，进一步引入了步骤S1中从预训练的多视图立体视觉网络提取邻域视图的可见性图，并得到如下式所示的可见性感知的特征一致性损失Lvfc：Finally, in order to deal with the visibility problem, the visibility map of the neighborhood view extracted from the pre-trained multi-view stereo vision network in step S1 is further introduced, and the visibility-aware feature consistency loss Lvfc is obtained as shown in the following formula:
其中，Np和M分别表示小批量(batch)中像素的数量和邻域视图的数量，Nq表示块qr中的像素数量；WhereNp and M represent the number of pixels and the number of neighborhood views in a mini-batch, respectively, andNq represents the number of pixels in a blockqr ;
S802：基于如下公式计算法向一致性损失Lnormal：S802: Calculate the normal consistency loss Lnormal based on the following formula:
其中，表示由步骤S6中得到的射线r的法向，/>表示从预训练的Ominidata模型中提取的单目法向；in, represents the normal direction of the ray r obtained in step S6, /> represents the monocular normal extracted from the pre-trained Ominidata model;
S803：基于如下公式计算几何偏置损失Lbias：S803: Calculate the geometric bias loss Lbias based on the following formula:
其中，表示通过步骤S801计算得到的学习到的SDF的零水平集点的位置，S表示小批量中的点/>的几何，|S|表示/>的个数，/>表示/>的绝对值；in, represents the position of the zero level set point of the learned SDF calculated in step S801, and S represents the point in the mini-batch/> The geometry of |S| represents/> The number of, /> Indicates/> The absolute value of
S804：基于如下公式计算平滑性损失Lsmooth：S804: Calculate the smoothness loss Lsmooth based on the following formula:
其中，n(p)表示通过步骤S5得到的法向，ε表示值很小的常数，X是均匀采样点与近表面点的集合；Wherein, n(p) represents the normal direction obtained by step S5, ε represents a very small constant, and X is a set of uniform sampling points and near-surface points;
S805：基于如下公式计算Eikonal损失Leikonal：S805: Calculate the Eikonal lossLeikonal based on the following formula:
S806：基于如下公式计算颜色损失Lrgb和S806: Calculate the color loss Lrgb and
其中，和/>分别表示通过步骤S6和步骤S7计算得到的颜色值，C(r)表示真值颜色值，R表示小批量中射线的集合。in, and/> They represent the color values calculated by step S6 and step S7 respectively, C(r) represents the true color value, and R represents the set of rays in the small batch.
S9表面模型获取步骤：在训练过程中不断重复步骤S2至S7的操作，直至达到预设的迭代次数；训练完成后，采用MarchingCube算法从几何网络学习到的SDF场的零水平级中提取场景的表面模型。S9 surface model acquisition step: during the training process, the operations of steps S2 to S7 are continuously repeated until the preset number of iterations is reached; after the training is completed, the MarchingCube algorithm is used to extract the surface model of the scene from the zero level of the SDF field learned by the geometric network.
本发明提供的神经隐式表面重建方法对重建的表面精度有显著增益，增益主要来自以下三个方面：首先通过可见性感知特征一致性损失来对学习到的SDF场的零水平级进行约束，可显著提高重建表面模型的几何一致性；同时，引入深度先验辅助采样策略，有助于定位表面上点的位置；在此基础上，引入了内部先验引导重要性渲染策略，可缓解由体渲染带来的有偏表面渲染问题，从而能够在进一步提高重建表面模型的保真度。The neural implicit surface reconstruction method provided by the present invention has a significant gain in the reconstructed surface accuracy, and the gain mainly comes from the following three aspects: first, the zero-level level of the learned SDF field is constrained by the visibility-aware feature consistency loss, which can significantly improve the geometric consistency of the reconstructed surface model; at the same time, the introduction of a depth prior auxiliary sampling strategy is helpful to locate the position of points on the surface; on this basis, an internal prior guided importance rendering strategy is introduced to alleviate the biased surface rendering problem caused by volume rendering, thereby being able to further improve the fidelity of the reconstructed surface model.
需要说明的是，在上述实施例中，没有详细描述的部分，可以通过常规技术手段实现故不再赘述。It should be noted that in the above embodiments, parts that are not described in detail can be implemented by conventional technical means and will not be described in detail.
本领域内的技术人员应明白，本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
本发明是参照根据本发明实施例的方法、设备(系统)、和计算机程序产品的流程图和/或方框图来描述。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框、以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式计算机或者其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present invention is described with reference to the flowchart and/or block diagram of the method, device (system), and computer program product according to the embodiment of the present invention. It should be understood that each process and/or box in the flowchart and/or block diagram, as well as the combination of the process and/or box in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded computer or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one process or multiple processes in the flowchart and/or one box or multiple boxes in the block diagram.
这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能。These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device that implements the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
这些计算机程序指令也可装载到计算机或其他可编程数据处理设备上，使得在计算机或其他可编程设备上执行一系列操作步骤以产生计算机实现的处理，从而在计算机或其他可编程设备上执行的指令提供用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的步骤。These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and/or one or more boxes in the block diagram.
尽管已描述了本发明的优选实施例，但本领域内的技术人员一旦得知了基本创造概念，则可对这些实施例做出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本发明范围的所有变更和修改。Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包括这些改动和变型在内。Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311806718.5ACN118037989B (en) | 2023-12-26 | 2023-12-26 | A multi-view neural implicit surface reconstruction method based on prior-driven |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311806718.5ACN118037989B (en) | 2023-12-26 | 2023-12-26 | A multi-view neural implicit surface reconstruction method based on prior-driven |
| Publication Number | Publication Date |
|---|---|
| CN118037989Atrue CN118037989A (en) | 2024-05-14 |
| CN118037989B CN118037989B (en) | 2024-11-08 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311806718.5AActiveCN118037989B (en) | 2023-12-26 | 2023-12-26 | A multi-view neural implicit surface reconstruction method based on prior-driven |
| Country | Link |
|---|---|
| CN (1) | CN118037989B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118736123A (en)* | 2024-06-27 | 2024-10-01 | 浙江大学 | High-fidelity reconstruction method for indoor scenes based on normal deflection network |
| CN119578009A (en)* | 2025-01-27 | 2025-03-07 | 上海漫格科技有限公司 | Meshing method and system for implicit expression model, computer device, computer readable storage medium and computer program product |
| CN119722908A (en)* | 2025-02-27 | 2025-03-28 | 成都信息工程大学 | Multi-view neural implicit surface reconstruction method based on dynamic sampling range |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220189104A1 (en)* | 2020-12-14 | 2022-06-16 | Raxium, Inc. | Methods and Systems for Rendering View-Dependent Images Using 2D Images |
| WO2023015414A1 (en)* | 2021-08-09 | 2023-02-16 | 中国科学院深圳先进技术研究院 | Method for eliminating uncertainty in self-supervised three-dimensional reconstruction |
| CN117036612A (en)* | 2023-08-18 | 2023-11-10 | 武汉创升无限数字科技有限公司 | Three-dimensional reconstruction method based on nerve radiation field |
| CN117274493A (en)* | 2023-09-25 | 2023-12-22 | 浙江工商大学 | Neural implicit surface reconstruction method and device integrating depth estimation |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20220189104A1 (en)* | 2020-12-14 | 2022-06-16 | Raxium, Inc. | Methods and Systems for Rendering View-Dependent Images Using 2D Images |
| WO2023015414A1 (en)* | 2021-08-09 | 2023-02-16 | 中国科学院深圳先进技术研究院 | Method for eliminating uncertainty in self-supervised three-dimensional reconstruction |
| CN117036612A (en)* | 2023-08-18 | 2023-11-10 | 武汉创升无限数字科技有限公司 | Three-dimensional reconstruction method based on nerve radiation field |
| CN117274493A (en)* | 2023-09-25 | 2023-12-22 | 浙江工商大学 | Neural implicit surface reconstruction method and device integrating depth estimation |
| Title |
|---|
| BOWEN SONG: "PINER: Prior-informed Implicit Neural Representation Learning for Test-time Adaptation in Sparse-view CT Reconstruction", 2023 IEEE/CVF WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION (WACV), 17 March 2023 (2023-03-17)* |
| WANJUAN SU: "PSDF: Prior-Driven Neural Implicit Surface Learning for Multi-view Reconstruction", JOURNAL OF LATEX CLASS FILES, vol. 14, no. 8, 23 January 2024 (2024-01-23)* |
| 李森涛 等: "基于体积渲染和几何一致性的多视图三维重建", 工业控制计算机, vol. 36, no. 10, 26 October 2023 (2023-10-26)* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118736123A (en)* | 2024-06-27 | 2024-10-01 | 浙江大学 | High-fidelity reconstruction method for indoor scenes based on normal deflection network |
| CN119578009A (en)* | 2025-01-27 | 2025-03-07 | 上海漫格科技有限公司 | Meshing method and system for implicit expression model, computer device, computer readable storage medium and computer program product |
| CN119722908A (en)* | 2025-02-27 | 2025-03-28 | 成都信息工程大学 | Multi-view neural implicit surface reconstruction method based on dynamic sampling range |
| CN119722908B (en)* | 2025-02-27 | 2025-05-06 | 成都信息工程大学 | Multi-view nerve implicit surface reconstruction method based on dynamic sampling range |
| Publication number | Publication date |
|---|---|
| CN118037989B (en) | 2024-11-08 |
| Publication | Publication Date | Title |
|---|---|---|
| Cherabier et al. | Learning priors for semantic 3d reconstruction | |
| CN118037989B (en) | A multi-view neural implicit surface reconstruction method based on prior-driven | |
| CN110689599A (en) | 3D visual saliency prediction method for generating countermeasure network based on non-local enhancement | |
| CN113850900B (en) | Method and system for recovering depth map based on image and geometric clues in three-dimensional reconstruction | |
| CN112329662B (en) | Multi-view saliency estimation method based on unsupervised learning | |
| US20240096001A1 (en) | Geometry-Free Neural Scene Representations Through Novel-View Synthesis | |
| Li et al. | Learning to optimize non-rigid tracking | |
| CN113962846B (en) | Image alignment method and device, computer readable storage medium and electronic equipment | |
| CN117237623B (en) | A method and system for semantic segmentation of UAV remote sensing images | |
| CN114494395A (en) | Method, Apparatus, Device and Storage Medium for Depth Map Generation Based on Plane Prior | |
| Xu et al. | AutoSegNet: an automated neural network for image segmentation | |
| Uddin et al. | A perceptually inspired new blind image denoising method using $ L_ {1} $ and perceptual loss | |
| Hirner et al. | FC-DCNN: A densely connected neural network for stereo estimation | |
| Kashyap et al. | Sparse representations for object-and ego-motion estimations in dynamic scenes | |
| CN118570273A (en) | A monocular depth estimation method based on surface normal vector and neural radiance field | |
| CN119173906A (en) | Method, electronic device and storage medium for predicting depth map by multi-view stereo vision system | |
| Durasov et al. | Double refinement network for efficient monocular depth estimation | |
| CN116342776A (en) | Three-dimensional scene decoupling method, electronic device and storage medium | |
| Sharma et al. | A novel 3d-unet deep learning framework based on high-dimensional bilateral grid for edge consistent single image depth estimation | |
| Cheng et al. | Two-branch convolutional sparse representation for stereo matching | |
| CN117689747B (en) | Multi-view nerve implicit surface reconstruction method based on point cloud guidance | |
| Zhao et al. | Generalizable 3D Gaussian Splatting for novel view synthesis | |
| Tsuji et al. | Non-guided depth completion with adversarial networks | |
| Lu et al. | Scene-aware feature matching | |
| CN114078155B (en) | Method and system for training neural networks using unlabeled paired images to derive object perspectives |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |