技术领域Technical field
本文涉及但不限于图像处理技术领域，尤指一种新视点图像合成方法、系统、电子设备和存储介质。This article relates to but is not limited to the field of image processing technology, especially a new perspective image synthesis method, system, electronic equipment and storage media.
背景技术Background technique
人工智能技术和硬件计算能力的不断的发展，给视频/图像处理技术带来了新的发展契机。突破实际拍摄视频/图像的视点约束，按需生成各种视点下的新图像，逐渐成为很多应用系统的必需功能。这种从输入视频/图像中不存在的视点下生成新视频/图像的技术，称为新视点合成技术。如何得到更具真实感的新图像，是新视点图像合成技术方案努力追求的重要方面。The continuous development of artificial intelligence technology and hardware computing capabilities has brought new development opportunities to video/image processing technology. Breaking through the viewpoint constraints of actual shooting videos/images and generating new images from various viewpoints on demand has gradually become a necessary function for many application systems. This technology of generating new videos/images from viewpoints that do not exist in the input video/image is called new viewpoint synthesis technology. How to obtain new images that are more realistic is an important aspect that new viewpoint image synthesis technology solutions strive to pursue.
发明内容Contents of the invention
本申请提供了一种新视点图像合成方法、系统、电子设备和存储介质，基于优化后的相机姿态信息进行神经辐射场训练，显著提升了训练效果，能够得到更具真实感的新视点合成图像。This application provides a new viewpoint image synthesis method, system, electronic equipment and storage medium, which performs neural radiation field training based on optimized camera posture information, significantly improves the training effect, and can obtain a more realistic new viewpoint synthetic image. .
本公开实施例提供一种新视点图像合成方法，包括：Embodiments of the present disclosure provide a new viewpoint image synthesis method, including:
获取样本图像和所述样本图像对应的第一相机姿态信息；Obtain a sample image and first camera posture information corresponding to the sample image;
通过空间匹配方法，对所述样本图像的第一相机姿态信息进行优化，得到所述样本图像对应的第二相机姿态信息；Optimize the first camera attitude information of the sample image through a spatial matching method to obtain the second camera attitude information corresponding to the sample image;
根据所述样本图像和所述第二相机姿态信息，对初始神经辐射场进行训练，得到训练后的神经辐射场；According to the sample image and the second camera posture information, the initial neural radiation field is trained to obtain a trained neural radiation field;
通过所述训练后的神经辐射场进行新视点渲染，得到新视点图像。New viewpoint rendering is performed through the trained neural radiation field to obtain a new viewpoint image.
本公开实施例还提供一种新视点图像合成系统，包括：Embodiments of the present disclosure also provide a new viewpoint image synthesis system, including:
客户端和服务器；client and server;
所述客户端设置为，获取样本图像和所述样本图像对应的第一相机姿态信息；The client is configured to obtain a sample image and first camera posture information corresponding to the sample image;
所述服务器设置为，通过空间匹配方法，对所述样本图像的第一相机姿态信息进行优化，得到所述样本图像对应的第二相机姿态信息；The server is configured to optimize the first camera posture information of the sample image through a spatial matching method to obtain the second camera posture information corresponding to the sample image;
所述服务器还设置为，根据所述样本图像和所述第二相机姿态信息，对初始神经辐射场进行训练，得到训练后的神经辐射场。The server is further configured to train an initial neural radiation field based on the sample image and the second camera posture information to obtain a trained neural radiation field.
本公开实施例还提供一种电子设备，包括，一个或多个处理器；An embodiment of the present disclosure also provides an electronic device, including one or more processors;
存储装置，用于存储一个或多个程序，a storage device for storing one or more programs,
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如本公开任一实施例所述的新视点图像合成方法。When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the new viewpoint image synthesis method as described in any embodiment of the present disclosure.
本公开实施例还提供一种计算机存储介质，所述存储介质中存储有计算机程序，其中，所述计算机程序被设置为运行时执行如本公开任一实施例所述的新视点图像合成方法。An embodiment of the present disclosure also provides a computer storage medium, and a computer program is stored in the storage medium, wherein the computer program is configured to execute the new viewpoint image synthesis method as described in any embodiment of the present disclosure when running.
本公开实施例提供的新视点图像合成系统框架，采用客户端和服务器端相结合的体系架构，分布式进行算力部署，能够支持灵活的渲染执行部署，满足更多应用场景的功能要求。The new perspective image synthesis system framework provided by the embodiments of the present disclosure adopts an architecture that combines client and server, and deploys computing power in a distributed manner. It can support flexible rendering execution deployment and meet the functional requirements of more application scenarios.
本申请的其它特征和优点将在随后的说明书中阐述，并且，部分地从说明书中变得显而易见，或者通过实施本申请而了解。本申请的其他优点可通过在说明书以及附图中所描述的方案来实现和获得。Additional features and advantages of the application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the application. Other advantages of the application can be realized and obtained by the solutions described in the specification and drawings.
附图说明Description of the drawings
附图用来提供对本申请技术方案的理解，并且构成说明书的一部分，与本申请的实施例一起用于解释本申请的技术方案，并不构成对本申请技术方案的限制。The drawings are used to provide an understanding of the technical solution of the present application and constitute a part of the specification. They are used to explain the technical solution of the present application together with the embodiments of the present application and do not constitute a limitation of the technical solution of the present application.
图1为本申请实施例提供的一种新视点图像合成方法的流程图；Figure 1 is a flow chart of a new viewpoint image synthesis method provided by an embodiment of the present application;
图2为本申请实施例提供的另一种新视点图像合成方法的流程图；Figure 2 is a flow chart of another new viewpoint image synthesis method provided by an embodiment of the present application;
图3为本申请实施例提供的另一种新视点图像合成方法的流程图；Figure 3 is a flow chart of another new viewpoint image synthesis method provided by an embodiment of the present application;
图4为本申请实施例提供的另一种新视点图像合成方法的流程图；Figure 4 is a flow chart of another new viewpoint image synthesis method provided by an embodiment of the present application;
图5为本申请实施例提供的另一种新视点图像合成方法的流程图；Figure 5 is a flow chart of another new viewpoint image synthesis method provided by an embodiment of the present application;
图6为本申请实施例提供的一种新视点图像合成系统框架图；Figure 6 is a framework diagram of a new viewpoint image synthesis system provided by an embodiment of the present application;
图7为本申请实施例提供的另一种新视点图像合成系统框架图。Figure 7 is a framework diagram of another new viewpoint image synthesis system provided by an embodiment of the present application.
具体实施方式Detailed ways
本申请描述了多个实施例，但是该描述是示例性的，而不是限制性的，并且对于本领域的普通技术人员来说显而易见的是，在本申请所描述的实施例包含的范围内可以有更多的实施例和实现方案。尽管在附图中示出了许多可能的特征组合，并在具体实施方式中进行了讨论，但是所公开的特征的许多其它组合方式也是可能的。除非特意加以限制的情况以外，任何实施例的任何特征或元件可以与任何其它实施例中的任何其他特征或元件结合使用，或可以替代任何其它实施例中的任何其他特征或元件。This application describes multiple embodiments, but the description is illustrative rather than restrictive, and it is obvious to those of ordinary skill in the art that within the scope of the embodiments described in this application, There are many more embodiments and implementations. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are possible. Unless expressly limited, any feature or element of any embodiment may be used in combination with, or may be substituted for, any other feature or element of any other embodiment.
本申请包括并设想了与本领域普通技术人员已知的特征和元件的组合。本申请已经公开的实施例、特征和元件也可以与任何常规特征或元件组合，以形成由权利要求限定的独特的发明方案。任何实施例的任何特征或元件也可以与来自其它发明方案的特征或元件组合，以形成另一个由权利要求限定的独特的发明方案。因此，应当理解，在本申请中示出和/或讨论的任何特征可以单独地或以任何适当的组合来实现。因此，除了根据所附权利要求及其等同替换所做的限制以外，实施例不受其它限制。此外，可以在所附权利要求的保护范围内进行各种修改和改变。This application includes and contemplates combinations with features and elements known to those of ordinary skill in the art. The embodiments, features and elements that have been disclosed in this application may also be combined with any conventional features or elements to form unique inventive solutions as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive solutions to form another unique inventive solution as defined by the claims. Therefore, it should be understood that any feature shown and/or discussed in this application may be implemented individually or in any suitable combination. Accordingly, the embodiments are not to be limited except by those appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.
此外，在描述具有代表性的实施例时，说明书可能已经将方法和/或过程呈现为特定的步骤序列。然而，在该方法或过程不依赖于本文所述步骤的特定顺序的程度上，该方法或过程不应限于所述的特定顺序的步骤。如本领域普通技术人员将理解的，其它的步骤顺序也是可能的。因此，说明书中阐述的步骤的特定顺序不应被解释为对权利要求的限制。此外，针对该方法和/或过程的权利要求不应限于按照所写顺序执行它们的步骤，本领域技术人员可以容易地理解，这些顺序可以变化，并且仍然保持在本申请实施例的精神和范围内。Additionally, in describing representative embodiments, the specification may have presented methods and/or processes as a specific sequence of steps. However, to the extent that the method or process does not rely on the specific order of steps described herein, the method or process should not be limited to the specific order of steps described. As one of ordinary skill in the art will appreciate, other sequences of steps are possible. Therefore, the specific order of steps set forth in the specification should not be construed as limiting the claims. Furthermore, claims directed to the method and/or process should not be limited to steps performing them in the order written, as those skilled in the art can readily understand that these orders may be varied and still remain within the spirit and scope of the embodiments of the present application. Inside.
本公开实施例提供一种新视点图像合成方法，如图1所示，包括：Embodiments of the present disclosure provide a new viewpoint image synthesis method, as shown in Figure 1, including:
步骤110，获取样本图像和所述样本图像对应的第一相机姿态信息；Step 110: Obtain the sample image and the first camera posture information corresponding to the sample image;
步骤120，通过空间匹配方法，对所述样本图像的第一相机姿态信息进行优化，得到所述样本图像对应的第二相机姿态信息；Step 120: Optimize the first camera posture information of the sample image through a spatial matching method to obtain the second camera posture information corresponding to the sample image;
步骤130，根据所述样本图像和所述第二相机姿态信息，对初始神经辐射场进行训练，得到训练后的神经辐射场；Step 130: Train the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field;
步骤140，通过所述训练后的神经辐射场进行新视点渲染，得到新视点图像。Step 140: Perform new viewpoint rendering using the trained neural radiation field to obtain a new viewpoint image.
一些示例性实施例中，步骤110包括：获取待处理图像，通过即时定位与地图构建方法，获取所述待处理图像对应的相机姿态信息；In some exemplary embodiments, step 110 includes: obtaining the image to be processed, and obtaining the camera posture information corresponding to the image to be processed through the real-time positioning and map construction method;
根据视点多样性原则，选取所述待处理图像中的多帧图像作为所述样本图像，所述样本图像对应的相机姿态信息为第一相机姿态信息。According to the principle of viewpoint diversity, multiple frame images in the image to be processed are selected as the sample images, and the camera posture information corresponding to the sample images is the first camera posture information.
一些示例性实施例中，所述即时定位与地图构建方法包括：SLAM(即时定位与地图构建，Simultaneous Localization and Mapping)算法。In some exemplary embodiments, the instant positioning and map construction method includes: SLAM (Simultaneous Localization and Mapping) algorithm.
可以理解，一些示例性实施例中，所述待处理图像包括所拍摄视频中的多个帧图像，也称为多个图像帧。即通过拍摄视频的方式获取样本图像的最原始来源。可以是面向目标物体的拍摄视频，或者是环绕目标物体一周的拍摄视频，或者，其他针对目标物体的自由路径的拍摄视频，不限于特定的拍摄方式。It can be understood that in some exemplary embodiments, the image to be processed includes multiple frame images in the captured video, also referred to as multiple image frames. That is, the original source of the sample image is obtained by shooting video. It can be a video shot facing the target object, or a video shot around the target object, or other videos shot on a free path of the target object, and is not limited to a specific shooting method.
一些示例性实施例中，所述根据视点多样性原则，选取所述待处理图像中的多帧图像作为所述样本图像，包括：In some exemplary embodiments, selecting multiple frame images in the image to be processed as the sample image according to the principle of viewpoint diversity includes:
根据所述待处理图像对应的相机姿态信息，按照视点多样性原则，从所述待处理图像中选取稀疏的多帧图像作为所述样本图像。According to the camera posture information corresponding to the image to be processed, and in accordance with the principle of viewpoint diversity, sparse multi-frame images are selected from the image to be processed as the sample image.
由于待处理图像一般包括多个视点下的帧图像，根据每个帧图像对应的相机姿态信息，可以知晓其对应的视点。按照视点多样性原则选取关键帧，在全部帧图像对应的视点范围内，分散选取多个帧图像，作为关键帧，使所选取的帧图像对应的视点对全部视点的覆盖率超过第一设定比例。其中，从全部帧图像中选择第二设定比例的帧图像作为关键帧。第一设定比例和第二设定比例根据需要灵活设定。例如，从全部帧图像中选取40％(第二设定比例)的图像作为关键帧，这些关键帧对应的视点覆盖了全部帧图像视点范围的80％(第一设定比例)，任意两个关键帧对应的视点可以相同，或者，不同。Since the image to be processed generally includes frame images from multiple viewpoints, the corresponding viewpoint can be known based on the camera posture information corresponding to each frame image. Select key frames according to the principle of viewpoint diversity, and select multiple frame images scatteredly within the viewpoint range corresponding to all frame images as key frames, so that the coverage rate of the viewpoints corresponding to the selected frame images for all viewpoints exceeds the first setting Proportion. Among them, the frame image with the second set ratio is selected as the key frame from all the frame images. The first setting ratio and the second setting ratio can be set flexibly according to needs. For example, select 40% (second set ratio) of images from all frame images as key frames. The viewpoints corresponding to these key frames cover 80% (first set proportion) of the viewpoint range of all frame images. Any two The viewpoints corresponding to key frames can be the same or different.
可以理解，从待处理图像中按照视点多样性原则选取帧图像作为关键帧，可以减少样本数量，并保持样本的丰富性，能够在减小训练数据量的情况下，保证训练的有效性，确保训练效果。It can be understood that selecting frame images as key frames from the images to be processed according to the principle of viewpoint diversity can reduce the number of samples and maintain the richness of the samples. It can ensure the effectiveness of training and ensure the effectiveness of training while reducing the amount of training data. training effect.
一些示例性实施例中，所述通过空间匹配方法，对所述样本图像的第一相机姿态信息进行优化，得到所述样本图像对应的第二相机姿态信息，包括：In some exemplary embodiments, the first camera pose information of the sample image is optimized through a spatial matching method to obtain the second camera pose information corresponding to the sample image, including:
提取所述样本图像的特征点；Extract feature points of the sample image;
根据所述第一相机姿态信息，建立初始地图；Establish an initial map based on the first camera attitude information;
根据所述初始地图，获取所述样本图像之间的空间距离；According to the initial map, obtain the spatial distance between the sample images;
对所述空间距离在空间距离阈值以内的所述样本图像的特征点进行匹配，得到匹配结果；Match the feature points of the sample image whose spatial distance is within a spatial distance threshold to obtain a matching result;
根据所述匹配结果，对所述第一相机姿态信息进行优化，得到所述第二相机姿态信息。According to the matching result, the first camera attitude information is optimized to obtain the second camera attitude information.
一些示例性实施例中，所述根据所述匹配结果，对所述第一相机姿态信息进行优化，得到所述第二相机姿态信息，包括：In some exemplary embodiments, optimizing the first camera posture information according to the matching result to obtain the second camera posture information includes:
通过光束法平差对所述第一相机姿态信息进行优化，得到所述第二相机姿态信息。The first camera attitude information is optimized through beam adjustment to obtain the second camera attitude information.
一些示例性实施例中，所述通过空间匹配方法，对所述样本图像的第一相机姿态信息进行优化，得到所述样本图像对应的第二相机姿态信息，包括：In some exemplary embodiments, the first camera pose information of the sample image is optimized through a spatial matching method to obtain the second camera pose information corresponding to the sample image, including:
提取所述样本图像的特征点；Extract feature points of the sample image;
根据所述样本图像的特征点和所述样本图像对应的第一相机姿态信息，建立初始地图；Establish an initial map based on the feature points of the sample image and the first camera posture information corresponding to the sample image;
根据所述初始地图，获取所述样本图像之间的空间距离；According to the initial map, obtain the spatial distance between the sample images;
对所述空间距离在空间距离阈值以内的所述样本图像的特征点进行匹配，得到匹配结果；Match the feature points of the sample image whose spatial distance is within a spatial distance threshold to obtain a matching result;
通过光束法平差对所述初始姿态变换进行优化，得到所述相机姿态变换。The initial attitude transformation is optimized through beam adjustment to obtain the camera attitude transformation.
可以理解，对所述空间距离在空间距离阈值以内的所述样本图像的特征点进行匹配，得到匹配结果，意味着在空间距离相近的关键帧之间进行特征点匹配，再采用光束法平差对所述第一相机姿态信息进行优化，可以减少重投影误差。相比于一些可实现的SFM算法中不考虑空间距离约束而进行特征点匹配后，将关键帧置入地图的方式，本公开实施例方案对空间距离在空间距离阈值以内的样本图像(关键帧)的特征点进行匹配后，再进行优化，不但提升了计算效果，也避免了由于重复纹理导致的距离较远的帧图像出现错误的匹配。It can be understood that matching the feature points of the sample image whose spatial distance is within the spatial distance threshold and obtaining the matching result means matching the feature points between key frames with similar spatial distance, and then using the beam method to adjust Optimizing the first camera attitude information can reduce reprojection errors. Compared with some achievable SFM algorithms that do not consider spatial distance constraints and perform feature point matching and then place key frames into the map, the embodiments of the present disclosure match sample images (key frames) whose spatial distance is within the spatial distance threshold. ) feature points are matched and then optimized, which not only improves the calculation effect, but also avoids erroneous matching of distant frame images due to repeated textures.
一些示例性实施例中，步骤130包括：In some exemplary embodiments, step 130 includes:
根据所述样本图像和所述第二相机姿态信息获取多个训练样本，其中，每个训练样本由所述样本图像的像素点所发射的光线及像素点对应的颜色构成；Obtain multiple training samples according to the sample image and the second camera posture information, wherein each training sample is composed of the light emitted by the pixels of the sample image and the color corresponding to the pixels;
根据所述多个训练样本，对所述初始神经辐射场进行训练。The initial neural radiation field is trained based on the plurality of training samples.
一些示例性实施例中，所述根据所述样本图像和所述第二相机姿态信息获取多个训练样本，包括：In some exemplary embodiments, obtaining multiple training samples based on the sample image and the second camera posture information includes:
根据所述像素点所在的样本图像对应的第二相机姿态信息及所述像素点在所述样本图像中的位置，确定所述像素点所发射的光线。The light emitted by the pixel is determined according to the second camera posture information corresponding to the sample image where the pixel is located and the position of the pixel in the sample image.
一些示例性实施例中，根据所述样本图像和所述第二相机姿态信息获取多个训练样本，包括：In some exemplary embodiments, obtaining multiple training samples based on the sample image and the second camera posture information includes:
将样本图像与第二相机姿态信息解析成每一个像素点发射的光线；Analyze the sample image and the pose information of the second camera into the light emitted by each pixel;
将每一个像素点发射的光线和该像素点的颜色，构成一个样本。The light emitted by each pixel and the color of the pixel form a sample.
一些示例性实施例中，一个样本图像的每一个像素点p的颜色为c，结合该样本图像对应的第二相机姿态信息和像素点位置，可以得到该像素点所发射的光线记为l(p,d)，其中p＝(x，y，z)为三维笛卡尔空间坐标系中该像素点的位置坐标，d＝(θ,φ)为球坐标系中该光线方向的立体角参数，其中，θ表示极角或纬度，它是从参考轴(通常是正z轴)到点的矢量与参考轴之间的夹角。θ的取值范围通常是0到π，表示方位角或经度，它是从参考平面上的某个基准方向(通常是正x轴)到点的投影与基准方向之间的夹角。/>的取值范围通常是0到2π。In some exemplary embodiments, the color of each pixel p of a sample image is c. Combining the second camera posture information and the pixel position corresponding to the sample image, the light emitted by the pixel can be obtained as l ( p, d), where p = (x, y, z) is the position coordinate of the pixel point in the three-dimensional Cartesian space coordinate system, d = (θ, φ) is the solid angle parameter of the light direction in the spherical coordinate system, where θ represents the polar angle or latitude, which is the angle between the vector from the reference axis (usually the positive z-axis) to the point and the reference axis. The value range of θ is usually 0 to π, Represents azimuth or longitude, which is the angle between the projection of a point from a datum direction on a reference plane (usually the positive x-axis) and the datum direction. /> The value range of is usually 0 to 2π.
可以理解，一个样本图像包括多个像素点，对应得到多个由像素点所发射光线和颜色构成的多个训练样本，所述多个样本图像得到更多的训练样本。It can be understood that a sample image includes multiple pixels, corresponding to multiple training samples composed of light rays and colors emitted by the pixels, and more training samples are obtained from the multiple sample images.
一些示例性实施例中，根据所述多个训练样本，对所述初始神经辐射场进行训练，包括：In some exemplary embodiments, training the initial neural radiation field according to the multiple training samples includes:
将全部训练样本随机打乱后，对初始神经辐射场进行训练。After randomly disrupting all training samples, the initial neural radiation field is trained.
其中，每一个输入样本(l，c)，在划分过后的空间中对坐标p进行空间变形并通过哈希编码查询到其位置，使用该结点的多层感知机对其进行编码。编码所得的特征f和d一起使用一个全局的多层感知机进行编码，输出颜色cpred和差异度disp。Among them, for each input sample (l, c), the coordinate p is spatially deformed in the divided space and its position is queried through hash coding, and the multi-layer perceptron of the node is used to encode it. The encoded features f and d are encoded together using a global multi-layer perceptron, and the color cpred and the difference disp are output.
一些示例性实施例中，进行训练所对应的损失函数In some exemplary embodiments, the loss function corresponding to training
其中，nr是样本数量。where nr is the sample size.
一些示例性实施例中，神经辐射场中的空间划分可以是基于均匀网格的空间划分，或者是基于八叉树的空间划分。In some exemplary embodiments, the spatial division in the neural radiation field may be a spatial division based on a uniform grid, or a spatial division based on an octree.
一些示例性实施例中，所述对初始神经辐射场进行训练，包括以下步骤：空间划分、空间变形、哈希编码和训练；In some exemplary embodiments, training the initial neural radiation field includes the following steps: spatial division, spatial deformation, hash coding and training;
一些示例性实施例中，空间划分包括：对于感兴趣区域通过八叉树进行空间划分，如果存在一个可见的相机到一个结点的距离小于其边长的λ倍(例如，λ取3)，则将该结点平均划分为八个子结点。重复这个过程直到每一个结点都不可再细分。每一个结点都包含一个多层感知机。In some exemplary embodiments, spatial division includes: performing spatial division through an octree for the area of interest, if the distance from a visible camera to a node is less than λ times its side length (for example, λ is 3), Then the node is evenly divided into eight sub-nodes. Repeat this process until every node cannot be subdivided. Each node contains a multilayer perceptron.
一些示例性实施例中，空间变形包括：对八叉树的每一个子结点进行空间变形；即为了更好地对空间进行表达，需要对八叉树的每一个子结点进行空间变形，包括：In some exemplary embodiments, spatial deformation includes: performing spatial deformation on each sub-node of the octree; that is, in order to better express the space, it is necessary to perform spatial deformation on each sub-node of the octree, include:
空间中所有相机记为{Ci|i＝1…nc}，nc为相机数量，将三维空间中的点变形至相机投影空间的函数在空间中均匀采样np个点{xj|j＝1…np}，则通过y＝G(x)得到变形后的点{yj|j＝1…np}。{yj}的协方差矩阵的前三个特征向量组成的矩阵记为M，则得到了最终的空间变形函数F(x)＝M·G(x)。All cameras in the space are recorded as {Ci |i=1...nc }, nc is the number of cameras, and is a function that deforms points in the three-dimensional space to the camera projection space Sampling np points {xj |j=1...np } uniformly in space, then obtain the deformed points {yj |j=1...np } through y=G(x). The matrix composed of the first three eigenvectors of the covariance matrix of {yj } is denoted as M, and the final spatial deformation function F(x)=M·G(x) is obtained.
通过空间划分和空间变形能够有效处理灵活拍摄方式下获取的各种样本图像，改善了一些可实现方案中，需要对样本图像采集的拍摄角度进行较多约束的问题，使得在自由拍摄方式下所获取的图像，都能作为有效样本参与训练，提升训练效果。Through space division and spatial deformation, various sample images obtained in flexible shooting modes can be effectively processed, which improves the problem of requiring more constraints on the shooting angle of sample image acquisition in some achievable solutions, making all the samples captured in free shooting mode The acquired images can be used as effective samples to participate in training to improve the training effect.
一些示例性实施例中，哈希编码包括：使用多重哈希函数对经过变形的空间进行编码，从而加速空间查询。In some exemplary embodiments, hash encoding includes encoding the transformed space using multiple hash functions to accelerate spatial queries.
一些示例性实施例中，训练包括：In some exemplary embodiments, training includes:
每一个输入样本(l，c)，在划分过后的空间中对坐标p进行空间变形并通过哈希编码查询到其位置，使用该结点的多层感知机对其进行编码。编码所得的特征f和d一起使用一个全局的多层感知机进行编码，输出颜色cpred和差异度disp。For each input sample (l, c), the coordinate p is spatially deformed in the divided space and its position is queried through hash coding, and the multi-layer perceptron of the node is used to encode it. The encoded features f and d are encoded together using a global multi-layer perceptron, and the color cpred and the difference disp are output.
一些示例性实施例中，进行训练的损失函数In some exemplary embodiments, the loss function for training
其中，nr是样本数量，f0，f1是一对在空间中随机采样得到的两个相邻八叉树结点提取的特征，nb是采样数量；一些示例性实施例中，nb设为10000。Where, nr is the number of samples, f0 and f1 are features extracted from a pair of two adjacent octree nodes randomly sampled in space, nb is the number of samples; in some exemplary embodiments, nb is set to 10000.
一些示例性实施例中，神经辐射场中的空间变形可以是基于规范化设备坐标的空间变形，或者是基于透视投影坐标系的空间变形。In some exemplary embodiments, the spatial deformation in the neural radiation field may be a spatial deformation based on normalized device coordinates, or a spatial deformation based on a perspective projection coordinate system.
一些示例性实施例中，步骤140包括：根据待渲染视点信息，通过所述训练后的神经辐射场，得到所述新视点图像；In some exemplary embodiments, step 140 includes: obtaining the new viewpoint image through the trained neural radiation field according to the viewpoint information to be rendered;
其中，所述待渲染视点信息包括待渲染视点的相机姿态及预览图像分辨率，所述新视点图像的分辨率不小于所述预览图像分辨率。Wherein, the viewpoint information to be rendered includes the camera posture of the viewpoint to be rendered and the preview image resolution, and the resolution of the new viewpoint image is not less than the preview image resolution.
一些示例性实施例中，根据待渲染视点信息，通过所述训练后的神经辐射场，得到所述新视点图像，包括：In some exemplary embodiments, the new viewpoint image is obtained through the trained neural radiation field according to the viewpoint information to be rendered, including:
根据待渲染视点信息l和图像高度h、宽度w，计算出整个图像上每一个像素点的射线；According to the viewpoint information to be rendered l and the image height h and width w, the ray of each pixel on the entire image is calculated;
针对每一个像素点，通过光线前进方法，沿射线进行采样，对训练好的神经辐射场给出的采样点的颜色值进行积分，得到该像素点的颜色值。For each pixel, the ray forward method is used to sample along the ray, and the color value of the sampling point given by the trained neural radiation field is integrated to obtain the color value of the pixel.
可以知晓，根据待渲染视点信息得到全部像素点的颜色值后，最终得到整幅图像，即根据待渲染视点信息渲染得到了新视点图像。It can be known that after obtaining the color values of all pixels based on the viewpoint information to be rendered, the entire image is finally obtained, that is, a new viewpoint image is rendered based on the viewpoint information to be rendered.
一些示例性实施例中，步骤140包括：根据所述训练后的神经辐射场，生成三维模型，其中，所述三维模型包括三维网格和纹理贴图；In some exemplary embodiments, step 140 includes: generating a three-dimensional model according to the trained neural radiation field, wherein the three-dimensional model includes a three-dimensional mesh and a texture map;
通过三维渲染方法对所述三维模型进行渲染，得到所述新视点图像。The three-dimensional model is rendered using a three-dimensional rendering method to obtain the new viewpoint image.
可以理解，三维模型包括的三维网格和纹理贴图能够被用于通过三维管线渲染得到新视点图像。一些示例性实施例中，通过所述训练后的神经辐射场，得到的新视点图像，记为第一新视点图像；通过三维渲染方法对所述三维模型进行渲染，得到的新视点图像，记为第二新视点图像。It can be understood that the three-dimensional mesh and texture map included in the three-dimensional model can be used to obtain a new viewpoint image through three-dimensional pipeline rendering. In some exemplary embodiments, the new viewpoint image obtained through the trained neural radiation field is denoted as the first new viewpoint image; the new viewpoint image obtained by rendering the three-dimensional model through the three-dimensional rendering method is denoted as is the second new viewpoint image.
一些示例性实施例中，根据所述训练后的神经辐射场，生成三维模型，包括：In some exemplary embodiments, a three-dimensional model is generated based on the trained neural radiation field, including:
根据神经辐射场的训练结果，通过提取等值面算法，计算出近似神经辐射场的三角形网格G＝{V,E}；其中顶点V＝{vi|i＝1…nv}，边E＝{ei|i＝1…ne}，nv为顶点数量，ne为边数量。According to the training results of the neural radiation field, through the isosurface extraction algorithm, a triangular mesh G={V,E} that approximates the neural radiation field is calculated; where the vertices V={vi |i=1...nv }, and the edges E={ei |i=1...ne }, nv is the number of vertices, and ne is the number of edges.
为每一个顶点vi赋予偏移量Δvi，权值向量wi，每一张输入图像Ij的相机姿态pi，在对应顶点vi位置的颜色值为cij，则通过最小化以下能量函数求解出最优的Δvi与ci：Assign offset Δvi to each vertex vi , weight vector wi , camera pose pi of each input image Ij , and the color value at the position of the corresponding vertex vi is cij , then by minimizing the following The energy function solves the optimal Δvi and ci :
其中vl是vi一邻域内相邻顶点的均值，nv为顶点数量，m为图像数量。Where vl is the mean value of adjacent vertices in the neighborhood of vi , nv is the number of vertices, and m is the number of images.
根据优化的结果，更新三角形网格G的顶点，并根据优化得到的权值向量wi生成三维模型所包括的纹理贴图；所述三角形网格G＝{V,E}即为三维模型所包括的三维网格。According to the optimization results, the vertices of the triangular mesh G are updated, and the texture map included in the three-dimensional model is generated according to the weight vector wi obtained by the optimization; the triangular mesh G = {V, E} is included in the three-dimensional model 3D grid.
本公开实施例还提供一种新视点图像合成方法，如图2所示，包括：Embodiments of the present disclosure also provide a new viewpoint image synthesis method, as shown in Figure 2, including:
步骤210，第一客户端获取样本图像和所述样本图像对应的第一相机姿态信息，并发送给服务器；Step 210: The first client obtains the sample image and the first camera posture information corresponding to the sample image, and sends it to the server;
步骤220，服务器通过空间匹配方法，对所述样本图像的第一相机姿态信息进行优化，得到所述样本图像对应的第二相机姿态信息；Step 220: The server optimizes the first camera posture information of the sample image through a spatial matching method to obtain the second camera posture information corresponding to the sample image;
步骤230，服务器根据所述样本图像和所述第二相机姿态信息，对初始神经辐射场进行训练，得到训练后的神经辐射场。Step 230: The server trains the initial neural radiation field based on the sample image and the second camera posture information to obtain a trained neural radiation field.
一些示例性实施例中，如图3所示，所述方法还包括：In some exemplary embodiments, as shown in Figure 3, the method further includes:
步骤2401，第二客户端获取待渲染视点信息，并发送至所述服务器；Step 2401: The second client obtains the viewpoint information to be rendered and sends it to the server;
步骤2501，服务器根据所述待渲染视点信息，通过所述训练后的神经辐射场进行渲染，得到新视点图像，记为第一新视点图像，并将所述新视点图像返回所述第二客户端。Step 2501: The server renders through the trained neural radiation field according to the viewpoint information to be rendered, obtains a new viewpoint image, records it as the first new viewpoint image, and returns the new viewpoint image to the second client. end.
一些示例性实施例中，步骤2401包括：第二客户端获取待渲染视点信息和目标图像分辨率，并发送至所述服务器；In some exemplary embodiments, step 2401 includes: the second client obtains the viewpoint information to be rendered and the target image resolution, and sends them to the server;
相应地，步骤2501包括：服务器根据所述待渲染视点信息和目标图像分辨率，通过所述训练后的神经辐射场进行渲染，得到新视点图像，记为第一新视点图像，并将所述新视点图像返回所述第二客户端。Correspondingly, step 2501 includes: the server renders through the trained neural radiation field according to the viewpoint information to be rendered and the target image resolution, obtains a new viewpoint image, records it as the first new viewpoint image, and records the new viewpoint image as the first new viewpoint image. The new viewpoint image is returned to the second client.
一些示例性实施例中，如图4所示，所述方法还包括：In some exemplary embodiments, as shown in Figure 4, the method further includes:
步骤260，服务器根据所述训练后的神经辐射场，生成三维模型，并将所述三维模型发送至第二客户端；其中，所述三维模型包括三维网格和纹理贴图；Step 260: The server generates a three-dimensional model based on the trained neural radiation field, and sends the three-dimensional model to the second client; wherein the three-dimensional model includes a three-dimensional grid and a texture map;
步骤270，第二客户端根据待渲染视点信息，通过三维渲染方法对所述三维模型进行渲染，得到新视点图像，记为第二新视点图像。Step 270: The second client renders the three-dimensional model through a three-dimensional rendering method according to the viewpoint information to be rendered, and obtains a new viewpoint image, which is recorded as a second new viewpoint image.
一些示例性实施例中，所述第一客户端和所述第二客户端是同一个客户端，或者，不同的客户端，不限于特定的方面。In some exemplary embodiments, the first client and the second client are the same client, or different clients, which are not limited to specific aspects.
一些示例性实施例中，步骤260还包括：服务器根据第二客户端的三维模型请求，将所述三维模型发送给所述第二客户端。In some exemplary embodiments, step 260 further includes: the server sending the three-dimensional model to the second client according to the second client's three-dimensional model request.
一些示例性实施例中，如图5所示，所述方法包括：In some exemplary embodiments, as shown in Figure 5, the method includes:
步骤2401，第二客户端获取多个候选的待渲染视点信息；Step 2401: The second client obtains multiple candidate viewpoint information to be rendered;
步骤2402，第二客户端将所述多个候选的待渲染视点信息和预览图像分辨率发送给服务器；Step 2402, the second client sends the plurality of candidate viewpoint information to be rendered and preview image resolution to the server;
步骤2501，服务器根据多个候选的待渲染视点信息和预览图像分辨率，通过所述训练后的神经辐射场进行渲染，得到多个候选的新视点图像，并将所述多个候选的新视点图像返回所述第二客户端；Step 2501: The server renders through the trained neural radiation field according to the multiple candidate viewpoint information to be rendered and the preview image resolution, obtains multiple candidate new viewpoint images, and converts the multiple candidate new viewpoint images into The image is returned to the second client;
步骤2502，第二客户端响应用户的选择指令，从所述多个候选的新视点图像中确定一个待渲染视点信息；Step 2502: The second client responds to the user's selection instruction and determines viewpoint information to be rendered from the plurality of candidate new viewpoint images;
步骤2503，第二客户端将所述待渲染视点信息和目标图像分辨率发送给服务器；Step 2503, the second client sends the viewpoint information to be rendered and the target image resolution to the server;
步骤2504，服务器根据所述待渲染视点信息和目标图像分辨率，通过所述训练后的神经辐射场进行渲染，得到新视点图像(记为第一新视点图像)，并将所述新视点图像返回所述第二客户端；Step 2504: The server renders through the trained neural radiation field according to the viewpoint information to be rendered and the target image resolution, obtains a new viewpoint image (recorded as the first new viewpoint image), and converts the new viewpoint image into Return to the second client;
其中，所述目标图像分辨率大于或等于所述预览图像分辨率。Wherein, the target image resolution is greater than or equal to the preview image resolution.
可以理解，根据该实施例方案，客户端可以先提交多个新视点的预览需求到服务器，根据对应的渲染结果，确定是否是自己所需要的新视点效果，在选定后，再由服务器按照更高的分辨率渲染生成最终的新视点图像。在预览阶段，为了提高服务器响应速度，减少服务器算力浪费和网络资源浪费，渲染生成分辨率更低的预览图像。It can be understood that according to this embodiment, the client can first submit preview requirements for multiple new viewpoints to the server, and determine whether it is the new viewpoint effect it needs based on the corresponding rendering results. After selection, the server will then Higher resolution rendering produces the final new viewpoint image. In the preview phase, in order to improve the server response speed and reduce the waste of server computing power and network resources, a preview image with a lower resolution is rendered and generated.
本公开实施例还提供一种新视点图像合成系统，如图6所示，包括：Embodiments of the present disclosure also provide a new viewpoint image synthesis system, as shown in Figure 6, including:
客户端610和服务器620；Client 610 and server 620;
所述客户端610设置为，获取样本图像和所述样本图像对应的第一相机姿态信息；The client 610 is configured to obtain a sample image and the first camera posture information corresponding to the sample image;
所述服务器620设置为，通过空间匹配方法，对所述样本图像的第一相机姿态信息进行优化，得到所述样本图像对应的第二相机姿态信息；The server 620 is configured to optimize the first camera posture information of the sample image through a spatial matching method to obtain the second camera posture information corresponding to the sample image;
所述服务器620还设置为，根据所述样本图像和所述第二相机姿态信息，对初始神经辐射场进行训练，得到训练后的神经辐射场。The server 620 is further configured to train the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field.
一些示例性实施例中，所述客户端610还设置为，获取待渲染视点信息，并发送至所述服务器220；In some exemplary embodiments, the client 610 is further configured to obtain the viewpoint information to be rendered and send it to the server 220;
所述服务器620还设置为，根据所述待渲染视点信息，通过所述训练后的神经辐射场进行渲染，得到新视点图像，记为第一新视点图像，并将所述新视点图像返回所述客户端610。The server 620 is also configured to perform rendering through the trained neural radiation field according to the viewpoint information to be rendered, obtain a new viewpoint image, which is recorded as the first new viewpoint image, and return the new viewpoint image to the new viewpoint image. Described client 610.
一些示例性实施例中，所述客户端610还设置为，显示所述新视点图像。In some exemplary embodiments, the client 610 is further configured to display the new viewpoint image.
可以理解，一些示例性实施例中，神经辐射场训练和第一新视点图像的渲染生成都在服务器端执行，可以充分利用服务器的强大运算能力，保证渲染效果，显著减小了客户端的运算压力。It can be understood that in some exemplary embodiments, the neural radiation field training and the rendering generation of the first new viewpoint image are both executed on the server side, which can fully utilize the powerful computing power of the server, ensure the rendering effect, and significantly reduce the computing pressure on the client. .
一些示例性实施例中，所述服务器620还设置为，根据所述训练后的神经辐射场，生成三维模型，并将所述三维模型发送至所述客户端610，其中，所述三维模型包括三维网格和纹理贴图；In some exemplary embodiments, the server 620 is further configured to generate a three-dimensional model according to the trained neural radiation field, and send the three-dimensional model to the client 610, where the three-dimensional model includes 3D mesh and texture mapping;
所述客户端610还设置为，通过三维渲染方法对所述三维模型进行渲染，得到新视点图像，记为第二新视点图像。The client 610 is also configured to render the three-dimensional model through a three-dimensional rendering method to obtain a new viewpoint image, which is recorded as a second new viewpoint image.
一些示例性实施例中，所述客户端610还设置为，获取待渲染视点信息；根据所述待渲染视点信息，通过三维渲染方法对所述三维模型进行渲染，得到新视点图像，记为第二新视点图像。In some exemplary embodiments, the client 610 is further configured to obtain viewpoint information to be rendered; render the three-dimensional model through a three-dimensional rendering method according to the viewpoint information to be rendered, and obtain a new viewpoint image, which is recorded as the third viewpoint image. Two new viewpoint images.
可以理解，一些示例性实施例中，神经辐射场训练在服务器端执行，第二新视点图像的渲染生成在客户端执行，可以克服一些应用场景下，客户端和服务器交互实时性较差，无法及时满足应用需要的情况下，可以采用客户端本地渲染生成新视点图像的方案。It can be understood that in some exemplary embodiments, the neural radiation field training is performed on the server side, and the rendering and generation of the second new viewpoint image is performed on the client side, which can overcome the poor real-time interaction between the client and the server in some application scenarios and cannot When application needs are met in a timely manner, the client can use local rendering to generate new viewpoint images.
需要说明的是，所述客户端包括一个或多个，提供样本图像的客户端与获取待渲染视点信息的客户端可以是同一个客户端，或者，是不同的客户端，不限于特定的方面。It should be noted that the client includes one or more, and the client that provides the sample image and the client that obtains the viewpoint information to be rendered may be the same client, or they may be different clients, and are not limited to specific aspects. .
一些示例性实施例中，所述客户端610还设置为，第二客户端获取多个候选的待渲染视点信息；将所述多个候选的待渲染视点信息和预览图像分辨率发送给服务器620；In some exemplary embodiments, the client 610 is further configured such that the second client obtains multiple candidate viewpoint information to be rendered and sends the plurality of candidate viewpoint information to be rendered and the preview image resolution to the server 620 ;
所述服务器620还设置为，服务器根据多个候选的待渲染视点信息和预览图像分辨率，通过所述训练后的神经辐射场进行渲染，得到多个候选的新视点图像，并将所述多个候选的新视点图像返回客户端610。The server 620 is also configured to perform rendering through the trained neural radiation field according to multiple candidate viewpoint information to be rendered and preview image resolution, obtain multiple candidate new viewpoint images, and convert the multiple candidate new viewpoint images. The candidate new viewpoint images are returned to the client 610.
一些示例性实施例中，所述客户端610还设置为，响应用户的选择指令，从所述多个候选的新视点图像中确定一个待渲染视点信息；将所述待渲染视点信息和目标图像分辨率发送给服务器620；In some exemplary embodiments, the client 610 is further configured to, in response to the user's selection instruction, determine a viewpoint information to be rendered from the plurality of candidate new viewpoint images; combine the viewpoint information to be rendered and the target image The resolution is sent to the server 620;
所述服务器620还设置为，服务器根据所述待渲染视点信息和目标图像分辨率，通过所述训练后的神经辐射场进行渲染，得到新视点图像，记为第一新视点图像，并将所述新视点图像返回所述第二客户端；The server 620 is also configured to perform rendering through the trained neural radiation field according to the viewpoint information to be rendered and the target image resolution to obtain a new viewpoint image, which is recorded as the first new viewpoint image, and the new viewpoint image is obtained. Return the new viewpoint image to the second client;
其中，所述目标图像分辨率大于或等于所述预览图像分辨率。Wherein, the target image resolution is greater than or equal to the preview image resolution.
本公开实施例还提供一种新视点图像合成系统，如图7所示，包括：客户端610和服务器620；Embodiments of the present disclosure also provide a new viewpoint image synthesis system, as shown in Figure 7, including: a client 610 and a server 620;
所述客户端610包括：图像获取模块6110，SLAM模块6120和关键帧选取模块6130；所述服务器620包括：姿态优化模块6210，神经辐射场训练模块6220，神经辐射场渲染模块6230和三维模型生成模块6240；The client 610 includes: image acquisition module 6110, SLAM module 6120 and key frame selection module 6130; the server 620 includes: posture optimization module 6210, neural radiation field training module 6220, neural radiation field rendering module 6230 and three-dimensional model generation module 6240;
其中，所述图像获取模块6110设置为，获取待处理图像；Wherein, the image acquisition module 6110 is configured to acquire the image to be processed;
所述SLAM模块6120设置为，采用SLAM算法获取所述待处理图像对应的相机姿态信息；The SLAM module 6120 is configured to use a SLAM algorithm to obtain the camera posture information corresponding to the image to be processed;
关键帧选取模块6130设置为，根据视点多样性原则，选取所述待处理图像中的多帧图像作为所述样本图像，所述样本图像对应的相机姿态信息为第一相机姿态信息；发送所述样本图像和所述样本图像对应的第一相机姿态信息给服务器620；The key frame selection module 6130 is configured to, according to the principle of viewpoint diversity, select multiple frame images in the image to be processed as the sample image, and the camera posture information corresponding to the sample image is the first camera posture information; send the Provide the sample image and the first camera posture information corresponding to the sample image to the server 620;
姿态优化模块6210设置为，通过空间匹配方法，对所述样本图像的第一相机姿态信息进行优化，得到所述样本图像对应的第二相机姿态信息。The posture optimization module 6210 is configured to optimize the first camera posture information of the sample image through a spatial matching method to obtain the second camera posture information corresponding to the sample image.
神经辐射场训练模块6220设置为，根据所述样本图像和所述第二相机姿态信息，对初始神经辐射场进行训练，得到训练后的神经辐射场。The neural radiation field training module 6220 is configured to train the initial neural radiation field according to the sample image and the second camera posture information to obtain a trained neural radiation field.
一些示例性实施例中，所述客户端还包括：交互模块6140，所述服务器还包括：神经辐射场渲染模块6230；In some exemplary embodiments, the client further includes: an interaction module 6140, and the server further includes: a neural radiation field rendering module 6230;
其中，交互模块6140设置为，获取待渲染视点信息，并发送至所述服务器；Among them, the interaction module 6140 is configured to obtain the viewpoint information to be rendered and send it to the server;
神经辐射场渲染模块6230设置为，根据所述待渲染视点信息，通过所述训练后的神经辐射场进行渲染，得到新视点图像，并将所述新视点图像返回所述客户端；The neural radiation field rendering module 6230 is configured to perform rendering through the trained neural radiation field according to the viewpoint information to be rendered, obtain a new viewpoint image, and return the new viewpoint image to the client;
交互模块6140还设置为，显示所述新视点图像。The interaction module 6140 is also configured to display the new viewpoint image.
一些示例性实施例中，所述客户端还包括：交互模块6140和实时渲染模块6150，所述服务器还包括：三维模型生成模块6240；In some exemplary embodiments, the client also includes: an interaction module 6140 and a real-time rendering module 6150, and the server also includes: a three-dimensional model generation module 6240;
其中，交互模块6140设置为，获取待渲染视点信息，并发送给所述实时渲染模块6150；Among them, the interaction module 6140 is configured to obtain the viewpoint information to be rendered and send it to the real-time rendering module 6150;
三维模型生成模块6240设置为，根据所述训练后的神经辐射场，生成三维模型，并将所述三维模型发送至所述客户端，其中，所述三维模型包括三维网格和纹理贴图；The three-dimensional model generation module 6240 is configured to generate a three-dimensional model according to the trained neural radiation field, and send the three-dimensional model to the client, where the three-dimensional model includes a three-dimensional grid and a texture map;
实时渲染模块6150，根据待渲染视点信息，通过三维渲染方法对所述三维模型进行渲染，得到新视点图像；The real-time rendering module 6150 renders the three-dimensional model through a three-dimensional rendering method according to the viewpoint information to be rendered, and obtains a new viewpoint image;
交互模块6140还设置为，显示所述新视点图像。The interaction module 6140 is also configured to display the new viewpoint image.
本公开实施例还提供一种电子设备，包括，一个或多个处理器；An embodiment of the present disclosure also provides an electronic device, including one or more processors;
存储装置，用于存储一个或多个程序，a storage device for storing one or more programs,
当所述一个或多个程序被所述一个或多个处理器执行，使得所述一个或多个处理器实现如本公开任一实施例所述的新视点图像合成方法。When the one or more programs are executed by the one or more processors, the one or more processors are caused to implement the new viewpoint image synthesis method as described in any embodiment of the present disclosure.
本公开实施例还提供一种计算机存储介质，所述存储介质中存储有计算机程序，其中，所述计算机程序被设置为运行时执行如本公开任一实施例所述的新视点图像合成方法。An embodiment of the present disclosure also provides a computer storage medium, and a computer program is stored in the storage medium, wherein the computer program is configured to execute the new viewpoint image synthesis method as described in any embodiment of the present disclosure when running.
可以看到，根据本公开实施例提供的新视点图像合成系统架构，以分布式部署方式，结合了服务器的算力优势进行神经辐射场模型训练，以及客户端的拍摄便利性和丰富性进行初始样本采集，避免了对客户端本身就并不强大的计算资源的占用，以高稳定性、合理的计算开销分布，使实时渲染的客户体验更佳。以客户端获取的第一相机姿态信息为基础，通过空间匹配方法，进行相机姿态优化后再进行训练，可以提高样本图像的相机姿态精度，减小匹配错误，提升了训练效果。一些示例性实施中提供了服务器渲染方案，另一些示例性实施例中提供了客户端基于三维模型的渲染方案，充分满足了灵活的渲染需求。It can be seen that according to the new viewpoint image synthesis system architecture provided by the embodiments of the present disclosure, in a distributed deployment manner, it combines the computing power advantage of the server for neural radiation field model training, and the shooting convenience and richness of the client for initial samples Collection avoids the occupation of computing resources that are not powerful on the client itself. With high stability and reasonable distribution of computing overhead, the customer experience of real-time rendering is better. Based on the first camera attitude information obtained by the client, through the spatial matching method, the camera attitude is optimized and then trained, which can improve the camera attitude accuracy of the sample image, reduce matching errors, and improve the training effect. Some exemplary embodiments provide a server rendering solution, and other exemplary embodiments provide a client-side three-dimensional model-based rendering solution, fully meeting flexible rendering requirements.
本领域普通技术人员可以理解，上文中所公开方法中的全部或某些步骤、系统、装置中的功能模块/单元可以被实施为软件、固件、硬件及其适当的组合。在硬件实施方式中，在以上描述中提及的功能模块/单元之间的划分不一定对应于物理组件的划分；例如，一个物理组件可以具有多个功能，或者一个功能或步骤可以由若干物理组件合作执行。某些组件或所有组件可以被实施为由处理器，如数字信号处理器或微处理器执行的软件，或者被实施为硬件，或者被实施为集成电路，如专用集成电路。这样的软件可以分布在计算机可读介质上，计算机可读介质可以包括计算机存储介质(或非暂时性介质)和通信介质(或暂时性介质)。如本领域普通技术人员公知的，术语计算机存储介质包括在用于存储信息(诸如计算机可读指令、数据结构、程序模块或其他数据)的任何方法或技术中实施的易失性和非易失性、可移除和不可移除介质。计算机存储介质包括但不限于RAM、ROM、EEPROM、闪存或其他存储器技术、CD-ROM、数字多功能盘(DVD)或其他光盘存储、磁盒、磁带、磁盘存储或其他磁存储装置、或者可以用于存储期望的信息并且可以被计算机访问的任何其他的介质。此外，本领域普通技术人员公知的是，通信介质通常包含计算机可读指令、数据结构、程序模块或者诸如载波或其他传输机制之类的调制数据信号中的其他数据，并且可包括任何信息递送介质。Those of ordinary skill in the art can understand that all or some steps, systems, and functional modules/units in the devices disclosed above can be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between functional modules/units mentioned in the above description does not necessarily correspond to the division of physical components; for example, one physical component may have multiple functions, or one function or step may consist of several physical components. Components execute cooperatively. Some or all of the components may be implemented as software executed by a processor, such as a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on computer-readable media, which may include computer storage media (or non-transitory media) and communication media (or transitory media). As is known to those of ordinary skill in the art, the term computer storage media includes volatile and nonvolatile media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. removable, removable and non-removable media. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, Digital Versatile Disk (DVD) or other optical disk storage, magnetic cassettes, tapes, disk storage or other magnetic storage devices, or may Any other medium used to store the desired information and that can be accessed by a computer. Additionally, it is known to those of ordinary skill in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media .
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311492488.XACN117576542A (en) | 2023-11-09 | 2023-11-09 | A new viewpoint image synthesis method, system, device and storage medium |
| PCT/CN2024/103328WO2025097814A1 (en) | 2023-11-09 | 2024-07-03 | New viewpoint image synthesis method and system, electronic device, and storage medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311492488.XACN117576542A (en) | 2023-11-09 | 2023-11-09 | A new viewpoint image synthesis method, system, device and storage medium |
| Publication Number | Publication Date |
|---|---|
| CN117576542Atrue CN117576542A (en) | 2024-02-20 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311492488.XAPendingCN117576542A (en) | 2023-11-09 | 2023-11-09 | A new viewpoint image synthesis method, system, device and storage medium |
| Country | Link |
|---|---|
| CN (1) | CN117576542A (en) |
| WO (1) | WO2025097814A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025097814A1 (en)* | 2023-11-09 | 2025-05-15 | 虹软科技股份有限公司 | New viewpoint image synthesis method and system, electronic device, and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB2613336B (en)* | 2021-11-24 | 2024-03-06 | Xyz Reality Ltd | Using a neural network scene representation for mapping |
| CN115631418B (en)* | 2022-11-18 | 2023-05-16 | 北京百度网讯科技有限公司 | Image processing method and device and training method of nerve radiation field |
| CN116468917A (en)* | 2023-03-17 | 2023-07-21 | 湖北星纪魅族科技有限公司 | Image processing method, electronic device and storage medium |
| CN117576542A (en)* | 2023-11-09 | 2024-02-20 | 虹软科技股份有限公司 | A new viewpoint image synthesis method, system, device and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025097814A1 (en)* | 2023-11-09 | 2025-05-15 | 虹软科技股份有限公司 | New viewpoint image synthesis method and system, electronic device, and storage medium |
| Publication number | Publication date |
|---|---|
| WO2025097814A1 (en) | 2025-05-15 |
| Publication | Publication Date | Title |
|---|---|---|
| CN114004941B (en) | Indoor scene three-dimensional reconstruction system and method based on nerve radiation field | |
| CN108335353B (en) | Three-dimensional reconstruction method, device and system, server and medium of dynamic scene | |
| US11669986B2 (en) | Generating enhanced three-dimensional object reconstruction models from sparse set of object images | |
| CN116051747A (en) | A method, equipment, and medium for reconstructing a 3D model of a building based on missing point cloud data | |
| WO2022198684A1 (en) | Methods and systems for training quantized neural radiance field | |
| Schöning et al. | Evaluation of multi-view 3D reconstruction software | |
| CN105825543B (en) | Point off density cloud generation method and system are regarded based on low altitude remote sensing image more | |
| CN117115359A (en) | Multi-view power grid three-dimensional space data reconstruction method based on depth map fusion | |
| CN109685879B (en) | Method, device, equipment and storage medium for determining multi-view image texture distribution | |
| GB2567245A (en) | Methods and apparatuses for depth rectification processing | |
| WO2023093085A1 (en) | Method and apparatus for reconstructing surface of object, and computer storage medium and computer program product | |
| CN114429527A (en) | Remote operation scene three-dimensional reconstruction method based on SLAM | |
| WO2024037562A1 (en) | Three-dimensional reconstruction method and apparatus, and computer-readable storage medium | |
| CN117274514A (en) | Remote sensing image generation method and device based on ground-air visual angle geometric transformation | |
| CN118397282A (en) | Three-dimensional point cloud robustness component segmentation method based on semantic SAM large model | |
| CN119006678A (en) | Three-dimensional Gaussian sputtering optimization method for pose-free input | |
| CN117058302A (en) | NeRF-based generalizable scene rendering method | |
| CN117576542A (en) | A new viewpoint image synthesis method, system, device and storage medium | |
| Chen et al. | Densefusion: Large-scale online dense pointcloud and dsm mapping for uavs | |
| CN115409949A (en) | Model training method, perspective image generation method, device, equipment and medium | |
| CN119863567A (en) | Real projective image generation method based on neural network and surface confidence | |
| CN119625162A (en) | Incremental optimal view selection method for neural radiance field based on hybrid uncertainty estimation | |
| CN119107404A (en) | A sparse perspective 3D model construction method, device, equipment and storage medium | |
| WO2025060587A1 (en) | Video jitter removal method, electronic device, system, and storage medium | |
| CN117893701A (en) | Underwater environment three-dimensional reconstruction method, device, system and storage medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |