技术领域Technical field
本申请涉及终端领域，具体涉及一种增强现实视频的处理方法与电子设备。This application relates to the field of terminals, and specifically to an augmented reality video processing method and electronic device.
背景技术Background technique
增强现实(augmented reality，AR)技术是一种实时地计算摄影机影像的位置及角度并加上相应图像的技术，是一种将真实世界信息和虚拟世界信息“无缝”集成的新技术，这种技术的目标是在屏幕上把虚拟世界套在现实世界并进行互动。Augmented reality (AR) technology is a technology that calculates the position and angle of camera images in real time and adds corresponding images. It is a new technology that "seamlessly" integrates real world information and virtual world information. This The goal of this technology is to put the virtual world into the real world on the screen and interact with it.
目前，在录制AR视频时由于无法将虚拟内容和真实物体的视频较好的融合，尤其是在摄场景中需要用户与虚拟内容进行交互时，需要多次的重复拍摄，费时费力。Currently, when recording AR videos, virtual content and real object videos cannot be well integrated. Especially when users need to interact with virtual content in a shooting scene, multiple repeated shots are required, which is time-consuming and labor-intensive.
因此，如何在录制AR视频时使得虚拟内容和真实物体内容较好的融合，提高AR视频的视频质量成为一个亟需解决的问题。Therefore, how to better integrate virtual content and real object content when recording AR videos and improve the video quality of AR videos has become an urgent problem that needs to be solved.
发明内容Contents of the invention
本申请提供了一种增强现实视频的处理方法与电子设备，能够在录制AR视频时使得虚拟内容和真实物体的视频较好融合，提高AR视频的视频质量。This application provides an augmented reality video processing method and electronic device, which can better integrate virtual content and real object videos when recording AR videos, and improve the video quality of AR videos.
第一方面，提供了一种增强现实视频的处理方法，包括：The first aspect provides an augmented reality video processing method, including:
获取原始视频与位姿信息，所述原始视频用于表示真实物体的视频，所述位姿信息用于表示终端设备获取所述原始视频时的位姿；根据所述原始视频与所述位姿信息生成虚拟平面，所述虚拟平面用于确定在所述原始视频中添加虚拟内容的位置信息；根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成AR视频。Obtain the original video and pose information, the original video is used to represent the video of the real object, and the pose information is used to represent the pose when the terminal device obtains the original video; according to the original video and the pose The information generates a virtual plane, the virtual plane is used to determine the location information of adding virtual content in the original video; and the AR video is generated by adding the virtual content in the original video according to the virtual plane.
在本申请的实施例中，在获取原始视频时可以获取原始视频对应的位姿信息；根据位姿信息与原始视频可以得到虚拟平面；在原始视频的图像帧中添加虚拟内容时，虚拟平面可以作为一个基准面，根据虚拟平面可以调整虚拟内容在原始视频中的位置，使得虚拟内容能够更好的融入到原始视频中，提高AR视频的视频质量。In the embodiment of the present application, when acquiring the original video, the pose information corresponding to the original video can be obtained; the virtual plane can be obtained according to the pose information and the original video; when virtual content is added to the image frame of the original video, the virtual plane can As a reference plane, the position of the virtual content in the original video can be adjusted according to the virtual plane, so that the virtual content can be better integrated into the original video and improve the video quality of the AR video.
应理解，位姿信息用于表示终端设备的摄像头在获取原始视频时的位姿；位姿信息可以包括姿态信息与位置信息。It should be understood that the pose information is used to represent the pose of the camera of the terminal device when acquiring the original video; the pose information may include pose information and position information.
结合第一方面，在第一方面的某些实现方式中，所述位姿信息包括三维姿态信息，还包括：With reference to the first aspect, in some implementations of the first aspect, the pose information includes three-dimensional pose information, and also includes:
通过四元数表示所述三维姿态信息。The three-dimensional attitude information is represented by quaternions.
在本申请的实施例中，可以将三维姿态信息转化为四元数来表示，从而避免将姿态信息通过三个参数表示所产生的歧义。In the embodiment of the present application, the three-dimensional posture information can be converted into a quaternion to represent it, thereby avoiding the ambiguity caused by expressing the posture information through three parameters.
结合第一方面，在第一方面的某些实现方式中，所述根据所述原始视频与所述位姿信息生成虚拟平面的信息，包括：With reference to the first aspect, in some implementations of the first aspect, generating virtual plane information based on the original video and the pose information includes:
根据所述原始视频中图像帧的位姿信息提取所述图像帧的特征点；Extract feature points of the image frame according to the pose information of the image frame in the original video;
根据所述特征点生成所述虚拟平面。The virtual plane is generated based on the feature points.
应理解，图像帧的特征点可以是指图像灰度值发生剧烈变化的点，或者在图像边缘上曲率较大的点；特征点可以用于标识图像中物体。It should be understood that the feature points of the image frame may refer to points where the gray value of the image changes drastically, or points with large curvature on the edges of the image; feature points may be used to identify objects in the image.
结合第一方面，在第一方面的某些实现方式中，还包括：Combined with the first aspect, some implementations of the first aspect also include:
保存所述位姿信息与所述虚拟平面的信息。The pose information and the information of the virtual plane are saved.
在本申请的实施例中，保存位姿信息与虚拟平面的信息可以使得在原始视频录制结束后，根据原始视频的位姿信息与虚拟平面的信息在原始视频中添加虚拟内容生成一个新的AR视频；由于保存了位姿信息与虚拟平面的信息，用户可以对原始视频进行多次不同的编辑，分别生成带不同虚拟内容的AR视频。In the embodiment of the present application, saving the pose information and the virtual plane information can enable adding virtual content to the original video to generate a new AR based on the pose information and the virtual plane information of the original video after the recording of the original video. Video; since the pose information and virtual plane information are saved, users can edit the original video multiple times to generate AR videos with different virtual content.
结合第一方面，在第一方面的某些实现方式中，所述保存所述位姿信息与所述虚拟平面的信息，包括：With reference to the first aspect, in some implementations of the first aspect, the saving of the pose information and the information of the virtual plane includes:
将所述位姿信息与所述虚拟平面的信息保存在二进制文件中。The pose information and the information of the virtual plane are saved in a binary file.
在一种可能的实现方式中，终端设备可以将位姿信息与虚拟平面的信息保存为独立的二进制文件。In a possible implementation, the terminal device can save the pose information and the virtual plane information as independent binary files.
在一种可能的实现方式中，可以将原始视频与原始视频对应的位姿信息与虚拟平面的信息保存在相同的目录下。In a possible implementation, the original video and the pose information corresponding to the original video and the virtual plane information can be saved in the same directory.
在一种可能的实现方式中，可以将原始视频对应的位姿信息与虚拟平面的信息与原始视频的命名相同保存在终端设备中。In a possible implementation, the pose information and virtual plane information corresponding to the original video can be saved in the terminal device with the same names as the original video.
在一种可能的实现方式中，可以通过每个图像帧的帧号作为标识，将原始视频对应的位姿信息与虚拟平面的信息保存在终端设备中。In one possible implementation, the frame number of each image frame can be used as an identifier to store the pose information and virtual plane information corresponding to the original video in the terminal device.
结合第一方面，在第一方面的某些实现方式中，所述保存所述位姿信息与所述虚拟平面的信息，包括：With reference to the first aspect, in some implementations of the first aspect, the saving of the pose information and the information of the virtual plane includes:
将所述位姿信息与所述虚拟平面的信息保存在所述原始视频对应的补充增强信息中。The pose information and the information of the virtual plane are stored in the supplementary enhancement information corresponding to the original video.
在一种可能的实现方式中，可以将位姿信息与虚拟平面的信息进行视频压缩编码的时保存至h.264或者h.265的补充增强信息中。In a possible implementation, the pose information and the virtual plane information can be stored in h.264 or h.265 supplementary enhancement information during video compression encoding.
结合第一方面，在第一方面的某些实现方式中，还包括：Combined with the first aspect, some implementations of the first aspect also include:
对保存的所述位姿信息与所述虚拟平面的信息进行压缩处理。Compression processing is performed on the saved pose information and the information on the virtual plane.
在本申请的实施例中，在保存位姿信息与虚拟平面的信息时可以对保存的信息进行压缩处理，从而能够有效的减少保存信息占用的内存空间。In embodiments of the present application, when saving pose information and virtual plane information, the saved information can be compressed, thereby effectively reducing the memory space occupied by the saved information.
在一种可能的实现方式中，可以采用以下的至少一种方式进行对保存位姿信息与虚拟平面的信息进行压缩处理：In a possible implementation, at least one of the following methods can be used to compress the information that saves the pose information and the virtual plane:
根据当前图像帧与前一图像帧的差保存位姿信息；或者，虚拟平面的平面编号可以采用无符号字符方式保存；或者，对于虚拟平面中顶点的描述，水平面可以保留一个点的Z轴信息删除其他点的Z轴信息，垂直面可以保留一个点的Y轴信息删除其他点的Y轴信息；或者，顶点的位置描述可以采用float16；或者，保存虚拟平面的信息时可以只保存当前视野范围内的平面。Save pose information based on the difference between the current image frame and the previous image frame; alternatively, the plane number of the virtual plane can be saved in the form of unsigned characters; or, for the description of the vertices in the virtual plane, the horizontal plane can retain the Z-axis information of a point Delete the Z-axis information of other points. The vertical plane can retain the Y-axis information of one point and delete the Y-axis information of other points; or, the position description of the vertex can be float16; or, when saving the information of the virtual plane, you can only save the current field of view range. inner plane.
结合第一方面，在第一方面的某些实现方式中，所述根据所述虚拟平面的信息在所述原始视频中添加所述虚拟内容生成AR视频，包括：With reference to the first aspect, in some implementations of the first aspect, adding the virtual content to the original video to generate an AR video according to the information of the virtual plane includes:
在所述原始视频录制完成后，根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成所述AR视频。After the original video recording is completed, the virtual content is added to the original video according to the virtual plane to generate the AR video.
结合第一方面，在第一方面的某些实现方式中，所述虚拟平面包括第一虚拟平面，所述第一虚拟平面是指第一图像帧对应的虚拟平面，所述第一图像帧是所述原始视频中的任意一个图像帧；With reference to the first aspect, in some implementations of the first aspect, the virtual plane includes a first virtual plane, the first virtual plane refers to a virtual plane corresponding to the first image frame, and the first image frame is Any image frame in the original video;
所述第一虚拟平面的信息包括图像帧的总数、所述第一虚拟平面的标识、所述第一虚拟平面包括的顶点数量以及所述第一虚拟平面包括的每一个顶点的位置信息，所述总数是指所述原始视频包括图像帧的总数。The information of the first virtual plane includes the total number of image frames, the identification of the first virtual plane, the number of vertices included in the first virtual plane, and the position information of each vertex included in the first virtual plane, so The total number refers to the total number of image frames included in the original video.
第二方面，提供了一种AR视频的处理装置，所述处理装置包括获取单元与处理单元；其中，所述获取单元用于获取原始视频与位姿信息，所述原始视频用于表示真实物体的视频，所述位姿信息用于表示终端设备获取所述原始视频时的位姿；所述处理单元用于根据所述原始视频与所述位姿信息生成虚拟平面，所述虚拟平面用于确定在所述原始视频中添加虚拟内容的位置信息；根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成AR视频。In a second aspect, an AR video processing device is provided. The processing device includes an acquisition unit and a processing unit; wherein the acquisition unit is used to acquire original video and pose information, and the original video is used to represent real objects. video, the pose information is used to represent the pose when the terminal device obtains the original video; the processing unit is used to generate a virtual plane according to the original video and the pose information, the virtual plane is used to Determine the location information of adding virtual content to the original video; add the virtual content to the original video according to the virtual plane to generate an AR video.
结合第二方面，在第二方面的某些实现方式中，所述位姿信息包括三维姿态信息，所述处理单元还用于：Combined with the second aspect, in some implementations of the second aspect, the pose information includes three-dimensional pose information, and the processing unit is also used to:
通过四元数表示所述三维姿态信息。The three-dimensional attitude information is represented by quaternions.
结合第二方面，在第二方面的某些实现方式中，所述处理单元具体用于：In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to:
根据所述原始视频中图像帧的位姿信息提取所述图像帧的特征点；Extract feature points of the image frame according to the pose information of the image frame in the original video;
根据所述特征点生成所述虚拟平面。The virtual plane is generated based on the feature points.
结合第二方面，在第二方面的某些实现方式中，所述处理单元还用于：Combined with the second aspect, in some implementations of the second aspect, the processing unit is also used to:
保存所述位姿信息与所述虚拟平面的信息。The pose information and the information of the virtual plane are saved.
结合第二方面，在第二方面的某些实现方式中，所述处理单元具体用于：In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to:
将所述位姿信息与所述虚拟平面的信息保存在二进制文件中。The pose information and the information of the virtual plane are saved in a binary file.
结合第二方面，在第二方面的某些实现方式中，所述处理单元具体用于：In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to:
将所述位姿信息与所述虚拟平面的信息保存在所述原始视频对应的补充增强信息中。The pose information and the information of the virtual plane are stored in the supplementary enhancement information corresponding to the original video.
结合第二方面，在第二方面的某些实现方式中，所述处理单元还用于：Combined with the second aspect, in some implementations of the second aspect, the processing unit is also used to:
对保存的所述位姿信息与所述虚拟平面的信息进行压缩处理。Compression processing is performed on the saved pose information and the information on the virtual plane.
结合第二方面，在第二方面的某些实现方式中，所述处理单元具体用于：In conjunction with the second aspect, in some implementations of the second aspect, the processing unit is specifically used to:
在所述原始视频录制完成后，根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成所述AR视频。After the original video recording is completed, the virtual content is added to the original video according to the virtual plane to generate the AR video.
结合第二方面，在第二方面的某些实现方式中，所述虚拟平面包括第一虚拟平面，所述第一虚拟平面是指第一图像帧对应的虚拟平面，所述第一图像帧是所述原始视频中的任意一个图像帧；With reference to the second aspect, in some implementations of the second aspect, the virtual plane includes a first virtual plane, the first virtual plane refers to a virtual plane corresponding to the first image frame, and the first image frame is Any image frame in the original video;
所述第一虚拟平面的信息包括图像帧的总数、所述第一虚拟平面的标识、所述第一虚拟平面包括的顶点数量以及所述第一虚拟平面包括的每一个顶点的位置信息，所述总数是指所述原始视频包括图像帧的总数。The information of the first virtual plane includes the total number of image frames, the identification of the first virtual plane, the number of vertices included in the first virtual plane, and the position information of each vertex included in the first virtual plane, so The total number refers to the total number of image frames included in the original video.
在一种可能的实现方式中，上述AR视频的处理装置可以是指芯片。In a possible implementation, the above AR video processing device may refer to a chip.
在上述处理装置为芯片时，获取单元可以是指输出接口、管脚或电路等；处理单元可以是指芯片内部的处理单元。When the above processing device is a chip, the acquisition unit may refer to an output interface, a pin or a circuit, etc.; the processing unit may refer to a processing unit inside the chip.
应理解，在上述第一方面中对相关内容的扩展、限定、解释和说明也适用于第二方面中相同的内容。It should be understood that the expansion, limitation, explanation and description of the relevant content in the above-mentioned first aspect also apply to the same content in the second aspect.
第三方面，提供了一种电子设备，所述电子设备包括：一个或多个处理器、存储器和显示屏；所述存储器与所述一个或多个处理器耦合，所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行：In a third aspect, an electronic device is provided. The electronic device includes: one or more processors, a memory, and a display screen; the memory is coupled to the one or more processors, and the memory is used to store a computer Program code, the computer program code comprising computer instructions invoked by the one or more processors to cause the electronic device to perform:
获取原始视频与位姿信息，所述原始视频用于表示真实物体的视频，所述位姿信息用于表示终端设备获取所述原始视频时的位姿；根据所述原始视频与所述位姿信息生成虚拟平面，所述虚拟平面用于确定在所述原始视频中添加虚拟内容的位置信息；根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成AR视频。Obtain the original video and pose information, the original video is used to represent the video of the real object, and the pose information is used to represent the pose when the terminal device obtains the original video; according to the original video and the pose The information generates a virtual plane, the virtual plane is used to determine the location information of adding virtual content in the original video; and the AR video is generated by adding the virtual content in the original video according to the virtual plane.
结合第三方面，在第三方面的某些实现方式中，所述位姿信息包括三维姿态信息，所述一个或多个处理器调用所述计算机指令以使得所述电子设备还执行：In conjunction with the third aspect, in some implementations of the third aspect, the pose information includes three-dimensional pose information, and the one or more processors invoke the computer instructions to cause the electronic device to also execute:
通过四元数表示所述三维姿态信息。The three-dimensional attitude information is represented by quaternions.
结合第三方面，在第三方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备还执行：In conjunction with the third aspect, in some implementations of the third aspect, the one or more processors invoke the computer instructions to cause the electronic device to also execute:
根据所述原始视频中图像帧的位姿信息提取所述图像帧的特征点；Extract feature points of the image frame according to the pose information of the image frame in the original video;
根据所述特征点生成所述虚拟平面。The virtual plane is generated based on the feature points.
结合第三方面，在第三方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备还执行：In conjunction with the third aspect, in some implementations of the third aspect, the one or more processors invoke the computer instructions to cause the electronic device to also execute:
保存所述位姿信息与所述虚拟平面的信息。The pose information and the information of the virtual plane are saved.
结合第三方面，在第三方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备还执行：In conjunction with the third aspect, in some implementations of the third aspect, the one or more processors invoke the computer instructions to cause the electronic device to also execute:
将所述位姿信息与所述虚拟平面的信息保存在二进制文件中。The pose information and the information of the virtual plane are saved in a binary file.
结合第三方面，在第三方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备还执行：In conjunction with the third aspect, in some implementations of the third aspect, the one or more processors invoke the computer instructions to cause the electronic device to also execute:
将所述位姿信息与所述虚拟平面的信息保存在所述原始视频对应的补充增强信息中。The pose information and the information of the virtual plane are stored in the supplementary enhancement information corresponding to the original video.
结合第三方面，在第三方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备还执行：In conjunction with the third aspect, in some implementations of the third aspect, the one or more processors invoke the computer instructions to cause the electronic device to also execute:
对保存的所述位姿信息与所述虚拟平面的信息进行压缩处理。Compression processing is performed on the saved pose information and the information on the virtual plane.
结合第三方面，在第三方面的某些实现方式中，所述一个或多个处理器调用所述计算机指令以使得所述电子设备还执行：In conjunction with the third aspect, in some implementations of the third aspect, the one or more processors invoke the computer instructions to cause the electronic device to also execute:
在所述原始视频录制完成后，根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成所述AR视频。After the original video recording is completed, the virtual content is added to the original video according to the virtual plane to generate the AR video.
结合第三方面，在第三方面的某些实现方式中，所述虚拟平面包括第一虚拟平面，所述第一虚拟平面是指第一图像帧对应的虚拟平面，所述第一图像帧是所述原始视频中的任意一个图像帧；With reference to the third aspect, in some implementations of the third aspect, the virtual plane includes a first virtual plane, the first virtual plane refers to a virtual plane corresponding to the first image frame, and the first image frame is Any image frame in the original video;
所述第一虚拟平面的信息包括图像帧的总数、所述第一虚拟平面的标识、所述第一虚拟平面包括的顶点数量以及所述第一虚拟平面包括的每一个顶点的位置信息，所述总数是指所述原始视频包括图像帧的总数。The information of the first virtual plane includes the total number of image frames, the identification of the first virtual plane, the number of vertices included in the first virtual plane, and the position information of each vertex included in the first virtual plane, so The total number refers to the total number of image frames included in the original video.
应理解，在上述第一方面中对相关内容的扩展、限定、解释和说明也适用于第三方面中相同的内容。It should be understood that the expansion, limitation, explanation and explanation of relevant content in the above-mentioned first aspect also apply to the same content in the third aspect.
第四方面，提供了一种电子设备，所述电子设备包括：一个或多个处理器、存储器和显示屏；所述存储器与所述一个或多个处理器耦合，所述存储器用于存储计算机程序代码，所述计算机程序代码包括计算机指令，所述一个或多个处理器调用所述计算机指令以使得所述电子设备执行第一方面中的任一种处理方法。In a fourth aspect, an electronic device is provided. The electronic device includes: one or more processors, a memory, and a display screen; the memory is coupled to the one or more processors, and the memory is used to store a computer Program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the electronic device to execute any one of the processing methods in the first aspect.
第五方面，提供了一种芯片系统，所述芯片系统应用于电子设备，所述芯片系统包括一个或多个处理器，所述处理器用于调用计算机指令以使得所述电子设备执行第一方面中的任一种处理方法。In a fifth aspect, a chip system is provided. The chip system is applied to an electronic device. The chip system includes one or more processors. The processor is used to call computer instructions to cause the electronic device to execute the first aspect. any of the processing methods.
第六方面，提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序代码，当所述计算机程序代码被电子设备运行时，使得该电子设备执行第一方面中的任一种处理方法。In a sixth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program code. When the computer program code is run by an electronic device, it causes the electronic device to perform any of the tasks in the first aspect. A way to deal with it.
第七方面，提供了一种计算机程序产品，所述计算机程序产品包括：计算机程序代码，当所述计算机程序代码被电子设备运行时，使得该电子设备执行第一方面中的任一种处理方法。In a seventh aspect, a computer program product is provided. The computer program product includes: computer program code. When the computer program code is run by an electronic device, it causes the electronic device to execute any one of the processing methods in the first aspect. .
在本申请的实施例中，由于在获取原始视频时可以获取原始视频对应的位姿信息；根据位姿信息与原始视频可以得到虚拟平面；在原始视频的图像帧中添加虚拟内容时，虚拟平面可以作为一个基准面，根据虚拟平面可以调整虚拟内容在原始视频中的位置；因此，在本申请的实施例中，通过虚拟平面使得虚拟内容能够更好的融入到原始视频中，从而提高生成的AR视频的视频质量。In the embodiment of the present application, since the pose information corresponding to the original video can be obtained when the original video is obtained; the virtual plane can be obtained according to the pose information and the original video; when virtual content is added to the image frame of the original video, the virtual plane It can be used as a reference plane, and the position of the virtual content in the original video can be adjusted according to the virtual plane; therefore, in the embodiment of the present application, the virtual plane allows the virtual content to be better integrated into the original video, thereby improving the generated Video quality of AR videos.
附图说明Description of drawings
图1是一种适用于本申请的装置的硬件系统的示意图；Figure 1 is a schematic diagram of a hardware system suitable for the device of the present application;
图2是一种适用于本申请的装置的软件系统的示意图；Figure 2 is a schematic diagram of a software system suitable for the device of the present application;
图3是本申请提供的一种应用场景的示意图；Figure 3 is a schematic diagram of an application scenario provided by this application;
图4是本申请提供的一种增强现实视频的处理方法的示意图；Figure 4 is a schematic diagram of an augmented reality video processing method provided by this application;
图5是本申请提供的一种AR视频处理的显示界面的示意图；Figure 5 is a schematic diagram of a display interface for AR video processing provided by this application;
图6是本申请提供的一种AR视频处理的显示界面的示意图；Figure 6 is a schematic diagram of a display interface for AR video processing provided by this application;
图7是本申请提供的一种AR视频处理的显示界面的示意图；Figure 7 is a schematic diagram of a display interface for AR video processing provided by this application;
图8是本申请提供的一种AR视频处理的显示界面的示意图；Figure 8 is a schematic diagram of a display interface for AR video processing provided by this application;
图9是本申请提供的一种AR视频处理的显示界面的示意图；Figure 9 is a schematic diagram of a display interface for AR video processing provided by this application;
图10是本申请提供的一种AR视频处理的显示界面的示意图；Figure 10 is a schematic diagram of a display interface for AR video processing provided by this application;
图11是本申请提供的一种AR视频处理的显示界面的示意图；Figure 11 is a schematic diagram of a display interface for AR video processing provided by this application;
图12是本申请提供的一种AR视频处理的显示界面的示意图；Figure 12 is a schematic diagram of a display interface for AR video processing provided by this application;
图13是本申请提供的一种增强现实视频的处理方法的示意图；Figure 13 is a schematic diagram of an augmented reality video processing method provided by this application;
图14是本申请提供的一种AR视频处理的显示界面的示意图；Figure 14 is a schematic diagram of a display interface for AR video processing provided by this application;
图15是本申请提供的一种AR视频处理的显示界面的示意图；Figure 15 is a schematic diagram of a display interface for AR video processing provided by this application;
图16是本申请提供的一种AR视频处理的显示界面的示意图；Figure 16 is a schematic diagram of a display interface for AR video processing provided by this application;
图17是本申请提供的一种AR视频处理的显示界面的示意图；Figure 17 is a schematic diagram of a display interface for AR video processing provided by this application;
图18是本申请提供的一种AR视频处理的显示界面的示意图；Figure 18 is a schematic diagram of a display interface for AR video processing provided by this application;
图19是本申请提供的一种AR视频处理的显示界面的示意图；Figure 19 is a schematic diagram of a display interface for AR video processing provided by this application;
图20是本申请提供的一种AR视频处理的显示界面的示意图；Figure 20 is a schematic diagram of a display interface for AR video processing provided by this application;
图21是本申请提供的一种AR视频处理的显示界面的示意图；Figure 21 is a schematic diagram of a display interface for AR video processing provided by this application;
图22是本申请提供的一种增强现实视频的处理装置的结构示意图；Figure 22 is a schematic structural diagram of an augmented reality video processing device provided by this application;
图23是本申请提供的一种电子设备的结构示意图。Figure 23 is a schematic structural diagram of an electronic device provided by this application.
具体实施方式Detailed ways
下面将结合附图，对本申请实施例中的技术方案进行描述。The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.
图1示出了一种适用于本申请的终端设备的硬件系统。Figure 1 shows a hardware system suitable for the terminal device of the present application.
终端设备100可以是手机、智慧屏、平板电脑、可穿戴电子设备、车载电子设备、增强现实(augmented reality，AR)设备、虚拟现实(virtual reality，VR)设备、笔记本电脑、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本、个人数字助理(personal digital assistant，PDA)、投影仪等等，本申请实施例对终端设备100的具体类型不作任何限制。The terminal device 100 may be a mobile phone, a smart screen, a tablet, a wearable electronic device, a vehicle-mounted electronic device, an augmented reality (AR) device, a virtual reality (VR) device, a notebook computer, or a super mobile personal computer ( Ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. The embodiment of the present application does not place any restrictions on the specific type of the terminal device 100.
终端设备100可以包括处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括压力传感器180A，陀螺仪传感器180B，气压传感器180C，磁传感器180D，加速度传感器180E，距离传感器180F，接近光传感器180G，指纹传感器180H，温度传感器180J，触摸传感器180K，环境光传感器180L，骨传导传感器180M等。The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, and an antenna 2. , mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone interface 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193, display screen 194, and Subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyro sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, and ambient light. Sensor 180L, bone conduction sensor 180M, etc.
需要说明的是，图1所示的结构并不构成对终端设备100的具体限定。在本申请另一些实施例中，终端设备100可以包括比图1所示的部件更多或更少的部件，或者，终端设备100可以包括图1所示的部件中某些部件的组合，或者，终端设备100可以包括图1所示的部件中某些部件的子部件。图1示的部件可以以硬件、软件、或软件和硬件的组合实现。It should be noted that the structure shown in FIG. 1 does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or less components than those shown in FIG. 1 , or the terminal device 100 may include a combination of some of the components shown in FIG. 1 , or , the terminal device 100 may include sub-components of some of the components shown in FIG. 1 . The components shown in Figure 1 may be implemented in hardware, software, or a combination of software and hardware.
处理器110可以包括一个或多个处理单元。例如，处理器110可以包括以下处理单元中的至少一个：应用处理器(application processor，AP)、调制解调处理器、图形处理器(graphics processing unit，GPU)、图像信号处理器(image signal processor，ISP)、控制器、视频编解码器、数字信号处理器(digital signal processor，DSP)、基带处理器、神经网络处理器(neural-network processing unit，NPU)。其中，不同的处理单元可以是独立的器件，也可以是集成的器件。Processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (image signal processor) , ISP), controller, video codec, digital signal processor (DSP), baseband processor, neural-network processing unit (NPU). Among them, different processing units can be independent devices or integrated devices.
控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。The controller can generate operation control signals based on the instruction operation code and timing signals to complete the control of fetching and executing instructions.
处理器110中还可以设置存储器，用于存储指令和数据。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in processor 110 is cache memory. This memory may hold instructions or data that have been recently used or recycled by processor 110 . If the processor 110 needs to use the instructions or data again, it can be called directly from the memory. Repeated access is avoided and the waiting time of the processor 110 is reduced, thus improving the efficiency of the system.
在一些实施例中，处理器110可以包括一个或多个接口。例如，处理器110可以包括以下接口中的至少一个：内部集成电路(inter-integrated circuit，I2C)接口、内部集成电路音频(inter-integrated circuit sound，I2S)接口、脉冲编码调制(pulse codemodulation，PCM)接口、通用异步接收传输器(universal asynchronous receiver/transmitter，UART)接口、移动产业处理器接口(mobile industry processor interface，MIPI)、通用输入输出(general-purpose input/output，GPIO)接口、SIM接口、USB接口。In some embodiments, processor 110 may include one or more interfaces. For example, the processor 110 may include at least one of the following interfaces: an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, and a pulse code modulation (PCM) interface. ) interface, universal asynchronous receiver/transmitter (UART) interface, mobile industry processor interface (MIPI), general-purpose input/output (GPIO) interface, SIM interface , USB interface.
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。处理器110可以通过不同的I2C总线接口分别耦合触摸传感器180K、充电器、闪光灯、摄像头193等。例如：处理器110可以通过I2C接口耦合触摸传感器180K，使处理器110与触摸传感器180K通过I2C总线接口通信，实现终端设备100的触摸功能。The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can separately couple the touch sensor 180K, charger, flash, camera 193, etc. through different I2C bus interfaces. For example, the processor 110 can be coupled to the touch sensor 180K through an I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to implement the touch function of the terminal device 100 .
I2S接口可以用于音频通信。在一些实施例中，处理器110可以包含多组I2S总线。处理器110可以通过I2S总线与音频模块170耦合，实现处理器110与音频模块170之间的通信。在一些实施例中，音频模块170可以通过I2S接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 can be coupled with the audio module 170 through the I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface to implement the function of answering calls through a Bluetooth headset.
PCM接口也可以用于音频通信，将模拟信号抽样，量化和编码。在一些实施例中，音频模块170与无线通信模块160可以通过PCM接口耦合。在一些实施例中，音频模块170也可以通过PCM接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。所述I2S接口和所述PCM接口都可以用于音频通信。The PCM interface can also be used for audio communications to sample, quantize and encode analog signals. In some embodiments, audio module 170 and wireless communication module 160 may be coupled through a PCM interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface to implement the function of answering calls through a Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与无线通信模块160。例如：处理器110通过UART接口与无线通信模块160中的蓝牙模块通信，实现蓝牙功能。在一些实施例中，音频模块170可以通过UART接口向无线通信模块160传递音频信号，实现通过蓝牙耳机播放音乐的功能。The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface to implement the function of playing music through the Bluetooth headset.
MIPI接口可以被用于连接处理器110与显示屏194和摄像头193等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)、显示屏串行接口(displayserial interface，DSI)等。在一些实施例中，处理器110和摄像头193通过CSI接口通信，实现终端设备100的拍摄功能。处理器110和显示屏194通过DSI接口通信，实现终端设备100的显示功能。The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . MIPI interfaces include camera serial interface (CSI), display serial interface (displayserial interface, DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate through the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate through the DSI interface to implement the display function of the terminal device 100.
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号接口，也可被配置为数据信号接口。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头193，显示屏194、无线通信模块160、音频模块170和传感器模块180。GPIO接口还可以被配置为I2C接口、I2S接口、UART接口或MIPI接口。The GPIO interface can be configured through software. The GPIO interface can be configured as a control signal interface or as a data signal interface. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display screen 194, the wireless communication module 160, the audio module 170 and the sensor module 180. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface or MIPI interface.
USB接口130是符合USB标准规范的接口，例如可以是迷你(Mini)USB接口、微型(Micro)USB接口或C型USB(USB Type C)接口。USB接口130可以用于连接充电器为终端设备100充电，也可以用于终端设备100与外围设备之间传输数据，还可以用于连接耳机以通过耳机播放音频。USB接口130还可以用于连接其他终端设备100，例如AR设备。The USB interface 130 is an interface that complies with the USB standard specification, and may be, for example, a Mini USB interface, a Micro USB interface or a USB Type C interface. The USB interface 130 can be used to connect a charger to charge the terminal device 100, to transmit data between the terminal device 100 and peripheral devices, and to connect a headset to play audio through the headset. The USB interface 130 can also be used to connect other terminal devices 100, such as AR devices.
图1所示的各模块间的连接关系只是示意性说明，并不构成对终端设备100的各模块间的连接关系的限定。可选地，终端设备100的各模块也可以采用上述实施例中多种连接方式的组合。The connection relationship between the modules shown in FIG. 1 is only a schematic illustration and does not constitute a limitation on the connection relationship between the modules of the terminal device 100. Optionally, each module of the terminal device 100 may also adopt a combination of various connection methods in the above embodiments.
充电管理模块140用于从充电器接收电力。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块140可以通过USB接口130接收有线充电器的电流。在一些无线充电的实施例中，充电管理模块140可以通过终端设备100的无线充电线圈接收电磁波(电流路径如虚线所示)。充电管理模块140为电池142充电的同时，还可以通过电源管理模块141为终端设备100供电。The charging management module 140 is used to receive power from the charger. Among them, the charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive current from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive electromagnetic waves through the wireless charging coil of the terminal device 100 (the current path is shown as a dotted line). While the charging management module 140 charges the battery 142, it can also provide power to the terminal device 100 through the power management module 141.
电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，显示屏194，摄像头193，和无线通信模块160等供电。电源管理模块141还可以用于监测电池容量、电池循环次数和电池健康状态(例如，漏电、阻抗)等参数。可选地，电源管理模块141可以设置于处理器110中，或者，电源管理模块141和充电管理模块140可以设置于同一个器件中。The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and/or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, the wireless communication module 160, and the like. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (eg, leakage, impedance). Alternatively, the power management module 141 may be provided in the processor 110, or the power management module 141 and the charging management module 140 may be provided in the same device.
终端设备100的无线通信功能可以通过天线1、天线2、移动通信模块150、无线通信模块160、调制解调处理器以及基带处理器等器件实现。The wireless communication function of the terminal device 100 can be implemented through components such as antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.
天线1和天线2用于发射和接收电磁波信号。终端设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 may be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example: Antenna 1 can be reused as a diversity antenna for a wireless LAN. In other embodiments, antennas may be used in conjunction with tuning switches.
移动通信模块150可以提供应用在终端设备100上的无线通信的解决方案，例如下列方案中的至少一个：第二代(2th generation，2G)移动通信解决方案、第三代(3thgeneration，3G)移动通信解决方案、第四代(4th generation，5G)移动通信解决方案、第五代(5th generation，5G)移动通信解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波和放大等处理，随后传送至调制解调处理器进行解调。移动通信模块150还可以放大经调制解调处理器调制后的信号，放大后的该信号经天线1转变为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。The mobile communication module 150 may provide a wireless communication solution applied to the terminal device 100, such as at least one of the following solutions: a second generation (2th generation, 2G) mobile communication solution, a third generation (3th generation, 3G) mobile communication solutions, fourth generation (4th generation, 5G) mobile communication solutions, fifth generation (5th generation, 5G) mobile communication solutions. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and then transmit them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor, and the amplified signal is converted into electromagnetic waves by the antenna 1 and radiated out. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be disposed in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be provided in the same device.
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(例如，扬声器170A、受话器170B)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。A modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (eg, speaker 170A, receiver 170B), or displays an image or video through the display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In other embodiments, the modem processor may be independent of the processor 110 and may be provided in the same device as the mobile communication module 150 or other functional modules.
与移动通信模块150类似，无线通信模块160也可以提供应用在终端设备100上的无线通信解决方案，例如下列方案中的至少一个：无线局域网(wireless local areanetworks，WLAN)、蓝牙(bluetooth，BT)、蓝牙低功耗(bluetooth low energy，BLE)、超宽带(ultra wide band，UWB)、全球导航卫星系统(global navigation satellite system，GNSS)、调频(frequency modulation，FM)、近场通信(near field communication，NFC)、红外(infrared，IR)技术。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，并将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频和放大，该信号经天线2转变为电磁波辐射出去。Similar to the mobile communication module 150, the wireless communication module 160 can also provide a wireless communication solution applied to the terminal device 100, such as at least one of the following solutions: wireless local area networks (WLAN), Bluetooth (bluetooth, BT) , Bluetooth low energy (BLE), ultra wide band (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (near field communication (NFC), infrared (infrared, IR) technology. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110, perform frequency modulation and amplification on it, and convert the signal into electromagnetic waves through the antenna 2 for radiation.
在一些实施例中，终端设备100的天线1和移动通信模块150耦合，终端设备100的天线2和无线通信模块160耦合，使得终端设备100可以通过无线通信技术与网络和其他电子设备通信。该无线通信技术可以包括以下通信技术中的至少一个：全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packetradio service，GPRS)，码分多址接入(code division multiple access，CDMA)，宽带码分多址(wideband code division multiple access，WCDMA)，时分码分多址(time-divisioncode division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，IR技术。该GNSS可以包括以下定位技术中的至少一个：全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigationsatellite system，GLONASS)，北斗卫星导航系统(beidou navigation satellitesystem，BDS)，准天顶卫星系统(quasi-zenith satellite system，QZSS)，星基增强系统(satellite based augmentation systems，SBAS)。In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 of the terminal device 100 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other electronic devices through wireless communication technology. The wireless communication technology may include at least one of the following communication technologies: global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (code division multiple access) access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, IR technology. The GNSS may include at least one of the following positioning technologies: global positioning system (GPS), global navigation satellite system (GLONASS), Beidou navigation satellite system (BDS), quasi-sky Quasi-zenith satellite system (QZSS), satellite based augmentation systems (SBAS).
终端设备100可以通过GPU、显示屏194以及应用处理器实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。The terminal device 100 can implement display functions through a GPU, a display screen 194, and an application processor. The GPU is an image processing microprocessor and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or alter display information.
显示屏194可以用于显示图像或视频。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)、有机发光二极管(organic light-emitting diode，OLED)、有源矩阵有机发光二极体(active-matrix organic light-emitting diode，AMOLED)、柔性发光二极管(flex light-emitting diode，FLED)、迷你发光二极管(mini light-emitting diode，Mini LED)、微型发光二极管(micro light-emitting diode，Micro LED)、微型OLED(Micro OLED)或量子点发光二极管(quantum dotlight emitting diodes，QLED)。在一些实施例中，终端设备100可以包括1个或N个显示屏194，N为大于1的正整数。Display 194 may be used to display images or videos. Display 194 includes a display panel. The display panel can use liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode (AMOLED), flexible Light-emitting diode (flex light-emitting diode, FLED), mini light-emitting diode (mini light-emitting diode, Mini LED), micro light-emitting diode (micro light-emitting diode, Micro LED), micro OLED (Micro OLED) or quantum dot light emission Diodes (quantum dotlight emitting diodes, QLED). In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
终端设备100可以通过ISP、摄像头193、视频编解码器、GPU、显示屏194以及应用处理器等实现拍摄功能。The terminal device 100 can implement the shooting function through the ISP, camera 193, video codec, GPU, display screen 194, application processor, etc.
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP可以对图像的噪点、亮度和色彩进行算法优化，ISP还可以优化拍摄场景的曝光和色温等参数。在一些实施例中，ISP可以设置在摄像头193中。The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera sensor through the lens, the optical signal is converted into an electrical signal, and the camera sensor passes the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can algorithmically optimize the noise, brightness and color of the image. ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP may be provided in the camera 193.
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的红绿蓝(red green blue，RGB)，YUV等格式的图像信号。在一些实施例中，终端设备100可以包括1个或N个摄像头193，N为大于1的正整数。Camera 193 is used to capture still images or video. The object passes through the lens to produce an optical image that is projected onto the photosensitive element. The photosensitive element may be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to convert it into a digital image signal. ISP outputs digital image signals to DSP for processing. DSP converts digital image signals into standard red green blue (RGB), YUV and other format image signals. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当终端设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the terminal device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.
视频编解码器用于对数字视频压缩或解压缩。终端设备100可以支持一种或多种视频编解码器。这样，终端设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1、MPEG2、MPEG3和MPEG4。Video codecs are used to compress or decompress digital video. The terminal device 100 may support one or more video codecs. In this way, the terminal device 100 can play or record videos in multiple encoding formats, such as: moving picture experts group (MPEG) 1, MPEG2, MPEG3, and MPEG4.
NPU是一种借鉴生物神经网络结构的处理器，例如借鉴人脑神经元之间传递模式对输入信息快速处理，还可以不断地自学习。通过NPU可以实现终端设备100的智能认知等功能，例如：图像识别、人脸识别、语音识别和文本理解。NPU is a processor that draws on the structure of biological neural networks. For example, it draws on the transmission pattern between neurons in the human brain to quickly process input information, and it can also continuously learn by itself. The NPU can realize intelligent cognitive functions of the terminal device 100, such as image recognition, face recognition, speech recognition and text understanding.
外部存储器接口120可以用于连接外部存储卡，例如安全数码(secure digital，SD)卡，实现扩展终端设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。The external memory interface 120 can be used to connect an external memory card, such as a secure digital (SD) card, to expand the storage capacity of the terminal device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to implement the data storage function. Such as saving music, videos, etc. files in external memory card.
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能(例如，声音播放功能和图像播放功能)所需的应用程序。存储数据区可存储终端设备100使用过程中所创建的数据(例如，音频数据和电话本)。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如：至少一个磁盘存储器件、闪存器件和通用闪存存储器(universal flash storage，UFS)等。处理器110通过运行存储在内部存储器121的指令和/或存储在设置于处理器中的存储器的指令，执行终端设备100的各种处理方法。Internal memory 121 may be used to store computer executable program code, which includes instructions. The internal memory 121 may include a program storage area and a data storage area. The stored program area may store an operating system and an application program required for at least one function (for example, a sound playback function and an image playback function). The storage data area may store data created during use of the terminal device 100 (eg, audio data and phone book). In addition, the internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, a flash memory device, and universal flash storage (UFS). The processor 110 executes various processing methods of the terminal device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.
终端设备100可以通过音频模块170、扬声器170A、受话器170B、麦克风170C、耳机接口170D以及应用处理器等实现音频功能，例如，音乐播放和录音。The terminal device 100 can implement audio functions, such as music playback and recording, through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor.
音频模块170用于将数字音频信息转换成模拟音频信号输出，也可以用于将模拟音频输入转换为数字音频信号。音频模块170还可以用于对音频信号编码和解码。在一些实施例中，音频模块170或者音频模块170的部分功能模块可以设置于处理器110中。The audio module 170 is used to convert digital audio information into analog audio signal output, and can also be used to convert analog audio input into digital audio signals. Audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 or some functional modules of the audio module 170 may be provided in the processor 110 .
扬声器170A，也称为喇叭，用于将音频电信号转换为声音信号。终端设备100可以通过扬声器170A收听音乐或免提通话。Speaker 170A, also called a horn, is used to convert audio electrical signals into sound signals. The terminal device 100 can listen to music or make hands-free calls through the speaker 170A.
受话器170B，也称为听筒，用于将音频电信号转换成声音信号。当用户使用终端设备100接听电话或语音信息时，可以通过将受话器170B靠近耳朵接听语音。Receiver 170B, also called an earpiece, is used to convert audio electrical signals into sound signals. When the user uses the terminal device 100 to answer a call or voice message, the user can listen to the voice by placing the receiver 170B close to the ear.
麦克风170C，也称为话筒或传声器，用于将声音信号转换为电信号。当用户拨打电话或发送语音信息时，可以通过靠近麦克风170C发声将声音信号输入麦克风170C。终端设备100可以设置至少一个麦克风170C。在另一些实施例中，终端设备100可以设置两个麦克风170C，以实现降噪功能。在另一些实施例中，终端设备100还可以设置三个、四个或更多麦克风170C，以实现识别声音来源和定向录音等功能。处理器110可以对麦克风170C输出的电信号进行处理，例如，音频模块170与无线通信模块160可以通过PCM接口耦合，麦克风170C将环境声音转换为电信号(如PCM信号)后，通过PCM接口将该电信号传输至处理器110；从处理器110对该电信号进行音量分析和频率分析，确定环境声音的音量和频率。Microphone 170C, also known as a microphone or microphone, is used to convert sound signals into electrical signals. When the user makes a call or sends a voice message, the sound signal can be input into the microphone 170C by speaking close to the microphone 170C. The terminal device 100 may be provided with at least one microphone 170C. In other embodiments, the terminal device 100 may be provided with two microphones 170C to implement the noise reduction function. In other embodiments, the terminal device 100 may also be provided with three, four or more microphones 170C to implement functions such as identifying sound sources and directional recording. The processor 110 can process the electrical signal output by the microphone 170C. For example, the audio module 170 and the wireless communication module 160 can be coupled through a PCM interface. After the microphone 170C converts the environmental sound into an electrical signal (such as a PCM signal), the The electrical signal is transmitted to the processor 110; the processor 110 performs volume analysis and frequency analysis on the electrical signal to determine the volume and frequency of the environmental sound.
耳机接口170D用于连接有线耳机。耳机接口170D可以是USB接口130，也可以是3.5mm的开放移动终端设备100平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of theUSA，CTIA)标准接口。The headphone interface 170D is used to connect wired headphones. The headphone interface 170D may be a USB interface 130, or may be a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.
压力传感器180A用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180A可以设置于显示屏194。压力传感器180A的种类很多，例如可以是电阻式压力传感器、电感式压力传感器或电容式压力传感器。电容式压力传感器可以是包括至少两个具有导电材料的平行板，当力作用于压力传感器180A，电极之间的电容改变，终端设备100根据电容的变化确定压力的强度。当触摸操作作用于显示屏194时，终端设备100根据压力传感器180A检测所述触摸操作。终端设备100也可以根据压力传感器180A的检测信号计算触摸的位置。在一些实施例中，作用于相同触摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。例如：当触摸操作强度小于第一压力阈值的触摸操作作用于短消息应用图标时，执行查看短消息的指令；当触摸操作强度大于或等于第一压力阈值的触摸操作作用于短消息应用图标时，执行新建短消息的指令。The pressure sensor 180A is used to sense pressure signals and can convert the pressure signals into electrical signals. In some embodiments, pressure sensor 180A may be disposed on display screen 194 . There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors or capacitive pressure sensors. The capacitive pressure sensor may include at least two parallel plates with conductive material. When force acts on the pressure sensor 180A, the capacitance between the electrodes changes, and the terminal device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 194, the terminal device 100 detects the touch operation according to the pressure sensor 180A. The terminal device 100 may also calculate the touched position based on the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch location but with different touch operation intensities may correspond to different operation instructions. For example: when the touch operation intensity is less than the first pressure threshold and acts on the short message application icon, the instruction to view the short message is executed; when the touch operation intensity is greater than or equal to the first pressure threshold and the touch operation acts on the short message application icon. , execute the command to create a new short message.
陀螺仪传感器180B可以用于确定终端设备100的运动姿态。在一些实施例中，可以通过陀螺仪传感器180B确定终端设备100围绕三个轴(即，x轴、y轴和z轴)的角速度。陀螺仪传感器180B可以用于拍摄防抖。例如，当快门被按下时，陀螺仪传感器180B检测终端设备100抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消终端设备100的抖动，实现防抖。陀螺仪传感器180B还可以用于导航和体感游戏等场景。The gyro sensor 180B may be used to determine the motion posture of the terminal device 100 . In some embodiments, the angular velocity of the terminal device 100 around three axes (ie, x-axis, y-axis, and z-axis) may be determined by the gyro sensor 180B. The gyro sensor 180B can be used for image stabilization. For example, when the shutter is pressed, the gyro sensor 180B detects the angle at which the terminal device 100 shakes, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to counteract the shake of the terminal device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used in scenarios such as navigation and somatosensory games.
气压传感器180C用于测量气压。在一些实施例中，终端设备100通过气压传感器180C测得的气压值计算海拔高度，辅助定位和导航。Air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.
磁传感器180D包括霍尔传感器。终端设备100可以利用磁传感器180D检测翻盖皮套的开合。在一些实施例中，当终端设备100是翻盖机时，终端设备100可以根据磁传感器180D检测翻盖的开合。终端设备100可以根据检测到的皮套的开合状态或翻盖的开合状态，设置翻盖自动解锁等特性。Magnetic sensor 180D includes a Hall sensor. The terminal device 100 can detect the opening and closing of the flip leather case using the magnetic sensor 180D. In some embodiments, when the terminal device 100 is a flip machine, the terminal device 100 may detect the opening and closing of the flip according to the magnetic sensor 180D. The terminal device 100 can set features such as automatic unlocking of the flip cover based on the detected opening and closing status of the leather case or the opening and closing status of the flip cover.
加速度传感器180E可检测终端设备100在各个方向上(一般为x轴、y轴和z轴)加速度的大小。当终端设备100静止时可检测出重力的大小及方向。加速度传感器180E还可以用于识别终端设备100的姿态，作为横竖屏切换和计步器等应用程序的输入参数。The acceleration sensor 180E can detect the acceleration of the terminal device 100 in various directions (generally the x-axis, y-axis, and z-axis). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E can also be used to identify the posture of the terminal device 100 as an input parameter for applications such as horizontal and vertical screen switching and pedometer.
距离传感器180F用于测量距离。终端设备100可以通过红外或激光测量距离。在一些实施例中，例如在拍摄场景中，终端设备100可以利用距离传感器180F测距以实现快速对焦。Distance sensor 180F is used to measure distance. The terminal device 100 can measure distance through infrared or laser. In some embodiments, for example, in a shooting scene, the terminal device 100 may utilize the distance sensor 180F to measure distance to achieve fast focusing.
接近光传感器180G可以包括例如发光二极管(light-emitting diode，LED)和光检测器，例如，光电二极管。LED可以是红外LED。终端设备100通过LED向外发射红外光。终端设备100使用光电二极管检测来自附近物体的红外反射光。当检测到反射光时，终端设备100可以确定附近存在物体。当检测不到反射光时，终端设备100可以确定附近没有物体。终端设备100可以利用接近光传感器180G检测用户是否手持终端设备100贴近耳朵通话，以便自动熄灭屏幕达到省电的目的。接近光传感器180G也可用于皮套模式或口袋模式的自动解锁与自动锁屏。The proximity light sensor 180G may include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The LED may be an infrared LED. The terminal device 100 emits infrared light outwardly through the LED. The terminal device 100 detects infrared reflected light from nearby objects using photodiodes. When the reflected light is detected, the terminal device 100 can determine that an object exists nearby. When no reflected light is detected, the terminal device 100 can determine that there is no object nearby. The terminal device 100 can use the proximity light sensor 180G to detect whether the user is holding the terminal device 100 close to the ear for talking, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used for automatic unlocking and automatic screen locking in holster mode or pocket mode.
环境光传感器180L用于感知环境光亮度。终端设备100可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180L也可用于拍照时自动调节白平衡。环境光传感器180L还可以与接近光传感器180G配合，检测终端设备100是否在口袋里，以防误触。The ambient light sensor 180L is used to sense ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in the pocket to prevent accidental touching.
指纹传感器180H用于采集指纹。终端设备100可以利用采集的指纹特性实现解锁、访问应用锁、拍照和接听来电等功能。Fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking photos, and answering incoming calls.
温度传感器180J用于检测温度。在一些实施例中，终端设备100利用温度传感器180J检测的温度，执行温度处理策略。例如，当温度传感器180J上报的温度超过阈值，终端设备100执行降低位于温度传感器180J附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，终端设备100对电池142加热，以避免低温导致终端设备100异常关机。在其他一些实施例中，当温度低于又一阈值时，终端设备100对电池142的输出电压执行升压，以避免低温导致的异常关机。Temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute the temperature processing policy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of a processor located near the temperature sensor 180J in order to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid the low temperature causing the terminal device 100 to shut down abnormally. In some other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs boosting on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
触摸传感器180K，也称为触控器件。触摸传感器180K可以设置于显示屏194，由触摸传感器180K与显示屏194组成触摸屏，触摸屏也称为触控屏。触摸传感器180K用于检测作用于其上或其附近的触摸操作。触摸传感器180K可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180K也可以设置于终端设备100的表面，并且与显示屏194设置于不同的位置。Touch sensor 180K, also known as touch device. The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen. The touch screen is also called a touch screen. The touch sensor 180K is used to detect a touch operation acted on or near the touch sensor 180K. The touch sensor 180K may pass the detected touch operation to the application processor to determine the touch event type. Visual output related to the touch operation may be provided through display screen 194 . In other embodiments, the touch sensor 180K may also be disposed on the surface of the terminal device 100 and at a different position from the display screen 194 .
骨传导传感器180M可以获取振动信号。在一些实施例中，骨传导传感器180M可以获取人体声部振动骨块的振动信号。骨传导传感器180M也可以接触人体脉搏，接收血压跳动信号。在一些实施例中，骨传导传感器180M也可以设置于耳机中，结合成骨传导耳机。音频模块170可以基于所述骨传导传感器180M获取的声部振动骨块的振动信号，解析出语音信号，实现语音功能。应用处理器可以基于所述骨传导传感器180M获取的血压跳动信号解析心率信息，实现心率检测功能。Bone conduction sensor 180M can acquire vibration signals. In some embodiments, the bone conduction sensor 180M can acquire the vibration signal of the vibrating bone mass of the human body's vocal part. The bone conduction sensor 180M can also contact the human body's pulse and receive blood pressure beating signals. In some embodiments, the bone conduction sensor 180M can also be provided in an earphone and combined into a bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vocal vibrating bone obtained by the bone conduction sensor 180M to implement the voice function. The application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180M to implement the heart rate detection function.
按键190包括开机键和音量键。按键190可以是机械按键，也可以是触摸式按键。终端设备100可以接收按键输入信号，实现于案件输入信号相关的功能。The buttons 190 include a power button and a volume button. The button 190 may be a mechanical button or a touch button. The terminal device 100 can receive key input signals and implement functions related to the case input signals.
马达191可以产生振动。马达191可以用于来电提示，也可以用于触摸反馈。马达191可以对作用于不同应用程序的触摸操作产生不同的振动反馈效果。对于作用于显示屏194的不同区域的触摸操作，马达191也可产生不同的振动反馈效果。不同的应用场景(例如，时间提醒、接收信息、闹钟和游戏)可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。The motor 191 can generate vibration. The motor 191 can be used for incoming call prompts and touch feedback. The motor 191 can produce different vibration feedback effects for touch operations acting on different applications. For touch operations on different areas of the display screen 194, the motor 191 can also produce different vibration feedback effects. Different application scenarios (such as time reminders, receiving information, alarm clocks, and games) can correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.
指示器192可以是指示灯，可以用于指示充电状态和电量变化，也可以用于指示消息、未接来电和通知。The indicator 192 may be an indicator light, which may be used to indicate charging status and power changes, or may be used to indicate messages, missed calls, and notifications.
SIM卡接口195用于连接SIM卡。SIM卡可以插入SIM卡接口195实现与终端设备100的接触，也可以从SIM卡接口195拔出实现与终端设备100的分离。终端设备100可以支持1个或N个SIM卡接口，N为大于1的正整数。同一个SIM卡接口195可以同时插入多张卡，所述多张卡的类型可以相同，也可以不同。SIM卡接口195也可以兼容外部存储卡。终端设备100通过SIM卡和网络交互，实现通话以及数据通信等功能。在一些实施例中，终端设备100采用嵌入式SIM(embedded-SIM，eSIM)卡，eSIM卡可以嵌在终端设备100中，不能和终端设备100分离。The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into the SIM card interface 195 to achieve contact with the terminal device 100 , or can be pulled out from the SIM card interface 195 to achieve separation from the terminal device 100 . The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. Multiple cards can be inserted into the same SIM card interface 195 at the same time, and the types of the multiple cards can be the same or different. The SIM card interface 195 is also compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an embedded SIM (embedded-SIM, eSIM) card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100 .
上文详细描述了终端设备100的硬件系统，下面介绍终端设备100的软件系统。软件系统可以采用分层架构、事件驱动架构、微核架构、微服务架构或云架构，本申请实施例以分层架构为例，示例性地描述终端设备100的软件系统。The hardware system of the terminal device 100 is described in detail above, and the software system of the terminal device 100 is introduced below. The software system may adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture or a cloud architecture. The embodiment of this application takes the layered architecture as an example to illustratively describe the software system of the terminal device 100.
如图2所示，采用分层架构的软件系统分成若干个层，每一层都有清晰的角色和分工。层与层之间通过软件接口通信。在一些实施例中，软件系统可以分为四层，从上至下分别为应用程序层、应用程序框架层、安卓运行时(Android Runtime)和系统库、以及内核层。As shown in Figure 2, a software system using a layered architecture is divided into several layers, and each layer has clear roles and division of labor. The layers communicate through software interfaces. In some embodiments, the software system can be divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.
应用程序层可以包括相机、图库、日历、通话、地图、导航、WLAN、蓝牙、音乐、视频、短信息等应用程序。The application layer can include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.
应用程序框架层为应用程序层的应用程序提供应用程序编程接口(ApplicationProgramming Interface，API)和编程框架。应用程序框架层可以包括一些预定义的函数。The application framework layer provides application programming interfaces (Application Programming Interface, API) and programming frameworks for applications in the application layer. The application framework layer can include some predefined functions.
例如，应用程序框架层包括窗口管理器、内容提供器、视图系统、资源管理器和通知管理器、同步定位与建图(Simultaneous Localization And Mapping，SLAM)位姿计算模块以及平面生成模块；应用程序框架层还可以包括电话管理器。For example, the application framework layer includes the window manager, content provider, view system, resource manager and notification manager, Simultaneous Localization And Mapping (SLAM) pose calculation module and plane generation module; application program The framework layer may also include a telephony manager.
SLAM位姿计算模块用于输出位姿信息与稀疏点云；其中，位姿信息是指终端设备的摄像头的位姿信息，终端设备的摄像头用于获取真实场景的视频；根据视频中任意一帧图像的位姿信息可以对该帧图像进行特征点提取，并通过计算得到稀疏点云。The SLAM pose calculation module is used to output pose information and sparse point clouds; among them, pose information refers to the pose information of the camera of the terminal device, and the camera of the terminal device is used to obtain the video of the real scene; according to any frame in the video The pose information of the image can be used to extract feature points of the frame image and obtain a sparse point cloud through calculation.
平面生成模块用于根据SLAM提供的稀疏点云，通过算法拟合生成虚拟平面；在真实场景中添加虚拟内容时，可以根据虚拟平面对虚拟内容的放置位置进行调整；例如，用户点击屏幕/手势操作放置虚拟内容时，用户的操作与生成的虚拟平面可以产生碰撞，确定虚拟内容的放置位置。应理解，本申请实施例提供的增强现实视频的处理方法对应的程序指令可以在SLAM位姿计算模块与平面生成模块中执行。The plane generation module is used to generate a virtual plane through algorithm fitting based on the sparse point cloud provided by SLAM; when adding virtual content in a real scene, the placement of the virtual content can be adjusted according to the virtual plane; for example, the user clicks the screen/gesture When placing virtual content, the user's operation can collide with the generated virtual plane to determine the placement position of the virtual content. It should be understood that the program instructions corresponding to the augmented reality video processing method provided by the embodiments of the present application can be executed in the SLAM pose calculation module and the plane generation module.
窗口管理器用于管理窗口程序。窗口管理器可以获取显示屏大小，判断是否有状态栏、锁定屏幕和截取屏幕。A window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, and capture the screen.
内容提供器用来存放和获取数据，并使这些数据可以被应用程序访问。所述数据可以包括视频、图像、音频、拨打和接听的电话、浏览历史和书签、以及电话簿。Content providers are used to store and retrieve data and make this data accessible to applications. The data may include video, images, audio, calls made and received, browsing history and bookmarks, and phone books.
视图系统包括可视控件，例如显示文字的控件和显示图片的控件。视图系统可用于构建应用程序。显示界面可以由一个或多个视图组成，例如，包括短信通知图标的显示界面，可以包括显示文字的视图以及显示图片的视图。The view system includes visual controls, such as controls that display text and controls that display pictures. A view system can be used to build applications. The display interface may be composed of one or more views. For example, a display interface including a text message notification icon may include a view for displaying text and a view for displaying pictures.
电话管理器用于提供终端设备100的通信功能，例如通话状态(接通或挂断)的管理。The phone manager is used to provide communication functions of the terminal device 100, such as management of call status (connected or hung up).
资源管理器为应用程序提供各种资源，比如本地化字符串、图标、图片、布局文件和视频文件。The resource manager provides various resources to the application, such as localized strings, icons, pictures, layout files, and video files.
通知管理器使应用程序可以在状态栏中显示通知信息，可以用于传达告知类型的消息，可以短暂停留后自动消失，无需用户交互。比如通知管理器被用于下载完成告知和消息提醒。通知管理器还可以管理以图表或者滚动条文本形式出现在系统顶部状态栏的通知，例如后台运行的应用程序的通知。通知管理器还可以管理以对话窗口形式出现在屏幕上的通知，例如在状态栏提示文本信息、发出提示音、电子设备振动以及指示灯闪烁。The notification manager allows applications to display notification information in the status bar, which can be used to convey notification-type messages and can automatically disappear after a short stay without user interaction. For example, the notification manager is used for download completion notifications and message reminders. The notification manager can also manage notifications that appear in the status bar at the top of the system in the form of graphics or scrollbar text, such as notifications for applications running in the background. The notification manager can also manage notifications that appear on the screen in the form of conversation windows, such as text messages in the status bar, beeps, electronic device vibrations, and indicator light flashes.
Android Runtime包括核心库和虚拟机。Android runtime负责安卓系统的调度和管理。Android Runtime includes core libraries and virtual machines. The Android runtime is responsible for the scheduling and management of the Android system.
核心库包含两部分：一部分是java语言需要调用的功能函数，另一部分是安卓的核心库。The core library contains two parts: one is the functional functions that need to be called by the Java language, and the other is the core library of Android.
应用程序层和应用程序框架层运行在虚拟机中。虚拟机将应用程序层和应用程序框架层的java文件执行为二进制文件。虚拟机用于执行对象生命周期的管理、堆栈管理、线程管理、安全和异常的管理、以及垃圾回收等功能。The application layer and application framework layer run in virtual machines. The virtual machine executes the java files of the application layer and application framework layer into binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
系统库可以包括多个功能模块，例如：表面管理器(surface manager)，媒体库(Media Libraries)，三维图形处理库(例如：针对嵌入式系统的开放图形库(opengraphics library for embedded systems，OpenGL ES)和2D图形引擎(例如：skia图形库(skia graphics library，SGL))。The system library can include multiple functional modules, such as: surface manager (surface manager), media libraries (Media Libraries), three-dimensional graphics processing libraries (such as: open graphics library for embedded systems (OpenGL ES) ) and 2D graphics engines (for example: skia graphics library (skia graphics library, SGL)).
表面管理器用于对显示子系统进行管理，并且为多个应用程序提供了2D图层和3D图层的融合。The surface manager is used to manage the display subsystem and provides the fusion of 2D and 3D layers for multiple applications.
媒体库支持多种音频格式的回放和录制、多种视频格式回放和录制以及静态图像文件。媒体库可以支持多种音视频编码格式，例如:MPEG4、H.264、动态图像专家组音频层面3(moving picture experts group audio layer III，MP3)、高级音频编码(advancedaudio coding，AAC)、自适应多码率(adaptive multi-rate，AMR)、联合图像专家组(jointphotographic experts group，JPG)和便携式网络图形(portable network graphics，PNG)。The media library supports playback and recording of multiple audio formats, playback and recording of multiple video formats, and still image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, moving picture experts group audio layer III (MP3), advanced audio coding (AAC), adaptive Adaptive multi-rate (AMR), joint photographic experts group (JPG) and portable network graphics (PNG).
三维图形处理库可以用于实现三维图形绘图、图像渲染、合成和图层处理。The 3D graphics processing library can be used to implement 3D graphics drawing, image rendering, composition and layer processing.
二维图形引擎是2D绘图的绘图引擎。2D Graphics Engine is a drawing engine for 2D drawing.
内核层是硬件和软件之间的层。内核层可以包括显示驱动、摄像头驱动、音频驱动和传感器驱动等驱动模块。The kernel layer is the layer between hardware and software. The kernel layer can include driver modules such as display driver, camera driver, audio driver, and sensor driver.
下面结合显示拍照场景，示例性说明终端设备100的软件系统和硬件系统的工作流程。The following exemplifies the workflow of the software system and hardware system of the terminal device 100 by combining the display of the photographing scene.
当用户在触摸传感器180K上进行触摸操作时，相应的硬件中断被发送至内核层，内核层将触摸操作加工成原始输入事件，原始输入事件例如包括触摸坐标和触摸操作的时间戳等信息。原始输入事件被存储在内核层，应用程序框架层从内核层获取原始输入事件，识别出原始输入事件对应的控件，并通知该控件对应的应用程序(application，APP)。例如，上述触摸操作为单击操作，上述控件对应的APP为相机APP，相机APP被单击操作唤醒后，可以通过API调用内核层的摄像头驱动，通过摄像头驱动控制摄像头193进行拍摄。When the user performs a touch operation on the touch sensor 180K, the corresponding hardware interrupt is sent to the kernel layer, and the kernel layer processes the touch operation into a raw input event. The raw input event includes, for example, touch coordinates and a timestamp of the touch operation. The original input events are stored in the kernel layer. The application framework layer obtains the original input events from the kernel layer, identifies the control corresponding to the original input event, and notifies the application (application, APP) corresponding to the control. For example, the above touch operation is a click operation, and the APP corresponding to the above control is a camera APP. After the camera APP is awakened by a click operation, the camera driver at the kernel layer can be called through the API, and the camera 193 can be controlled through the camera driver to take pictures.
目前，在录制AR视频时由于无法将虚拟内容和真实物体的视频很好的融合，尤其是在摄场景中需要用户与虚拟内容进行交互时，需要多次的重复拍摄，费时费力。Currently, when recording AR videos, virtual content and real object videos cannot be well integrated. Especially when users need to interact with virtual content in a shooting scene, multiple repeated shots are required, which is time-consuming and labor-intensive.
有鉴于此，本申请提供了一种AR视频的处理方法，通过在获取原始视频时获取原始视频对应的位姿信息；根据位姿信息与原始视频可以得到虚拟平面；在原始视频的图像帧中添加虚拟内容时，虚拟平面可以作为一个基准面，根据虚拟平面可以调整虚拟内容在原始视频中的位置，使得虚拟内容能够更好的融入到原始视频中，提高AR视频的视频质量。In view of this, this application provides an AR video processing method by obtaining the pose information corresponding to the original video when acquiring the original video; a virtual plane can be obtained based on the pose information and the original video; in the image frames of the original video When adding virtual content, the virtual plane can be used as a reference plane. The position of the virtual content in the original video can be adjusted according to the virtual plane, so that the virtual content can be better integrated into the original video and improve the video quality of the AR video.
下面以终端设备100为例，结合图3至图21对本申请提供的增强现实视频的处理方法进行详细描述。Taking the terminal device 100 as an example, the augmented reality video processing method provided by this application will be described in detail below with reference to FIGS. 3 to 21 .
图3是本申请的应用场景的示意图；如图3所示，本申请实施例提供的AR视频的处理方法可以应用于AR视频领域；可以获取原始视频通过AR视频处理可以得到目标视频；其中，原始视频可以是指用户拍摄的真实物体的视频，目标视频可以是指在原始视频中添加虚拟内容后得到的AR视频。Figure 3 is a schematic diagram of the application scenario of the present application; as shown in Figure 3, the AR video processing method provided by the embodiment of the present application can be applied to the field of AR video; the original video can be obtained and the target video can be obtained through AR video processing; wherein, The original video may refer to a video of a real object taken by the user, and the target video may refer to an AR video obtained by adding virtual content to the original video.
示例性地，本申请实施例提供的AR视频的处理方法也可以在应用程序(Application，APP)中运行从而执行AR视频编辑；比如，AR视频APP可以执行本申请的AR视频的处理方法。或者，本申请实施例提供的AR视频的处理方法也可以集成在终端设备的相机中；比如，在终端设备的相机的设置中可以选择AR视频模式，从而实现本申请实施例提供的AR视频的处理方法；下面分别对这两种实现方式进行详细描述。For example, the AR video processing method provided by the embodiment of the present application can also be run in an application (Application, APP) to perform AR video editing; for example, the AR video APP can execute the AR video processing method of the present application. Alternatively, the AR video processing method provided by the embodiments of the present application can also be integrated into the camera of the terminal device; for example, the AR video mode can be selected in the camera settings of the terminal device, thereby realizing the AR video processing method provided by the embodiments of the present application. Processing methods; the two implementation methods are described in detail below.
实现方式一：通过应用程序实现本申请实施例的AR视频的处理方法。Implementation method one: implement the AR video processing method in the embodiment of this application through an application program.
如图4所示，图4是本申请实施例提供的AR视频的处理方法的示意性流程图；该处理方法200包括步骤S210至步骤S260，下面分别对这些步骤进行详细的描述。As shown in Figure 4, Figure 4 is a schematic flow chart of an AR video processing method provided by an embodiment of the present application; the processing method 200 includes steps S210 to S260, and these steps are described in detail below.
步骤S210、运行AR视频APP。Step S210: Run the AR video APP.
例如，用户可以点击终端设备的显示界面中的AR视频APP；响应于用户的点击操作，终端设备可以运行AR视频APP；如图5所示，图5示出了终端设备的一种图形用户界面(graphical user interface，GUI)，该GUI可以为终端设备的桌面310。当终端设备检测到用户点击桌面310上的AR视频APP的图标320的操作后，可以运行AR视频APP，显示如图6所示的另一GUI；图6所示的显示界面330上可以包括拍摄取景框340，拍摄取景框340内可以实时显示预览图像；拍摄界面上还可以包括用于指示拍摄的控件350，以及其它拍摄控件。For example, the user can click on the AR video APP in the display interface of the terminal device; in response to the user's click operation, the terminal device can run the AR video APP; as shown in Figure 5, Figure 5 shows a graphical user interface of the terminal device (graphical user interface, GUI), the GUI may be the desktop 310 of the terminal device. When the terminal device detects the user's operation of clicking the icon 320 of the AR video APP on the desktop 310, the AR video APP can be run to display another GUI as shown in Figure 6; the display interface 330 shown in Figure 6 can include shooting Viewfinder frame 340, the preview image can be displayed in real time in the shooting viewfinder frame 340; the shooting interface can also include a control 350 for instructing shooting, and other shooting controls.
在一个示例中，终端设备检测到用户点击显示界面上的AR视频APP的图标的操作，可以启动AR视频APP，显示AR视频APP的显示界面；在显示界面上可以包括拍摄取景框；例如，在录像模式下，拍摄取景框可以为部分屏幕，或者也可以为整个显示屏。在预览状态下，即可以是用户打开AR视频APP且未按下拍摄按钮之前，拍摄取景框内均可以实时显示预览图像。In one example, the terminal device detects the user's operation of clicking the icon of the AR video APP on the display interface, can start the AR video APP, and display the display interface of the AR video APP; the display interface can include a shooting viewfinder; for example, on In video recording mode, the shooting viewfinder can be part of the screen or the entire display. In the preview state, that is, before the user opens the AR video APP and does not press the shooting button, the preview image can be displayed in real time in the shooting viewfinder.
还应理解，上述通过AR视频APP进行举例描述，本申请实施例对应用程序的名称不作任何限定。It should also be understood that the above description is based on an example of an AR video APP, and the embodiment of the present application does not place any restrictions on the name of the application.
步骤S220、获取原始视频与位姿信息。Step S220: Obtain the original video and pose information.
例如，如图7所示终端设备检测到用户点击拍摄的控件350的操作，开始录制拍摄取景框中显示的图像。For example, as shown in FIG. 7 , the terminal device detects the user's operation of clicking the shooting control 350 and starts recording the image displayed in the shooting viewfinder.
应理解，用户用于指示拍摄的行为可以包括按下拍摄按钮，也可以包括用户设备通过语音指示终端设备进行拍摄行为，或者，还可以包括用户其它的指示终端设备进行拍摄行为。上述为举例说明，并不对本申请作任何限定。It should be understood that the user's behavior of instructing shooting may include pressing a shooting button, or the user device may instruct the terminal device to perform shooting behavior through voice, or may also include other user instructing the terminal device to perform shooting behavior. The above are examples and do not limit this application in any way.
示例性地，位姿信息可以用于表示终端设备的摄像头在获取原始视频时的位姿；位姿信息可以包括姿态信息与位置信息。For example, the pose information can be used to represent the pose of the camera of the terminal device when acquiring the original video; the pose information can include pose information and position information.
例如，终端设备可以通过如图1所示的陀螺仪传感器180B获取每帧图像对应的位姿信息。For example, the terminal device can obtain the pose information corresponding to each frame of image through the gyroscope sensor 180B as shown in FIG. 1 .
步骤S230、保存位姿信息与虚拟平面的信息。Step S230: Save pose information and virtual plane information.
其中，保存的位姿信息可以是指原始视频中每个图像帧对应的位姿信息。The saved pose information may refer to the pose information corresponding to each image frame in the original video.
示例性地，根据原始视频中任意一个图像帧的位姿信息可以对该图像帧进行特征点提取，并通过计算得到稀疏点云；根据稀疏点云通过算法拟合可以生成虚拟平面；在真实物体的视频中添加虚拟内容时，可以根据虚拟平面对虚拟内容的放置位置进行调整。For example, feature points of any image frame in the original video can be extracted based on the pose information of the image frame, and a sparse point cloud can be obtained through calculation; a virtual plane can be generated through algorithm fitting based on the sparse point cloud; in real objects When adding virtual content to a video, the placement of the virtual content can be adjusted according to the virtual plane.
在本申请的实施例中，保存位姿信息与虚拟平面的信息可以使得在原始视频录制结束后，根据原始视频的位姿信息与虚拟平面的信息在原始视频中添加虚拟内容生成一个新的AR视频；由于保存了位姿信息与虚拟平面的信息，用户可以对原始视频进行多次不同的编辑，分别生成包括不同虚拟内容的AR视频。In the embodiment of the present application, saving the pose information and the virtual plane information can enable adding virtual content to the original video to generate a new AR based on the pose information and the virtual plane information of the original video after the recording of the original video. Video; since the pose information and virtual plane information are saved, users can edit the original video multiple times and generate AR videos containing different virtual contents.
在一个示例中，在本申请的AR视频的处理方法中可以将获取的三维姿态信息通过为四元数进行表示，从而能够避免通过三个参数表示姿态所产生歧义。In one example, in the AR video processing method of the present application, the acquired three-dimensional posture information can be expressed as a quaternion, thereby avoiding the ambiguity caused by expressing the posture through three parameters.
其中，四元数可以是指由实数加上三个虚数单位i、j、k组成；比如，四元数都可以是1、i、j和k的线性组合，即四元数一般可表示为a+bi+cj+dk，其中a、b、c、d均表示实数；i、j、k可以表示旋转；其中，i旋转可以表示X轴与Y轴相交平面中X轴正向向Y轴正向的旋转，j旋转可以表示Z轴与X轴相交平面中Z轴正向向X轴正向的旋转，k旋转可以表示Y轴与Z轴相交平面中Y轴正向向Z轴正向的旋转。Among them, quaternions can refer to real numbers plus three imaginary units i, j, and k; for example, quaternions can be linear combinations of 1, i, j, and k, that is, quaternions can generally be expressed as a+bi+cj+dk, where a, b, c, d all represent real numbers; i, j, k can represent rotation; among them, i rotation can represent the positive direction of the X-axis toward the Y-axis in the plane where the X-axis and the Y-axis intersect Forward rotation, j rotation can represent the rotation from the positive direction of the Z axis to the positive direction of the X axis in the plane where the Z axis intersects the X axis, and k rotation can represent the positive direction of the Y axis to the positive direction of the Z axis in the plane where the Y axis intersects the Z axis. of rotation.
示例性地，在终端设备接收到用户指示拍摄的指令；比如，用户在终端设备上点击视频录制时，终端设备可以启动位姿计算的初始化工作；在未成功初始化前位姿可以表示为(位置x/y/z，旋转四元数)，即可以表示为(0,0,0,0,0,0,0)，虚拟平面的信息为(数量0)；初始化成功时，指定图像帧(初始化起始帧)位姿表示为(0,0,0,0,0,0,0)，虚拟平面的信息表示为(数量x，平面编号0，平面0点数n,点0的位置X1,Y1,Z1,…,点n的位置Xn,Yn,Zn)。For example, when the terminal device receives an instruction from the user to instruct shooting; for example, when the user clicks video recording on the terminal device, the terminal device can start the initialization work of pose calculation; the pose before unsuccessful initialization can be expressed as (position x/y/z, rotation quaternion), which can be expressed as (0,0,0,0,0,0,0), the information of the virtual plane is (number 0); when the initialization is successful, specify the image frame ( Initialization starting frame) pose is expressed as (0,0,0,0,0,0,0), virtual plane information is expressed as (number x, plane number 0, plane 0 point number n, point 0 position X1 ,Y1 ,Z1 ,..., the position of point n Xn ,Yn ,Zn ).
其中，数量x表示虚拟平面的总数量，即视频中包括的图像帧的总数；平面编号0可以用于表示多个虚拟平面中的第一个虚拟平面；平面0点数n可以用于表示第一个虚拟平面中包括顶点的数量为n；点0的位置X1,Y1,Z1用于表示第一个虚拟平面中包括顶点0的位置信息；点n的位置Xn,Yn,Zn用于表示第一个虚拟平面中包括顶点n的位置信息。Among them, the number The number of vertices included in each virtual plane is n; the position X1 , Y1 , Z1 of point 0 is used to represent the position information including vertex 0 in the first virtual plane; the position Xn , Yn , Z of point nn is used to represent the position information of the first virtual plane including vertex n.
应理解，虚拟平面的信息可以包括改虚拟平面中包括的所有顶点的位置信息。It should be understood that the information of the virtual plane may include position information of all vertices included in the virtual plane.
例如，在视频录制过程中，获取的当前图像帧对应的位姿信息可以表示为(X,Y,Z,q0,q1,q2,q3)，虚拟平面的信息可以表示为(数量x,平面编号A,平面A点数n,点0的位置X1,Y1,Z1,…,点q的位置Xq,Yq,Zq)。For example, during the video recording process, the pose information corresponding to the current image frame obtained can be expressed as (X, Y, Z, q0 , q1 , q2 , q3 ), and the information of the virtual plane can be expressed as (number x, plane number A, plane A point number n, the position of point 0 X1 , Y1 , Z1 ,..., the position of point q Xq , Yq , Zq ).
其中，x,y,z可以分别表示获取当前图像帧的摄像头在x轴、y轴以及z轴的坐标；q0,q1,q2,q3表示旋转四元数；比如，可以表示为俯仰角、方位角、旋转角以及欧拉角；数量x表示平面的总数量；平面编号A可以用于表示当前图像帧对应的虚拟平面的标识；平面A点数n用于表示当前图像帧对应的虚拟平面中包括顶点的数量为n；点0的位置X1,Y1,Z1可以用于表示当前图像帧对应的虚拟平面中包括顶点0的位置信息；点n的位置Xn,Yn,Zn用于表示当前图像帧对应的虚拟平面包括的顶点n的位置信息。Among them, x, y, and z can respectively represent the coordinates of the camera that obtains the current image frame on the x-axis, y-axis, and z-axis; q0 , q1 , q2 , q3 represent rotation quaternions; for example, it can be expressed as Pitch angle, azimuth angle, rotation angle and Euler angle; the quantity x represents the total number of planes; the plane number A can be used to represent the identification of the virtual plane corresponding to the current image frame; the number of plane A points n is used to represent the virtual plane corresponding to the current image frame The number of vertices included in the virtual plane is n; the position X1 , Y1 , Z1 of point 0 can be used to represent the position information of vertex 0 included in the virtual plane corresponding to the current image frame; the position Xn , Yn of point n ,Zn is used to represent the position information of vertex n included in the virtual plane corresponding to the current image frame.
在一个示例中，可以获取原始视频中的一个图像帧；根据该图像帧的位姿信息可以对该图像帧进行特征点提取，并通过计算得到稀疏点云；根据稀疏点云信息可以拟合生成虚拟平面；在对视频添加虚拟内容时，可以根据虚拟平面对视频中添加的虚拟内容的所在位置进行调整。In one example, an image frame in the original video can be obtained; the feature points of the image frame can be extracted based on the pose information of the image frame, and a sparse point cloud can be obtained through calculation; the sparse point cloud information can be fitted and generated Virtual plane; when adding virtual content to a video, the location of the virtual content added to the video can be adjusted according to the virtual plane.
例如，在用户点击屏幕/手势操作放置虚拟内容时，用户的操作与生成的虚拟平面可以产生碰撞，确定虚拟内容的放置位置。For example, when the user clicks on the screen/gestures to place virtual content, the user's operation can collide with the generated virtual plane to determine the placement position of the virtual content.
在本申请的实施例中，在获取位姿信息与虚拟平面的信息后，终端设备可以保存位姿信息与虚拟平面的信息。In embodiments of the present application, after acquiring the pose information and the information of the virtual plane, the terminal device may save the pose information and the information of the virtual plane.
在一个示例中，自定义信息包括上述位姿信息与虚拟平面的信息，终端设备可以将自定义信息保存为独立保存为独立的二进制文件(binary,bin)。In one example, the custom information includes the above-mentioned pose information and virtual plane information, and the terminal device can save the custom information as an independent binary file (binary, bin).
例如，可以将原始视频与原始视频对应的自定义信息保存在相同的目录下。For example, the original video and the custom information corresponding to the original video can be saved in the same directory.
例如，可以将原始视频对应的自定义信息与原始视频的命名相同保存在终端设备中。For example, the custom information corresponding to the original video can be saved in the terminal device with the same name as the original video.
例如，可以通过一个图像帧的帧号作为标识，将原始视频对应的自定义信息保存在终端设备中。For example, the frame number of an image frame can be used as an identifier to save the customized information corresponding to the original video in the terminal device.
示例性地，可以根据以下数据格式将原始视频中每个图像帧对应的自定义信息保存为独立的bin文件：For example, the custom information corresponding to each image frame in the original video can be saved as a separate bin file according to the following data format:
帧号：Frame num:unsigned int32；Frame number: Frame num:unsigned int32;
位姿信息：(数据1，数据2，数据3，数据4，数据5，数据6，数据7)；其中，数据1～数据7可以是float格式的数据；Pose information: (data 1, data 2, data 3, data 4, data 5, data 6, data 7); among them, data 1 to data 7 can be data in float format;
虚拟平面的信息：(num:unsigned int32；planeNum0:unsigned int32；planeNumPoint:unsigned int32；point0(float,float,float)…pointN(float,float,float)…planeNumN…)；Virtual plane information: (num: unsigned int32; planeNum0: unsigned int32; planeNumPoint: unsigned int32; point0(float, float, float)…pointN(float, float, float)…planeNumN…);
例如，在对原始视频进行编辑时，可以同时加载原始视频和上述bin文件；根据帧号对原始视频中的图像帧与该图像帧对应的自定义信息进行同步对齐。For example, when editing the original video, you can load the original video and the above-mentioned bin file at the same time; synchronize the image frames in the original video with the custom information corresponding to the image frames according to the frame number.
在一个示例中，自定义信息可以包括上述位姿信息与虚拟平面的信息，终端设备可以将自定义信息保存到原始视频对应视频码流中的补充增强信息中。In one example, the custom information may include the above-mentioned pose information and virtual plane information, and the terminal device may save the custom information into supplementary enhancement information in the video code stream corresponding to the original video.
例如，可以将以下信息进行视频压缩编码的时候存入到h.264/h.265的SEI信息中：For example, the following information can be stored in h.264/h.265 SEI information during video compression encoding:
位姿信息：(float,float,float,float,float,float,float)；Pose information: (float, float, float, float, float, float, float);
虚拟平面的信息：(num:unsigned int32；planeNum0:unsigned int32；planeNumPoint:unsigned int32；point0(float,float,float)…pointN(float,float,float)…planeNumN…)。Virtual plane information: (num: unsigned int32; planeNum0: unsigned int32; planeNumPoint: unsigned int32; point0(float, float, float)…pointN(float, float, float)…planeNumN…).
将自定义信息存入视频压缩编码的SEI信息的情况下，在执行步骤S250在进行编辑视频解码时，可以按照上述格式进行自定义信息的解码。In the case where the custom information is stored in the SEI information of the video compression encoding, when decoding the edited video in step S250, the custom information can be decoded according to the above format.
在本申请的实施例中，为了减少保存上述位姿信息与虚拟平面的信息所占用终端设备的存储空间，可以采用以下的至少一种方式进行对自定义信息进行压缩处理：In the embodiment of the present application, in order to reduce the storage space of the terminal device occupied by saving the above pose information and virtual plane information, at least one of the following methods can be used to compress the custom information:
根据当前图像帧与前一图像帧的差保存位姿信息；或者，虚拟平面的平面编号可以采用无符号字符(unsigned char)方式保存；或者，对于虚拟平面中顶点的描述，水平面可以保留一个点的Z轴信息删除其他点的Z轴信息，垂直面可以保留一个点的Y轴信息删除其他点的Y轴信息；或者，顶点的位置描述可以采用float16；或者，保存虚拟平面的信息时可以只保存当前视野范围内的平面。Save pose information based on the difference between the current image frame and the previous image frame; alternatively, the plane number of the virtual plane can be saved in the form of unsigned characters (unsigned char); or, for the description of the vertices in the virtual plane, a point can be retained on the horizontal plane The Z-axis information deletes the Z-axis information of other points. The vertical plane can retain the Y-axis information of one point and delete the Y-axis information of other points; alternatively, the position description of the vertex can be float16; or, when saving the information of the virtual plane, only Save the plane within the current field of view.
在本申请的实施例中，通过AR视频APP获取原始视频一方面是为了在录制视频时能够生成以及保存原始视频的位姿信息与虚拟平面的信息；另一方面，在原始视频录制结束后，可以对原始视频中每个图像帧进行编辑；比如，添加虚拟内容。In the embodiment of the present application, the purpose of obtaining the original video through the AR video APP is to generate and save the pose information and virtual plane information of the original video when recording the video; on the other hand, after the original video recording is completed, Each image frame in the original video can be edited; for example, virtual content can be added.
步骤S240、原始视频录制结束。Step S240: The original video recording ends.
例如，如图8所示终端设备检测到用户再次点击拍摄的控件350的操作，结束本次视频的录制；比如，本次录制视频为20秒。For example, as shown in Figure 8, the terminal device detects that the user clicks the shooting control 350 again, and ends the recording of this video; for example, the video recorded this time is 20 seconds.
步骤S250、打开虚拟平面的可视化界面，对原始视频进行编辑。Step S250: Open the visualization interface of the virtual plane and edit the original video.
应理解，在对原始视频中的任意一个图像帧进行编辑时，终端设备可以调用保存的该图像帧对应的自定义信息；即调用该图像帧的位姿信息与平面信息。It should be understood that when editing any image frame in the original video, the terminal device can call the saved custom information corresponding to the image frame; that is, call the pose information and plane information of the image frame.
例如，提取原始视频中第8秒的任意一个图像帧，如图9所示显示界面330还可以包括编辑选项360；在终端设备检测到用户点击编辑选项360后，终端设备可以显示编辑模式的界面，如图10所示；终端设备检测到用户点击编辑模式界面上用于指示AR内容选择361后，显示如图11所示的显示界面；在图11的显示界面中还包括显示平面选项362，终端设备检测到用户可以点击显示平面选项362的操作，在显示界面中显示生成的虚拟平面363，参见图12；在本申请的实施例中，终端设备的显示界面上可以向用户提供用于放置虚拟内容的可视化平面；例如，在用户添加虚拟内容的过程中，在显示界面上可以显示虚拟平面363；在用户点击屏幕/手势操作放置虚拟内容时，用户的操作与虚拟平面363产生碰撞，从而确定虚拟内容的放置位置，如图12所示。For example, any image frame at the 8th second of the original video is extracted. As shown in Figure 9, the display interface 330 can also include an editing option 360; after the terminal device detects that the user clicks the editing option 360, the terminal device can display an editing mode interface. , as shown in Figure 10; after the terminal device detects that the user clicks on the edit mode interface to indicate AR content selection 361, the display interface as shown in Figure 11 is displayed; the display interface in Figure 11 also includes a display plane option 362, The terminal device detects that the user can click on the display plane option 362, and displays the generated virtual plane 363 in the display interface, see Figure 12; in the embodiment of the present application, the display interface of the terminal device can provide the user with a method for placing The visualization plane of virtual content; for example, when the user adds virtual content, the virtual plane 363 can be displayed on the display interface; when the user clicks on the screen/gestures to place virtual content, the user's operation collides with the virtual plane 363, thus Determine where to place the virtual content, as shown in Figure 12.
应理解，在对虚拟内容进行编辑比如调整虚拟内容的位置时，可以在界面中显示虚拟平面363；在完成编辑后，虚拟平面363并不会出现在AR视频中；虚拟平面363可以作为一个参考平面，用于用户确定虚拟内容在视频中的添加位置。It should be understood that when editing virtual content, such as adjusting the position of the virtual content, the virtual plane 363 can be displayed in the interface; after the editing is completed, the virtual plane 363 will not appear in the AR video; the virtual plane 363 can be used as a reference. A plane used by the user to determine where virtual content is added in the video.
步骤S260、生成包括虚拟内容的AR视频。Step S260: Generate an AR video including virtual content.
示例性地，用户可以对原始视频中的每一个图像帧进行编辑；比如，可以在每一个图像帧中添加虚拟内容，对每一个图像帧中的虚拟内容的位置信息进行调整；从而生成带虚拟内容的AR视频。For example, the user can edit each image frame in the original video; for example, virtual content can be added to each image frame, and the position information of the virtual content in each image frame can be adjusted; thereby generating a video with virtual content. AR video of content.
在一个示例中，用户可以播放原始视频，用户点击暂停键可以提取当前图像帧并对当前图像帧进行编辑，即在当前图像帧中添加虚拟内容；当用户再次点击播放按钮时，当前图像帧编辑完成。In one example, the user can play the original video, and the user clicks the pause button to extract the current image frame and edit the current image frame, that is, adding virtual content to the current image frame; when the user clicks the play button again, the current image frame is edited Finish.
在本申请的实施例中，在获取原始视频时可以获取原始视频对应的位姿信息；根据位姿信息与原始视频可以得到虚拟平面；在原始视频的图像帧中添加虚拟内容时，虚拟平面可以作为一个基准面，根据虚拟平面可以调整虚拟内容在原始视频中的位置，使得虚拟内容能够更好的融入到原始视频中，提高AR视频的视频质量。In the embodiment of the present application, when acquiring the original video, the pose information corresponding to the original video can be obtained; the virtual plane can be obtained according to the pose information and the original video; when virtual content is added to the image frame of the original video, the virtual plane can As a reference plane, the position of the virtual content in the original video can be adjusted according to the virtual plane, so that the virtual content can be better integrated into the original video and improve the video quality of the AR video.
实现方式二：将本申请实施例的AR视频的处理方法集成在终端设备的相机的模式中。Implementation Mode 2: Integrate the AR video processing method of the embodiment of the present application into the camera mode of the terminal device.
如图13所示，图13是本申请实施例提供的AR视频的处理方法的示意性流程图；该处理方法400包括步骤S410至步骤S470，下面分别对这些步骤进行详细的描述。As shown in Figure 13, Figure 13 is a schematic flow chart of an AR video processing method provided by an embodiment of the present application; the processing method 400 includes steps S410 to S470, and these steps are described in detail below.
步骤S410、运行终端设备的相机。Step S410: Run the camera of the terminal device.
例如，终端设备检查到用户点击相机的操作；响应于用户的点击操作，终端设备可以运行相机。For example, the terminal device detects the user's click operation on the camera; in response to the user's click operation, the terminal device can run the camera.
图14示出了终端设备的一种GUI，该GUI可以为终端设备的桌面510；当终端设备检测到用户点击桌面510上的相机的图标520的操作后，可以运行相机显示如图15所示的另一GUI，该GUI可以是相机的显示界面530；该显示界面530上可以包括拍摄取景框540、指示拍摄的控件550，以及其它拍摄控件，其中，拍摄取景框540内可以实时显示预览图像。Figure 14 shows a GUI of a terminal device, which can be the desktop 510 of the terminal device; when the terminal device detects the user's operation of clicking the camera icon 520 on the desktop 510, the camera display can be run as shown in Figure 15 Another GUI, the GUI may be the display interface 530 of the camera; the display interface 530 may include a shooting viewfinder frame 540, a control 550 for instructing shooting, and other shooting controls, wherein the preview image may be displayed in real time in the shooting viewfinder frame 540 .
步骤S420、选择AR拍摄模式。Step S420: Select the AR shooting mode.
例如，终端设备可以是检测到用户指示AR拍摄模式的操作。其中，AR拍摄模式可以是指在可以对原始视频进行处理添加虚拟内容的拍摄模式。For example, the terminal device may detect an operation in which the user indicates the AR shooting mode. Among them, the AR shooting mode may refer to a shooting mode that can process the original video and add virtual content.
如图16所示，拍摄界面上还包括设置560，在终端设备检测到用户点击设置560后，终端设备显示设置模式界面，如图17所示；终端设备检测到用户点击设置模式界面上用于指示AR视频561后，终端设备进入AR拍摄模式。As shown in Figure 16, the shooting interface also includes setting 560. After the terminal device detects that the user clicks on setting 560, the terminal device displays the setting mode interface, as shown in Figure 17; the terminal device detects that the user clicks on the setting mode interface for After instructing AR video 561, the terminal device enters AR shooting mode.
步骤S430、获取原始视频与位姿信息。Step S430: Obtain the original video and pose information.
例如，如图17所示终端设备检测到用户点击拍摄的控件550的操作，开始录制拍摄取景框中显示的图像。For example, as shown in FIG. 17 , the terminal device detects the user's operation of clicking the shooting control 550 and starts recording the image displayed in the shooting viewfinder.
应理解，用户用于指示拍摄的行为可以包括按下拍摄按钮，也可以包括用户设备通过语音指示终端设备进行拍摄的行为，或者，还可以包括用户其它的指示终端设备进行拍摄行为；上述为举例说明，并不对本申请作任何限定。It should be understood that the user's behavior of instructing the shooting may include pressing the shooting button, the user device may also include the user device instructing the terminal device to shoot via voice, or may also include the user's other behavior of instructing the terminal device to shoot; the above are examples. description does not limit this application in any way.
示例性地，位姿信息可以用于表示终端设备的摄像头在获取原始视频时的位姿；位姿信息可以包括姿态信息与位置信息。For example, the pose information can be used to represent the pose of the camera of the terminal device when acquiring the original video; the pose information can include pose information and position information.
例如，终端设备可以通过如图1所示的陀螺仪传感器180B获取每帧图像对应的位姿信息。For example, the terminal device can obtain the pose information corresponding to each frame of image through the gyroscope sensor 180B as shown in FIG. 1 .
步骤S440、保存位姿信息与虚拟平面的信息。Step S440: Save pose information and virtual plane information.
其中，保存的位姿信息可以是指原始视频中每个图像帧对应的位姿信息。The saved pose information may refer to the pose information corresponding to each image frame in the original video.
示例性地，根据原始视频中任意一个图像帧的位姿信息可以对该图像帧进行特征点提取，并通过计算得到稀疏点云；根据稀疏点云通过算法拟合可以生成虚拟平面；在真实场景中添加虚拟内容时，可以根据虚拟平面对虚拟内容的放置位置进行调整。For example, feature points can be extracted from the image frame based on the pose information of any image frame in the original video, and a sparse point cloud can be obtained through calculation; a virtual plane can be generated through algorithm fitting based on the sparse point cloud; in a real scene When adding virtual content, you can adjust the placement of the virtual content according to the virtual plane.
在本申请的实施例中，保存位姿信息与虚拟平面的信息可以使得在原始视频录制结束后，根据原始视频的位姿信息与虚拟平面的信息在原始视频中添加虚拟内容生成一个新的AR视频；由于保存了位姿信息与虚拟平面的信息，用户可以对原始视频进行多次不同的编辑，分别生成包括不同虚拟内容的AR视频。In the embodiment of the present application, saving the pose information and the virtual plane information can enable adding virtual content to the original video to generate a new AR based on the pose information and the virtual plane information of the original video after the recording of the original video. Video; since the pose information and virtual plane information are saved, users can edit the original video multiple times and generate AR videos containing different virtual contents.
在一个示例中，在本申请的AR视频的处理方法中可以将获取的三维姿态信息通过为四元数进行表示，从而能够避免通过三个参数表示姿态所产生歧义。In one example, in the AR video processing method of the present application, the acquired three-dimensional posture information can be expressed as a quaternion, thereby avoiding the ambiguity caused by expressing the posture through three parameters.
其中，四元数可以是指由实数加上三个虚数单位i、j、k组成；比如，四元数都可以是1、i、j和k的线性组合，即四元数一般可表示为a+bi+cj+dk，其中a、b、c、d均表示实数；i、j、k可以表示旋转；其中，i旋转可以表示X轴与Y轴相交平面中X轴正向向Y轴正向的旋转，j旋转可以表示Z轴与X轴相交平面中Z轴正向向X轴正向的旋转，k旋转可以表示Y轴与Z轴相交平面中Y轴正向向Z轴正向的旋转。Among them, quaternions can refer to real numbers plus three imaginary units i, j, and k; for example, quaternions can be linear combinations of 1, i, j, and k, that is, quaternions can generally be expressed as a+bi+cj+dk, where a, b, c, d all represent real numbers; i, j, k can represent rotation; among them, i rotation can represent the positive direction of the X-axis toward the Y-axis in the plane where the X-axis and the Y-axis intersect Forward rotation, j rotation can represent the rotation from the positive direction of the Z axis to the positive direction of the X axis in the plane where the Z axis intersects the X axis, and k rotation can represent the positive direction of the Y axis to the positive direction of the Z axis in the plane where the Y axis intersects the Z axis. of rotation.
示例性地，在终端设备接收到用户指示拍摄的指令；比如，用户在终端设备上点击视频录制时，终端设备可以启动位姿计算的初始化工作；在未成功初始化前位姿可以表示为(位置x/y/z，旋转四元数)，即可以表示为(0,0,0,0,0,0,0)，虚拟平面的信息为(数量0)；初始化成功时，指定图像帧(初始化起始帧)位姿表示为(0,0,0,0,0,0,0)，虚拟平面的信息表示为(数量x，平面编号0,平面0点数n,点0的位置X1,Y1,Z1,…,点n的位置Xn,Yn,Zn)。For example, when the terminal device receives an instruction from the user to instruct shooting; for example, when the user clicks video recording on the terminal device, the terminal device can start the initialization work of pose calculation; the pose before unsuccessful initialization can be expressed as (position x/y/z, rotation quaternion), which can be expressed as (0,0,0,0,0,0,0), the information of the virtual plane is (number 0); when the initialization is successful, specify the image frame ( Initialization starting frame) pose is expressed as (0,0,0,0,0,0,0), and the information of the virtual plane is expressed as (number x, plane number 0, plane 0 point number n, point 0 position X1 ,Y1 ,Z1 ,..., the position of point n Xn ,Yn ,Zn ).
其中，数量x表示虚拟平面的总数量，即视频中包括的图像帧的总数；平面编号0可以用于表示多个虚拟平面中的第一个虚拟平面；平面0点数n可以用于表示第一个虚拟平面中包括顶点的数量为n；点0的位置X1,Y1,Z1用于表示第一个虚拟平面中包括顶点0的位置信息；点n的位置Xn,Yn,Zn用于表示第一个虚拟平面中包括顶点n的位置信息。Among them, the number The number of vertices included in each virtual plane is n; the position X1 , Y1 , Z1 of point 0 is used to represent the position information including vertex 0 in the first virtual plane; the position Xn , Yn , Z of point nn is used to represent the position information of the first virtual plane including vertex n.
应理解，虚拟平面的信息可以包括改虚拟平面中包括的所有顶点的位置信息。It should be understood that the information of the virtual plane may include position information of all vertices included in the virtual plane.
例如，在视频录制过程中，获取的当前图像帧对应的位姿信息可以表示为(X,Y,Z,q0,q1,q2,q3)，虚拟平面的信息可以表示为(数量x,平面编号A,平面A点数n,点0的位置X1,Y1,Z1,…,点q的位置Xq,Yq,Zq)。For example, during the video recording process, the pose information corresponding to the current image frame obtained can be expressed as (X, Y, Z, q0 , q1 , q2 , q3 ), and the information of the virtual plane can be expressed as (number x, plane number A, plane A point number n, the position of point 0 X1 , Y1 , Z1 ,..., the position of point q Xq , Yq , Zq ).
其中，x,y,z可以分别表示获取当前图像帧的摄像头在x轴、y轴以及z轴的坐标；q0,q1,q2,q3表示旋转四元数；比如，可以表示为俯仰角、方位角、旋转角以及欧拉角；数量x表示平面的总数量；平面编号A可以用于表示当前图像帧对应的虚拟平面的标识；平面A点数n用于表示当前图像帧对应的虚拟平面中包括顶点的数量为n；点0的位置X1,Y1,Z1可以用于表示当前图像帧对应的虚拟平面中包括顶点0的位置信息；点n的位置Xn,Yn,Zn用于表示当前图像帧对应的虚拟平面包括的顶点n的位置信息。Among them, x, y, and z can respectively represent the coordinates of the camera that obtains the current image frame on the x-axis, y-axis, and z-axis; q0 , q1 , q2 , q3 represent rotation quaternions; for example, it can be expressed as Pitch angle, azimuth angle, rotation angle and Euler angle; the quantity x represents the total number of planes; the plane number A can be used to represent the identification of the virtual plane corresponding to the current image frame; the number of plane A points n is used to represent the virtual plane corresponding to the current image frame The number of vertices included in the virtual plane is n; the position X1 , Y1 , Z1 of point 0 can be used to represent the position information of vertex 0 included in the virtual plane corresponding to the current image frame; the position Xn , Yn of point n ,Zn is used to represent the position information of vertex n included in the virtual plane corresponding to the current image frame.
在一个示例中，可以获取原始视频中的一个图像帧；根据该图像帧的位姿信息可以对该图像帧进行特征点提取，并通过计算得到稀疏点云；根据稀疏点云信息可以拟合生成虚拟平面；在对视频添加虚拟内容时，可以根据虚拟平面对视频中添加的虚拟内容的所在位置进行调整。In one example, an image frame in the original video can be obtained; the feature points of the image frame can be extracted based on the pose information of the image frame, and a sparse point cloud can be obtained through calculation; the sparse point cloud information can be fitted and generated Virtual plane; when adding virtual content to a video, the location of the virtual content added to the video can be adjusted according to the virtual plane.
例如，在用户点击屏幕/手势操作放置虚拟内容时，用户的操作与生成的虚拟平面可以产生碰撞，确定虚拟内容的放置位置。For example, when the user clicks on the screen/gestures to place virtual content, the user's operation can collide with the generated virtual plane to determine the placement position of the virtual content.
在本申请的实施例中，在获取位姿信息与虚拟平面的信息后，终端设备可以保存位姿信息与虚拟平面的信息。In embodiments of the present application, after acquiring the pose information and the information of the virtual plane, the terminal device may save the pose information and the information of the virtual plane.
在一个示例中，自定义信息包括上述位姿信息与虚拟平面的信息，终端设备可以将自定义信息保存为独立保存为独立的二进制文件(binary,bin)。In one example, the custom information includes the above-mentioned pose information and virtual plane information, and the terminal device can save the custom information as an independent binary file (binary, bin).
例如，可以将原始视频与原始视频对应的自定义信息保存在相同的目录下。For example, the original video and the custom information corresponding to the original video can be saved in the same directory.
例如，可以将原始视频对应的自定义信息与原始视频的命名相同保存在终端设备中。For example, the custom information corresponding to the original video can be saved in the terminal device with the same name as the original video.
例如，可以通过一个图像帧的帧号作为标识，将原始视频对应的自定义信息保存在终端设备中。For example, the frame number of an image frame can be used as an identifier to save the customized information corresponding to the original video in the terminal device.
示例性地，可以根据以下数据格式将原始视频中每个图像帧对应的自定义信息保存为独立的bin文件：For example, the custom information corresponding to each image frame in the original video can be saved as a separate bin file according to the following data format:
帧号：Frame num:unsigned int32；Frame number: Frame num:unsigned int32;
位姿信息：(数据1，数据2，数据3，数据4，数据5，数据6，数据7)；其中，数据1～数据7可以是float格式的数据；Pose information: (data 1, data 2, data 3, data 4, data 5, data 6, data 7); among them, data 1 to data 7 can be data in float format;
虚拟平面的信息：(num:unsignedint32；planeNum0:unsigned int32；planeNumPoint:unsigned int32；point0(float,float,float)…pointN(float,float,float)…planeNumN…)；Virtual plane information: (num: unsignedint32; planeNum0: unsigned int32; planeNumPoint: unsigned int32; point0(float, float, float)…pointN(float, float, float)…planeNumN…);
例如，在对原始视频进行编辑时，可以同时加载原始视频和上述bin文件；根据帧号对原始视频中的图像帧与该图像帧对应的自定义信息进行同步对齐。For example, when editing the original video, you can load the original video and the above-mentioned bin file at the same time; synchronize the image frames in the original video with the custom information corresponding to the image frames according to the frame number.
在一个示例中，自定义信息可以包括上述位姿信息与虚拟平面的信息，终端设备可以将自定义信息保存到原始视频对应视频码流中的补充增强信息中。In one example, the custom information may include the above-mentioned pose information and virtual plane information, and the terminal device may save the custom information into supplementary enhancement information in the video code stream corresponding to the original video.
例如，可以将以下信息进行视频压缩编码的时候存入到h.264/h.265的SEI信息中：For example, the following information can be stored in h.264/h.265 SEI information during video compression encoding:
位姿信息：(float,float,float,float,float,float,float)；Pose information: (float, float, float, float, float, float, float);
虚拟平面的信息：(num:unsignedint32；planeNum0:unsigned int32；planeNumPoint:unsigned int32；point0(float,float,float)…pointN(float,float,float)…planeNumN…)。Virtual plane information: (num: unsignedint32; planeNum0: unsigned int32; planeNumPoint: unsigned int32; point0(float, float, float)…pointN(float, float, float)…planeNumN…).
将自定义信息存入视频压缩编码的SEI信息的情况下，在执行步骤S250在进行编辑视频解码时，可以按照上述格式进行自定义信息的解码。In the case where the custom information is stored in the SEI information of the video compression encoding, when decoding the edited video in step S250, the custom information can be decoded according to the above format.
在本申请的实施例中，为了减少保存上述位姿信息与虚拟平面的信息所占用终端设备的存储空间，可以采用以下的至少一种方式进行对自定义信息进行压缩处理：In the embodiment of the present application, in order to reduce the storage space of the terminal device occupied by saving the above pose information and virtual plane information, at least one of the following methods can be used to compress the custom information:
根据当前图像帧与前一图像帧的差保存位姿信息；或者，虚拟平面的平面编号可以采用无符号字符(unsigned char)方式保存；或者，对于虚拟平面中顶点的描述，水平面可以保留一个点的Z轴信息删除其他点的Z轴信息，垂直面可以保留一个点的Y轴信息删除其他点的Y轴信息；或者，顶点的位置描述可以采用float16；或者，保存虚拟平面的信息时可以只保存当前视野范围内的平面。Save pose information based on the difference between the current image frame and the previous image frame; alternatively, the plane number of the virtual plane can be saved in the form of unsigned characters (unsigned char); or, for the description of the vertices in the virtual plane, a point can be retained on the horizontal plane The Z-axis information deletes the Z-axis information of other points. The vertical plane can retain the Y-axis information of one point and delete the Y-axis information of other points; alternatively, the position description of the vertex can be float16; or, when saving the information of the virtual plane, only Save the plane within the current field of view.
步骤S450、原始视频录制结束。Step S450: The original video recording ends.
例如，如图19所示终端设备检测到用户再次点击拍摄的控件550的操作，结束本次视频的录制；比如，本次录制视频为20秒。For example, as shown in Figure 19, the terminal device detects that the user clicks the shooting control 550 again, and ends the recording of this video; for example, the video recorded this time is 20 seconds.
步骤S460、对原始视频进行编辑。Step S460: Edit the original video.
应理解，在对原始视频中的任意一个图像帧进行编辑时，终端设备可以调用保存的该图像帧对应的自定义信息；即调用该图像帧的位姿信息与平面信息。It should be understood that when editing any image frame in the original video, the terminal device can call the saved custom information corresponding to the image frame; that is, call the pose information and plane information of the image frame.
例如，通过虚拟平面的可视化界面对原始视频进行编辑；可以提取原始视频中第8秒的任意一帧图像，如图20所示显示界面还可以包括显示平面选项570，终端设备检测到用户点击显示平面选项570的操作，在显示界面中可以显示生成的虚拟平面562，如图21所示。For example, the original video is edited through the virtual plane visual interface; any frame of the 8th second in the original video can be extracted. As shown in Figure 20, the display interface can also include a display plane option 570. The terminal device detects that the user clicks the display By operating the plane option 570, the generated virtual plane 562 can be displayed in the display interface, as shown in Figure 21.
例如，在用户添加虚拟内容的过程中，在显示界面上可以显示虚拟平面562；在用户点击屏幕/手势操作放置虚拟内容时，用户的操作与虚拟平面562产生碰撞，从而确定虚拟内容的放置位置。For example, when the user is adding virtual content, the virtual plane 562 may be displayed on the display interface; when the user clicks on the screen/gestures to place virtual content, the user's operation collides with the virtual plane 562, thereby determining the placement location of the virtual content. .
应理解，在对虚拟内容进行编辑比如调整虚拟内容的位置时，可以在界面中显示虚拟平面562；在完成编辑后，虚拟平面562并不会出现在AR视频中；虚拟平面562用于用户确定虚拟内容在视频中的添加位置。It should be understood that when editing virtual content, such as adjusting the position of the virtual content, the virtual plane 562 can be displayed in the interface; after the editing is completed, the virtual plane 562 will not appear in the AR video; the virtual plane 562 is used for the user to determine Where in the video the virtual content is added.
步骤S470、生成包括虚拟内容的AR视频。Step S470: Generate an AR video including virtual content.
示例性地，用户可以对原始视频中的每一个图像帧进行编辑；比如，可以在每一个图像帧中添加虚拟内容，对每一个图像帧中的虚拟内容的位置信息进行调整；从而生成带虚拟内容的AR视频。For example, the user can edit each image frame in the original video; for example, virtual content can be added to each image frame, and the position information of the virtual content in each image frame can be adjusted; thereby generating a video with virtual content. AR video of content.
在一个示例中，用户可以播放原始视频，用户点击暂停键可以提取当前图像帧并对当前图像帧进行编辑，即在当前图像帧中添加虚拟内容；当用户再次点击播放按钮时，当前图像帧编辑完成。In one example, the user can play the original video, and the user clicks the pause button to extract the current image frame and edit the current image frame, that is, adding virtual content to the current image frame; when the user clicks the play button again, the current image frame is edited Finish.
在本申请的实施例中，在获取原始视频时可以获取原始视频对应的位姿信息；根据位姿信息与原始视频可以得到虚拟平面；在原始视频的图像帧中添加虚拟内容时，虚拟平面可以作为一个基准面，根据虚拟平面可以调整虚拟内容在原始视频中的位置，使得虚拟内容能够更好的融入到原始视频中，提高AR视频的视频质量。In the embodiment of the present application, when acquiring the original video, the pose information corresponding to the original video can be obtained; the virtual plane can be obtained according to the pose information and the original video; when virtual content is added to the image frame of the original video, the virtual plane can As a reference plane, the position of the virtual content in the original video can be adjusted according to the virtual plane, so that the virtual content can be better integrated into the original video and improve the video quality of the AR video.
应理解，上述举例说明是为了帮助本领域技术人员理解本申请实施例，而非要将本申请实施例限于所例示的具体数值或具体场景。本领域技术人员根据所给出的上述举例说明，显然可以进行各种等价的修改或变化，这样的修改或变化也落入本申请实施例的范围内。It should be understood that the above examples are to help those skilled in the art understand the embodiments of the present application, but are not intended to limit the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.
上文结合图1至图21，详细描述了本申请实施例的AR视频的处理方法，下面将结合图22和图23，详细描述本申请的装置实施例。应理解，本申请实施例中的装置可以执行前述本申请实施例的AR视频的处理方法，即以下各种产品的具体工作过程，可以参考前述方法实施例中的对应过程。The AR video processing method according to the embodiment of the present application is described in detail above with reference to FIGS. 1 to 21 . Next, the device embodiment of the present application will be described in detail with reference to FIGS. 22 and 23 . It should be understood that the device in the embodiment of the present application can perform the AR video processing method of the foregoing embodiment of the present application, that is, for the specific working processes of the following various products, reference can be made to the corresponding processes in the foregoing method embodiments.
图22是本申请提供的一种增强现实视频的处理装置的结构示意图。该处理装置600包括获取单元610和处理单元620。Figure 22 is a schematic structural diagram of an augmented reality video processing device provided by this application. The processing device 600 includes an acquisition unit 610 and a processing unit 620.
其中，获取单元610获取原始视频与位姿信息，所述原始视频用于表示真实物体的视频，所述位姿信息用于表示终端设备获取所述原始视频时的位姿；处理单元620用于根据所述原始视频与所述位姿信息生成虚拟平面，所述虚拟平面用于确定在所述原始视频中添加虚拟内容的位置信息；根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成AR视频。Among them, the acquisition unit 610 acquires the original video and pose information, the original video is used to represent the video of the real object, and the pose information is used to represent the pose when the terminal device acquires the original video; the processing unit 620 is used to A virtual plane is generated according to the original video and the pose information, and the virtual plane is used to determine the position information of adding virtual content in the original video; and the virtual plane is added to the original video according to the virtual plane. Content-generated AR videos.
可选地，作为一个实施例，所述位姿信息包括三维姿态信息，所述处理单元620还用于：Optionally, as an embodiment, the pose information includes three-dimensional pose information, and the processing unit 620 is further configured to:
通过四元数表示所述三维姿态信息。The three-dimensional attitude information is represented by quaternions.
可选地，作为一个实施例，所述处理单元620具体用于：Optionally, as an embodiment, the processing unit 620 is specifically used to:
根据所述原始视频中图像帧的位姿信息提取所述图像帧的特征点；Extract feature points of the image frame according to the pose information of the image frame in the original video;
根据所述特征点生成所述虚拟平面。The virtual plane is generated based on the feature points.
可选地，作为一个实施例，所述处理单元620还用于：Optionally, as an embodiment, the processing unit 620 is also used to:
保存所述位姿信息与所述虚拟平面的信息。The pose information and the information of the virtual plane are saved.
可选地，作为一个实施例，所述处理单元620具体用于：Optionally, as an embodiment, the processing unit 620 is specifically used to:
将所述位姿信息与所述虚拟平面的信息保存在二进制文件中。The pose information and the information of the virtual plane are saved in a binary file.
可选地，作为一个实施例，所述处理单元620具体用于：Optionally, as an embodiment, the processing unit 620 is specifically used to:
将所述位姿信息与所述虚拟平面的信息保存在所述原始视频对应的补充增强信息中。The pose information and the information of the virtual plane are stored in the supplementary enhancement information corresponding to the original video.
可选地，作为一个实施例，所述处理单元620还用于：Optionally, as an embodiment, the processing unit 620 is also used to:
对保存的所述位姿信息与所述虚拟平面的信息进行压缩处理。Compression processing is performed on the saved pose information and the information on the virtual plane.
可选地，作为一个实施例，所述处理单元620具体用于：Optionally, as an embodiment, the processing unit 620 is specifically used to:
在所述原始视频录制完成后，根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成所述AR视频。After the original video recording is completed, the virtual content is added to the original video according to the virtual plane to generate the AR video.
可选地，作为一个实施例，所述虚拟平面包括第一虚拟平面，所述第一虚拟平面是指第一图像帧对应的虚拟平面，所述第一图像帧是所述原始视频中的任意一个图像帧；Optionally, as an embodiment, the virtual plane includes a first virtual plane, the first virtual plane refers to a virtual plane corresponding to a first image frame, and the first image frame is any part of the original video. an image frame;
所述第一虚拟平面的信息包括图像帧的总数、所述第一虚拟平面的标识、所述第一虚拟平面包括的顶点数量以及所述第一虚拟平面包括的每一个顶点的位置信息，所述总数是指所述原始视频包括图像帧的总数。The information of the first virtual plane includes the total number of image frames, the identification of the first virtual plane, the number of vertices included in the first virtual plane, and the position information of each vertex included in the first virtual plane, so The total number refers to the total number of image frames included in the original video.
需要说明的是，上述处理装置600以功能单元的形式体现。这里的术语“单元”可以通过软件和/或硬件形式实现，对此不作具体限定。It should be noted that the above-mentioned processing device 600 is embodied in the form of a functional unit. The term "unit" here can be implemented in the form of software and/or hardware, and is not specifically limited.
例如，“单元”可以是实现上述功能的软件程序、硬件电路或二者结合。所述硬件电路可能包括应用特有集成电路(application specific integrated circuit，ASIC)、电子电路、用于执行一个或多个软件或固件程序的处理器(例如共享处理器、专有处理器或组处理器等)和存储器、合并逻辑电路和/或其它支持所描述的功能的合适组件。For example, a "unit" may be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuitry may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor) for executing one or more software or firmware programs. etc.) and memory, merged logic circuitry, and/or other suitable components to support the described functionality.
因此，在本申请的实施例中描述的各示例的单元，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请的范围。Therefore, the units of each example described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered beyond the scope of this application.
图23示出了本申请提供的一种电子设备的结构示意图。图23中的虚线表示该单元或该模块为可选的。电子设备700可用于实现上述方法实施例中描述的处理方法。Figure 23 shows a schematic structural diagram of an electronic device provided by this application. The dashed line in Figure 23 indicates that the unit or module is optional. The electronic device 700 may be used to implement the processing method described in the above method embodiment.
电子设备700包括一个或多个处理器701，该一个或多个处理器702可支持电子设备700实现方法实施例中的方法。处理器701可以是通用处理器或者专用处理器。例如，处理器701可以是中央处理器(central processing unit，CPU)、数字信号处理器(digitalsignal processor，DSP)、专用集成电路(application specific integrated circuit，ASIC)、现场可编程门阵列(field programmable gate array，FPGA)或者其它可编程逻辑器件，如分立门、晶体管逻辑器件或分立硬件组件。The electronic device 700 includes one or more processors 701, and the one or more processors 702 can support the electronic device 700 to implement the method in the method embodiment. Processor 701 may be a general-purpose processor or a special-purpose processor. For example, the processor 701 may be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a field programmable gate array. array, FPGA) or other programmable logic devices such as discrete gates, transistor logic devices or discrete hardware components.
处理器701可以用于对电子设备700进行控制，执行软件程序，处理软件程序的数据。电子设备700还可以包括通信单元705，用以实现信号的输入(接收)和输出(发送)。The processor 701 can be used to control the electronic device 700, execute software programs, and process data of the software programs. The electronic device 700 may also include a communication unit 705 to implement input (reception) and output (transmission) of signals.
例如，电子设备700可以是芯片，通信单元705可以是该芯片的输入和/或输出电路，或者，通信单元705可以是该芯片的通信接口，该芯片可以作为终端设备或其它电子设备的组成部分。For example, the electronic device 700 may be a chip, and the communication unit 705 may be an input and/or output circuit of the chip, or the communication unit 705 may be a communication interface of the chip, and the chip may be used as a component of a terminal device or other electronic device. .
又例如，电子设备700可以是终端设备，通信单元705可以是该终端设备的收发器，或者，通信单元705可以是该终端设备的收发电路。For another example, the electronic device 700 may be a terminal device, and the communication unit 705 may be a transceiver of the terminal device, or the communication unit 705 may be a transceiver circuit of the terminal device.
电子设备700中可以包括一个或多个存储器702，其上存有程序704，程序704可被处理器701运行，生成指令703，使得处理器701根据指令703执行上述方法实施例中描述的AR视频的处理方法。The electronic device 700 may include one or more memories 702 on which a program 704 is stored. The program 704 may be run by the processor 701 to generate an instruction 703, so that the processor 701 executes the AR video described in the above method embodiment according to the instruction 703. processing method.
可选地，存储器702中还可以存储有数据。可选地，处理器701还可以读取存储器702中存储的数据，该数据可以与程序704存储在相同的存储地址，该数据也可以与程序704存储在不同的存储地址。Optionally, data may also be stored in the memory 702 . Optionally, the processor 701 can also read data stored in the memory 702. The data may be stored at the same storage address as the program 704, or the data may be stored at a different storage address than the program 704.
处理器701和存储器702可以单独设置，也可以集成在一起；例如，集成在终端设备的系统级芯片(system on chip，SOC)上。The processor 701 and the memory 702 can be provided separately or integrated together; for example, integrated on a system on chip (SOC) of the terminal device.
示例性地，存储器702可以用于存储本申请实施例中提供的AR视频的处理方法的相关程序704，处理器701可以用于在AR视频编辑时调用存储器702中存储的AR视频的处理方法的相关程序704，执行本申请实施例的AR视频的处理；例如，获取原始视频与位姿信息，所述原始视频用于表示真实物体的视频，所述位姿信息用于表示终端设备获取所述原始视频时的位姿；处理单元用于根据所述原始视频与所述位姿信息生成虚拟平面，所述虚拟平面用于确定在所述原始视频中添加虚拟内容的位置信息；根据所述虚拟平面在所述原始视频中添加所述虚拟内容生成AR视频。For example, the memory 702 can be used to store the relevant program 704 of the AR video processing method provided in the embodiment of the present application, and the processor 701 can be used to call the AR video processing method stored in the memory 702 when editing the AR video. Related program 704 executes AR video processing in the embodiment of the present application; for example, obtaining the original video and pose information, the original video is used to represent the video of the real object, and the pose information is used to represent the terminal device acquiring the The pose of the original video; the processing unit is configured to generate a virtual plane according to the original video and the pose information, and the virtual plane is used to determine the position information of adding virtual content in the original video; according to the virtual The plane adds the virtual content to the original video to generate an AR video.
本申请还提供了一种计算机程序产品，该计算机程序产品被处理器701执行时实现本申请中任一方法实施例所述的处理方法。This application also provides a computer program product, which when executed by the processor 701 implements the processing method described in any method embodiment in this application.
该计算机程序产品可以存储在存储器702中，例如是程序704，程序704经过预处理、编译、汇编和链接等处理过程最终被转换为能够被处理器701执行的可执行目标文件。The computer program product may be stored in the memory 702, such as a program 704. The program 704 is finally converted into an executable object file that can be executed by the processor 701 through processes such as preprocessing, compilation, assembly, and linking.
本申请还提供了一种计算机可读存储介质，其上存储有计算机程序，该计算机程序被计算机执行时实现本申请中任一方法实施例所述的方法。该计算机程序可以是高级语言程序，也可以是可执行目标程序。This application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a computer, the method described in any method embodiment of this application is implemented. The computer program may be a high-level language program or an executable object program.
可选地，该计算机可读存储介质例如是存储器702。存储器702可以是易失性存储器或非易失性存储器，或者，存储器702可以同时包括易失性存储器和非易失性存储器。其中，非易失性存储器可以是只读存储器(read-only memory，ROM)、可编程只读存储器(programmable ROM，PROM)、可擦除可编程只读存储器(erasable PROM，EPROM)、电可擦除可编程只读存储器(electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(random access memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(static RAM，SRAM)、动态随机存取存储器(dynamic RAM，DRAM)、同步动态随机存取存储器(synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(double data rate SDRAM，DDR SDRAM)、增强型同步动态随机存取存储器(enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(synchlinkDRAM，SLDRAM)和直接内存总线随机存取存储器(direct rambus RAM，DR RAM)。Optionally, the computer-readable storage medium is, for example, memory 702. Memory 702 may be volatile memory or non-volatile memory, or memory 702 may include both volatile memory and non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically removable memory. Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of illustration, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synchlinkDRAM, SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).
本领域的技术人员可以清楚地了解到，为了描述的方便和简洁，上述描述的装置和设备的具体工作过程以及产生的技术效果，可以参考前述方法实施例中对应的过程和技术效果，在此不再赘述。Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes and technical effects produced by the above-described devices and equipment can be referred to the corresponding processes and technical effects in the foregoing method embodiments. Herein No longer.
在本申请所提供的几个实施例中，所揭露的系统、装置和方法，可以通过其它的方式实现。例如，以上所描述的方法实施例的一些特征可以忽略，或不执行。以上所描述的装置实施例仅仅是示意性的，单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，多个单元或组件可以结合或者可以集成到另一个系统。另外，各单元之间的耦合或各个组件之间的耦合可以是直接耦合，也可以是间接耦合，上述耦合包括电的、机械的或其它形式的连接。In several embodiments provided in this application, the disclosed systems, devices and methods can be implemented in other ways. For example, some features of the method embodiments described above may be omitted, or not performed. The device embodiments described above are only illustrative, and the division of units is only a logical function division. In actual implementation, there may be other division methods, and multiple units or components may be combined or integrated into another system. In addition, the coupling between units or the coupling between components may be direct coupling or indirect coupling, and the above-mentioned coupling includes electrical, mechanical or other forms of connection.
应理解，在本申请的各种实施例中，各过程的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本申请的实施例的实施过程构成任何限定。It should be understood that in the various embodiments of the present application, the size of the sequence numbers of each process does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be used in the embodiments of the present application. The implementation process constitutes any limitation.
另外，本文中术语“系统”和“网络”在本文中常被可互换使用。本文中的术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中字符“/”，一般表示前后关联对象是一种“或”的关系。Additionally, the terms "system" and "network" are often used interchangeably herein. The term "and/or" in this article is just an association relationship describing related objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, A and B exist simultaneously, alone There are three situations B. In addition, the character "/" in this article generally indicates that the related objects are an "or" relationship.
总之，以上所述仅为本申请技术方案的较佳实施例而已，并非用于限定本申请的保护范围。凡在本申请的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本申请的保护范围之内。In short, the above descriptions are only preferred embodiments of the technical solution of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of this application shall be included in the protection scope of this application.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110831693.9ACN115686182B (en) | 2021-07-22 | 2021-07-22 | Processing method of augmented reality video and electronic equipment |
| PCT/CN2022/089308WO2023000746A1 (en) | 2021-07-22 | 2022-04-26 | Augmented reality video processing method and electronic device |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110831693.9ACN115686182B (en) | 2021-07-22 | 2021-07-22 | Processing method of augmented reality video and electronic equipment |
| Publication Number | Publication Date |
|---|---|
| CN115686182A CN115686182A (en) | 2023-02-03 |
| CN115686182Btrue CN115686182B (en) | 2024-02-27 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110831693.9AActiveCN115686182B (en) | 2021-07-22 | 2021-07-22 | Processing method of augmented reality video and electronic equipment |
| Country | Link |
|---|---|
| CN (1) | CN115686182B (en) |
| WO (1) | WO2023000746A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI831665B (en)* | 2023-04-10 | 2024-02-01 | 晶達光電股份有限公司 | Display with usb type-c specification |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104102678A (en)* | 2013-04-15 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Method and device for realizing augmented reality |
| CN107835436A (en)* | 2017-09-25 | 2018-03-23 | 北京航空航天大学 | A kind of real-time virtual reality fusion live broadcast system and method based on WebGL |
| CN110827411A (en)* | 2018-08-09 | 2020-02-21 | 北京微播视界科技有限公司 | Self-adaptive environment augmented reality model display method, device, equipment and storage medium |
| CN111372098A (en)* | 2015-01-21 | 2020-07-03 | 微软技术许可有限责任公司 | User equipment, system, method and readable medium for shared scene grid data synchronization |
| CN112039937A (en)* | 2019-06-03 | 2020-12-04 | 华为技术有限公司 | Display method, position determination method and device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10679415B2 (en)* | 2017-07-05 | 2020-06-09 | Qualcomm Incorporated | Enhanced signaling of regions of interest in container files and video bitstreams |
| GB201805650D0 (en)* | 2018-04-05 | 2018-05-23 | Holome Tech Limited | Method and apparatus for generating augmented reality images |
| US10818093B2 (en)* | 2018-05-25 | 2020-10-27 | Tiff's Treats Holdings, Inc. | Apparatus, method, and system for presentation of multimedia content including augmented reality content |
| US10740960B2 (en)* | 2019-01-11 | 2020-08-11 | Microsoft Technology Licensing, Llc | Virtual object placement for augmented reality |
| CN110378990B (en)* | 2019-07-03 | 2023-01-10 | 北京悉见科技有限公司 | Augmented reality scene presentation method, device and storage medium |
| CN110879979B (en)* | 2019-11-13 | 2024-01-02 | 泉州师范学院 | Augmented reality system based on mobile terminal |
| CN112882576B (en)* | 2021-02-26 | 2023-07-25 | 北京市商汤科技开发有限公司 | AR interaction method and device, electronic equipment and storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104102678A (en)* | 2013-04-15 | 2014-10-15 | 腾讯科技（深圳）有限公司 | Method and device for realizing augmented reality |
| CN111372098A (en)* | 2015-01-21 | 2020-07-03 | 微软技术许可有限责任公司 | User equipment, system, method and readable medium for shared scene grid data synchronization |
| CN107835436A (en)* | 2017-09-25 | 2018-03-23 | 北京航空航天大学 | A kind of real-time virtual reality fusion live broadcast system and method based on WebGL |
| CN110827411A (en)* | 2018-08-09 | 2020-02-21 | 北京微播视界科技有限公司 | Self-adaptive environment augmented reality model display method, device, equipment and storage medium |
| CN112039937A (en)* | 2019-06-03 | 2020-12-04 | 华为技术有限公司 | Display method, position determination method and device |
| Publication number | Publication date |
|---|---|
| WO2023000746A1 (en) | 2023-01-26 |
| CN115686182A (en) | 2023-02-03 |
| Publication | Publication Date | Title |
|---|---|---|
| US11669242B2 (en) | Screenshot method and electronic device | |
| US11450322B2 (en) | Speech control method and electronic device | |
| CN114679537B (en) | A shooting method and terminal | |
| CN115866121B (en) | Application interface interaction method, electronic device and computer readable storage medium | |
| US12425701B2 (en) | Video synthesis method and apparatus, electronic device, and storage medium | |
| WO2020259452A1 (en) | Full-screen display method for mobile terminal, and apparatus | |
| CN114650363B (en) | Image display method and electronic device | |
| WO2020253719A1 (en) | Screen recording method and electronic device | |
| WO2020000448A1 (en) | Flexible screen display method and terminal | |
| WO2020029306A1 (en) | Image capture method and electronic device | |
| CN112637477A (en) | Image processing method and electronic equipment | |
| CN113641271B (en) | Application window management method, terminal device and computer readable storage medium | |
| CN112150499B (en) | Image processing method and related device | |
| CN114089932B (en) | Multi-screen display method, device, terminal equipment and storage medium | |
| CN113448658A (en) | Screen capture processing method, graphical user interface and terminal | |
| CN115967851A (en) | Quick photographing method, electronic device and computer-readable storage medium | |
| WO2021204103A1 (en) | Picture preview method, electronic device, and storage medium | |
| CN114756184A (en) | Collaborative display method, terminal device and computer-readable storage medium | |
| CN113542574A (en) | Shooting preview method, terminal, storage medium and electronic device under zoom | |
| CN116339569A (en) | Split screen display method, folding screen device and computer readable storage medium | |
| CN115686182B (en) | Processing method of augmented reality video and electronic equipment | |
| CN113495733A (en) | Theme pack installation method and device, electronic equipment and computer readable storage medium | |
| CN116522400B (en) | Image processing method and terminal equipment | |
| CN117435309A (en) | Task migration method, electronic equipment and system | |
| WO2025060749A1 (en) | Photographing method, and electronic device and storage medium |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address | Address after:Unit 3401, unit a, building 6, Shenye Zhongcheng, No. 8089, Hongli West Road, Donghai community, Xiangmihu street, Futian District, Shenzhen, Guangdong 518040 Patentee after:Honor Terminal Co.,Ltd. Country or region after:China Address before:3401, unit a, building 6, Shenye Zhongcheng, No. 8089, Hongli West Road, Donghai community, Xiangmihu street, Futian District, Shenzhen, Guangdong Patentee before:Honor Device Co.,Ltd. Country or region before:China | |
| CP03 | Change of name, title or address |