“本申请要求于2019年3月26日提交国家知识产权局、申请号为201910234517.X、发明名称为“3D光照估计方法及电子设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中”。"This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office on March 26, 2019, with the application number 201910234517.X and the title of the invention "3D illumination estimation method and electronic device", the entire contents of which are incorporated by reference in this application".
技术领域technical field
本申请涉及增强现实技术领域，尤其涉及多场景的3D光照估计方法及电子设备。The present application relates to the field of augmented reality technology, in particular to a multi-scene 3D illumination estimation method and electronic equipment.
背景技术Background technique
增强现实(augmented reality，AR)技术是在虚拟现实基础上发展起来的新技术，是通过计算机系统提供的信息增加用户对现实世界感知的技术，并将计算机生成的虚拟对象、场景或系统提示信息叠加到真实场景中，从而实现对现实的“增强”，是一种将真实世界信息和虚拟世界信息“无缝”集成的新技术。因此，如何将虚拟对象的渲染效果与环境协调，对AR产品的用户体验有重要意义。利用光照估计对虚拟对象进行渲染，是“无缝”AR的重要组成部分。Augmented reality (augmented reality, AR) technology is a new technology developed on the basis of virtual reality. It is a technology that increases users' perception of the real world through the information provided by the computer system, and uses computer-generated virtual objects, scenes or system prompt information. It is a new technology that "seamlessly" integrates real world information and virtual world information. Therefore, how to coordinate the rendering effect of virtual objects with the environment is of great significance to the user experience of AR products. Rendering virtual objects using lighting estimation is an important part of "seamless" AR.
目前已有厂商推出了相关解决方案，例如：Google推出的AR应用的软件工具包ARCore，和苹果推出的AR应用的软件工具包ARKit。但是通过上述软件，电子设备仅能获取到较少的信息，例如：环境光的色彩补偿信息和强度信息等，根据上述信息估计出的光照信息是一个平均光照信息(即每个方向的光照信息一样)，因此根据上述信息渲染的虚拟对象的光照分布与真实环境中的光照分布相差较大，增强现实效果不真实，用户体验差。At present, manufacturers have launched related solutions, such as ARCore, a software toolkit for AR applications launched by Google, and ARKit, a software toolkit for AR applications launched by Apple. However, through the above-mentioned software, electronic equipment can only obtain less information, such as: color compensation information and intensity information of ambient light, etc., and the illumination information estimated according to the above information is an average illumination information (that is, the illumination information of each direction Therefore, the light distribution of the virtual object rendered according to the above information is quite different from the light distribution in the real environment, the augmented reality effect is unreal, and the user experience is poor.
发明内容Contents of the invention
本申请实施例提供3D光照估计方法及电子设备，可以根据至少一个球谐系数估计出更接近真实场景的光照效果，以使得虚拟对象与真实场景更有效地融合，提升了用户体验。Embodiments of the present application provide a 3D illumination estimation method and an electronic device, which can estimate an illumination effect closer to a real scene according to at least one spherical harmonic coefficient, so that a virtual object and a real scene can be more effectively integrated, and user experience is improved.
为达到上述目的，本申请的实施例采用如下技术方案：In order to achieve the above object, the embodiments of the present application adopt the following technical solutions:
第一方面，本申请实施例提供一种3D光照估计方法，该方法包括：获取第一场景的图像信息；其中，该第一场景的图像信息包括该第一场景中像素点的颜色信息和该第一场景中像素点的深度信息；根据该第一场景的图像信息建立第一模拟场景；其中，该第一模拟场景包括至少两个光照探测球以及点云；根据该第一模拟场景包括的点云的信息估计光照探测球的球谐系数；其中，该点云的信息包括该点云中的点的位置信息和辐照度信息；根据该光照探测球的球谐系数估计该虚拟对象的位置的球谐系数。In a first aspect, an embodiment of the present application provides a 3D illumination estimation method, the method comprising: acquiring image information of a first scene; wherein the image information of the first scene includes color information of pixels in the first scene and the Depth information of pixels in the first scene; establish a first simulation scene according to the image information of the first scene; wherein, the first simulation scene includes at least two light detection balls and point clouds; according to the first simulation scene included Estimate the spherical harmonic coefficient of the light detection sphere from the information of the point cloud; wherein, the information of the point cloud includes the position information and irradiance information of the points in the point cloud; estimate the spherical harmonic coefficient of the virtual object according to the spherical harmonic coefficient of the light detection sphere The spherical harmonic coefficients of the position.
上述第一方面提供的技术方案，电子设备可以通过获取第一场景的图像信息，根据第一场景的图像信息建立第一模拟场景，并根据第一模拟场景包括的点云的信息估计光照探测球的球谐系数，再根据光照探测球的球谐系统估计虚拟对象的位置的球谐系数，其中，球谐系数可以包括更多的信息量，例如：光强的方向等。因此，电子设备可以估计出更接近真实场景的光照效果，以使得虚拟对象与真实场景更有效地融合，提升了用户体验。In the technical solution provided by the first aspect above, the electronic device can obtain the image information of the first scene, establish the first simulation scene according to the image information of the first scene, and estimate the illumination detection ball according to the point cloud information included in the first simulation scene. Then estimate the spherical harmonic coefficient of the position of the virtual object according to the spherical harmonic system of the light detection sphere, where the spherical harmonic coefficient can include more information, such as the direction of light intensity, etc. Therefore, the electronic device can estimate a lighting effect that is closer to the real scene, so that the virtual object can be more effectively integrated with the real scene, thereby improving user experience.
结合第一方面，在第一种可能的实现方式中，对于第一光照探测球，根据该第一模拟场景包括的点云的信息估计该第一光照探测球的球谐系数，包括：根据点云的信息获取该第一光照探测球的第一辐照度；根据该第一光照探测球的第一辐照度估计该第一光照探测球的球谐系数；该第一光照探测球为该第一模拟场景包括的任一光照探测球。基于第一方面的第一种可能的实现方式，电子设备可以根据点云的信息获取任一光照探测球的辐照度，根据任一光照探测球的辐照度估计任一光照探测球的球谐系数，以便电子设备根据光照探测球的球谐系数估计虚拟对象的位置的球谐系数。With reference to the first aspect, in a first possible implementation manner, for the first illumination probe sphere, estimating the spherical harmonic coefficient of the first illumination probe sphere according to point cloud information included in the first simulation scene includes: The cloud information acquires the first irradiance of the first illumination probe sphere; estimates the spherical harmonic coefficient of the first illumination probe sphere according to the first irradiance of the first illumination probe sphere; the first illumination probe sphere is the Any light probe sphere included in the first simulation scene. Based on the first possible implementation of the first aspect, the electronic device can obtain the irradiance of any light detection sphere according to the point cloud information, and estimate the irradiance of any light detection sphere according to the irradiance of any light detection sphere harmonic coefficients, so that the electronic device can estimate the spherical harmonic coefficients of the position of the virtual object according to the spherical harmonic coefficients of the light detection sphere.
结合第一方面的第一种可能的实现方式，在第二种可能的实现方式中，该根据点云的信息获取第一光照探测球的第一辐照度，包括：根据第一有效点的信息获取第一光照探测球的第一辐照度，其中，第一有效点是点云中在该第一光照探测球的可见范围内的点。基于第一方面的第二种可能的实现方式，电子设备可以根据第一有效点的信息获取第一光照探测球的第一辐照度，以便电子设备根据第一光照探测球的第一辐照度估计第一光照探测球的球谐系数。With reference to the first possible implementation of the first aspect, in the second possible implementation, the acquisition of the first irradiance of the first illumination probe sphere based on point cloud information includes: The information acquires the first irradiance of the first illumination probe sphere, wherein the first valid point is a point in the point cloud within the visible range of the first illumination probe sphere. Based on the second possible implementation of the first aspect, the electronic device can acquire the first irradiance of the first illumination detection sphere according to the information of the first effective point, so that the electronic device can obtain the first irradiance of the first illumination detection sphere according to the Estimate the spherical harmonic coefficients of the first illuminated probe sphere.
结合第一方面以及第一方面的各种可能的实现方式，在第三种可能的实现方式中，该根据该光照探测球的球谐系数估计该虚拟对象的位置的球谐系数，包括：对该光照探测球的球谐系数进行加权求和，得到该虚拟对象的位置的球谐系数。基于第一方面的第三种可能的实现方式，电子设备可以将光照探测球的球谐系数进行加权求和，得到虚拟对象的位置的球谐系数，以便电子设备根据虚拟对象位置的球谐系数估计真实场景中虚拟对象的光照效果。In combination with the first aspect and various possible implementations of the first aspect, in a third possible implementation, estimating the spherical harmonic coefficient of the position of the virtual object according to the spherical harmonic coefficient of the illumination detection sphere includes: The spherical harmonic coefficients of the illumination detection sphere are weighted and summed to obtain the spherical harmonic coefficients of the position of the virtual object. Based on the third possible implementation of the first aspect, the electronic device can weight and sum the spherical harmonic coefficients of the light detection sphere to obtain the spherical harmonic coefficient of the position of the virtual object, so that the electronic device can use the spherical harmonic coefficient of the position of the virtual object Estimate lighting effects of virtual objects in real scenes.
结合第一方面的第三种可能的实现方式，在第四种可能的实现方式中，该光照探测球与该虚拟对象间的距离小于或等于第一距离。基于第一方面的第四种可能的实现方式，电子设备可以将与虚拟对象间的距离小于或等于第一距离的光照探测球的球谐系数加权求和，得到虚拟对象的位置的球谐系数。With reference to the third possible implementation manner of the first aspect, in a fourth possible implementation manner, the distance between the illumination detection sphere and the virtual object is less than or equal to the first distance. Based on the fourth possible implementation of the first aspect, the electronic device may weight and sum the spherical harmonic coefficients of the light detection spheres whose distance to the virtual object is less than or equal to the first distance, to obtain the spherical harmonic coefficient of the position of the virtual object .
结合第一方面以及第一方面的各种可能的实现方式，在第五种可能的实现方式中，该方法还包括：根据用户输入获取虚拟对象的位置；或者，预设置虚拟对象的位置。基于第一方面的第五种可能的实现方式，电子设备可以根据用户输入获取虚拟对象的位置，电子设备还可以预设置虚拟对象的位置。With reference to the first aspect and various possible implementation manners of the first aspect, in a fifth possible implementation manner, the method further includes: acquiring the position of the virtual object according to user input; or, presetting the position of the virtual object. Based on a fifth possible implementation manner of the first aspect, the electronic device may acquire the position of the virtual object according to user input, and the electronic device may also preset the position of the virtual object.
第二方面，本申请实施例提供一种3D光照估计方法，该方法包括：获取该第一场景的图像信息；其中，该第一场景的图像信息包括该第一场景中像素点的颜色信息；根据该第一场景的图像信息获取该第一场景对应的天空图；其中，该第一场景对应的天空图用于指示该第一场景的光照分布；根据该天空图的信息估计该虚拟对象的球谐系数，其中，该天空图的信息包括天空图的球谐系数。In a second aspect, an embodiment of the present application provides a 3D illumination estimation method, the method comprising: acquiring image information of the first scene; wherein the image information of the first scene includes color information of pixels in the first scene; Acquiring a sky map corresponding to the first scene according to the image information of the first scene; wherein, the sky map corresponding to the first scene is used to indicate the illumination distribution of the first scene; estimating the virtual object according to the information of the sky map spherical harmonic coefficients, wherein the information of the sky map includes the spherical harmonic coefficients of the sky map.
上述第一方面提供的技术方案，电子设备可以通过获取第一场景的图像信息，根据第一场景的图像信息获取第一场景对应的天空图，根据天空图的信息估计虚拟对象的球谐系数，其中，球谐系数可以包括更多的信息量，例如：光强的方向等。因此，电子设备可以估计出更接近真实场景的光照效果，以使得虚拟对象与真实场景更有效地融合，提升了用户体验。In the technical solution provided by the first aspect above, the electronic device can acquire the image information of the first scene, obtain the sky map corresponding to the first scene according to the image information of the first scene, and estimate the spherical harmonic coefficient of the virtual object according to the information of the sky map, Wherein, the spherical harmonic coefficient may include more information, for example, the direction of light intensity and the like. Therefore, the electronic device can estimate a lighting effect that is closer to the real scene, so that the virtual object can be more effectively integrated with the real scene, thereby improving user experience.
结合第二方面，在第一种可能的实现方式中，该根据该天空图的信息估计该虚拟对象的球谐系数，包括：将该天空图的球谐系数作为该虚拟对象的球谐系数。基于第二方面的第一种可能的实现方式，电子设备可以将天空图的球谐系数作为虚拟对象的球谐系数，以便电子设备根据虚拟对象的球谐系数估计真实场景中虚拟对象的光照效果。With reference to the second aspect, in a first possible implementation manner, estimating the spherical harmonic coefficients of the virtual object according to the information of the sky map includes: using the spherical harmonic coefficients of the sky map as the spherical harmonic coefficients of the virtual object. Based on the first possible implementation of the second aspect, the electronic device can use the spherical harmonic coefficients of the sky map as the spherical harmonic coefficients of the virtual object, so that the electronic device can estimate the lighting effect of the virtual object in the real scene according to the spherical harmonic coefficients of the virtual object .
结合第二方面以及第二方面的第一种可能的实现方式，在第二种可能的实现方式中，该方法还包括：根据用户输入获取虚拟对象的位置；或者，预设置虚拟对象的位置。基于第二方面第二种可能的实现方式，电子设备可以根据用户输入获取虚拟对象的位置，或者，电子设备可以预设置虚拟对象的位置。With reference to the second aspect and the first possible implementation manner of the second aspect, in a second possible implementation manner, the method further includes: acquiring the position of the virtual object according to user input; or, presetting the position of the virtual object. Based on a second possible implementation manner of the second aspect, the electronic device may acquire the position of the virtual object according to user input, or the electronic device may preset the position of the virtual object.
第三方面，本申请实施例提供了一种电子设备，该电子设备具有实现上述第一方面所述的方法和功能。该功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。In a third aspect, an embodiment of the present application provides an electronic device, the electronic device has the method and function for realizing the above-mentioned first aspect. This function may be implemented by hardware, or may be implemented by executing corresponding software on the hardware. The hardware or software includes one or more modules corresponding to the above functions.
第四方面，本申请实施例提供了一种电子设备，该电子设备具有实现上述第二方面所述的方法和功能。该功能可以通过硬件实现，也可以通过硬件执行相应的软件实现。该硬件或软件包括一个或多个与上述功能相对应的模块。In a fourth aspect, the embodiment of the present application provides an electronic device, the electronic device has the method and function for realizing the above-mentioned second aspect. This function may be implemented by hardware, or may be implemented by executing corresponding software on the hardware. The hardware or software includes one or more modules corresponding to the above functions.
第五方面，本申请实施例提供了一种电子设备，包括：至少一个处理器以及至少一个存储器，该至少一个存储器与该至少一个处理器耦合；该至少一个存储器用于存储计算机程序，使得该计算机程序被该至少一个处理器执行时实现如第一方面及其各种可能的实现方式所述的3D光照估计方法。In a fifth aspect, the embodiment of the present application provides an electronic device, including: at least one processor and at least one memory, the at least one memory is coupled to the at least one processor; the at least one memory is used to store a computer program, so that the When the computer program is executed by the at least one processor, the 3D illumination estimation method described in the first aspect and various possible implementation manners thereof is realized.
第六方面，本申请实施例提供了一种电子设备，包括：至少一个处理器以及至少一个存储器，该至少一个存储器与该至少一个处理器耦合；该至少一个存储器用于存储计算机程序，使得该计算机程序被该至少一个处理器执行时实现如第二方面及其各种可能的实现方式所述的3D光照估计方法。In a sixth aspect, an embodiment of the present application provides an electronic device, including: at least one processor and at least one memory, the at least one memory is coupled to the at least one processor; the at least one memory is used to store a computer program, so that the When the computer program is executed by the at least one processor, the 3D illumination estimation method described in the second aspect and various possible implementation manners thereof is realized.
第七方面，本申请提供了一种系统芯片，该系统芯片可以应用在电子设备中，该系统芯片包括：至少一个处理器，涉及的程序指令在该至少一个处理器中执行，以实现根据第一方面的方法及其任一设计中的电子设备的功能。可选的，该系统芯片还可以包括至少一个存储器，该存储器存储有涉及的程序指令。In a seventh aspect, the present application provides a system chip, which can be applied in electronic equipment, and the system chip includes: at least one processor, and related program instructions are executed in the at least one processor, so as to implement the system chip according to the first A method and function of an electronic device in any design thereof. Optionally, the system chip may further include at least one memory, and the memory stores related program instructions.
第八方面，本申请提供了一种系统芯片，该系统芯片可以应用在电子设备中，该系统芯片包括：至少一个处理器，涉及的程序指令在该至少一个处理器中执行，以实现根据第一方面的方法及其任一设计中的电子设备的功能。可选的，该系统芯片还可以包括至少一个存储器，该存储器存储有涉及的程序指令。In an eighth aspect, the present application provides a system chip, which can be applied in electronic equipment. The system chip includes: at least one processor, and related program instructions are executed in the at least one processor, so as to implement the system chip according to the first aspect. A method and function of an electronic device in any design thereof. Optionally, the system chip may further include at least one memory, and the memory stores related program instructions.
第九方面，本申请实施例提供了一种计算机可读存储介质，如计算机非瞬态的可读存储介质。其上储存有计算机程序，当该计算机程序在计算机上运行时，使得计算机执行上述第一方面的任一种可能的方法。例如，该计算机可以是至少一个存储节点。In a ninth aspect, the embodiment of the present application provides a computer-readable storage medium, such as a non-transitory computer-readable storage medium. A computer program is stored thereon, and when the computer program is run on the computer, the computer is made to execute any possible method of the first aspect above. For example, the computer can be at least one storage node.
第十方面，本申请实施例提供了一种计算机可读存储介质，如计算机非瞬态的可读存储介质。其上储存有计算机程序，当该计算机程序在计算机上运行时，使得计算机执行上述第二方面的任一种可能的方法。例如，该计算机可以是至少一个存储节点。In a tenth aspect, the embodiment of the present application provides a computer-readable storage medium, such as a non-transitory computer-readable storage medium. A computer program is stored thereon, and when the computer program is run on the computer, the computer is made to execute any possible method of the second aspect above. For example, the computer can be at least one storage node.
第十一方面，本申请实施例提供了一种计算机程序产品，当其在计算机上运行时，使得第一方面提供的任一方法被执行。例如，该计算机可以是至少一个存储节点。In an eleventh aspect, the embodiment of the present application provides a computer program product, which enables any method provided in the first aspect to be executed when it is run on a computer. For example, the computer can be at least one storage node.
第十二方面，本申请实施例提供了一种计算机程序产品，当其在计算机上运行时，使得第二方面提供的任一方法被执行。例如，该计算机可以是至少一个存储节点。In a twelfth aspect, the embodiment of the present application provides a computer program product, which enables any method provided in the second aspect to be executed when it is run on a computer. For example, the computer can be at least one storage node.
可以理解的，上述提供的任一种电子设备或系统芯片或计算机存储介质或计算机程序产品等均用于执行上文所提供的对应的方法，因此，其所能达到的有益效果可参考对应的方法中的有益效果，此处不再赘述。It can be understood that any electronic device or system chip or computer storage medium or computer program product provided above is used to execute the corresponding method provided above, therefore, the beneficial effects that it can achieve can refer to the corresponding The beneficial effects in the method will not be repeated here.
附图说明Description of drawings
图1为本申请实施例提供的电子设备的硬件结构示意图；FIG. 1 is a schematic diagram of a hardware structure of an electronic device provided in an embodiment of the present application;
图2(a)为本申请实施例提供的3D光照估计方法的原理示意图一；Fig. 2 (a) is a schematic diagram of the principle of the 3D illumination estimation method provided by the embodiment of the present application;
图2(b)为本申请实施例提供的3D光照估计方法的原理示意图二；Fig. 2(b) is a schematic diagram 2 of the principle of the 3D illumination estimation method provided by the embodiment of the present application;
图2(c)为本申请实施例提供的3D光照估计方法的原理示意图三；Fig. 2(c) is a schematic diagram 3 of the principle of the 3D illumination estimation method provided by the embodiment of the present application;
图3(a)为本申请实施例提供的显示界面实例示意图一；Figure 3(a) is a schematic diagram of a display interface example 1 provided by the embodiment of the present application;
图3(b)为本申请实施例提供的显示界面实例示意图二；Figure 3(b) is a second schematic diagram of a display interface example provided by the embodiment of the present application;
图3(c)为本申请实施例提供的显示界面实例示意图三；Figure 3(c) is a schematic diagram of a third example of a display interface provided by the embodiment of the present application;
图3(d)为本申请实施例提供的显示界面实例示意图四；Fig. 3(d) is a schematic diagram 4 of a display interface example provided by the embodiment of the present application;
图4(a)为本申请实施例提供的场景类型示意图一；Figure 4(a) is a schematic diagram of the scene type provided by the embodiment of the present application;
图4(b)为本申请实施例提供的场景类型示意图二；Figure 4(b) is a second schematic diagram of the scene type provided by the embodiment of the present application;
图4(c)为本申请实施例提供的场景类型示意图三；Figure 4(c) is a schematic diagram of the third scene type provided by the embodiment of the present application;
图5为本申请实施例提供的3D光照估计方法流程示意图一；FIG. 5 is a first schematic flow diagram of the 3D illumination estimation method provided by the embodiment of the present application;
图6(a)为本申请实施例提供的显示界面实例示意图五；Figure 6(a) is a schematic diagram five of an example of a display interface provided by the embodiment of the present application;
图6(b)为本申请实施例提供的显示界面实例示意图六；Figure 6(b) is a schematic diagram six of an example of a display interface provided by the embodiment of the present application;
图7为本申请实施例提供的光照探测球的示意图一；Fig. 7 is a schematic diagram 1 of the illumination detection ball provided by the embodiment of the present application;
图8为本申请实施例提供的光照探测球的示意图二；Fig. 8 is a second schematic diagram of the illumination detection ball provided by the embodiment of the present application;
图9为本申请实施例提供的显示界面实例示意图七；FIG. 9 is a schematic diagram VII of a display interface example provided by the embodiment of the present application;
图10为本申请实施例提供的3D光照估计方法流程示意图二；FIG. 10 is the second schematic flow diagram of the 3D illumination estimation method provided by the embodiment of the present application;
图11为本申请实施例提供的天空图示意图；FIG. 11 is a schematic diagram of a sky map provided in an embodiment of the present application;
图12为本申请实施例提供的3D光照估计方法流程示意图三；FIG. 12 is the third schematic flow chart of the 3D illumination estimation method provided by the embodiment of the present application;
图13为本申请实施例提供的电子设备的结构示意图一；FIG. 13 is a first structural schematic diagram of an electronic device provided by an embodiment of the present application;
图14为本申请实施例提供的电子设备的结构示意图二。FIG. 14 is a second structural schematic diagram of the electronic device provided by the embodiment of the present application.
具体实施方式Detailed ways
下面结合本申请实施例中的附图对本申请实施例进行描述。本申请实施例的实施方式部分使用的术语仅用于对本申请的具体实施例进行解释，而非旨在限定本申请。Embodiments of the present application are described below with reference to the drawings in the embodiments of the present application. The terms used in the implementation of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
对本申请实施例涉及的电子设备进行介绍。本申请实施例提供的电子设备可以为具有增强现实(augmented reality，AR)的任一设备，如：可以为便携式计算机(如手机)、笔记本电脑、个人计算机(personal computer，PC)、可穿戴电子设备(如智能手表)、平板电脑、AR设备等。The electronic devices involved in the embodiments of the present application are introduced. The electronic device provided by the embodiment of the present application can be any device with augmented reality (augmented reality, AR), such as: it can be a portable computer (such as a mobile phone), a notebook computer, a personal computer (personal computer, PC), a wearable electronic device Devices (such as smart watches), tablets, AR devices, etc.
请参阅图1，图1是本申请实施例提供的一种电子设备的结构示意图。图1中的电子设备100具体可以包括：处理器110，外部存储器接口120，内部存储器121，通用串行总线(universal serial bus，USB)接口130，充电管理模块140，电源管理模块141，电池142，天线1，天线2，移动通信模块150，无线通信模块160，音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，传感器模块180，按键190，马达191，指示器192，摄像头193，显示屏194，以及用户标识模块(subscriber identification module，SIM)卡接口195等。其中传感器模块180可以包括深度传感器180A，压力传感器180B，陀螺仪传感器180C，气压传感器180D，磁传感器180E，加速度传感器180F，距离传感器180G，接近光传感器180H，指纹传感器180J，温度传感器180K，触摸传感器180L，环境光传感器180M，骨传导传感器180N等。Please refer to FIG. 1 . FIG. 1 is a schematic structural diagram of an electronic device provided by an embodiment of the present application. The electronic device 100 in FIG. 1 may specifically include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (universal serial bus, USB) interface 130, a charging management module 140, a power management module 141, and a battery 142 , antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, audio module 170, speaker 170A, receiver 170B, microphone 170C, earphone jack 170D, sensor module 180, button 190, motor 191, indicator 192, camera 193 , a display screen 194, and a subscriber identification module (subscriber identification module, SIM) card interface 195, etc. The sensor module 180 may include a depth sensor 180A, a pressure sensor 180B, a gyroscope sensor 180C, an air pressure sensor 180D, a magnetic sensor 180E, an acceleration sensor 180F, a distance sensor 180G, a proximity sensor 180H, a fingerprint sensor 180J, a temperature sensor 180K, and a touch sensor. 180L, ambient light sensor 180M, bone conduction sensor 180N, etc.
可以理解的是，本申请实施例示意的结构并不构成对电子设备100的具体限定。在本申请另一些实施例中，电子设备100可以包括比图示更多或更少的部件，或者组合某些部件，或者拆分某些部件，或者不同的部件布置。图示的部件可以以硬件，软件或软件和硬件的组合实现。It can be understood that, the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100 . In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine certain components, or separate certain components, or arrange different components. The illustrated components can be realized in hardware, software or a combination of software and hardware.
处理器110可以包括一个或多个处理单元，例如：处理器110可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processingunit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processing unit (graphics processing unit, GPU), an image signal processor ( image signal processor (ISP), controller, video codec, digital signal processor (digital signal processor, DSP), baseband processor, and/or neural network processor (neural-network processing unit, NPU), etc. Wherein, different processing units may be independent devices, or may be integrated in one or more processors.
控制器可以根据指令操作码和时序信号，产生操作控制信号，完成取指令和执行指令的控制。The controller can generate an operation control signal according to the instruction opcode and timing signal, and complete the control of fetching and executing the instruction.
处理器110中还可以设置存储器，用于存储指令和数据。存储器可以存储用于实现六个模块化功能的指令：检测指令、连接指令、信息管理指令、分析指令、数据传输指令和通知指令，并由处理器110来控制执行。在一些实施例中，处理器110中的存储器为高速缓冲存储器。该存储器可以保存处理器110刚用过或循环使用的指令或数据。如果处理器110需要再次使用该指令或数据，可从所述存储器中直接调用。避免了重复存取，减少了处理器110的等待时间，因而提高了系统的效率。A memory may also be provided in the processor 110 for storing instructions and data. The memory can store instructions for realizing six modular functions: detection instruction, connection instruction, information management instruction, analysis instruction, data transmission instruction and notification instruction, and the execution is controlled by the processor 110 . In some embodiments, the memory in processor 110 is a cache memory. The memory may hold instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can be called directly from the memory. Repeated access is avoided, and the waiting time of the processor 110 is reduced, thereby improving the efficiency of the system.
在一些实施例中，处理器110可以包括一个或多个接口。接口可以包括集成电路(inter-integrated circuit，I2C)接口，集成电路内置音频(inter-integrated circuitsound，I2S)接口，脉冲编码调制(pulse code modulation，PCM)接口，通用异步收发传输器(universal asynchronous receiver/transmitter，UART)接口，移动产业处理器接口(mobile industry processor interface，MIPI)，通用输入输出(general-purposeinput/output，GPIO)接口，用户标识模块(subscriber identity module，SIM)接口，和/或通用串行总线(universal serial bus，USB)接口等。In some embodiments, processor 110 may include one or more interfaces. The interface may include an integrated circuit (inter-integrated circuit, I2C) interface, an integrated circuit built-in audio (inter-integrated circuitsound, I2S) interface, a pulse code modulation (pulse code modulation, PCM) interface, a universal asynchronous receiver (universal asynchronous receiver) /transmitter, UART) interface, mobile industry processor interface (mobile industry processor interface, MIPI), general-purpose input and output (general-purpose input/output, GPIO) interface, subscriber identity module (subscriber identity module, SIM) interface, and/or A universal serial bus (universal serial bus, USB) interface, etc.
I2C接口是一种双向同步串行总线，包括一根串行数据线(serial data line，SDA)和一根串行时钟线(derail clock line，SCL)。在一些实施例中，处理器110可以包含多组I2C总线。处理器110可以通过不同的I2C总线接口分别耦合触摸传感器180K，充电器，闪光灯，摄像头193等。例如：处理器110可以通过I2C接口耦合触摸传感器180K，使处理器110与触摸传感器180K通过I2C总线接口通信，实现电子设备100的触摸功能。The I2C interface is a bidirectional synchronous serial bus, including a serial data line (serial data line, SDA) and a serial clock line (derail clock line, SCL). In some embodiments, processor 110 may include multiple sets of I2C buses. The processor 110 can be respectively coupled to the touch sensor 180K, the charger, the flashlight, the camera 193 and the like through different I2C bus interfaces. For example, the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 and the touch sensor 180K communicate through the I2C bus interface to realize the touch function of the electronic device 100 .
I2S接口可以用于音频通信。在一些实施例中，处理器110可以包含多组I2S总线。处理器110可以通过I2S总线与音频模块170耦合，实现处理器110与音频模块170之间的通信。在一些实施例中，音频模块170可以通过I2S接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。The I2S interface can be used for audio communication. In some embodiments, processor 110 may include multiple sets of I2S buses. The processor 110 may be coupled to the audio module 170 through an I2S bus to implement communication between the processor 110 and the audio module 170 . In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the I2S interface, so as to realize the function of answering calls through the Bluetooth headset.
PCM接口也可以用于音频通信，将模拟信号抽样，量化和编码。在一些实施例中，音频模块170与无线通信模块160可以通过PCM总线接口耦合。在一些实施例中，音频模块170也可以通过PCM接口向无线通信模块160传递音频信号，实现通过蓝牙耳机接听电话的功能。所述I2S接口和所述PCM接口都可以用于音频通信。The PCM interface can also be used for audio communication, sampling, quantizing and encoding the analog signal. In some embodiments, the audio module 170 and the wireless communication module 160 may be coupled through a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 through the PCM interface, so as to realize the function of answering calls through the Bluetooth headset. Both the I2S interface and the PCM interface can be used for audio communication.
UART接口是一种通用串行数据总线，用于异步通信。该总线可以为双向通信总线。它将要传输的数据在串行通信与并行通信之间转换。在一些实施例中，UART接口通常被用于连接处理器110与无线通信模块160。例如：处理器110通过UART接口与无线通信模块160中的蓝牙模块通信，实现蓝牙功能。在一些实施例中，音频模块170可以通过UART接口向无线通信模块160传递音频信号，实现通过蓝牙耳机播放音乐的功能。The UART interface is a universal serial data bus used for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, a UART interface is generally used to connect the processor 110 and the wireless communication module 160 . For example: the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to realize the Bluetooth function. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 160 through the UART interface, so as to realize the function of playing music through the Bluetooth headset.
MIPI接口可以被用于连接处理器110与显示屏194，摄像头193等外围器件。MIPI接口包括摄像头串行接口(camera serial interface，CSI)，显示屏串行接口(displayserial interface，DSI)等。在一些实施例中，处理器110和摄像头193通过CSI接口通信，实现电子设备100的拍摄功能。处理器110和显示屏194通过DSI接口通信，实现电子设备100的显示功能。The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193 . The MIPI interface includes a camera serial interface (camera serial interface, CSI), a display serial interface (displayserial interface, DSI), and the like. In some embodiments, the processor 110 communicates with the camera 193 through the CSI interface to realize the shooting function of the electronic device 100 . The processor 110 communicates with the display screen 194 through the DSI interface to realize the display function of the electronic device 100 .
GPIO接口可以通过软件配置。GPIO接口可以被配置为控制信号，也可被配置为数据信号。在一些实施例中，GPIO接口可以用于连接处理器110与摄像头193，显示屏194，无线通信模块160，音频模块170，传感器模块180等。GPIO接口还可以被配置为I2C接口，I2S接口，UART接口，MIPI接口等。The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193 , the display screen 194 , the wireless communication module 160 , the audio module 170 , the sensor module 180 and so on. The GPIO interface can also be configured as an I2C interface, I2S interface, UART interface, MIPI interface, etc.
USB接口130是符合USB标准规范的接口，具体可以是MiniUSB接口，MicroUSB接口，USBTypeC接口等。USB接口130可以用于连接充电器为电子设备100充电，也可以用于电子设备100与外围设备之间传输数据。也可以用于连接耳机，通过耳机播放音频。该接口还可以用于连接其他电子设备，例如AR设备等。The USB interface 130 is an interface conforming to the USB standard specification, specifically, it may be a MiniUSB interface, a MicroUSB interface, a USB Type C interface, and the like. The USB interface 130 can be used to connect a charger to charge the electronic device 100 , and can also be used to transmit data between the electronic device 100 and peripheral devices. It can also be used to connect headphones and play audio through them. This interface can also be used to connect other electronic devices, such as AR devices.
可以理解的是，本发明实施例示意的各模块间的接口连接关系，只是示意性说明，并不构成对电子设备100的结构限定。在本申请另一些实施例中，电子设备100也可以采用上述实施例中不同的接口连接方式，或多种接口连接方式的组合。It can be understood that the interface connection relationship between the modules shown in the embodiment of the present invention is only a schematic illustration, and does not constitute a structural limitation of the electronic device 100 . In other embodiments of the present application, the electronic device 100 may also adopt different interface connection manners in the foregoing embodiments, or a combination of multiple interface connection manners.
充电管理模块140用于从充电器接收充电输入。其中，充电器可以是无线充电器，也可以是有线充电器。在一些有线充电的实施例中，充电管理模块140可以通过USB接口130接收有线充电器的充电输入。在一些无线充电的实施例中，充电管理模块140可以通过电子设备100的无线充电线圈接收无线充电输入。充电管理模块140为电池142充电的同时，还可以通过电源管理模块141为电子设备供电。The charging management module 140 is configured to receive a charging input from a charger. Wherein, the charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger through the USB interface 130 . In some wireless charging embodiments, the charging management module 140 may receive a wireless charging input through a wireless charging coil of the electronic device 100 . While the charging management module 140 is charging the battery 142 , it can also provide power for electronic devices through the power management module 141 .
电源管理模块141用于连接电池142，充电管理模块140与处理器110。电源管理模块141接收电池142和/或充电管理模块140的输入，为处理器110，内部存储器121，显示屏194，摄像头193，和无线通信模块160等供电。电源管理模块141还可以用于监测电池容量，电池循环次数，电池健康状态(漏电，阻抗)等参数。在其他一些实施例中，电源管理模块141也可以设置于处理器110中。在另一些实施例中，电源管理模块141和充电管理模块140也可以设置于同一个器件中。The power management module 141 is used for connecting the battery 142 , the charging management module 140 and the processor 110 . The power management module 141 receives the input from the battery 142 and/or the charging management module 140 to provide power for the processor 110 , the internal memory 121 , the display screen 194 , the camera 193 , and the wireless communication module 160 . The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle times, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 may also be disposed in the processor 110 . In some other embodiments, the power management module 141 and the charging management module 140 may also be set in the same device.
电子设备100的无线通信功能可以通过天线1，天线2，移动通信模块150，无线通信模块160，调制解调处理器以及基带处理器等实现。The wireless communication function of the electronic device 100 can be realized by the antenna 1 , the antenna 2 , the mobile communication module 150 , the wireless communication module 160 , a modem processor, a baseband processor, and the like.
天线1和天线2用于发射和接收电磁波信号。电子设备100中的每个天线可用于覆盖单个或多个通信频带。不同的天线还可以复用，以提高天线的利用率。例如：可以将天线1复用为无线局域网的分集天线。在另外一些实施例中，天线可以和调谐开关结合使用。Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 may be used to cover single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: Antenna 1 can be multiplexed as a diversity antenna of a wireless local area network. In other embodiments, the antenna may be used in conjunction with a tuning switch.
移动通信模块150可以提供应用在电子设备100上的包括2G/3G/4G/5G等无线通信的解决方案。移动通信模块150可以包括至少一个滤波器，开关，功率放大器，低噪声放大器(low noise amplifier，LNA)等。移动通信模块150可以由天线1接收电磁波，并对接收的电磁波进行滤波，放大等处理，传送至调制解调处理器进行解调。移动通信模块150还可以对经调制解调处理器调制后的信号放大，经天线1转为电磁波辐射出去。在一些实施例中，移动通信模块150的至少部分功能模块可以被设置于处理器110中。在一些实施例中，移动通信模块150的至少部分功能模块可以与处理器110的至少部分模块被设置在同一个器件中。The mobile communication module 150 can provide wireless communication solutions including 2G/3G/4G/5G applied on the electronic device 100 . The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (low noise amplifier, LNA) and the like. The mobile communication module 150 can receive electromagnetic waves through the antenna 1, filter and amplify the received electromagnetic waves, and send them to the modem processor for demodulation. The mobile communication module 150 can also amplify the signals modulated by the modem processor, and convert them into electromagnetic waves through the antenna 1 for radiation. In some embodiments, at least part of the functional modules of the mobile communication module 150 may be set in the processor 110 . In some embodiments, at least part of the functional modules of the mobile communication module 150 and at least part of the modules of the processor 110 may be set in the same device.
调制解调处理器可以包括调制器和解调器。其中，调制器用于将待发送的低频基带信号调制成中高频信号。解调器用于将接收的电磁波信号解调为低频基带信号。随后解调器将解调得到的低频基带信号传送至基带处理器处理。低频基带信号经基带处理器处理后，被传递给应用处理器。应用处理器通过音频设备(不限于扬声器170A，受话器170B等)输出声音信号，或通过显示屏194显示图像或视频。在一些实施例中，调制解调处理器可以是独立的器件。在另一些实施例中，调制解调处理器可以独立于处理器110，与移动通信模块150或其他功能模块设置在同一个器件中。A modem processor may include a modulator and a demodulator. Wherein, the modulator is used for modulating the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low frequency baseband signal. Then the demodulator sends the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal is passed to the application processor after being processed by the baseband processor. The application processor outputs sound signals through audio equipment (not limited to speaker 170A, receiver 170B, etc.), or displays images or videos through display screen 194 . In some embodiments, the modem processor may be a stand-alone device. In some other embodiments, the modem processor may be independent from the processor 110, and be set in the same device as the mobile communication module 150 or other functional modules.
无线通信模块160可以提供应用在电子设备100上的包括无线局域网(wirelesslocal area networks，WLAN)(如无线保真(wireless fidelity，Wi-Fi)网络)，蓝牙(bluetooth，BT)，全球导航卫星系统(global navigation satellite system，GNSS)，调频(frequency modulation，FM)，近距离无线通信技术(near field communication，NFC)，红外技术(infrared，IR)等无线通信的解决方案。无线通信模块160可以是集成至少一个通信处理模块的一个或多个器件。无线通信模块160经由天线2接收电磁波，将电磁波信号调频以及滤波处理，将处理后的信号发送到处理器110。无线通信模块160还可以从处理器110接收待发送的信号，对其进行调频，放大，经天线2转为电磁波辐射出去。The wireless communication module 160 can provide applications on the electronic device 100 including wireless local area networks (wireless local area networks, WLAN) (such as wireless fidelity (wireless fidelity, Wi-Fi) network), bluetooth (bluetooth, BT), global navigation satellite system (global navigation satellite system, GNSS), frequency modulation (frequency modulation, FM), near field communication technology (near field communication, NFC), infrared technology (infrared, IR) and other wireless communication solutions. The wireless communication module 160 may be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2 , frequency-modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110 . The wireless communication module 160 can also receive the signal to be sent from the processor 110 , frequency-modulate it, amplify it, and convert it into electromagnetic waves through the antenna 2 for radiation.
在一些实施例中，电子设备100的天线1和移动通信模块150耦合，天线2和无线通信模块160耦合，使得电子设备100可以通过无线通信技术与网络以及其他设备通信。所述无线通信技术可以包括全球移动通讯系统(global system for mobile communications，GSM)，通用分组无线服务(general packet radio service，GPRS)，码分多址接入(codedivision multiple access，CDMA)，宽带码分多址(wideband code division multipleaccess，WCDMA)，时分码分多址(time-division code division multiple access，TD-SCDMA)，长期演进(long term evolution，LTE)，BT，GNSS，WLAN，NFC，FM，和/或IR技术等。所述GNSS可以包括全球卫星定位系统(global positioning system，GPS)，全球导航卫星系统(global navigation satellite system，GLONASS)，北斗卫星导航系统(beidounavigation satellite system，BDS)，准天顶卫星系统(quasi-zenith satellitesystem，QZSS)和/或星基增强系统(satellite based augmentation systems，SBAS)。In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (general packet radio service, GPRS), code division multiple access (codedivision multiple access, CDMA), wideband code Wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM , and/or IR technology, etc. The GNSS may include a global positioning system (global positioning system, GPS), a global navigation satellite system (global navigation satellite system, GLONASS), a Beidou satellite navigation system (beidounavigation satellite system, BDS), a quasi-zenith satellite system (quasi- zenith satellite system (QZSS) and/or satellite based augmentation systems (SBAS).
电子设备100通过GPU，显示屏194，以及应用处理器等实现显示功能。GPU为图像处理的微处理器，连接显示屏194和应用处理器。GPU用于执行数学和几何计算，用于图形渲染。处理器110可包括一个或多个GPU，其执行程序指令以生成或改变显示信息。The electronic device 100 realizes the display function through the GPU, the display screen 194 , and the application processor. The GPU is a microprocessor for image processing, and is connected to the display screen 194 and the application processor. GPUs are used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
显示屏194用于显示图像，视频等。显示屏194包括显示面板。显示面板可以采用液晶显示屏(liquid crystal display，LCD)，有机发光二极管(organic light-emittingdiode，OLED)，有源矩阵有机发光二极体或主动矩阵有机发光二极体(active-matrixorganic light emitting diode，AMOLED)，柔性发光二极管(flexlight-emitting diode，FLED)，Miniled，MicroLed，Micro-oLed，量子点发光二极管(quantum dot light emittingdiodes，QLED)等。在一些实施例中，电子设备100可以包括1个或N个显示屏194，N为大于1的正整数。The display screen 194 is used to display images, videos and the like. The display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (active-matrix organic light emitting diode). , AMOLED), flexible light-emitting diode (flexlight-emitting diode, FLED), Miniled, MicroLed, Micro-oLed, quantum dot light-emitting diodes (quantum dot light emittingdiodes, QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194 , where N is a positive integer greater than 1.
电子设备100可以通过ISP，摄像头193，视频编解码器，GPU，显示屏194以及应用处理器等实现拍摄功能。The electronic device 100 can realize the shooting function through the ISP, the camera 193 , the video codec, the GPU, the display screen 194 and the application processor.
ISP用于处理摄像头193反馈的数据。例如，拍照时，打开快门，光线通过镜头被传递到摄像头感光元件上，光信号转换为电信号，摄像头感光元件将所述电信号传递给ISP处理，转化为肉眼可见的图像。ISP还可以对图像的噪点，亮度，肤色进行算法优化。ISP还可以对拍摄场景的曝光，色温等参数优化。在一些实施例中，ISP可以设置在摄像头193中。The ISP is used for processing the data fed back by the camera 193 . For example, when taking a picture, open the shutter, the light is transmitted to the photosensitive element of the camera through the lens, and the light signal is converted into an electrical signal, and the photosensitive element of the camera transmits the electrical signal to the ISP for processing, and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on image noise, brightness, and skin color. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, the ISP may be located in the camera 193 .
摄像头193用于捕获静态图像或视频。物体通过镜头生成光学图像投射到感光元件。感光元件可以是电荷耦合器件(charge coupled device，CCD)或互补金属氧化物半导体(complementary metal-oxide-semiconductor，CMOS)光电晶体管。感光元件把光信号转换成电信号，之后将电信号传递给ISP转换成数字图像信号。ISP将数字图像信号输出到DSP加工处理。DSP将数字图像信号转换成标准的RGB，YUV等格式的图像信号。在一些实施例中，电子设备100可以包括1个或N个摄像头193，N为大于1的正整数，若电子设备100包括N个摄像头，N个摄像头中有一个是主摄像头。Camera 193 is used to capture still images or video. The object generates an optical image through the lens and projects it to the photosensitive element. The photosensitive element may be a charge coupled device (charge coupled device, CCD) or a complementary metal-oxide-semiconductor (complementary metal-oxide-semiconductor, CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. DSP converts digital image signals into standard RGB, YUV and other image signals. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. If the electronic device 100 includes N cameras, one of the N cameras is the main camera.
数字信号处理器用于处理数字信号，除了可以处理数字图像信号，还可以处理其他数字信号。例如，当电子设备100在频点选择时，数字信号处理器用于对频点能量进行傅里叶变换等。Digital signal processors are used to process digital signals. In addition to digital image signals, they can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the energy of the frequency point.
视频编解码器用于对数字视频压缩或解压缩。电子设备100可以支持一种或多种视频编解码器。这样，电子设备100可以播放或录制多种编码格式的视频，例如：动态图像专家组(moving picture experts group，MPEG)1，MPEG2，MPEG3，MPEG4等。Video codecs are used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in various encoding formats, for example: moving picture experts group (moving picture experts group, MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
NPU为神经网络(neural-network，NN)计算处理器，通过借鉴生物神经网络结构，例如借鉴人脑神经元之间传递模式，对输入信息快速处理，还可以不断的自学习。通过NPU可以实现电子设备100的智能认知等应用，例如：图像识别，人脸识别，语音识别，文本理解等。The NPU is a neural-network (NN) computing processor. By referring to the structure of biological neural networks, such as the transfer mode between neurons in the human brain, it can quickly process input information and continuously learn by itself. Applications such as intelligent cognition of the electronic device 100 can be realized through the NPU, such as image recognition, face recognition, speech recognition, text understanding, and the like.
外部存储器接口120可以用于连接外部存储卡，例如Micro SD卡，实现扩展电子设备100的存储能力。外部存储卡通过外部存储器接口120与处理器110通信，实现数据存储功能。例如将音乐，视频等文件保存在外部存储卡中。The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100 . The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function. Such as saving music, video and other files in the external memory card.
内部存储器121可以用于存储计算机可执行程序代码，所述可执行程序代码包括指令。内部存储器121可以包括存储程序区和存储数据区。其中，存储程序区可存储操作系统，至少一个功能所需的应用程序(比如声音播放功能，图像播放功能等)等。存储数据区可存储电子设备100使用过程中所创建的数据(比如音频数据，电话本等)等。此外，内部存储器121可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件，闪存器件，通用闪存存储器(universal flash storage，UFS)等。处理器110通过运行存储在内部存储器121的指令，和/或存储在设置于处理器中的存储器的指令，执行电子设备100的各种功能应用以及数据处理。The internal memory 121 may be used to store computer-executable program codes including instructions. The internal memory 121 may include an area for storing programs and an area for storing data. Wherein, the stored program area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.) and the like. The storage data area can store data created during the use of the electronic device 100 (such as audio data, phonebook, etc.) and the like. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one magnetic disk storage device, flash memory device, universal flash storage (universal flash storage, UFS) and the like. The processor 110 executes various functional applications and data processing of the electronic device 100 by executing instructions stored in the internal memory 121 and/or instructions stored in a memory provided in the processor.
电子设备100可以通过音频模块170，扬声器170A，受话器170B，麦克风170C，耳机接口170D，以及应用处理器等实现音频功能。例如音乐播放，录音等。The electronic device 100 can implement audio functions through the audio module 170 , the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor. Such as music playback, recording, etc.
音频模块170用于将数字音频信息转换成模拟音频信号输出，也用于将模拟音频输入转换为数字音频信号。音频模块170还可以用于对音频信号编码和解码。在一些实施例中，音频模块170可以设置于处理器110中，或将音频模块170的部分功能模块设置于处理器110中。The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signal. The audio module 170 may also be used to encode and decode audio signals. In some embodiments, the audio module 170 may be set in the processor 110 , or some functional modules of the audio module 170 may be set in the processor 110 .
扬声器170A，也称“喇叭”，用于将音频电信号转换为声音信号。电子设备100可以通过扬声器170A收听音乐，或收听免提通话。Speaker 170A, also referred to as a "horn", is used to convert audio electrical signals into sound signals. Electronic device 100 can listen to music through speaker 170A, or listen to hands-free calls.
受话器170B，也称“听筒”，用于将音频电信号转换成声音信号。当电子设备100接听电话或语音信息时，可以通过将受话器170B靠近人耳接听语音。Receiver 170B, also called "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the receiver 170B can be placed close to the human ear to receive the voice.
麦克风170C，也称“话筒”，“传声器”，用于将声音信号转换为电信号。当拨打电话或发送语音信息时，用户可以通过人嘴靠近麦克风170C发声，将声音信号输入到麦克风170C。电子设备100可以设置至少一个麦克风170C。在另一些实施例中，电子设备100可以设置两个麦克风170C，除了采集声音信号，还可以实现降噪功能。在另一些实施例中，电子设备100还可以设置三个，四个或更多麦克风170C，实现采集声音信号，降噪，还可以识别声音来源，实现定向录音功能等。The microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can put his mouth close to the microphone 170C to make a sound, and input the sound signal to the microphone 170C. The electronic device 100 may be provided with at least one microphone 170C. In some other embodiments, the electronic device 100 may be provided with two microphones 170C, which may also implement a noise reduction function in addition to collecting sound signals. In some other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify sound sources, and realize directional recording functions, etc.
耳机接口170D用于连接有线耳机。耳机接口170D可以是USB接口130，也可以是3.5mm的开放移动电子设备平台(open mobile terminal platform，OMTP)标准接口，美国蜂窝电信工业协会(cellular telecommunications industry association of the USA，CTIA)标准接口。The earphone interface 170D is used for connecting wired earphones. The earphone interface 170D may be the USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.
深度传感器180A用于获取景物的深度信息。在一些实施例中，深度传感器可以设置于摄像头193。The depth sensor 180A is used to acquire the depth information of the scene. In some embodiments, a depth sensor can be provided on the camera 193 .
压力传感器180B用于感受压力信号，可以将压力信号转换成电信号。在一些实施例中，压力传感器180B可以设置于显示屏194。压力传感器180B的种类很多，如电阻式压力传感器，电感式压力传感器，电容式压力传感器等。电容式压力传感器可以是包括至少两个具有导电材料的平行板。当有力作用于压力传感器180B，电极之间的电容改变。电子设备100根据电容的变化确定压力的强度。当有触摸操作作用于显示屏194，电子设备100根据压力传感器180B检测所述触摸操作强度。电子设备100也可以根据压力传感器180B的检测信号计算触摸的位置。在一些实施例中，作用于相同触摸位置，但不同触摸操作强度的触摸操作，可以对应不同的操作指令。例如：当有触摸操作强度小于第一压力阈值的触摸操作作用于短消息应用图标时，执行查看短消息的指令。当有触摸操作强度大于或等于第一压力阈值的触摸操作作用于短消息应用图标时，执行新建短消息的指令。The pressure sensor 180B is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180B may be located on the display screen 194 . There are many types of pressure sensors 180B, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may be comprised of at least two parallel plates with conductive material. When a force is applied to the pressure sensor 180B, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the electronic device 100 detects the intensity of the touch operation according to the pressure sensor 180B. The electronic device 100 may also calculate the touched position according to the detection signal of the pressure sensor 180B. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities may correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view short messages is executed. When a touch operation whose intensity is greater than or equal to the first pressure threshold acts on the icon of the short message application, the instruction of creating a new short message is executed.
陀螺仪传感器180C可以用于确定电子设备100的运动姿态。在一些实施例中，可以通过陀螺仪传感器180C确定电子设备100围绕三个轴(即，x，y和z轴)的角速度。陀螺仪传感器180C可以用于拍摄防抖。示例性的，当按下快门，陀螺仪传感器180C检测电子设备100抖动的角度，根据角度计算出镜头模组需要补偿的距离，让镜头通过反向运动抵消电子设备100的抖动，实现防抖。陀螺仪传感器180C还可以用于导航，体感游戏场景。The gyro sensor 180C can be used to determine the motion posture of the electronic device 100 . In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y and z axes) may be determined by the gyro sensor 180C. The gyro sensor 180C can be used for image stabilization. Exemplarily, when the shutter is pressed, the gyro sensor 180C detects the shaking angle of the electronic device 100, calculates the distance that the lens module needs to compensate according to the angle, and allows the lens to counteract the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180C can also be used for navigation and somatosensory game scenes.
气压传感器180D用于测量气压。在一些实施例中，电子设备100通过气压传感器180D测得的气压值计算海拔高度，辅助定位和导航。The air pressure sensor 180D is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude based on the air pressure value measured by the air pressure sensor 180D to assist positioning and navigation.
磁传感器180E包括霍尔传感器。电子设备100可以利用磁传感器180E检测翻盖皮套的开合。在一些实施例中，当电子设备100是翻盖机时，电子设备100可以根据磁传感器180E检测翻盖的开合。进而根据检测到的皮套的开合状态或翻盖的开合状态，设置翻盖自动解锁等特性。The magnetic sensor 180E includes a Hall sensor. The electronic device 100 may detect opening and closing of the flip leather case by using the magnetic sensor 180E. In some embodiments, when the electronic device 100 is a flip machine, the electronic device 100 may detect opening and closing of the flip according to the magnetic sensor 180E. Furthermore, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, features such as automatic unlocking of the flip cover are set.
加速度传感器180F可检测电子设备100在各个方向上(一般为三轴)加速度的大小。当电子设备100静止时可检测出重力的大小及方向。还可以用于识别电子设备姿态，应用于横竖屏切换，计步器等应用。The acceleration sensor 180F can detect the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of electronic devices, and can be used in applications such as horizontal and vertical screen switching, pedometers, etc.
距离传感器180G，用于测量距离。电子设备100可以通过红外或激光测量距离。在一些实施例中，拍摄场景，电子设备100可以利用距离传感器180G测距以实现快速对焦。The distance sensor 180G is used for measuring the distance. The electronic device 100 may measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180G for distance measurement to achieve fast focusing.
接近光传感器180H可以包括例如发光二极管(LED)和光检测器，例如光电二极管。发光二极管可以是红外发光二极管。电子设备100通过发光二极管向外发射红外光。电子设备100使用光电二极管检测来自附近物体的红外反射光。当检测到充分的反射光时，可以确定电子设备100附近有物体。当检测到不充分的反射光时，电子设备100可以确定电子设备100附近没有物体。电子设备100可以利用接近光传感器180H检测用户手持电子设备100贴近耳朵通话，以便自动熄灭屏幕达到省电的目的。接近光传感器180H也可用于皮套模式，口袋模式自动解锁与锁屏。The proximity light sensor 180H may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diodes may be infrared light emitting diodes. The electronic device 100 emits infrared light through the light emitting diode. Electronic device 100 uses photodiodes to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it may be determined that there is an object near the electronic device 100 . When insufficient reflected light is detected, the electronic device 100 may determine that there is no object near the electronic device 100 . The electronic device 100 can use the proximity light sensor 180H to detect that the user is holding the electronic device 100 close to the ear to make a call, so as to automatically turn off the screen to save power. Proximity light sensor 180H can also be used in leather case mode, automatic unlock and lock screen in pocket mode.
指纹传感器180J用于采集指纹。电子设备100可以利用采集的指纹特性实现指纹解锁，访问应用锁，指纹拍照，指纹接听来电等。The fingerprint sensor 180J is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access to application locks, take pictures with fingerprints, answer incoming calls with fingerprints, and the like.
温度传感器180K用于检测温度。在一些实施例中，电子设备100利用温度传感器180K检测的温度，执行温度处理策略。例如，当温度传感器180K上报的温度超过阈值，电子设备100执行降低位于温度传感器180K附近的处理器的性能，以便降低功耗实施热保护。在另一些实施例中，当温度低于另一阈值时，电子设备100对电池142加热，以避免低温导致电子设备100异常关机。在其他一些实施例中，当温度低于又一阈值时，电子设备100对电池142的输出电压执行升压，以避免低温导致的异常关机。The temperature sensor 180K is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180K to implement a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180K exceeds the threshold, the electronic device 100 may reduce the performance of the processor located near the temperature sensor 180K, so as to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to prevent the electronic device 100 from being shut down abnormally due to the low temperature. In some other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.
触摸传感器180L，也称“触控器件”。触摸传感器180L可以设置于显示屏194，由触摸传感器180L与显示屏194组成触摸屏，也称“触控屏”。触摸传感器180L用于检测作用于其上或附近的触摸操作。触摸传感器可以将检测到的触摸操作传递给应用处理器，以确定触摸事件类型。可以通过显示屏194提供与触摸操作相关的视觉输出。在另一些实施例中，触摸传感器180L也可以设置于电子设备100的表面，与显示屏194所处的位置不同。The touch sensor 180L is also called "touch device". The touch sensor 180L can be disposed on the display screen 194, and the touch sensor 180L and the display screen 194 form a touch screen, also called “touch screen”. The touch sensor 180L is used to detect a touch operation on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194 . In other embodiments, the touch sensor 180L may also be disposed on the surface of the electronic device 100 , which is different from the position of the display screen 194 .
环境光传感器180M用于感知环境光亮度。电子设备100可以根据感知的环境光亮度自适应调节显示屏194亮度。环境光传感器180M也可用于拍照时自动调节白平衡。环境光传感器180M还可以与接近光传感器180G配合，检测电子设备100是否在口袋里，以防误触。The ambient light sensor 180M is used for sensing ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180M can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180M can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in the pocket, so as to prevent accidental touch.
骨传导传感器180N可以获取振动信号。在一些实施例中，骨传导传感器180N可以获取人体声部振动骨块的振动信号。骨传导传感器180N也可以接触人体脉搏，接收血压跳动信号。在一些实施例中，骨传导传感器180N也可以设置于耳机中，结合成骨传导耳机。音频模块170可以基于所述骨传导传感器180N获取的声部振动骨块的振动信号，解析出语音信号，实现语音功能。应用处理器可以基于所述骨传导传感器180N获取的血压跳动信号解析心率信息，实现心率检测功能。The bone conduction sensor 180N can acquire vibration signals. In some embodiments, the bone conduction sensor 180N can acquire the vibration signal of the vibrating bone mass of the human voice. The bone conduction sensor 180N can also contact the pulse of the human body and receive the blood pressure beating signal. In some embodiments, the bone conduction sensor 180N can also be disposed in the earphone, combined into a bone conduction earphone. The audio module 170 can analyze the voice signal based on the vibration signal of the vibrating bone mass of the vocal part acquired by the bone conduction sensor 180N, so as to realize the voice function. The application processor can analyze the heart rate information based on the blood pressure beating signal acquired by the bone conduction sensor 180N, so as to realize the heart rate detection function.
按键190包括开机键，音量键等。按键190可以是机械按键。也可以是触摸式按键。电子设备100可以接收按键输入，产生与电子设备100的用户设置以及功能控制有关的键信号输入。The keys 190 include a power key, a volume key and the like. The key 190 may be a mechanical key. It can also be a touch button. The electronic device 100 can receive key input and generate key signal input related to user settings and function control of the electronic device 100 .
马达191可以产生振动提示。马达191可以用于来电振动提示，也可以用于触摸振动反馈。例如，作用于不同应用(例如拍照，音频播放等)的触摸操作，可以对应不同的振动反馈效果。作用于显示屏194不同区域的触摸操作，马达191也可对应不同的振动反馈效果。不同的应用场景(例如：时间提醒，接收信息，闹钟，游戏等)也可以对应不同的振动反馈效果。触摸振动反馈效果还可以支持自定义。The motor 191 can generate a vibrating reminder. The motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations applied to different applications (such as taking pictures, playing audio, etc.) may correspond to different vibration feedback effects. The motor 191 may also correspond to different vibration feedback effects for touch operations acting on different areas of the display screen 194 . Different application scenarios (for example: time reminder, receiving information, alarm clock, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.
指示器192可以是指示灯，可以用于指示充电状态，电量变化，也可以用于指示消息，未接来电，通知等。The indicator 192 can be an indicator light, and can be used to indicate charging status, power change, and can also be used to indicate messages, missed calls, notifications, and the like.
SIM卡接口195用于连接SIM卡。SIM卡可以通过插入SIM卡接口195，或从SIM卡接口195拔出，实现和电子设备100的接触和分离。电子设备100可以支持1个或N个SIM卡接口，N为大于1的正整数。SIM卡接口195可以支持Nano SIM卡，Micro SIM卡，SIM卡等。同一个SIM卡接口195可以同时插入多张卡。所述多张卡的类型可以相同，也可以不同。SIM卡接口195也可以兼容不同类型的SIM卡。SIM卡接口195也可以兼容外部存储卡。电子设备100通过SIM卡和网络交互，实现通话以及数据通信等功能。在一些实施例中，电子设备100采用eSIM，即：嵌入式SIM卡。eSIM卡可以嵌在电子设备100中，不能和电子设备100分离。The SIM card interface 195 is used for connecting a SIM card. The SIM card can be connected and separated from the electronic device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195 . The electronic device 100 may support 1 or N SIM card interfaces, where N is a positive integer greater than 1. SIM card interface 195 can support Nano SIM card, Micro SIM card, SIM card and so on. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the multiple cards may be the same or different. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to implement functions such as calling and data communication. In some embodiments, the electronic device 100 adopts an eSIM, that is, an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100 .
下面将结合图2(a)、图2(b)和图2(c)介绍本申请提供的3D光照估计方法的具体原理。本申请实施例提供3D光照估计方法及电子设备，可以应用于图1所示的电子设备100。该电子设备100上安装有AR应用软件，该AR应用软件可以是应用于多个领域的应用软件，例如：医疗领域、军事领域、游戏领域、通信领域、教育领域等多个领域。AR应用软件可以执行本申请实施例提供的3D光照估计算法，例如：当用户想使用AR功能时，用户可以通过物理按键或者点击电子设备100上的AR应用图标打开电子设备100上的AR应用，电子设备100可以确定当前真实场景的场景类型(例如：室内场景、室外场景、人脸场景或其他场景)，对于不同的场景类型获取不同的图像信息，并采用不同的光照估计算法获取一个或多个球谐系数，再根据该一个或多个球谐系数估计该当前真实场景中虚拟对象的光照效果。对于室内场景，电子设备100还可以根据多个球谐系数估计该场景中指定位置的光照效果。The specific principle of the 3D illumination estimation method provided by the present application will be introduced below with reference to FIG. 2(a), FIG. 2(b) and FIG. 2(c). Embodiments of the present application provide a 3D illumination estimation method and an electronic device, which may be applied to the electronic device 100 shown in FIG. 1 . AR application software is installed on the electronic device 100, and the AR application software may be application software applied in multiple fields, such as medical field, military field, game field, communication field, education field and other fields. The AR application software can execute the 3D illumination estimation algorithm provided by the embodiment of the present application. For example, when the user wants to use the AR function, the user can open the AR application on the electronic device 100 through a physical button or click on the AR application icon on the electronic device 100, The electronic device 100 can determine the scene type of the current real scene (for example: indoor scene, outdoor scene, face scene or other scenes), obtain different image information for different scene types, and use different illumination estimation algorithms to obtain one or more spherical harmonic coefficients, and then estimate the lighting effect of the virtual object in the current real scene according to the one or more spherical harmonic coefficients. For an indoor scene, the electronic device 100 may also estimate the lighting effect of a specified location in the scene according to a plurality of spherical harmonic coefficients.
如图2(a)所示，用户可以通过点击(例如：单击)电子设备100上的应用图标101打开电子设备100上的AR应用。As shown in FIG. 2( a ), the user can open the AR application on the electronic device 100 by clicking (for example: clicking) the application icon 101 on the electronic device 100 .
如图2(b)所示，进入AR应用后，电子设备100可以确定当前真实场景的场景类型，并通过摄像头193获取当前真实场景102的图像信息，并采用与当前真实场景102的场景类型对应的光照估计算法获取一个或多个球谐系数，再根据该一个或多个球谐系数估计虚拟对象的球谐系数。As shown in Figure 2(b), after entering the AR application, the electronic device 100 can determine the scene type of the current real scene, and obtain the image information of the current real scene 102 through the camera 193, and use the scene type corresponding to the current real scene 102 The lighting estimation algorithm of the algorithm obtains one or more spherical harmonic coefficients, and then estimates the spherical harmonic coefficients of the virtual object according to the one or more spherical harmonic coefficients.
如图2(c)所示，电子设备100可以根据虚拟对象的球谐系数渲染该虚拟对象，并将渲染好的虚拟对象103显示在电子设备100的显示屏上。As shown in FIG. 2( c ), the electronic device 100 may render the virtual object according to the spherical harmonic coefficients of the virtual object, and display the rendered virtual object 103 on the display screen of the electronic device 100 .
综上所述，电子设备100在估计该真实场景中虚拟对象的光照效果时，对于不同的场景类型可以获取不同的图像信息，并采用不同的光照估计算法获取一个或多个球谐系数，其中，球谐系数可以包括更多的信息量，例如：光强的方向等。因此，通过本方案，电子设备100可以估计出更接近真实场景的光照效果，以使得虚拟对象与真实场景更有效地融合，提升了用户体验。To sum up, when the electronic device 100 estimates the lighting effect of the virtual object in the real scene, it can obtain different image information for different scene types, and use different lighting estimation algorithms to obtain one or more spherical harmonic coefficients, where , spherical harmonic coefficients can include more information, such as: the direction of light intensity, etc. Therefore, through this solution, the electronic device 100 can estimate the lighting effect that is closer to the real scene, so that the virtual object can be more effectively integrated with the real scene, and user experience can be improved.
下面将结合说明书附图对本申请实施例提供的3D光照估计方法进行描述。The 3D illumination estimation method provided by the embodiment of the present application will be described below with reference to the accompanying drawings.
可以理解的，本申请实施例中，电子设备可以执行本申请实施例中的部分或全部步骤，这些步骤或操作仅是示例，本申请实施例还可以执行其它操作或者各种操作的变形。此外，各个步骤可以按照本申请实施例呈现的不同的顺序来执行，并且有可能并非要执行本申请实施例中的全部操作。It can be understood that, in the embodiment of the present application, the electronic device may perform some or all of the steps in the embodiment of the present application, these steps or operations are only examples, and the embodiment of the present application may also perform other operations or variations of various operations. In addition, each step may be performed in a different order presented in the embodiment of the present application, and it may not be necessary to perform all operations in the embodiment of the present application.
首先，进入AR应用后，电子设备可以自动打开相机应用，或者通过用户的操作打开相机应用来确定当前真实场景的场景类型。电子设备可以调用相机应用中的场景识别功能确定当前真实场景的场景类型。其中，场景识别功能可以是华为手机中的人工智能摄影大师MasterAI，该场景识别功能还可以是其他类型的应用，本申请实施例不限定。当前真实场景可以是用户通过电子设备的摄像头观察到的场景。该场景识别功能可以根据场景识别、人脸检测的结果，按照预定义的规则推荐适合当前真实场景的拍照模式。电子设备可以根据该拍照模式确定当前真实场景的场景类型。例如，若拍照模式是室内模式，电子设备可以确定当前真实场景的场景类型是室内场景；若拍照模式是室外模式，电子设备可以确定当前真实场景的场景类型是室外场景；若拍照模式是人脸模式，电子设备可以确定当前真实场景的场景类型是人脸场景。First, after entering the AR application, the electronic device can automatically open the camera application, or open the camera application through the user's operation to determine the scene type of the current real scene. The electronic device can call the scene recognition function in the camera application to determine the scene type of the current real scene. Wherein, the scene recognition function may be the artificial intelligence photography master MasterAI in the Huawei mobile phone, and the scene recognition function may also be other types of applications, which are not limited in this embodiment of the application. The current real scene may be a scene observed by the user through the camera of the electronic device. The scene recognition function can recommend a photo mode suitable for the current real scene according to the results of scene recognition and face detection according to predefined rules. The electronic device can determine the scene type of the current real scene according to the photographing mode. For example, if the photographing mode is an indoor mode, the electronic device may determine that the scene type of the current real scene is an indoor scene; if the photographing mode is an outdoor mode, the electronic device may determine that the scene type of the current real scene is an outdoor scene; mode, the electronic device can determine that the scene type of the current real scene is a face scene.
其中，室内场景可以包括人物或者动物居住或者活动的房屋建筑、交通工具等内部空间环境；室外场景可以包括房屋建筑内部之外的自然和人工场景；人脸场景可以包括人脸占当前真实场景的比例大于或等于阈值(例如：30％)的场景。Among them, the indoor scene can include internal space environments such as houses and vehicles where people or animals live or move; outdoor scenes can include natural and artificial scenes outside the interior of houses and buildings; Scenes with a ratio greater than or equal to a threshold (eg: 30%).
请参见图3(a)，用户可以通过点击(例如：单击)电子设备的操作界面302上的AR图标301进入AR应用，进入AR应用后，电子设备可以自动打开相机应用，并调用相机应用中的场景识别功能确定当前真实场景的场景类型。Referring to FIG. 3(a), the user can click (for example: click) the AR icon 301 on the operation interface 302 of the electronic device to enter the AR application. After entering the AR application, the electronic device can automatically open the camera application and call the camera application. The scene recognition function in determines the scene type of the current real scene.
请参见图3(b)，用户可以通过点击(例如：单击)电子设备的操作界面302上的AR图标301进入AR应用，进入AR应用后，用户可以通过点击信息提示框303中的按钮“是”打开相机应用，并调用相机应用中的场景识别功能确定真实场景的场景类型。Please refer to FIG. 3(b), the user can click (for example: click) the AR icon 301 on the operation interface 302 of the electronic device to enter the AR application. After entering the AR application, the user can click the button " Yes" to open the camera application, and call the scene recognition function in the camera application to determine the scene type of the real scene.
请参见图3(c)，若用户点击信息提示框303中的按钮“否”，电子设备不会打开相机应用，而是会进入AR应用的操作界面304。Referring to FIG. 3( c ), if the user clicks the button “No” in the information prompt box 303 , the electronic device will not open the camera application, but will enter the operation interface 304 of the AR application.
请参见图3(d)，用户可以通过点击(例如：单击)电子设备的操作界面302上的AR图标301进入AR应用，进入AR应用后，用户可以通过点击AR应用的操作界面304上AR功能按钮305打开相机应用，并调用相机应用中的场景识别功能确定当前真实场景的场景类型。Referring to FIG. 3(d), the user can click (for example: click) the AR icon 301 on the operation interface 302 of the electronic device to enter the AR application. After entering the AR application, the user can click on the AR icon 301 on the operation interface 304 of the AR application. The function button 305 opens the camera application, and invokes the scene recognition function in the camera application to determine the scene type of the current real scene.
下面以当前真实场景的场景类型分别为室内场景、室外场景和人脸场景为例介绍该3D光照估计方法。In the following, the 3D illumination estimation method will be introduced by taking the scene types of the current real scene as indoor scene, outdoor scene and human face scene as examples.
若电子设备确定的真实场景的场景类型是室内场景(如图4(a)所示的场景)，该3D光照估计方法的实现过程可以参考下述图5所示方法对应的实施例。If the scene type of the real scene determined by the electronic device is an indoor scene (such as the scene shown in FIG. 4(a)), the implementation process of the 3D illumination estimation method can refer to the embodiment corresponding to the method shown in FIG. 5 below.
若电子设备确定的真实场景的场景类型是室外场景(如图4(b)所示的场景)，该3D光照估计方法的实现过程可以参考下述图10所示方法对应的实施例。If the scene type of the real scene determined by the electronic device is an outdoor scene (such as the scene shown in FIG. 4(b)), the implementation process of the 3D illumination estimation method can refer to the embodiment corresponding to the method shown in FIG. 10 below.
若电子设备确定的真实场景的场景类型是人脸场景(如图4(c)所示的场景)，该3D光照估计方法的实现过程可以参考下述图12所示方法对应的实施例。If the scene type of the real scene determined by the electronic device is a face scene (such as the scene shown in FIG. 4(c)), the implementation process of the 3D illumination estimation method can refer to the embodiment corresponding to the method shown in FIG. 12 below.
需要说明的是，图4(a)、图4(b)和图4(c)仅为室内场景、室外场景和人脸场景的一个示例，本领域技术人员应该理解，室内场景、室外场景和人脸场景还可以是包括其他内容的的场景，本申请实施例不进行具体限制。It should be noted that Fig. 4(a), Fig. 4(b) and Fig. 4(c) are only an example of indoor scene, outdoor scene and human face scene, and those skilled in the art should understand that indoor scene, outdoor scene and The face scene may also be a scene including other content, which is not specifically limited in this embodiment of the present application.
如图5所示，为本申请实施例提供的一种3D光照估计方法，该3D光照估计方法适用于AR技术，该3D光照估计方法包括以下步骤：As shown in Figure 5, it is a 3D illumination estimation method provided in the embodiment of the present application, the 3D illumination estimation method is suitable for AR technology, and the 3D illumination estimation method includes the following steps:
步骤501、电子设备获取第一场景的图像信息。Step 501, the electronic device acquires image information of a first scene.
其中，该第一场景可以是当前真实的场景，若第一场景的场景类型是室内场景，该第一场景的图像信息可以包括第一场景中像素点的颜色信息和第一场景中像素点的深度信息。Wherein, the first scene may be the current real scene. If the scene type of the first scene is an indoor scene, the image information of the first scene may include the color information of the pixels in the first scene and the color information of the pixels in the first scene. depth information.
其中，该第一场景中像素点的颜色信息可以用于指示第一场景中像素点的颜色，例如：第一场景中像素点的颜色信息可以是第一场景中像素点的红色(red，R)、绿色(green，G)、蓝色(blue，B)的颜色值。该第一场景中像素点的颜色信息可以通过电子设备的主摄像头获取，例如：电子设备的主摄像头可以拍摄至少一张第一场景的图像或第一场景的视频，电子设备根据该至少一张第一场景的图像或第一场景的视频获取第一场景中像素点的颜色信息。若电子设备的主摄像头拍摄了两张或两张以上的图像，电子设备可以获取每张图像中像素点的颜色信息，并将两张或两张以上图像中像素点的颜色信息的平均值作为第一场景中像素点的颜色信息。若电子设备的主摄像头拍摄了10帧的第一场景的视频，电子设备可以获取每帧视频中像素点的颜色信息，并将10帧视频中像素点的颜色信息的平均值作为第一场景中像素点的颜色信息。Wherein, the color information of the pixels in the first scene can be used to indicate the color of the pixels in the first scene, for example: the color information of the pixels in the first scene can be the red (red, R ), green (green, G), blue (blue, B) color values. The color information of the pixels in the first scene can be obtained through the main camera of the electronic device, for example: the main camera of the electronic device can take at least one image or video of the first scene, and the electronic device The image of the first scene or the video of the first scene acquires color information of pixels in the first scene. If the main camera of the electronic device takes two or more images, the electronic device can obtain the color information of the pixels in each image, and use the average value of the color information of the pixels in the two or more images as Color information of pixels in the first scene. If the main camera of the electronic device captures 10 frames of the video of the first scene, the electronic device can obtain the color information of the pixels in each frame of video, and use the average value of the color information of the pixels in the 10 frames of video as the first scene. The color information of the pixel.
其中，该第一场景中像素点的深度信息可以用于指示第一场景中像素点的位置，例如：第一场景中像素点的深度信息可以是第一场景中像素点在第一场景的空间坐标。该第一场景中像素点的深度信息可以通过电子设备的深度传感器获取，例如：电子设备的深度传感器可以拍摄至少一张第一场景的图像或第一场景的视频，电子设备根据该至少一张第一场景的图像或第一场景的视频获取第一场景中像素点的深度信息。若电子设备的深度传感器拍摄了两张或两张以上的图像，电子设备可以获取每张图像中像素点的深度信息，并将两张或两张以上图像中像素点的深度信息的平均值作为第一场景中像素点的深度信息。若电子设备的深度传感器拍摄了10帧的第一场景的视频，电子设备可以获取每帧视频中像素点的深度信息，并将10帧视频中像素点的深度信息的平均值作为第一场景中像素点的深度信息。Wherein, the depth information of the pixels in the first scene can be used to indicate the position of the pixels in the first scene, for example: the depth information of the pixels in the first scene can be the space of the pixels in the first scene in the first scene coordinate. The depth information of the pixels in the first scene can be obtained through the depth sensor of the electronic device, for example: the depth sensor of the electronic device can take at least one image or video of the first scene, and the electronic device can capture at least one image or video of the first scene according to the at least one Depth information of pixels in the first scene is obtained from the image of the first scene or the video of the first scene. If the depth sensor of the electronic device takes two or more images, the electronic device can obtain the depth information of the pixels in each image, and use the average value of the depth information of the pixels in the two or more images as Depth information of pixels in the first scene. If the depth sensor of the electronic device captures 10 frames of the video of the first scene, the electronic device can obtain the depth information of the pixels in each frame of video, and use the average value of the depth information of the pixels in the 10 frames of video as the first scene. Pixel depth information.
需要说明的是，电子设备的主摄像头拍摄的至少一张第一场景的图像或第一场景的视频和电子设备的深度传感器拍摄的至少一张第一场景的图像或第一场景的视频有相同的位置参考。It should be noted that the at least one image of the first scene or the video of the first scene captured by the main camera of the electronic device and the at least one image of the first scene or video of the first scene captured by the depth sensor of the electronic device have the same The location reference for .
示例性的，该深度传感器可以是飞行时间(time of fight，ToF)传感器。电子设备可以根据ToF获得的第一场景中像素点的深度信息以较少的帧数快速建立第一模拟场景。Exemplarily, the depth sensor may be a time of flight (time of fight, ToF) sensor. The electronic device can quickly establish the first simulation scene with a small number of frames according to the depth information of the pixels in the first scene obtained by ToF.
可选的，电子设备在确定第一场景的场景类型是室内场景时，自动通过电子设备的主摄像头和深度传感器拍摄至少一张第一场景的图像或第一场景的视频，再根据该至少一张第一场景的图像或第一场景的视频获取第一场景的图像信息。Optionally, when the electronic device determines that the scene type of the first scene is an indoor scene, it automatically takes at least one image of the first scene or a video of the first scene through the main camera and the depth sensor of the electronic device, and then according to the at least one An image of the first scene or a video of the first scene is used to obtain the image information of the first scene.
例如，请参见图6(a)，用户可以通过点击AR应用界面601上的AR功能按钮602进入相机应用，电子设备可以在确定第一场景的场景类型是室内场景时，自动通过电子设备的主摄像头和深度传感器拍摄第一场景的视频，再根据该第一场景的视频获取第一场景的图像信息。For example, referring to FIG. 6(a), the user can click the AR function button 602 on the AR application interface 601 to enter the camera application, and the electronic device can automatically pass through the electronic device's main The camera and the depth sensor shoot the video of the first scene, and then acquire the image information of the first scene according to the video of the first scene.
可选的，电子设备在确定第一场景的场景类型是室内场景时，通过用户手动拍摄至少一张第一场景的图像或第一场景的视频，再根据该至少一张第一场景的图像或第一场景的视频获取第一场景的图像信息。Optionally, when the electronic device determines that the scene type of the first scene is an indoor scene, the user manually takes at least one image of the first scene or a video of the first scene, and then according to the at least one image or video of the first scene The video of the first scene acquires the image information of the first scene.
例如，请参见图6(b)，用户可以点击信息提示框603中的按钮“确定”，再在拍照模式下通过点击拍照按钮604通过电子设备的主摄像头和深度传感器拍摄一张照片，再根据拍摄的照片获取第一场景的图像信息。For example, referring to FIG. 6(b), the user can click the button "OK" in the information prompt box 603, and then click the camera button 604 in the camera mode to take a photo through the main camera and depth sensor of the electronic device, and then according to The captured photo acquires image information of the first scene.
步骤502、电子设备根据第一场景的图像信息建立第一模拟场景。Step 502, the electronic device creates a first simulation scene according to the image information of the first scene.
其中，第一模拟场景可以是根据第一场景的图像信息建立的一个三维虚拟场景。第一模拟场景可以包括至少两个光照探测球以及点云。Wherein, the first simulation scene may be a three-dimensional virtual scene established according to the image information of the first scene. The first simulated scene may include at least two light probe spheres and a point cloud.
其中，光照探测球可以为第一模拟场景中，用于指示该光照探测球所处位置的球谐系数的虚拟球体。Wherein, the illumination probe sphere may be a virtual sphere of spherical harmonic coefficients used to indicate the location of the illumination probe sphere in the first simulation scene.
可选的，光照探测球还可以称为光照探头(light probe)或者以其他名称命名，本申请实施例不予限制。光照探测球的球谐系数包括该光照探测球的颜色信息和该光照探测球的位置信息，该光照探测球的颜色信息用于指示该光照探测球的颜色；该光照探测球的位置信息用于指示该光照探测球在第一模拟场景中的位置。Optionally, the light probe ball may also be called a light probe (light probe) or be named by other names, which is not limited in this embodiment of the present application. The spherical harmonic coefficient of the illumination detection sphere includes the color information of the illumination detection sphere and the position information of the illumination detection sphere, the color information of the illumination detection sphere is used to indicate the color of the illumination detection sphere; the position information of the illumination detection sphere is used for Indicates the position of the light probe sphere in the first simulated scene.
需要说明的是，光照探测球的球谐系数包括的光照探测球的颜色信息可以包括光照探测球的颜色值和光照方向。本领域技术人员应该理解，本申请实施例中的球谐系数包括的颜色信息都包括颜色值和光照方向，下文不再赘述。It should be noted that the color information of the illumination probe sphere included in the spherical harmonic coefficient of the illumination probe sphere may include a color value and an illumination direction of the illumination probe sphere. Those skilled in the art should understand that the color information included in the spherical harmonic coefficients in the embodiments of the present application includes a color value and an illumination direction, which will not be described in detail below.
可选的，光照探测球在第一模拟场景中的个数和位置是预设置的。Optionally, the number and positions of the light detection spheres in the first simulation scene are preset.
其中，该点云可以为第一模拟场景中，用于指示第一模拟场景中对象的几何形态、物体属性和颜色分布的点的集合。其中，物体属性可以包括光源和非光源。例如，点云按物体属性可以分为光源点云和非光源点云，该光源点云为被深度学习算法(例如：卷积神经网络(convolutionalneuralnetworks，CNN))识别到的属于光源物体上的点的集合，该非光源点云为被深度学习算法(例如：CNN)识别到的属于非光源物体上的点的集合。其中，光源物体可以包括灯光、窗口或强烈的反光等。Wherein, the point cloud may be a collection of points in the first simulation scene used to indicate the geometry, object properties and color distribution of the objects in the first simulation scene. Wherein, the object attribute may include light source and non-light source. For example, point cloud can be divided into light source point cloud and non-light source point cloud according to object attributes. The light source point cloud is a point on a light source object identified by a deep learning algorithm (for example: convolutional neural networks (CNN)). A collection of non-light source point clouds is a collection of points on non-light source objects identified by a deep learning algorithm (for example: CNN). Wherein, the light source object may include lights, windows, or strong reflections.
可选的，点云是将第一场景的图像信息数字化后的采样点的集合。点云的信息可以包括点云中每个点的位置信息和辐照度(irradiance)信息，该点的位置信息用于指示该点在第一模拟场景中的位置，该点的辐照度信息用于指示该点的辐照度。Optionally, the point cloud is a collection of sampling points after digitizing the image information of the first scene. The information of the point cloud can include position information and irradiance (irradiance) information of each point in the point cloud, the position information of this point is used to indicate the position of this point in the first simulation scene, the irradiance information of this point Indicates the irradiance at the point.
需要说明的是，电子设备可以根据第一场景的图像信息获取点云的信息。It should be noted that the electronic device may acquire point cloud information according to the image information of the first scene.
示例性的，图7是一个模拟场景，该光照探头可以是图7中示出的球体，点云可以是图7中示出的黑点，该点云是图7所示的模拟场景中的一部分点云。Exemplarily, Fig. 7 is a simulated scene, the light probe can be the sphere shown in Fig. 7, the point cloud can be the black point shown in Fig. 7, and the point cloud is in the simulated scene shown in Fig. 7 Part of a point cloud.
需要说明的是，第一模拟场景中可以包括比图7所示的光照探测球更多或者更少的光照探测球，第一模拟场景中可以包括比图7所示的点云更多的点云，本申请实施例不进行具体限定。It should be noted that the first simulation scene may include more or fewer light probe spheres than the light probe sphere shown in FIG. 7 , and the first simulation scene may include more points than the point cloud shown in FIG. 7 Cloud, the embodiment of this application does not specifically limit it.
步骤503、电子设备根据第一模拟场景包括的点云的信息估计光照探测球的球谐系数。Step 503, the electronic device estimates the spherical harmonic coefficient of the illumination probe sphere according to the point cloud information included in the first simulation scene.
可选的，对于第一光照探测球，该第一光照探测球为第一模拟场景包括的任一光照探测球，根据第一模拟场景包括的点云的信息估计该第一光照探测球的球谐系数，可以包括步骤1和步骤2：Optionally, for the first illumination probe sphere, the first illumination probe sphere is any illumination probe sphere included in the first simulation scene, and the ball of the first illumination probe sphere is estimated according to the point cloud information included in the first simulation scene Harmonic coefficients, can include step 1 and step 2:
步骤1、电子设备根据点云的信息获取第一光照探测球的第一辐照度。Step 1. The electronic device acquires the first irradiance of the first light detection sphere according to the point cloud information.
可选的，对于点云中的任一点，电子设备根据该点的位置信息和辐照度信息获取该点在第一光照探测球的第二辐照度。电子设备根据点云中每个点在第一光照探测球的第二辐照度获取第一光照探测球的第一辐照度。Optionally, for any point in the point cloud, the electronic device acquires the second irradiance of the point on the first light detection sphere according to the position information and irradiance information of the point. The electronic device acquires the first irradiance of the first light probing sphere according to the second irradiance of each point in the point cloud on the first light probing sphere.
示例一，电子设备根据点云中每个点在第一光照探测球的第二辐照度获取第一光照探测球的第一辐照度可以包括：电子设备将点云中的每个点在第一光照探测球的第二辐照度加权求和，得到该第一光照探测球的第一辐照度。例如：若点云中包括6个点，6个点分别在第一光照探测球位置处的第二辐照度以及加权系数如表1所示。第一辐照度＝q1*L1+q2*L2+q3*L3+q4*L4+q5*L5+q6*L6。Example 1, obtaining the first irradiance of the first light detection sphere by the electronic device according to the second irradiance of each point in the point cloud at the first light detection sphere may include: the electronic device takes each point in the point cloud at The weighted sum of the second irradiance of the first light detection sphere is obtained to obtain the first irradiance of the first light detection sphere. For example: if the point cloud includes 6 points, the second irradiance and weighting coefficients of the 6 points at the position of the first light detection ball are shown in Table 1. The first irradiance=q1*L1+q2*L2+q3*L3+q4*L4+q5*L5+q6*L6.
表1Table 1
其中，每个点的加权系数可以根据每个点和第一光照探测球在第一模拟场景中的位置确定。例如，若该点与该第一光照探测球之间的直线距离越大，该点的加权系数越小；若该点与该第一光照探测球之间的直线距离越小，该点的加权系数越大；若该点与该第一光照探测球之间的夹角越大，该点的加权系数越小；若该点与该第一光照探测球之间的夹角越小，该点的加权系数越大。Wherein, the weighting coefficient of each point may be determined according to each point and the position of the first illumination detection sphere in the first simulation scene. For example, if the straight-line distance between the point and the first light detection ball is larger, the weighting coefficient of the point is smaller; if the straight-line distance between the point and the first light detection ball is smaller, the weighting coefficient of the point is smaller. The larger the coefficient is; if the angle between the point and the first light detection ball is larger, the weight coefficient of the point is smaller; if the angle between the point and the first light detection ball is smaller, the point The larger the weighting coefficient is.
其中，若点云中的任一点在第一模拟场景中的位置坐标为(x1，y1，z1)，第一光照探测球在第一模拟场景中的位置坐标为(x2，y2，z2)，则该点与该第一光照探测球之间的直线距离可以是若该点与该第一光照探测球之间的夹角为θ，则Among them, if the position coordinates of any point in the point cloud in the first simulation scene are (x1 , y1 , z1 ), the position coordinates of the first light detection ball in the first simulation scene are (x2 , y2 , z2 ), then the straight-line distance between this point and the first light detection sphere can be If the angle between this point and the first light detection ball is θ, then
示例二，电子设备根据点云中每个点在第一光照探测球的第二辐照度获取第一光照探测球的第一辐照度可以包括：电子设备将第一有效点在第一光照探测球的第二辐照度加权求和，得到该第一光照探测球的第一辐照度。其中，第一有效点可以是点云中在第一光照探测球的可见范围内的点。Example 2, obtaining the first irradiance of the first illumination probe sphere by the electronic device according to the second irradiance of each point in the point cloud on the first illumination probe sphere may include: The second irradiance of the probe sphere is weighted and summed to obtain the first irradiance of the first illumination probe sphere. Wherein, the first effective point may be a point in the point cloud within the visible range of the first illumination probe sphere.
可选的，该第一光照探测球的可见范围是预设置的，或者是电子设备根据第一光照探测球在第一场景中的位置和点云中的点在第一场景中的位置确定的。Optionally, the visible range of the first light detection ball is preset, or is determined by the electronic device according to the position of the first light detection ball in the first scene and the position of the point in the point cloud in the first scene .
如图8所示，点云中的任一点801在光照探测球LP802和LP803的可见范围内，点云中的任一点801不在光照探测球LP804的可见范围内。As shown in FIG. 8 , any point 801 in the point cloud is within the visible range of the light probe spheres LP802 and LP803 , and any point 801 in the point cloud is not within the visible range of the light probe sphere LP804 .
示例性的，若点云中包括9个点，其中有6个点在第一光照探测球的可见范围内。9个点分别在第一光照探测球位置处的第二辐照度以及加权系数如表2所示。第一辐照度＝q1*L1+q2*L2+q3*L3+q4*L4+q5*L5+q6*L6。Exemplarily, if the point cloud includes 9 points, 6 of them are within the visible range of the first light detection sphere. Table 2 shows the second irradiance and weighting coefficients of the nine points respectively at the position of the first light detection ball. The first irradiance=q1*L1+q2*L2+q3*L3+q4*L4+q5*L5+q6*L6.
表2Table 2
需要说明的是，若点云中的某一个点不在第一光照探测球的可见范围内，该点相对于该第一光照探测球的加权系数为0，该点在该第一光照探测球位置处的第二辐照度可以为0也可以是一个很小的值，即该点在该第一光照探测球位置处的第二辐照度可以忽略不计。It should be noted that if a point in the point cloud is not within the visible range of the first illumination probe sphere, the weighting coefficient of the point relative to the first illumination probe sphere is 0, and the point is at the position of the first illumination probe sphere The second irradiance at can be 0 or a very small value, that is, the second irradiance of this point at the position of the first light detection ball can be ignored.
其中，每个点的加权系数可以根据每个点和第一光照探测球在第一模拟场景中的位置确定。例如，若该点与该第一光照探测球之间的直线距离越大，该点的加权系数越小；若该点与该第一光照探测球之间的直线距离越小，该点的加权系数越大；若该点与该第一光照探测球之间的夹角越大，该点的加权系数越小；若该点与该第一光照探测球之间的夹角越小，该点的加权系数越大。Wherein, the weighting coefficient of each point may be determined according to each point and the position of the first illumination detection sphere in the first simulation scene. For example, if the straight-line distance between the point and the first light detection ball is larger, the weighting coefficient of the point is smaller; if the straight-line distance between the point and the first light detection ball is smaller, the weighting coefficient of the point is smaller. The larger the coefficient is; if the angle between the point and the first light detection ball is larger, the weight coefficient of the point is smaller; if the angle between the point and the first light detection ball is smaller, the point The larger the weighting coefficient is.
其中，关于该点与该第一光照探测球之间的直线距离和该点与该第一光照探测球之间的夹角的介绍可以参考上述示例一中对应的介绍，此处不再赘述。For the introduction of the straight-line distance between the point and the first light detection sphere and the angle between the point and the first light detection sphere, reference may be made to the corresponding introduction in the first example above, which will not be repeated here.
示例三，电子设备根据点云中每个点在第一光照探测球的第二辐照度获取第一光照探测球的第一辐照度可以包括：电子设备将第二有效点在第一光照探测球的第二辐照度加权求和，得到该第一光照探测球的第一辐照度。其中，第二有效点可以是点云中在第一光照探测球的可见范围内，且在第一光照探测球的第二辐照度大于或等于预设阈值的点。例如，若点云中包括9个点。9个点分别在第一光照探测球位置处的第二辐照度以及加权系数如表3所示。第一辐照度＝q1*L1+q2*L2+q3*L3+q4*L4+q5*L5。Example 3, obtaining the first irradiance of the first illumination probe sphere by the electronic device according to the second irradiance of each point in the point cloud on the first illumination probe sphere may include: The second irradiance of the probe sphere is weighted and summed to obtain the first irradiance of the first illumination probe sphere. Wherein, the second effective point may be a point in the point cloud that is within the visible range of the first illumination probe sphere and whose second irradiance on the first illumination probe sphere is greater than or equal to a preset threshold. For example, if the point cloud includes 9 points. Table 3 shows the second irradiance and weighting coefficients of the nine points respectively at the position of the first light detection ball. The first irradiance=q1*L1+q2*L2+q3*L3+q4*L4+q5*L5.
表3table 3
需要说明的是，若点云中的某一个点在第一光照探测球的的第二辐照度小于预设阈值，该点相对于该第一光照探测球的加权系数可以为0也可以为一个很小的值。It should be noted that if the second irradiance of a certain point in the point cloud on the first light detection sphere is less than the preset threshold, the weighting coefficient of this point relative to the first light detection sphere can be 0 or can be A very small value.
其中，每个点的加权系数可以根据每个点和第一光照探测球在第一模拟场景中的位置确定。例如，若该点与该第一光照探测球之间的直线距离越大，该点的加权系数越小；若该点与该第一光照探测球之间的直线距离越小，该点的加权系数越大；若该点与该第一光照探测球之间的夹角越大，该点的加权系数越小；若该点与该第一光照探测球之间的夹角越小，该点的加权系数越大。Wherein, the weighting coefficient of each point may be determined according to each point and the position of the first illumination detection sphere in the first simulation scene. For example, if the straight-line distance between the point and the first light detection ball is larger, the weighting coefficient of the point is smaller; if the straight-line distance between the point and the first light detection ball is smaller, the weighting coefficient of the point is smaller. The larger the coefficient is; if the angle between the point and the first light detection ball is larger, the weight coefficient of the point is smaller; if the angle between the point and the first light detection ball is smaller, the point The larger the weighting coefficient is.
其中，关于该点与该第一光照探测球之间的直线距离和该点与该第一光照探测球之间的夹角的介绍可以参考上述示例一中对应的介绍，此处不再赘述。For the introduction of the straight-line distance between the point and the first light detection sphere and the angle between the point and the first light detection sphere, reference may be made to the corresponding introduction in the first example above, which will not be repeated here.
步骤2、电子设备根据第一光照探测球的第一辐照度估计第一光照探测球的球谐系数。Step 2. The electronic device estimates the spherical harmonic coefficient of the first illumination detection sphere according to the first irradiance of the first illumination detection sphere.
可选的，第一光照探测球的球谐系数包括三个颜色通道(R、G和B)的球谐系数。Optionally, the spherical harmonic coefficients of the first illumination detection sphere include spherical harmonic coefficients of three color channels (R, G, and B).
其中，第一光照探测球在一个颜色通道的球谐系数可以是0～2阶共9维浮点型系数(也称为L2 spherical harmonics)。因此，第一光照探测球的球谐系数可以是3*9维。Wherein, the spherical harmonic coefficients of the first illumination detection sphere in one color channel may be 0-2 order 9-dimensional floating-point coefficients (also called L2 spherical harmonics). Therefore, the spherical harmonic coefficient of the first illumination probe sphere may be 3*9 dimensions.
可选的，第一光照探测球的任一个颜色通道的球谐系数可以表示为第一光照探测球的三个颜色通道的球谐系数可以表示为：Optionally, the spherical harmonic coefficient of any color channel of the first light detection sphere can be expressed as The spherical harmonic coefficients of the three color channels of the first light probe sphere can be expressed as:
下面以第一光照探测球的任一个颜色通道为例介绍电子设备根据第一辐照度估计第一光照探测球在该颜色通道的球谐系数。The following takes any color channel of the first illumination detection sphere as an example to introduce the estimation of the spherical harmonic coefficient of the first illumination detection sphere in the color channel by the electronic device according to the first irradiance.
若第一光照探测球的任一个颜色通道的球谐系数为电子设备可以根据下述公式计算第一光照探测球在该颜色通道的球谐系数的元素：If the spherical harmonic coefficient of any color channel of the first light probe sphere is The electronic device can calculate the elements of the spherical harmonic coefficient of the first illumination detection sphere in the color channel according to the following formula:
其中，xj为第一光照探测球上的任一方向，L(xj)表示第一光照探测球上任一方向上的第三辐照度，Ylm(xj)表示第一光照探测球上任一方向上的球谐基函数。Among them, xj is any direction on the first light probing sphere, L(xj ) represents the third irradiance on any direction on the first light probing sphere, Ylm (xj ) represents the first light probing sphere Spherical harmonic basis functions in either direction.
需要说明的是，电子设备可以在第一光照探测球上选择N个方向，该N个方向可以覆盖第一光照探测球的球体，N为正整数，0＜j≤N。电子设备可以根据第一光照探测球的第一辐照度获取第一光照探测球上任一方向上的第三辐照度L(xj)。电子设备可以根据下述公式获取Ylm(xj)。It should be noted that the electronic device may select N directions on the first illumination detection sphere, and the N directions may cover the sphere of the first illumination detection sphere, where N is a positive integer, and 0<j≤N. The electronic device may acquire the third irradiance L(xj ) in any direction on the first light detection sphere according to the first irradiance of the first light detection sphere. The electronic device can obtain Ylm (xj ) according to the following formula.
其中，是xj在第一光照探测球的球面上的球坐标，Ylm是l和m的球谐函数，0≤m≤l，i是虚数单位，Plm是伴随勒让德多项式，Pl(x)是l阶勒让德多项式，in, is the spherical coordinate of xj on the spherical surface of the first light detection sphere, Ylm is the spherical harmonic function of l and m, 0≤m≤l, i is the imaginary unit, Plm is the associated Legendre polynomial, Pl (x) is a Legendre polynomial of order l,
可选的，电子设备根据上述公式计算第一光照探测球在三个颜色通道的球谐系数，即第一光照探测球的球谐系数。Optionally, the electronic device calculates the spherical harmonic coefficients of the first illumination detection sphere in the three color channels according to the above formula, that is, the spherical harmonic coefficient of the first illumination detection sphere.
可选的，电子设备根据步骤5031和5032估计任一个光照探测球的球谐系数。Optionally, the electronic device estimates the spherical harmonic coefficient of any illumination detection sphere according to steps 5031 and 5032.
步骤504、电子设备根据光照探测球的球谐系数估计虚拟对象的位置的球谐系数。Step 504, the electronic device estimates the spherical harmonic coefficient of the position of the virtual object according to the spherical harmonic coefficient of the illumination detection sphere.
其中，虚拟对象可以是用户想通过AR应用显示在电子设备的显示屏上的由电子数据制作的可视化对象。例如，虚拟对象可以是一张图片、3D模型等。Wherein, the virtual object may be a visualized object made of electronic data that the user wants to display on the display screen of the electronic device through the AR application. For example, the virtual object can be a picture, a 3D model, and the like.
可选的，电子设备根据用户输入获取虚拟对象的位置；或者，电子设备预设置虚拟对象的位置。其中，虚拟对象的位置用于指示虚拟对象在第一模拟场景的空间坐标。Optionally, the electronic device acquires the position of the virtual object according to user input; or, the electronic device presets the position of the virtual object. Wherein, the position of the virtual object is used to indicate the space coordinates of the virtual object in the first simulation scene.
例如：电子设备可以在获取第一场景的图像信息后，通过图9中的提示框901提示用户输入虚拟对象的位置，用户可以通过点击(例如：单击)第一场景的图像信息中的茶几902，输入虚拟对象的位置，电子设备可以将渲染好的虚拟对象放置在用户输入的位置。For example: after the electronic device acquires the image information of the first scene, it can prompt the user to input the position of the virtual object through the prompt box 901 in FIG. 902. Input the position of the virtual object, and the electronic device may place the rendered virtual object at the position input by the user.
在一种示例中，电子设备根据光照探测球的球谐系数估计虚拟对象的球谐系数可以包括：电子设备对光照探测球的球谐系数进行加权求和，得到虚拟对象的球谐系数。In an example, the electronic device estimating the spherical harmonic coefficients of the virtual object according to the spherical harmonic coefficients of the illumination detection sphere may include: the electronic device performs weighted summation of the spherical harmonic coefficients of the illumination detection sphere to obtain the spherical harmonic coefficients of the virtual object.
示例性的，若第一模拟场景包括5个光照探测球，5个光照探测球的球谐系数和加权系数如表4所示，则虚拟对象的球谐系数为0*y1+0.3*y2+0.1*y3+0.2*y4+0.4*y5+0*y6。其中，6个加权系数可以根据6个光照探测球与虚拟对象的距离确定。Exemplarily, if the first simulation scene includes 5 light probe spheres, and the spherical harmonic coefficients and weighting coefficients of the 5 light probe spheres are as shown in Table 4, then the spherical harmonic coefficient of the virtual object is 0*y1+0.3*y2+ 0.1*y3+0.2*y4+0.4*y5+0*y6. Among them, the 6 weighting coefficients can be determined according to the distances between the 6 light detection spheres and the virtual object.
表4Table 4
在又一种示例中，电子设备根据光照探测球的球谐系数估计虚拟对象的球谐系数可以包括：电子设备对光照探测球的球谐系数进行加权求和，得到虚拟对象的球谐系数。其中，该光照探测球与虚拟对象间的距离小于或等于第一距离。In yet another example, the electronic device estimating the spherical harmonic coefficients of the virtual object according to the spherical harmonic coefficients of the illumination detection sphere may include: the electronic device performs weighted summation of the spherical harmonic coefficients of the illumination detection sphere to obtain the spherical harmonic coefficients of the virtual object. Wherein, the distance between the light detection ball and the virtual object is less than or equal to the first distance.
例如，若第一距离为0.5m，第一模拟场景包括5个光照探测球，5个光照探测球与虚拟对象间的距离、5个光照探测球的球谐系数、加权系数如表5所示，则虚拟对象的球谐系数＝0.3*y2+0.1*y3+0.2*y4+0.4*y5。其中，6个加权系数可以根据6个光照探测球与虚拟对象的距离确定。For example, if the first distance is 0.5m, the first simulation scene includes 5 light detection spheres, the distance between the 5 light detection spheres and the virtual object, the spherical harmonic coefficients and weighting coefficients of the 5 light detection spheres are shown in Table 5 , then the spherical harmonic coefficient of the virtual object=0.3*y2+0.1*y3+0.2*y4+0.4*y5. Among them, the 6 weighting coefficients can be determined according to the distances between the 6 light detection spheres and the virtual object.
表5table 5
需要说明的是，在实际应用中，电子设备可以根据步骤5031和5032，估计与虚拟对象间的距离小于或等于第一距离的光照探测球的球谐系数，再根据与虚拟对象间的距离小于或等于第一距离的光照探测球的球谐系数估计虚拟对象的球谐系数，而不用估计与虚拟对象间的距离大于第一距离的光照探测球的球谐系数。It should be noted that, in practical applications, the electronic device can estimate the spherical harmonic coefficients of the light detection sphere whose distance to the virtual object is less than or equal to the first distance according to steps 5031 and 5032, and then according to the distance to the virtual object less than Or estimate the spherical harmonic coefficient of the virtual object by the spherical harmonic coefficient of the light detection sphere equal to the first distance, instead of estimating the spherical harmonic coefficient of the light detection sphere whose distance from the virtual object is greater than the first distance.
可选的，电子设备可以根据虚拟对象的球谐系数渲染虚拟对象。Optionally, the electronic device may render the virtual object according to the spherical harmonic coefficients of the virtual object.
电子设备在确定第一场景的场景类型为室内场景后，通过电子设备的主摄像头和深度传感器获取第一场景的图像信息，根据第一场景的图像信息建立包括至少两个光照探测球和点云的第一模拟场景，再根据第一模拟场景包括的点云的信息估计光照探测球的球谐系数，最后根据光照探测球的球谐系数估计所述虚拟对象的球谐系数，因为虚拟对象的球谐系数是根据光源点云和非光源点云的球谐系数估计的，并且该球谐系数包括光照方向，所以电子设备可以根据光照探测球的球谐系数估计出更接近真实场景的光照效果，以使得虚拟对象与真实场景更有效地融合，提升了用户体验。After the electronic device determines that the scene type of the first scene is an indoor scene, the electronic device obtains the image information of the first scene through the main camera and the depth sensor of the electronic device, and establishes an image including at least two illumination detection spheres and a point cloud according to the image information of the first scene. The first simulation scene, and then estimate the spherical harmonic coefficient of the illumination detection sphere according to the point cloud information included in the first simulation scene, and finally estimate the spherical harmonic coefficient of the virtual object according to the spherical harmonic coefficient of the illumination detection sphere, because the virtual object The spherical harmonic coefficient is estimated based on the spherical harmonic coefficient of the light source point cloud and the non-light source point cloud, and the spherical harmonic coefficient includes the direction of illumination, so the electronic device can estimate the illumination effect closer to the real scene based on the spherical harmonic coefficient of the light detection sphere , so that the virtual object can be more effectively integrated with the real scene, and the user experience is improved.
以上图5是对室内场景的光照估计进行说明，但是由于实际应用中存在室外和人脸等场景，当检测到室外场景时，可以采用下述图10所示方法。Figure 5 above illustrates the illumination estimation of indoor scenes, but since there are outdoor scenes and human faces in practical applications, when an outdoor scene is detected, the method shown in Figure 10 below can be used.
如图10所示，为本申请实施例提供的一种3D光照估计方法，该3D光照估计方法包括以下步骤：As shown in Figure 10, it is a 3D illumination estimation method provided in the embodiment of the present application, and the 3D illumination estimation method includes the following steps:
步骤1001、电子设备获取第一场景的图像信息。Step 1001, the electronic device acquires image information of a first scene.
其中，该第一场景可以是当前真实的场景，若第一场景的场景类型是室外场景，该第一场景的图像信息可以包括第一场景中像素点的颜色信息。Wherein, the first scene may be a current real scene, and if the scene type of the first scene is an outdoor scene, the image information of the first scene may include color information of pixels in the first scene.
其中，该第一场景中像素点的颜色信息可以用于指示第一场景中像素点的颜色，例如：第一场景中像素点的颜色信息可以是第一场景中像素点的RGB颜色值。该第一场景中像素点的颜色信息可以通过电子设备的一个或多个摄像头获取(例如：广角摄像头、长焦摄像头等)，例如：电子设备的一个或多个摄像头可以拍摄至少一张第一场景的图像或第一场景的视频，电子设备根据该至少一张第一场景的图像或第一场景的视频获取第一场景中像素点的颜色信息。若电子设备的一个或多个摄像头拍摄了两张或两张以上的图像，电子设备可以获取每张图像中像素点的颜色信息，并将两张或两张以上图像中像素点的颜色信息的平均值作为第一场景中像素点的颜色信息。若电子设备的一个或多个摄像头拍摄了10帧的第一场景的视频，电子设备可以获取每帧视频中像素点的颜色信息，并将10帧视频中像素点的颜色信息的平均值作为第一场景中像素点的颜色信息。Wherein, the color information of the pixel in the first scene may be used to indicate the color of the pixel in the first scene, for example: the color information of the pixel in the first scene may be the RGB color value of the pixel in the first scene. The color information of the pixels in the first scene can be obtained by one or more cameras of the electronic device (for example: wide-angle camera, telephoto camera, etc.), for example: one or more cameras of the electronic device can take at least one first The image of the scene or the video of the first scene, the electronic device obtains the color information of the pixels in the first scene according to the at least one image of the first scene or the video of the first scene. If one or more cameras of the electronic device take two or more images, the electronic device can obtain the color information of the pixels in each image, and combine the color information of the pixels in the two or more images The average value is used as color information of pixels in the first scene. If one or more cameras of the electronic device capture 10 frames of the video of the first scene, the electronic device can obtain the color information of the pixels in each frame of video, and use the average value of the color information of the pixels in the 10 frames of video as the first Color information of pixels in a scene.
示例性的，电子设备可以在确定第一场景的场景类型是室外场景时，自动通过电子设备的摄像头拍摄至少一张第一场景的图像或第一场景的视频，再根据该至少一张第一场景的图像或第一场景的视频获取第一场景的图像信息。Exemplarily, when the electronic device determines that the scene type of the first scene is an outdoor scene, it may automatically take at least one image or video of the first scene through the camera of the electronic device, and then The image of the scene or the video of the first scene acquires the image information of the first scene.
示例性的，电子设备可以在确定第一场景的场景类型是室外场景时，通过用户手动拍摄至少一张第一场景的图像或第一场景的视频，再根据该至少一张第一场景的图像或第一场景的视频获取第一场景的图像信息。Exemplarily, when the electronic device determines that the scene type of the first scene is an outdoor scene, the user manually takes at least one image or video of the first scene, and then according to the at least one image of the first scene Or the video of the first scene acquires the image information of the first scene.
步骤1002、电子设备根据第一场景的图像信息获取第一场景对应的天空图。Step 1002, the electronic device acquires a sky map corresponding to the first scene according to the image information of the first scene.
其中，该第一场景对应的天空图用于指示该第一场景的光照分布。例如，图11中第一行是不同光照分布对应的天空图，第二行是用第一行的天空图渲染的虚拟对象的图片。Wherein, the sky map corresponding to the first scene is used to indicate the illumination distribution of the first scene. For example, the first row in Figure 11 is the sky map corresponding to different lighting distributions, and the second row is the picture of the virtual object rendered with the sky map in the first row.
可选的，电子设备通过将第一场景的图像信息输入第一卷积神经网络(convolutional neural networks，CNN)获取第一场景对应的天空图。Optionally, the electronic device obtains the sky map corresponding to the first scene by inputting the image information of the first scene into a first convolutional neural network (convolutional neural networks, CNN).
示例性的，第一场景的图像信息可以经过第一CNN中的多层函数，如卷积层(convolutional layer)、池化层(pooling layer)、线性整流层(relu layer)、批归一化层(batch normalization layer)和损失函数层等之后，得到第一场景的图像信息对应的天空图。Exemplarily, the image information of the first scene may pass through multi-layer functions in the first CNN, such as convolutional layer, pooling layer, linear rectification layer (relu layer), batch normalization After batch normalization layer and loss function layer, etc., the sky map corresponding to the image information of the first scene is obtained.
步骤1003、电子设备根据天空图的信息估计虚拟对象的球谐系数。Step 1003, the electronic device estimates the spherical harmonic coefficients of the virtual object according to the information of the sky map.
其中，天空图的信息可以包括天空图的球谐系数。Wherein, the information of the sky map may include spherical harmonic coefficients of the sky map.
其中，虚拟对象可以是用户想通过AR应用显示在电子设备的显示屏上的由电子数据制作的可视化对象。例如，虚拟对象可以是一张图片、3D模型等。Wherein, the virtual object may be a visualized object made of electronic data that the user wants to display on the display screen of the electronic device through the AR application. For example, the virtual object can be a picture, a 3D model, and the like.
可选的，电子设备根据用户输入获取虚拟对象的位置；或者，电子设备预设置虚拟对象的位置。其中，虚拟对象的位置用于指示虚拟对象在第一模拟场景的空间坐标。Optionally, the electronic device acquires the position of the virtual object according to user input; or, the electronic device presets the position of the virtual object. Wherein, the position of the virtual object is used to indicate the space coordinates of the virtual object in the first simulation scene.
可选的，电子设备根据天空图的信息获取天空图的球谐系数，将该天空图的球谐系数作为虚拟对象的球谐系数，电子设备根据虚拟对象的球谐系数渲染虚拟对象。例如：电子设备根据第一场景的球谐系数获取虚拟对象的光照主方向、补偿色温和光照强度，电子设备根据该光照主方向、该补偿色温和该光照强度渲染虚拟对象。Optionally, the electronic device acquires the spherical harmonic coefficients of the sky map according to the information of the sky map, uses the spherical harmonic coefficients of the sky map as the spherical harmonic coefficients of the virtual object, and the electronic device renders the virtual object according to the spherical harmonic coefficients of the virtual object. For example, the electronic device obtains the main direction of illumination, the compensation color temperature and the light intensity of the virtual object according to the spherical harmonic coefficient of the first scene, and the electronic device renders the virtual object according to the main direction of light, the compensation color temperature and the light intensity.
电子设备在确定第一场景的场景类型为室外场景后，通过电子设备的一个或多个摄像头取第一场景的图像信息，根据第一场景的图像信息获取第一场景对应的天空图，再根据天空图的信息估计虚拟对象的球谐系数，因为虚拟对象的球谐系数包括光照方向，所以电子设备可以根据光照探测球的球谐系数估计出更接近真实场景的光照效果，以使得虚拟对象与真实场景更有效地融合，提升了用户体验。After determining that the scene type of the first scene is an outdoor scene, the electronic device obtains the image information of the first scene through one or more cameras of the electronic device, obtains the sky map corresponding to the first scene according to the image information of the first scene, and then according to The information of the sky map estimates the spherical harmonic coefficient of the virtual object, because the spherical harmonic coefficient of the virtual object includes the direction of illumination, so the electronic device can estimate the illumination effect closer to the real scene according to the spherical harmonic coefficient of the light detection sphere, so that the virtual object and The real scene is more effectively integrated and the user experience is improved.
以上图5、图10分别对室内场景、室外场景的光照估计进行说明，但是由于实际应用中存在人脸等场景，当检测到人脸场景时，可以采用下述图12所示方法。Figure 5 and Figure 10 above illustrate the illumination estimation of indoor scenes and outdoor scenes respectively. However, since there are scenes such as human faces in practical applications, when a human face scene is detected, the method shown in Figure 12 below can be used.
如图12所示，为本申请实施例提供的一种3D光照估计方法，该3D光照估计方法包括以下步骤：As shown in Figure 12, it is a 3D illumination estimation method provided in the embodiment of the present application, and the 3D illumination estimation method includes the following steps:
步骤1201、电子设备获取第一场景的图像信息。Step 1201, the electronic device acquires image information of a first scene.
其中，该第一场景可以是当前真实的场景，若第一场景的场景类型是人脸场景，该第一场景的图像信息可以包括第一场景中像素点的颜色信息。Wherein, the first scene may be a current real scene, and if the scene type of the first scene is a face scene, the image information of the first scene may include color information of pixels in the first scene.
需要说明的是，若第一场景的场景类型是人脸场景，第一场景中像素点的颜色信息的具体介绍可以参考步骤1001中的介绍。It should be noted that, if the scene type of the first scene is a face scene, the specific introduction of the color information of the pixels in the first scene can refer to the introduction in step 1001 .
示例性的，电子设备可以在确定第一场景的场景类型是人脸场景时，自动通过电子设备的摄像头拍摄至少一张第一场景的图像或第一场景的视频，再根据该至少一张第一场景的图像或第一场景的视频获取第一场景的图像信息。Exemplarily, when the electronic device determines that the scene type of the first scene is a face scene, it can automatically take at least one image or video of the first scene through the camera of the electronic device, and then according to the at least one first scene An image of a scene or a video of the first scene acquires image information of the first scene.
示例性的，电子设备可以在确定第一场景的场景类型是人脸场景时，通过用户手动拍摄至少一张第一场景的图像或第一场景的视频，再根据该至少一张第一场景的图像或第一场景的视频获取第一场景的图像信息。Exemplarily, when the electronic device determines that the scene type of the first scene is a face scene, the user manually takes at least one image or video of the first scene, and then according to the at least one image of the first scene The image or the video of the first scene acquires the image information of the first scene.
步骤1202、电子设备根据第一场景的图像信息估计虚拟对象的球谐系数。Step 1202, the electronic device estimates spherical harmonic coefficients of the virtual object according to the image information of the first scene.
可选的，电子设备根据第一场景的图像信息获取第一场景的球谐系数，该第一场景的球谐系数即为虚拟对象的球谐系数。电子设备可以根据虚拟对象的球谐系数渲染虚拟对象。例如：电子设备根据第一场景的球谐系数获取虚拟对象的光照主方向、补偿色温和光照强度，电子设备根据该光照主方向、该补偿色温和该光照强度渲染虚拟对象。Optionally, the electronic device acquires the spherical harmonic coefficients of the first scene according to the image information of the first scene, and the spherical harmonic coefficients of the first scene are the spherical harmonic coefficients of the virtual object. The electronic device may render the virtual object according to the spherical harmonic coefficients of the virtual object. For example, the electronic device obtains the main direction of illumination, the compensation color temperature and the light intensity of the virtual object according to the spherical harmonic coefficient of the first scene, and the electronic device renders the virtual object according to the main direction of light, the compensation color temperature and the light intensity.
可选的，电子设备通过将第一场景的图像信息输入第二CNN获取第一场景的球谐系数。Optionally, the electronic device obtains the spherical harmonic coefficients of the first scene by inputting the image information of the first scene into the second CNN.
示例性的，第一场景的图像信息可以经过第二CNN中的多层函数，如卷积层(convolutional layer)、池化层(pooling layer)、线性整流层(relu layer)、批归一化层(batch normalization layer)和损失函数层等之后，可以得到第一场景的球谐系数。Exemplarily, the image information of the first scene may pass through multi-layer functions in the second CNN, such as convolutional layer, pooling layer, linear rectification layer (relu layer), batch normalization After batch normalization layer and loss function layer, etc., the spherical harmonic coefficients of the first scene can be obtained.
可选的，该第一场景的球谐系数是第一场景中人脸的球谐系数。Optionally, the spherical harmonic coefficients of the first scene are spherical harmonic coefficients of faces in the first scene.
电子设备在确定第一场景的场景类型为人脸场景后，通过电子设备的一个或多个摄像头取第一场景的图像信息，根据第一场景的图像信息估计虚拟对象的球谐系数，因为虚拟对象的球谐系数包括光照方向，所以电子设备可以根据光照探测球的球谐系数估计出更接近真实场景的光照效果，以使得虚拟对象与真实场景更有效地融合，提升了用户体验。After the electronic device determines that the scene type of the first scene is a face scene, the image information of the first scene is obtained through one or more cameras of the electronic device, and the spherical harmonic coefficient of the virtual object is estimated according to the image information of the first scene, because the virtual object The spherical harmonic coefficient of the light detection sphere includes the direction of light, so the electronic device can estimate the lighting effect closer to the real scene according to the spherical harmonic coefficient of the light detection sphere, so that the virtual object and the real scene can be more effectively integrated, and the user experience is improved.
可以理解的是，上述电子设备为了实现上述功能，其包含了执行各个功能相应的硬件结构和/或软件模块。本领域技术人员应该很容易意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，本申请实施例能够以硬件或硬件和计算机软件的结合形式来实现。某个功能究竟以硬件还是计算机软件驱动硬件的方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本申请实施例的范围。It can be understood that, in order to realize the above-mentioned functions, the above-mentioned electronic device includes corresponding hardware structures and/or software modules for performing each function. Those skilled in the art should easily realize that the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software in combination with the example units and algorithm steps described in the embodiments disclosed herein. Whether a certain function is executed by hardware or computer software drives hardware depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the embodiments of the present application.
本申请实施例可以根据上述方法示例对上述电子设备进行功能模块的划分，例如，可以对应各个功能划分各个功能模块，也可以将两个或两个以上的功能集成在一个处理模块中。上述集成的模块既可以采用硬件的形式实现，也可以采用软件功能模块的形式实现。需要说明的是，本申请实施例中对模块的划分是示意性的，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式。The embodiments of the present application may divide the above-mentioned electronic device into functional modules according to the above-mentioned method examples. For example, each functional module may be divided corresponding to each function, or two or more functions may be integrated into one processing module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software function modules. It should be noted that the division of modules in the embodiment of the present application is schematic, and is only a logical function division, and there may be other division methods in actual implementation.
示例性的，在采用对应各个功能划分各个功能模块的情况下，图13示出了一种电子设备130的结构示意图。该电子设备130包括：获取模块1301、建立模块1302以及估计模块1303。Exemplarily, in the case of dividing each functional module corresponding to each function, FIG. 13 shows a schematic structural diagram of an electronic device 130 . The electronic device 130 includes: an acquisition module 1301 , an establishment module 1302 and an estimation module 1303 .
获取模块1301，用于获取第一场景的图像信息；其中，该第一场景的图像信息包括该第一场景中像素点的颜色信息和该第一场景中像素点的深度信息。The obtaining module 1301 is configured to obtain image information of a first scene; wherein, the image information of the first scene includes color information of pixels in the first scene and depth information of pixels in the first scene.
建立模块1302，用于根据该第一场景的图像信息建立第一模拟场景；其中，该第一模拟场景包括至少两个光照探测球以及点云。The establishment module 1302 is configured to establish a first simulation scene according to the image information of the first scene; wherein, the first simulation scene includes at least two light probe spheres and a point cloud.
估计模块1303，用于根据该第一模拟场景包括的点云的信息估计光照探测球的球谐系数；其中，该点云的信息包括该点云中的点的位置信息和辐照度信息。The estimation module 1303 is configured to estimate the spherical harmonic coefficient of the illumination probe sphere according to the point cloud information included in the first simulation scene; wherein the point cloud information includes position information and irradiance information of points in the point cloud.
估计模块1303，还用于根据该光照探测球的球谐系数估计该虚拟对象的位置的球谐系数。The estimation module 1303 is further configured to estimate the spherical harmonic coefficient of the position of the virtual object according to the spherical harmonic coefficient of the illumination detection sphere.
可选的，估计模块1303，具体用于根据点云的信息获取该第一光照探测球的第一辐照度；估计模块1303，还具体用于根据该第一光照探测球的第一辐照度估计该第一光照探测球的球谐系数；其中，该第一光照探测球为该第一模拟场景包括的任一光照探测球。Optionally, the estimation module 1303 is specifically configured to obtain the first irradiance of the first illumination detection sphere according to point cloud information; the estimation module 1303 is also specifically configured to obtain the first irradiance of the first illumination detection sphere according to the Estimate the spherical harmonic coefficient of the first illumination probe sphere; wherein, the first illumination probe sphere is any illumination probe sphere included in the first simulation scene.
可选的，获取模块1301，具体用于根据第一有效点的信息获取第一光照探测球的第一辐照度，其中，第一有效点是点云中在该第一光照探测球的可见范围内的点。Optionally, the acquisition module 1301 is specifically configured to acquire the first irradiance of the first illumination probe sphere according to the information of the first effective point, where the first effective point is the visible area of the first illumination probe sphere in the point cloud. points in the range.
可选的，估计模块1303，还具体用于对该光照探测球的球谐系数进行加权求和，得到该虚拟对象的位置的球谐系数。Optionally, the estimation module 1303 is also specifically configured to perform weighted summation of the spherical harmonic coefficients of the illumination detection sphere to obtain the spherical harmonic coefficients of the position of the virtual object.
可选的，该光照探测球与该虚拟对象间的距离小于或等于第一距离。Optionally, the distance between the light detection sphere and the virtual object is less than or equal to the first distance.
可选的，获取模块1301，还用于根据用户输入获取虚拟对象的位置；或者，预设置虚拟对象的位置。Optionally, the acquiring module 1301 is also configured to acquire the position of the virtual object according to user input; or, preset the position of the virtual object.
其中，上述方法实施例涉及的各操作的所有相关内容均可以援引到对应功能模块的功能描述，在此不再赘述。Wherein, all relevant content of each operation involved in the above method embodiment can be referred to the function description of the corresponding function module, and will not be repeated here.
在本实施例中，该电子设备130以采用集成的方式划分各个功能模块的形式来呈现。这里的“模块”可以指特定ASIC，电路，执行一个或多个软件或固件程序的处理器和存储器，集成逻辑电路，和/或其他可以提供上述功能的器件。在一个简单的实施例中，本领域的技术人员可以想到该电子设备130可以采用图1所示的形式。In this embodiment, the electronic device 130 is presented in a form of dividing various functional modules in an integrated manner. A "module" here may refer to a specific ASIC, a circuit, a processor and a memory executing one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the functions described above. In a simple embodiment, those skilled in the art can imagine that the electronic device 130 can take the form shown in FIG. 1 .
比如，图1中的处理器110可以通过调用内部存储器121中存储的计算机执行指令，使得电子设备130执行上述方法实施例中的3D光照估计方法。For example, the processor 110 in FIG. 1 may cause the electronic device 130 to execute the 3D illumination estimation method in the foregoing method embodiments by invoking computer-executed instructions stored in the internal memory 121 .
示例性的，图13中的获取模块1301、建立模块1302以及估计模块1303的功能/实现过程可以通过图1中的处理器110调用内部存储器121中存储的计算机执行指令来实现。Exemplarily, the functions/implementation process of the acquisition module 1301 , the establishment module 1302 and the estimation module 1303 in FIG. 13 can be implemented by the processor 110 in FIG. 1 invoking computer-executed instructions stored in the internal memory 121 .
由于本实施例提供的电子设备130可执行上述的3D光照估计方法，因此其所能获得的技术效果可参考上述方法实施例，在此不再赘述。Since the electronic device 130 provided in this embodiment can execute the above-mentioned 3D illumination estimation method, the technical effect it can obtain can refer to the above-mentioned method embodiment, and will not be repeated here.
示例性的，在采用对应各个功能划分各个功能模块的情况下，图14示出了一种电子设备140的结构示意图。该电子设备140包括：获取模块1401以及估计模块1402。Exemplarily, FIG. 14 shows a schematic structural diagram of an electronic device 140 in the case of dividing each functional module corresponding to each function. The electronic device 140 includes: an acquisition module 1401 and an estimation module 1402 .
获取模块1401，用于获取该第一场景的图像信息；其中，该第一场景的图像信息包括该第一场景中像素点的颜色信息。The obtaining module 1401 is configured to obtain image information of the first scene; wherein, the image information of the first scene includes color information of pixels in the first scene.
获取模块1401，还用于根据该第一场景的图像信息获取该第一场景对应的天空图；其中，该第一场景对应的天空图用于指示该第一场景的光照分布。The acquiring module 1401 is further configured to acquire a sky map corresponding to the first scene according to the image information of the first scene; wherein, the sky map corresponding to the first scene is used to indicate the illumination distribution of the first scene.
估计模块1402，用于根据该天空图的信息估计该虚拟对象的球谐系数，其中，该天空图的信息包括天空图的球谐系数。The estimation module 1402 is configured to estimate the spherical harmonic coefficients of the virtual object according to the information of the sky map, wherein the information of the sky map includes the spherical harmonic coefficients of the sky map.
可选的，估计模块1402，具体用于将该天空图的球谐系数作为该虚拟对象的球谐系数。Optionally, the estimation module 1402 is specifically configured to use the spherical harmonic coefficients of the sky map as the spherical harmonic coefficients of the virtual object.
可选的，获取模块1401，还用于根据用户输入获取虚拟对象的位置；或者，预设置虚拟对象的位置。Optionally, the acquiring module 1401 is also configured to acquire the position of the virtual object according to user input; or, preset the position of the virtual object.
其中，上述方法实施例涉及的各操作的所有相关内容均可以援引到对应功能模块的功能描述，在此不再赘述。Wherein, all relevant content of each operation involved in the above method embodiment can be referred to the function description of the corresponding function module, and will not be repeated here.
在本实施例中，该电子设备140以采用集成的方式划分各个功能模块的形式来呈现。这里的“模块”可以指特定ASIC，电路，执行一个或多个软件或固件程序的处理器和存储器，集成逻辑电路，和/或其他可以提供上述功能的器件。在一个简单的实施例中，本领域的技术人员可以想到该电子设备140可以采用图1所示的形式。In this embodiment, the electronic device 140 is presented in the form of dividing various functional modules in an integrated manner. A "module" here may refer to a specific ASIC, a circuit, a processor and a memory executing one or more software or firmware programs, an integrated logic circuit, and/or other devices that can provide the functions described above. In a simple embodiment, those skilled in the art can imagine that the electronic device 140 can take the form shown in FIG. 1 .
比如，图1中的处理器110可以通过调用内部存储器121中存储的计算机执行指令，使得电子设备140执行上述方法实施例中的3D光照估计方法。For example, the processor 110 in FIG. 1 may cause the electronic device 140 to execute the 3D illumination estimation method in the foregoing method embodiments by invoking computer-executed instructions stored in the internal memory 121 .
示例性的，图14中的获取模块1401以及估计模块1402的功能/实现过程可以通过图1中的处理器110调用内部存储器121中存储的计算机执行指令来实现。Exemplarily, the functions/implementation process of the acquiring module 1401 and the estimating module 1402 in FIG. 14 may be implemented by the processor 110 in FIG. 1 invoking computer-executed instructions stored in the internal memory 121 .
由于本实施例提供的电子设备140可执行上述的3D光照估计方法，因此其所能获得的技术效果可参考上述方法实施例，在此不再赘述。Since the electronic device 140 provided in this embodiment can execute the above-mentioned 3D illumination estimation method, the technical effect it can obtain can refer to the above-mentioned method embodiment, and will not be repeated here.
通过以上的实施方式的描述，所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将装置的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and brevity of the description, only the division of the above-mentioned functional modules is used as an example for illustration. In practical applications, the above-mentioned functions can be allocated according to needs It is completed by different functional modules, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.
在本申请所提供的几个实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述模块或单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个装置，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be Incorporation or may be integrated into another device, or some features may be omitted, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是一个物理单元或多个物理单元，即可以位于一个地方，或者也可以分布到多个不同地方。可以根据实际的需要选择其中的部分或者全部单元来实现本实施例方案的目的。The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may be one physical unit or multiple physical units, that is, it may be located in one place, or may be distributed to multiple different places . Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.
另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。上述集成的单元既可以使用硬件的形式实现，也可以使用软件功能单元的形式实现。In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.
所述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个可读取存储介质中。基于这样的理解，本申请实施例的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的全部或部分可以以软件产品的形式体现出来，该软件产品存储在一个存储介质中，包括若干指令用以使得一个设备(可以是单片机，芯片等)或处理器(processor)执行本申请各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。If the integrated unit is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the software product is stored in a storage medium Among them, several instructions are included to make a device (which may be a single-chip microcomputer, a chip, etc.) or a processor (processor) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: various media capable of storing program codes such as U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk.
以上所述，仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何在本申请揭露的技术范围内的变化或替换，都应涵盖在本申请的保护范围之内。因此，本申请的保护范围应以所述权利要求的保护范围为准。The above is only a specific implementation of the application, but the protection scope of the application is not limited thereto, and any changes or replacements within the technical scope disclosed in the application should be covered within the protection scope of the application . Therefore, the protection scope of the present application should be determined by the protection scope of the claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910234517X | 2019-03-26 | ||
| CN201910234517 | 2019-03-26 |
| Publication Number | Publication Date |
|---|---|
| CN110458902Atrue CN110458902A (en) | 2019-11-15 |
| CN110458902B CN110458902B (en) | 2022-04-05 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910586485.XAActiveCN110458902B (en) | 2019-03-26 | 2019-07-01 | 3D illumination estimation method and electronic equipment |
| Country | Link |
|---|---|
| CN (1) | CN110458902B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021103137A1 (en)* | 2019-11-28 | 2021-06-03 | 浙江大学 | Indoor scene illumination estimation model, method and device, and storage medium and rendering method |
| WO2021151380A1 (en)* | 2020-01-30 | 2021-08-05 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method for rendering virtual object based on illumination estimation, method for training neural network, and related products |
| CN113298588A (en)* | 2020-06-19 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Method and device for providing object information and electronic equipment |
| CN113537193A (en)* | 2021-07-15 | 2021-10-22 | Oppo广东移动通信有限公司 | Illumination estimation method, illumination estimation device, storage medium, and electronic apparatus |
| CN113537194A (en)* | 2021-07-15 | 2021-10-22 | Oppo广东移动通信有限公司 | Illumination estimation method, illumination estimation device, storage medium and electronic device |
| CN114125310A (en)* | 2022-01-26 | 2022-03-01 | 荣耀终端有限公司 | Photographing method, terminal device and cloud server |
| CN114979457A (en)* | 2021-02-26 | 2022-08-30 | 华为技术有限公司 | Image processing method and related device |
| CN115375827A (en)* | 2022-07-21 | 2022-11-22 | 荣耀终端有限公司 | Illumination Estimation Method and Electronic Device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110182352A1 (en)* | 2005-03-31 | 2011-07-28 | Pace Charles P | Feature-Based Video Compression |
| CN104268923A (en)* | 2014-09-04 | 2015-01-07 | 无锡梵天信息技术股份有限公司 | Illumination method based on picture level images |
| US20150279113A1 (en)* | 2014-03-25 | 2015-10-01 | Metaio Gmbh | Method and system for representing a virtual object in a view of a real environment |
| CN108235053A (en)* | 2016-12-19 | 2018-06-29 | 中国电信股份有限公司 | Interactive rendering intent, equipment and system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110182352A1 (en)* | 2005-03-31 | 2011-07-28 | Pace Charles P | Feature-Based Video Compression |
| US20150279113A1 (en)* | 2014-03-25 | 2015-10-01 | Metaio Gmbh | Method and system for representing a virtual object in a view of a real environment |
| CN106133796A (en)* | 2014-03-25 | 2016-11-16 | Metaio有限公司 | Method and system for representing virtual objects in a view of a real environment |
| CN104268923A (en)* | 2014-09-04 | 2015-01-07 | 无锡梵天信息技术股份有限公司 | Illumination method based on picture level images |
| CN108235053A (en)* | 2016-12-19 | 2018-06-29 | 中国电信股份有限公司 | Interactive rendering intent, equipment and system |
| Title |
|---|
| 何菲菲: ""基于图像的室外场景虚实融合关键技术研究"", 《中国优秀硕士学位全文数据库 信息科技辑》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021103137A1 (en)* | 2019-11-28 | 2021-06-03 | 浙江大学 | Indoor scene illumination estimation model, method and device, and storage medium and rendering method |
| WO2021151380A1 (en)* | 2020-01-30 | 2021-08-05 | Guangdong Oppo Mobile Telecommunications Corp., Ltd. | Method for rendering virtual object based on illumination estimation, method for training neural network, and related products |
| CN113298588A (en)* | 2020-06-19 | 2021-08-24 | 阿里巴巴集团控股有限公司 | Method and device for providing object information and electronic equipment |
| CN114979457A (en)* | 2021-02-26 | 2022-08-30 | 华为技术有限公司 | Image processing method and related device |
| CN114979457B (en)* | 2021-02-26 | 2023-04-07 | 华为技术有限公司 | Image processing method and related device |
| CN113537193A (en)* | 2021-07-15 | 2021-10-22 | Oppo广东移动通信有限公司 | Illumination estimation method, illumination estimation device, storage medium, and electronic apparatus |
| CN113537194A (en)* | 2021-07-15 | 2021-10-22 | Oppo广东移动通信有限公司 | Illumination estimation method, illumination estimation device, storage medium and electronic device |
| CN113537194B (en)* | 2021-07-15 | 2024-08-02 | Oppo广东移动通信有限公司 | Illumination estimation method, illumination estimation device, storage medium, and electronic apparatus |
| CN114125310A (en)* | 2022-01-26 | 2022-03-01 | 荣耀终端有限公司 | Photographing method, terminal device and cloud server |
| CN115375827A (en)* | 2022-07-21 | 2022-11-22 | 荣耀终端有限公司 | Illumination Estimation Method and Electronic Device |
| CN115375827B (en)* | 2022-07-21 | 2023-09-15 | 荣耀终端有限公司 | Illumination estimation method and electronic device |
| Publication number | Publication date |
|---|---|
| CN110458902B (en) | 2022-04-05 |
| Publication | Publication Date | Title |
|---|---|---|
| CN110458902B (en) | 3D illumination estimation method and electronic equipment | |
| US12086957B2 (en) | Image bloom processing method and apparatus, and storage medium | |
| CN113395382B (en) | A method for data interaction between devices and related devices | |
| CN114727220B (en) | Equipment searching method and electronic equipment | |
| CN111770282B (en) | Image processing method and device, computer readable medium and terminal equipment | |
| CN111835907A (en) | A method, device and system for transferring services across electronic devices | |
| CN114880251B (en) | Storage unit access method, access device and terminal equipment | |
| WO2022100685A1 (en) | Drawing command processing method and related device therefor | |
| WO2022193989A1 (en) | Operation method and apparatus for electronic device and electronic device | |
| WO2022156555A1 (en) | Screen brightness adjustment method, apparatus, and terminal device | |
| WO2023005882A1 (en) | Photographing method, photographing parameter training method, electronic device, and storage medium | |
| WO2022022319A1 (en) | Image processing method, electronic device, image processing system and chip system | |
| WO2023005706A1 (en) | Device control method, electronic device, and storage medium | |
| CN113518189B (en) | Shooting method, system, electronic device and storage medium | |
| CN113496477B (en) | Screen detection method and electronic equipment | |
| CN112037157A (en) | Data processing method and device, computer readable medium and electronic equipment | |
| CN113391735A (en) | Display form adjusting method and device, electronic equipment and storage medium | |
| CN114661258A (en) | Adaptive display method, electronic device, and storage medium | |
| CN109285563B (en) | Voice data processing method and device in online translation process | |
| WO2023030067A1 (en) | Remote control method, remote control device and controlled device | |
| CN114584652B (en) | A user graphical interface display method, device, computer equipment and storage medium | |
| CN114332331A (en) | Image processing method and device | |
| CN115599198A (en) | Figure image display method, electronic device and storage medium | |
| CN116419003A (en) | Building coding method and related device | |
| CN117119314A (en) | Image processing method and related electronic equipment |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |