技术领域Technical Field
本申请属于人工智能技术领域，具体涉及一种模型训练方法、图像曝光方法、装置及电子设备。The present application belongs to the field of artificial intelligence technology, and specifically relates to a model training method, an image exposure method, a device and an electronic device.
背景技术Background technique
目前，随着电子设备拍摄性能快速发展，以及人们对于图像审美的要求逐渐提高，人们越来越希望拍摄到的场景接近人眼看到的场景，即真实性。其中有一个比较重要的影响因素为：曝光控制，主要用于调整图像亮度，当前拍摄功能都是由相机自动控制曝光。At present, with the rapid development of electronic equipment shooting performance and people's increasing requirements for image aesthetics, people increasingly hope that the scenes they shoot are close to what the human eye sees, that is, authenticity. One of the more important influencing factors is exposure control, which is mainly used to adjust the image brightness. The current shooting function is automatically controlled by the camera.
相机在自动控制曝光时，分为测光、曝光调整两个主要步骤，其中测光指测量当前物体的反射率，以得到图像亮度，再根据图像亮度进行曝光设置，使成像的亮度接近真实的场景亮度。在曝光设置时，现有技术中有一个“18％灰”原则，即假设图像整体的平均反射率为18％，这是因为，18％灰可以将人们眼中的绝大部分景物都能够展现出来，相机依此作为曝光依据，来获取曝光结果。其中，若拍摄物体的实际反射率较高，比如雪景、白色桌面等，则会存在欠曝光的情况，导致图像整体偏灰；而当拍摄物体为反射率较低，比如黑色桌面，黑色汽车等，则会存在过曝光的情况，导致图像整体偏亮。同时，若拍摄场景存在光源，仍然按照“18％灰”的假设进行曝光设置，就会导致图像整体偏暗。可见，在现有技术中，存在曝光结果不准确的问题，从而导致拍摄图像质量差。When the camera automatically controls exposure, it is divided into two main steps: light metering and exposure adjustment. Light metering refers to measuring the reflectivity of the current object to obtain the image brightness, and then setting the exposure according to the image brightness so that the image brightness is close to the actual scene brightness. When setting the exposure, there is a "18% gray" principle in the prior art, that is, assuming that the average reflectivity of the entire image is 18%. This is because 18% gray can show most of the scenery in people's eyes. The camera uses this as the exposure basis to obtain the exposure result. Among them, if the actual reflectivity of the photographed object is high, such as snow scenes, white desktops, etc., there will be underexposure, resulting in the overall gray image; and when the photographed object has a low reflectivity, such as a black desktop, a black car, etc., there will be overexposure, resulting in the overall bright image. At the same time, if there is a light source in the shooting scene, the exposure setting is still performed according to the "18% gray" assumption, which will cause the overall image to be dark. It can be seen that in the prior art, there is a problem of inaccurate exposure results, which leads to poor quality of the captured image.
发明内容Summary of the invention
本申请实施例的目的是提供一种模型训练方法，能够准确地输出拍摄物体的反射率，而不是默认的灰卡的反射率，同时可以识别出存在光源的区域，使得拍摄过程中测量图像亮度的准确率提高，从而可以提高设置曝光参数的准确率，避免出现过曝光、欠曝光的现象，提高图像质量。The purpose of the embodiment of the present application is to provide a model training method that can accurately output the reflectivity of the photographed object instead of the default reflectivity of the gray card, and at the same time can identify the area where the light source exists, so that the accuracy of measuring the image brightness during the shooting process is improved, thereby improving the accuracy of setting exposure parameters, avoiding overexposure and underexposure, and improving image quality.
第一方面，本申请实施例提供了一种模型训练方法，该方法包括：将一张训练图像的N个网格中每个网格的光源预测值、与所述N个网格相关的真实反射率输入第一模型；其中，所述N个网格中光源网格的光源预测值为1，所述N个网格中非光源网格的光源预测值为0，N为正整数；将所述训练图像输入所述第一模型，通过所述第一模型，根据所述训练图像的N个网格的图像特征，输出所述N个网格中每个网格的光源概率、与所述N个网格相关的训练反射率；其中，所述光源概率为网格为光源网格的概率；根据所述训练图像的N个网格中每个网格的光源预测值、与所述N个网格相关的真实反射率、所述N个网格中每个网格的光源概率和与所述N个网格相关的训练反射率，对所述第一模型进行训练，得到第二模型。In a first aspect, an embodiment of the present application provides a model training method, the method comprising: inputting a light source prediction value of each of N grids in a training image and a true reflectivity associated with the N grids into a first model; wherein the light source prediction value of the light source grid in the N grids is 1, and the light source prediction value of the non-light source grid in the N grids is 0, and N is a positive integer; inputting the training image into the first model, and through the first model, outputting the light source probability of each of the N grids and the training reflectivity associated with the N grids according to the image features of the N grids in the training image; wherein the light source probability is the probability that a grid is a light source grid; training the first model according to the light source prediction value of each of the N grids in the training image, the true reflectivity associated with the N grids, the light source probability of each of the N grids and the training reflectivity associated with the N grids to obtain a second model.
第二方面，本申请实施例提供了一种图像曝光方法，该方法包括：将一张预览图像输入第一方面所述的第二模型，通过所述第二模型输出：所述预览图像的N个网格中每个网格的光源概率和与所述N个网格相关的训练反射率；根据所述N个网格中与非光源网格相关的训练反射率和灰卡的反射率，计算亮度比例；其中，将光源概率小于第二阈值的网格确定为非光源网格；根据参考图像亮度和所述亮度比例，设置曝光参数；其中，所述参考图像亮度是对所述预览图像进行测光得到的。In a second aspect, an embodiment of the present application provides an image exposure method, the method comprising: inputting a preview image into the second model described in the first aspect, and outputting through the second model: the light source probability of each grid in N grids of the preview image and the training reflectivity associated with the N grids; calculating the brightness ratio based on the training reflectivity associated with non-light source grids in the N grids and the reflectivity of the gray card; wherein the grids whose light source probability is less than a second threshold are determined as non-light source grids; setting exposure parameters based on the reference image brightness and the brightness ratio; wherein the reference image brightness is obtained by measuring the preview image.
第三方面，本申请实施例提供了一种模型训练装置，该装置包括：输入模块，用于将一张训练图像的N个网格中每个网格的光源预测值、与所述N个网格相关的真实反射率输入第一模型；其中，所述N个网格中光源网格的光源预测值为1，所述N个网格中非光源网格的光源预测值为0，N为正整数；输出模块，用于将所述训练图像输入所述第一模型，通过所述第一模型，根据所述训练图像的N个网格的图像特征，输出所述N个网格中每个网格的光源概率、与所述N个网格相关的训练反射率；其中，所述光源概率为网格为光源网格的概率；训练模块，用于根据所述训练图像的N个网格中每个网格的光源预测值、与所述N个网格相关的真实反射率、所述N个网格中每个网格的光源概率和与所述N个网格相关的训练反射率，对所述第一模型进行训练，得到第二模型。In a third aspect, an embodiment of the present application provides a model training device, which includes: an input module, used to input the light source prediction value of each grid in N grids of a training image and the real reflectivity associated with the N grids into a first model; wherein the light source prediction value of the light source grid in the N grids is 1, and the light source prediction value of the non-light source grid in the N grids is 0, and N is a positive integer; an output module, used to input the training image into the first model, and through the first model, according to the image features of the N grids of the training image, output the light source probability of each grid in the N grids and the training reflectivity associated with the N grids; wherein the light source probability is the probability that a grid is a light source grid; a training module, used to train the first model according to the light source prediction value of each grid in the N grids of the training image, the real reflectivity associated with the N grids, the light source probability of each grid in the N grids and the training reflectivity associated with the N grids, to obtain a second model.
第四方面，本申请实施例提供了一种图像曝光装置，该装置包括：输出模块，用于将一张预览图像输入第一方面所述的第二模型，通过所述第二模型输出：所述预览图像的N个网格中每个网格的光源概率和与所述N个网格相关的训练反射率；计算模块，用于根据所述N个网格中与非光源网格相关的训练反射率和灰卡的反射率，计算亮度比例；其中，将光源概率小于第二阈值的网格确定为非光源网格；设置模块，用于根据参考图像亮度和所述亮度比例，设置曝光参数；其中，所述参考图像亮度是对所述预览图像进行测光得到的。。In a fourth aspect, an embodiment of the present application provides an image exposure device, the device comprising: an output module, used to input a preview image into the second model described in the first aspect, and output through the second model: the light source probability of each grid in the N grids of the preview image and the training reflectivity associated with the N grids; a calculation module, used to calculate the brightness ratio according to the training reflectivity associated with the non-light source grids in the N grids and the reflectivity of the gray card; wherein the grid with a light source probability less than a second threshold is determined as a non-light source grid; a setting module, used to set exposure parameters according to the reference image brightness and the brightness ratio; wherein the reference image brightness is obtained by measuring the preview image. .
第五方面，本申请实施例提供了一种电子设备，包括处理器和存储器，所述存储器存储可在所述处理器上运行的程序或指令，所述程序或指令被所述处理器执行时实现如第一方面所述的步骤。In a fifth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps described in the first aspect are implemented.
第六方面，本申请实施例提供了一种电子设备，包括处理器和存储器，所述存储器存储可在所述处理器上运行的程序或指令，所述程序或指令被所述处理器执行时实现如第二方面所述的步骤。In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps described in the second aspect are implemented.
第七方面，本申请实施例提供了一种可读存储介质，所述可读存储介质上存储程序或指令，所述程序或指令被处理器执行时实现如第一方面所述的方法的步骤。In a seventh aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.
第八方面，本申请实施例提供了一种可读存储介质，所述可读存储介质上存储程序或指令，所述程序或指令被处理器执行时实现如第二方面所述的方法的步骤In an eighth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the second aspect are implemented.
第九方面，本申请实施例提供了一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现如第一方面所述的方法。In a ninth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
第十方面，本申请实施例提供了一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现如第一方面所述的方法。In the tenth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.
第十一方面，本申请实施例提供了一种芯片，所述芯片包括处理器和通信接口，所述通信接口和所述处理器耦合，所述处理器用于运行程序或指令，实现如第二方面所述的方法。In the eleventh aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the second aspect.
第十二方面，本申请实施例提供一种计算机程序产品，该程序产品被存储在存储介质中，该程序产品被至少一个处理器执行以实现如第一方面所述的方法。In a twelfth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.
第十三方面，本申请实施例提供一种计算机程序产品，该程序产品被存储在存储介质中，该程序产品被至少一个处理器执行以实现如第二方面所述的方法。In a thirteenth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the second aspect.
在本申请的实施例中，首先准备训练数据，训练数据包括一张训练图像、训练图像中每个网格的光源预测值、与N个网格相关的真实反射率，其中，网格是光源网格时，光源预测值为1，网格是非光源网格时，光源预测值为0。然后将训练图像、与N个网格相关的真实反射率和每个网格的光源预测值输入待训练的第一模型，同时第一模型对训练图像进行图像特征提取，输出每个网格为光源网格的光源概率和与N个网格相关的训练反射率。最后，基于光源预测值、真实反射率、光源概率、训练反射率，对第一模型进行训练，得到第二模型，使得第二模型输出的光源概率尽可能地接近输入的光源预测值、输出的训练反射率尽可能地接近输入的真实反射率。可见，在本申请的实施例中，对模型进行训练后，模型可以准确地输出拍摄物体的反射率，而不是默认的灰卡的反射率，同时可以识别出存在光源的区域，使得拍摄过程中测量图像亮度的准确率提高，从而可以提高设置曝光参数的准确率，避免出现过曝光、欠曝光的现象，提升图像质量。In an embodiment of the present application, training data is first prepared, and the training data includes a training image, a light source prediction value of each grid in the training image, and a true reflectivity associated with N grids, wherein when a grid is a light source grid, the light source prediction value is 1, and when a grid is a non-light source grid, the light source prediction value is 0. Then the training image, the true reflectivity associated with the N grids, and the light source prediction value of each grid are input into the first model to be trained, and at the same time, the first model extracts image features from the training image, and outputs the light source probability that each grid is a light source grid and the training reflectivity associated with the N grids. Finally, based on the light source prediction value, the true reflectivity, the light source probability, and the training reflectivity, the first model is trained to obtain a second model, so that the light source probability output by the second model is as close as possible to the input light source prediction value, and the output training reflectivity is as close as possible to the input true reflectivity. It can be seen that in the embodiments of the present application, after the model is trained, the model can accurately output the reflectivity of the photographed object instead of the reflectivity of the default gray card, and can also identify areas where light sources exist, thereby improving the accuracy of measuring image brightness during the shooting process, thereby improving the accuracy of setting exposure parameters, avoiding overexposure and underexposure, and improving image quality.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
图1是本申请的一些实施例提供的模型训练方法的流程图；FIG1 is a flow chart of a model training method provided by some embodiments of the present application;
图2是本申请的一些实施例提供的模型训练方法的示意图；FIG2 is a schematic diagram of a model training method provided by some embodiments of the present application;
图3是本申请的一些实施例提供的图像曝光方法的流程图；FIG3 is a flow chart of an image exposure method provided by some embodiments of the present application;
图4是本申请的一些实施例提供的模型训练装置的的框图；FIG4 is a block diagram of a model training device provided in some embodiments of the present application;
图5是本申请的一些实施例提供的图像曝光装置的的框图；FIG5 is a block diagram of an image exposure device provided by some embodiments of the present application;
图6是本申请的一些实施例提供的电子设备的硬件结构示意图；FIG6 is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application;
图7是本申请的一些实施例提供的电子设备的硬件结构示意图。FIG. 7 is a schematic diagram of the hardware structure of an electronic device provided in some embodiments of the present application.
具体实施方式Detailed ways
下面将结合本申请实施例的附图，对本申请实施例的技术方案进行清楚地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请的实施例，本领域普通技术人员获得的所有其他实施例，都属于本申请保护的范围。The following will be combined with the drawings of the embodiments of the present application to clearly describe the technical solutions of the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present application belong to the scope of protection of this application.
本申请的说明书和权利要求书的术语“第一”、“第二”等是用于区别类似的对象，而不用于描述特定的顺序或先后次序。应该理解这样使用的术语在适当情况下可以互换，以便本申请的实施例能够以除了在这里图示或描述的那些以外的顺序实施，且“第一”、“第二”等所区分的对象通常为一类，并不限定对象的个数，例如目标对象可以是一个，也可以是多个。此外，说明书以及权利要求中“和/或”表示所连接对象的至少其中之一，字符“/”，一般表示前后关联对象是一种“或”的关系。The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally a class, and the number of objects is not limited. For example, the target object can be one or more. In addition, "and/or" in the specification and claims represents at least one of the connected objects, and the character "/" generally indicates that the objects associated before and after are in an "or" relationship.
下面结合附图，通过具体的实施例及其应用场景对本申请实施例提供的模型训练方法进行详细地说明。The model training method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
需要说明的是，本申请实施例提供的模型训练方法，执行主体可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备等电子设备。本申请的一些实施例中以电子设备为执行主体执行模型训练方法为例，说明本申请实施例提供的模型训练方法。It should be noted that the model training method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the model training method provided in the embodiments of the present application is illustrated by taking the electronic device as the execution subject to execute the model training method.
本申请实施例提供的模型训练方法可以应用于拍摄图像中涉及到模型训练的场景，其中，一种具体的应用场景是，在拍摄图像之前，对第一模型进行训练，得到第二模型，在拍摄图像时，由第二模型输出拍摄场景中的物体反射率和光源区域，可用于调整测光得到图像亮度。The model training method provided in the embodiment of the present application can be applied to scenarios involving model training in captured images. A specific application scenario is that before capturing an image, a first model is trained to obtain a second model. When capturing an image, the second model outputs the reflectivity of objects and the light source area in the captured scene, which can be used to adjust the light measurement to obtain image brightness.
如图1所示，示出了本申请一个实施例的模型训练方法的流程图，以该方法应用于电子设备进行举例，该方法包括：As shown in FIG. 1 , a flow chart of a model training method according to an embodiment of the present application is shown. Taking the method applied to an electronic device as an example, the method includes:
步骤110：将一张训练图像的N个网格中每个网格的光源预测值、与N个网格相关的真实反射率输入第一模型。其中，所述N个网格中光源网格的光源预测值为1，N个网格中非光源网格的光源预测值为0，N为正整数。Step 110: Input the light source prediction value of each grid in N grids of a training image and the true reflectivity associated with the N grids into the first model, wherein the light source prediction value of the light source grid in the N grids is 1, the light source prediction value of the non-light source grid in the N grids is 0, and N is a positive integer.
在该步骤中，利用一张训练图像对第一模型进行训练，当使用多张训练图像对第一模型进行训练时，可重复本实施例的各个步骤。In this step, the first model is trained using one training image. When multiple training images are used to train the first model, the steps of this embodiment can be repeated.
可选地，相机测光时，通常使用64*48的网格来测光，因此，N≤64*48，或者N稍大于64*48一些也可以。例如，N的取值可以是以下任一个：16*16、14*14、64*64、48×48。其中，N为2的整数倍，方便后续计算。Optionally, when the camera measures light, a 64*48 grid is usually used for measuring light, so N≤64*48, or N slightly larger than 64*48 is also acceptable. For example, the value of N can be any of the following: 16*16, 14*14, 64*64, 48×48. Among them, N is an integer multiple of 2, which is convenient for subsequent calculations.
可参考地，在将训练图像输入第一模型之前，对训练图像进行处理，将训练图像划分为16*16个网格，并计算出N个网格的真实反射率。进一步地，根据N个网格的真实反射率，确定每个网格是否为光源网格。其中。光源网格即存在光源的网格。For reference, before inputting the training image into the first model, the training image is processed, the training image is divided into 16*16 grids, and the true reflectivity of the N grids is calculated. Further, according to the true reflectivity of the N grids, it is determined whether each grid is a light source grid. Among them, the light source grid is a grid where a light source exists.
在后续的步骤中，需要通过第一模型针对训练图像输出每个网格为光源网格的概率，概率的取值范围为0-1，因此，在对训练图像进行处理的过程中，设置光源网格的光源预测值为1，设置非光源网格的光源预测值为0，以便于与概率进行比较。In the subsequent steps, the first model needs to output the probability that each grid is a light source grid for the training image, and the probability value range is 0-1. Therefore, in the process of processing the training image, the light source prediction value of the light source grid is set to 1, and the light source prediction value of the non-light source grid is set to 0, so as to facilitate comparison with the probability.
步骤120：将训练图像输入第一模型，通过第一模型，根据训练图像的N个网格的图像特征，输出N个网格中每个网格的光源概率、与N个网格相关的训练反射率。其中，光源概率为网格为光源网格的概率。Step 120: Input the training image into the first model, and output the light source probability of each grid in the N grids and the training reflectivity associated with the N grids according to the image features of the N grids in the training image through the first model. The light source probability is the probability that the grid is a light source grid.
在该步骤中，一方面，将训练图像的每个网格的光源预测值和与N个网格相关的真实反射率输入第一模型，用于作为训练的标准。In this step, on the one hand, the light source prediction value of each grid of the training image and the true reflectivity associated with the N grids are input into the first model to be used as training criteria.
在该步骤中，另一方面，将训练图像输入第一模型，通过第一模型提取训练图像的图像特征，再由第一模型根据训练图像的每个网格的图像特征，输出每个网格的光源概率和与N个网格相关的训练反射率。In this step, on the other hand, the training image is input into the first model, the image features of the training image are extracted by the first model, and then the first model outputs the light source probability of each grid and the training reflectivity related to N grids based on the image features of each grid in the training image.
其中，在拍摄场景中的物体是否为光源，以及物体的反射率，均取决于物体本身的性质，比如物体材质、表面粗糙程度，因此，在该步骤中，可以基于提取的图像特征，分析出每个网格所拍摄的是什么物体，然后根据物体的性质，输出每个网格的光源概率和训练反射率。Among them, whether the object in the shooting scene is a light source and the reflectivity of the object depend on the properties of the object itself, such as the material of the object and the roughness of the surface. Therefore, in this step, based on the extracted image features, it is possible to analyze what object is photographed in each grid, and then output the light source probability and training reflectivity of each grid according to the properties of the object.
例如，其中一个网格，经图像特征提取，识别到该网格中所拍摄的物体是黑板的一小部分，从而根据黑板的性质，输出黑板的光源概率和训练反射率。For example, in one of the grids, after image feature extraction, it is recognized that the object photographed in the grid is a small part of the blackboard, so according to the properties of the blackboard, the light source probability and training reflectivity of the blackboard are output.
步骤130：根据训练图像的N个网格中每个网格的光源预测值、与N个网格相关的真实反射率、N个网格中每个网格的光源概率和与N个网格相关的训练反射率，对第一模型进行训练，得到第二模型。Step 130: Train the first model according to the light source prediction value of each of the N grids in the training image, the true reflectivity associated with the N grids, the light source probability of each of the N grids and the training reflectivity associated with the N grids to obtain the second model.
在该步骤中，根据每个网格的光源预测值和每个网格的光源概率之间的差异，以及训练反射率和真实反射率之间的差异，对第一模型进行训练，得到第二模型，确保第二模型输出的光源概率与对应的光源预测值尽量接近，以及输出的训练反射率与对应的真实反射率尽量接近。In this step, the first model is trained according to the difference between the light source prediction value of each grid and the light source probability of each grid, and the difference between the training reflectivity and the true reflectivity, to obtain the second model, ensuring that the light source probability output by the second model is as close as possible to the corresponding light source prediction value, and the output training reflectivity is as close as possible to the corresponding true reflectivity.
在本申请的实施例中，首先准备训练数据，训练数据包括一张训练图像、训练图像中每个网格的光源预测值、与N个网格相关的真实反射率，其中，网格是光源网格时，光源预测值为1，网格是非光源网格时，光源预测值为0。然后将训练图像、与N个网格相关的真实反射率和每个网格的光源预测值输入待训练的第一模型，同时第一模型对训练图像进行图像特征提取，输出每个网格为光源网格的光源概率和与N个网格相关的训练反射率。最后，基于光源预测值、真实反射率、光源概率、训练反射率，对第一模型进行训练，得到第二模型，使得第二模型输出的光源概率尽可能地接近输入的光源预测值、输出的训练反射率尽可能地接近输入的真实反射率。可见，在本申请的实施例中，对模型进行训练后，模型可以准确地输出拍摄物体的反射率，而不是默认的灰卡的反射率，同时可以识别出存在光源的区域，使得拍摄过程中测量图像亮度的准确率提高，从而可以提高设置曝光参数的准确率，避免出现过曝光、欠曝光的现象，提升图像质量。In an embodiment of the present application, training data is first prepared, and the training data includes a training image, a light source prediction value of each grid in the training image, and a true reflectivity associated with N grids, wherein when a grid is a light source grid, the light source prediction value is 1, and when a grid is a non-light source grid, the light source prediction value is 0. Then the training image, the true reflectivity associated with the N grids, and the light source prediction value of each grid are input into the first model to be trained, and at the same time, the first model extracts image features from the training image, and outputs the light source probability that each grid is a light source grid and the training reflectivity associated with the N grids. Finally, based on the light source prediction value, the true reflectivity, the light source probability, and the training reflectivity, the first model is trained to obtain a second model, so that the light source probability output by the second model is as close as possible to the input light source prediction value, and the output training reflectivity is as close as possible to the input true reflectivity. It can be seen that in the embodiments of the present application, after the model is trained, the model can accurately output the reflectivity of the photographed object instead of the reflectivity of the default gray card, and can also identify areas where light sources exist, thereby improving the accuracy of measuring image brightness during the shooting process, thereby improving the accuracy of setting exposure parameters, avoiding overexposure and underexposure, and improving image quality.
在本申请另一个实施例的模型训练方法的流程中，与N个网格相关的真实反射率包括：与N个网格对应的N个真实反射率；在步骤110之前，该方法还包括：In the process of the model training method of another embodiment of the present application, the real reflectivity associated with N grids includes: N real reflectivities corresponding to the N grids; before step 110, the method further includes:
步骤A1：获取一组测试图像中测试区域的第一亮度和第二亮度。其中，第一亮度是测试区域放置灰卡时的亮度，第二亮度是测试区域未放置灰卡时的亮度。Step A1: Obtain a first brightness and a second brightness of a test area in a set of test images, wherein the first brightness is the brightness when a gray card is placed in the test area, and the second brightness is the brightness when no gray card is placed in the test area.
步骤A2：根据第一亮度、第二亮度和灰卡的反射率，计算训练图像中测试区域的真实反射率。其中，训练图像为测试区域未放置灰卡的测试图像。Step A2: Calculate the true reflectivity of the test area in the training image according to the first brightness, the second brightness and the reflectivity of the gray card, wherein the training image is a test image without a gray card placed in the test area.
可选地，在相机专业模式下，使用自动曝光功能拍摄同一场景，同一场景拍摄两次，一次拍摄时，将灰卡放置在所拍摄的物体表面，在另一次拍摄时，将灰卡取走。其中，放置灰卡的区域为测试区域，测试区域不可以是光源区域，且测试区域可以代表当前场景的平均照度。基于此，对一个场景拍摄时，得到一组测试图像，包括在测试区域放置灰卡的一张测试图像和在测试区域不放置会卡的一张测试图像，从而分别获取两张测试图像中测试区域的第一亮度和第二亮度。Optionally, in the professional mode of the camera, use the automatic exposure function to shoot the same scene twice. During one shooting, place a gray card on the surface of the object being shot, and during the other shooting, take the gray card away. The area where the gray card is placed is the test area. The test area cannot be a light source area, and the test area can represent the average illumination of the current scene. Based on this, when shooting a scene, a set of test images is obtained, including a test image in which a gray card is placed in the test area and a test image in which a gray card is not placed in the test area, thereby obtaining the first brightness and the second brightness of the test area in the two test images respectively.
其中，灰卡或者测试区域占测试图像的占比不可过小，以至少占比1/16以上为优。The proportion of the gray card or test area in the test image should not be too small, and it is preferred that the proportion is at least 1/16.
可选地，对于每张测试图像，分别保存DNG格式和JPG格式。Optionally, for each test image, save it in DNG format and JPG format respectively.
进一步地，参考公式(1)：Further, referring to formula (1):
lux*r*K＝brightess(1)lux*r*K＝brightness(1)
其中，在公式(1)中，lux表示入射光照度，r表示物体反射率，K表示相机相关常数，brightness表示图像亮度。In formula (1), lux represents the incident light illumination, r represents the object reflectivity, K represents the camera related constant, and brightness represents the image brightness.
对于放置灰卡的测试图像，利用灰卡的反射率得到测试区域的反射率为18％，基于DNG格式的该测试图像，获取测试区域的图像亮度，从而利用公式(1)可以计算出入射光照度lux，即For the test image with the gray card placed, the reflectivity of the test area is 18% using the reflectivity of the gray card. Based on the test image in DNG format, the image brightness of the test area is obtained, and the incident light illuminance lux can be calculated using formula (1), that is,
例如，测试区域的图像亮度的值是36，For example, the image brightness value of the test area is 36.
对于未放置灰卡的测试图像，也就是后续的训练图像，基于DNG格式的该测试图像，获取测试区域的图像亮度，已知入射光照度，从而利用公式(1)可以计算出物体反射率r，即测试区域的反射率，也就是步骤A2中的测试区域的真实反射率，其中，For the test image without gray card, that is, the subsequent training image, the image brightness of the test area is obtained based on the test image in DNG format. The incident light illumination is known, so the object reflectivity r, that is, the reflectivity of the test area, can be calculated using formula (1), that is, the real reflectivity of the test area in step A2, where:
例如，测试区域的图像亮度的值是50，For example, the image brightness value of the test area is 50,
或者，拍摄者手持反射式测光表，测量拍摄物体的亮度(Brighttarget)；再将灰卡摆放在拍摄物体的表面，尽可能保证灰卡所受的光照和拍摄物体的光照一致，再手持反射式测光表，测量灰卡的亮度(Brightgray)。Alternatively, the photographer holds a reflective light meter to measure the brightness of the object (Bright target); then places a gray card on the surface of the object, ensuring as much as possible that the light received by the gray card is consistent with that of the object, and then holds the reflective light meter to measure the brightness of the gray card (Bright gray).
进一步地，参考公式(2)：Further, referring to formula (2):
其中，在公式(2)中，Bright_target表示物体的亮度，Bright_gray表示灰卡的亮度，18表示灰卡的反射率，r表示物体的反射率。In formula (2), Bright_target represents the brightness of the object, Bright_gray represents the brightness of the gray card, 18 represents the reflectivity of the gray card, and r represents the reflectivity of the object.
例如，物体的亮度的值是50，灰卡的亮度的值是36，计算得到物体的反射率为25％。For example, the brightness value of the object is 50, and the brightness value of the gray card is 36. The calculated reflectivity of the object is 25%.
可见，基于公式(2)，可以得到物体表面的反射率，也就是步骤A2中的测试区域的真实反射率。It can be seen that based on formula (2), the reflectivity of the object surface can be obtained, that is, the actual reflectivity of the test area in step A2.
其中，还需要使用电子设备对该拍摄物体进行拍摄，以保存拍摄图像的DNG格式和JPG格式，以用于后续的步骤中。It is also necessary to use an electronic device to photograph the object to save the photographed image in DNG format and JPG format for use in subsequent steps.
步骤A3：将训练图像划分为N个网格。其中，测试区域包括至少一个网格。Step A3: Divide the training image into N grids, wherein the test area includes at least one grid.
如图2所示，将训练图像201划分为16*16的大小，从而得到16*16个网格。As shown in FIG. 2 , the training image 201 is divided into 16*16 grids, thereby obtaining 16*16 grids.
步骤A4：在测试区域中，确定种子网格。其中，种子网格的真实反射率与测试区域的真实反射率相同。Step A4: In the test area, a seed grid is determined, wherein the true reflectivity of the seed grid is the same as the true reflectivity of the test area.
如图2所示，测试区域202对应为之前放置会卡的区域，测试区域202至少占比整个训练图像的1/16以上，因此在16*16个网格中，可以保证某一个网格的面积较大程度地被测试区域填充，如某一个网格的95％面积是位于测试区域内的，从而，在所有的网格中，找到被测试区域填充的面积最大的网格，作为种子网格(seedgrid)。对应地，将测试区域的真实反射率，作为该为种子网格的真实反射率。As shown in FIG2 , the test area 202 corresponds to the area where the card was previously placed. The test area 202 accounts for at least 1/16 of the entire training image. Therefore, in the 16*16 grids, it can be guaranteed that the area of a certain grid is filled by the test area to a large extent. For example, 95% of the area of a certain grid is located in the test area. Therefore, among all the grids, the grid with the largest area filled by the test area is found as the seed grid. Correspondingly, the true reflectivity of the test area is used as the true reflectivity of the seed grid.
步骤A5：根据种子网格的真实反射率、种子网格的亮度、除种子网格以外的每个网格的亮度、N个网格的直方图信息和N个网格的语义向量，计算除种子网格以外的每个网格的真实反射率。Step A5: Calculate the true reflectivity of each grid except the seed grid according to the true reflectivity of the seed grid, the brightness of the seed grid, the brightness of each grid except the seed grid, the histogram information of the N grids and the semantic vectors of the N grids.
首先，以种子网格的真实反射率为基础，计算除种子网格以外的每个网格的初始反射率。First, based on the true reflectivity of the seed grid, the initial reflectivity of each grid except the seed grid is calculated.
参考公式(3)：Refer to formula (3):
其中，在公式(3)中，Bright_grid表示除种子网格以外的网格的亮度，Bright_seed_grid表示种子网格的亮度，origin_albedo表示除种子网格以外的网格的初始反射率，seed_albedo表示种子网格的真实反射率。In formula (3), Bright_grid represents the brightness of the grids other than the seed grid, Bright_seed_grid represents the brightness of the seed grid, origin_albedo represents the initial reflectivity of the grids other than the seed grid, and seed_albedo represents the true reflectivity of the seed grid.
所有网格亮度可以基于DNG格式的训练图像获取。All grid brightness can be obtained based on the training images in DNG format.
例如，如下表1，表示种子网格的真实反射率已知，其周边的8个网格的初始反射率未知。其中，种子网格的真实反射率为18％。For example, as shown in Table 1 below, the actual reflectivity of the seed grid is known, while the initial reflectivities of the 8 surrounding grids are unknown. The actual reflectivity of the seed grid is 18%.
表1Table 1
如下表2，表示种子网格与其周边的8个网格的亮度。Table 2 below shows the brightness of the seed grid and the eight grids surrounding it.
表2Table 2
如下表3，表示种子网格的真实反射率和其周边的8个网格的初始反射率。Table 3 below shows the actual reflectivity of the seed grid and the initial reflectivity of the eight surrounding grids.
表3table 3
以表中的左上角的网格的初始反射率的计算过程为例，网格的亮度的值为30，种子网格的亮度的值为35，种子网格的真实反射率的值为18，因此，网格的初始反射率的值为：Taking the calculation process of the initial reflectivity of the grid in the upper left corner of the table as an example, the brightness value of the grid is 30, the brightness value of the seed grid is 35, and the actual reflectivity value of the seed grid is 18. Therefore, the initial reflectivity value of the grid is:
然后，对初始反射率(origin albedo)进行纠正，可以得到除种子网格以外的所有网格的真实反射率。Then, the origin albedo is corrected to obtain the true albedo of all grids except the seed grid.
需要说明的是，在理想状态下，如果可以拿一个超级巨大的灰卡覆盖全部拍摄场景，则初始反射率就是真实反射率。但由于资源受限，现实世界无法找到这么大的灰卡，并且拍摄场景会存在光照不均的问题，比如某个位置存在光源，或者有阴影遮挡，另外光照也会受到物体距离光源的距离、入射角度等问题导致光照不均，因此需要初始反射率进行纠正。It should be noted that, ideally, if a super-large gray card can cover the entire shooting scene, the initial reflectivity is the real reflectivity. However, due to limited resources, such a large gray card cannot be found in the real world, and the shooting scene may have uneven lighting problems, such as the presence of a light source in a certain position, or shadows. In addition, the lighting may also be affected by the distance between the object and the light source, the angle of incidence, and other issues, resulting in uneven lighting. Therefore, the initial reflectivity is needed to correct it.
在纠正过程中，从DNG格式的训练图像中获取N个网格的直方图信息，从JPG格式的训练图像中获取N个网格的语义向量。其中，可以使用cv函数库中的cv2.cvtColor(image_stats,cv2.COLOR_BGR2GRAY)函数将JPG格式图像转为raw图，再使用cv2.calcHist函数获取直方图信息。另外，使用经过预训练的开源图像语义向量模型，比如clip模型，将JPG格式的训练图像输入给该模型，得到语义向量表示。clip模型共包含两个模态，一个是文本模态的文本编码(text encoder)，一个是视觉模态的图像编码(image encoder)，这里的语义向量表示即为经过视觉模态的图像编码模块编码之后得到的嵌入(embedding)向量。During the correction process, the histogram information of N grids is obtained from the training image in DNG format, and the semantic vectors of N grids are obtained from the training image in JPG format. Among them, the cv2.cvtColor(image_stats,cv2.COLOR_BGR2GRAY) function in the cv function library can be used to convert the JPG format image into a raw image, and then the cv2.calcHist function is used to obtain the histogram information. In addition, a pre-trained open source image semantic vector model, such as the clip model, is used to input the training image in JPG format into the model to obtain a semantic vector representation. The clip model contains two modalities, one is the text encoder of the text modality, and the other is the image encoder of the visual modality. The semantic vector representation here is the embedding vector obtained after encoding by the image encoding module of the visual modality.
进一步地，以种子网格作为基准，使用cv2.compareHist函数，计算除种子网格以外的每个网格的直方图信息与种子网格的直方图信息之间的相似度，得到如下表4所示的相似度：Furthermore, taking the seed grid as a benchmark, the cv2.compareHist function is used to calculate the similarity between the histogram information of each grid except the seed grid and the histogram information of the seed grid, and the similarity is obtained as shown in Table 4 below:
表4Table 4
同理，以种子网格作为基准，使用余弦相似度计算除种子网格以外的每个网格的语义向量与种子网格的语义向量之间的相似度，得到如下表5所示的相似度。Similarly, taking the seed grid as the benchmark, the cosine similarity is used to calculate the similarity between the semantic vector of each grid except the seed grid and the semantic vector of the seed grid, and the similarity is obtained as shown in Table 5 below.
表5table 5
具体分为三种情况对初始反射率进行纠正。There are three specific cases to correct the initial reflectivity.
如下表6所示，第1-3列表示9个网格的初始反射率，第4-6列表示9个网格的直方图信息相似度，第7-9列表示9个网格的语义向量相似度。As shown in Table 6 below, columns 1-3 represent the initial reflectance of the 9 grids, columns 4-6 represent the histogram information similarity of the 9 grids, and columns 7-9 represent the semantic vector similarity of the 9 grids.
表6Table 6
如下表7所示，第1-3列表示9个网格的初始反射率，第4-6列表示9个网格的直方图信息相似度或高或低的情况，第7-9列表示9个网格的语义向量相似度或高或低的情况。其中，相似度大于某阈值，如0.5，则相似度高；反之，则相似度低。As shown in Table 7 below, columns 1-3 represent the initial reflectivity of the 9 grids, columns 4-6 represent the high or low similarity of the histogram information of the 9 grids, and columns 7-9 represent the high or low similarity of the semantic vectors of the 9 grids. If the similarity is greater than a certain threshold, such as 0.5, the similarity is high; otherwise, the similarity is low.
表7Table 7
第一种情况，对于除种子网格以外的其中一个网格，其两个相似度均高，将对该网格的初始反射率经纠正得到的真实反射率设置为种子网格的真实反射率相同。如下表8所示，均设置为18％。In the first case, for one of the grids other than the seed grid, both similarities are high, and the true reflectivity obtained by correcting the initial reflectivity of the grid is set to be the same as the true reflectivity of the seed grid. As shown in Table 8 below, both are set to 18%.
表8Table 8
第二种情况，对于除种子网格以外的其中一个网格，其一个相似度高，其另一个相似度低，如下表9所示，利用公式(4)，对该网格的初始反射率进行纠正，以得到真实反射率，如纠正后得到19.232％。In the second case, for one of the grids other than the seed grid, one similarity is high and the other similarity is low, as shown in Table 9 below. Using formula (4), the initial reflectivity of the grid is corrected to obtain the true reflectivity, such as 19.232% after correction.
表9Table 9
其中，参考公式(4)：Wherein, refer to formula (4):
other_grid_final_albedo＝other_grid_final_albedo＝
origin_albedo+(seed_albedo–origin_albedo)*similarity(4)origin_albedo+(seed_albedo–origin_albedo)*similarity(4)
在公式(4)中，other_grid_final_albedo表示除种子网格以外的网格的真实反射率，origin_albedo表示除种子网格以外的网格的初始反射率，seed_albedo表示种子网格的真实反射率，similarity表示除种子网格以外的网格的两个相似度中的较大者。In formula (4), other_grid_final_albedo represents the true reflectivity of the grid other than the seed grid, origin_albedo represents the initial reflectivity of the grid other than the seed grid, seed_albedo represents the true reflectivity of the seed grid, and similarity represents the larger of the two similarities of the grid other than the seed grid.
例如，表中的左下角的网格的初始反射率的值为19.54，种子网格的真实反射率的值为18，两个相似度中的较大者为0.8，则网格的真实反射率的值为：19.54+(18-19.54)*0.8＝18.303。For example, the initial reflectivity value of the grid in the lower left corner of the table is 19.54, the actual reflectivity value of the seed grid is 18, and the larger of the two similarities is 0.8, then the actual reflectivity value of the grid is: 19.54+(18-19.54)*0.8=18.303.
第三种情况，对于除种子网格以外的其中一个网格，其两个相似度均低，说明该网格和种子网格完全不同，如下表10所示，将该网格的初始反射率作为真实反射率，如纠正后还是131％、123％、65％。In the third case, for one of the grids other than the seed grid, both similarities are low, indicating that the grid is completely different from the seed grid, as shown in Table 10 below. The initial reflectivity of the grid is taken as the true reflectivity, which is still 131%, 123%, and 65% after correction.
表10Table 10
步骤A6：将真实反射率大于第一阈值的网格确定为光源网格，以及将真实反射率小于或者等于第一阈值的网格确定为非光源网格。Step A6: Determine a grid whose real reflectivity is greater than a first threshold as a light source grid, and determine a grid whose real reflectivity is less than or equal to the first threshold as a non-light source grid.
在该步骤中，将真实反射率大于100％的网格确定为光源网格。In this step, the meshes whose true reflectivity is greater than 100% are determined as light source meshes.
对应地，得到光源预测值的表格，如下表11所示。Correspondingly, a table of light source prediction values is obtained, as shown in Table 11 below.
表11Table 11
将训练图像、真实反射率和光源预测值输入第一模型。The training images, true reflectance, and illuminant predictions are input into the first model.
例如，真实反射率的集合＝[18，18，131，18，18，123，For example, the set of true reflectances = [18, 18, 131, 18, 18, 123,
18.303……]，光源预测值的集合＝[0，0，1，0，0，1，0，0，0……]。18.303……], the set of light source prediction values = [0, 0, 1, 0, 0, 1, 0, 0, 0……].
在本实施例中，提供了准备训练数据的方案。在已知灰卡的反射率的前提下，将灰卡放置在拍摄物体表面，以及在拍摄物体表面不放置灰卡，放置区域作为测试区域，从而计算出不放置灰卡的测试区域的真实反射率。进一步地，将不放置会卡时得到的图像作为训练图像，将训练图像划分为N个网格，找到被测试区域占据面积最大的网格作为种子网格，默认种子网格的真实反射率与计算的测试区域的真实反射率相同，再利用种子网格与除种子网格以外的网格之间的亮度关系，计算出除种子网格以外的网格的初始反射率；最后利用所有网格的直方图信息和语义向量，对初始反射率进行纠正，得到所有网格的真实反射率。其中，若真实反射率大于100％，则认为对应的网格是光源网格。可见，本实施例提供的训练数据的准备方案，可以确保用于模型训练的真实反射率和光源预测值是准确的，从而确保训练效果。In this embodiment, a scheme for preparing training data is provided. Under the premise of knowing the reflectivity of the gray card, the gray card is placed on the surface of the photographed object, and the gray card is not placed on the surface of the photographed object, and the placement area is used as the test area, so as to calculate the true reflectivity of the test area without the gray card. Further, the image obtained when the gray card is not placed is used as the training image, and the training image is divided into N grids, and the grid with the largest area occupied by the tested area is found as the seed grid. The true reflectivity of the seed grid is assumed to be the same as the calculated true reflectivity of the test area, and then the brightness relationship between the seed grid and the grids other than the seed grid is used to calculate the initial reflectivity of the grids other than the seed grid; finally, the histogram information and semantic vectors of all grids are used to correct the initial reflectivity to obtain the true reflectivity of all grids. Among them, if the true reflectivity is greater than 100%, it is considered that the corresponding grid is a light source grid. It can be seen that the training data preparation scheme provided in this embodiment can ensure that the true reflectivity and light source prediction value used for model training are accurate, thereby ensuring the training effect.
在本申请另一个实施例的模型训练方法的流程中，与N个网格相关的训练反射率包括：与N个网格对应的N个训练反射率；步骤120，包括：In the process of the model training method of another embodiment of the present application, the training reflectivity associated with N grids includes: N training reflectivity corresponding to the N grids; step 120 includes:
子步骤B1：通过第一模型的模型骨干的至少两层结构，逐层对训练图像提取图像特征。其中，图像特征包括图像深层特征和图像浅层特征。Sub-step B1: extracting image features from the training image layer by layer through at least two layers of the model backbone of the first model, wherein the image features include deep image features and shallow image features.
可选地，图像浅层特征包括图像的颜色、纹理、形状、边缘等；深层特征包括图像的视觉特征，即人所能够理解并看到的东西，比如人脸、花草、树木等。Optionally, shallow features of the image include color, texture, shape, edge, etc. of the image; deep features include visual features of the image, that is, things that people can understand and see, such as faces, flowers, trees, etc.
可选地，通过MobilenetV2作为模型骨干(backbone)提取图像特征。Optionally, MobilenetV2 is used as the model backbone to extract image features.
例如，将训练图像输入第一模型，首先输出为224*224*3的图像数据，其中224*244为图像的高和宽，3代表图像的红(R)绿(G)蓝(B)通道的像素信息，每个像素信息的大小为0-255的像素值；然后经过一层conv2d卷积网络，提取浅层图像特征，由3通道，扩展为16通道，卷积层的设置为：输入通道数(inp)＝3，输出通道数(oup)＝3，卷积核大小(kernel_size)＝3，步长(stride)＝1。进一步地，经过堆叠的bneck结构逐步获取深层图像特征，每个bneck都包含逐点卷积(pw)层、深度分离卷积(dw)层、线性(linear)层，对于每一个bneck，由多个最小倒残差结构(InvertedResidual)单元组成。pw层和dw层均涉及到操作：二维卷积(conv2d)、二维正则化(batchnorm2d)、relu6激活函数；linear层涉及到操作：conv2d和batchnorm2d。其中，对于这样的benck结构，一共堆叠6层，每一层的参数配置为：For example, the training image is input into the first model, and the output is 224*224*3 image data, where 224*244 is the height and width of the image, 3 represents the pixel information of the red (R), green (G), and blue (B) channels of the image, and the size of each pixel information is a pixel value of 0-255; then, after a layer of conv2d convolutional network, shallow image features are extracted, and the channels are expanded from 3 to 16 channels. The settings of the convolution layer are: the number of input channels (inp) = 3, the number of output channels (oup) = 3, the convolution kernel size (kernel_size) = 3, and the stride (stride) = 1. Further, deep image features are gradually obtained through the stacked bneck structure. Each bneck contains a point-by-point convolution (pw) layer, a depth separation convolution (dw) layer, and a linear (linear) layer. For each bneck, it is composed of multiple minimum inverted residual structures (InvertedResidual) units. The pw layer and the dw layer both involve the following operations: two-dimensional convolution (conv2d), two-dimensional regularization (batchnorm2d), and relu6 activation function; the linear layer involves the following operations: conv2d and batchnorm2d. For such a Benck structure, a total of 6 layers are stacked, and the parameter configuration of each layer is:
其中，每一行是一个列表，每个列表包含4个元素，第一个元素为InvertedResidual单元的数量，第二个元素为输出通道数，第三个元素为卷积核大小，第四个元素为扩展倍率，其中，隐藏层数量＝输入通道数*扩展倍率。Among them, each row is a list, each list contains 4 elements, the first element is the number of InvertedResidual units, the second element is the number of output channels, the third element is the convolution kernel size, and the fourth element is the expansion factor, where the number of hidden layers = the number of input channels * expansion factor.
子步骤B2：对模型骨干的最后一层结构提取的图像特征进行上采样操作。Sub-step B2: Up-sample the image features extracted from the last layer of the model backbone.
子步骤B3：对上采样后的图像特征和模型骨干的倒数第二层结构提取的图像特征依次进行拼接操作、融合操作、维度扩展操作以及全连接操作，并输出N个网格中每个网格的光源概率。Sub-step B3: Perform concatenation, fusion, dimension expansion, and full connection operations on the upsampled image features and the image features extracted from the penultimate layer structure of the model backbone, and output the light source probability of each grid in the N grids.
到模型骨干的最后一层时，语义特征已经被高度抽象，此时图像大小在7*7左右，相比于网格的数量，如16*16，比较小，因此可以融合模型骨干倒数第二层的特征信息进行综合处理，其中，模型骨干倒数第二层的图像大小在14*14左右，具体处理步骤包括：对最后一层的图像特征进行上采样(upsample)操作，得到14*14的大小，将倒数第二层的图像特征与经过上采样的图像特征进行拼接(cat)操作，以拼接到一起，将拼接后的图像特征经过conv2d_1x1的卷积网络进行特征融合，此时图像大小仍然为14*14，将特征融合后的网络经过conv2d进行维度扩展，由14*14扩展到16*16，最后接一个全连接层+sigmoid激活函数，大小为256，按照从左到右、从上到下的顺序来计算每个网格的光源概率，光源概率的取值范围在0-1之间。By the last layer of the model backbone, the semantic features have been highly abstracted. At this time, the image size is about 7*7, which is relatively small compared to the number of grids, such as 16*16. Therefore, the feature information of the penultimate layer of the model backbone can be fused for comprehensive processing. The image size of the penultimate layer of the model backbone is about 14*14. The specific processing steps include: upsampling the image features of the last layer to obtain a size of 14*14, concatenating the image features of the penultimate layer with the upsampled image features to splice them together, and fusing the spliced image features through a conv2d_1x1 convolutional network. At this time, the image size is still 14*14. The network after feature fusion is expanded from 14*14 to 16*16 through conv2d. Finally, a fully connected layer + sigmoid activation function is connected with a size of 256. The light source probability of each grid is calculated from left to right and from top to bottom. The value range of the light source probability is between 0 and 1.
子步骤B4：对模型骨干的最后一层结构提取的图像特征依次进行卷积操作、池化操作、平铺操作以及全连接操作，并输出与N个网格相关的训练反射率。Sub-step B4: Perform convolution operations, pooling operations, tiling operations, and full connection operations on the image features extracted from the last layer of the model backbone, and output the training reflectivities associated with N grids.
在该步骤中，在模型骨干的最后一层提取的图像特征的基础上，经过conv2d，将特征由7*7扩展到16*16，经过一层池化(pool)层，将不同通道的特征进行融合，此时特征大小仍然为16*16，经过一层平铺(flatten)层，将特征平铺，方便与最后一层全连接层做计算，最后加一个全连接层，大小为256，按照从左到右、从上到下的位置存放每个网格的训练反射率。In this step, based on the image features extracted by the last layer of the model backbone, conv2d is used to expand the features from 7*7 to 16*16. After a pooling layer, the features of different channels are fused. At this time, the feature size is still 16*16. After a flatten layer, the features are flattened to facilitate calculation with the last fully connected layer. Finally, a fully connected layer is added with a size of 256 to store the training reflectivity of each grid from left to right and from top to bottom.
在本实施例中，以mobilenetv2作为模型骨干提取图像特征，其中分出两个分支：一个分支将最后一层提取的图像特征以及倒数第二层提取的图像特征进行融合、卷积、全连接层等操作，最后输出每个网格的光源概率；另一个分支将最后一层提取的图像特征通过卷积、池化、平铺、全连接等操作，最后输出每个网格的训练反射率。In this embodiment, mobilenetv2 is used as the model backbone to extract image features, and two branches are divided: one branch performs operations such as fusion, convolution, and full connection on the image features extracted from the last layer and the image features extracted from the penultimate layer, and finally outputs the light source probability of each grid; the other branch performs operations such as convolution, pooling, tiling, and full connection on the image features extracted from the last layer, and finally outputs the training reflectivity of each grid.
在本申请另一个实施例的模型训练方法的流程中，与N个网格相关的训练反射率包括：与N个网格对应的N个训练反射率；步骤130，包括：In the process of the model training method of another embodiment of the present application, the training reflectivity associated with N grids includes: N training reflectivity corresponding to the N grids; step 130 includes:
子步骤C1：根据训练图像的N个网格中每个网格的光源预测值和N个网格中每个网格的光源概率，计算第一损失信息。Sub-step C1: Calculate first loss information according to the light source prediction value of each grid in the N grids of the training image and the light source probability of each grid in the N grids.
可选地，通过交叉熵损失函数，计算一个网格的光源概率与该网格的光源预测值之间的损失信息，再计算所有网格的损失信息的平均值，得到第一损失信息，表示为grid_is_light_loss。Optionally, the loss information between the light source probability of a grid and the light source prediction value of the grid is calculated through the cross entropy loss function, and then the average value of the loss information of all grids is calculated to obtain the first loss information, which is expressed as grid_is_light_loss.
如下表12所示，第1-3列表示每个网格的光源预测值，第4-6列表示为每个网格的光源概率。As shown in Table 12 below, columns 1 to 3 represent the light source prediction value of each grid, and columns 4 to 6 represent the light source probability of each grid.
表12Table 12
如下表13所示，表示每个光源概率与对应的光源预测值之间的损失信息。As shown in Table 13 below, the loss information between each light source probability and the corresponding light source prediction value is represented.
表13Table 13
计算平均值后，grid_is_light_loss＝0.06。After calculating the average, grid_is_light_loss=0.06.
子步骤C2：根据与训练图像的N个网格对应的第一乘积和与训练图像的N个网格对应的第二乘积，计算第二损失信息。其中，对于每个网格，第一乘积为1减去光源概率的差值与训练反射率相乘得到的，第二乘积为1减去光源预测值的差值与真实反射率相乘得到的。Sub-step C2: Calculate the second loss information according to the first product corresponding to the N grids of the training image and the second product corresponding to the N grids of the training image, wherein for each grid, the first product is 1 minus the difference of the light source probability multiplied by the training reflectivity, and the second product is 1 minus the difference of the light source prediction value multiplied by the actual reflectivity.
在该步骤中，对于一个网格，将1-光源概率的差值与训练反射率相乘，可以计算出非光源网格的训练反射率；将1-光源预测值的差值与真实反射率相乘，可以计算出非光源网格的真实反射率；将非光源网格的训练反射率与非光源网格的真实反射率经过L2损失函数计算，得到非光源网格的训练反射率与真实反射率之间的损失信息，再计算所有损失信息的平均值后，得到第二损失信息，表示为grid_albedo_loss。In this step, for a grid, the training reflectivity of the non-light source grid can be calculated by multiplying the difference of 1-light source probability with the training reflectivity; the real reflectivity of the non-light source grid can be calculated by multiplying the difference of 1-light source prediction value with the real reflectivity; the training reflectivity of the non-light source grid and the real reflectivity of the non-light source grid are calculated by the L2 loss function to obtain the loss information between the training reflectivity and the real reflectivity of the non-light source grid, and then the average of all loss information is calculated to obtain the second loss information, which is expressed as grid_albedo_loss.
其中，因光源本身会发光，本身不具有反射率，因此，该步骤的目的在于，计算非光源网格的反射率损失。因光源网格的光源预测值为1，用1减去光源预测值后为0，再与真实反射率相乘，可以将真实反射率置为0，从而仅保留非光源网格的真实反射率，最终达到在非光源网格的真实反射率与训练反射率之间计算损失信息。Among them, since the light source itself emits light and has no reflectivity, the purpose of this step is to calculate the reflectivity loss of the non-light source mesh. Since the light source prediction value of the light source mesh is 1, 1 minus the light source prediction value is 0, and then multiplied by the true reflectivity, the true reflectivity can be set to 0, thereby retaining only the true reflectivity of the non-light source mesh, and finally calculating the loss information between the true reflectivity of the non-light source mesh and the training reflectivity.
具体地，如下表14所示，第1-3列表示每个网格的1减去光源概率的差值，第4-6列表示每个网格的训练反射率。Specifically, as shown in Table 14 below, columns 1-3 represent the difference of 1 minus the light source probability for each grid, and columns 4-6 represent the training reflectivity of each grid.
表14Table 14
如下表15所示，表示1减去光源概率的差值与训练反射率的第一乘积。As shown in Table 15 below, it represents the first product of the difference between 1 and the light source probability and the training reflectivity.
表15Table 15
如下表16所示，第1-3列表示每个网格的1减去光源预测值的差值，第4-6列表示每个网格的真实反射率。As shown in Table 16 below, columns 1-3 represent the difference between 1 and the light source prediction value for each grid, and columns 4-6 represent the actual reflectivity of each grid.
表16Table 16
如下表17所示，表示1减去光源预测值的差值与真实反射率的第二乘积。As shown in Table 17 below, it represents the second product of the difference between 1 and the light source prediction value and the actual reflectance.
表17Table 17
如下表18所示，表示通过均方误差损失函数得到的每个网格的第一乘积与第二乘积之间的损失信息。As shown in Table 18 below, the loss information between the first product and the second product of each grid obtained by the mean square error loss function is represented.
表18Table 18
计算平均值后，grid_albedo_loss＝194.11。After calculating the average, grid_albedo_loss=194.11.
子步骤C3：计算第一损失信息和第二损失信息的加权和值，得到第三损失信息。Sub-step C3: Calculate the weighted sum of the first loss information and the second loss information to obtain third loss information.
第三损失信息表示最终损失，计算方式参考公式(5)：The third loss information indicates the final loss, and the calculation method is based on formula (5):
loss 1＝ alpha_A * grid_is_light_loss + alpha_B * grid_albedo_loss(5)loss 1 = alpha_A * grid_is_light_loss + alpha_B * grid_albedo_loss(5)
其中，在公式(5)中，loss 1表示第三损失信息，alpha_A表示第一损失信息的权重，alpha_B表示第二损失信息的权重，grid_is_light_loss表示第一损失信息，grid_albedo_loss表示第二损失信息。In formula (5), loss 1 represents the third loss information, alpha_A represents the weight of the first loss information, alpha_B represents the weight of the second loss information, grid_is_light_loss represents the first loss information, and grid_albedo_loss represents the second loss information.
其中，为了避免第二损失信息表示的数值过大导致第一损失信息不起作用以及梯度爆炸的问题，对第二损失信息*0.01来加权，因此得到：loss1＝0.06*1+194.11*0.01＝2.0011。In order to avoid the problem that the value represented by the second loss information is too large, causing the first loss information to become ineffective and the gradient to explode, the second loss information is weighted by *0.01, so we get: loss1=0.06*1+194.11*0.01=2.0011.
子步骤C4：根据第三损失信息，对第一模型进行训练。Sub-step C4: training the first model according to the third loss information.
在本实施例中，一方面，计算基于光源概率和光源预测值得到的第一损失信息，另一方面，计算基于训练反射率和真实反射率得到的第二损失信息，从而加权求和两部分损失信息，得到第三损失信息，以对第一模型进行训练，以使得训练后得到的第二模型输出的各个网格的光源概率和每个网格的训练反射率是准确的。In this embodiment, on the one hand, the first loss information obtained based on the light source probability and the light source prediction value is calculated, and on the other hand, the second loss information obtained based on the training reflectivity and the actual reflectivity is calculated, so as to weightedly sum the two parts of the loss information to obtain the third loss information, so as to train the first model so that the light source probability of each grid and the training reflectivity of each grid output by the second model obtained after training are accurate.
在本申请另一个实施例的模型训练方法的流程中，与N个网格相关的真实反射率包括：计算所有非光源网格的真实反射率的均值得到的一个真实反射率；与N个网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的一个训练反射率。In the process of the model training method of another embodiment of the present application, the real reflectivity associated with N grids includes: a real reflectivity obtained by calculating the average value of the real reflectivity of all non-light source grids; the training reflectivity associated with N grids includes: a training reflectivity obtained by calculating the average value of the training reflectivity of all non-light source grids.
可选地，在准备训练数据的过程中，基于得到的每个网格的真实反射率和每个网格的光源预测值，计算其中的所有非光源网格的真实反射率的平均值，将该平均值输入第一模型，该平均值表示为nolight_average_albedo。Optionally, in the process of preparing training data, based on the obtained true reflectivity of each grid and the light source prediction value of each grid, the average value of the true reflectivity of all non-light source grids is calculated, and the average value is input into the first model. The average value is represented as nolight_average_albedo.
例如，nolight_average_albedo＝8％。For example, nolight_average_albedo=8%.
对应地，将训练图像输入第一模型后，将第一模型中的输出每个网格的光源概率的全连接层结果与输出每个网格的训练反射率的全连接层结果相乘，此时维度为16*16，然后再接一个全连接层，权重为256*1，得到最终的1*1的向量，该向量存放所有非光源网格的训练反射率的平均值。其中，第一模型默认为光源概率小于某阈值的网格为非光源网格，阈值如0.5。Correspondingly, after the training image is input into the first model, the fully connected layer result of the output light source probability of each grid in the first model is multiplied by the fully connected layer result of the output training reflectivity of each grid, and the dimension is 16*16 at this time, and then a fully connected layer is connected with a weight of 256*1 to obtain the final 1*1 vector, which stores the average value of the training reflectivity of all non-light source grids. Among them, the first model defaults to grids with light source probability less than a certain threshold as non-light source grids, and the threshold is 0.5.
将所有非光源网格的真实反射率的平均值输入第一模型中，以用于与第一模型输出的所有非光源网格的训练反射率的平均值之间计算损失信息，可见，第一模型不需要输出每个光源网格的训练反射率，只需要输出一个平均值，可以降低第一模型处理任务的难度。The average value of the true reflectivity of all non-light source meshes is input into the first model to calculate the loss information with the average value of the training reflectivity of all non-light source meshes output by the first model. It can be seen that the first model does not need to output the training reflectivity of each light source mesh, but only needs to output an average value, which can reduce the difficulty of the first model processing task.
步骤130，包括：Step 130 includes:
子步骤D1：根据训练图像的N个网格中每个网格的光源预测值和N个网格中每个网格的光源概率，计算第一损失信息。Sub-step D1: Calculate first loss information according to the light source prediction value of each grid in the N grids of the training image and the light source probability of each grid in the N grids.
第一损失信息的计算方式同上一实施例，在此不再赘述。The calculation method of the first loss information is the same as that of the previous embodiment and will not be repeated here.
子步骤D2：根据真实反射率和训练反射率，计算第四损失信息。Sub-step D2: Calculate the fourth loss information according to the real reflectivity and the training reflectivity.
真实反射率表示所有非光源网格的真实反射率的平均值，训练反射率表示所有非光源网格的训练反射率的平均值。The true reflectivity represents the average of the true reflectivity of all non-light meshes, and the trained reflectivity represents the average of the trained reflectivity of all non-light meshes.
将该步骤中的真实反射率和训练反射率经过L2损失计算得到第四损失信息，表示为albedo_loss。The actual reflectivity and the training reflectivity in this step are calculated through L2 loss to obtain the fourth loss information, which is expressed as albedo_loss.
子步骤D3：计算第一损失信息和第四损失信息的加权和值，得到第五损失信息。Sub-step D3: Calculate the weighted sum of the first loss information and the fourth loss information to obtain the fifth loss information.
可选地，第五损失信息的计算方式参考公式(6)：Optionally, the fifth loss information is calculated by referring to formula (6):
loss 2＝grid_is_light_loss+alpha_C*albedo_loss(6)loss 2 = grid_is_light_loss + alpha_C*albedo_loss (6)
其中，在公式(6)中，loss2表示第五损失信息，alpha_C表示第四损失信息的权重，grid_is_light_loss表示第一损失信息，albedo_loss表示第四损失信息。In formula (6), loss2 represents the fifth loss information, alpha_C represents the weight of the fourth loss information, grid_is_light_loss represents the first loss information, and albedo_loss represents the fourth loss information.
为了避免第四损失信息表示的数值过大导致第一损失信息不起作用以及梯度爆炸的问题，对第四损失信息*0.01来加权，例如：loss2＝0.06+194.11*0.01＝2.0011。In order to avoid the problem that the value represented by the fourth loss information is too large, causing the first loss information to become ineffective and the gradient to explode, the fourth loss information is weighted by * 0.01, for example: loss2 = 0.06 + 194.11 * 0.01 = 2.0011.
子步骤D4：根据第五损失信息，第一模型进行训练。Sub-step D4: According to the fifth loss information, the first model is trained.
在本实施例中，第一模型直接输出每个网格的光源概率和非光源网格的训练反射率的平均值，而不再需要单独输出每个网格的训练反射率。其中，在模型内部对光源概率和训练反射率进行融合，从而使得模型的处理任务难度降低，提高模型输出的准确率。In this embodiment, the first model directly outputs the light source probability of each grid and the average value of the training reflectivity of the non-light source grid, and no longer needs to output the training reflectivity of each grid separately. The light source probability and the training reflectivity are fused within the model, thereby reducing the difficulty of the model's processing tasks and improving the accuracy of the model output.
下面结合附图，通过具体的实施例及其应用场景对本申请实施例提供的图像曝光方法进行详细地说明。The image exposure method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
需要说明的是，本申请实施例提供的图像曝光方法，执行主体可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备等电子设备。本申请的一些实施例中以电子设备为执行主体执行图像曝光方法为例，说明本申请实施例提供的图像曝光方法。It should be noted that the image exposure method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the image exposure method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the image exposure method as an example.
本申请实施例提供的图像曝光方法可以应用于拍摄图像的场景，其中，一种具体的应用场景是，在拍摄图像时，第二模型输出拍摄场景中的物体反射率和光源区域，用于对测光得到的图像亮度进行调整，再利用调整后的图像亮度设置曝光参数。The image exposure method provided in the embodiment of the present application can be applied to the scene of shooting images. Among them, a specific application scenario is that when shooting an image, the second model outputs the reflectivity of the objects and the light source area in the shooting scene, which is used to adjust the image brightness obtained by metering, and then use the adjusted image brightness to set the exposure parameters.
如图3所示，示出了本申请一个实施例的图像曝光方法的流程图，以该方法应用于电子设备进行举例，该方法包括：As shown in FIG3 , a flow chart of an image exposure method according to an embodiment of the present application is shown. Taking the method applied to an electronic device as an example, the method includes:
步骤210：将一张预览图像输入基于上述任一实施例的模型训练方法得到的第二模型，通过第二模型输出：预览图像的N个网格中每个网格的光源概率和与N个网格相关的训练反射率。Step 210: Input a preview image into a second model obtained by the model training method based on any of the above embodiments, and output through the second model: the light source probability of each grid in the N grids of the preview image and the training reflectivity associated with the N grids.
例如，用户打开相机程序，屏幕显示拍摄预览界面，拍摄预览界面显示预览图像。For example, a user opens a camera program, and the screen displays a shooting preview interface that displays a preview image.
通过前述实施例的模型训练方法训练得到的第二模型，输出预览图像的N个网格的光源概率和与N个网格相关的训练反射率。The second model trained by the model training method of the aforementioned embodiment outputs the light source probabilities of N grids of the preview image and the training reflectivities associated with the N grids.
步骤220：根据N个网格中与非光源网格相关的训练反射率和灰卡的反射率，计算亮度比例。其中，将光源概率小于第二阈值的网格确定为非光源网格。Step 220: Calculate the brightness ratio according to the training reflectance related to the non-light source grid in the N grids and the reflectance of the gray card, wherein the grid whose light source probability is less than the second threshold is determined as the non-light source grid.
可选地，第二阈值可以是0.5。Optionally, the second threshold may be 0.5.
在该步骤中，亮度比例的计算公式参考公式(7)：In this step, the calculation formula of the brightness ratio refers to formula (7):
其中，在公式(7)中，P表示亮度比例，Q表示与非光源网格相关的训练反射率，18表示灰卡的反射率。Wherein, in formula (7), P represents the brightness ratio, Q represents the training reflectivity associated with the non-light source grid, and 18 represents the reflectivity of the gray card.
步骤230：根据参考图像亮度和亮度比例，设置曝光参数。Step 230: setting exposure parameters according to the reference image brightness and brightness ratio.
其中，参考图像亮度是对预览图像进行测光得到的。The reference image brightness is obtained by measuring the preview image.
在该步骤中，若亮度比例是正数，则基于测光得到的参考图像亮度，按照亮度比例进行提高；反之，若亮度比例是负数，则基于测光得到的参考图像亮度，按照亮度比例进行降低。In this step, if the brightness ratio is a positive number, the brightness of the reference image obtained based on the light metering is increased according to the brightness ratio; conversely, if the brightness ratio is a negative number, the brightness of the reference image obtained based on the light metering is reduced according to the brightness ratio.
例如，与非光源网格的训练反射率是30％，灰卡的反射率是18％，30％高于18％，则将测光得到的参考图像亮度提高：(30-18)/18＝12/18＝66％。进一步地，根据提高后的图像亮度设置曝光参数，以实现自动曝光控制。For example, the training reflectivity of the non-light source grid is 30%, the reflectivity of the gray card is 18%, and 30% is higher than 18%. Then the brightness of the reference image obtained by light measurement is increased by: (30-18)/18=12/18=66%. Further, the exposure parameters are set according to the increased image brightness to realize automatic exposure control.
其中，自动曝光控制是许多数码相机的默认设置，在这种曝光模式下相机将自动控制拍摄曝光，用户不需做任何事情，相机上的传感器可根据景物反射回来的光线强度自动设置光圈值和快门速度。曝光控制主要用来调节景物整体亮度，如果拍摄对象过于黯淡，可以调节曝光补偿标尺来增加亮度。Among them, automatic exposure control is the default setting of many digital cameras. In this exposure mode, the camera will automatically control the shooting exposure. The user does not need to do anything. The sensor on the camera can automatically set the aperture value and shutter speed according to the intensity of the light reflected back from the scene. Exposure control is mainly used to adjust the overall brightness of the scene. If the subject is too dim, the exposure compensation scale can be adjusted to increase the brightness.
在本申请的实施例中，基于模型训练方法训练好的第二模型，在拍摄过程中，将预览图像输入至第二模型中，通过第二模型输出预览图像的N个网格中每个网格的光源概率和与N个网格相关的训练反射率，再基于其中与非光源网格相关的训练反射率和灰卡的反射率计算亮度比例，亮度比例和参考图像亮度进行结合，得到的图像亮度和真实场景亮度更加接近，此时再设置曝光参数，可以确保曝光结果准确，最终使得拍摄的图像更接近真实场景，提高图像质量。In an embodiment of the present application, based on a second model trained by a model training method, during the shooting process, the preview image is input into the second model, and the light source probability of each grid in N grids of the preview image and the training reflectivity associated with the N grids are output by the second model. Then, the brightness ratio is calculated based on the training reflectivity associated with the non-light source grids and the reflectivity of the gray card. The brightness ratio is combined with the reference image brightness, and the obtained image brightness is closer to the real scene brightness. At this time, setting the exposure parameters can ensure that the exposure result is accurate, and ultimately the captured image is closer to the real scene, thereby improving the image quality.
在本申请另一个实施例的图像曝光方法的流程中，与N个网格相关的训练反射率包括：与N个网格对应的N个训练反射率；步骤220，包括：In the process of the image exposure method of another embodiment of the present application, the training reflectances associated with N grids include: N training reflectances corresponding to the N grids; step 220 includes:
子步骤E1：根据N个网格中每个非光源网格的训练反射率，计算所有非光源网格的训练反射率的平均值。Sub-step E1: Calculate the average of the training reflectances of all non-light source grids according to the training reflectance of each non-light source grid in the N grids.
在该步骤中，第二模型输出每个非光源网格的训练反射率，计算所有非光源网格的训练反射率的平均值。In this step, the second model outputs the training reflectivity of each non-light source mesh, and calculates the average of the training reflectivity of all non-light source meshes.
子步骤E2：根据平均值和灰卡的反射率，计算亮度比例。Sub-step E2: Calculate the brightness ratio based on the average value and the reflectivity of the gray card.
在该步骤中，将平均值作为与非光源网格相关的训练反射率代入公式(7)，计算亮度比例。In this step, the average value is substituted into formula (7) as the training reflectance associated with the non-light source mesh to calculate the brightness ratio.
在本实施例中，基于全局测光得到的一个图像亮度，结合计算所有非光源网格的训练反射率的平均值得到的一个亮度比例，可以对图像的整体亮度进行提高或者降低，以使得图像曝光效果得到改善。In this embodiment, based on an image brightness obtained by global photometry and a brightness ratio obtained by calculating the average value of the training reflectivity of all non-light source grids, the overall brightness of the image can be increased or decreased to improve the image exposure effect.
在本申请另一个实施例的图像曝光方法的流程中，与N个网格相关的训练反射率包括：与N个网格对应的N个训练反射率；步骤220，包括：In the process of the image exposure method of another embodiment of the present application, the training reflectances associated with N grids include: N training reflectances corresponding to the N grids; step 220 includes:
子步骤F1：根据N个网格中每个非光源网格的训练反射率和灰卡的反射率，计算与每个非光源网格对应的亮度比例。Sub-step F1: Calculate the brightness ratio corresponding to each non-light source grid in the N grids according to the training reflectivity of each non-light source grid and the reflectivity of the gray card.
可选地，本实施例适用于高动态范围(High Dynamic Range Imaging，HDR)图像的拍摄场景。Optionally, this embodiment is applicable to a shooting scene of a High Dynamic Range (HDR) image.
在该步骤中，第二模型输出每个非光源网格的训练反射率，每个训练反射率分别作与非光源网格相关的训练反射率代入公式(7)，计算出与每个非光源网格对应的亮度比例。In this step, the second model outputs the training reflectivity of each non-light source grid, and each training reflectivity is substituted into formula (7) as the training reflectivity related to the non-light source grid to calculate the brightness ratio corresponding to each non-light source grid.
步骤230，包括：Step 230 includes:
子步骤F2：根据与每个非光源网格对应的亮度比例和与每个非光源网格对应的参考图像亮度，对与非光源网格对应的不同图像区域分别设置曝光参数。Sub-step F2: setting exposure parameters for different image areas corresponding to the non-light source grids respectively according to the brightness ratio corresponding to each non-light source grid and the reference image brightness corresponding to each non-light source grid.
在测光过程中，针对每个非光源网格，得到一个参考图像亮度。During the light metering process, a reference image brightness is obtained for each non-light source grid.
在该步骤中，针对每个非光源网格的参考图像亮度和亮度比例，单独设置曝光参数。In this step, exposure parameters are set individually for the reference image brightness and brightness ratio of each non-light source grid.
在本实施例中，同一图像区域内，多个非光源网格的训练反射率相同或者接近，因此与多个非光源网格对应的亮度比例也相同或者接近，再结合这个图像区域内的多个非光源网格的参考图像亮度相同或者接近，可以到达按照不同的图像区域单独设置曝光参数的效果，提高拍摄效果。In this embodiment, within the same image area, the training reflectivities of multiple non-light source grids are the same or close, so the brightness ratios corresponding to the multiple non-light source grids are also the same or close. Combined with the fact that the reference image brightness of the multiple non-light source grids within this image area is the same or close, the exposure parameters can be set separately according to different image areas to improve the shooting effect.
在本申请另一个实施例的图像曝光方法的流程中，与N个网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的训练反射率；In the process of the image exposure method of another embodiment of the present application, the training reflectance associated with the N grids includes: calculating the training reflectance obtained by averaging the training reflectances of all non-light source grids;
N个网格中与非光源网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的训练反射率。The training reflectances associated with the non-light source grids in the N grids include: the training reflectance obtained by calculating the average of the training reflectances of all the non-light source grids.
在本实施例中，基于第二模型的结构，可以直接输出所有非光源网格的训练反射率的平均值，从而直接利用输出的平均值计算亮度比例，可以降低模型的处理任务难度。In this embodiment, based on the structure of the second model, the average value of the training reflectance of all non-light source grids can be directly output, so that the brightness ratio can be directly calculated using the output average value, which can reduce the difficulty of the model's processing tasks.
在本申请另一个实施例的图像曝光方法的流程中，步骤230，包括：In the process of the image exposure method of another embodiment of the present application, step 230 includes:
步骤H1：根据参考图像亮度和亮度比例，确定实际图像亮度。其中，实际图像亮度小于第三阈值。Step H1: Determine the actual image brightness according to the reference image brightness and the brightness ratio, wherein the actual image brightness is less than a third threshold.
可选地，第三阈值为255。Optionally, the third threshold is 255.
可参考地，根据参考图像亮度和亮度比例，确定的图像亮度高于255，则自动设置为255，从而最终确定的实际图像亮度即255。For reference, according to the reference image brightness and the brightness ratio, if the determined image brightness is higher than 255, it is automatically set to 255, so that the actual image brightness finally determined is 255.
步骤H2：根据实际图像亮度，设置曝光参数。Step H2: Set exposure parameters according to actual image brightness.
在本实施例中，在提高图像亮度时，需要对与光源网格对应的图像区域做亮度保护，避免过曝光或者欠曝光现象发生。In this embodiment, when the image brightness is increased, it is necessary to perform brightness protection on the image area corresponding to the light source grid to avoid overexposure or underexposure.
在本申请另一个实施例的图像曝光方法的流程中，在步骤210之后，该方法还包括：In the process of the image exposure method of another embodiment of the present application, after step 210, the method further includes:
步骤I1：分别设置预览图像中光源网格的测光权重和非光源网格的测光权重。其中，光源网格的测光权重小于非光源网格的测光权重。Step I1: respectively set the photometric weight of the light source grid and the photometric weight of the non-light source grid in the preview image, wherein the photometric weight of the light source grid is smaller than the photometric weight of the non-light source grid.
步骤I2：根据光源网格的测光权重和非光源网格的测光权重，对预览图像进行测光。Step I2: Measure the preview image according to the metering weight of the light source grid and the metering weight of the non-light source grid.
对预览图像测光后，得到参考图像亮度。可以是针对不同的图像区域分别得到不同的参考图像亮度，也可以是针对整个图像得到一个平均后的参考图像亮度。After measuring the preview image, a reference image brightness is obtained, which may be different reference image brightnesses for different image regions, or an average reference image brightness for the entire image.
在本实施例中，对于预览图像中存在光源的图像区域，选择测光时忽略该部分，或者降低其测光权重，这样可以使测光更准确，尤其是对于存在大面积灯箱、光源等逆光场景，测光更准。In this embodiment, for the image area where there is a light source in the preview image, this part is ignored during light metering, or its light metering weight is reduced. This can make the light metering more accurate, especially for backlit scenes with large-area light boxes, light sources, etc.
综上，本申请通过深度学习技术，实现分网格的光源检测和反射率识别，将两个任务进行联合学习，模型能够同时识别各个网格的反射率以及是否存在光源，不需要借助额外的语义分割模型或者显著性检测模型，任务设计和模型结构都比较简单，创新性地改善了图像反射率标注的过程，降低任务难度，且提升预测准确率，相机可以根据得到的每个网格的反射率以及各个网格是否存在光源，来进行曝光控制和测光控制。本申请不局限于人工智能(Artificial Intelligence，AI)控制自动曝光(Auto Exposure，AE)的应用场景，也可以用于AI控制自动白平衡(Auto White Balance，AWB)的应用场景，更多地，还可以利用本申请中的模型，来识别拍摄场景的最优相机参数信息，比如色调，景深等。In summary, this application uses deep learning technology to achieve grid-based light source detection and reflectivity recognition, and jointly learns the two tasks. The model can simultaneously identify the reflectivity of each grid and whether there is a light source, without the need for additional semantic segmentation models or saliency detection models. The task design and model structure are relatively simple, and the image reflectivity annotation process is innovatively improved, the task difficulty is reduced, and the prediction accuracy is improved. The camera can perform exposure control and light metering control based on the reflectivity of each grid and whether there is a light source in each grid. This application is not limited to the application scenario of artificial intelligence (AI) controlling automatic exposure (AE), but can also be used in the application scenario of AI controlling automatic white balance (AWB). Moreover, the model in this application can also be used to identify the optimal camera parameter information of the shooting scene, such as hue, depth of field, etc.
本申请实施例提供的模型训练方法，执行主体可以为模型训练装置。本申请实施例中以模型训练装置执行模型训练方法为例，说明本申请实施例提供的模型训练装置。The model training method provided in the embodiment of the present application can be executed by a model training device. In the embodiment of the present application, the model training device executing the model training method is taken as an example to illustrate the model training device provided in the embodiment of the present application.
图4示出了本申请一个实施例的模型训练装置400的框图，该装置包括：FIG4 shows a block diagram of a model training device 400 according to an embodiment of the present application, the device comprising:
输入模块401，用于将一张训练图像的N个网格中每个网格的光源预测值、与N个网格相关的真实反射率输入第一模型；其中，N个网格中光源网格的光源预测值为1，N个网格中非光源网格的光源预测值为0，N为正整数；An input module 401 is used to input the light source prediction value of each grid in N grids of a training image and the real reflectivity related to the N grids into a first model; wherein the light source prediction value of the light source grid in the N grids is 1, the light source prediction value of the non-light source grid in the N grids is 0, and N is a positive integer;
输出模块402，用于将训练图像输入第一模型，通过第一模型，根据训练图像的N个网格的图像特征，输出N个网格中每个网格的光源概率、与N个网格相关的训练反射率；其中，光源概率为网格为光源网格的概率；The output module 402 is used to input the training image into the first model, and output the light source probability of each grid in the N grids and the training reflectivity related to the N grids according to the image features of the N grids in the training image through the first model; wherein the light source probability is the probability that the grid is a light source grid;
训练模块403，用于根据训练图像的N个网格中每个网格的光源预测值、与N个网格相关的真实反射率、N个网格中每个网格的光源概率和与N个网格相关的训练反射率，对第一模型进行训练，得到第二模型。The training module 403 is used to train the first model according to the light source prediction value of each grid in the N grids of the training image, the true reflectivity associated with the N grids, the light source probability of each grid in the N grids and the training reflectivity associated with the N grids to obtain the second model.
在本申请的实施例中，首先准备训练数据，训练数据包括一张训练图像、训练图像中每个网格的光源预测值、与N个网格相关的真实反射率，其中，网格是光源网格时，光源预测值为1，网格是非光源网格时，光源预测值为0。然后将训练图像、与N个网格相关的真实反射率和每个网格的光源预测值输入待训练的第一模型，同时第一模型对训练图像进行图像特征提取，输出每个网格为光源网格的光源概率和与N个网格相关的训练反射率。最后，基于光源预测值、真实反射率、光源概率、训练反射率，对第一模型进行训练，得到第二模型，使得第二模型输出的光源概率尽可能地接近输入的光源预测值、输出的训练反射率尽可能地接近输入的真实反射率。可见，在本申请的实施例中，对模型进行训练后，模型可以准确地输出拍摄物体的反射率，而不是默认的灰卡的反射率，同时可以识别出存在光源的区域，使得拍摄过程中测量图像亮度的准确率提高，从而可以提高设置曝光参数的准确率，避免出现过曝光、欠曝光的现象，提升图像质量。In an embodiment of the present application, training data is first prepared, and the training data includes a training image, a light source prediction value of each grid in the training image, and a true reflectivity associated with N grids, wherein when a grid is a light source grid, the light source prediction value is 1, and when a grid is a non-light source grid, the light source prediction value is 0. Then the training image, the true reflectivity associated with the N grids, and the light source prediction value of each grid are input into the first model to be trained, and at the same time, the first model extracts image features from the training image, and outputs the light source probability that each grid is a light source grid and the training reflectivity associated with the N grids. Finally, based on the light source prediction value, the true reflectivity, the light source probability, and the training reflectivity, the first model is trained to obtain a second model, so that the light source probability output by the second model is as close as possible to the input light source prediction value, and the output training reflectivity is as close as possible to the input true reflectivity. It can be seen that in the embodiments of the present application, after the model is trained, the model can accurately output the reflectivity of the photographed object instead of the reflectivity of the default gray card, and can also identify areas where light sources exist, thereby improving the accuracy of measuring image brightness during the shooting process, thereby improving the accuracy of setting exposure parameters, avoiding overexposure and underexposure, and improving image quality.
可选地，与N个网格相关的真实反射率包括：与N个网格对应的N个真实反射率；Optionally, the real reflectivities associated with the N grids include: N real reflectivities corresponding to the N grids;
该装置还包括：The device also includes:
获取模块，用于获取一组测试图像中测试区域的第一亮度和第二亮度；其中，第一亮度是测试区域放置灰卡时的亮度，第二亮度是测试区域未放置灰卡时的亮度；An acquisition module, used to acquire a first brightness and a second brightness of a test area in a set of test images; wherein the first brightness is the brightness when a gray card is placed in the test area, and the second brightness is the brightness when no gray card is placed in the test area;
计算模块，用于根据第一亮度、第二亮度和灰卡的反射率，计算训练图像中测试区域的真实反射率；其中，训练图像为测试区域未放置灰卡的测试图像；A calculation module, used to calculate the real reflectivity of the test area in the training image according to the first brightness, the second brightness and the reflectivity of the gray card; wherein the training image is a test image in which no gray card is placed in the test area;
划分模块，用于将训练图像划分为N个网格；其中，测试区域包括至少一个网格；A division module, used to divide the training image into N grids; wherein the test area includes at least one grid;
确定模块，用于在测试区域中，确定种子网格；其中，种子网格的真实反射率与测试区域的真实反射率相同；A determination module is used to determine a seed grid in a test area, wherein the true reflectivity of the seed grid is the same as the true reflectivity of the test area;
计算模块，还用于根据种子网格的真实反射率、种子网格的亮度、除种子网格以外的每个网格的亮度、N个网格的直方图信息和N个网格的语义向量，计算除种子网格以外的每个网格的真实反射率；The calculation module is further used to calculate the true reflectivity of each grid except the seed grid according to the true reflectivity of the seed grid, the brightness of the seed grid, the brightness of each grid except the seed grid, the histogram information of the N grids and the semantic vectors of the N grids;
确定模块，还用于将真实反射率大于第一阈值的网格确定为光源网格，以及将真实反射率小于或者等于第一阈值的网格确定为非光源网格。The determination module is further configured to determine a grid whose real reflectivity is greater than a first threshold as a light source grid, and to determine a grid whose real reflectivity is less than or equal to the first threshold as a non-light source grid.
可选地，与N个网格相关的训练反射率包括：与N个网格对应的N个训练反射率；Optionally, the training reflectivities associated with the N grids include: N training reflectivities corresponding to the N grids;
该装置还包括：The device also includes:
提取模块，用于通过第一模型的模型骨干的至少两层结构，逐层对训练图像提取图像特征；其中，图像特征包括图像深层特征和图像浅层特征；An extraction module, used to extract image features from the training image layer by layer through at least two layers of the model backbone of the first model; wherein the image features include deep image features and shallow image features;
操作模块，用于对模型骨干的最后一层结构提取的图像特征进行上采样操作；An operation module, used to perform upsampling operations on the image features extracted from the last layer structure of the model backbone;
输出模块402，还用于：The output module 402 is further used for:
对上采样后的图像特征和模型骨干的倒数第二层结构提取的图像特征依次进行拼接操作、融合操作、维度扩展操作以及全连接操作，并输出N个网格中每个网格的光源概率；The upsampled image features and the image features extracted by the penultimate layer structure of the model backbone are sequentially concatenated, fused, dimensionally expanded, and fully connected, and the light source probability of each grid in N grids is output;
对模型骨干的最后一层结构提取的图像特征依次进行卷积操作、池化操作、平铺操作以及全连接操作，并输出与N个网格相关的训练反射率。The image features extracted by the last layer of the model backbone are sequentially subjected to convolution, pooling, tiling, and full connection operations, and the training reflectivity associated with N grids is output.
可选地，与N个网格相关的训练反射率包括：与N个网格对应的N个训练反射率；Optionally, the training reflectivities associated with the N grids include: N training reflectivities corresponding to the N grids;
该装置还包括：The device also includes:
计算模块，用于：Compute module for:
根据训练图像的N个网格中每个网格的光源预测值和N个网格中每个网格的光源概率，计算第一损失信息；Calculate first loss information according to the light source prediction value of each grid in the N grids of the training image and the light source probability of each grid in the N grids;
根据与训练图像的N个网格对应的第一乘积和与训练图像的N个网格对应的第二乘积，计算第二损失信息；其中，对于每个网格，第一乘积为1减去光源概率的差值与训练反射率相乘得到的，第二乘积为1减去光源预测值的差值与真实反射率相乘得到的；Calculating second loss information according to first products corresponding to N grids of the training image and second products corresponding to N grids of the training image; wherein, for each grid, the first product is 1 minus the difference between the light source probability and the training reflectivity, and the second product is 1 minus the difference between the light source prediction value and the actual reflectivity;
计算第一损失信息和第二损失信息的加权和值，得到第三损失信息；Calculate a weighted sum of the first loss information and the second loss information to obtain third loss information;
训练模块403，还用于根据第三损失信息，对第一模型进行训练。The training module 403 is further used to train the first model according to the third loss information.
可选地，与N个网格相关的真实反射率包括：计算所有非光源网格的真实反射率的均值得到的一个真实反射率；与N个网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的一个训练反射率；Optionally, the real reflectivity associated with the N grids includes: a real reflectivity obtained by calculating the average of the real reflectivity of all non-light source grids; the training reflectivity associated with the N grids includes: a training reflectivity obtained by calculating the average of the training reflectivity of all non-light source grids;
该装置还包括：The device also includes:
计算模块，用于：Compute module for:
根据训练图像的N个网格中每个网格的光源预测值和N个网格中每个网格的光源概率，计算第一损失信息；Calculate first loss information according to the light source prediction value of each grid in the N grids of the training image and the light source probability of each grid in the N grids;
根据真实反射率和训练反射率，计算第四损失信息；Calculate the fourth loss information according to the real reflectivity and the training reflectivity;
计算第一损失信息和第四损失信息的加权和值，得到第五损失信息；Calculating a weighted sum of the first loss information and the fourth loss information to obtain fifth loss information;
训练模块403，还用于根据第五损失信息，第一模型进行训练。The training module 403 is further used to train the first model according to the fifth loss information.
本申请实施例提供的图像曝光方法，执行主体可以为图像曝光装置。本申请实施例中以图像曝光装置执行图像曝光方法为例，说明本申请实施例提供的图像曝光装置。The image exposure method provided in the embodiment of the present application can be performed by an image exposure device. In the embodiment of the present application, the image exposure device performing the image exposure method is taken as an example to illustrate the image exposure device provided in the embodiment of the present application.
图5示出了本申请一个实施例的图像曝光装置500的框图，该装置包括：FIG5 shows a block diagram of an image exposure device 500 according to an embodiment of the present application, the device comprising:
输出模块501，用于将一张预览图像输入基于上述任一实施例的模型训练方法得到的第二模型，通过第二模型输出：预览图像的N个网格中每个网格的光源概率和与N个网格相关的训练反射率；An output module 501 is used to input a preview image into a second model obtained by the model training method based on any of the above embodiments, and output through the second model: the light source probability of each grid in N grids of the preview image and the training reflectivity associated with the N grids;
计算模块502，用于根据N个网格中与非光源网格相关的训练反射率和灰卡的反射率，计算亮度比例；其中，将光源概率小于第二阈值的网格确定为非光源网格；The calculation module 502 is used to calculate the brightness ratio according to the training reflectance related to the non-light source grid and the reflectance of the gray card in the N grids; wherein the grid whose light source probability is less than the second threshold is determined as the non-light source grid;
设置模块503，用于根据参考图像亮度和亮度比例，设置曝光参数；A setting module 503, used to set exposure parameters according to the brightness and brightness ratio of the reference image;
其中，参考图像亮度是对预览图像进行测光得到的。The reference image brightness is obtained by measuring the preview image.
在本申请的实施例中，基于模型训练方法训练好的第二模型，在拍摄过程中，将预览图像输入至第二模型中，通过第二模型输出预览图像的N个网格中每个网格的光源概率和与所述N个网格相关的训练反射率，再基于其中与非光源网格相关的训练反射率和灰卡的反射率计算亮度比例，亮度比例和参考图像亮度进行结合，得到的图像亮度和真实场景亮度更加接近，此时再设置曝光参数，可以确保曝光结果准确，最终使得拍摄的图像更接近真实场景，提高图像质量。In an embodiment of the present application, based on a second model trained by a model training method, during the shooting process, the preview image is input into the second model, and the light source probability of each grid in N grids of the preview image and the training reflectivity associated with the N grids are output by the second model. Then, the brightness ratio is calculated based on the training reflectivity associated with the non-light source grids and the reflectivity of the gray card. The brightness ratio is combined with the reference image brightness, and the obtained image brightness is closer to the real scene brightness. At this time, setting the exposure parameters can ensure that the exposure result is accurate, and ultimately the captured image is closer to the real scene, thereby improving the image quality.
可选地，与N个网格相关的训练反射率包括：与N个网格对应的N个训练反射率；Optionally, the training reflectivities associated with the N grids include: N training reflectivities corresponding to the N grids;
计算模块502，还用于：The calculation module 502 is further configured to:
根据N个网格中每个非光源网格的训练反射率，计算所有非光源网格的训练反射率的平均值；According to the training reflectivity of each non-light source grid in the N grids, the average of the training reflectivity of all non-light source grids is calculated;
根据平均值和灰卡的反射率，计算亮度比例。Calculate the brightness ratio based on the average value and the reflectance of the gray card.
设置模块503，还用于根据参考图像亮度和亮度比例的平均值，设置曝光参数。The setting module 503 is further used to set exposure parameters according to the average values of the brightness and brightness ratio of the reference image.
可选地，与N个网格相关的训练反射率包括：与N个网格对应的N个训练反射率；Optionally, the training reflectivities associated with the N grids include: N training reflectivities corresponding to the N grids;
计算模块502，还用于根据N个网格中每个非光源网格的训练反射率和灰卡的反射率，计算与每个非光源网格对应的亮度比例；The calculation module 502 is further used to calculate the brightness ratio corresponding to each non-light source grid according to the training reflectivity of each non-light source grid in the N grids and the reflectivity of the gray card;
设置模块503，还用于根据与每个非光源网格对应的亮度比例和与每个非光源网格对应的参考图像亮度，对与非光源网格对应的不同图像区域分别设置曝光参数。The setting module 503 is further used to set exposure parameters for different image areas corresponding to the non-light source grids respectively according to the brightness ratio corresponding to each non-light source grid and the reference image brightness corresponding to each non-light source grid.
可选地，与N个网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的训练反射率；Optionally, the training reflectances associated with the N grids include: a training reflectance obtained by calculating an average of the training reflectances of all non-light source grids;
N个网格中与非光源网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的训练反射率。The training reflectances associated with the non-light source grids in the N grids include: the training reflectance obtained by calculating the average of the training reflectances of all the non-light source grids.
可选地，该装置还包括：Optionally, the device further comprises:
确定模块，用于根据参考图像亮度和亮度比例，确定实际图像亮度；其中，实际图像亮度小于第三阈值；A determination module, used to determine the actual image brightness according to the reference image brightness and the brightness ratio; wherein the actual image brightness is less than a third threshold;
设置模块，还用于根据实际图像亮度，设置曝光参数。The setting module is also used to set exposure parameters according to the actual image brightness.
可选地，设置模块503，还用于分别设置预览图像中光源网格的测光权重和非光源网格的测光权重；其中，光源网格的测光权重小于非光源网格的测光权重；Optionally, the setting module 503 is further used to respectively set the photometric weight of the light source grid and the photometric weight of the non-light source grid in the preview image; wherein the photometric weight of the light source grid is smaller than the photometric weight of the non-light source grid;
该装置还包括：The device also includes:
测光模块，用于根据光源网格的测光权重和非光源网格的测光权重，对预览图像进行测光。The photometry module is used to measure the preview image according to the photometry weight of the light source grid and the photometry weight of the non-light source grid.
本申请实施例中的装置可以是电子设备，也可以是电子设备中的部件，例如集成电路或芯片。该电子设备可以是终端，也可以为除终端之外的其他设备。示例性的，电子设备可以为手机、平板电脑、笔记本电脑、掌上电脑、车载电子设备、移动上网装置(MobileInternet Device，MID)、增强现实(augmented reality，AR)/虚拟现实(virtual reality，VR)设备、机器人、可穿戴设备、超级移动个人计算机(ultra-mobile personal computer，UMPC)、上网本或者个人数字助理(personal digital assistant，PDA)等，还可以为服务器、网络附属存储器(Network Attached Storage，NAS)、个人计算机(personal computer，PC)、电视机(television，TV)、柜员机或者自助机等，本申请实施例不作具体限定。The device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR)/virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.
本申请实施例的装置可以为具有动作系统的装置。该动作系统可以为安卓(Android)动作系统，可以为ios动作系统，还可以为其他可能的动作系统，本申请实施例不作具体限定。The device of the embodiment of the present application may be a device having an action system. The action system may be an Android action system, an iOS action system, or other possible action systems, which are not specifically limited in the embodiment of the present application.
本申请实施例提供的装置能够实现上述方法实施例实现的各个过程，实现相同的技术效果，为避免重复，这里不再赘述。The device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and achieve the same technical effect. To avoid repetition, it will not be repeated here.
可选地，如图6所示，本申请实施例还提供一种电子设备100，包括处理器101，存储器102，存储在存储器102上并可在处理器101上运行的程序或指令，该程序或指令被处理器101执行时实现上述任一模型训练方法或者图像曝光方法实施例的各个步骤，且能达到相同的技术效果，为避免重复，这里不再赘述。Optionally, as shown in Figure 6, an embodiment of the present application also provides an electronic device 100, including a processor 101, a memory 102, and a program or instruction stored in the memory 102 and executable on the processor 101. When the program or instruction is executed by the processor 101, each step of any of the above-mentioned model training methods or image exposure method embodiments is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
需要说明的是，本申请实施例的电子设备包括上述的移动电子设备和非移动电子设备。It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.
图7为实现本申请实施例的一种电子设备的硬件结构示意图。FIG. 7 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.
该电子设备1000包括但不限于：射频单元1001、网络模块1002、音频输出单元1003、输入单元1004、传感器1005、显示单元1006、用户输入单元1007、接口单元1008、存储器1009、处理器1010、摄像头1011等部件。The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, a processor 1010, a camera 1011 and other components.
本领域技术人员可以理解，电子设备1000还可以包括给各个部件供电的电源(比如电池)，电源可以通过电源管理系统与处理器1010逻辑相连，从而通过电源管理系统实现管理充电、放电、以及功耗管理等功能。图7中示出的电子设备结构并不构成对电子设备的限定，电子设备可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置，在此不再赘述。Those skilled in the art can understand that the electronic device 1000 can also include a power source (such as a battery) for supplying power to each component, and the power source can be logically connected to the processor 1010 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The electronic device structure shown in FIG7 does not constitute a limitation on the electronic device, and the electronic device can include more or fewer components than shown in the figure, or combine certain components, or arrange components differently, which will not be described in detail here.
其中，一个电子设备1000用于执行模型训练方法，处理器1010，用于将一张训练图像的N个网格中每个网格的光源预测值、与所述N个网格相关的真实反射率输入第一模型；其中，所述N个网格中光源网格的光源预测值为1，所述N个网格中非光源网格的光源预测值为0，N为正整数；将所述训练图像输入所述第一模型，通过所述第一模型，根据所述训练图像的N个网格的图像特征，输出所述N个网格中每个网格的光源概率、与所述N个网格相关的训练反射率；其中，所述光源概率为网格为光源网格的概率；根据所述训练图像的N个网格中每个网格的光源预测值、与所述N个网格相关的真实反射率、所述N个网格中每个网格的光源概率和与所述N个网格相关的训练反射率，对所述第一模型进行训练，得到第二模型。Among them, an electronic device 1000 is used to execute the model training method, and the processor 1010 is used to input the light source prediction value of each grid in N grids of a training image and the real reflectivity related to the N grids into a first model; wherein the light source prediction value of the light source grid in the N grids is 1, and the light source prediction value of the non-light source grid in the N grids is 0, and N is a positive integer; the training image is input into the first model, and through the first model, according to the image features of the N grids of the training image, the light source probability of each grid in the N grids and the training reflectivity related to the N grids are output; wherein the light source probability is the probability that the grid is a light source grid; according to the light source prediction value of each grid in the N grids of the training image, the real reflectivity related to the N grids, the light source probability of each grid in the N grids and the training reflectivity related to the N grids, the first model is trained to obtain a second model.
在本申请的实施例中，首先准备训练数据，训练数据包括一张训练图像、训练图像中每个网格的光源预测值、与N个网格相关的真实反射率，其中，网格是光源网格时，光源预测值为1，网格是非光源网格时，光源预测值为0。然后将训练图像、与N个网格相关的真实反射率和每个网格的光源预测值输入待训练的第一模型，同时第一模型对训练图像进行图像特征提取，输出每个网格为光源网格的光源概率和与N个网格相关的训练反射率。最后，基于光源预测值、真实反射率、光源概率、训练反射率，对第一模型进行训练，得到第二模型，使得第二模型输出的光源概率尽可能地接近输入的光源预测值、输出的训练反射率尽可能地接近输入的真实反射率。可见，在本申请的实施例中，对模型进行训练后，模型可以准确地输出拍摄物体的反射率，而不是默认的灰卡的反射率，同时可以识别出存在光源的区域，使得拍摄过程中测量图像亮度的准确率提高，从而可以提高设置曝光参数的准确率，避免出现过曝光、欠曝光的现象，提升图像质量。In an embodiment of the present application, training data is first prepared, and the training data includes a training image, a light source prediction value of each grid in the training image, and a true reflectivity associated with N grids, wherein when a grid is a light source grid, the light source prediction value is 1, and when a grid is a non-light source grid, the light source prediction value is 0. Then the training image, the true reflectivity associated with the N grids, and the light source prediction value of each grid are input into the first model to be trained, and at the same time, the first model extracts image features from the training image, and outputs the light source probability that each grid is a light source grid and the training reflectivity associated with the N grids. Finally, based on the light source prediction value, the true reflectivity, the light source probability, and the training reflectivity, the first model is trained to obtain a second model, so that the light source probability output by the second model is as close as possible to the input light source prediction value, and the output training reflectivity is as close as possible to the input true reflectivity. It can be seen that in the embodiments of the present application, after the model is trained, the model can accurately output the reflectivity of the photographed object instead of the reflectivity of the default gray card, and can also identify areas where light sources exist, thereby improving the accuracy of measuring image brightness during the shooting process, thereby improving the accuracy of setting exposure parameters, avoiding overexposure and underexposure, and improving image quality.
可选地，与所述N个网格相关的真实反射率包括：与所述N个网格对应的N个真实反射率；处理器1010，还用于获取一组测试图像中测试区域的第一亮度和第二亮度；其中，所述第一亮度是所述测试区域放置灰卡时的亮度，所述第二亮度是所述测试区域未放置灰卡时的亮度；根据所述第一亮度、所述第二亮度和所述灰卡的反射率，计算训练图像中所述测试区域的真实反射率；其中，所述训练图像为所述测试区域未放置所述灰卡的测试图像；将所述训练图像划分为N个网格；其中，所述测试区域包括至少一个网格；在所述测试区域中，确定种子网格；其中，所述种子网格的真实反射率与所述测试区域的真实反射率相同；根据所述种子网格的真实反射率、所述种子网格的亮度、除所述种子网格以外的每个网格的亮度、所述N个网格的直方图信息和所述N个网格的语义向量，计算除所述种子网格以外的每个网格的真实反射率；将真实反射率大于第一阈值的网格确定为光源网格，以及将真实反射率小于或者等于所述第一阈值的网格确定为非光源网格。Optionally, the real reflectivity associated with the N grids includes: N real reflectivities corresponding to the N grids; the processor 1010 is further used to obtain a first brightness and a second brightness of a test area in a set of test images; wherein the first brightness is the brightness when a gray card is placed in the test area, and the second brightness is the brightness when no gray card is placed in the test area; according to the first brightness, the second brightness and the reflectivity of the gray card, the real reflectivity of the test area in the training image is calculated; wherein the training image is a test image in which the gray card is not placed in the test area; the training image is divided into N grids; wherein the test area includes at least one grid; in the test area, a seed grid is determined; wherein the real reflectivity of the seed grid is the same as the real reflectivity of the test area; according to the real reflectivity of the seed grid, the brightness of the seed grid, the brightness of each grid except the seed grid, the histogram information of the N grids and the semantic vectors of the N grids, the real reflectivity of each grid except the seed grid is calculated; a grid whose real reflectivity is greater than a first threshold is determined as a light source grid, and a grid whose real reflectivity is less than or equal to the first threshold is determined as a non-light source grid.
可选地，与所述N个网格相关的训练反射率包括：与所述N个网格对应的N个训练反射率；处理器1010，还用于通过所述第一模型的模型骨干的至少两层结构，逐层对所述训练图像提取图像特征；其中，所述图像特征包括图像深层特征和图像浅层特征；对所述模型骨干的最后一层结构提取的图像特征进行上采样操作；对上采样后的图像特征和所述模型骨干的倒数第二层结构提取的图像特征依次进行拼接操作、融合操作、维度扩展操作以及全连接操作，并输出所述N个网格中每个网格的光源概率；对所述模型骨干的最后一层结构提取的图像特征依次进行卷积操作、池化操作、平铺操作以及全连接操作，并输出与所述N个网格相关的训练反射率。Optionally, the training reflectivities associated with the N grids include: N training reflectivities corresponding to the N grids; the processor 1010 is also used to extract image features from the training image layer by layer through at least two layers of the model backbone of the first model; wherein the image features include deep image features and shallow image features; upsampling the image features extracted from the last layer of the model backbone; sequentially performing concatenation, fusion, dimensionality expansion and full connection operations on the upsampled image features and the image features extracted from the penultimate layer of the model backbone, and outputting the light source probability of each grid in the N grids; sequentially performing convolution, pooling, tiling and full connection operations on the image features extracted from the last layer of the model backbone, and outputting the training reflectivities associated with the N grids.
可选地，与所述N个网格相关的训练反射率包括：与所述N个网格对应的N个训练反射率；处理器1010，还用于根据所述训练图像的N个网格中每个网格的光源预测值和所述N个网格中每个网格的光源概率，计算第一损失信息；根据与所述训练图像的N个网格对应的第一乘积和与所述训练图像的N个网格对应的第二乘积，计算第二损失信息；其中，对于每个网格，所述第一乘积为1减去光源概率的差值与训练反射率相乘得到的，所述第二乘积为1减去光源预测值的差值与真实反射率相乘得到的；计算所述第一损失信息和所述第二损失信息的加权和值，得到第三损失信息；根据所述第三损失信息，对所述第一模型进行训练。Optionally, the training reflectivities associated with the N grids include: N training reflectivities corresponding to the N grids; the processor 1010 is further used to calculate first loss information based on the light source prediction value of each grid in the N grids of the training image and the light source probability of each grid in the N grids; calculate second loss information based on a first product corresponding to the N grids of the training image and a second product corresponding to the N grids of the training image; wherein, for each grid, the first product is 1 minus the difference between the light source probability and the training reflectivity, and the second product is 1 minus the difference between the light source prediction value and the true reflectivity; calculate the weighted sum of the first loss information and the second loss information to obtain third loss information; and train the first model based on the third loss information.
可选地，与所述N个网格相关的真实反射率包括：计算所有非光源网格的真实反射率的均值得到的一个真实反射率；与所述N个网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的一个训练反射率；处理器1010，还用于根据所述训练图像的N个网格中每个网格的光源预测值和所述N个网格中每个网格的光源概率，计算第一损失信息；根据真实反射率和训练反射率，计算第四损失信息；计算所述第一损失信息和所述第四损失信息的加权和值，得到第五损失信息；根据所述第五损失信息，所述第一模型进行训练。Optionally, the real reflectivity associated with the N grids includes: a real reflectivity obtained by calculating the average of the real reflectivity of all non-light source grids; the training reflectivity associated with the N grids includes: a training reflectivity obtained by calculating the average of the training reflectivity of all non-light source grids; the processor 1010 is also used to calculate the first loss information based on the light source prediction value of each grid in the N grids of the training image and the light source probability of each grid in the N grids; calculate the fourth loss information based on the real reflectivity and the training reflectivity; calculate the weighted sum of the first loss information and the fourth loss information to obtain the fifth loss information; and train the first model based on the fifth loss information.
其中，一个电子设备1000用于执行图像曝光方法，处理器1010，用于将一张预览图像输入基于上述模型训练方法训练得到的第二模型，通过所述第二模型输出：所述预览图像的N个网格中每个网格的光源概率和与所述N个网格相关的训练反射率；根据所述N个网格中与非光源网格相关的训练反射率和灰卡的反射率，计算亮度比例；其中，将光源概率小于第二阈值的网格确定为非光源网格；根据参考图像亮度和所述亮度比例，设置曝光参数；其中，所述参考图像亮度是对所述预览图像进行测光得到的。Among them, an electronic device 1000 is used to execute the image exposure method, and the processor 1010 is used to input a preview image into a second model trained based on the above-mentioned model training method, and output through the second model: the light source probability of each grid in N grids of the preview image and the training reflectivity related to the N grids; the brightness ratio is calculated according to the training reflectivity related to the non-light source grids in the N grids and the reflectivity of the gray card; wherein the grid whose light source probability is less than the second threshold is determined as the non-light source grid; the exposure parameters are set according to the reference image brightness and the brightness ratio; wherein the reference image brightness is obtained by measuring the preview image.
在本申请的实施例中，基于模型训练方法训练好的第二模型，在拍摄过程中，将预览图像输入至第二模型中，通过第二模型输出预览图像的N个网格中每个网格的光源概率和与所述N个网格相关的训练反射率，再基于其中与非光源网格相关的训练反射率和灰卡的反射率计算亮度比例，亮度比例和参考图像亮度进行结合，得到的图像亮度和真实场景亮度更加接近，此时再设置曝光参数，可以确保曝光结果准确，最终使得拍摄的图像更接近真实场景，提高图像质量。In an embodiment of the present application, based on a second model trained by a model training method, during the shooting process, the preview image is input into the second model, and the light source probability of each grid in N grids of the preview image and the training reflectivity associated with the N grids are output by the second model. Then, the brightness ratio is calculated based on the training reflectivity associated with the non-light source grids and the reflectivity of the gray card. The brightness ratio is combined with the reference image brightness, and the obtained image brightness is closer to the real scene brightness. At this time, setting the exposure parameters can ensure that the exposure result is accurate, and ultimately the captured image is closer to the real scene, thereby improving the image quality.
可选地，所述与所述N个网格相关的训练反射率包括：与所述N个网格对应的N个训练反射率；处理器1010，还用于根据所述N个网格中每个非光源网格的训练反射率，计算所有非光源网格的训练反射率的平均值；根据所述平均值和灰卡的反射率，计算亮度比例。Optionally, the training reflectivity associated with the N grids includes: N training reflectivity corresponding to the N grids; the processor 1010 is further used to calculate the average value of the training reflectivity of all non-light source grids based on the training reflectivity of each non-light source grid in the N grids; and calculate the brightness ratio based on the average value and the reflectivity of the gray card.
可选地，所述与所述N个网格相关的训练反射率包括：与所述N个网格对应的N个训练反射率；处理器1010，还用于根据所述N个网格中每个非光源网格的训练反射率和灰卡的反射率，计算与每个非光源网格对应的亮度比例；根据与每个非光源网格对应的亮度比例和与每个非光源网格对应的参考图像亮度，对与非光源网格对应的不同图像区域分别设置曝光参数。Optionally, the training reflectivities associated with the N grids include: N training reflectivities corresponding to the N grids; the processor 1010 is further used to calculate the brightness ratio corresponding to each non-light source grid in the N grids based on the training reflectivity of each non-light source grid and the reflectivity of the gray card; and to set exposure parameters for different image areas corresponding to the non-light source grids respectively according to the brightness ratio corresponding to each non-light source grid and the reference image brightness corresponding to each non-light source grid.
可选地，所述与所述N个网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的训练反射率；所述N个网格中与非光源网格相关的训练反射率包括：计算所有非光源网格的训练反射率的均值得到的训练反射率。Optionally, the training reflectivity associated with the N grids includes: the training reflectivity obtained by calculating the average of the training reflectivity of all non-light source grids; the training reflectivity associated with the non-light source grids in the N grids includes: the training reflectivity obtained by calculating the average of the training reflectivity of all non-light source grids.
可选地，处理器1010，还用于根据参考图像亮度和所述亮度比例，确定实际图像亮度；其中，所述实际图像亮度小于第三阈值；根据所述实际图像亮度，设置曝光参数。Optionally, the processor 1010 is further used to determine the actual image brightness based on the reference image brightness and the brightness ratio; wherein the actual image brightness is less than a third threshold; and set exposure parameters based on the actual image brightness.
可选地，处理器1010，还用于分别设置所述预览图像中光源网格的测光权重和非光源网格的测光权重；其中，光源网格的测光权重小于非光源网格的测光权重；根据光源网格的测光权重和非光源网格的测光权重，对所述预览图像进行测光。Optionally, the processor 1010 is further used to respectively set the photometry weight of the light source grid and the photometry weight of the non-light source grid in the preview image; wherein the photometry weight of the light source grid is less than the photometry weight of the non-light source grid; and the preview image is photometered according to the photometry weight of the light source grid and the photometry weight of the non-light source grid.
综上，本申请通过深度学习技术，实现分网格的光源检测和反射率识别，将两个任务进行联合学习，模型能够同时识别各个网格的反射率以及是否存在光源，不需要借助额外的语义分割模型或者显著性检测模型，任务设计和模型结构都比较简单，创新性地改善了图像反射率标注的过程，降低任务难度，且提升预测准确率，相机可以根据得到的每个网格的反射率以及各个网格是否存在光源，来进行曝光控制和测光控制。本申请不局限于人工智能(Artificial Intelligence，AI)控制自动曝光(Auto Exposure，AE)的应用场景，也可以用于AI控制自动白平衡(Auto White Balance，AWB)的应用场景，更多地，还可以利用本申请中的模型，来识别拍摄场景的最优相机参数信息，比如色调，景深等。In summary, this application uses deep learning technology to achieve grid-based light source detection and reflectivity recognition, and jointly learns the two tasks. The model can simultaneously identify the reflectivity of each grid and whether there is a light source, without the need for additional semantic segmentation models or saliency detection models. The task design and model structure are relatively simple, and the image reflectivity annotation process is innovatively improved, the task difficulty is reduced, and the prediction accuracy is improved. The camera can perform exposure control and light metering control based on the reflectivity of each grid and whether there is a light source in each grid. This application is not limited to application scenarios where artificial intelligence (AI) controls automatic exposure (AE), but can also be used in application scenarios where AI controls automatic white balance (AWB). Furthermore, the model in this application can also be used to identify the optimal camera parameter information for the shooting scene, such as hue, depth of field, etc.
应理解的是，本申请实施例中，输入单元1004可以包括图形处理器(GraphicsProcessing Unit，GPU)10041和麦克风10042，图形处理器10041对在视频图像捕获模式或图像捕获模式中由图像捕获装置(如摄像头)获得的静态图片或视频图像的图像数据进行处理。显示单元1006可包括显示面板10061，可以采用液晶显示器、有机发光二极管等形式来配置显示面板10061。用户输入单元1007包括触控面板10071以及其他输入设备10072中的至少一种。触控面板10071，也称为触摸屏。触控面板10071可包括触摸检测装置和触摸控制器两个部分。其他输入设备10072可以包括但不限于物理键盘、功能键(比如音量控制按键、开关按键等)、轨迹球、鼠标、动作杆，在此不再赘述。存储器1009可用于存储软件程序以及各种数据，包括但不限于应用程序和动作系统。处理器1010可集成应用处理器和调制解调处理器，其中，应用处理器主要处理动作系统、用户页面和应用程序等，调制解调处理器主要处理无线通信。可以理解的是，上述调制解调处理器也可以不集成到处理器1010中。It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video image obtained by the image capture device (such as a camera) in the video image capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and an action rod, which will not be repeated here. The memory 1009 can be used to store software programs and various data, including but not limited to applications and action systems. The processor 1010 may integrate an application processor and a modem processor, wherein the application processor mainly processes the action system, user pages and application programs, and the modem processor mainly processes wireless communications. It is understandable that the modem processor may not be integrated into the processor 1010.
存储器1009可用于存储软件程序以及各种数据。存储器1009可主要包括存储程序或指令的第一存储区和存储数据的第二存储区，其中，第一存储区可存储操作系统、至少一个功能所需的应用程序或指令(比如声音播放功能、图像播放功能等)等。此外，存储器1009可以包括易失性存储器或非易失性存储器，或者，存储器1009可以包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM，DDRSDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(Synch link DRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DRRAM)。本申请实施例中的存储器1009包括但不限于这些和任意其它适合类型的存储器。The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.
处理器1010可包括一个或多个处理单元；可选地，处理器1010集成应用处理器和调制解调处理器，其中，应用处理器主要处理涉及操作系统、用户界面和应用程序等的操作，调制解调处理器主要处理无线通信信号，如基带处理器。可以理解的是，上述调制解调处理器也可以不集成到处理器1010中。The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.
本申请实施例还提供一种可读存储介质，可读存储介质上存储有程序或指令，该程序或指令被处理器执行时实现上述模型训练方法或者图像曝光方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned model training method or image exposure method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
其中，处理器为上述实施例中的电子设备中的处理器。可读存储介质，包括计算机可读存储介质，如计算机只读存储器ROM、随机存取存储器RAM、磁碟或者光盘等。The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
本申请实施例另提供了一种芯片，芯片包括处理器和通信接口，通信接口和处理器耦合，处理器用于运行程序或指令，实现上述模型训练方法或者图像曝光方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned model training method or image exposure method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
应理解，本申请实施例提到的芯片还可以称为系统级芯片、系统芯片、芯片系统或片上系统芯片等。It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.
本申请实施例提供一种计算机程序产品，该程序产品被存储在存储介质中，该程序产品被至少一个处理器执行以实现如上述模型训练方法或者图像曝光方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned model training method or image exposure method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
需要说明的是，在本文中，术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。此外，需要指出的是，本申请实施方式中的方法和装置的范围不限按示出或讨论的顺序来执行功能，还可包括根据所涉及的功能按基本同时的方式或按相反的顺序来执行功能，例如，可以按不同于所描述的次序来执行所描述的方法，并且还可以添加、省去、或组合各种步骤。另外，参照某些示例所描述的特征可在其他示例中被组合。It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises one..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
通过以上的实施方式的描述，本领域的技术人员可以清楚地了解到上述实施例方法可借助软件加必需的通用硬件平台的方式来实现，当然也可以通过硬件，但很多情况下前者是更佳的实施方式。基于这样的理解，本申请的技术方案本质上或者说对现有技术做出贡献的部分可以以计算机软件产品的形式体现出来，该计算机软件产品存储在一个存储介质(如ROM/RAM、磁碟、光盘)中，包括若干指令用以使得一台终端(可以是手机，计算机，服务器，或者网络设备等)执行本申请各个实施例的方法。Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.
上面结合附图对本申请的实施例进行了描述，但是本申请并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本申请的启示下，在不脱离本申请宗旨和权利要求所保护的范围情况下，还可做出很多形式，均属于本申请的保护之内。The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410490069.0ACN118200735A (en) | 2024-04-23 | 2024-04-23 | Model training method, image exposure method, device and electronic equipment |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410490069.0ACN118200735A (en) | 2024-04-23 | 2024-04-23 | Model training method, image exposure method, device and electronic equipment |
| Publication Number | Publication Date |
|---|---|
| CN118200735Atrue CN118200735A (en) | 2024-06-14 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410490069.0APendingCN118200735A (en) | 2024-04-23 | 2024-04-23 | Model training method, image exposure method, device and electronic equipment |
| Country | Link |
|---|---|
| CN (1) | CN118200735A (en) |
| Publication | Publication Date | Title |
|---|---|---|
| US11882357B2 (en) | Image display method and device | |
| US11457138B2 (en) | Method and device for image processing, method for training object detection model | |
| CN113888437B (en) | Image processing method, device, electronic device and computer readable storage medium | |
| CN109493350B (en) | Portrait segmentation method and device | |
| CN110663045B (en) | Method, electronic system and medium for automatic exposure adjustment of digital images | |
| CN113518210B (en) | Method and device for automatic white balance of image | |
| US20240273909A1 (en) | Camera shooting parameter adjustment method and apparatus, and electronic device | |
| CN114372931A (en) | A target object blurring method, device, storage medium and electronic device | |
| CN113507570B (en) | Exposure compensation method and device and electronic equipment | |
| CN113824884B (en) | Shooting method and device, shooting equipment and computer readable storage medium | |
| WO2021128593A1 (en) | Facial image processing method, apparatus, and system | |
| US20230289930A1 (en) | Systems and Methods for Lightweight Machine Learning for Image Illumination Control | |
| JP2023078061A (en) | Imaging exposure control method and apparatus, device and storage medium | |
| US20250045887A1 (en) | Systems and methods to generate high dynamic range scenes | |
| CN117710273A (en) | Image enhancement model construction method, image enhancement method, device and medium | |
| CN118200735A (en) | Model training method, image exposure method, device and electronic equipment | |
| CN113344011B (en) | A color constancy method based on cascade fusion feature confidence weighting | |
| CN115580781A (en) | Exposure parameter adjustment method, device, electronic equipment and storage medium | |
| CN118450265B (en) | Image processing method and related equipment | |
| CN117014561B (en) | Information fusion method, training method of variable learning and electronic equipment | |
| CN118135073A (en) | A three-dimensional scene lighting rendering method, device, equipment and storage medium | |
| CN115883986A (en) | Image processing method and device | |
| JP2025530977A (en) | Relighting outdoor images using machine learning | |
| CN119068304A (en) | Training model method, image processing method, electronic device and storage medium | |
| CN116363017A (en) | Image processing method and device |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |