技术领域Technical Field
本发明涉及光电子技术领域，具体涉及一种图像传感器、图像传感器的控制方法、光电设备及介质。The present invention relates to the field of optoelectronic technology, and in particular to an image sensor, a control method for the image sensor, an optoelectronic device and a medium.
背景技术Background Art
随着事件型视觉传感器（Event-based Vision Sensor，简称EVS）的兴起，融合了有源积分型像素传感器（Active Pixel Sensor，简称APS）与事件型视觉传感器的复合型图像传感器随之应运而生。With the rise of event-based vision sensors (EVS), composite image sensors that combine active pixel sensors (APS) and event-based vision sensors have emerged.
在相关技术中，会交错设置固定的EVS像素和APS像素，从而实现EVS与APS的相互融合。但是这种方案导致采集的EVS图像和APS图像的分辨率不高，影响了APS图像和EVS图像的成像质量。In the related art, fixed EVS pixels and APS pixels are staggered to achieve mutual fusion of EVS and APS. However, this solution results in low resolution of the collected EVS images and APS images, affecting the imaging quality of the APS images and EVS images.
发明内容Summary of the invention
本发明实施例提供一种图像传感器、图像传感器的控制方法、光电设备及介质，旨在有效提升EVS模式和APS模式融合成像质量。The embodiments of the present invention provide an image sensor, a control method of the image sensor, an optoelectronic device, and a medium, aiming to effectively improve the quality of fused imaging in the EVS mode and the APS mode.
第一方面，本发明实施例提供一种图像传感器，所述图像传感器包括：In a first aspect, an embodiment of the present invention provides an image sensor, the image sensor comprising:
像素，配置有光电转换模块、第一传输支路以及第二传输支路；所述光电转换模块基于控制信号将接收到的入射光转换为光强积分信号或光强对数信号，通过所述第一传输支路输出所述光强积分信号，或通过所述第二传输支路输出所述光强对数信号；A pixel is configured with a photoelectric conversion module, a first transmission branch and a second transmission branch; the photoelectric conversion module converts the received incident light into a light intensity integral signal or a light intensity logarithmic signal based on a control signal, outputs the light intensity integral signal through the first transmission branch, or outputs the light intensity logarithmic signal through the second transmission branch;
处理模块，包括有源积分型像素读出单元和事件型像素读出单元，所述有源积分型像素读出单元接收所述第一传输支路输出的所述光强积分信号，处理得到有源图像数据，所述事件型像素读出单元接收所述第二传输支路输出的所述光强对数信号，处理得到事件图像数据。The processing module includes an active integration type pixel readout unit and an event type pixel readout unit. The active integration type pixel readout unit receives the light intensity integration signal output by the first transmission branch and processes it to obtain active image data. The event type pixel readout unit receives the light intensity logarithmic signal output by the second transmission branch and processes it to obtain event image data.
第二方面，本发明实施例提供一种图像传感器的控制方法，应用于如上所述的图像传感器，所述方法包括：In a second aspect, an embodiment of the present invention provides a method for controlling an image sensor, which is applied to the image sensor as described above, and the method includes:
获取图像传感器配置信息；Get image sensor configuration information;
根据所述图像传感器配置信息确定所述图像传感器中各个像素的目标传感模式；determining a target sensing mode for each pixel in the image sensor according to the image sensor configuration information;
若所述目标传感模式为有源积分型像素模式，控制所述光电转换模块连通所述第一传输支路，以使所述光电转换模块通过所述第一传输支路输出光强积分信号至所述有源积分型像素读出单元，由所述光强积分信号处理得到所述像素对应的有源图像数据；If the target sensing mode is an active integral pixel mode, controlling the photoelectric conversion module to be connected to the first transmission branch, so that the photoelectric conversion module outputs a light intensity integral signal to the active integral pixel readout unit through the first transmission branch, and the active image data corresponding to the pixel is obtained by processing the light intensity integral signal;
若所述目标传感模式为事件型像素模式，控制所述光电转换模块连通所述第二传输支路，以使所述光电转换模块通过所述第二传输支路输出光强对数信号至所述事件型像素读出单元，由所述光强对数信号处理得到所述像素对应的事件图像数据。If the target sensing mode is an event-type pixel mode, the photoelectric conversion module is controlled to connect to the second transmission branch so that the photoelectric conversion module outputs a light intensity logarithmic signal to the event-type pixel readout unit through the second transmission branch, and the event image data corresponding to the pixel is obtained by processing the light intensity logarithmic signal.
第三方面，本发明实施例还提供一种光电设备，包括图像传感器、处理器和存储器，所述存储器存储有计算机程序，当所述计算机程序被所述处理器执行时，使得所述处理器执行上述的图像传感器的控制方法的步骤。In a third aspect, an embodiment of the present invention further provides an optoelectronic device, comprising an image sensor, a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned image sensor control method.
第四方面，本发明实施例还提供一种计算机可读存储介质，所述计算机可读存储介质包括计算机程序，当所述计算机程序在电子设备上运行时，所述计算机程序用于使所述电子设备执行上所述图像传感器的控制方法的步骤。In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium includes a computer program. When the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of the control method of the image sensor.
本发明在像素中配置有光电转换模块、第一传输支路以及第二传输支路，通过光电转换模块可以基于控制信号将其接收到的入射光转换为光强积分信号或光强对数信号，并将光强积分信号通过第一传输支路输出给对应的有源积分型像素读出单元，或者将光强对数信号通过第二传输支路输出给对应的事件型像素读出单元，对应处理可以得到APS图像数据或者EVS图像数据，从而基于一个像素即可以生成EVS图像数据，又可以生成APS图像数据，实现单个像素兼容输出APS图像数据和EVS图像数据，通过这样的像素构成兼容APS模式和EVS模式的复合型图像传感器，使得兼容APS模式和EVS模式的复合型图像传感器生成的图像数据分辨率更高，有效地提高了复合型图像传感器的APS模式和EVS模式融合成像质量。The present invention configures a photoelectric conversion module, a first transmission branch and a second transmission branch in a pixel. The photoelectric conversion module can convert the incident light it receives into a light intensity integral signal or a light intensity logarithmic signal based on a control signal, and output the light intensity integral signal to the corresponding active integral pixel readout unit through the first transmission branch, or output the light intensity logarithmic signal to the corresponding event-type pixel readout unit through the second transmission branch. The corresponding processing can obtain APS image data or EVS image data, so that both EVS image data and APS image data can be generated based on one pixel, and a single pixel can be compatible with outputting APS image data and EVS image data. A composite image sensor compatible with the APS mode and the EVS mode is formed by such pixels, so that the image data generated by the composite image sensor compatible with the APS mode and the EVS mode has higher resolution, which effectively improves the APS mode and EVS mode fusion imaging quality of the composite image sensor.
附图说明BRIEF DESCRIPTION OF THE DRAWINGS
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
图1是本发明中图像传感器的结构简图；FIG1 is a schematic diagram of the structure of an image sensor in the present invention;
图2是本发明中图像传感器的结构示意图；FIG2 is a schematic diagram of the structure of an image sensor in the present invention;
图3是本发明实施例中像素的第一电路图；FIG3 is a first circuit diagram of a pixel in an embodiment of the present invention;
图4是本发明实施例中涉及的控制信号的时序图；FIG4 is a timing diagram of control signals involved in an embodiment of the present invention;
图5是本发明实施例中像素的第二电路图；FIG5 is a second circuit diagram of a pixel in an embodiment of the present invention;
图6是本发明一可选实施例中图像传感器的控制方法流程图；6 is a flow chart of a method for controlling an image sensor in an optional embodiment of the present invention;
图7是本发明实施例中涉及的第一传感阵列的结构示意图；FIG7 is a schematic diagram of the structure of a first sensor array involved in an embodiment of the present invention;
图8是本发明实施例中涉及的第一传感阵列在HVS图像模式下的结构示意图；FIG8 is a schematic structural diagram of a first sensor array in an HVS image mode involved in an embodiment of the present invention;
图9是本发明实施例中涉及的第二传感阵列的结构示意图；FIG9 is a schematic diagram of the structure of a second sensor array involved in an embodiment of the present invention;
图10是本发明实施例中涉及的第二传感阵列在HVS图像模式下的结构示意图；10 is a schematic diagram of the structure of the second sensor array in the HVS image mode involved in an embodiment of the present invention;
图11是本发明实施例中涉及的第三传感阵列的结构示意图；FIG11 is a schematic diagram of the structure of a third sensor array involved in an embodiment of the present invention;
图12是本发明实施例中涉及的第三传感阵列在HVS图像模式下的结构示意图；12 is a schematic structural diagram of a third sensor array in an HVS image mode according to an embodiment of the present invention;
图13是本发明实施例涉及的在APS图像模式下的像素传感模式示意图；FIG13 is a schematic diagram of a pixel sensing mode in an APS image mode according to an embodiment of the present invention;
图14是本发明实施例涉及的在全尺寸EVS图像模式下的像素传感模式示意图；14 is a schematic diagram of a pixel sensing mode in a full-size EVS image mode according to an embodiment of the present invention;
图15是本发明实施例涉及的在部分尺寸EVS图像模式下的第一像素传感模式示意图；15 is a schematic diagram of a first pixel sensing mode in a partial-size EVS image mode according to an embodiment of the present invention;
图16是本发明实施例涉及的在部分尺寸EVS图像模式下的第二像素传感模式示意图；16 is a schematic diagram of a second pixel sensing mode in a partial-size EVS image mode according to an embodiment of the present invention;
图17是本发明实施例涉及的在HVS图像模式下的第一像素传感模式示意图；17 is a schematic diagram of a first pixel sensing mode in an HVS image mode according to an embodiment of the present invention;
图18是本发明实施例涉及的HVS图像模式下的第二像素传感模式示意图；18 is a schematic diagram of a second pixel sensing mode in an HVS image mode according to an embodiment of the present invention;
图19是本发明实施例涉及的折线传感阵列在APS图像模式下的像素传感模式示意图；FIG19 is a schematic diagram of a pixel sensing mode of a fold line sensing array in an APS image mode according to an embodiment of the present invention;
图20是本发明实施例涉及的折线传感阵列在HVS图像模式下的像素传感模式示意图；FIG20 is a schematic diagram of a pixel sensing mode of a fold line sensing array in an HVS image mode according to an embodiment of the present invention;
图21是本发明实施例中涉及的第一折线像素阵列的结构示意图；FIG21 is a schematic structural diagram of a first fold line pixel array involved in an embodiment of the present invention;
图22是本发明实施例中第一折线像素阵列在HVS图像模式下的结构示意图；FIG22 is a schematic diagram of the structure of the first fold line pixel array in the HVS image mode according to an embodiment of the present invention;
图23是本发明实施例中涉及的第二折线像素阵列的结构示意图；FIG23 is a schematic diagram of the structure of a second fold line pixel array involved in an embodiment of the present invention;
图24是本发明实施例中第二折线像素阵列在HVS图像模式下的结构示意图；24 is a schematic diagram of the structure of the second fold line pixel array in the HVS image mode according to an embodiment of the present invention;
图25是本发明实施例中涉及的第三折线像素阵列的结构示意图；FIG25 is a schematic diagram of the structure of a third fold line pixel array involved in an embodiment of the present invention;
图26是本发明实施例中第三折线像素阵列在HVS图像模式下的结构示意图；26 is a schematic diagram of the structure of a third fold line pixel array in HVS image mode according to an embodiment of the present invention;
图27是本发明一可选实施例中光电传感设备的结构示意图。FIG. 27 is a schematic diagram of the structure of a photoelectric sensing device in an optional embodiment of the present invention.
具体实施方式DETAILED DESCRIPTION
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明的一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
另外，在本发明中如涉及“第一”、“第二”等的描述仅用于描述目的，而不能理解为指示或暗示其相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。另外，各个实施例之间的技术方案可以相互结合，但是必须是以本领域普通技术人员能够实现为基础，当技术方案的结合出现相互矛盾或无法实现时应当认为这种技术方案的结合不存在，也不在本发明要求的保护范围之内。In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
随着事件型视觉传感器（Event-based Vision Sensor，简称EVS）的兴起，融合了有源积分型像素传感器（Active Pixel Sensor，简称APS）与事件型视觉传感器的复合型图像传感器随之应运而生。With the rise of event-based vision sensors (EVS), composite image sensors that combine active pixel sensors (APS) and event-based vision sensors have emerged.
但是，相关的融合方案存在不少缺陷。However, the relevant fusion solutions have many defects.
（1）第一种融合方案为：对EVS中光电二极管所产生的光电流进行复制以形成APS，从而实现EVS与APS的相互融合。但是，这种方案在光电流的复制过程中容易引入大量的噪声，从而严重影响了APS的成像质量；而且，无法实现单个像素同时输出有源积分型图像数据（以下称为APS图像数据）和事件型图像数据（以下称为EVS图像数据）。(1) The first fusion scheme is to replicate the photocurrent generated by the photodiode in the EVS to form an APS, thereby realizing the mutual fusion of the EVS and the APS. However, this scheme easily introduces a large amount of noise in the process of photocurrent replication, which seriously affects the imaging quality of the APS; moreover, it is impossible to achieve a single pixel outputting active integral image data (hereinafter referred to as APS image data) and event image data (hereinafter referred to as EVS image data) at the same time.
（2）第二种融合方案为：交错设置固定的有源积分型像素（以下称为EVS像素）和事件型像素（以下称为APS像素），从而实现EVS与APS的相互融合。但是，这种方案的分辨率不高，对APS图像和EVS图像的成像质量均会造成影响；EVS像素和APS像素都需要用于检测光的光电转换器件，而光电转换器件会占据图像传感器的大部分空间，从而增加了图像传感器的尺寸；同时，也无法实现单个像素兼容输出APS图像数据和EVS图像数据。(2) The second fusion scheme is to stagger fixed active integration pixels (hereinafter referred to as EVS pixels) and event pixels (hereinafter referred to as APS pixels) to achieve mutual fusion of EVS and APS. However, the resolution of this scheme is not high, which will affect the imaging quality of both APS and EVS images; both EVS pixels and APS pixels require photoelectric conversion devices for detecting light, and photoelectric conversion devices will occupy most of the space of the image sensor, thereby increasing the size of the image sensor; at the same time, it is impossible to achieve compatible output of APS image data and EVS image data by a single pixel.
因此，目前亟需一种显示模式灵活可调且单个像素能兼容输出APS图像数据和EVS图像数据的高质量复合型图像传感器。Therefore, there is an urgent need for a high-quality composite image sensor with a flexible and adjustable display mode and a single pixel that can output both APS image data and EVS image data.
为了解决上述问题，本发明实施例提出一种图像传感器。In order to solve the above problems, an embodiment of the present invention provides an image sensor.
参照图1，在一实施例中，图像传感器包括：1 , in one embodiment, the image sensor includes:
像素，被配置有光电转换模块、第一传输支路以及第二传输支路；光电转换模块基于控制信号将接收到的入射光转换为光强积分信号或光强对数信号，通过第一传输支路输出光强积分信号，或通过第二传输支路输出光强对数信号；The pixel is configured with a photoelectric conversion module, a first transmission branch and a second transmission branch; the photoelectric conversion module converts the received incident light into a light intensity integral signal or a light intensity logarithmic signal based on a control signal, outputs the light intensity integral signal through the first transmission branch, or outputs the light intensity logarithmic signal through the second transmission branch;
处理模块，包括有源积分型像素读出单元和事件型像素读出单元，有源积分型像素读出单元接收第一传输支路输出的光强积分信号，处理得到APS图像数据，事件型像素读出单元接收第二传输支路输出的光强对数信号，处理得到EVS图像数据。The processing module includes an active integration pixel readout unit and an event type pixel readout unit. The active integration pixel readout unit receives the light intensity integration signal output by the first transmission branch and processes it to obtain APS image data. The event type pixel readout unit receives the light intensity logarithmic signal output by the second transmission branch and processes it to obtain EVS image data.
在本实施例中，该图像传感器包括像素阵列，像素阵列对应有多个阵列设置的像素，如图1所示，每个像素对应产生的图像数据可以用于合成图像。而像素中包括用于将接收到的入射光转换为光强积分信号或光强对数信号的光电转换模块，而光电转换模块输出光强积分信号或光强对数信号可以通过控制信号灵活控制，以及与光电转换模块连接的第一传输支路和第二传输支路。需要说明的是，在本实施例中，连接一般是指两个部件或电路间具有电气连接。In this embodiment, the image sensor includes a pixel array, and the pixel array corresponds to a plurality of pixels arranged in an array, as shown in FIG1 , and the image data corresponding to each pixel can be used to synthesize an image. The pixel includes a photoelectric conversion module for converting the received incident light into a light intensity integral signal or a light intensity logarithmic signal, and the light intensity integral signal or the light intensity logarithmic signal output by the photoelectric conversion module can be flexibly controlled by a control signal, as well as a first transmission branch and a second transmission branch connected to the photoelectric conversion module. It should be noted that in this embodiment, connection generally refers to an electrical connection between two components or circuits.
第一传输支路和第二传输支路可以基于不同的控制信号分别与光电转换模块进行连通，同样地，光电转换模块也可以基于不同的控制信号将入射光转换为光强积分信号或光强对数信号。控制信号是由控制器根据对该像素输出的图像数据类型需求而生成并发送的。控制信号包括对光电转换模块的控制信号，以及对第一传输支路和第二传输支路与光电转换模块之间连通关系的控制信号。The first transmission branch and the second transmission branch can be connected to the photoelectric conversion module based on different control signals. Similarly, the photoelectric conversion module can also convert the incident light into a light intensity integral signal or a light intensity logarithmic signal based on different control signals. The control signal is generated and sent by the controller according to the image data type requirement of the pixel output. The control signal includes a control signal for the photoelectric conversion module and a control signal for the connection relationship between the first transmission branch and the second transmission branch and the photoelectric conversion module.
若需要像素输出APS图像数据，图像传感器的控制器则可以向像素输出有源积分型像素模式（以下称为APS模式）对应的控制信号，控制光电转换模块与第一传输支路接通，以及控制光电转换模块将接收到的入射光转换为光强积分信号，从而使得像素可以通过第一传输支路输出光强积分信号。If the pixel is required to output APS image data, the controller of the image sensor can output a control signal corresponding to the active integral pixel mode (hereinafter referred to as the APS mode) to the pixel, control the photoelectric conversion module to be connected to the first transmission branch, and control the photoelectric conversion module to convert the received incident light into a light intensity integral signal, so that the pixel can output the light intensity integral signal through the first transmission branch.
若需要像素输出EVS图像数据，图像传感器的控制器则可以向像素输出事件型像素模式（以下称为EVS模式）对应的控制信号，控制光电转换模块与第二传输支路接通，以及控制光电转换模块将接收到的入射光转换为光强对数信号，从而像素可以通过第二传输支路输出光强对数信号。If the pixel is required to output EVS image data, the controller of the image sensor can output a control signal corresponding to an event-type pixel mode (hereinafter referred to as the EVS mode) to the pixel, control the photoelectric conversion module to be connected to the second transmission branch, and control the photoelectric conversion module to convert the received incident light into a logarithmic light intensity signal, so that the pixel can output a logarithmic light intensity signal through the second transmission branch.
该图像传感器还包括处理模块，该处理模块将像素输出的电学信号转换为图像数据。对应地，处理模块包括有源积分型像素读出单元和事件型像素读出单元，有源积分型像素读出单元与第一传输支路连接，接收第一传输支路输出的光强积分信号，事件型像素读出单元与第二传输支路连接，接收第二传输支路输出的光强对数信号。The image sensor also includes a processing module, which converts the electrical signal output by the pixel into image data. Correspondingly, the processing module includes an active integration type pixel readout unit and an event type pixel readout unit, the active integration type pixel readout unit is connected to the first transmission branch and receives the light intensity integration signal output by the first transmission branch, and the event type pixel readout unit is connected to the second transmission branch and receives the light intensity logarithmic signal output by the second transmission branch.
在本实施例公开的技术方案中，基于控制信号可以使光电转换模块基于接收到的入射光，可控地形成光强积分信号或光强对数信号，然后可以控制光电转换模块将光强积分信号通过第一传输支路输出给对应的有源积分型像素读出单元，或者控制光电转换模块将光强对数信号通过第二传输支路输出给对应的事件型像素读出单元，对应处理可以得到APS图像数据或者EVS图像数据，从而基于一个像素即可以生成EVS图像数据，又可以生成APS图像数据，实现单个像素兼容输出APS图像数据和EVS图像数据，通过这样的像素构成兼容APS模式和EVS模式的复合型图像传感器，使得兼容APS模式和EVS模式的复合型图像传感器生成的图像数据分辨率更高，不仅减少了复合型图像传感器的体积，还有效地提高了复合型图像传感器的成像质量。另外，配合控制器的对应控制，能使得上述复合型图像传感器的工作模式灵活可调，特别是事件型图像模式和混合图像模式下，需要进行事件感知的区域位置和对应分辨率灵活可调，适用于多种应用场景。In the technical solution disclosed in the present embodiment, based on the control signal, the photoelectric conversion module can be controllably formed into a light intensity integral signal or a light intensity logarithmic signal based on the received incident light, and then the photoelectric conversion module can be controlled to output the light intensity integral signal to the corresponding active integral pixel readout unit through the first transmission branch, or the photoelectric conversion module can be controlled to output the light intensity logarithmic signal to the corresponding event-type pixel readout unit through the second transmission branch. The corresponding processing can obtain APS image data or EVS image data, so that both EVS image data and APS image data can be generated based on one pixel, and a single pixel can be compatible with outputting APS image data and EVS image data. A composite image sensor compatible with the APS mode and the EVS mode is formed by such pixels, so that the image data generated by the composite image sensor compatible with the APS mode and the EVS mode has a higher resolution, which not only reduces the volume of the composite image sensor, but also effectively improves the imaging quality of the composite image sensor. In addition, with the corresponding control of the controller, the working mode of the above-mentioned composite image sensor can be flexibly adjusted, especially in the event image mode and the mixed image mode, the area position and the corresponding resolution required for event perception can be flexibly adjusted, which is suitable for a variety of application scenarios.
在一些可选实施例中，参照图2，有源积分型像素读出单元包括有源积分型像素读出电路（即APS读出电路）等，事件型像素读出单元包括事件型像素后端电路（即EVS后端电路）、事件型像素阵列驱动电路（即EVS阵列驱动电路）与行扫仲裁电路、事件型像素列读出电路（即EVS列读出电路）与列扫仲裁电路等。事件型像素后端电路阵列（即EVS后端电路阵列）与EVS阵列驱动电路与行扫仲裁电路连接，EVS后端电路阵列包括多个呈阵列设置的EVS后端电路。像素的第一传输支路与APS读出电路连接，像素的第二传输支路与EVS后端电路连接。In some optional embodiments, referring to FIG. 2 , the active integral pixel readout unit includes an active integral pixel readout circuit (i.e., an APS readout circuit), etc., and the event pixel readout unit includes an event pixel backend circuit (i.e., an EVS backend circuit), an event pixel array drive circuit (i.e., an EVS array drive circuit) and a row scan arbitration circuit, an event pixel column readout circuit (i.e., an EVS column readout circuit) and a column scan arbitration circuit, etc. The event pixel backend circuit array (i.e., an EVS backend circuit array) is connected to the EVS array drive circuit and the row scan arbitration circuit, and the EVS backend circuit array includes a plurality of EVS backend circuits arranged in an array. The first transmission branch of the pixel is connected to the APS readout circuit, and the second transmission branch of the pixel is connected to the EVS backend circuit.
在一些可选实施例中，图像传感器包括像素阵列，像素阵列包括多个呈阵列设置的像素单元，像素单元可以包括一个或多个上述的像素。对应地，事件型像素读出单元中包括EVS后端电路阵列。EVS后端电路阵列可以通过键合连接的方式接像素阵列，以获得像素阵列输出的信号，结合事件型像素读出单元中的其他部件处理得到像素阵列对应的EVS图像数据。其中，每个像素可以分别与对应的EVS后端电路连接，同时每个像素还可以分别与对应的APS读出电路连接，以使每个像素的传感模式分别包括APS模式和EVS模式。图像传感器中还可以包括控制器，控制器是根据需求调整每个像素的传感模式，以调节图像传感器的工作模式的部件。In some optional embodiments, the image sensor includes a pixel array, the pixel array includes a plurality of pixel units arranged in an array, and the pixel unit may include one or more of the above-mentioned pixels. Correspondingly, the event-type pixel readout unit includes an EVS back-end circuit array. The EVS back-end circuit array can be connected to the pixel array by bonding to obtain the signal output by the pixel array, and the EVS image data corresponding to the pixel array is obtained by processing in combination with other components in the event-type pixel readout unit. Among them, each pixel can be respectively connected to the corresponding EVS back-end circuit, and each pixel can also be respectively connected to the corresponding APS readout circuit, so that the sensing mode of each pixel includes APS mode and EVS mode respectively. The image sensor may also include a controller, which is a component that adjusts the sensing mode of each pixel according to demand to adjust the working mode of the image sensor.
在一些可选实施例中，EVS后端电路阵列与EVS阵列驱动电路与行扫仲裁电路、EVS列读出电路与列扫仲裁电路分别连接，通过EVS列读出电路与行扫仲裁电路对各个EVS后端电路进行驱动并行扫仲裁，通过EVS列读出电路与列扫仲裁电路对各个EVS后端电路进行列扫仲裁。EVS后端电路对像素输出的光强对数信号进行采样和比较，当检测到的光强对数信号超过设定阈值后会产生有关光强强弱变化的事件信号，再通过EVS阵列驱动电路与行扫仲裁电路和EVS列读出电路与列扫仲裁电路读出事件信号，该事件信号为数字信号，可以记为事件型数字信号，该事件型数字信号可以作为该像素及其对应像素单元、光电转换模块等主体对应的EVS图像数据。In some optional embodiments, the EVS back-end circuit array and the EVS array driving circuit are connected to the row scanning arbitration circuit, the EVS column readout circuit and the column scanning arbitration circuit respectively, and each EVS back-end circuit is driven to perform parallel scanning arbitration through the EVS column readout circuit and the row scanning arbitration circuit, and each EVS back-end circuit is subjected to column scanning arbitration through the EVS column readout circuit and the column scanning arbitration circuit. The EVS back-end circuit samples and compares the light intensity logarithmic signal output by the pixel, and when the detected light intensity logarithmic signal exceeds the set threshold, an event signal related to the change of light intensity is generated, and then the event signal is read out through the EVS array driving circuit and the row scanning arbitration circuit and the EVS column readout circuit and the column scanning arbitration circuit. The event signal is a digital signal, which can be recorded as an event-type digital signal, and the event-type digital signal can be used as the EVS image data corresponding to the pixel and its corresponding pixel unit, photoelectric conversion module and other subjects.
在一些可选实施例中，APS读出电路也可以通过键合连接的方式接像素阵列。通过APS读出电路对各个像素产生的光强积分信号以及复位信号进行采样和放大处理，再通过模数转换器ADC转换成数字信号，该数字信号记为有源积分型数字信号，也是该像素及其对应像素、光电转换模块等主体对应的APS图像数据。In some optional embodiments, the APS readout circuit can also be connected to the pixel array by bonding. The light intensity integral signal and reset signal generated by each pixel are sampled and amplified by the APS readout circuit, and then converted into a digital signal by an analog-to-digital converter ADC. The digital signal is recorded as an active integral digital signal, which is also the APS image data corresponding to the pixel and its corresponding pixel, photoelectric conversion module and other entities.
在一些可选实施例中，图像传感器还包括像素驱动电路，像素驱动电路通过键合连接的方式接像素阵列排布而组成的像素阵列，与像素中的像素连接。在控制器的控制下，通过像素驱动电路对像素阵列中各像素的像素进行驱动控制，实现像素中各光电转换模块的采样操作与复位操作，以生成光强积分信号或光强对数信号。In some optional embodiments, the image sensor further includes a pixel driving circuit, which is connected to the pixel array formed by the pixel array arrangement by bonding connection, and is connected to the pixels in the pixel array. Under the control of the controller, the pixel driving circuit drives and controls the pixels of each pixel in the pixel array to implement the sampling operation and reset operation of each photoelectric conversion module in the pixel, so as to generate a light intensity integral signal or a light intensity logarithmic signal.
在一些可选实施例中，图像传感器还包括处理器，处理器与上述模数转换器ADC、EVS列读出电路与列扫仲裁电路分别连接，处理器接收像素在EVS模式时，EVS列读出电路与列扫仲裁电路对应输出的EVS图像数据，或者像素在APS模式时，模数转换器ADC输出的APS图像数据。处理器在接收到两种模式的数字信号后，在控制器的控制下对EVS图像数据和APS图像这两种数字信号数据进行数据处理，并通过数据交织电路对以上两种数据做数据交织并通过高速接口输出，最终实现图像传感器的APS成像、EVS成像或APS和EVS融合成像。In some optional embodiments, the image sensor further includes a processor, which is connected to the above-mentioned analog-to-digital converter ADC, the EVS column readout circuit and the column scan arbitration circuit respectively, and the processor receives the EVS image data correspondingly output by the EVS column readout circuit and the column scan arbitration circuit when the pixel is in the EVS mode, or the APS image data output by the analog-to-digital converter ADC when the pixel is in the APS mode. After receiving the digital signals of the two modes, the processor processes the two digital signal data, the EVS image data and the APS image, under the control of the controller, and interleaves the above two data through the data interleaving circuit and outputs them through the high-speed interface, finally realizing the APS imaging, EVS imaging or APS and EVS fusion imaging of the image sensor.
其中，处理器包括图像信号处理器（ISP）及事件信号处理器（ESP)，图像信号处理器用于对APS模式下的APS图像数据进行后期处理，如暗场校正、线性校正、坏点校正、数字增益控制、白平衡、自动曝光控制、自动对焦数据处理、降噪、数据压缩等。Among them, the processor includes an image signal processor (ISP) and an event signal processor (ESP). The image signal processor is used to perform post-processing on the APS image data in the APS mode, such as dark field correction, linear correction, bad pixel correction, digital gain control, white balance, automatic exposure control, autofocus data processing, noise reduction, data compression, etc.
需要说明的是，每个像素分别与APS读出电路、EVS后端电路连接，这使得每个像素的传感模式可以是APS模式和EVS模式中的一个。与此对应地，控制器根据不同需求可以将图像传感器配置为不同的工作模式。It should be noted that each pixel is connected to the APS readout circuit and the EVS back-end circuit respectively, so that the sensing mode of each pixel can be one of the APS mode and the EVS mode. Correspondingly, the controller can configure the image sensor to different working modes according to different requirements.
可选地，光电转换模块包括光电输出电路和复位电路，光电输出电路和复位电路之间形成浮动扩散节点FD，第一传输支路以及第二传输支路通过浮动扩散节点FD接光电转换模块；Optionally, the photoelectric conversion module includes a photoelectric output circuit and a reset circuit, a floating diffusion node FD is formed between the photoelectric output circuit and the reset circuit, and the first transmission branch and the second transmission branch are connected to the photoelectric conversion module through the floating diffusion node FD;
在第一传输支路与光电转换模块基于控制信号接通时，若光电转换模块处于复位清空状态，复位电路复位光电输出电路；若光电转换模块处于曝光积分状态，光电输出电路基于入射光产生积分电荷；若光电转换模块处于读出状态，光电输出电路基于积分电荷在浮动扩散节点形成光强积分信号；When the first transmission branch is connected to the photoelectric conversion module based on the control signal, if the photoelectric conversion module is in a reset clearing state, the reset circuit resets the photoelectric output circuit; if the photoelectric conversion module is in an exposure integration state, the photoelectric output circuit generates an integrated charge based on the incident light; if the photoelectric conversion module is in a readout state, the photoelectric output circuit forms a light intensity integration signal at the floating diffusion node based on the integrated charge;
在第二传输支路与光电转换模块基于控制信号接通时，复位电路向光电转换模块提供基准电压，光电输出电路基于入射光输出的光电流与基准电压在浮动扩散节点形成光强对数信号，并向第二传输支路输出光强对数信号。When the second transmission branch and the photoelectric conversion module are connected based on the control signal, the reset circuit provides a reference voltage to the photoelectric conversion module, and the photoelectric output circuit forms a light intensity logarithmic signal at the floating diffusion node based on the photocurrent output by the incident light and the reference voltage, and outputs the light intensity logarithmic signal to the second transmission branch.
在本实施例中，像素的光电转换模块包括光电输出电路和复位电路，光电输出电路可以基于光电效应，基于其接收到的入射光在光电二极管产生并累积电荷或者通过光电二极管形成光电流，对应的在浮动扩散节点FD产生积分电压或者对数电压，参照图3。复位电路与光电输出电路连接，复位电路可以对光电输出电路进行复位，从而清空光电输出电路因为光电效应而累积的积分电荷，将光电输出电路恢复至未产生光电效应的电学状态。光电输出电路和复位电路之间形成浮动扩散节点FD，通过对光电输出电路和复位电路的控制，可以在浮动扩散节点形成不同的电压信号，电压信号中包括生成图像数据所需要的光强积分信号、光强对数信号和复位信号等。In this embodiment, the photoelectric conversion module of the pixel includes a photoelectric output circuit and a reset circuit. The photoelectric output circuit can generate and accumulate charges in the photodiode based on the incident light received by the photoelectric effect or form a photocurrent through the photodiode, and correspondingly generate an integral voltage or a logarithmic voltage at the floating diffusion node FD, refer to Figure 3. The reset circuit is connected to the photoelectric output circuit, and the reset circuit can reset the photoelectric output circuit, thereby clearing the integral charge accumulated by the photoelectric output circuit due to the photoelectric effect, and restoring the photoelectric output circuit to an electrical state where no photoelectric effect is generated. A floating diffusion node FD is formed between the photoelectric output circuit and the reset circuit. By controlling the photoelectric output circuit and the reset circuit, different voltage signals can be formed at the floating diffusion node, and the voltage signal includes the light intensity integral signal, light intensity logarithmic signal and reset signal required to generate image data.
若需要得到像素对应APS图像数据，则需要像素处于APS模式，APS模式要求第一传输支路与光电转换模块接通，以及像素输出光电积分信号。在第一传输支路将光电转换模块和有源积分型像素单元接通时，第一传输支路才能将光电转换模块生成的光强积分信号输出至有源积分型像素读出单元，由有源积分型像素读出单元将其转化为像素对应APS图像数据。而当第一传输支路与光电转换模块接通时，则表明像素处于APS模式，像素或者像素中光电转换模块需要在APS模式下，使得像素或像素中光电转换模块可以通过第一传输支路输出光强积分信号。If you need to obtain APS image data corresponding to the pixel, the pixel needs to be in APS mode. The APS mode requires the first transmission branch to be connected to the photoelectric conversion module and the pixel to output a photoelectric integration signal. When the first transmission branch connects the photoelectric conversion module and the active integration pixel unit, the first transmission branch can output the light intensity integration signal generated by the photoelectric conversion module to the active integration pixel readout unit, which converts it into APS image data corresponding to the pixel. When the first transmission branch is connected to the photoelectric conversion module, it indicates that the pixel is in APS mode, and the pixel or the photoelectric conversion module in the pixel needs to be in APS mode, so that the pixel or the photoelectric conversion module in the pixel can output the light intensity integration signal through the first transmission branch.
具体地，在APS模式下，光电转换模块存在三种状态：复位清空状态，曝光积分状态，读出状态，分别通过复位清空信号、曝光积分信号、读出信号控制光电转换模块进入不同的状态。Specifically, in the APS mode, the photoelectric conversion module has three states: reset clear state, exposure integration state, and readout state. The photoelectric conversion module is controlled to enter different states by a reset clear signal, an exposure integration signal, and a readout signal, respectively.
具体地，若第一传输支路与光电转换模块接通时，控制器通过复位清空信号、曝光积分信号、读出信号控制光电转换模块依次进入复位清空状态，曝光积分状态，读出状态。Specifically, if the first transmission branch is connected to the photoelectric conversion module, the controller controls the photoelectric conversion module to enter the reset and clear state, the exposure and integration state, and the readout state in sequence through the reset and clear signal, the exposure and integration signal, and the readout signal.
当光电转换模块接收到复位清空信号时，光电转换模块进入复位清空状态，当光电转换模块进入复位清空状态，通过复位电路清空光电输出电路因为曝光而产生的电荷，从而实现复位电路对光电输出电路复位。然后控制器向光电转换模块发送曝光积分信号。When the photoelectric conversion module receives the reset clear signal, the photoelectric conversion module enters the reset clear state. When the photoelectric conversion module enters the reset clear state, the reset circuit clears the charge generated by the photoelectric output circuit due to exposure, thereby realizing the reset circuit resetting the photoelectric output circuit. Then the controller sends an exposure integration signal to the photoelectric conversion module.
当光电转换模块接收到曝光积分信号时，光电转换模块进入曝光积分状态。当光电转换模块进入曝光积分状态，光电输出电路基于当前接收到的入射光形成积分电荷，积分电荷是通过光电效应不断产生并累积的电荷。When the photoelectric conversion module receives the exposure integration signal, the photoelectric conversion module enters the exposure integration state. When the photoelectric conversion module enters the exposure integration state, the photoelectric output circuit forms an integral charge based on the currently received incident light. The integral charge is the charge that is continuously generated and accumulated through the photoelectric effect.
当光电转换模块处于曝光积分状态达到预设时长时，光电转换模块结束曝光，控制器向光电转换模块发送读出信号。当光电转换模块接收到读出信号时，光电转换模块进入读出状态，光电输出电路将在曝光积分状态下产生的积分电荷释放，在浮动扩散节点FD产生电压，形成光强积分信号，光强积分信号将通过第一传输支路传输至有源积分型像素读出单元，通过处理得到该光电转换模块对应的APS图像数据。When the photoelectric conversion module is in the exposure integration state for a preset time, the photoelectric conversion module ends the exposure, and the controller sends a readout signal to the photoelectric conversion module. When the photoelectric conversion module receives the readout signal, the photoelectric conversion module enters the readout state, and the photoelectric output circuit releases the integrated charge generated in the exposure integration state, generates a voltage at the floating diffusion node FD, and forms a light intensity integration signal. The light intensity integration signal is transmitted to the active integration pixel readout unit through the first transmission branch, and the APS image data corresponding to the photoelectric conversion module is obtained through processing.
若需要得到像素对应EVS图像数据，则需要像素处于EVS模式，EVS模式要求第二传输支路与光电转换模块接通，以及像素输出光电变化信号。在第二传输支路将光电转换模块和事件型像素读出单元接通时，第二传输支路才能将光电转换模块生成的光强对数信号输出至事件型像素读出单元，由事件型像素读出单元将其转化为像素对应EVS图像数据。而当第二传输支路与光电转换模块接通时，则表明像素处于EVS模式，使得像素或者像素中光电转换模块，通过第二传输支路输出光强对数信号。If you need to obtain the EVS image data corresponding to the pixel, the pixel needs to be in EVS mode. The EVS mode requires the second transmission branch to be connected to the photoelectric conversion module and the pixel to output a photoelectric change signal. When the second transmission branch connects the photoelectric conversion module and the event-type pixel readout unit, the second transmission branch can output the light intensity logarithmic signal generated by the photoelectric conversion module to the event-type pixel readout unit, and the event-type pixel readout unit converts it into the EVS image data corresponding to the pixel. When the second transmission branch is connected to the photoelectric conversion module, it indicates that the pixel is in EVS mode, so that the pixel or the photoelectric conversion module in the pixel outputs a light intensity logarithmic signal through the second transmission branch.
复位电路可以持续向光电转换模块提供基准电压，光电输出电路对入射光进行光电感应后转化的电荷会使其形成光电流，由于光电流和基准电压，此时会在浮动扩散节点FD上产生一个与上述光电流呈log关系的电压，称为对数电压，该对数电压可以作为光强对数信号。事件型像素读出单元对该光强对数信号进行转换、采样并判断光强是否发生了变化，若变化超过了设定阈值，则会对应产生光强变化的事件数据，继续进行向后端传输处理，最终得到光电转换模块对应的EVS图像数据。The reset circuit can continuously provide a reference voltage to the photoelectric conversion module. The charge converted by the photoelectric output circuit after photoelectric sensing of the incident light will form a photocurrent. Due to the photocurrent and the reference voltage, a voltage with a logarithmic relationship with the above-mentioned photocurrent will be generated on the floating diffusion node FD at this time. This voltage is called a logarithmic voltage, which can be used as a light intensity logarithmic signal. The event-type pixel readout unit converts and samples the light intensity logarithmic signal and determines whether the light intensity has changed. If the change exceeds the set threshold, the event data corresponding to the light intensity change will be generated, and the data will continue to be transmitted to the back end for processing, and finally the EVS image data corresponding to the photoelectric conversion module will be obtained.
在本实施例公开的技术方案中，通过光电转换模块与传输支路的连通状态，控制光电输出电路与复位电路形成不同的光学信号，从而实现了各像素的传感模式的可调性，有效地提高了复合型图像传感器的成像质量。In the technical solution disclosed in this embodiment, the photoelectric output circuit and the reset circuit are controlled to form different optical signals through the connectivity state of the photoelectric conversion module and the transmission branch, thereby achieving the adjustability of the sensing mode of each pixel and effectively improving the imaging quality of the composite image sensor.
可选地，参照图3，光电输出电路包括光电二极管PD、传输晶体管TG，复位电路包括复位晶体管RST和第一复位电源VPIX；Optionally, referring to FIG. 3 , the photoelectric output circuit includes a photodiode PD, a transfer transistor TG, and the reset circuit includes a reset transistor RST and a first reset power supply VPIX;
光电二极管的阳极接地，光电二极管的阴极接传输晶体管的第一端，复位晶体管的第一端接传输晶体管的第二端，复位晶体管的第二端接第一复位电源，传输晶体管的控制端接像素控制信号，复位晶体管的控制端接复位控制信号；The anode of the photodiode is grounded, the cathode of the photodiode is connected to the first end of the transfer transistor, the first end of the reset transistor is connected to the second end of the transfer transistor, the second end of the reset transistor is connected to the first reset power supply, the control end of the transfer transistor is connected to the pixel control signal, and the control end of the reset transistor is connected to the reset control signal;
当像素控制信号为高电平状态、复位控制信号为高电平状态时，即为复位清空信号，光电转换模块进入复位清空状态；When the pixel control signal is in a high level state and the reset control signal is in a high level state, it is a reset clear signal, and the photoelectric conversion module enters a reset clear state;
当像素控制信号为低电平状态、复位控制信号为低电平状态时，即为曝光积分信号，光电转换模块进入曝光积分状态；When the pixel control signal is in a low level state and the reset control signal is in a low level state, it is an exposure integration signal, and the photoelectric conversion module enters an exposure integration state;
当像素控制信号为高电平状态、复位控制信号为低电平状态时，即为读出信号，光电转换模块进入读出状态。When the pixel control signal is in a high level state and the reset control signal is in a low level state, it is a readout signal, and the photoelectric conversion module enters a readout state.
在本实施例中，参照图3，光电输出电路包括光电二极管PD，该光电二极管PD的阳极接地，光电二极管PD的阴极接传输晶体管TG的第一端，这样当光电二极管PD通过光电效应产生电荷后，会输出电压值传输晶体管，若传输晶体管接通，则会输出至浮动扩散节点FD。传输晶体管TG的第二端与接复位电路中复位晶体管RST的第一端，复位电路除了包括复位晶体管RST之外，还包括第一复位电源VPIX，第一复位电源VPIX可以向复位晶体管RST提供稳定电压作为复位电压。传输晶体管TG的第二端和复位晶体管RST的第一端之间，则会形成浮动扩散节点FD，该浮动扩散节点FD也是第一传输支路以及第二传输支路接入光电转换模块的节点。In this embodiment, referring to FIG. 3 , the photoelectric output circuit includes a photodiode PD, the anode of the photodiode PD is grounded, and the cathode of the photodiode PD is connected to the first end of the transfer transistor TG, so that when the photodiode PD generates charge through the photoelectric effect, it will output a voltage value to the transfer transistor, and if the transfer transistor is turned on, it will be output to the floating diffusion node FD. The second end of the transfer transistor TG is connected to the first end of the reset transistor RST in the reset circuit. In addition to the reset transistor RST, the reset circuit also includes a first reset power supply VPIX, and the first reset power supply VPIX can provide a stable voltage as a reset voltage to the reset transistor RST. A floating diffusion node FD is formed between the second end of the transfer transistor TG and the first end of the reset transistor RST, and the floating diffusion node FD is also the node where the first transmission branch and the second transmission branch are connected to the photoelectric conversion module.
传输晶体管TG、复位晶体管RST的第三端作为控制端接控制器，通过向传输晶体管TG的控制端输出像素控制信号TX，通过向复位晶体管RST的控制端输入复位控制信号RS。像素控制信号TX和复位控制信号RS可以是脉冲信号，像素控制信号TX和复位控制信号RS中均会出现交替高电平状态和低电平状态，通过高电平状态控制对应晶体管的联通，以及通过低电平状态控制对应晶体管的关断，像素控制信号TX和复位控制信号RS的具体时序可以参照图4。The third terminals of the transmission transistor TG and the reset transistor RST are connected to the controller as control terminals, and the pixel control signal TX is output to the control terminal of the transmission transistor TG, and the reset control signal RS is input to the control terminal of the reset transistor RST. The pixel control signal TX and the reset control signal RS can be pulse signals, and both the pixel control signal TX and the reset control signal RS can alternately have high level states and low level states, and the corresponding transistors are controlled to be connected through the high level state, and the corresponding transistors are controlled to be turned off through the low level state. The specific timing of the pixel control signal TX and the reset control signal RS can refer to Figure 4.
在APS模式下，第一传输支路与光电转换模块接通，控制器向光电转换模块发送的复位清空信号包括像素控制信号TX的高电平状态，以及复位控制信号RS的高电平状态。当像素控制信号TX是高电平状态、复位控制信号RS是高电平状态时，高电平状态可以使传输晶体管TG以及复位晶体管RST接通，此时，复位晶体管RST的第一端（即浮动扩散节点FD）被拉高，光电二极管PD被复位。In the APS mode, the first transmission branch is connected to the photoelectric conversion module, and the reset clear signal sent by the controller to the photoelectric conversion module includes the high level state of the pixel control signal TX and the high level state of the reset control signal RS. When the pixel control signal TX is in the high level state and the reset control signal RS is in the high level state, the high level state can turn on the transmission transistor TG and the reset transistor RST. At this time, the first end of the reset transistor RST (i.e., the floating diffusion node FD) is pulled high, and the photodiode PD is reset.
控制器向光电转换模块发送的曝光积分信号包括像素控制信号TX的低电平状态，以及复位控制信号RS的低电平状态。当像素控制信号TX是低电平状态、复位控制信号RS是低电平状态时，传输晶体管TG关断以及复位晶体管RST关断，光电二极管PD基于光电效应进行曝光，光电二极管PD持续一段时间的曝光以累积积分电荷。The exposure integration signal sent by the controller to the photoelectric conversion module includes the low level state of the pixel control signal TX and the low level state of the reset control signal RS. When the pixel control signal TX is in the low level state and the reset control signal RS is in the low level state, the transmission transistor TG is turned off and the reset transistor RST is turned off, and the photodiode PD is exposed based on the photoelectric effect, and the photodiode PD is exposed for a period of time to accumulate integrated charge.
控制器向光电转换模块发送的读出信号包括像素控制信号TX的高电平状态和复位控制信号RS的低电平状态，当像素控制信号TX是高电平状态、复位控制信号RS是低电平状态时，传输晶体管TG接通以及复位晶体管RST关断。传输晶体管TG的连通，可以将光电二极管PD中在曝光积分状态下产生的电荷转移到浮动扩散节点FD处，而后像素控制信号TX由高电平状态切换为低电平状态，传输晶体管TG关断，此时，浮动扩散节点FD处的寄生电容对积分电荷进行电荷与电压转换，形成的线性积分电压信号作为光电积分信号。The readout signal sent by the controller to the photoelectric conversion module includes the high level state of the pixel control signal TX and the low level state of the reset control signal RS. When the pixel control signal TX is in the high level state and the reset control signal RS is in the low level state, the transfer transistor TG is turned on and the reset transistor RST is turned off. The connection of the transfer transistor TG can transfer the charge generated in the photodiode PD in the exposure integration state to the floating diffusion node FD, and then the pixel control signal TX is switched from the high level state to the low level state, and the transfer transistor TG is turned off. At this time, the parasitic capacitance at the floating diffusion node FD converts the integrated charge into a voltage, and the linear integrated voltage signal formed is used as the photoelectric integration signal.
进一步地，当像素控制信号为低电平状态、复位控制信号为高电平状态时，复位电路在浮动扩散节点形成复位信号；Further, when the pixel control signal is in a low level state and the reset control signal is in a high level state, the reset circuit forms a reset signal at the floating diffusion node;
有源积分型像素读出单元基于接收到的复位信号和光强积分信号，处理得到有源图像数据。The active integration pixel readout unit processes the received reset signal and the light intensity integration signal to obtain active image data.
在本实施例中，读出信号还包括复位控制信号RS的高电平状态，以及像素控制信号TX的低电平状态，当像素控制信号TX是低电平状态、复位控制信号RS是高电平状态时，传输晶体管TG关断，复位晶体管RST连通，复位电路可以在浮动扩散节点FD形成复位信号，并通过第一传输支路输出。有源积分型像素读出单元可以接收到相邻的不同时刻下，光电转换模块输出的复位信号和光强积分信号。In this embodiment, the readout signal also includes the high level state of the reset control signal RS and the low level state of the pixel control signal TX. When the pixel control signal TX is in the low level state and the reset control signal RS is in the high level state, the transmission transistor TG is turned off and the reset transistor RST is connected. The reset circuit can form a reset signal at the floating diffusion node FD and output it through the first transmission branch. The active integral pixel readout unit can receive the reset signal and the light intensity integration signal output by the photoelectric conversion module at different adjacent moments.
可选地，参照图4，当像素控制信号TX为低电平状态、复位控制信号RS为低电平状态时，即为曝光积分信号，光电转换模块进入曝光积分状态，曝光积分状态需要持续一定时长，在曝光积分状态结束后，将会进入读出状态，首先像素控制信号TX保持低电平状态，而复位控制信号RS将会被置为高电平状态，使得复位电路可以在浮动扩散节点FD形成复位信号，并通过第一传输支路输出，然后在将复位控制信号RS置为低电平状态后，再将像素控制信号TX置为高电平状态，此时，基于曝光积分状态的积分电荷，浮动扩散节点FD会形成光电积分信号，并通过第一传输支路输出。这样在曝光积分状态后，先输出复位信号，再输出光电积分信号的对应控制流程将更加流畅，效率更高。Optionally, referring to FIG. 4 , when the pixel control signal TX is in a low level state and the reset control signal RS is in a low level state, that is, the exposure integration signal, the photoelectric conversion module enters the exposure integration state, and the exposure integration state needs to last for a certain period of time. After the exposure integration state ends, it will enter the readout state. First, the pixel control signal TX remains in a low level state, and the reset control signal RS will be set to a high level state, so that the reset circuit can form a reset signal at the floating diffusion node FD and output it through the first transmission branch. Then, after the reset control signal RS is set to a low level state, the pixel control signal TX is set to a high level state. At this time, based on the integrated charge of the exposure integration state, the floating diffusion node FD will form a photoelectric integration signal and output it through the first transmission branch. In this way, after the exposure integration state, the corresponding control process of outputting the reset signal first and then outputting the photoelectric integration signal will be smoother and more efficient.
有源积分型像素读出单元中的APS读出电路对复位信号和光电积分信号进行采样，然后通过有源积分型像素读出单元中的模数转换器ADC对采样后的光电积分电压信号与复位信号之间的差值进行数字量化，进而可以得到该光电转换模块对应的有源图像数据（即APS图像数据）。该光电输出单元对应的APS图像数据也属于对应像素的APS图像数据。The APS readout circuit in the active integration pixel readout unit samples the reset signal and the photoelectric integration signal, and then the difference between the sampled photoelectric integration voltage signal and the reset signal is digitally quantized by the analog-to-digital converter ADC in the active integration pixel readout unit, thereby obtaining the active image data (i.e., APS image data) corresponding to the photoelectric conversion module. The APS image data corresponding to the photoelectric output unit also belongs to the APS image data of the corresponding pixel.
需要说明的是，若EVS模式下，第二传输支路与光电转换模块接通，像素控制信号TX和复位控制信号RS持续输出高电平状态，这样传输晶体管TG和复位晶体管RST持续开通，复位控制信号RS为行逻辑电路提供的基准电压，当复位晶体管RST输出的光电流通过复位晶体管RST和传输晶体管TG时会在浮动扩散节点FD上产生一个与光电流呈log关系的电压，记作对数电压。该对数电压可以作为光强对数信号输出至事件型像素读出单元的EVS后端电路，EVS后端电路对该对数电压进行采样并判断光强是否发生了变化，如果转换后的对数电压超过设定的阈值，EVS后端电路会产生对应的事件数据，并通过事件输出总线EBUS向后级传输，经过后级处理可以得到光电输出电路对应的事件图像数据（即EVS图像数据），该光电输出单元对应的EVS图像数据也属于对应像素的EVS图像数据。It should be noted that, if in the EVS mode, the second transmission branch is connected to the photoelectric conversion module, the pixel control signal TX and the reset control signal RS continuously output a high level state, so that the transmission transistor TG and the reset transistor RST are continuously turned on, and the reset control signal RS is a reference voltage provided by the row logic circuit. When the photocurrent output by the reset transistor RST passes through the reset transistor RST and the transmission transistor TG, a voltage having a logarithmic relationship with the photocurrent is generated on the floating diffusion node FD, which is recorded as a logarithmic voltage. The logarithmic voltage can be output as a light intensity logarithmic signal to the EVS back-end circuit of the event-type pixel readout unit. The EVS back-end circuit samples the logarithmic voltage and determines whether the light intensity has changed. If the converted logarithmic voltage exceeds the set threshold, the EVS back-end circuit will generate corresponding event data and transmit it to the subsequent stage through the event output bus EBUS. After the subsequent stage processing, the event image data corresponding to the photoelectric output circuit (i.e., EVS image data) can be obtained. The EVS image data corresponding to the photoelectric output unit also belongs to the EVS image data of the corresponding pixel.
进一步地，在EVS后端电路中，可以定义2bit数据定义为on、off，表示当前对数电压与需要对比的前一次对数电压之间的关系，当本次采样的对数电压比与之比较的前一个对数电压的高于1个阈值时定义为on，当低于1个阈值时为off，当前后两个对数电压的差值介于+/-一个阈值范围内定义为无事件。也就是说，可以简单理解为：10代表光与之前相比变强了；01代表与之前相比光变弱了；00 光强没有发生变化；11异常数据，定义为坏点，处理掉。这里面只有10和01被认为是发生了光强变化事件，其他两种情况是未发生光强变化事件。当发生光强变化事件后，会将此次采样的对数电压，也即光强对数信号，存储在一个电容上，下一次采样时与这个对数电压进行比较，以确定下一个光强变化事件。Furthermore, in the EVS back-end circuit, 2-bit data can be defined as on and off, indicating the relationship between the current logarithmic voltage and the previous logarithmic voltage to be compared. When the logarithmic voltage of this sampling is higher than the previous logarithmic voltage compared with it by 1 threshold, it is defined as on, and when it is lower than 1 threshold, it is off. The difference between the current and the next two logarithmic voltages is defined as no event within +/- a threshold range. In other words, it can be simply understood as: 10 means that the light has become stronger than before; 01 means that the light has become weaker than before; 00 means that the light intensity has not changed; 11 is abnormal data, defined as a bad pixel, and processed. Only 10 and 01 are considered to have light intensity change events, and the other two situations are that no light intensity change events have occurred. When a light intensity change event occurs, the logarithmic voltage sampled this time, that is, the light intensity logarithmic signal, will be stored on a capacitor, and compared with this logarithmic voltage at the next sampling to determine the next light intensity change event.
在本实施例中，通过传输晶体管TG的控制端接像素控制信号TX，复位晶体管RST的控制端接复位控制信号RS，统一控制光电转化电路的输出模式，快速实现了对各像素传感模式的调整，有效地提高了复合型图像传感器的成像质量。In this embodiment, the control terminal of the transmission transistor TG is connected to the pixel control signal TX, and the control terminal of the reset transistor RST is connected to the reset control signal RS, so as to uniformly control the output mode of the photoelectric conversion circuit, quickly adjust the sensing mode of each pixel, and effectively improve the imaging quality of the composite image sensor.
可选地，参照图3，图像传感器还包括：跟随晶体管SF，跟随晶体管SF的第一端接第一传输支路和第二传输支路，跟随晶体管SF的控制端接浮动扩散节点FD，跟随晶体管SF的第二端接第二复位电源VSF。Optionally, referring to FIG. 3 , the image sensor further includes: a follower transistor SF, a first terminal of the follower transistor SF connected to the first transmission branch and the second transmission branch, a control terminal of the follower transistor SF connected to the floating diffusion node FD, and a second terminal of the follower transistor SF connected to the second reset power supply VSF.
跟随晶体管SF的控制端接浮动扩散节点FD，也即与传输晶体管TG的第二端连接，跟随晶体管SF接收在浮动扩散节点FD形成的信号，包括线性积分电压信号、光强对数信号等，可以对线性积分电压信号，也即光强积分信号进行放大处理或跟随输出，还可以对对数电压，也即光强对数信号进行放大处理或跟随输出，以使有源积分型像素读出单元和事件型像素读出单元可以更好地对电压信号进行采样和处理，计算出更准确的图像数据，从而提高图像传感器的成像质量。The control terminal of the follower transistor SF is connected to the floating diffusion node FD, that is, it is connected to the second end of the transfer transistor TG. The follower transistor SF receives the signal formed at the floating diffusion node FD, including a linear integral voltage signal, a light intensity logarithmic signal, etc. The linear integral voltage signal, that is, the light intensity integral signal, can be amplified or followed output, and the logarithmic voltage, that is, the light intensity logarithmic signal can be amplified or followed output, so that the active integral pixel readout unit and the event pixel readout unit can better sample and process the voltage signal and calculate more accurate image data, thereby improving the imaging quality of the image sensor.
可选地，参照图3，第一传输支路上设置有第一选择晶体管SEL1，第一选择晶体管SEL1的第一端接有源积分型像素读出单元，第一选择晶体管SEL1的第二端接跟随晶体管SF的第一端，第一选择晶体管SEL1的控制端接第一读出控制信号READ1，第一选择晶体管SEL1基于第一读出控制信号READ1连通或关断；Optionally, referring to FIG3 , a first selection transistor SEL1 is provided on the first transmission branch, a first terminal of the first selection transistor SEL1 is connected to the active integral pixel readout unit, a second terminal of the first selection transistor SEL1 is connected to the first terminal of the follower transistor SF, a control terminal of the first selection transistor SEL1 is connected to the first readout control signal READ1, and the first selection transistor SEL1 is turned on or off based on the first readout control signal READ1;
第二传输支路上设置有第二选择晶体管SEL2，第二选择晶体管SEL2的第一端接事件型像素读出单元，第二选择晶体管SEL2的第二端接跟随晶体管SF的第一端，第二选择晶体管SEL2的控制端接第二读出控制信号READ2，第二选择晶体管SEL2基于第二读出控制信号READ2连通或关断。A second selection transistor SEL2 is provided on the second transmission branch, a first end of the second selection transistor SEL2 is connected to the event-type pixel readout unit, a second end of the second selection transistor SEL2 is connected to the first end of the follower transistor SF, a control end of the second selection transistor SEL2 is connected to the second readout control signal READ2, and the second selection transistor SEL2 is connected or disconnected based on the second readout control signal READ2.
在本实施例中，有源积分型像素读出单元包括列共享总线ABUS，以及未在图3中示出的与列共享总线ABUS连通的APS读出电路等的APS数据处理部件。事件型像素读出单元包括EVS后端电路、行共享总线EBUS，以及未示出的事件型像素阵列驱动电路与行扫仲裁电路等EVS数据处理部件。在跟随晶体管SF和列共享总线ABUS之间的电路是第一传输支路，在跟随晶体管SF和EVS后端电路之间的电路是第二传输支路。In this embodiment, the active integration type pixel readout unit includes a column sharing bus ABUS, and an APS data processing component such as an APS readout circuit connected to the column sharing bus ABUS, which is not shown in FIG3. The event type pixel readout unit includes an EVS back-end circuit, a row sharing bus EBUS, and EVS data processing components such as an event type pixel array drive circuit and a row scanning arbitration circuit, which are not shown. The circuit between the follower transistor SF and the column sharing bus ABUS is a first transmission branch, and the circuit between the follower transistor SF and the EVS back-end circuit is a second transmission branch.
在第一传输支路上还可以设置有第一选择晶体管SEL1，在第二传输支路上还可以设置有第二选择晶体管SEL2。第一选择晶体管SEL1的第一端接有源积分型像素读出单元，第一选择晶体管SEL1的第二端接跟随晶体管SF的第一端，第一选择晶体管SEL1的控制端接第一读出控制信号READ1。A first selection transistor SEL1 may be further provided on the first transmission branch, and a second selection transistor SEL2 may be further provided on the second transmission branch. A first terminal of the first selection transistor SEL1 is connected to the active integral pixel readout unit, a second terminal of the first selection transistor SEL1 is connected to the first terminal of the follower transistor SF, and a control terminal of the first selection transistor SEL1 is connected to the first readout control signal READ1.
控制器通过向第一选择晶体管SEL1输出第一读出控制信号READ1来控制第一选择晶体管SEL1连通或关断，通过向第一选择晶体管SEL1输出第二读出控制信号READ2来控制第二选择晶体管SEL2连通或关断。当第一读出控制信号READ1为高电平状态时，第一选择晶体管SEL1连通，使得第一传输支路与光电转换模块接通，反之为低电平状态时，第一选择晶体管SEL1关断，使得第二传输支路与光电转换模块断开；当第二读出控制信号READ2为高电平状态时，第二选择晶体管SEL2连通，使得第二传输支路与光电转换模块接通，反之为低电平状态时，第二选择晶体管SEL2关断，使得第二传输支路与光电转换模块断开，第一读出控制信号READ1和第二读出控制信号READ2、像素控制信号TX和复位控制信号RS之间的时序关系可以参照图4。The controller controls the first selection transistor SEL1 to be connected or disconnected by outputting a first readout control signal READ1 to the first selection transistor SEL1, and controls the second selection transistor SEL2 to be connected or disconnected by outputting a second readout control signal READ2 to the first selection transistor SEL1. When the first readout control signal READ1 is in a high level state, the first selection transistor SEL1 is connected, so that the first transmission branch is connected to the photoelectric conversion module, and vice versa, when it is in a low level state, the first selection transistor SEL1 is turned off, so that the second transmission branch is disconnected from the photoelectric conversion module; when the second readout control signal READ2 is in a high level state, the second selection transistor SEL2 is connected, so that the second transmission branch is connected to the photoelectric conversion module, and vice versa, when it is in a low level state, the second selection transistor SEL2 is turned off, so that the second transmission branch is disconnected from the photoelectric conversion module. The timing relationship between the first readout control signal READ1 and the second readout control signal READ2, the pixel control signal TX and the reset control signal RS can refer to Figure 4.
在本实施公开的技术方案中，通过第一读出控制信号READ1和第二读出控制信号READ2实现了自动控制光电转换模块与传输支路的接通，使得各像素的传感模式更加灵活可调，进一步提高图像传感器工作模式的灵活性以及成像质量。In the technical solution disclosed in this embodiment, the connection between the photoelectric conversion module and the transmission branch is automatically controlled through the first readout control signal READ1 and the second readout control signal READ2, so that the sensing mode of each pixel is more flexible and adjustable, further improving the flexibility of the image sensor working mode and the imaging quality.
可选地，第二选择晶体管的第一端接尾流电源；Optionally, the first terminal of the second selection transistor is connected to the tail current power supply;
在第二读出控制信号为高电平状态，且第一读出控制信号为低电平状态时，跟随晶体管与尾流电源构成源极跟随器，对光电转换模块输出的光强对数信号进行放大并跟随输出至第二传输支路。When the second readout control signal is in a high level state and the first readout control signal is in a low level state, the follower transistor and the tail current power supply form a source follower to amplify the light intensity logarithmic signal output by the photoelectric conversion module and follow the output to the second transmission branch.
参照图3，第二选择晶体管SEL2的第一端还接尾流电源I0，尾流电源I0与EVS后端电路相邻设置。在第二读出控制信号READ2为高电平状态，且第一读出控制信号READ1为低电平状态时，第二选择晶体管SEL2连通，第一选择晶体管SEL1关断。跟随晶体管SF与尾流电源I0构成源极跟随器，对光电转换模块输出的光强对数信号进行放大并跟随输出至第二传输支路。Referring to Figure 3, the first end of the second selection transistor SEL2 is also connected to the tail current power supply I0, and the tail current power supply I0 is arranged adjacent to the EVS back-end circuit. When the second readout control signal READ2 is in a high level state and the first readout control signal READ1 is in a low level state, the second selection transistor SEL2 is connected and the first selection transistor SEL1 is turned off. The follower transistor SF and the tail current power supply I0 form a source follower, which amplifies the light intensity logarithmic signal output by the photoelectric conversion module and follows the output to the second transmission branch.
进一步地，第一选择晶体管SEL1的第一端也可以接尾流电源，该第一尾流电源可以接在ABUS线上面（图中未示出），在第一读出控制信号READ1为高电平状态，且第二读出控制信号READ2为低电平状态时，第一选择晶体管SEL1连通，第二选择晶体管SEL2关断。跟随晶体管SF同样可以与尾流电源构成源极跟随器，对光电转换模块输出的光强积分信号进行放大并跟随输出至第一传输支路。使得跟随晶体管SF管无论在哪个模式下，其都接入尾流电源使其正常工作，从而对光强积分信号或光强对数信号进行放大或跟随，以便有源积分型像素读出单元和事件型像素读出单元更精确地处理这些电压信号，得到准确的图像数据。Furthermore, the first end of the first selection transistor SEL1 can also be connected to a tail current power supply, and the first tail current power supply can be connected to the ABUS line (not shown in the figure). When the first readout control signal READ1 is in a high level state and the second readout control signal READ2 is in a low level state, the first selection transistor SEL1 is connected and the second selection transistor SEL2 is turned off. The follower transistor SF can also form a source follower with the tail current power supply to amplify the light intensity integral signal output by the photoelectric conversion module and follow the output to the first transmission branch. No matter which mode the follower transistor SF is in, it is connected to the tail current power supply to make it work normally, thereby amplifying or following the light intensity integral signal or the light intensity logarithmic signal, so that the active integral pixel readout unit and the event pixel readout unit can process these voltage signals more accurately and obtain accurate image data.
在本实施例中，当像素在EVS模式下，光电转换模块在FD产生的光强对数信号是对数电压，基本上没有驱动能力，需要通过源极跟随器对器进行放大和跟随，以使信号可以向后面电路传递，从而可以提高得到EVS图像数据的准确性和稳定性，进一步提高成像质量。In this embodiment, when the pixel is in the EVS mode, the logarithmic signal of the light intensity generated by the photoelectric conversion module in the FD is a logarithmic voltage, which basically has no driving capability and needs to be amplified and followed by a source follower so that the signal can be transmitted to the subsequent circuit, thereby improving the accuracy and stability of the EVS image data and further improving the imaging quality.
可选地，参照图5，基于以上任意一实施例，在本发明的另一实施例提供的图像传感器中，每个像素被配置有至少四个相邻的光电输出电路，各个光电输出电路的接入复位电路、第一传输支路及第二传输支路。Optionally, referring to Figure 5, based on any of the above embodiments, in an image sensor provided in another embodiment of the present invention, each pixel is configured with at least four adjacent photoelectric output circuits, and each photoelectric output circuit is connected to a reset circuit, a first transmission branch and a second transmission branch.
在本实施例中，为了增大像素中光电输出电路的有效感光面积以及降低像素成本，可以让多个光电输出电路共用选择跟随输出单元，选择跟随输出单元包括复位电路、源极跟随晶体管SF、第一传输支路和第二传输支路，至少四个光电输出电路及其共用的选择跟随输出单元则可以组成一个像素。In this embodiment, in order to increase the effective photosensitivity area of the photoelectric output circuit in the pixel and reduce the pixel cost, multiple photoelectric output circuits can share a selection follower output unit, and the selection follower output unit includes a reset circuit, a source follower transistor SF, a first transmission branch and a second transmission branch. At least four photoelectric output circuits and their shared selection follower output unit can form a pixel.
像素中配置的至少四个光电输出电路（每个光电输出电路分别包括光电二极管PD与传输晶体管TG）通过同一浮动扩散节点FD与选择跟随输出单元中，复位电路的复位晶体管RST以及源极跟随晶体管SF及其连接的第一传输支路和第二传输支路相连。At least four photoelectric output circuits configured in the pixel (each photoelectric output circuit includes a photodiode PD and a transfer transistor TG) are connected to the reset transistor RST of the reset circuit and the source follower transistor SF and the first transmission branch and the second transmission branch connected thereto in the selection follower output unit through the same floating diffusion node FD.
具体地，如图5所示，相邻的4个光电输出电路共用1个选择跟随输出单元：第一个光电输出电路包括光电二极管PD0、传输晶体管TG0，光电二极管PD0的阳极接地，光电二极管PD0的阴极接传输晶体管TG0的第一端，传输晶体管TG0的控制端接像素控制信号TX0，传输晶体管TG0的第二端接复位晶体管RST的第一端；第二个光电输出电路包括光电二极管PD1、传输晶体管TG1，光电二极管PD1的阳极接地，光电二极管PD1的阴极接传输晶体管TG1的第一端，传输晶体管TG1的控制端接像素控制信号TX1，传输晶体管TG1的第二端接复位晶体管RST的第一端；第三个光电输出电路包括光电二极管PD2、传输晶体管TG2，光电二极管PD2的阳极接地，光电二极管PD2的阴极接传输晶体管TG2的第一端，传输晶体管TG2的控制端接像素控制信号TX2，传输晶体管TG2的第二端接复位晶体管RST的第一端；第四个光电输出电路包括光电二极管PD3、传输晶体管TG3，光电二极管PD3的阳极接地，光电二极管PD3的阴极接传输晶体管TG3的第一端，传输晶体管TG3的控制端接像素控制信号TX3，传输晶体管TG3的第二端接复位晶体管RST的第一端；四个光电输出电路共用复位晶体管RST和源极跟随晶体管SF及其连接的第一传输支路和第二传输支路。Specifically, as shown in FIG5 , four adjacent photoelectric output circuits share one select-follow output unit: the first photoelectric output circuit includes a photodiode PD0 and a transfer transistor TG0, the anode of the photodiode PD0 is grounded, the cathode of the photodiode PD0 is connected to the first end of the transfer transistor TG0, the control end of the transfer transistor TG0 is connected to the pixel control signal TX0, and the second end of the transfer transistor TG0 is connected to the first end of the reset transistor RST; the second photoelectric output circuit includes a photodiode PD1 and a transfer transistor TG1, the anode of the photodiode PD1 is grounded, the cathode of the photodiode PD1 is connected to the first end of the transfer transistor TG1, the control end of the transfer transistor TG1 is connected to the pixel control signal TX1, and the second end of the transfer transistor TG1 is connected to the first end of the reset transistor RST; the third photoelectric output circuit includes a photodiode PD0 and a transfer transistor TG1, the anode of the photodiode PD1 is grounded, the cathode of the photodiode PD1 is connected to the first end of the transfer transistor TG1, the control end of the transfer transistor TG1 is connected to the pixel control signal TX1, and the second end of the transfer transistor TG1 is connected to the first end of the reset transistor RST; The output circuit includes a photodiode PD2 and a transmission transistor TG2, wherein the anode of the photodiode PD2 is grounded, the cathode of the photodiode PD2 is connected to the first end of the transmission transistor TG2, the control end of the transmission transistor TG2 is connected to the pixel control signal TX2, and the second end of the transmission transistor TG2 is connected to the first end of the reset transistor RST; the fourth photoelectric output circuit includes a photodiode PD3 and a transmission transistor TG3, wherein the anode of the photodiode PD3 is grounded, the cathode of the photodiode PD3 is connected to the first end of the transmission transistor TG3, the control end of the transmission transistor TG3 is connected to the pixel control signal TX3, and the second end of the transmission transistor TG3 is connected to the first end of the reset transistor RST; the four photoelectric output circuits share the reset transistor RST and the source follower transistor SF and the first transmission branch and the second transmission branch connected thereto.
需要说明的是，图像传感器的像素阵列可以包括多个呈阵列设置的像素。而每个像素可以包括多个相邻设置的光电输出电路，多个光电输出电路与一个选择跟随输出单元连接，如至少四个光电输出电路与一个选择跟随输出单元中的复位电路组成光电转换模块输出信号。It should be noted that the pixel array of the image sensor may include a plurality of pixels arranged in an array. Each pixel may include a plurality of adjacently arranged photoelectric output circuits, and the plurality of photoelectric output circuits are connected to a select-follow output unit, such as at least four photoelectric output circuits and a reset circuit in a select-follow output unit to form a photoelectric conversion module output signal.
光电输出电路基于光电效应对入射光进行采样，基于选择跟随输出单元对应能够可控地得到光强积分信号、光强对数信号或者复位信号等电压信号。通过处理模块中有源积分型像素读出单元和事件型像素读出单元的处理，可以基于光电输出电路对应的电压信号，得到光电输出电路对应的APS图像数据或者EVS图像数据。即控制光电输出电路和复位电路可以生成光电输出电路对应的光强积分信号或光强对数信号，对应可以通过有源积分型像素读出单元或事件型像素读出单元生成电输出电路对应的不同类型的图像数据。The photoelectric output circuit samples the incident light based on the photoelectric effect, and can controllably obtain voltage signals such as light intensity integral signal, light intensity logarithmic signal or reset signal based on the selection of the follower output unit. Through the processing of the active integral pixel readout unit and the event pixel readout unit in the processing module, the APS image data or EVS image data corresponding to the photoelectric output circuit can be obtained based on the voltage signal corresponding to the photoelectric output circuit. That is, controlling the photoelectric output circuit and the reset circuit can generate a light intensity integral signal or a light intensity logarithmic signal corresponding to the photoelectric output circuit, and correspondingly, different types of image data corresponding to the electrical output circuit can be generated through the active integral pixel readout unit or the event pixel readout unit.
基于光电输出电路可以进一步划分出子像素，通过对光电输出电路的可控性采样处理以及其对应选择跟随输出单元，可以输出光电输出电路对应的光强积分信号或光强对数信号，因此，一个光电输出电路及其对应的选择跟随输出单元可以构成一个子像素，控制像素的传感模式还可以包括控制子像素的传感模式，也即控制像素电路中光电转换电路对应模式，进而以子像素级输出APS图像数据，从而可以进一步提高图像的分辨率。Sub-pixels can be further divided based on the photoelectric output circuit. Through controllable sampling processing of the photoelectric output circuit and its corresponding selection-follow output unit, a light intensity integral signal or a light intensity logarithmic signal corresponding to the photoelectric output circuit can be output. Therefore, a photoelectric output circuit and its corresponding selection-follow output unit can constitute a sub-pixel. The sensing mode of controlling the pixel can also include the sensing mode of controlling the sub-pixel, that is, the corresponding mode of the photoelectric conversion circuit in the control pixel circuit, and then the APS image data is output at the sub-pixel level, so that the resolution of the image can be further improved.
可选地，参照图7，多个光电输出电路的感光面上设置有不同颜色的滤光片；Optionally, referring to FIG. 7 , filters of different colors are provided on the photosensitive surfaces of the plurality of photoelectric output circuits;
在第一传输支路与光电转换模块基于控制信号接通时，各光电输出电路分别基于通过其对应滤光片的入射光产生积分电荷，各光电输出电路分别在浮动扩散节点FD形成对应的光强积分信号；When the first transmission branch and the photoelectric conversion module are connected based on the control signal, each photoelectric output circuit generates an integrated charge based on the incident light passing through its corresponding filter, and each photoelectric output circuit forms a corresponding light intensity integrated signal at the floating diffusion node FD;
在第二传输支路与光电转换模块基于控制信号接通时，各光电输出电路对入射光进行光电感应后并联输出的光电流与基准电压在浮动扩散节点FD形成光强对数信号。When the second transmission branch and the photoelectric conversion module are connected based on the control signal, the photocurrent and the reference voltage output in parallel after each photoelectric output circuit performs photoelectric sensing on the incident light form a light intensity logarithmic signal at the floating diffusion node FD.
在本实施例中，每个光电输出电路的感光面上，即基于光电输出电路的光电二极管PD的表面，对应设置有微透镜、滤光片，可以是一对一或者一对多设置，微透镜的作用是将入射光汇聚到光电二极管PD上，而滤光片是一种表现颜色的光学滤光片，滤光片可以安装在微透镜与光电二极管PD之间，对入射光进行滤光处理，使得后端的光电输出电路对某个颜色或者波段的光信号进行光电转换。In this embodiment, a microlens and a filter are correspondingly arranged on the photosensitive surface of each photoelectric output circuit, that is, the surface of the photodiode PD based on the photoelectric output circuit, which can be a one-to-one or a one-to-many arrangement. The function of the microlens is to converge the incident light onto the photodiode PD, and the filter is an optical filter that expresses color. The filter can be installed between the microlens and the photodiode PD to filter the incident light, so that the rear-end photoelectric output circuit performs photoelectric conversion on the light signal of a certain color or band.
在一些实施例中，每个像素被配置有至少四个光电输出电路，对应配置了不同颜色的滤光片，如RGB-IR、RGBW等，也可以是多个像素的光电输出电路之间，对应配置了不同颜色的滤光片，如Quard Bayer滤光片阵列。使得像素组成的像素阵列，可以基于通过不同颜色滤光片对不同波段的光信号进行采样。In some embodiments, each pixel is configured with at least four photoelectric output circuits, corresponding to which filters of different colors are configured, such as RGB-IR, RGBW, etc., or between the photoelectric output circuits of multiple pixels, filters of different colors are correspondingly configured, such as a Quard Bayer filter array. This allows the pixel array composed of pixels to sample light signals of different wavelengths through filters of different colors.
在一些实施例中，若像素在APS模式下时，第一传输支路与光电转换模块接通，像素配置的至少四个光电输出电路可以分别进入曝光积分状态。而光电输出电路感光面上可以设置颜色不同的滤波片，其需要对不同波段的光波进行采样，而这些光电输出电路共用选择跟随输出单元，即共用同一个第一传输支路，从而可以获得不同颜色分量的APS图像数据，以便后续合成彩色APS图像。因此在APS模式下，像素被配置的多个光电输出电路需要分别进入曝光状态，然后再分别进入读出状态对应输出光强积分信号。这样各光电输出电路的光电二极管可以分别基于通过其对应滤光片的入射光产生积分电荷，从而分别对各光电输出电路的积分电荷进行读出，在浮动扩散节点FD形成滤光片对应颜色分量的光强积分信号，结合该光电输出电路的复位信号，可以得到该滤光片对应颜色通道下的APS图像数据，这样图像传感器可以生成图像质量更高的彩色APS图像。In some embodiments, if the pixel is in the APS mode, the first transmission branch is connected to the photoelectric conversion module, and at least four photoelectric output circuits configured in the pixel can enter the exposure integration state respectively. Filters of different colors can be set on the photosensitive surface of the photoelectric output circuit, which needs to sample light waves of different bands, and these photoelectric output circuits share the selection follower output unit, that is, share the same first transmission branch, so that APS image data of different color components can be obtained for subsequent synthesis of color APS images. Therefore, in the APS mode, the multiple photoelectric output circuits configured by the pixel need to enter the exposure state respectively, and then enter the readout state respectively to output the light intensity integral signal. In this way, the photodiode of each photoelectric output circuit can generate an integral charge based on the incident light passing through its corresponding filter, so as to read out the integral charge of each photoelectric output circuit respectively, and form the light intensity integral signal of the color component corresponding to the filter at the floating diffusion node FD. Combined with the reset signal of the photoelectric output circuit, the APS image data under the color channel corresponding to the filter can be obtained, so that the image sensor can generate a color APS image with higher image quality.
在一些实施例中，若像素在EVS模式下时，第二传输支路与光电转换模块接通，而像素的各光电输出电路的光电二极管PD处于并联状态，可同步进行光电转换并读出，等效于一个大的光电二极管PD进行EVS成像。需要说明的是，在EVS模式下，一般不需要关注颜色或者波段的区分，只关注对应像素对应入射光的光强变化，像素的各光电输出电路可以同时对入射光进行光电感应，进行光电反应后并联输出光电流，结合与复位电路提供的基准电压输出光强对数信号，当并联输出光电流以形成光强对数信号时，可以提高事件传感帧率。In some embodiments, if the pixel is in EVS mode, the second transmission branch is connected to the photoelectric conversion module, and the photodiodes PD of each photoelectric output circuit of the pixel are in parallel, and photoelectric conversion and readout can be performed synchronously, which is equivalent to a large photodiode PD for EVS imaging. It should be noted that in EVS mode, it is generally not necessary to pay attention to the distinction between colors or bands, but only to the intensity change of the incident light corresponding to the corresponding pixel. Each photoelectric output circuit of the pixel can simultaneously perform photoelectric sensing on the incident light, output photocurrent in parallel after photoelectric reaction, and output the light intensity logarithmic signal in combination with the reference voltage provided by the reset circuit. When the photocurrent is output in parallel to form a light intensity logarithmic signal, the event sensing frame rate can be improved.
可选地，图像传感器包括多个阵列设置的像素，每个像素对应的多个光电输出电路之间呈阵列排布或者折线排布。Optionally, the image sensor includes a plurality of pixels arranged in an array, and a plurality of photoelectric output circuits corresponding to each pixel are arranged in an array or in a zigzag arrangement.
在本实施例中，图像传感器包括像素阵列，像素阵列包括多个呈阵列排布设置的像素组成，而各像素中包括的光电输出电路也可以是呈阵列排布设置，或者各像素中包括的光电输出电路也可以是呈折线排布设置。In this embodiment, the image sensor includes a pixel array, which includes a plurality of pixels arranged in an array, and the photoelectric output circuit included in each pixel may also be arranged in an array, or the photoelectric output circuit included in each pixel may also be arranged in a zigzag line.
参见图7－图18，像素包括多个相邻的光电输出电路，而多个相邻的光电输出电路可以呈阵列排布，参见图19－图26，像素中多个相邻的光电输出电路可以呈折线排布。而参见图7－图26可知，像素之间是可以呈阵列排布设置的。Referring to Figures 7 to 18, a pixel includes a plurality of adjacent photoelectric output circuits, and the plurality of adjacent photoelectric output circuits can be arranged in an array, and referring to Figures 19 to 26, the plurality of adjacent photoelectric output circuits in a pixel can be arranged in a zigzag line. Referring to Figures 7 to 26, it can be seen that the pixels can be arranged in an array.
基于像素中各光电输出电路产生的电荷或者电流，能够处理得到对应的图像数据，该图像数据也组成了其所在像素采集的图像数据，而多个阵列设置的像素对应产生的图像数据能够组成图像传感器采集的图像。Based on the charge or current generated by each photoelectric output circuit in the pixel, the corresponding image data can be processed and obtained, and the image data also constitutes the image data collected by the pixel where it is located, and the image data corresponding to the pixels arranged in multiple arrays can constitute the image collected by the image sensor.
每个像素中光电输出电路的个数和光电输出电路的分布方式，可以根据需求设定，光电输出电路的个数还可以是8个、16个等其他数目，各个光电输出电路可以是按照阵列或者折线等其他形状排布，可根据实际需求灵活调整，在此不作限定。但是，结合图像传感器工作模式的设定，折线排布可以减少像素的损失，以提高图像质量。具体效果结合图像传感器的控制方法进行说明。The number of photoelectric output circuits in each pixel and the distribution of the photoelectric output circuits can be set according to demand. The number of photoelectric output circuits can also be 8, 16 or other numbers. Each photoelectric output circuit can be arranged in an array or other shapes such as a fold line. It can be flexibly adjusted according to actual needs and is not limited here. However, combined with the setting of the image sensor working mode, the fold line arrangement can reduce the loss of pixels to improve the image quality. The specific effect is explained in conjunction with the control method of the image sensor.
可选地，基于上述任一实施例，在本发明的另一实施例中，图像传感器包括：多个堆叠设置的晶圆，光电输出电路的光电二极管设置在表层晶圆上，光电输出电路的传输晶体管、复位电路或处理模块设置在任意一个晶圆上。Optionally, based on any of the above embodiments, in another embodiment of the present invention, the image sensor includes: a plurality of stacked wafers, the photodiode of the photoelectric output circuit is arranged on the surface wafer, and the transfer transistor, reset circuit or processing module of the photoelectric output circuit is arranged on any one of the wafers.
在本实施例中，图像传感器中各个部件可以堆叠设置，其部件设置在多个堆叠设置的晶圆上。光电输出电路需要基于光电效应输出电压信号，因而将光电输出电路，尤其是光电输出电路中的光电二极管PD设置在表层晶圆上，以使光电输出电路的光电二极管PD可以更好地接收入射光。而其他部件可以根据需求设置在包括表层晶圆在内的任意一个晶圆上。In this embodiment, the components of the image sensor can be stacked and arranged on multiple stacked wafers. The photoelectric output circuit needs to output a voltage signal based on the photoelectric effect, so the photoelectric output circuit, especially the photodiode PD in the photoelectric output circuit, is arranged on the surface wafer so that the photodiode PD of the photoelectric output circuit can better receive the incident light. Other components can be arranged on any wafer including the surface wafer according to requirements.
此外，如图2所示，图像传感器还包括电源管理电路及偏置电路，电源管理电路为整颗芯片的像素以及电路提供不同的电源电压（如复位电源VPIX及复位电源VSF），偏置电路为整颗芯片提供参考电压（或者偏置电压）和参考电流。In addition, as shown in FIG2 , the image sensor also includes a power management circuit and a bias circuit. The power management circuit provides different power supply voltages (such as reset power supply VPIX and reset power supply VSF) for the pixels and circuits of the entire chip, and the bias circuit provides a reference voltage (or bias voltage) and a reference current for the entire chip.
在一些具体实施例中，图像传感器具有两层堆叠设置的晶圆。光电输出电路设置在第一层晶圆上，而第一层晶圆可以不只有光电输出电路，还可以有电源管理电路、模数转换器ADC的行列控制、EVS后端电路阵列和数字信号处理部分等部件或电路。In some specific embodiments, the image sensor has two stacked wafers. The photoelectric output circuit is arranged on the first wafer, and the first wafer may not only have the photoelectric output circuit, but also have components or circuits such as a power management circuit, a row and column control of an analog-to-digital converter ADC, an EVS back-end circuit array, and a digital signal processing part.
在一些具体实施例中，图像传感器具有三层堆叠设置的晶圆。光电输出电路设置于第一层晶圆，EVS后端电路部分设置于第二层晶圆，剩下的所有电路设置于第三层晶圆。或者，光电输出电路和选择跟随输出单元设置于第一层晶圆，EVS后端电路部分设置于第二层晶圆，剩下的所有电路设置于第三层晶圆。In some specific embodiments, the image sensor has three layers of stacked wafers. The photoelectric output circuit is arranged on the first layer of wafer, the EVS back-end circuit part is arranged on the second layer of wafer, and all the remaining circuits are arranged on the third layer of wafer. Alternatively, the photoelectric output circuit and the select follower output unit are arranged on the first layer of wafer, the EVS back-end circuit part is arranged on the second layer of wafer, and all the remaining circuits are arranged on the third layer of wafer.
可选地，基于上述任意一个实施例，在另一实施例中还提供一种图像传感器的控制方法。Optionally, based on any one of the above embodiments, a method for controlling an image sensor is further provided in another embodiment.
具体地，本实施例将从图像传感器的控制器的角度进行描述，该图像传感器的控制器具体可以集成在图像传感器或者光电设备中，即本发明实施例图像传感器的控制方法可以由图像传感器或者光电设备执行。Specifically, this embodiment will be described from the perspective of the image sensor controller, which can be integrated into the image sensor or optoelectronic device, that is, the control method of the image sensor in the embodiment of the present invention can be executed by the image sensor or optoelectronic device.
以下结合附图分别进行详细说明，本实施例中以执行主体是图像传感器为例。需说明的是，以下实施例的描述顺序不作为对实施例优选顺序的限定。虽然在流程图中示出了逻辑顺序，但是在某些情况下，可以以不同于附图所示的顺序执行所示出或描述的步骤。The following is a detailed description in conjunction with the accompanying drawings. In this embodiment, the execution subject is an image sensor as an example. It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments. Although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in an order different from that shown in the accompanying drawings.
请参考图6，该图像传感器的控制方法的具体流程可以如下步骤S10~步骤S40，其中：Please refer to FIG. 6 , the specific process of the control method of the image sensor may be as follows: Step S10 to Step S40, wherein:
步骤S10，获取图像传感器配置信息；Step S10, obtaining image sensor configuration information;
步骤S20，根据图像传感器配置信息确定图像传感器中各个像素的目标传感模式；Step S20, determining a target sensing mode of each pixel in the image sensor according to the image sensor configuration information;
步骤S30，若目标传感模式为有源积分型像素模式，控制光电转换模块连通第一传输支路，以使光电转换模块通过第一传输支路输出光强积分信号至有源积分型像素读出单元，由光强积分信号处理得到像素对应的有源图像数据；Step S30, if the target sensing mode is the active integral pixel mode, control the photoelectric conversion module to connect to the first transmission branch, so that the photoelectric conversion module outputs the light intensity integral signal to the active integral pixel readout unit through the first transmission branch, and obtains the active image data corresponding to the pixel by processing the light intensity integral signal;
步骤S40，若目标传感模式为事件型像素模式，控制光电转换模块连通第二传输支路，以使光电转换模块通过第二传输支路输出光强对数信号至事件型像素读出单元，由光强对数信号处理得到像素对应的事件图像数据。Step S40, if the target sensing mode is the event-type pixel mode, control the photoelectric conversion module to connect to the second transmission branch, so that the photoelectric conversion module outputs a light intensity logarithmic signal to the event-type pixel readout unit through the second transmission branch, and obtains event image data corresponding to the pixel by processing the light intensity logarithmic signal.
在本实施例中，通过不同的成像需求，可以触发图像传感器的图像传感器配置信息，通过图像传感器配置信息可以选择图像传感器中的部分或全部像素工作于APS模式或EVS模式，从而调整图像工作器的工作模式。In this embodiment, the image sensor configuration information of the image sensor can be triggered by different imaging requirements. Through the image sensor configuration information, some or all pixels in the image sensor can be selected to operate in APS mode or EVS mode, thereby adjusting the working mode of the image worker.
对接收到的图像传感器配置信息进行分析，图像传感器配置信息可以包括图像传感器待执行的工作模式，或者可以包括需要工作于APS模式下像素的相关信息，以及需要工作于EVS模式下的像素，相关信息可以包括像素的数量、位置等信息，通过相关信息可以确定出图像传感器中部分或全部的像素。The received image sensor configuration information is analyzed. The image sensor configuration information may include the working mode to be executed by the image sensor, or may include relevant information of pixels that need to work in the APS mode, and pixels that need to work in the EVS mode. The relevant information may include the number and position of pixels, etc., and some or all of the pixels in the image sensor can be determined through the relevant information.
进而可以根据图像传感器配置信息确定出图像传感器的像素阵列中各个像素当前需要工作的传感模式，即像素的目标传感模式，进而可以确定像素对应的目标传感模式，并通过控制信号控制像素进入目标传感模式，控制信号可以包括上述的像素控制信号TX、复位控制信号RS、第一读出控制信号READ1和第二读出控制信号READ2等。Furthermore, the sensing mode in which each pixel in the pixel array of the image sensor currently needs to work, that is, the target sensing mode of the pixel, can be determined based on the image sensor configuration information, and then the target sensing mode corresponding to the pixel can be determined, and the pixel can be controlled to enter the target sensing mode through a control signal. The control signal may include the above-mentioned pixel control signal TX, reset control signal RS, first read control signal READ1 and second read control signal READ2, etc.
若像素的目标传感模式为APS模式时，则控制像素的光电转换模块连通像素的第一传输支路，并向光电转换模块的各个光电输出电路发送APS模式对应的控制信号，以使各个光电输出电路形成对应的光强积分信号，从而光电转换模块可以通过第一传输支路将光强积分信号发送至有源积分像素读出单元，从而得到该像素对应的APS图像数据，像素对应的APS图像数据也是像素所在APS像素单元的APS图像数据。其中，APS像素单元包括一个或多个呈阵列排布的处于APS模式下的像素。If the target sensing mode of the pixel is the APS mode, the photoelectric conversion module of the pixel is controlled to connect to the first transmission branch of the pixel, and the control signal corresponding to the APS mode is sent to each photoelectric output circuit of the photoelectric conversion module, so that each photoelectric output circuit forms a corresponding light intensity integration signal, so that the photoelectric conversion module can send the light intensity integration signal to the active integration pixel readout unit through the first transmission branch, thereby obtaining the APS image data corresponding to the pixel, and the APS image data corresponding to the pixel is also the APS image data of the APS pixel unit where the pixel is located. Among them, the APS pixel unit includes one or more pixels arranged in an array in the APS mode.
若像素的目标传感模式为EVS模式时，则控制像素的光电转换模块连通像素的第二传输支路，并向光电转换模块发送EVS模式对应的控制信号，以使光电转换模块的各光电输出电路形成光强对数信号，可以通过第二传输支路将光强对数信号发送至事件型像素读出单元。从而得到像素对应的EVS图像数据，像素对应的EVS图像数据也是像素所在事件像素单元的EVS图像数据，其中，事件像素单元包括一个或多个呈阵列排布的处于EVS模式下的像素。由于像素在EVS模式下，其对应的光电输出电路一般是并联，各光电输出电路中的光电二极管PD等效一个大的光电二极管，使得该像素为整体的EVS像素，得到该EVS像素对应的EVS图像数据。If the target sensing mode of the pixel is the EVS mode, the photoelectric conversion module of the pixel is controlled to connect to the second transmission branch of the pixel, and a control signal corresponding to the EVS mode is sent to the photoelectric conversion module, so that each photoelectric output circuit of the photoelectric conversion module forms a light intensity logarithmic signal, and the light intensity logarithmic signal can be sent to the event-type pixel readout unit through the second transmission branch. Thus, the EVS image data corresponding to the pixel is obtained, and the EVS image data corresponding to the pixel is also the EVS image data of the event pixel unit where the pixel is located, wherein the event pixel unit includes one or more pixels arranged in an array in the EVS mode. Since the pixel is in the EVS mode, its corresponding photoelectric output circuit is generally connected in parallel, and the photodiode PD in each photoelectric output circuit is equivalent to a large photodiode, so that the pixel is an overall EVS pixel, and the EVS image data corresponding to the EVS pixel is obtained.
进一步地，上述图像传感器的控制方法中，步骤S20，包括：Furthermore, in the above-mentioned image sensor control method, step S20 includes:
根据图像传感器配置信息确定图像传感器的工作模式；determining an operating mode of the image sensor according to the image sensor configuration information;
若图像传感器的工作模式为有源型图像模式，则确定图像传感器中全部或部分像素的目标传感模式为有源积分型像素模式，目标传感模式不是有源积分型像素模式的像素不工作；If the working mode of the image sensor is the active image mode, determining that the target sensing mode of all or part of the pixels in the image sensor is the active integral pixel mode, and the pixels whose target sensing mode is not the active integral pixel mode do not work;
若图像传感器的工作模式为事件型图像模式，则确定图像传感器中全部或部分像素的目标传感模式为事件型像素模式，目标传感模式不是事件型像素模式的像素不工作；If the working mode of the image sensor is the event-based image mode, determining that the target sensing mode of all or part of the pixels in the image sensor is the event-based pixel mode, and pixels whose target sensing mode is not the event-based pixel mode do not work;
若图像传感器的工作模式为混合图像模式，则确定图像传感器中的第一像素的目标传感模式为有源积分型像素模式，第二像素的目标传感模式为事件型像素模式。If the working mode of the image sensor is the mixed image mode, it is determined that the target sensing mode of the first pixel in the image sensor is the active integration pixel mode, and the target sensing mode of the second pixel is the event pixel mode.
在本实施例中，根据图像传感器配置信息，可以调整图像传感器的工作模式。图像传感器的工作模式通过调整各像素的传感模式实现，各像素不同传感模式的排列组合，可以实现不同的工作模式。为处于需求的工作模式，调整图像传感器中各像素至目标传感模式，从而满足不同的图像采集需求。In this embodiment, the working mode of the image sensor can be adjusted according to the configuration information of the image sensor. The working mode of the image sensor is realized by adjusting the sensing mode of each pixel. The arrangement and combination of different sensing modes of each pixel can realize different working modes. To be in the required working mode, each pixel in the image sensor is adjusted to the target sensing mode, thereby meeting different image acquisition requirements.
图像传感器的工作模式可以包括以下三种：The image sensor can work in three modes:
1、有源型图像模式（可记为APS图像模式）：像素阵列中的所有像素均工作在APS模式，为全尺寸APS图像模式，或者像素阵列中的部分像素工作在EVS模式而其他像素不工作，为部分尺寸APS图像模式。1. Active image mode (can be recorded as APS image mode): All pixels in the pixel array work in APS mode, which is full-size APS image mode, or some pixels in the pixel array work in EVS mode while other pixels do not work, which is partial-size APS image mode.
2、事件型图像模式（可记为EVS图像模式）：像素阵列中的所有像素均工作在EVS模式，为全尺寸EVS图像模式，或者像素阵列中的部分像素工作在EVS模式而其他像素不工作，为部分尺寸EVS图像模式。2. Event-type image mode (can be recorded as EVS image mode): All pixels in the pixel array work in EVS mode, which is the full-size EVS image mode, or some pixels in the pixel array work in EVS mode while other pixels do not work, which is the partial-size EVS image mode.
3、混合图像模式（可记为HVS图像模式）：像素阵列中的部分像素工作在APS模式，而像素阵列中的其他像素工作在EVS模式。3. Hybrid image mode (which can be recorded as HVS image mode): some pixels in the pixel array work in APS mode, while other pixels in the pixel array work in EVS mode.
若响应于图像传感器配置信息，确定图像传感器的工作模式为APS图像模式，则根据第一预设规则，从图像传感器中选择全部或部分的像素，将这些像素的目标传感模式设置为APS模式，这些目标传感模式设置为APS模式的像素可以被称为有源像素（以下称为APS像素），而未被选择到的像素的目标传感模式不是APS模式，目标传感模式不是APS模式的非有源像素的其他像素不工作；If, in response to the image sensor configuration information, it is determined that the working mode of the image sensor is the APS image mode, then according to a first preset rule, all or part of the pixels are selected from the image sensor, and the target sensing mode of these pixels is set to the APS mode. The pixels whose target sensing mode is set to the APS mode may be referred to as active pixels (hereinafter referred to as APS pixels), and the target sensing mode of the pixels that are not selected is not the APS mode, and other pixels that are not active pixels whose target sensing mode is not the APS mode do not work;
若响应于图像传感器配置信息，确定图像传感器的工作模式为EVS图像模式，则根据第二预设规则，从图像传感器中选择全部或部分的像素，将这些像素的目标传感模式设置为EVS模式，这些目标传感模式设置为EVS模式的像素可以被称为事件像素（以下称为EVS像素），而未被选择到的像素的目标传感模式不是EVS模式，目标传感模式不是EVS模式的非事件像素的其他像素不工作；If, in response to the image sensor configuration information, it is determined that the working mode of the image sensor is the EVS image mode, then according to a second preset rule, all or part of the pixels are selected from the image sensor, and the target sensing mode of these pixels is set to the EVS mode. The pixels whose target sensing mode is set to the EVS mode may be referred to as event pixels (hereinafter referred to as EVS pixels), and the target sensing mode of the pixels that are not selected is not the EVS mode, and other pixels that are not event pixels whose target sensing mode is not the EVS mode do not work;
若响应于图像传感器配置信息，确定图像传感器的工作模式为HVS图像模式，则根据第三预设规则从图像传感器中选择部分像素作为第一像素，从图像传感器的非第一像素中选择全部或部分像素作为第二像素，进而将第一像素的目标传感模式设置为APS模式，第二像素的目标传感模式设置为EVS模式，使得第一像素为APS像素，第二像素为EVS像素，非APS像素和非EVS像素的其他像素不工作。If, in response to the image sensor configuration information, it is determined that the operating mode of the image sensor is the HVS image mode, then according to a third preset rule, some pixels are selected from the image sensor as first pixels, and all or some pixels are selected from non-first pixels of the image sensor as second pixels, and then the target sensing mode of the first pixel is set to the APS mode, and the target sensing mode of the second pixel is set to the EVS mode, so that the first pixel is an APS pixel, the second pixel is an EVS pixel, and other pixels that are not APS pixels and non-EVS pixels do not work.
其中，第一预设规则、第二预设规则和第三预设规则可以预先设置，也可以根据用户实时触发的图像传感器配置信息确定。The first preset rule, the second preset rule and the third preset rule may be preset, or may be determined according to image sensor configuration information triggered in real time by a user.
需要说明的是，图像传感器的工作模式取决于应用需求，尤其是在部分尺寸EVS图像模式或者HVS图像模式下，工作在EVS模式下的像素的比例和位置也区别于应用需求。该需求可以综合APS图像和EVS图像的精度、能耗进行取舍，在部分尺寸EVS图像模式下，通过事件信号处理器的处理，可以弥补EVS图像缺失点并获得全比例EVS图像，而在HVS图像模式下，通过图像信号处理器的处理，可以弥补APS图像缺失点并可以获得全画幅APS图像，通过事件信号处理器的处理，可以弥补EVS图像缺失点并可以获得全比例EVS图像，当图像传感器工作在APS图像模式、EVS图像模式或HVS图像模式下时，行控制逻辑可以对整个像素阵列提供不同的控制信号、偏置电压、参考基准电压和电源。It should be noted that the working mode of the image sensor depends on the application requirements, especially in the partial-size EVS image mode or HVS image mode, the proportion and position of the pixels working in the EVS mode are also different from the application requirements. This requirement can be traded off by comprehensively considering the accuracy and energy consumption of the APS image and the EVS image. In the partial-size EVS image mode, the missing points of the EVS image can be compensated and the full-scale EVS image can be obtained through the processing of the event signal processor, while in the HVS image mode, the missing points of the APS image can be compensated and the full-frame APS image can be obtained through the processing of the image signal processor. The missing points of the EVS image can be compensated and the full-scale EVS image can be obtained through the processing of the event signal processor. When the image sensor works in the APS image mode, the EVS image mode or the HVS image mode, the row control logic can provide different control signals, bias voltages, reference base voltages and power supplies to the entire pixel array.
同时，在部分尺寸EVS图像模式或者HVS图像模式下，工作在EVS模式下的多个像素可以是部分区块集中设置，可以是按行或者列分散设置，可以是按对角线等斜线或者折线段分散设置，还可以是离散多点式阵列设置，可根据实际需求，基于控制器的控制灵活设计，在此不作限定。At the same time, in the partial-size EVS image mode or the HVS image mode, multiple pixels working in the EVS mode can be concentrated in partial blocks, can be dispersed in rows or columns, can be dispersed in diagonal lines or broken line segments, or can be set in a discrete multi-point array. It can be flexibly designed based on the control of the controller according to actual needs, and is not limited here.
这样在定义了图像传感器的三种工作模式的基础上，可以根据需求选择图像传感器的工作模式并可以对每个工作模式下的具体显示内容进行灵活调节，使得图像传感器的工作模式灵活可调，使其成为兼容多种工作模式的高质量复合型图像传感器。In this way, based on the definition of the three working modes of the image sensor, the working mode of the image sensor can be selected according to needs and the specific display content under each working mode can be flexibly adjusted, making the working mode of the image sensor flexible and adjustable, making it a high-quality composite image sensor compatible with multiple working modes.
进一步地，上述方法还包括：Furthermore, the above method also includes:
若工作模式为混合图像模式或事件型图像模式，则基于事件像素单元中各事件像素对应的事件图像数据，生成目标事件图像数据；If the working mode is the mixed image mode or the event image mode, target event image data is generated based on the event image data corresponding to each event pixel in the event pixel unit;
其中，事件像素单元包括至少一个事件像素，事件像素是图像传感器中目标传感模式为事件型像素模式的像素，事件像素单元的感知颜色信息与目标感知颜色信息相同，感知颜色信息基于事件像素单元中各事件像素对应的第一滤光片颜色确定。Among them, the event pixel unit includes at least one event pixel, the event pixel is a pixel in the image sensor whose target sensing mode is an event-type pixel mode, the perceived color information of the event pixel unit is the same as the target perceived color information, and the perceived color information is determined based on the first filter color corresponding to each event pixel in the event pixel unit.
在本实施例中，在本实施例中，若图像传感器的工作模式为HVS图像模式或EVS图像模式，则图像传感器中则会包括处于EVS模式的EVS像素。这些EVS像素可以集中或分散设置于图像传感器中。由于在图像传感器中，每个像素的光电数据电路的感光面上设置有不同颜色的滤光片，那么EVS像素中的多个光电输出电路的感光面上也可以设置有不同颜色的滤光片，但若单个EVS像素对应的多个滤光片，可能并非包含了完整所需的多个目标颜色（例如红、绿、蓝RGB三原色）的滤光片，那么单个EVS像素不能有效地对各种所需要的目标颜色分别进行感知采样，即EVS像素可能不具有满足需求的感知不同颜色光信号变化的事件感知能力，因而需要结合多个能感知不同颜色的事件像素，组成事件像素单元（以下称为EVS像素单元），基于EVS像素单元中各个EVS像素能有效对各种所需的目标颜色（如红绿蓝）的光信号分别进行感知采样。In this embodiment, in this embodiment, if the working mode of the image sensor is the HVS image mode or the EVS image mode, the image sensor will include EVS pixels in the EVS mode. These EVS pixels can be arranged in the image sensor in a concentrated or dispersed manner. Since filters of different colors are arranged on the photosensitive surface of the photoelectric data circuit of each pixel in the image sensor, filters of different colors can also be arranged on the photosensitive surface of multiple photoelectric output circuits in the EVS pixel, but if the multiple filters corresponding to a single EVS pixel may not include filters of multiple target colors (such as red, green, and blue RGB primary colors) required in full, then a single EVS pixel cannot effectively sense and sample various required target colors separately, that is, the EVS pixel may not have the event sensing capability of sensing changes in light signals of different colors that meets the requirements, and therefore, it is necessary to combine multiple event pixels that can sense different colors to form an event pixel unit (hereinafter referred to as an EVS pixel unit), based on the fact that each EVS pixel in the EVS pixel unit can effectively sense and sample light signals of various required target colors (such as red, green, and blue).
其中，基于EVS像素单元的颜色感知信息是基于EVS像素单元所包括的各EVS像素中的若干光电输出电路的第一滤光片颜色确定的。单个像素中可能会包括若干光电输出电路，对应也会包括若干颜色的滤光片，因此，像素的各光电输出电路基于其滤光片，能够感知若干颜色，根据各光电输出电路对应的滤光片颜色，可以确定像素的感知颜色信息，感知颜色信息也包括感知颜色类型和/或感知颜色架构，感知颜色类型是指像素可以感知到的一种或多种不同波段光信号的颜色类型，感知颜色架构是指像素中不同感知颜色对应光电输出电路之间的排布方式和/或不同感知颜色对应光电输出电路的数量。因此，基于EVS像素单元中各EVS像素的感知颜色信息，可以组合确定EVS像素单元的感知颜色信息。Among them, the color perception information based on the EVS pixel unit is determined based on the first filter color of several photoelectric output circuits in each EVS pixel included in the EVS pixel unit. A single pixel may include several photoelectric output circuits, and correspondingly also includes filters of several colors. Therefore, each photoelectric output circuit of the pixel can perceive several colors based on its filter. According to the filter color corresponding to each photoelectric output circuit, the perceived color information of the pixel can be determined. The perceived color information also includes the perceived color type and/or the perceived color architecture. The perceived color type refers to the color type of one or more different bands of light signals that the pixel can perceive. The perceived color architecture refers to the arrangement between the photoelectric output circuits corresponding to different perceived colors in the pixel and/or the number of photoelectric output circuits corresponding to different perceived colors. Therefore, based on the perceived color information of each EVS pixel in the EVS pixel unit, the perceived color information of the EVS pixel unit can be determined in combination.
而在本实施例中，组成EVS像素单元需要满足的条件是，其感知颜色信息与预设需求对应的目标颜色感知信息相同。In the present embodiment, the condition that needs to be met for constituting the EVS pixel unit is that its perceived color information is the same as the target color perception information corresponding to the preset requirement.
目标感知颜色信息是基于预设的感知需求设定的，目标颜色信息可以包括目标感知颜色类型和/或目标感知颜色架构。目标感知颜色类型是指EVS像素单元需要感知到的一种或多种不同波段光信号的颜色类型，例如红绿蓝或红外光等，可以理解的是，为了达成该感知目标颜色的效果，至少需要为图像传感器上多个像素或多个光信号输出电路设置不同目标感知颜色类型对应的滤光片。而目标感知颜色架构是指EVS像素单元中需要满足的不同目标感知颜色对应光电输出电路之间的排布方式和/或不同目标感知颜色对应EVS光电输出电路的数量。The target perceived color information is set based on preset perception requirements, and the target color information may include a target perceived color type and/or a target perceived color architecture. The target perceived color type refers to the color type of one or more different bands of light signals that the EVS pixel unit needs to perceive, such as red, green, blue or infrared light, etc. It is understandable that in order to achieve the effect of perceiving the target color, at least filters corresponding to different target perceived color types need to be set for multiple pixels or multiple light signal output circuits on the image sensor. The target perceived color architecture refers to the arrangement of photoelectric output circuits corresponding to different target perceived colors that need to be met in the EVS pixel unit and/or the number of EVS photoelectric output circuits corresponding to different target perceived colors.
在一些实施例中，由于图像传感器中滤光片与各光信号输出电路的结构信息是固定的，因此，可以预先确定图像传感器中可以组成的EVS像素单元的预设像素的集合，在确定工作模式为HVS图像模式或者EVS图像模式后，可以根据需求从这些预设像素的集合中，选择目标集合，并将目标集合内的像素的目标传感模式设置为EVS模式。In some embodiments, since the structural information of the filter and each optical signal output circuit in the image sensor is fixed, the set of preset pixels that can constitute the EVS pixel unit in the image sensor can be determined in advance. After determining that the working mode is the HVS image mode or the EVS image mode, a target set can be selected from these preset pixel sets as needed, and the target sensing mode of the pixels in the target set can be set to the EVS mode.
在一些实施例中，可以根据需求实时地从图像传感器中，选择像素作为事件像素，然后根据这些事件像素对应的第一滤光片颜色，组成满足目标感知颜色信息的EVS像素单元。In some embodiments, pixels can be selected from the image sensor as event pixels in real time as required, and then EVS pixel units that meet the target perceived color information are formed according to the first filter colors corresponding to these event pixels.
具体地，若图像传感器的工作模式为HVS图像模式或EVS图像模式，则根据需求设置图像传感器中需要处于EVS模式的EVS像素。这些EVS像素可以集中或分散设置于图像传感器中。若EVS像素的感知颜色信息与目标感知颜色信息相同，则将单个EVS像素作为EVS像素单元；若EVS像素的感知颜色信息与目标感知颜色信息不相同，则需要查询其他的EVS像素的感知颜色信息，以与其他EVS像素进行组合，得到EVS像素单元，使EVS像素单元整体的感知颜色信息，与目标颜色信息相同。Specifically, if the working mode of the image sensor is the HVS image mode or the EVS image mode, the EVS pixels in the image sensor that need to be in the EVS mode are set according to the requirements. These EVS pixels can be set in the image sensor in a concentrated or dispersed manner. If the perceived color information of the EVS pixel is the same as the target perceived color information, a single EVS pixel is used as an EVS pixel unit; if the perceived color information of the EVS pixel is different from the target perceived color information, it is necessary to query the perceived color information of other EVS pixels to combine with other EVS pixels to obtain an EVS pixel unit, so that the overall perceived color information of the EVS pixel unit is the same as the target color information.
在本实施中，由于EVS像素单元对应的感知颜色信息与目标感知颜色信息相同，那么EVS像素单元中各EVS像素采集到的事件图像数据整体上满足目标感知颜色信息对应的感知需求。基于EVS像素单元中各EVS像素采集到的事件图像数据，可以得到EVS像素单元对应的、满足感知需求的目标事件图像数据。以EVS像素单元为一个完整的图像像素，利用EVS像素单元的目标事件图像数据，生成图像传感器采集的目标图像数据，从而使得图像传感器成像满足对目标颜色对应光信号的事件感知需求。In this embodiment, since the perceived color information corresponding to the EVS pixel unit is the same as the target perceived color information, the event image data collected by each EVS pixel in the EVS pixel unit as a whole meets the perception requirements corresponding to the target perceived color information. Based on the event image data collected by each EVS pixel in the EVS pixel unit, the target event image data corresponding to the EVS pixel unit that meets the perception requirements can be obtained. Taking the EVS pixel unit as a complete image pixel, the target event image data collected by the image sensor is generated using the target event image data of the EVS pixel unit, so that the image sensor imaging meets the event perception requirements for the light signal corresponding to the target color.
进一步地，若工作模式为混合图像模式或有源型图像模式，则基于有源像素单元中各有源像素对应的第二滤光片颜色确定目标感知颜色信息；Further, if the working mode is a mixed image mode or an active image mode, the target perceived color information is determined based on the second filter color corresponding to each active pixel in the active pixel unit;
其中，有源像素单元包括至少一个有源像素，有源像素是图像传感器中，目标传感模式为有源积分型像素模式的像素。The active pixel unit includes at least one active pixel, and the active pixel is a pixel in the image sensor whose target sensing mode is an active integral pixel mode.
在本实施例中，若工作模式为HVS图像模式，则图像传感器中会既包含EVS像素，又包含APS像素，根据有源积分的感知需求，一个或多个相邻的APS像素也可以组成APS像素单元，APS像素单元中各APS像素的各光电输出电路设置有不同颜色的滤光片，使得APS像素单元具有不同颜色的感光能力。In this embodiment, if the working mode is the HVS image mode, the image sensor will include both EVS pixels and APS pixels. According to the perception requirements of active integration, one or more adjacent APS pixels can also form an APS pixel unit. Each photoelectric output circuit of each APS pixel in the APS pixel unit is provided with a filter of a different color, so that the APS pixel unit has a photosensitivity of different colors.
在一些实施例中，APS像素单元中，至少包括一个设置有红色滤光片的光电输出电路，至少包括一个设置有绿色滤光片的光电输出电路，以及至少包括一个设置有蓝色滤光片的光电输出电路，以使APS像素单元能获得彩色的目标有源图像数据。In some embodiments, the APS pixel unit includes at least one photoelectric output circuit provided with a red filter, at least one photoelectric output circuit provided with a green filter, and at least one photoelectric output circuit provided with a blue filter, so that the APS pixel unit can obtain color target active image data.
基于APS像素单元中包含的若干APS像素对应的第二滤光片颜色，可以确定APS像素单元的感光颜色信息。在HVS图像模式下的图像传感器中，可以要求其EVS像素单元和APS像素单元的感知颜色信息保持一致，以保证HVS图像模式下不同类型的图像数据具有一致性。Based on the second filter colors corresponding to the APS pixels included in the APS pixel unit, the photosensitive color information of the APS pixel unit can be determined. In the image sensor in the HVS image mode, the perceived color information of the EVS pixel unit and the APS pixel unit can be required to be consistent to ensure the consistency of different types of image data in the HVS image mode.
因此，获取图像传感器的APS像素单元所包含APS像素中，各光电输出电路对应的第二滤光片颜色，根据第二滤光片颜色确定APS像素单元的感知颜色信息，APS像素单元的感知颜色信息可以包括APS像素单元能够感知的一种或多种不同波段光信号的颜色类型和/或颜色架构。将APS像素单元的感知颜色信息设置为EVS像素单元的目标感知颜色信息，以使得APS像素单元和EVS像素单元对不同颜色光信号的感知能力一致。APS像素单元基于其包含的APS像素对应的APS图像数据，可以得APS像素单元的目标有源图像数据，APS像素单元的目标有源图像数据与EVS像素单元的目标事件图像数据所包含的颜色效果一致，进而根据APS像素单元的目标有源图像数据和EVS像素单元的目标事件图像数据，可以融合生成图像传感器在HVS图像模式下采集到的目标图像数据，这样融合两种类型的图像数据的效果更好，能够进一步提高复合型图像传感器，融合两种传感模式时的成像质量。Therefore, the second filter color corresponding to each photoelectric output circuit in the APS pixel included in the APS pixel unit of the image sensor is obtained, and the perceived color information of the APS pixel unit is determined according to the second filter color. The perceived color information of the APS pixel unit may include the color type and/or color architecture of one or more different band light signals that the APS pixel unit can perceive. The perceived color information of the APS pixel unit is set as the target perceived color information of the EVS pixel unit, so that the perception capabilities of the APS pixel unit and the EVS pixel unit to light signals of different colors are consistent. Based on the APS image data corresponding to the APS pixels included in the APS pixel unit, the target active image data of the APS pixel unit can be obtained. The target active image data of the APS pixel unit is consistent with the color effect included in the target event image data of the EVS pixel unit. Then, according to the target active image data of the APS pixel unit and the target event image data of the EVS pixel unit, the target image data collected by the image sensor in the HVS image mode can be fused and generated. In this way, the effect of fusing the two types of image data is better, which can further improve the imaging quality of the composite image sensor when fusing the two sensing modes.
进一步地，事件像素单元中，相邻事件像素的感知颜色信息不同。Furthermore, in the event pixel unit, the perceived color information of adjacent event pixels is different.
在本实施例中，EVS像素单元中包括多个EVS像素时，会存在有两个EVS像素相邻，其中，当像素之间的距离小于预设距离时，两个像素相邻。相邻EVS像素之间所对应的传感区域则会对某一颜色光信号的感知效果更好，同时其对其他颜色光信号的感知效果将会较差，当摄入该传感区域的光信号颜色单一，并且属于该传感区域感知效果较差的颜色光信号时，该传感区域对光强变化事件感知能力较弱，影像成像质量。因此，设置出的EVS像素单元中，相邻的EVS像素对应的感知颜色信息不同。可以理解的是，EVS像素对应的感知颜色信息是基于EVS像素中各光电输出电路对应的滤光片颜色确定的，而EVS像素对应的感知颜色信息，也影响了其所在的EVS像素单元的感知颜色信息。In this embodiment, when the EVS pixel unit includes multiple EVS pixels, there will be two EVS pixels adjacent to each other, wherein the two pixels are adjacent when the distance between the pixels is less than the preset distance. The sensing area corresponding to the adjacent EVS pixels will have a better perception effect on a certain color light signal, while its perception effect on other color light signals will be poor. When the light signal entering the sensing area is of a single color and belongs to a color light signal with a poor perception effect in the sensing area, the sensing area has a weak ability to perceive light intensity change events, and the image quality is poor. Therefore, in the set EVS pixel unit, the perceived color information corresponding to adjacent EVS pixels is different. It can be understood that the perceived color information corresponding to the EVS pixel is determined based on the filter color corresponding to each photoelectric output circuit in the EVS pixel, and the perceived color information corresponding to the EVS pixel also affects the perceived color information of the EVS pixel unit in which it is located.
这样可以使得事件像素单元中，具有不同感知颜色信息的EVS像素交错设置，从而保证图像传感器不同传感区域的事件感知能力，从而提高图像传感器的成像质量。In this way, the EVS pixels with different perceived color information can be staggered in the event pixel unit, thereby ensuring the event perception capability of different sensing areas of the image sensor, thereby improving the imaging quality of the image sensor.
基于上述各实施例，以下提供多种的图像传感器及其控制方法的具体实施方式：Based on the above embodiments, various specific implementations of image sensors and control methods thereof are provided below:
示例性地，图像传感器包括传感阵列，传感阵列可以包括从下到上依次层叠设置的多个像素组成的像素阵列、彩色滤镜阵列及微透镜阵列。其中，像素阵列具体可以包括多个像素的光电输出电路，彩色滤镜阵列可以包括多个呈阵列交错设置的彩色滤光片，微透镜阵列可以包括多个呈阵列设置的微透镜。Exemplarily, the image sensor includes a sensor array, which may include a pixel array composed of a plurality of pixels stacked in sequence from bottom to top, a color filter array, and a microlens array. Specifically, the pixel array may include a plurality of photoelectric output circuits of pixels, the color filter array may include a plurality of color filters staggered in an array, and the microlens array may include a plurality of microlenses arranged in an array.
对应地，像素的每个光电输出电路上，设置有彩色滤镜阵列中的一个滤光片和微透镜阵列中至少一个微透镜。由于像素中各光电输出电路之间可以阵列排布或折线排布，对应还有滤光片和微透镜，共用选择跟随输出单元，各光电输出电路均可以独立于选择跟随输出单元结合，形成光强积分信号或光强对数信号。因此，可以用光电输出电路定义出子像素，为了方便理解，后续将光电转换电路称为子像素。Correspondingly, each photoelectric output circuit of the pixel is provided with a filter in the color filter array and at least one microlens in the microlens array. Since the photoelectric output circuits in the pixel can be arranged in an array or a zigzag line, and there are corresponding filters and microlenses, and a shared selection-follow output unit, each photoelectric output circuit can be independently combined with the selection-follow output unit to form a light intensity integral signal or a light intensity logarithmic signal. Therefore, a sub-pixel can be defined by a photoelectric output circuit. For ease of understanding, the photoelectric conversion circuit will be referred to as a sub-pixel in the following.
其中，彩色滤镜阵列可以为基于RGB三原色的不同颜色滤光片组成的阵列，可以是Bayer阵列、Quard Bayer阵列及RYYB阵列等；可以理解的是，出于其他设计需求，彩色滤镜阵列还可以略做变化，如提升感光能力的RGBW阵列，又如适应较暗光信号的RGBIR阵列等，在此不作限定。Among them, the color filter array can be an array composed of different color filters based on the three primary colors of RGB, which can be a Bayer array, a Quard Bayer array, and a RYYB array, etc.; it can be understood that for other design requirements, the color filter array can also be slightly changed, such as an RGBW array for improving photosensitivity, or an RGBIR array for adapting to darker light signals, etc., which is not limited here.
图像传感器所生成的目标图像数据中的子图像数据与EVS像素单元或APS像素单元对应的，这些像素单元对应可以由一个或多个像素组成。通过不同的滤光片的设置，使像素各个子像素可以采集相同或不同颜色通道对应的光强积分信息，像素对应可以得到一个或多个颜色分量对应的图像数据，不同像素对应的颜色分量不相同，从而可以组合多个像素得到像素单元，使得像素单元能够采集到多种颜色对应的目标有源图像数据或目标事件图像数据。The sub-image data in the target image data generated by the image sensor corresponds to the EVS pixel unit or the APS pixel unit, and these pixel units can be composed of one or more pixels. Through the setting of different filters, each sub-pixel of the pixel can collect the light intensity integral information corresponding to the same or different color channels, and the pixel can obtain image data corresponding to one or more color components. The color components corresponding to different pixels are different, so that multiple pixels can be combined to obtain a pixel unit, so that the pixel unit can collect target active image data or target event image data corresponding to multiple colors.
为了得到真实的彩色图像数据。对于组成像素单元的一个或多个像素中，至少包括一个设置有红色滤光片的子像素、至少包括一个设置有蓝色滤光片的子像素、至少包括一个设置有绿色滤光片的子像素。使得每个像素单元都能获得红色分量对应的图像数据、蓝色分量对应的图像数据以及绿色分量对应的图像数据。使得图像单元能够得到真实的彩色目标有源图像数据或目标事件图像数据，让图像传感器基于多个彩色目标有源图像数据和/或目标事件图像数据所生成的目标图像数据，颜色更加真实，图像质量更好。In order to obtain real color image data. For one or more pixels constituting a pixel unit, at least one sub-pixel is provided with a red filter, at least one sub-pixel is provided with a blue filter, and at least one sub-pixel is provided with a green filter. Each pixel unit can obtain image data corresponding to the red component, image data corresponding to the blue component, and image data corresponding to the green component. The image unit can obtain real color target active image data or target event image data, so that the target image data generated by the image sensor based on multiple color target active image data and/or target event image data has more realistic colors and better image quality.
示例一：如图7所示的传感阵列中，像素阵列对应的彩色滤镜阵列为Bayer阵列，相邻的1个红色滤光片R、2个绿色滤光片G及1个蓝色滤光片B所在的4个子像素，可以构成一个像素，即1个红色子像素、2个绿色子像素及1个蓝色子像素构成一个像素。此处，每个像素的2×2共4个子像素之间为阵列排布，4个子像素交接处的圆圈表征像素的中心，且每个像素中4个子像素的光电输出电路共用选择跟随输出单元，选择跟随输出单元包括复位电路、跟随晶体管SF和第二复位电源VSF、第一传输支路和第二传输支路等，其中，复位电路还包括复位晶体管RST和第一复位电源VPIX。Example 1: In the sensor array shown in FIG7 , the color filter array corresponding to the pixel array is a Bayer array, and the four sub-pixels where one adjacent red filter R, two green filters G, and one blue filter B are located can constitute one pixel, that is, one red sub-pixel, two green sub-pixels, and one blue sub-pixel constitute one pixel. Here, the 2×2 sub-pixels of each pixel are arranged in an array, and the circle at the intersection of the four sub-pixels represents the center of the pixel, and the photoelectric output circuits of the four sub-pixels in each pixel share a selection follower output unit, which includes a reset circuit, a follower transistor SF, a second reset power supply VSF, a first transmission branch, a second transmission branch, etc., wherein the reset circuit also includes a reset transistor RST and a first reset power supply VPIX.
详细地，如图7所示的传感阵列中，所对应的像素阵列处于HVS图像模式时如图8所示，像素阵列中的部分像素工作在APS模式，对应APS像素包括红色子像素R、绿色子像素G及蓝色子像素B，而像素阵列中的其他像素工作在EVS模式，对应子像素共用选择跟随输出单元，并联为EVS像素，记作E。In detail, in the sensor array shown in FIG7 , when the corresponding pixel array is in the HVS image mode as shown in FIG8 , some pixels in the pixel array operate in the APS mode, and the corresponding APS pixels include red sub-pixels R, green sub-pixels G, and blue sub-pixels B, while other pixels in the pixel array operate in the EVS mode, and the corresponding sub-pixels share a selection follower output unit and are connected in parallel as EVS pixels, denoted as E.
需要说明的是，每个像素中子像素的个数和子像素的分布方式不局限于如图7所示的排布方式，子像素的个数还可以是8个、16个等其他数目，各个子像素可以是按照阵列排布或者折线排布等其他形状排布，可根据实际需求灵活调整，在此不作限定。It should be noted that the number of sub-pixels in each pixel and the distribution of the sub-pixels are not limited to the arrangement shown in Figure 7. The number of sub-pixels can also be 8, 16, or other numbers. The individual sub-pixels can be arranged in other shapes such as array arrangement or zigzag arrangement, and can be flexibly adjusted according to actual needs, and are not limited here.
像素阵列中的部分像素工作在EVS模式下，且EVS像素E对应的4个子像素刚好对应一个完整的APS像素，基于RGB三原色的4个子像素，每个EVS像素E均能对可见光进行有效感知，此时，像素E的位置和个数可根据实际需求灵活选择。Some pixels in the pixel array work in EVS mode, and the four sub-pixels corresponding to the EVS pixel E just correspond to a complete APS pixel. Based on the four sub-pixels of the three primary colors of RGB, each EVS pixel E can effectively perceive visible light. At this time, the position and number of pixels E can be flexibly selected according to actual needs.
示例二：如图9所示的传感阵列中，像素阵列对应的彩色滤镜阵列为Quard Bayer阵列，相邻的4个红色滤光片R、8个绿色滤光片G及4个蓝色滤光片B所对应的16个子像素构成一个像素单元，即4个红色子像素、8个绿色子像素及4个蓝色子像素构成一个像素单元，此处，每个像素单元为4×4共16个子像素的阵列排布方式，其在Bayer阵列的基础上对每个子像素进行了4倍放大拓展，呈2×2阵列排布设置的4个红色子像素等效于一个复合红色像素，呈2×2阵列排布设置的4个绿色子像素等效于一个复合绿色像素，呈2×2阵列排布设置的4个绿色子像素等效于一个复合绿色像素，呈2×2阵列排布设置的4个蓝色子像素等效于一个复合蓝色像素，1个复合红色像素、2个复合绿色像素及1个复合蓝色像素构成一个像素单元。Example 2: In the sensor array shown in FIG. 9 , the color filter array corresponding to the pixel array is a Quard Bayer array, and 16 sub-pixels corresponding to 4 adjacent red filters R, 8 green filters G, and 4 blue filters B constitute a pixel unit, that is, 4 red sub-pixels, 8 green sub-pixels, and 4 blue sub-pixels constitute a pixel unit. Here, each pixel unit is a 4×4 array arrangement of 16 sub-pixels, which magnifies and expands each sub-pixel by 4 times on the basis of the Bayer array. The 4 red sub-pixels arranged in a 2×2 array are equivalent to a composite red pixel, the 4 green sub-pixels arranged in a 2×2 array are equivalent to a composite green pixel, the 4 green sub-pixels arranged in a 2×2 array are equivalent to a composite green pixel, the 4 blue sub-pixels arranged in a 2×2 array are equivalent to a composite blue pixel, and 1 composite red pixel, 2 composite green pixels, and 1 composite blue pixel constitute a pixel unit.
此时，每个像素（复合红色像素、复合绿色像素或者复合蓝色像素）中4个子像素共用选择跟随输出单元，这使得该复合像素既能作为APS像素单元的部分，又能作为EVS像素。其中，4个子像素交接处的圆圈表征像素的中心。At this time, the four sub-pixels in each pixel (composite red pixel, composite green pixel or composite blue pixel) share the select-follow output unit, which makes the composite pixel both part of the APS pixel unit and as an EVS pixel. The circle at the intersection of the four sub-pixels represents the center of the pixel.
详细地，如图9所示的传感阵列中，所对应的像素阵列处于HVS图像模式时如图10所示，像素阵列中的部分像素工作在APS模式，对应包括4个红色子像素R、4个绿色像素G或者4个蓝色子像素B的像素，属于APS像素单元的部分，而像素阵列中的其他像素工作在EVS模式，这些像素的子像素处于EVS模式且并联，形成一个大的EVS像素记为E。In detail, in the sensor array shown in FIG9 , when the corresponding pixel array is in the HVS image mode as shown in FIG10 , some pixels in the pixel array operate in the APS mode, corresponding to pixels including 4 red sub-pixels R, 4 green pixels G or 4 blue sub-pixels B, which are part of the APS pixel unit, while other pixels in the pixel array operate in the EVS mode, and the sub-pixels of these pixels are in the EVS mode and connected in parallel to form a large EVS pixel denoted as E.
示例三：如图11所示的传感阵列中，像素阵列对应的彩色滤镜阵列为Bayer阵列，相邻的2个红色滤光片R、8个绿色滤光片G、2个蓝色滤光片B及4个白色滤光片W所对应的16个子像素构成一个像素单元，即2个红色子像素、8个绿色子像素、2个蓝色子像素及4个白色子像素构成一个像素单元。此处，每个像素为4×4共16个子像素的呈阵列排布方式，呈2×2正方形设置的1个红色子像素、2个绿色子像素及1个白色子像素构成1个第一类像素，呈2×2正方形设置的2个绿色子像素、1个蓝色子像素及1个白色子像素构成1个第二类像素，相邻的2个第一类像素与2个第二类像素构成一个像素单元。Example 3: In the sensor array shown in FIG11 , the color filter array corresponding to the pixel array is a Bayer array, and the 16 sub-pixels corresponding to the two adjacent red filters R, eight green filters G, two blue filters B, and four white filters W constitute a pixel unit, that is, two red sub-pixels, eight green sub-pixels, two blue sub-pixels, and four white sub-pixels constitute a pixel unit. Here, each pixel is arranged in an array of 4×4 with a total of 16 sub-pixels, and one red sub-pixel, two green sub-pixels, and one white sub-pixel arranged in a 2×2 square form a first-type pixel, and two green sub-pixels, one blue sub-pixel, and one white sub-pixel arranged in a 2×2 square form a second-type pixel, and two adjacent first-type pixels and two second-type pixels constitute a pixel unit.
此时，每个第一类像素或者第二类像素中4个子像素共用选择跟随输出单元，这使得这些像素既能作为APS像素单元的部分，又能作为EVS像素。其中，4个子像素交接处的圆圈表征了像素的中心。At this time, the four sub-pixels in each first-type pixel or second-type pixel share the selection follower output unit, which allows these pixels to be part of the APS pixel unit and also as EVS pixels. The circle at the intersection of the four sub-pixels represents the center of the pixel.
详细地，如图11所示的传感阵列中，所对应的像素阵列处于HVS图像模式时如图12所示，像素阵列中的部分像素工作在APS模式，而像素阵列中的其他像素工作在EVS模式，记为E。In detail, in the sensing array shown in FIG. 11 , when the corresponding pixel array is in the HVS image mode, as shown in FIG. 12 , some pixels in the pixel array operate in the APS mode, while other pixels in the pixel array operate in the EVS mode, which is denoted as E.
以下基于上述示例的传感阵列（如图7－图12所示），对其相应图像传感器的三种工作模式进行详细展开说明。Based on the sensing array of the above example (as shown in FIG. 7 to FIG. 12 ), the three working modes of the corresponding image sensor are described in detail below.
1、APS图像模式：如图13所示，像素阵列中的所有像素均工作在APS模式，为全尺寸APS图像模式，对应的子像素均记为像素A。1. APS image mode: As shown in FIG. 13 , all pixels in the pixel array operate in the APS mode, which is a full-size APS image mode, and the corresponding sub-pixels are all recorded as pixel A.
2、EVS图像模式：如图14所示，像素阵列中的所有像素均工作在EVS模式，为全尺寸EVS图像模式；或者如图15、图16所示，像素阵列中的部分像素工作在EVS模式而其他像素不工作，为部分尺寸EVS图像模式。2. EVS image mode: As shown in FIG. 14 , all pixels in the pixel array operate in the EVS mode, which is a full-size EVS image mode; or as shown in FIGS. 15 and 16 , some pixels in the pixel array operate in the EVS mode while other pixels do not operate, which is a partial-size EVS image mode.
其中，工作在EVS模式下且选择跟随输出单元的4个子像素等效为像素E，此时4个子像素对应的光电二极管PD处于并联状态，可等效于一个大的光电二极管PD进行EVS成像，作为一个更大的EVS像素。Among them, the four sub-pixels working in the EVS mode and selected to follow the output unit are equivalent to pixel E. At this time, the photodiodes PD corresponding to the four sub-pixels are in a parallel state, which can be equivalent to a large photodiode PD for EVS imaging, as a larger EVS pixel.
3、HVS图像模式：如图17或者图18所示，像素阵列中的部分像素工作在APS模式，而像素阵列中的其他像素工作在EVS模式。3. HVS image mode: As shown in FIG. 17 or FIG. 18 , some pixels in the pixel array operate in the APS mode, while other pixels in the pixel array operate in the EVS mode.
其中，工作在APS模式的子像素可以记为APS像素A，工作在EVS模式下且共用选择跟随输出单元的4个子像素可以等效为EVS像素E。Among them, the sub-pixel working in the APS mode can be recorded as APS pixel A, and the four sub-pixels working in the EVS mode and sharing the select-follow-output unit can be equivalent to EVS pixel E.
在部分尺寸EVS图像模式和或者HVS图像模式下，工作在EVS模式下的多个像素或者子像素可以是部分区块集中设置，可以是按行或者列分散设置，可以是按对角线等斜线或者折线段分散设置，还可以是离散多点式阵列设置，可根据实际需求，基于控制器的控制灵活设计，在此不作限定。In the partial-size EVS image mode and/or HVS image mode, multiple pixels or sub-pixels operating in the EVS mode may be concentrated in a partial block, dispersed by rows or columns, dispersed by diagonal lines, oblique lines, or broken line segments, or may be arranged in a discrete multi-point array. The design may be flexibly based on the control of the controller according to actual needs and is not limited here.
其中，在多个EVS像素分散设置时，如果单个EVS像素对应的多个子像素并非一个完整地包含RGB三原色的APS像素单元，而是构成APS像素单元的一个APS像素，即单个EVS像素无法完全感知APS像素单元所能感知的颜色，则使相邻的多个（如4个）EVS像素构成EVS像素单元，并使其与对应的APS像素单元一致，使得相邻的多个EVS像素E组成的EVS像素单元能有效对各种所需颜色（如红绿蓝）或者波段（如红外）的光信号分别进行感知采样，从而EVS像素单元可以等效于一个更大的EVS像素，以提升其事件感知灵敏度。Among them, when multiple EVS pixels are dispersedly arranged, if the multiple sub-pixels corresponding to a single EVS pixel are not an APS pixel unit that completely contains the three primary colors of RGB, but an APS pixel that constitutes an APS pixel unit, that is, a single EVS pixel cannot fully perceive the color that the APS pixel unit can perceive, then multiple adjacent (such as 4) EVS pixels constitute an EVS pixel unit, and make them consistent with the corresponding APS pixel unit, so that the EVS pixel unit composed of multiple adjacent EVS pixels E can effectively perceive and sample light signals of various required colors (such as red, green and blue) or bands (such as infrared), so that the EVS pixel unit can be equivalent to a larger EVS pixel to improve its event perception sensitivity.
详细地，如图8所示，像素阵列中的部分像素工作在EVS模式下，且EVS像素E对应的4个子像素刚好对应一个完整的APS像素，即包含基于RGB三原色的4个子像素，每个EVS像素E均能对可见光进行有效感知，此时，像素E的位置和个数可根据实际需求灵活选择。In detail, as shown in FIG8 , some pixels in the pixel array operate in the EVS mode, and the four sub-pixels corresponding to the EVS pixel E just correspond to a complete APS pixel, that is, it includes four sub-pixels based on the three primary colors of RGB. Each EVS pixel E can effectively perceive visible light. At this time, the position and number of pixels E can be flexibly selected according to actual needs.
详细地，如图10所示，像素阵列中的部分像素工作在EVS模式下，且EVS像素E对应的4个子像素刚好对应一个完整的APS像素单元的1/4，其中，完整APS像素单元的要求是其对应至少一个蓝色子像素、至少一个绿色子像素和至少一个红色子像素。每个EVS像素只能对红光、绿光及蓝光中的一种进行感知，如果相邻的多个EVS像素均只能对某一种颜色的光（如红光）进行感知，则当该区域的光中其他颜色的光的占比更大或者完全是其他颜色的光（如蓝光）时，该区域的多个EVS像素的事件感知能力较弱或者完全无法做到事件感知，因此，使得相邻的4个分别感知红光、绿光及蓝光的EVS像素组成EVS像素单元，并按照对应的APS像素单元的架构分布，形成RGGB布局，等效于一个更大的事件像素，该事件像素单元能有效对各种可见光进行动态事件感知。In detail, as shown in FIG10 , some pixels in the pixel array work in the EVS mode, and the four sub-pixels corresponding to the EVS pixel E just correspond to 1/4 of a complete APS pixel unit, wherein the requirement for a complete APS pixel unit is that it corresponds to at least one blue sub-pixel, at least one green sub-pixel, and at least one red sub-pixel. Each EVS pixel can only sense one of red light, green light, and blue light. If multiple adjacent EVS pixels can only sense light of a certain color (such as red light), then when the proportion of light of other colors in the light of the area is greater or it is completely other colors (such as blue light), the event sensing ability of multiple EVS pixels in the area is weak or completely unable to sense events. Therefore, the four adjacent EVS pixels that sense red light, green light, and blue light respectively form an EVS pixel unit, and are distributed according to the architecture of the corresponding APS pixel unit to form an RGGB layout, which is equivalent to a larger event pixel. The event pixel unit can effectively sense dynamic events of various visible lights.
详细地，如图12所示，像素阵列中的部分像素工作在EVS模式下，且EVS像素对应的4个子像素刚好对应一个完整的APS像素的1/4，EVS像素对红光或者蓝光的感知偏弱，如果相邻的多个EVS像素E的结构完全一样，那么其对应的颜色感知能力则是一样的，均对某一种颜色的光（如红光）的感知偏弱，则当该区域的光中该颜色的光的占比更大或者完全是该颜色的光时，该区域的多个EVS像素E的事件感知能力较弱，因此，使得相邻的4个EVS像素的感光能力交错设置，将感知红光偏弱的EVS像素与感知绿光偏弱的EVS像素E进行互补，并按照对应的APS像素的架构分布，形成RGBW布局，等效于一个更大的事件像素，该事件像素能有效对各种可见光进行动态事件感知。In detail, as shown in FIG12 , some pixels in the pixel array operate in the EVS mode, and the four sub-pixels corresponding to the EVS pixels just correspond to 1/4 of a complete APS pixel. The EVS pixels are weakly sensitive to red light or blue light. If the structures of multiple adjacent EVS pixels E are exactly the same, then their corresponding color perception capabilities are the same, and they all perceive a certain color of light (such as red light) weakly. When the light of this color accounts for a larger proportion of the light in this area or is entirely light of this color, the event perception capabilities of the multiple EVS pixels E in this area are weaker. Therefore, the photosensitivity of the four adjacent EVS pixels is staggered, and the EVS pixels that perceive red light weakly are complemented with the EVS pixels E that perceive green light weakly, and are distributed according to the architecture of the corresponding APS pixels to form an RGBW layout, which is equivalent to a larger event pixel. The event pixel can effectively perceive dynamic events of various visible lights.
示例四：像素对应的光电输出电路呈折线排布，也即像素呈阵列排布，而像素对应子像素呈折线排布。Example 4: The photoelectric output circuits corresponding to the pixels are arranged in a zigzag line, that is, the pixels are arranged in an array, and the sub-pixels corresponding to the pixels are arranged in a zigzag line.
参见图19－图20，连接四点的折线段表示共用选择跟随输出单元的4个子像素。工作在APS模式的子像素记为A，工作在EVS模式的子像素记为E。由于跨两列的折线或者跨两行的折线对应的子像素共用选择跟随输出单元，在APS图像模式下，折线排布的子像素构成一个APS像素或者APS像素单元的部分，而在EVS图像模式下，折线排布的子像素构成一个EVS像素或者EVS像素单元的部分。图像传感器可以工作在如图19所示的APS图像模式下，也可以工作在如图20所示的HVS图像模式下。Referring to FIG. 19 and FIG. 20, the broken line segment connecting the four points represents four sub-pixels that share a selection-follow output unit. The sub-pixel working in the APS mode is denoted as A, and the sub-pixel working in the EVS mode is denoted as E. Since the sub-pixels corresponding to the broken lines across two columns or across two rows share a selection-follow output unit, in the APS image mode, the sub-pixels arranged in the broken lines constitute an APS pixel or part of an APS pixel unit, while in the EVS image mode, the sub-pixels arranged in the broken lines constitute an EVS pixel or part of an EVS pixel unit. The image sensor can work in the APS image mode as shown in FIG. 19, or in the HVS image mode as shown in FIG. 20.
示例性地，像素阵列对应的彩色滤镜阵列同样可以是Bayer阵列、Quard Bayer阵列或者RGBW阵列等。Exemplarily, the color filter array corresponding to the pixel array may also be a Bayer array, a Quard Bayer array, or an RGBW array.
如图21所示，彩色滤镜阵列为Bayer阵列，相邻的1个红色滤光片R、2个绿色滤光片G及1个蓝色滤光片B所对应的4个子像素构成一个像素，即1个红色子像素、2个绿色子像素及1个蓝色子像素构成一个像素，此处，每个像素为连接四点的折线段布局方式，且每个像素中4个子像素的子像素共用选择跟随输出单元，这使得该像素既能作为APS像素，又能作为EVS像素。As shown in FIG. 21 , the color filter array is a Bayer array, and four sub-pixels corresponding to one adjacent red filter R, two green filters G, and one blue filter B constitute one pixel, that is, one red sub-pixel, two green sub-pixels, and one blue sub-pixel constitute one pixel. Here, each pixel is laid out in a broken line segment connecting four points, and the four sub-pixels in each pixel share a selection follower output unit, which allows the pixel to be used as both an APS pixel and an EVS pixel.
详细地，如图21所示的传感阵列中，所对应的像素阵列处于HVS图像模式时如图22所示，像素阵列中的部分像素工作在APS模式，APS像素包括红色子像素R、绿色子像素G及蓝色子像素B，而像素阵列中的其他像素工作在EVS模式，对应子像素记为E。In detail, in the sensor array shown in FIG21 , when the corresponding pixel array is in the HVS image mode as shown in FIG22 , some pixels in the pixel array operate in the APS mode, and the APS pixels include red sub-pixels R, green sub-pixels G, and blue sub-pixels B, while other pixels in the pixel array operate in the EVS mode, and the corresponding sub-pixels are denoted as E.
详细地，如图23所示，彩色滤镜阵列为Quard Bayer阵列，每个像素为四段并排设置的连接四点的折线排布，每段折线段表示1个像素，4个子像素构成一个像素。此时相邻的4个红色滤光片R、8个绿色滤光片G及4个蓝色滤光片B所对应的16个子像素构成一个像素单元，即4个红色子像素构成的一个红色像素、8个绿色子像素构成的两个绿色像素，以及4个蓝色子像素构成的一个蓝色像素，合并为一个像素单元，此时，每个像素中4个子像素的子像素共用选择跟随输出单元，这使得该像素既能作为APS或APS像素单元的部分，又能作为EVS像素。In detail, as shown in FIG23, the color filter array is a Quard Bayer array, and each pixel is arranged as four folded lines connecting four points arranged side by side, each folded line segment represents one pixel, and four sub-pixels constitute one pixel. At this time, the 16 sub-pixels corresponding to the four adjacent red filters R, eight green filters G, and four blue filters B constitute a pixel unit, that is, a red pixel composed of four red sub-pixels, two green pixels composed of eight green sub-pixels, and a blue pixel composed of four blue sub-pixels are combined into one pixel unit. At this time, the sub-pixels of the four sub-pixels in each pixel share a selection follower output unit, which enables the pixel to be used as both an APS or part of an APS pixel unit and an EVS pixel.
详细地，如图23所示的传感阵列中，所对应的像素阵列处于HVS图像模式时如图24所示，像素阵列中的部分像素工作在APS模式，对应像素为红色子像素R、绿色子像素G及蓝色子像素B像素，属于APS像素单元的部分，而像素阵列中的其他像素工作在EVS模式，对应子像素记为E。In detail, in the sensor array shown in Figure 23, when the corresponding pixel array is in the HVS image mode as shown in Figure 24, some pixels in the pixel array operate in the APS mode, and the corresponding pixels are the red sub-pixel R, the green sub-pixel G and the blue sub-pixel B pixels, which are part of the APS pixel unit, while the other pixels in the pixel array operate in the EVS mode, and the corresponding sub-pixels are denoted as E.
详细地，如图25所示，彩色滤镜阵列为RGBW阵列，每个像素为四段并排设置的连接四点的折线排布，每段折线段表示1个像素，4个子像素构成一个像素。此时相邻的2个红色滤光片R、8个绿色滤光片G、2个蓝色滤光片B及4个白色滤光片W所对应的16个子像素构成一个像素单元，即2个红色子像素、8个绿色子像素、2个蓝色子像素及4个白色子像素构成一个像素单元。每个像素中4个子像素共用选择跟随输出单元，这使得该像素既能作为APS像素或APS像素单元的部分，又能作为EVS像素。In detail, as shown in FIG25 , the color filter array is an RGBW array, and each pixel is arranged as four folded lines connecting four points arranged side by side, each folded line segment represents one pixel, and four sub-pixels constitute one pixel. At this time, the 16 sub-pixels corresponding to the two adjacent red filters R, eight green filters G, two blue filters B, and four white filters W constitute one pixel unit, that is, two red sub-pixels, eight green sub-pixels, two blue sub-pixels, and four white sub-pixels constitute one pixel unit. The four sub-pixels in each pixel share a selection-following output unit, which enables the pixel to be used as both an APS pixel or part of an APS pixel unit, and an EVS pixel.
详细地，如图25所示的传感阵列中，所对应的像素阵列处于HVS图像模式时如图26所示，像素阵列中共用选择跟随输出单元的4个白色子像素（与白色滤光片W对应）所在对应的像素工作在EVS模式，因而这些子像素记为E，而其他像素工作在APS模式。In detail, in the sensor array shown in FIG25 , when the corresponding pixel array is in the HVS image mode as shown in FIG26 , the corresponding pixels of the four white sub-pixels (corresponding to the white filter W) in the pixel array that share the selection follow output unit operate in the EVS mode, and thus these sub-pixels are recorded as E, while the other pixels operate in the APS mode.
由此可见，通过折线排布像素对应的光电输出电路，在HVS图像模式下虽然损失了一半的红色子像素和蓝色子像素，但未损失绿色子像素。从而在HVS图像模式下，可以保证APS图像质量最小损失，同时获得EVS图像数据，提高了图像传感器整体的图像质量。如图27所示，图27为本发明实施例提供的光电设备1100的结构示意图。It can be seen that by arranging the photoelectric output circuits corresponding to the pixels in a zigzag line, although half of the red sub-pixels and blue sub-pixels are lost in the HVS image mode, the green sub-pixels are not lost. Therefore, in the HVS image mode, the APS image quality can be guaranteed to be minimally lost, while the EVS image data is obtained, thereby improving the overall image quality of the image sensor. As shown in FIG. 27 , FIG. 27 is a schematic diagram of the structure of the optoelectronic device 1100 provided in an embodiment of the present invention.
该光电设备1100包括有一个或者一个以上处理核心的处理器1101、有一个或一个以上计算机可读存储介质的存储器1102及存储在存储器1102上并可在处理器上运行的计算机程序。其中，处理器1101与存储器1102电性连接。本领域技术人员可以理解，图中示出的光电设备结构并不构成对光电设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。The optoelectronic device 1100 includes a processor 1101 having one or more processing cores, a memory 1102 having one or more computer-readable storage media, and a computer program stored in the memory 1102 and executable on the processor. The processor 1101 is electrically connected to the memory 1102. Those skilled in the art will appreciate that the optoelectronic device structure shown in the figure does not constitute a limitation on the optoelectronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.
处理器1101是光电设备1100的控制中心，利用各种接口和线路连接整个光电设备1100的各个部分，通过运行或加载存储在存储器1102内的软件程序和/或单元，以及调用存储在存储器1102内的数据，执行光电设备1100的各种功能和处理数据，从而对光电设备1100进行整体监控。处理器1101可以是处理器CPU、图形处理器GPU、网络处理器（NetworkProcessor，NP)等，可以实现或者执行本发明实施例中公开的各方法、步骤及逻辑框图。The processor 1101 is the control center of the optoelectronic device 1100, and uses various interfaces and lines to connect various parts of the entire optoelectronic device 1100. By running or loading software programs and/or units stored in the memory 1102, and calling data stored in the memory 1102, the processor 1101 executes various functions of the optoelectronic device 1100 and processes data, thereby monitoring the optoelectronic device 1100 as a whole. The processor 1101 can be a processor CPU, a graphics processor GPU, a network processor (Network Processor, NP), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present invention.
在本发明实施例中，光电设备1100中的处理器1101会按照如下的步骤，将一个或一个以上的应用程序的进程对应的指令加载到存储器1102中，并由处理器1101来运行存储在存储器1102中的应用程序，从而实现各种功能，例如：In the embodiment of the present invention, the processor 1101 in the optoelectronic device 1100 will load instructions corresponding to the processes of one or more application programs into the memory 1102 according to the following steps, and the processor 1101 will run the application programs stored in the memory 1102 to implement various functions, such as:
获取图像传感器配置信息；Get image sensor configuration information;
根据所述图像传感器配置信息确定所述图像传感器中各个像素的目标传感模式；determining a target sensing mode for each pixel in the image sensor according to the image sensor configuration information;
若所述目标传感模式为有源积分型像素模式，控制所述光电转换模块连通所述第一传输支路，以使所述光电转换模块通过所述第一传输支路输出光强积分信号至所述有源积分型像素读出单元，得到所述像素对应的有源图像数据；If the target sensing mode is an active integral pixel mode, controlling the photoelectric conversion module to connect to the first transmission branch, so that the photoelectric conversion module outputs a light intensity integral signal to the active integral pixel readout unit through the first transmission branch, and obtains active image data corresponding to the pixel;
若所述目标传感模式为事件型像素模式，控制所述光电转换模块连通所述第二传输支路，以使所述光电转换模块通过所述第二传输支路输出光强对数信号至所述事件型像素读出单元，得到所述像素对应的事件图像数据。If the target sensing mode is an event-type pixel mode, the photoelectric conversion module is controlled to connect to the second transmission branch so that the photoelectric conversion module outputs a light intensity logarithmic signal to the event-type pixel readout unit through the second transmission branch to obtain event image data corresponding to the pixel.
以上各个操作的具体实施可参见前面的实施例，在此不再赘述。The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
如图27所示，光电设备1100还包括：图像传感器1103，处理器1101分别与图像传感器1103电性连接，图像传感器1103可以是如上述实施例涉及的图像传感器。本领域技术人员可以理解，图27中示出的光电设备结构并不构成对光电设备的限定，可以包括比图示更多或更少的部件，或者组合某些部件，或者不同的部件布置。As shown in FIG27 , the optoelectronic device 1100 further includes: an image sensor 1103, and the processor 1101 is electrically connected to the image sensor 1103, respectively. The image sensor 1103 may be an image sensor as described in the above embodiment. Those skilled in the art may understand that the optoelectronic device structure shown in FIG27 does not constitute a limitation on the optoelectronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
在上述实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
本领域普通技术人员可以理解，上述实施例的各种方法中的全部或部分步骤可以通过指令来完成，或通过指令控制相关的硬件来完成，该指令可以存储于一计算机可读存储介质中，并由处理器进行加载和执行。A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
为此，本发明实施例提供一种计算机可读存储介质，其中存储有多条计算机程序，该计算机程序能够被处理器进行加载，以执行本发明实施例所提供的任一种图像传感器的控制方法。该计算机程序可以执行如下图像传感器的控制方法的步骤：To this end, an embodiment of the present invention provides a computer-readable storage medium, in which a plurality of computer programs are stored, and the computer program can be loaded by a processor to execute any image sensor control method provided by an embodiment of the present invention. The computer program can execute the following steps of the image sensor control method:
获取图像传感器配置信息；Get image sensor configuration information;
根据所述图像传感器配置信息确定所述图像传感器中各个像素的目标传感模式；determining a target sensing mode for each pixel in the image sensor according to the image sensor configuration information;
若所述目标传感模式为有源积分型像素模式，控制所述光电转换模块连通所述第一传输支路，以使所述光电转换模块通过所述第一传输支路输出光强积分信号至所述有源积分型像素读出单元，得到所述像素对应的有源图像数据；If the target sensing mode is an active integral pixel mode, controlling the photoelectric conversion module to connect to the first transmission branch, so that the photoelectric conversion module outputs a light intensity integral signal to the active integral pixel readout unit through the first transmission branch, and obtains active image data corresponding to the pixel;
若所述目标传感模式为事件型像素模式，控制所述光电转换模块连通所述第二传输支路，以使所述光电转换模块通过所述第二传输支路输出光强对数信号至所述事件型像素读出单元，得到所述像素对应的事件图像数据。If the target sensing mode is an event-type pixel mode, the photoelectric conversion module is controlled to connect to the second transmission branch so that the photoelectric conversion module outputs a light intensity logarithmic signal to the event-type pixel readout unit through the second transmission branch to obtain event image data corresponding to the pixel.
以上各个操作的具体实施可参见前面的实施例，在此不再赘述。The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
其中，该计算机可读存储介质可以包括：只读存储器（ROM，Read Only Memory）、随机存取记忆体（RAM，Random Access Memory）、磁盘或光盘等。The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
由于该计算机可读存储介质中所存储的计算机程序，可以执行本发明实施例所提供的任一种图像传感器的控制方法，因此，可以实现本发明实施例所提供的任一种图像传感器的控制方法所能实现的有益效果，详见前面的实施例，在此不再赘述。Since the computer program stored in the computer-readable storage medium can execute any image sensor control method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any image sensor control method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
在上述图像传感器的控制装置、计算机可读存储介质、光电设备、计算机程序产品实施例中，对各个实施例的描述都各有侧重，某个实施例中没有详述的部分，可以参见其他实施例的相关描述。所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的图像传感器的控制装置、计算机可读存储介质、计算机程序产品、光电设备及其相应单元的具体工作过程及可带来的有益效果，可以参考如上实施例中图像传感器的控制方法的说明，具体在此不再赘述。In the above-mentioned control device of the image sensor, the computer-readable storage medium, the optoelectronic device, and the computer program product embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, reference can be made to the relevant description of other embodiments. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process and beneficial effects of the control device of the image sensor, the computer-readable storage medium, the computer program product, the optoelectronic device and its corresponding units described above can refer to the description of the control method of the image sensor in the above embodiment, and will not be repeated here.
以上对本发明实施例所提供的一种图像传感器、图像传感器的控制方法、光电设备、计算机可读存储介质以及计算机程序产品进行了详细介绍，本文中应用了具体个例对本发明的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本发明的方法及其核心思想；同时，对于本领域的技术人员，依据本发明的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本发明的限制。The above is a detailed introduction to an image sensor, an image sensor control method, an optoelectronic device, a computer-readable storage medium, and a computer program product provided in the embodiments of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411158945.6ACN118828238A (en) | 2024-08-22 | 2024-08-22 | Image sensor, image sensor control method, optoelectronic device and medium |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202411158945.6ACN118828238A (en) | 2024-08-22 | 2024-08-22 | Image sensor, image sensor control method, optoelectronic device and medium |
| Publication Number | Publication Date |
|---|---|
| CN118828238Atrue CN118828238A (en) | 2024-10-22 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202411158945.6APendingCN118828238A (en) | 2024-08-22 | 2024-08-22 | Image sensor, image sensor control method, optoelectronic device and medium |
| Country | Link |
|---|---|
| CN (1) | CN118828238A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119729242A (en)* | 2024-12-27 | 2025-03-28 | 维沃移动通信有限公司 | Image sensor, electronic device, and image generation method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104253983A (en)* | 2008-12-08 | 2014-12-31 | 索尼株式会社 | Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus |
| CN113971635A (en)* | 2021-10-27 | 2022-01-25 | 深圳锐视智芯科技有限公司 | Image array format conversion method and device, electronic equipment and storage medium |
| CN114051108A (en)* | 2021-12-10 | 2022-02-15 | 深圳锐视智芯科技有限公司 | A mode switchable image sensor and image sensor control method |
| CN116489523A (en)* | 2022-01-21 | 2023-07-25 | 三星电子株式会社 | Image sensor and electronic device including image sensor |
| CN116684756A (en)* | 2023-05-05 | 2023-09-01 | 天津天芯微系统集成研究院有限公司 | Dynamic visual image sensor based on column-level ADC quantization and operation method thereof |
| WO2023174653A1 (en)* | 2022-03-17 | 2023-09-21 | Sony Semiconductor Solutions Corporation | Hybrid image and event sensing with rolling shutter compensation |
| CN116801123A (en)* | 2023-07-26 | 2023-09-22 | 上海传英信息技术有限公司 | Image sensor, image processing device and intelligent terminal |
| CN118075631A (en)* | 2024-02-28 | 2024-05-24 | 深圳大学 | Multi-mode low-power consumption reading interface circuit suitable for dynamic vision sensor |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104253983A (en)* | 2008-12-08 | 2014-12-31 | 索尼株式会社 | Solid-state imaging device, method for processing signal of solid-state imaging device, and imaging apparatus |
| CN113971635A (en)* | 2021-10-27 | 2022-01-25 | 深圳锐视智芯科技有限公司 | Image array format conversion method and device, electronic equipment and storage medium |
| CN114051108A (en)* | 2021-12-10 | 2022-02-15 | 深圳锐视智芯科技有限公司 | A mode switchable image sensor and image sensor control method |
| CN116489523A (en)* | 2022-01-21 | 2023-07-25 | 三星电子株式会社 | Image sensor and electronic device including image sensor |
| WO2023174653A1 (en)* | 2022-03-17 | 2023-09-21 | Sony Semiconductor Solutions Corporation | Hybrid image and event sensing with rolling shutter compensation |
| CN116684756A (en)* | 2023-05-05 | 2023-09-01 | 天津天芯微系统集成研究院有限公司 | Dynamic visual image sensor based on column-level ADC quantization and operation method thereof |
| CN116801123A (en)* | 2023-07-26 | 2023-09-22 | 上海传英信息技术有限公司 | Image sensor, image processing device and intelligent terminal |
| CN118075631A (en)* | 2024-02-28 | 2024-05-24 | 深圳大学 | Multi-mode low-power consumption reading interface circuit suitable for dynamic vision sensor |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN119729242A (en)* | 2024-12-27 | 2025-03-28 | 维沃移动通信有限公司 | Image sensor, electronic device, and image generation method |
| Publication | Publication Date | Title |
|---|---|---|
| US8913166B2 (en) | Solid-state imaging apparatus | |
| CN110113546B (en) | Imaging system and combination and readout method of adjacent pixel units in pixel array | |
| EP2302905B1 (en) | Imaging device and camera system | |
| JP4881987B2 (en) | Solid-state imaging device and imaging device | |
| US8350940B2 (en) | Image sensors and color filter arrays for charge summing and interlaced readout modes | |
| CN204633945U (en) | Image sensor, imaging system and processor system | |
| JP4611296B2 (en) | Charge binning image sensor | |
| JP6239975B2 (en) | Solid-state imaging device and imaging system using the same | |
| US7663680B2 (en) | Arrangement of circuits in pixels, each circuit shared by a plurality of pixels, in image sensing apparatus | |
| KR101696463B1 (en) | Solid-state imaging device, signal processing method thereof and image capturing apparatus | |
| CN102148943B (en) | Solid-state image sensor and imaging apparatus | |
| JPWO2013145765A1 (en) | Imaging unit, imaging apparatus, and imaging control program | |
| JP2017022624A (en) | Imaging device, driving method therefor, and imaging apparatus | |
| JP2014175832A (en) | Solid state image pickup device | |
| JP2022192062A (en) | Image sensor, camera module, mobile terminal and image collection method | |
| KR20150118214A (en) | Image sensing device | |
| CN207251823U (en) | Imaging device and imaging system | |
| CN118828238A (en) | Image sensor, image sensor control method, optoelectronic device and medium | |
| JP6508805B2 (en) | Imaging device and imaging device | |
| CN120238766A (en) | Image sensor, method for acquiring image, chip system and electronic device | |
| JP2009130574A (en) | Imaging apparatus and solid-state imaging device driving method |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |