技术领域technical field
本发明涉及摄像模块以及图像信号处理方法，特别是涉及省略以机械方式进行焦点调节的焦点调节机构，并且得到高分辨率的图像信号的技术。The present invention relates to a camera module and an image signal processing method, in particular to a technique for obtaining a high-resolution image signal by omitting a focus adjustment mechanism for mechanical focus adjustment.
背景技术Background technique
以往提出了一种摄像装置(专利文献1)，其通过在拍摄光学系统的光路中插入调制相位的光波面调制元件来对焦点深度进行放大，对由于焦点深度的放大而模糊的图像(大的点像)，实施基于具有复原处理参数的核(kernel)的解卷积处理，由此复原为高分辨率的图像(小的点像)。该摄像元件具有包括光波面调制元件的光学系统、对通过了光学系统的被摄体像进行拍摄的摄像元件、和对来自摄像元件的被摄体的图像数据实施规定的处理的图像处理装置，图像处理装置将选择的特定区域的分散图像复原作为没有分散的图像信号。Conventionally, an imaging device has been proposed (Patent Document 1), which amplifies the depth of focus by inserting an optical wavefront modulation element that modulates the phase in the optical path of the imaging optical system, and blurred images due to the magnification of the depth of focus (large Point image) is restored to a high-resolution image (small point image) by performing deconvolution processing with a kernel (kernel) having restoration processing parameters. The imaging element has an optical system including an optical wavefront modulation element, an imaging element that captures an image of a subject passing through the optical system, and an image processing device that performs predetermined processing on image data of the subject from the imaging element, The image processing device restores the scattered image of the selected specific area as an image signal without dispersion.
选择的特定区域例如在图像内预先决定。另外，特定区域也能构成为：从图像内提取特征部位，通过选择该部分来决定。The selected specific area is predetermined, for example, within the image. In addition, the specific area can be determined by extracting a feature part from the image and selecting the part.
光波面调制元件例如是相位板，其具有三维的曲面，被配置在物体侧透镜与成像透镜间，使基于成像透镜而向摄像元件的受光面成像的波面变形。或者，作为相位板的其他例子，还存在折射率变化的光学元件(例如折射率分布型波面调制透镜)，厚度、折射率根据向透镜表面的编码(coding)而变化的光学元件(例如波面调制混合透镜)等。The optical wavefront modulation element is, for example, a phase plate, which has a three-dimensional curved surface, is disposed between the object side lens and the imaging lens, and deforms the wavefront imaged on the light receiving surface of the imaging element by the imaging lens. Alternatively, as another example of a phase plate, there are optical elements whose refractive index changes (such as a refractive index distribution type wavefront modulation lens), and optical elements whose thickness and refractive index change according to coding (coding) on the lens surface (such as a wavefront modulation lens). mixing lens), etc.
专利文献2以及3表示摄像器件的瑕疵修正的一个例子。作为修正的方法的一个例子，存在如下所述的“回填”：针对各个缺陷单元，对在上下以及左右方向相邻的四个单元的像素信号进行简单平均，对缺陷单元的像素信号进行修正，即，将该缺陷像素信号置换为修正值。Patent Documents 2 and 3 show an example of defect correction of an imaging device. As an example of a correction method, there is "backfill" as follows: For each defective unit, the pixel signals of four adjacent units in the vertical and left-right directions are simply averaged, and the pixel signal of the defective unit is corrected, That is, the defective pixel signal is replaced with a correction value.
专利文献4表示适于静态图像或者运动图像拍摄的图像信号的读出方式的一个例子。作为适合于运动图像拍摄的信号的读出方式，例如存在色差线依次方式。色差线依次方式是在排列的像素之上配置补色滤光器，读出像素的行之中奇数行的像素的信号来作为A场的信号，读出偶数行的像素的信号来作为B场信号，针对1帧以A以及B这两个场信号读出。另一方面，在静态图像拍摄中，大多采用将拜耳排列了RGB的彩色滤光器配置在像素上，按每一行依次读出来读出所有像素的信号的方式。Patent Document 4 shows an example of a method for reading out image signals suitable for shooting still images or moving images. As a method of reading signals suitable for moving image shooting, there is, for example, a color difference line sequential method. The color-difference line sequential method is to arrange a complementary color filter on the arranged pixels, read out the signal of the pixels in the odd-numbered rows among the rows of pixels as the signal of the A field, and read the signal of the pixels in the even-numbered row as the signal of the B field , read out with two field signals of A and B for one frame. On the other hand, in still image shooting, a method of arranging RGB color filters in a Bayer arrangement on pixels and sequentially reading out signals of all pixels for each row is often used.
由于不清楚散焦量，所以不管有没有对焦，PSF(Point SpreadFunction)都具有相同展宽的特殊的光学系统被称作EDof(Extended Depthof Field，放大的焦点深度)(专利文献5～7)。Since the amount of defocus is not known, a special optical system in which PSF (Point Spread Function) has the same spread regardless of focus is called EDof (Extended Depthof Field, enlarged depth of focus) (Patent Documents 5 to 7).
专利文献1：特开2009-159535号公报Patent Document 1: JP-A-2009-159535
专利文献2：特开2000-184289号公报Patent Document 2: JP-A-2000-184289
专利文献3：特开2004-23683号公报Patent Document 3: JP-A-2004-23683
专利文献4：特开2010-171868号公报Patent Document 4: JP-A-2010-171868
专利文献5：特开2009-10944号公报Patent Document 5: JP-A-2009-10944
专利文献6：特开2009-187092号公报Patent Document 6: JP-A-2009-187092
专利文献7：特开2009-188676号公报Patent Document 7: JP-A-2009-188676
如专利文献2以及3所记载的那样，通常，摄像元件包含缺陷像素(瑕疵)。瑕疵包括：与摄像信号无关地始终输出固定的信号的白瑕疵以及黑瑕疵、输出与周边像素存在一定以上增益的不同信号的调制瑕疵、由读出电路的缺陷产生的线瑕疵等。As described in Patent Documents 2 and 3, generally, an imaging element includes defective pixels (defects). Defects include white and black defects in which a fixed signal is always output regardless of the imaging signal, modulation defects in which a signal having a gain different from that of peripheral pixels is output, and line defects caused by defects in the readout circuit.
这些之中，等级大的瑕疵需要无条件地修正。在由EDof光学系统得到的模糊图像的复原时，由于如专利文献1那样进行提高低电平信号(MTF)的修正处理，所以若不修正到与通常的瑕疵不同的微小瑕疵，则会对复原后的图像的画质造成影响。但是，在整个画面上修正微小等级的瑕疵在时间上是困难的。Among these, flaws with a high level need to be unconditionally corrected. In the restoration of the blurred image obtained by the EDof optical system, since the correction process of increasing the low-level signal (MTF) is performed as in Patent Document 1, unless a small defect different from a normal defect is not corrected, it will affect the restoration affect the image quality of the resulting image. However, it is time-consuming to correct small-scale artifacts across the entire screen.
发明内容Contents of the invention
本发明是鉴于上述状况而做出的发明，其目的在于提供一种仅限于对由焦点深度被放大的光学系统得到的图像的复原所需的区域进行详细的瑕疵修正的技术。The present invention was made in view of the above-mentioned circumstances, and an object of the present invention is to provide a technique for performing detailed defect correction only on an area required for restoration of an image obtained by an optical system with an enlarged depth of focus.
本发明的一个方式提供一种摄像模块，具备：透镜部，其通过规定的光学相位调制机构而被放大了焦点深度；摄像元件，其安装有在规定的成像面以矩阵状排列的各像素所对应的红、蓝以及绿的彩色滤光器，通过各像素对经由透镜部而在规定的成像面成像的光学像进行光电变换，从而能够输出与各像素对应的颜色的模拟图像信号；AD变换部，其将从摄像元件输出的每个像素的模拟图像信号变换为每个像素的数字图像信号并输出；修正部，其对规定区域的像素群所对应的数字图像信号的低等级的缺陷进行修正；和复原处理部，其实施从通过修正部修正了缺陷后的规定区域的像素群所对应的数字图像信号中，除去由光学相位调制机构引起的被摄体像的劣化的复原处理。One aspect of the present invention provides an imaging module including: a lens unit whose depth of focus is enlarged by a predetermined optical phase modulation mechanism; Corresponding red, blue, and green color filters, through each pixel, photoelectrically converts the optical image formed on the specified imaging surface through the lens unit, so that the analog image signal of the color corresponding to each pixel can be output; AD conversion a part that converts the analog image signal of each pixel output from the imaging element into a digital image signal of each pixel and outputs it; a correction; and a restoration processing unit that performs restoration processing for removing degradation of the subject image caused by the optical phase modulation mechanism from the digital image signal corresponding to the pixel group in the predetermined area after the defect has been corrected by the correction unit.
作为优选方式，修正部对规定区域的像素群以及规定区域以外的像素群所对应的数字图像信号的高等级的缺陷进行修正。Preferably, the correcting unit corrects high-level defects of the digital image signal corresponding to the pixel groups in the predetermined area and the pixel groups outside the predetermined area.
作为优选方式，还具备判断部，其判断是选择静态图像拍摄模式还是选择运动图像拍摄模式，在判断部判断选择静态图像拍摄模式的情况下，修正部对静态图像信号的读出像素群之中规定区域的像素群所对应的数字图像信号的低等级的缺陷进行修正，在判断部判断选择运动图像拍摄模式的情况下，修正部对运动图像信号的读出像素群的全部所对应的数字图像信号的高等级的缺陷进行修正。As a preferable form, it further includes a judging section that judges whether to select the still image shooting mode or the moving image shooting mode, and when the judging section judges that the still image shooting mode is selected, the correction section reads out the still image signal from among the pixel groups The low-level defect of the digital image signal corresponding to the pixel group of the predetermined area is corrected. High-level bugs in the signal are corrected.
本发明提供一种图像信号处理方法，其中，摄像模块具备：透镜部，其通过规定的光学相位调制机构而被放大了焦点深度；摄像元件，其安装有在规定的成像面以矩阵状排列的各像素所对应的红、蓝以及绿的彩色滤光器，通过各像素对经由透镜部而在规定的成像面成像的光学像进行光电变换，从而能够输出与各像素对应的颜色的模拟图像信号；和AD变换部，其将从摄像元件输出的每个像素的模拟图像信号变换为每个像素的数字图像信号并输出，摄像模块执行以下步骤：对规定区域的像素群所对应的数字图像信号的低等级的缺陷进行修正的步骤；和实施从修正了缺陷后的规定区域的像素群所对应的数字图像信号中，除去由光学相位调制机构引起的被摄体像的劣化的复原处理的步骤。The present invention provides an image signal processing method, wherein the camera module includes: a lens unit whose depth of focus is magnified by a predetermined optical phase modulation mechanism; The red, blue, and green color filters corresponding to each pixel can output an analog image signal of the color corresponding to each pixel by photoelectrically converting the optical image formed on a predetermined imaging plane through the lens unit by each pixel. and an AD conversion section, which converts the analog image signal of each pixel output from the imaging element into a digital image signal of each pixel and outputs it, and the camera module performs the following steps: digital image signals corresponding to the pixel groups in the specified area A step of correcting low-level defects; and a step of performing restoration processing for removing degradation of the subject image caused by the optical phase modulation mechanism from the digital image signal corresponding to the pixel group in the predetermined area after the defect has been corrected .
发明效果Invention effect
在本发明中，由于针对被复原处理的规定区域的图像信号，在进行了修正更微小的瑕疵的处理的基础上实施复原处理，所以能防止微小瑕疵因复原处理而明显化，并且与修正整个图像的微小瑕疵的情况相比，能缩短修正所需的时间来完成处理。In the present invention, since the restoration process is performed on the image signal of the predetermined area that has been restored, the restoration process is performed on the basis of the processing for correcting even finer flaws, so it is possible to prevent the fine flaws from becoming conspicuous due to the restoration processing, and it is consistent with the entire correction process. Compared with the case of minor defects in the image, the time required for correction can be shortened to complete the process.
附图说明Description of drawings
图1是本发明所涉及的摄像模块的框图；Fig. 1 is the block diagram of camera module involved in the present invention;
图2是表示透镜部的光学系统的一个例子的图；FIG. 2 is a diagram showing an example of an optical system of a lens unit;
图3是表示第1实施方式的复原处理的流程图；FIG. 3 is a flowchart showing restoration processing in the first embodiment;
图4是表示第2实施方式的复原处理的流程图；FIG. 4 is a flowchart showing restoration processing in the second embodiment;
图5是表示第3实施方式的复原处理的流程图；FIG. 5 is a flowchart showing restoration processing in the third embodiment;
图6是表示摄像装置的一个例子的框图。FIG. 6 is a block diagram showing an example of an imaging device.
图中：1...摄像模块，10...透镜部，10A...拍摄透镜，11...光学滤光器，12...摄像元件，14...AD变换部，20...复原处理块，21...复原处理块判断部，22...瑕疵表选择部，24...瑕疵修正处理部，25...图像复原处理部。In the figure: 1...camera module, 10...lens unit, 10A...shooting lens, 11...optical filter, 12...camera element, 14...AD conversion unit, 20. .. Restoration processing block, 21... Restoration processing block judging part, 22... Defect table selection part, 24... Defect correction processing part, 25... Image restoration processing part.
具体实施方式detailed description
[第1实施方式][the first embodiment]
图1是表示本发明的摄像模块1的第1实施方式的框图。FIG. 1 is a block diagram showing a first embodiment of a camera module 1 of the present invention.
如图1所示，摄像模块1由透镜部10、摄像元件12、AD变换部14、复原处理块20构成。As shown in FIG. 1 , the imaging module 1 is composed of a lens unit 10 , an imaging element 12 , an AD conversion unit 14 , and a restoration processing block 20 .
图2是表示透镜部10的光学系统的一个例子的图。透镜部10如图2所示，由单焦点固定的拍摄透镜10A、和在瞳孔位置插入的光学滤光器11构成。光学滤光器11是调制相位的部件，使拍摄透镜EDoF化，来得到被放大的焦点深度(Extended Depth OfFocus：EDoF)。FIG. 2 is a diagram showing an example of an optical system of the lens unit 10 . As shown in FIG. 2 , the lens unit 10 is composed of a single focus fixed imaging lens 10A and an optical filter 11 inserted at the pupil position. The optical filter 11 is a component for modulating the phase, and makes the imaging lens EDoF to obtain an enlarged depth of focus (Extended Depth Of Focus: EDoF).
此外，在光学滤光器11的附近，配置了未图示的光圈。另外，光学滤光器11可以是1个，也可以是组合多个而得到的部件。另外，光学滤光器11只不过是光学相位调制机构的一个例子，也可以采用其他的部件，例如可以采用专利文献1那样的各种光波面调制元件。In addition, a not-shown diaphragm is disposed near the optical filter 11 . In addition, one optical filter 11 may be used, or a combination of a plurality may be used. In addition, the optical filter 11 is only an example of an optical phase modulation mechanism, and other components may be used, for example, various optical wavefront modulation elements as described in Patent Document 1 may be used.
该透镜部10能够省略以机械方式进行焦点调节的焦点调节机构，从而能实现小型化，适合搭载于带照相功能的移动电话、便携信息终端。The lens unit 10 can omit a focus adjustment mechanism that mechanically adjusts the focus, thereby achieving miniaturization, and is suitable for mounting on mobile phones and portable information terminals with a camera function.
透过了被EDoF化的透镜部10的光学像被成像于摄像元件12，在此被变换为电信号。The optical image transmitted through the EDoF-enabled lens unit 10 is formed on the imaging element 12, where it is converted into an electrical signal.
摄像元件12是彩色摄像元件，按照每个像素将红(R)、绿(G)、蓝(B)3原色的原色滤光器以规定的图案排列成矩阵状(拜耳排列、G条纹R/G完美方格图案(perfectly checkered pattern)，蜂窝排列等)，摄像元件12由C-MOS(complementary metal-oxide semiconductor)图像传感器或者CCD(charge-coupled device)图像传感器构成。经由透镜部10而入射到摄像元件12的受光面的光学像由在该受光面排列的各光电二极管被变换为与入射光量对应的量的信号电荷。然后，在各光电二极管中积蓄的R、G、B的信号电荷作为每个像素的电压信号(图像信号)被依次输出。The imaging element 12 is a color imaging element, and the primary color filters of the three primary colors of red (R), green (G), and blue (B) are arranged in a matrix in a predetermined pattern for each pixel (Bayer arrangement, G stripe R/ (perfectly checked pattern, honeycomb arrangement, etc.), the imaging element 12 is composed of a C-MOS (complementary metal-oxide semiconductor) image sensor or a CCD (charge-coupled device) image sensor. The optical image incident on the light-receiving surface of the imaging element 12 via the lens unit 10 is converted into signal charges corresponding to the amount of incident light by the photodiodes arrayed on the light-receiving surface. Then, the signal charges of R, G, and B accumulated in each photodiode are sequentially output as a voltage signal (image signal) for each pixel.
AD变换部14将从摄像元件12按每个像素输出的模拟的RGB图像信号变换为数字的RGB图像信号。由AD变换部14变换为数字的图像信号后的数字图像信号被施加给复原处理块20。从复原处理块20输出的图像信号被施加给去马赛克(demosaicing)信号处理部26。去马赛克信号处理部26进行去马赛克处理，补充从马赛克状的图像数据缺失的颜色成分，生成RGB的各成分齐备的图像信号即RAW数据。The AD converter 14 converts the analog RGB image signals output from the imaging element 12 for each pixel into digital RGB image signals. The digital image signal converted into a digital image signal by the AD converter 14 is applied to the restoration processing block 20 . The image signal output from the restoration processing block 20 is applied to a demosaicing signal processing section 26 . The demosaic signal processing unit 26 performs demosaic processing to supplement color components missing from the mosaic-like image data, and generates RAW data which is an image signal having all RGB components.
复原处理块20包括复原处理块判断部21、瑕疵表选择部22、瑕疵数据存储部23、瑕疵修正处理部24、图像复原处理部25。The restoration processing block 20 includes a restoration processing block judgment unit 21 , a defect table selection unit 22 , a defect data storage unit 23 , a defect correction processing unit 24 , and an image restoration processing unit 25 .
复原处理块判断部21判断从AD变换部14输入的图像信号是从复原处理的对象的块(像素群)输出的图像信号，还是从复原处理的对象以外的块输出的图像信号。该判断与专利文献1公开的复原处理的所需位置的选择同样地进行。例如，将从存在于画面中央等预先决定的区域内的块、存在于从图像内提取出的脸部等特征部位的块的像素读出的图像信号，判断为从复原处理的对象的块输出的图像信号。The restoration processing block judgment unit 21 judges whether the image signal input from the AD conversion unit 14 is an image signal output from a block (pixel group) to be restored, or an image signal output from a block other than the restoration to be processed. This determination is performed in the same manner as the selection of the desired position for the restoration process disclosed in Patent Document 1. FIG. For example, an image signal read from pixels of a block existing in a predetermined area such as the center of the screen or a block existing in a feature part such as a face extracted from an image is judged to be output from a block to be restored. image signal.
块是覆盖静止图像的像素群整体的N个部分像素群区域。块的尺寸与复原处理的单位一致。例如，若解卷积滤波器的核尺寸为7×7，则1块的单位成为7×7像素。A block is N partial pixel group areas covering the entire pixel group of a still image. The size of the block corresponds to the unit of restoration processing. For example, if the kernel size of the deconvolution filter is 7×7, the unit of one block is 7×7 pixels.
瑕疵表选择部22基于复原处理块判断部21的判断结果，从存储于瑕疵数据存储部23的表之中，选择用于瑕疵修正的表。瑕疵数据存储部23存储有第1瑕疵表23a、第2瑕疵表23b、第3瑕疵表23c。The defect table selection unit 22 selects a table for defect correction from among the tables stored in the defect data storage unit 23 based on the determination result of the restoration processing block determination unit 21 . The defect data storage unit 23 stores a first defect table 23a, a second defect table 23b, and a third defect table 23c.
第1瑕疵表23a存储当进行运动图像拍摄时读出图像信号的像素所对应的第1等级的瑕疵(像素缺陷)的坐标。第1等级的瑕疵是醒目到需要修正的程度的瑕疵。第2瑕疵表23b存储当进行静态图像拍摄时读出图像信号的像素所对应的第1等级的瑕疵的坐标。第1瑕疵表23a与第2瑕疵表23b分别表示静态图像信号以及运动图像信号的读出像素中的瑕疵的坐标，但在存储第1等级的瑕疵的坐标这一点上是共通的。第3瑕疵表23c存储当进行静态图像拍摄时读出图像信号的像素所对应的第2等级的瑕疵的坐标。第2等级的瑕疵是比第1等级的瑕疵更微小的瑕疵。瑕疵的等级表示视觉上的瑕疵的程度，与瑕疵的种类无关。作为瑕疵的种类，包括白瑕疵、黑瑕疵、调制瑕疵、线瑕疵等。The first defect table 23 a stores coordinates of first-level defects (pixel defects) corresponding to pixels from which an image signal is read out when a moving image is captured. Class 1 blemishes are blemishes that are conspicuous enough to require correction. The second blemish table 23b stores the coordinates of the first-level blemish corresponding to the pixel from which the image signal is read out when the still image is captured. The first blemish table 23a and the second blemish table 23b respectively indicate the coordinates of blemishes in the readout pixels of the still image signal and the moving image signal, but are common in that they store the coordinates of blemishes of the first level. The third blemish table 23 c stores the coordinates of blemishes of the second level corresponding to pixels from which an image signal is read out when a still image is captured. The second-grade flaws are finer than the first-grade flaws. The level of flaws indicates the degree of visual flaws, regardless of the kind of flaws. The types of flaws include white flaws, black flaws, modulation flaws, line flaws, and the like.
另外，第1等级的瑕疵与第2等级的瑕疵的定义方法没有特别限定。例如，将摄像元件12的像素(群)的瑕疵从尺寸大的开始排序，将从前端最大尺寸的瑕疵开始按顺序变小的规定个数的瑕疵(例如从前端开始到第100的瑕疵)定义为第1等级的瑕疵，将比第1等级的瑕疵中最小的瑕疵进一步按规定顺序的尺寸小的瑕疵(例如从前端开始的第101到第200的瑕疵)定义为第2等级的瑕疵。或者，也可以定义为第1等级的瑕疵以外全部为第2等级的瑕疵。在拍摄图像的记录时压缩图像数据的情况下，也可以根据记录图像的压缩率来决定修正怎样程度的微小瑕疵。例如，若是FINE模式的静态图像记录，则将从前端开始的第101到第300为止的瑕疵定义为第2等级的瑕疵，若是Normal模式的静态图像记录，则将从前端开始的第101到第200为止的瑕疵定义为第2等级的瑕疵。In addition, the definition method of the flaw of the 1st rank and the flaw of the 2nd rank is not specifically limited. For example, the defects of the pixels (groups) of the image pickup device 12 are sorted from the largest in size, and a predetermined number of defects (for example, defects from the leading end to the 100th defect) that decrease in order from the largest defect at the tip are defined. The flaws of the first grade are defined as the flaws of the second grade that are smaller in size than the smallest flaws of the first grade flaws in a predetermined order (for example, the 101st to 200th flaws from the front end). Alternatively, it may be defined as flaws other than first-grade flaws that are second-grade flaws. In the case of compressing image data at the time of recording of captured images, it is also possible to determine the extent to which minute flaws are corrected according to the compression rate of recorded images. For example, for still image recording in FINE mode, the 101st to 300th blemishes from the top are defined as second-level blemishes, and for still image recording in Normal mode, the 101st to 300th blemishes from the top are defined Flaws up to 200 are defined as second-grade flaws.
瑕疵修正处理部24从瑕疵表选择部22选择的表中取得瑕疵的坐标信息，基于该瑕疵的坐标信息，进行图像信号的瑕疵修正。瑕疵修正的方法可以采用公知的方法。例如，可以如专利文献2以及3所记载的那样，通过对存在于缺陷像素周边的像素的“回填”来进行瑕疵修正。The defect correction processing unit 24 acquires the coordinate information of the defect from the table selected by the defect table selection unit 22 , and performs defect correction of the image signal based on the coordinate information of the defect. As a method of defect correction, a known method can be used. For example, as described in Patent Documents 2 and 3, defect correction can be performed by "backfilling" pixels existing around a defective pixel.
图像复原处理部25针对瑕疵修正后的图像信号实施复原处理，将需要复原处理的块的分散图像信号复原为没有分散的图像信号。例如，如专利文献1所记载的那样，在复原处理中进行如下处理：提升1次图像的空间频率中的调制传递函数(MTF：Modulation Transfer Function)，进行接近没有采用相位板的情况下的MTF特性的修正，形成高精细的最终图像。The image restoration processing unit 25 performs restoration processing on the image signal after the defect correction, and restores the dispersed image signal of the block requiring restoration processing to an image signal without dispersion. For example, as described in Patent Document 1, in the restoration process, the modulation transfer function (MTF: Modulation Transfer Function) in the spatial frequency of the primary image is raised to approach the MTF when the phase plate is not used. Correction of characteristics to form a high-definition final image.
接下来，对复原处理块20进行的复原处理进行说明。图3是表示第1实施方式的复原处理块20进行的复原处理的流程图。该处理对应于静态图像拍摄模式的选择而开始。Next, restoration processing performed by the restoration processing block 20 will be described. FIG. 3 is a flowchart showing restoration processing performed by the restoration processing block 20 in the first embodiment. This process starts corresponding to selection of the still image shooting mode.
在S1中，从AD变换部14，对复原处理块判断部21与瑕疵修正处理部24施加数字图像信号。该图像信号分别由静态图像的读出像素、例如偶数以及奇数行的像素输出。In S1 , a digital image signal is applied from the AD conversion unit 14 to the restoration processing block determination unit 21 and the blemish correction processing unit 24 . The image signals are respectively output from readout pixels of the still image, for example, pixels of even-numbered and odd-numbered rows.
S2中，复原处理块判断部21根据从AD变换部14输入的图像信号，判断复原处理的对象的块与复原处理的对象以外的块。对块赋予连续编号n(n＝1～N)。In S2 , the restoration processing block judgment unit 21 judges a block subject to restoration processing and a block not subject to restoration processing based on the image signal input from the AD conversion unit 14 . Serial numbers n (n=1 to N) are assigned to the blocks.
在S3中，瑕疵表选择部22设定连续编号n＝1。In S3, the defect table selection unit 22 sets the serial number n=1.
在S4中，瑕疵表选择部22判断被赋予了连续编号n的块是否被判断为进行复原处理的块。在“是”的情况下进入S5，在“否”的情况下进入S6。In S4, the defect table selection unit 22 judges whether or not the block assigned the serial number n is judged as a block to be restored. In the case of "Yes", it goes to S5, and in the case of "No", it goes to S6.
在S5中，瑕疵表选择部22选择第3瑕疵表23c。瑕疵修正处理部24从瑕疵表选择部22选择的表(在此为第3瑕疵表23c)取得瑕疵的坐标信息。然后，基于该瑕疵的坐标信息，仅对来自复原对象块的图像信号进行第2等级的瑕疵的修正。进一步，在第2等级的瑕疵的修正前后，瑕疵表选择部22选择第2瑕疵表23b。瑕疵修正处理部24从瑕疵表选择部22选择的表(在此为第2瑕疵表23b)取得瑕疵的坐标信息。然后，基于该瑕疵的坐标信息，仅对来自复原对象块的图像信号进行第1等级的瑕疵的修正。由此，针对来自复原对象块的图像信号，既修正了大的第1等级的瑕疵，又修正了微小的第2等级的瑕疵。In S5, the defect table selection part 22 selects the 3rd defect table 23c. The flaw correction processing part 24 acquires the coordinate information of a flaw from the table selected by the flaw table selection part 22 (here, the 3rd flaw table 23c). Then, based on the coordinate information of the defect, only the image signal from the block to be restored is corrected for the defect of the second level. Furthermore, the defect table selection part 22 selects the 2nd defect table 23b before and after correction of the defect of the 2nd level. The flaw correction processing part 24 acquires the coordinate information of a flaw from the table selected by the flaw table selection part 22 (here, the 2nd flaw table 23b). Then, based on the coordinate information of the defect, only the image signal from the block to be restored is corrected for the first-level defect. As a result, both the large first-level flaws and the slight second-level flaws are corrected for the image signal from the block to be restored.
在S6中，瑕疵表选择部22将n增加1。In S6, the defect table selection unit 22 increments n by one.
在S7中，瑕疵表选择部22判断是否是n＝N。在“是”的情况下进入S8，在“否”的情况下返回S4。In S7, the defect table selection unit 22 judges whether or not n=N. In the case of "Yes", it progresses to S8, and in the case of "No", it returns to S4.
在S8中，图像复原处理部25仅对从复原处理的对象的块输出的图像信号(也包括瑕疵修正结束的图像信号)实施复原处理，针对复原对象块生成没有模糊的静止图像信号。另外，图像复原处理部25将来自复原处理的对象以外的块的图像信号保持原样地输入到去马赛克信号处理部26。In S8 , the image restoration processing unit 25 performs restoration processing only on the image signal output from the block to be restored (including the image signal after blemish correction), and generates a still image signal without blurring for the block to be restored. Also, the image restoration processing unit 25 inputs the image signals from blocks other than the restoration processing target to the demosaic signal processing unit 26 as they are.
在S9中，来自图像复原处理部25的图像信号(包括复原后的静止图像信号与复原的对象以外的静态图像信号双方)被输入到去马赛克信号处理部26，实施去马赛克处理，输出RAW数据。In S9, the image signal (including both the restored still image signal and the still image signal not to be restored) from the image restoration processing unit 25 is input to the demosaic signal processing unit 26, and the demosaic processing is performed to output RAW data. .
通过以上的处理，针对需要复原的块，在进行了修正更微小的瑕疵的处理的基础上实施复原处理，所以能防止微小的瑕疵因复原处理而明显化，并且与修正整个图像的微小瑕疵的情况相比，能缩短修正所需的时间来完成处理。Through the above processing, for the block that needs to be restored, the restoration process is performed on the basis of the processing of correcting the finer flaws, so it is possible to prevent the fine flaws from becoming conspicuous due to the restoration process, and to correct the fine flaws of the entire image. Compared with the case, the time required for the correction to complete the processing can be shortened.
[第2实施方式][the second embodiment]
图4是表示第2实施方式的复原处理块40进行的复原处理的流程图。关于与第1实施方式相同的处理步骤，省略其说明。FIG. 4 is a flowchart showing restoration processing performed by the restoration processing block 40 in the second embodiment. Descriptions of the same processing steps as those in the first embodiment are omitted.
S10～S12分别与S1～3相同。S10-S12 are the same as S1-3 respectively.
S13与S4的判断相同。其中，“是”的情况下进入S14，“否”的情况下进入S15。The determination of S13 is the same as that of S4. However, it progresses to S14 in the case of "Yes", and progresses to S15 in the case of "No".
S14与S5相同。The S14 is the same as the S5.
在S15中，瑕疵表选择部22选择第2瑕疵表23b。瑕疵修正处理部24从瑕疵表选择部22选择的表(在此为第2瑕疵表23b)取得瑕疵的坐标信息。然后，基于该瑕疵的坐标信息，进行图像信号整体的第1等级的瑕疵的修正。In S15, the defect table selection unit 22 selects the second defect table 23b. The flaw correction processing part 24 acquires the coordinate information of a flaw from the table selected by the flaw table selection part 22 (here, the 2nd flaw table 23b). Then, based on the coordinate information of the defect, correction of the first-level defect of the entire image signal is performed.
S16～S19分别与S6～S9相同。其中，S17中为“是”的情况下进入S18，为“否”的情况下返回S13。S16 to S19 are the same as S6 to S9, respectively. However, if it is "YES" in S17, it will progress to S18, and if it is "No", it will return to S13.
通过以上的处理，由于针对需要复原的块，在进行了修正更微小的瑕疵的处理的基础上实施复原处理，所以能防止微小的瑕疵因复原处理而明显化，并且与修正整个图像的微小瑕疵的情况相比，能缩短修正所需的时间来完成处理。另外，针对不需要复原的块，仅进行修正显眼的瑕疵的处理。Through the above processing, for the block that needs to be restored, the restoration process is performed on the basis of the processing of correcting the finer flaws, so it is possible to prevent the fine flaws from becoming conspicuous due to the restoration process, and it is consistent with the correction of the fine flaws of the entire image. The time required for correction to complete processing can be shortened compared to the case of Also, for blocks that do not need to be restored, only conspicuous defects are corrected.
[第3实施方式][the third embodiment]
图5是表示第3实施方式的复原处理块40进行的复原处理的流程图。针对与第1至第2实施方式与相同的处理步骤，省略其说明。FIG. 5 is a flowchart showing restoration processing performed by the restoration processing block 40 in the third embodiment. Descriptions of the same processing steps as those in the first to second embodiments are omitted.
在S20中，复原处理块判断部21判断是否指示了运动图像模式或者静态图像模式的某一个来作为拍摄模式的用户选择。在运动图像模式的情况下进入S21，在静态图像模式的情况下进入S26。In S20 , the restoration processing block judging unit 21 judges whether one of the moving image mode and the still image mode has been designated as the user's selection of the shooting mode. It proceeds to S21 in the case of moving image mode, and proceeds to S26 in the case of still image mode.
在S21中，从AD变换部14，对复原处理块判断部21与瑕疵修正处理部24施加数字图像信号。该图像信号从运动图像的读出像素，例如从奇数行的像素定期地输出。In S21 , the digital image signal is applied from the AD conversion unit 14 to the restoration processing block determination unit 21 and the blemish correction processing unit 24 . This image signal is periodically output from readout pixels of a moving image, for example, from pixels of odd-numbered rows.
在S22中，瑕疵表选择部22选择第1瑕疵表23a。In S22, the defect table selection part 22 selects the 1st defect table 23a.
在S23中，瑕疵修正处理部24从瑕疵表选择部22选择的表(在此为第1瑕疵表23a)取得瑕疵的坐标信息。然后，基于该瑕疵的坐标信息，进行运动图像信号的第1等级的瑕疵修正。该瑕疵修正针对全部图像信号而进行。In S23, the flaw correction processing part 24 acquires the coordinate information of a flaw from the table selected by the flaw table selection part 22 (here, the 1st flaw table 23a). Then, based on the coordinate information of the defect, the first-level defect correction of the video signal is performed. This defect correction is performed on all image signals.
在S24中，图像复原处理部25对运动图像的读出像素所对应的全部的图像信号(也包括瑕疵修正结束的图像信号)实施复原处理，生成没有模糊的运动图像帧的图像信号。In S24 , the image restoration processing unit 25 performs restoration processing on all image signals (including image signals whose blemishes have been corrected) corresponding to the read pixels of the moving image to generate image signals of moving image frames without blurring.
在S25中，复原后的运动图像帧信号被输入到去马赛克信号处理部26，被实施去马赛克处理。该去马赛克处理后的运动图像帧信号能被作为实时取景图像而显示，或者能作为运动图像数据被记录。In S25 , the restored video frame signal is input to the demosaic signal processing unit 26 and subjected to demosaic processing. The demosaiced moving image frame signal can be displayed as a live view image, or can be recorded as moving image data.
在S26执行第1实施方式或者第2实施方式的某一复原处理。执行哪一个处理是任意的，可以通过用户设定等来选择。In S26, any restoration process of the first embodiment or the second embodiment is executed. Which process to execute is arbitrary and can be selected by user settings or the like.
在S27中，复原处理块判断部21判断用户是否输入了结束拍摄模式的指示。在“是”的情况下结束处理，在“否”的情况下返回S20。In S27, the restoration processing block determination unit 21 determines whether or not the user has input an instruction to end the shooting mode. When it is "Yes", the processing is terminated, and when it is "No", it returns to S20.
[第4实施方式][the fourth embodiment]
图6是表示使用了摄像模块1的摄像装置的一个例子的框图。FIG. 6 is a block diagram showing an example of an imaging device using the imaging module 1 .
图6所示的摄像装置100是被装入了图1所示的摄像模块1的装置，除了摄像模块以外，具有与通常的数字摄像机等相同的结构。The imaging device 100 shown in FIG. 6 is a device in which the imaging module 1 shown in FIG. 1 is incorporated, and has the same configuration as a general digital video camera or the like except for the imaging module.
中央处理装置(CPU：central processing unit)102是按照来自操作部104的操作输入以及规定的程序，统一控制装置整体的部分，也作为实施自动曝光(AE)运算、白平衡(WB)调整运算等各种运算的运算机构发挥作用。The central processing unit (CPU: central processing unit) 102 is a part that collectively controls the entire device in accordance with the operation input from the operation unit 104 and a predetermined program, and is also used for performing automatic exposure (AE) calculations, white balance (WB) adjustment calculations, etc. The arithmetic mechanism of various calculations plays a role.
在CPU102上，经由总线103以及存储器·接口(存储器I/F)106连接了RAM(Random Access Memory)108以及ROM(Read Only Memory)110。RAM108被用作程序的展开区域以及CPU102的运算作业用区域，并且被用作图像数据的临时存储区域。ROM110中，存储有CPU102执行的程序以及控制所需的各种数据、与摄像动作相关的各种常量/信息等。CPU102控制上述第1～3实施方式的复原处理的流程的程序被存储在ROM110中。A RAM (Random Access Memory) 108 and a ROM (Read Only Memory) 110 are connected to the CPU 102 via a bus 103 and a memory/interface (memory I/F) 106 . The RAM 108 is used as a program development area and an area for calculation work of the CPU 102 , and is also used as a temporary storage area for image data. The ROM 110 stores programs executed by the CPU 102 , various data necessary for control, various constants/information related to imaging operations, and the like. Programs for the CPU 102 to control the flow of restoration processing in the first to third embodiments described above are stored in the ROM 110 .
摄像模块1根据来自CPU102的指令进行拍摄动作等，输出RGB的RAW数据。该RAW数据经由总线103以及存储器I/F106临时保存在RAM108中。The imaging module 1 performs an imaging operation and the like in accordance with an instruction from the CPU 102 , and outputs RGB RAW data. This RAW data is temporarily stored in RAM 108 via bus 103 and memory I/F 106 .
保存在RAM108中的RGB的RAW数据被输入到图像处理块30中，被实施各种图像处理。图像处理包括伽马修正、对比度调整处理、锐度调整处理、噪声减少处理、颜色修正处理、YC变换等各种处理。The RGB RAW data stored in the RAM 108 is input to the image processing block 30 and subjected to various image processing. Image processing includes various processing such as gamma correction, contrast adjustment processing, sharpness adjustment processing, noise reduction processing, color correction processing, and YC conversion.
另外，在选择了RAW数据记录的情况下，RAW数据以RAW文件的格式，经由外部存储器·接口(外部存储器I/F)114被记录于存储卡116。Also, when RAW data recording is selected, the RAW data is recorded in the memory card 116 via the external memory/interface (external memory I/F) 114 in the format of a RAW file.
操作部104包括快门按钮、选择拍摄模式与再生模式的模式选择开关、使菜单画面显示于显示部(LCD：liquid crystal display)118的菜单按钮、用于从菜单画面选择所希望的项目的多功能方向键等。来自操作部104的输出信号经由总线103被输入到CPU102，CPU102基于来自操作部104的输入信号，实施拍摄、再生等适当的处理。The operation unit 104 includes a shutter button, a mode selection switch for selecting a shooting mode and a playback mode, a menu button for displaying a menu screen on a display unit (LCD: liquid crystal display) 118, and a multi-function button for selecting a desired item from the menu screen. Arrow keys etc. An output signal from the operation unit 104 is input to the CPU 102 via the bus 103 , and the CPU 102 performs appropriate processing such as imaging and reproduction based on the input signal from the operation unit 104 .
摄像装置100包括用于向被摄体照射闪光的闪光装置120，闪光装置120根据来自CPU102的发光指令，从充电部122接受电源的提供，来照射闪光。The imaging device 100 includes a flash device 120 for irradiating a subject with flash light, and the flash device 120 receives power from a charging unit 122 in accordance with a light emission instruction from the CPU 102 to emit flash light.
在图像处理块30中处理的图像数据(静态图像或者运动图像)被提供给图像记录部31，在此，按照规定的压缩格式(例如，JPEG(JointPhotographic Experts Group)方式、MPEG(Moving Picture Experts Group)方式)压缩该图像数据。被压缩后的图像数据以图像文件(例如JPEG文件、MPEG文件)的格式，经由外部存储器I/F114被记录于存储卡116中。当在图像记录部31中记录拍摄图像时压缩图像数据的情况下，也可以根据记录图像的压缩率来决定修正怎样程度的微小瑕疵。The image data (still image or moving image) processed in the image processing block 30 is provided to the image recording unit 31, where, according to a prescribed compression format (for example, JPEG (Joint Photographic Experts Group) method, MPEG (Moving Picture Experts Group) ) way) to compress the image data. The compressed image data is recorded in the memory card 116 via the external memory I/F 114 in the format of an image file (eg, JPEG file, MPEG file). In the case of compressing image data when recording captured images in the image recording unit 31 , it is also possible to determine the extent to which minute flaws are corrected according to the compression ratio of the recorded images.
另外，在LCD118LCD上，根据经由接口126而施加的运动图像信号，在摄像准备中显示影像(实时取景图像)，另外，在再生模式时，读出记录于存储卡116的JPEG文件、MPEG文件、RAW文件等图像文件，显示图像。另外，存储于JPEG文件、MPEG文件中的压缩图像数据通过压缩扩展处理电路124进行扩展处理，之后输出到LCD118，存储于RAW文件中的RAW数据通过图像处理块30进行了RAW显影后被输出到LCD118。In addition, on the LCD 118LCD, video images (live view images) are displayed during imaging preparations based on moving image signals applied via the interface 126, and JPEG files, MPEG files, Image files such as RAW files, display images. In addition, compressed image data stored in a JPEG file or an MPEG file is subjected to expansion processing by the compression/expansion processing circuit 124, and then output to the LCD 118, and RAW data stored in a RAW file is subjected to RAW development by the image processing block 30, and then output to LCD118.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2010-189115 | 2010-08-26 | ||
| JP2010189115AJP5513310B2 (en) | 2010-08-26 | 2010-08-26 | Imaging module, image signal processing method, restoration processing apparatus and method |
| Publication Number | Publication Date |
|---|---|
| CN102387294A CN102387294A (en) | 2012-03-21 |
| CN102387294Btrue CN102387294B (en) | 2016-09-21 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201110245255.0AExpired - Fee RelatedCN102387294B (en) | 2010-08-26 | 2011-08-24 | Photographing module and image-signal processing method |
| Country | Link |
|---|---|
| JP (1) | JP5513310B2 (en) |
| CN (1) | CN102387294B (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2014103852A1 (en)* | 2012-12-28 | 2014-07-03 | 富士フイルム株式会社 | Pixel correction method and image capture device |
| US9726876B2 (en)* | 2013-11-27 | 2017-08-08 | Mitutoyo Corporation | Machine vision inspection system and method for obtaining an image with an extended depth of field |
| WO2017181381A1 (en)* | 2016-04-21 | 2017-10-26 | 华为技术有限公司 | Method and photographing device for acquiring depth information |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6819359B1 (en)* | 1999-02-03 | 2004-11-16 | Fuji Photo Film Co., Ltd. | Method and apparatus for controlling the processing of signals containing defective pixels in accordance with imaging operation mode |
| CN101316322A (en)* | 2007-05-31 | 2008-12-03 | 佳能株式会社 | Image processing apparatus and image processing method |
| JP2009159535A (en)* | 2007-12-27 | 2009-07-16 | Kyocera Corp | Imaging apparatus and image processing method |
| CN101490694A (en)* | 2005-11-10 | 2009-07-22 | 蒂-布勒技术有限公司 | Image enhancement in the mosaic domain |
| CN101505373A (en)* | 2008-02-05 | 2009-08-12 | 富士胶片株式会社 | Image capturing apparatus, image capturing method and computer readable medium |
| CN101790040A (en)* | 2010-03-05 | 2010-07-28 | 圆展科技股份有限公司 | real camera |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP4508740B2 (en)* | 2004-06-22 | 2010-07-21 | キヤノン株式会社 | Image processing device |
| JPWO2009119838A1 (en)* | 2008-03-27 | 2011-07-28 | 京セラ株式会社 | Optical system, imaging device, and information code reader |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6819359B1 (en)* | 1999-02-03 | 2004-11-16 | Fuji Photo Film Co., Ltd. | Method and apparatus for controlling the processing of signals containing defective pixels in accordance with imaging operation mode |
| CN101490694A (en)* | 2005-11-10 | 2009-07-22 | 蒂-布勒技术有限公司 | Image enhancement in the mosaic domain |
| CN101316322A (en)* | 2007-05-31 | 2008-12-03 | 佳能株式会社 | Image processing apparatus and image processing method |
| JP2009159535A (en)* | 2007-12-27 | 2009-07-16 | Kyocera Corp | Imaging apparatus and image processing method |
| CN101505373A (en)* | 2008-02-05 | 2009-08-12 | 富士胶片株式会社 | Image capturing apparatus, image capturing method and computer readable medium |
| CN101790040A (en)* | 2010-03-05 | 2010-07-28 | 圆展科技股份有限公司 | real camera |
| Publication number | Publication date |
|---|---|
| JP5513310B2 (en) | 2014-06-04 |
| CN102387294A (en) | 2012-03-21 |
| JP2012049760A (en) | 2012-03-08 |
| Publication | Publication Date | Title |
|---|---|---|
| US8593509B2 (en) | Three-dimensional imaging device and viewpoint image restoration method | |
| JP5816015B2 (en) | Solid-state imaging device and camera module | |
| US8350948B2 (en) | Image device which bypasses blurring restoration during a through image | |
| JP6141587B2 (en) | Lens shading correction factor calculation method and lens shading correction method and apparatus using the same | |
| CN104969545B (en) | Image processing device, imaging device, image processing method, and program | |
| JP2009153013A (en) | Imaging apparatus, color noise reduction method, and color noise reduction program | |
| JP2004343519A (en) | Imaging device and image processing device | |
| JPWO2013046828A1 (en) | Image processing apparatus, method, program, and imaging apparatus | |
| JP5514042B2 (en) | Imaging module, image signal processing method, and imaging apparatus | |
| JP5159715B2 (en) | Image processing device | |
| CN102387294B (en) | Photographing module and image-signal processing method | |
| JP4817529B2 (en) | Imaging apparatus and image processing method | |
| JP5095434B2 (en) | Imaging apparatus and image processing apparatus | |
| US8736722B2 (en) | Enhanced image capture sharpening | |
| JP5299159B2 (en) | Imaging apparatus and program | |
| JP2012216947A (en) | Image pickup device and image processing method | |
| US8804014B2 (en) | Imaging device for reducing color moire | |
| JP4449692B2 (en) | Electronic camera | |
| JP4687750B2 (en) | Digital camera and image signal processing storage medium | |
| JP5333163B2 (en) | Imaging device | |
| JP2013055459A (en) | Imaging device, image processing device, and program | |
| JP5515653B2 (en) | Imaging apparatus and image processing apparatus | |
| JP5513311B2 (en) | Imaging module, image processing apparatus, and image signal processing method | |
| WO2012165087A1 (en) | Image capture device and ghosting correction method | |
| WO2022044915A1 (en) | Image processing device, imaging device, image processing method, and image processing program |
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | Granted publication date:20160921 | |
| CF01 | Termination of patent right due to non-payment of annual fee |