Detailed Description
The following description is of the best mode for carrying out the invention and is intended to illustrate the general spirit of the invention and not to limit the invention. Reference must be made to the following claims for their true scope of the invention.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of further features, integers, steps, operations, elements, components, and/or groups thereof.
Use of the terms "first," "second," "third," and the like in the claims is used to modify a claim element without indicating a priority, precedence, or order between elements, or the order in which a method step is performed, but is intended to distinguish one element from another element having a same name.
FIG. 1 is a block diagram of a computing device according to an embodiment of the present invention. The system architecture can be implemented in desktop computers, notebook computers, tablet computers, mobile phones, digital cameras, digital video recorders, etc., and at least includes a processing unit 110. The processing unit 110 may be implemented in numerous ways, such as in dedicated hardware circuitry or in general purpose hardware (e.g., a single processor, multiple processors with parallel processing capability, a graphics processor, or other processor with computing capability), and when executing hardware (hardware), firmware (firmware), or software (software) instructions, provides the functionality described hereinafter. The processing unit 110 may be integrated in an Image Signal Processor (ISP), and may control the camera module 190 through the camera module controller 170 to capture an Image. The camera module 190 may include an image sensor, such as a complementary metal-oxide-semiconductor (CMOS) sensor, a charge-coupled device (CCD) sensor, etc., for sensing an image formed by the intensities of red, green, and blue light, and read electronics for collecting the sensed data from the image sensor. The volatile Memory 150, such as a Dynamic Random Access Memory (DRAM), is used for storing data required in the execution process, such as variables, data tables (data tables), and the like.
FIG. 2 is a flowchart illustrating an image correction method according to an embodiment of the present invention. The method is performed by the processing unit 110 when executing the relevant hardware, firmware or software instructions. FIG. 3 is a schematic diagram of a calibration plate according to an embodiment of the invention. In order to correct the image taken by the camera module 190, the embodiment of the present invention provides a checker board (checkerboard)30 like a checkerboard (checkerboard). In some embodiments, the length and/or width of the calibration plate is adjustable. Generally, the size of the calibration board is preferably such that the entire photographing lens can be covered. Before the camera module 190 leaves the factory, the processing unit 110 drives the camera module controller 170 to control the camera module 190 to photograph the calibration board 30, so that the camera module 190 obtains a photographed image and stores the photographed image in the image buffer 130 (step S210). FIG. 4 is a diagram of a captured image according to an embodiment of the present invention. However, the image sensor senses the photographed image 40 generated by the light penetrating through the lens in the camera module 190. The photographed image 40 is a calibration plate image including distortion (distortion) including radial distortion and tangential distortion. The radial distortion is due to the shape of the lens in camera module 190 such that light rays farther from the center of the lens bend more greatly as they pass through the lens and light rays closer to the center of the lens bend less. The tangential distortion is caused by the assembly error of the camera module 190, mainly because the lens and the image sensor of the camera module 190 are not arranged in parallel.
In order to correct the output of the camera, the processing unit 110 determines the corner of the captured image 40 and a distortion center (step S230), and determines the optimal parameters corresponding to the camera module 190And(step S250), applying the optimal parametersAndand the distortion center obtains the maximum likelihood point coordinates of the angular point in the shot image 40 after the radial distortion and the tangential distortion are eliminated(step S270) according to the maximum likelihood point coordinates of the corner pointsCoordinates of maximum likelihood points of pixels in the captured image 40 other than the corner points are obtained (step S280) and a mapping table is generated and stored in the nonvolatile storage device 160 based on the above calculation results (step S290).
In step S230, the corner point is an extreme point, i.e. a point particularly prominent on the specific property. The corner point may be the intersection of two lines (e.g. the intersection of any two lines in fig. 3) or may be a point located on different things in two adjacent main directions. When the captured image is obtained by capturing a calibration plate, the corner point is the intersection of two edges (see the examples of fig. 4 and 5 for details). The angular points (horns) and distortion centers (x) of the captured image 40 can be determined by those skilled in the art with reference to known algorithmse，ye). For example, Richard Hartley and Sing Bing Kang published in the academic journal IEEE Transactions on Pattern Analysis and Machine understanding of August 2007, Vol.29, No.8, pages 1309 to 1321, the article Parameter-Free Radial disorder Correction with Center of diagnosis evaluation. Fig. 5 is a schematic view of a corner point according to an embodiment of the present invention.
In step S250, the processing unit110 selects one of the sets of parameters and calculates all corner points in the captured image 40 using the selected parameters to produce an energy function. The processing unit 110 generates a plurality of energy functions by using other sets of parameters in the plurality of sets of parameters, and when the energy functions corresponding to all sets are calculated, the set of parameters corresponding to the minimum energy function is used as the optimal parameter, which can be expressed asAndFIG. 6 is a flow chart of a method for determining parameters according to an embodiment of the present invention, the method includes an outer loop (steps S611 to S615) and an inner loop (steps S631 to S637). in each of the outer loops, the processing unit 110 selects a set α from the m sets of parametersj＝(cj，pj)TAnd βj＝(β1j，β2j，β3j)TJ is greater than or equal to 0 and less than or equal to m-1 (step S611), wherein m sets of parameters can be preset according to empirical values, wherein the first parameter α can be used for simulating the curved surface of the radial surface distortion, and the second parameter β can be used for simulating the direction of the optical axisjAnd βjCalculating the coordinates P' (step S631 to step S637) corresponding to the n corner points after removing the radial distortion, wherein the processing unit 110 may sample the n corner points in fig. 5, for example, a column (column) or/and a row (row) in fig. 5jAnd βjWhen all the sets of the first parameter α and the second parameter β are processed (yes in step S615), an outer loop is skipped, and the parameters corresponding to the minimum energy function are taken as the optimal parameters, wherein the optimal parameters include the first optimal parameterAnd a second optimum parameterAnd(step S650).
In each pass of the inner loop, the processing unit 110 selects the first (next) corner point P in fig. 5i＝(xi，yi) I is 0. ltoreq. n-1 (step S631), the parameter α is selectedjAnd βjSubstituting into the surface equation to calculate the corner point PiZ is a depth value ofi(step S633), and using the depth value ziAnd calculating the coordinate P 'with the radial distortion removed by the distance h between the camera module 190 and the calibration plate 30'i(step S635). When all corner points have been processed (yes in step S637), the inner loop is skipped.
In step S633, the depth value ziThe following surface equation can be used for calculation:
s.t.
wherein z isiDepth value, x, representing the ith corner pointiX-coordinate value, y, representing the ith corner pointiY-coordinate value representing the ith corner, cjAnd pjRepresents the jth first parameter α 1j、β2jAnd β 3jRepresenting the jth second parameter β.
In step S635, the coordinates P 'with the distorted radial surface are removed'iCan be usedThe following formula calculates:
wherein, P'iRepresents the coordinate of the ith corner point after the radial distortion is eliminated, h represents the distance between the camera module 190 and the calibration board 30, and xiX-coordinate value, y, representing the ith corner pointiY-coordinate value, z, representing the ith corner pointiRepresenting the depth value of the ith corner point.
In step S613, the corresponding parameter αjAnd βjEnergy function E ofjThe following formula can be used for calculation:
coordinates of median angular point after eliminating radial surface distortionThe following formula can be used for calculation:
wherein h represents the distance between the camera module 190 and the calibration board 30, and xkX-coordinate value, y, representing the kth cornerkY-coordinate value, z, representing the kth corner pointkDepth value, x, representing the kth corner pointk-1X-coordinate value, y, representing the (k-1) th corner pointk-1Y-coordinate value, z, representing the (k-1) th corner pointk-1Depth value, x, representing the k-1 cornerk+1X-coordinate value representing the (k + 1) th corner point，yk+1Y-coordinate value representing the (k + 1) th corner point, and zk+1Representing the depth value of the (k + 1) th corner point. The positional relationship among the (k-1) th, the (k) th and the (k + 1) th corner points may be any three collinear and adjacent corner points, such as the four cases shown in table 1.
Table 1
When the kth corner point is an edge corner point, i.e. the kth side has no (k-1) th and/or (k + 1) th corner point, the coordinates of the median corner pointThe values are respectively:
in step S270, the processing unit 110 executes step S250 to determine the optimal parametersAndintroducing the surface equation to calculate the coordinates corresponding to the corner points in FIG. 5 after removing the radial distortion, and then applying the principle of equidistant space between adjacent corner points to calculate the coordinates corresponding to the corner points in FIG. 5 after removing the radial distortion and the tangential distortion to calculate the coordinates of the maximum likelihood points corresponding to the corner points in FIG. 5 after removing the radial distortion and the tangential distortionDetailed calculation processThe description is as follows: FIG. 7 is a flowchart of a method for determining maximum likelihood point coordinates according to an embodiment of the invention. The processing unit 110 obtains optimal parameters including a first optimal parameterAnd a second optimum parameterWherein,and(step S710), a loop is repeatedly executed to use the optimal parametersAndcalculating the best undistorted radial coordinates P 'corresponding to all corner points in FIG. 5'u，v(steps S731 to S737). In each pass of the loop, the processing unit 110 selects the first (next) corner point P in fig. 5u，v＝(xu，v，yu，v) U is more than or equal to 0 and less than or equal to U-1, V is more than or equal to 0 and less than or equal to V-1, U represents the total number of rows (rows) of angular points, V represents the total number of columns (columns) of angular points (step S731), Pu，vRepresents the v-th column corner point of the u-th row, xu，vX coordinate value representing the v column corner of the u row, and yu，vAnd a y coordinate value representing the v column corner of the u row. Then, the processing unit 110 will optimize the parametersAndangular point P calculated by substituting curved surface equationu，vBest depth value zu，v(step S733), and using the optimal depth value zu，vAnd take picturesThe optimal coordinate P 'with the radial distortion removed is calculated from the distance h between the image head module 190 and the calibration plate 30'u，v(step S735). When all corner points have been processed (yes route in step S737), the loop is exited. Fig. 8 is a schematic diagram of a corner point after radial distortion is removed according to an embodiment of the present invention.
In step S733, corner Point Pu，vBest depth value zu，vThe following surface equation can be used for calculation:
s.t.
wherein z isu，vDepth value, x, representing the corner point of the v-th rowu，vX-coordinate value, y, representing the corner point of the u-th row and v-th columnu，vY-coordinate value representing the angle point of the v-th column of the u-th row and the optimum parameterAndwherein,and
in step S735, the best coordinates P 'after the distortion of the radial surface is removed'u，vThe following formula can be used for calculation:
wherein, P'u，vRepresents the optimal coordinates of the u-th row and v-th column corner points after radial distortion removal, h represents the distance between the camera module 190 and the verification board 30, and xu，vX-coordinate value, y, representing the corner point of the u-th row and v-th columnu，vY-coordinate value representing the v-th column corner of the u-th row, and zu，vRepresents the optimal depth value of the v-th column corner of the u-th row.
When the detected corner point is removed from the radial distortion (i.e. after the corner point of fig. 8 is obtained) (yes in step S737), the processing unit 110 calculates the column average x ″ -of the corner point after the radial distortion is removedvAnd the line average y ″)u(step S750), the nearest distortion center (x) is obtainede，ye) Index _ x1 and index _ x2 of the two columns and the nearest distortion center (x)e，ye) Index _ y1 and index _ y2 of two rows, and the distortion center (x)e，ye) Can be calculated in step S230 (step S760) according to the distortion center (x)e，ye) And calculating the basic value x of the x axis by the information of two adjacent rows and two adjacent columnsbaseAnd step value xstepAnd the basic value y of the y-axisbaseAnd a step value ystep(step S770), and generating maximum likelihood point coordinates corresponding to all the corner points in the photographed image 40 after removing the radial distortion and the tangential distortion therefrom(step S790). Fig. 9 is a schematic diagram of an angular point after radial distortion and tangential distortion are removed according to an embodiment of the present invention, and it should be noted that the schematic diagram of the angular point shown in fig. 9 is inclined because the optical axis is not perpendicular to the calibration plate, and when the optical axis is perpendicular to the calibration plate or almost perpendicular to the calibration plate, the inclination condition shown in fig. 9 disappears or is very insignificant.
In step S750, the column average x ″ of the corner points with the distorted radial surface is eliminatedvAnd the line average y ″)uThe following formula can be used for calculation:
wherein U represents a total number of rows (rows) of corner points, V represents a total number of columns (columns) of corner points, x'u，vX-coordinate values representing the u-th row v-th column corner points after the radial distortion is eliminated, and y'u，vAnd representing the y-coordinate value of the angular point of the ith row and the vth column after radial surface distortion is eliminated.
In step S760, the index values index _ x1, index _ x2, index _ y1, and index _ y2 may be obtained using the following equations:
wherein x iseX-coordinate value, y, representing center of distortioneY-coordinate values representing the center of distortion, U representing the total number of rows of corner points, V representing the total number of columns of corner points, x ″vRepresenting the average value of the x-coordinate values of the v-th column corner points after the distortion of the radial plane is eliminated, and y ″uRepresenting the average value of the y-coordinate values of the u-th line of corner points after the radial plane distortion is eliminated,represents the average value of the x-coordinate values of the column corner points of the index _ x1 after the radial plane distortion is eliminated,represents the average value of the x-coordinate values of the column corner points of the index _ x2 after the radial plane distortion is eliminated,an average value of the removed radial plane distortion y-coordinate values representing the line corner point of the index _ y1, anAnd the mean value of the y-coordinate values of the first index _ y2 line corner point after the radial plane distortion is eliminated.
In step S770, a base value x of the x-axis is calculatedbaseAnd step value xstepAnd the basic value y of the y-axisbaseAnd a step value ystepWherein the basic value x of the x-axis/y-axisbase/ybaseMean value of x-coordinate value/y-coordinate value of one column/row corner point closer to distortion center, x-coordinate valuestep/ystepMeans the difference between the x-coordinate value/y-coordinate value of the index value of two rows/columns near the distortion center after the radial distortion is eliminated. In one embodiment, the base value x of the x-axisbaseAnd step value xstepAnd the basic value y of the y-axisbaseAnd a step value ystepThe following formula can be used for calculation:
wherein,represents the average value of the x-coordinate values of the column corner points of the index _ x1 after the radial plane distortion is eliminated,represents the average value of the x-coordinate values of the column corner points of the index _ x2 after the radial plane distortion is eliminated,an average value of the removed radial plane distortion y-coordinate values representing the line corner point of the index _ y1, anAnd the mean value of the y-coordinate values of the first index _ y2 line corner point after the radial plane distortion is eliminated.
In step S790, the maximum likelihood point coordinates after removal of the radial distortion and the tangential distortion corresponding to all the corner points in the captured image 40The following formula can be used for calculation:
wherein, the value range of r and s can be expressed as:
index_x1≤r≤V-1-index_x1，
index_y1≤s≤U-1-index_y1
wherein,x-coordinate values representing the (index _ x1+ r) -th column corner points after removing the radial distortion and the tangential distortion,and the y-coordinate values represent the (index _ y1+ s) th row corner points after radial distortion and tangential distortion are eliminated, U represents the total number of rows of corner points, and V represents the total number of columns of corner points.
In step S280, those skilled in the art can rely on the maximum likelihood point coordinates of the corner pointThe coordinates of the maximum likelihood points of the pixels in fig. 9 except for the corner points are calculated using a known algorithm (e.g., interpolation).
In step S290, the non-volatile memory device 160 can be a flash memory or other memory device that does not cause the disappearance of the mapping table due to the power failure. The mapping table may include a plurality of storage cells, the number and location of the storage cells corresponding to the number and location of image sensors of the image sensor array. For example, when the image sensor array includes mxn image sensors, the mapping table includes mxn memory cells, m and n are integers greater than 0, and m and n may be the same or different integers. Each cell records reference position information of a pixel of a captured image. Suppose storage cell [ i, j ] records [ k, l ]: in detail, when the storage cell [ i, j ] is the corner determined in step S230, then [ k, l ] may include information of the maximum likelihood point coordinates after radial distortion and tangential distortion removal corresponding to the corner calculated in step S270. When the storage bin [ i, j ] is not the corner determined in step S230, then [ k, l ] may include information for calculating the maximum likelihood point coordinates corresponding to the pixel in step S280. In some embodiments, the storage information of the storage cell [ i, j ] may represent that the reference position of the pixel [ i, j ] of the captured image is [ k, l ], where i, k are any integer from 0 to m-1, and j, l are any integer from 0 to n-1. In some embodiments, the storage information of the storage cell [ i, j ] may represent that the reference position of the pixel [ i, j ] of the captured image is [ i + k, j + l ], where i is any integer from 0 to m-1, k is any integer (positive integer, 0 or negative integer) and i + k is from 0 to m-1, and j is any integer from 0 to n-1, l is any integer and j + l is from 0 to n-1. In some embodiments, to reduce the storage space, the mapping table may store only the information of the maximum likelihood point coordinates after removing the radial distortion and the tangential distortion corresponding to the corner point determined in step S230. In some embodiments, the mapping table may store only information of the optimal coordinates of the corner point determined in step S735 after removing the radial distortion.
After the shipment of the camera module 190, the processing unit 110 may take the original image from the camera module 190 and generate the adjustment image according to the reference position information of the mapping table in the nonvolatile memory device 160. In one example, the processing unit 110 may take the value of the pixel [ k, l ] in the original image as the value of the pixel [ i, j ] in the adjusted image. In another example, the processing unit 110 may obtain the value of the pixel [ i + k, j + l ] in the original image as the value of the pixel [ i, j ] in the adjusted image. In yet another example, the processing unit 110 may calculate the value of the pixel [ k, l ] and the values of the neighboring pixels in the original image using a smoothing algorithm (smoothing algorithm) and take the calculation result as the value of the pixel [ i, j ] in the adjusted image. In yet another example, the processing unit 110 may use a smoothing algorithm to calculate the value of the pixel [ i + k, j + l ] and the values of the neighboring pixels in the original image and use the calculation result as the value of the pixel [ i, j ] in the adjusted image.
In one aspect of the present invention, the processing unit 110 drives the camera module controller 170 to control the camera module 190 to photograph the calibration plate 30, so that the camera module 190 takes the photographed image 40 containing the distortion; eliminating distortion of the photographed image 40 using an algorithm for generating reference position information corresponding to a plurality of pixels of the photographed image 40; and stores the mapping table to the non-volatile storage 160. In an alternative embodiment, the processing unit 110 may generate an adjustment model according to the adjustment result, wherein the adjustment model includes a plurality of mathematical formulas or algorithms and parameters thereof for reducing distortion in the original image. However, it should be noted that when the distortion contained in the image imaged on the image sensor array is difficult to be simulated using the mathematical formula and its parameters, the generated adjustment model will not effectively eliminate the distortion contained in the image. Unlike the above embodiments, the mapping table of the embodiment of the present invention includes a plurality of storage cells, and each storage cell records reference position information of one pixel, so as to solve the above-mentioned defects.
In another aspect of the invention, the processing unit 110 performs the correction using two stages: determining the optimal parameters of the camera module 190 according to the information of the corner points; and obtaining the distortion-removed maximum likelihood point coordinates of the plurality of pixels in the photographed image 40 using the optimum parameters and the distortion center. Finally, the processing unit 110 stores a mapping table to the nonvolatile storage device 160, wherein the mapping table contains information of the maximum likelihood point coordinates after the distortion is removed. In still another aspect of the present invention, the processing unit 110 obtains maximum likelihood point coordinates after removing distortion in the following manner: obtaining a plurality of distortion-eliminated maximum likelihood point coordinates corresponding to the corner points in the shot image by using the optimal parameters and the distortion center; and obtaining the maximum likelihood point coordinates of the pixels except the corner points in the shot image according to the maximum likelihood point coordinates of the corner points. In some alternative embodiments, the calibration method requires that the calibration plate is photographed from different angles to generate a plurality of images, and then an adjustment model is generated according to the corner points and the distortion center information of the plurality of images, wherein the adjustment model includes a plurality of mathematical formulas or algorithms and parameters thereof. Unlike the above-described embodiment, the two-stage correction of the embodiment of the present invention requires only one-time photographing of the calibration board 30 to obtain distortion-removed maximum likelihood point coordinates of all pixels in the photographed image 40.
Although fig. 1 includes the above-described elements, it is not excluded that more additional elements may be used to achieve better technical results without departing from the spirit of the invention. Moreover, although the process steps of FIGS. 2, 7 and 8 are performed in a particular order, those skilled in the art can modify the order of the steps without departing from the spirit of the invention to achieve the same result, and therefore the invention is not limited to use in only the order described above.
While the present invention has been described with reference to the above embodiments, it should be noted that the description is not intended to limit the invention. Rather, this invention covers modifications and similar arrangements apparent to those skilled in the art. The scope of the claims is, therefore, to be construed in the broadest manner to include all such obvious modifications and similar arrangements.