Summary of the invention
The purpose of the present invention is to provide a kind of based on the microlens array type optical field camera geometric parameter mark for focusing picture pointMethod is determined to solve the geometric parameter of camera.
To achieve the above object, the present invention provides a kind of based on the microlens array type optical field camera geometric parameters for focusing picture pointNumber scaling method, the microlens array type optical field camera geometrical parameter calibration method based on focusing picture point includes following stepRapid: S1 obtains object point and focuses picture point about main lens according to the focal imaging index path of microlens array type optical field cameraMapping relations；S2 obtains focusing picture point and detector picture point according to the focal imaging index path of microlens array type optical field cameraMapping relations about lenticule；S3 solves the coordinate for focusing picture point according to the detector picture point that detection obtains；S4, according to S3The coordinate of the focusing picture point of acquisition solves camera internal parameter matrix and external parameter matrix in peg model；And S5, lead toThe camera internal parameter matrix and external parameter matrix of S4 acquisition are crossed, the geometric parameter of microlens array type optical field camera is demarcated,The geometric parameter includes at a distance from the spacing and main lens to lenticule of microlens array and detector plane.
Further, " mapping relations of object point and focusing picture point about main lens " in S1 are expressed as formula (1), are set:Object point is expressed as M, and coordinate is (xw,yw,zw)；It focuses picture point and is expressed as m ', coordinate is (u ', v '):
Sm '=A [R t] M, (1)
In formula (1), s is scale factor；M=[xw,yw,zw,1]T；M '=[u ', v ', 1]T；[R t] indicates microlens arrayThe external parameter matrix of type light-field camera, A indicate the inner parameter matrix of microlens array type optical field camera；
[R t [formula (3) and formula (2) being expressed as:
In formula (2) and formula (3), q1、q2、q3Respectively indicate three seats in camera coordinates system and world coordinate system conversion processThe rotation angle of parameter, tx、ty、tzIndicate the distance that the origin of world coordinate system is translated along three reference axis；
The formula (4) that A is expressed as:
In formula (4), dx × dy is the Pixel Dimensions of detector, u0、v0For seat of the main lens centre under image coordinate systemMark, θ are the out of plumb inclination angle of two reference axis；
Wherein: camera coordinates system is using the center of main lens as origin O, xc、ycAxis is parallel to detector plane, zcAxis is verticalIn detector plane；World coordinate system is to demarcate the center of object plane as origin Ow, xw、ywAxis is parallel to calibration object plane, zwAxisPerpendicular to calibration object plane；Image coordinate system is using detector plane center as origin Oc, u, v axis is parallel to xc、ycAxis.
Further, " the focusing picture point and mapping relations of the detector picture point about lenticule " in S2 is formula (8), ifFixed: detector picture point includes picture point m1With picture point m2, picture point m1It is (u in the coordinate of image coordinate system1,v1), it is corresponding micro-The center of mirror is (p in the coordinate of image coordinate system1,q1)；Picture point m2It is (u in the coordinate of image coordinate system2,v2), it is correspondingLenticule center is (p in the coordinate of image coordinate system2,q2)；
Sm '=sT-1m1=A [R t] M (8)
In formula (8): m1It is expressed as picture point m1Coordinate vector；T is picture point m ' and picture point m1Between transition matrix；
m1It is expressed as m1=[u1,v1,1]T；
The formula (6) and formula (7) that T is expressed as:
δ=b/a=(u1-u2)/(p1-p2)-1 (7)。
Further, the detector picture point that S3 is specifically obtained according to detection, and being focused into using microlens array typeAs the geometrical relationship in index path, the coordinate for focusing picture point is solved；
" the detector picture point that detection obtains " is picture point m1、m2；
" geometrical relationship in the focal imaging index path of microlens array type " is expressed as formula (9):
Formula (10) then can be obtained according to formula (8):
Firstly, by picture point m1、m2Image coordinate system coordinate and corresponding lenticule center image coordinate system seatIt marks substitution formula (7), δ value is calculated；
Then, the δ value being calculated is substituted into formula (9), solves and focuses picture point m ' coordinate (u ', v ').
Further, S4 is specifically included: S41, if peg model plane z in world coordinate systemw=0, it substitutes into formula (1),3 × 3 homography matrix H can be obtained with corresponding focusing picture point m ' coordinate according to object point M, the maximum for being solution matrix H is seeminglySo estimate, the focusing picture point in iterative image seeks lsqnonlin；S42, according to r1With r2Constraint condition, willThe r obtained by matrix H1With r2Expression formula substitute into, then in equation that symmetrical matrix B and its calculating formula are brought into, finallyTo 2 × 6 matrix Vs, the homography matrix h of three width images is substituted intoiCan solution matrix b, and then the unknown ginseng in solution matrix ANumber；S43 has solved matrix A, and matrix A is substituted into r1With r2Expression formula can solve r1、r2, and associate(d) matrix H can be solvedUnknown parameter in matrix [R t] out；S44 demarcates n width image, and the focusing picture point of each image has m, to above-mentionedThe parameter iteration optimization that method for solving is found out, seeks lsqnonlin, is carried out with the parameter found out to above-mentioned stepsNonlinear optimization.
Further, S41 is specifically included:
If peg model plane z in world coordinate systemw=0, (1) formula is substituted into, then there are formula (11):
3 × 3 homography matrix H can be obtained with corresponding focusing picture point m ' coordinate according to object point M, form is as follows:
H=[h1 h2 h3]=λ A [r1 r2 t] (12)
In formula (12), hi=[hi1,hi2,hi3]T, λ is any scalar；
For the maximal possibility estimation of solution matrix H, the focusing picture point in iterative image seeks lsqnonlin:
In formula (13), m 'iThe actual coordinate vector of picture point is focused for i-th in same piece image,According toFormula (11), formula (12) are by object point MiIt is calculated.
Further, S42 is specifically included:
By the property of spin matrix it is found that r in matrix R1With r2Meet following two constraint condition:
From formula (12)
r1=λ A-1h1,r2=λ A-1h2 (15)
In formula (15), λ=1/ | | A-1h1| |=1/ | | A-1h2| |, formula (15) substitution formula (14) can be obtained:
It enables
B is symmetrical matrix, can be calculated formula (18):
In formula (18):
B=[B11,B12,B22,B13,B23,B33]T (19)
vij=[hi1hj1,hi1hj2+hi2hj1,hi2hj2,hi3hj1+hi1hj3,hi3hj2+hi2hj3,hi3hj3]T (20)
Formula (17) and formula (18), which are substituted into formula (16), can obtain formula (21):
There are formula (22):
In formula (22), V is 2 × 6 matrixes, contains 6 unknown parameters, substitutes into the homography matrix h of three width imagesiIt can solveMatrix b, and then the unknown parameter in solution matrix A:
γ=- B12α2β/λ (26)
u0=γ v0/β-B13α2/λ (27)
So that it is determined that the inner parameter matrix A of microlens array type optical field camera.
Further, S43 is specifically included:
S42, which is solved matrix A substitution formula (15), can solve r1、r2, specific calculation expression is formula (29):
r3=r1×r2 (29)
Formula (30) can be obtained by formula (12):
T=λ A-1h3 (30)。
Further, S44 is specifically included:
N width image is demarcated, the focusing picture point of each image has m, the parameter that above-mentioned method for solving is found out intoRow iteration optimization, seeks lsqnonlin:
In formula (31), m 'ijFor the focusing picpointed coordinate of i-th j-th of angle point of width image,According toFormula (8) by the i-th width image j-th of object point coordinate vector MjAnd Ai, Ri, tiIt is calculated, the initial value of A and [R t] are by S41It is calculated to S43.
Further, S5 is specifically included:
The diameter d of main lens pupil diameter D and lenticule image meets following formula:
D/L=d/b (32)
Formula (33) can be obtained by formula (7):
A=b/ δ (33)
Formula (32) and formula (33) are substituted into α=(L-a)/(dx) in formula (4), formula (34) and formula (35) can be acquired:
L=a+ α dx (34)
B=α δ ddx/ (δ D-d) (35)
In formula (34) and formula (35), parameter alpha, δ become known parameters.
Model is divided into two conjugate relations by method provided by the invention, detects angle point, by angle point according to the second conjugationRelationship obtains focusing picture point, then the geometric parameter of microlens array type optical field camera, this method are calculated by the second conjugate relationModel coupling degree is low, calculates easy.
Specific embodiment
In the accompanying drawings, same or similar element is indicated using same or similar label or there is same or like functionElement.The embodiment of the present invention is described in detail with reference to the accompanying drawing.
As shown in Figure 1, provided in this embodiment based on the microlens array type optical field camera geometric parameter mark for focusing picture pointDetermine method the following steps are included:
S1 obtains object point and focuses picture point about master according to the focal imaging index path of microlens array type optical field cameraThe mapping relations of lens, that is, " the first conjugate relation " mentioned below.
S2 obtains focusing picture point and detector picture point according to the focal imaging index path of microlens array type optical field cameraAbout the mapping relations of lenticule, that is, " the second conjugate relation " mentioned below.
S3 solves the seat for focusing picture point using inverse conversion matrix or geometrical relationship according to the detector picture point that detection obtainsMark.
S4 solves the camera internal in peg model using iterative algorithm and joins according to the coordinate of the S3 focusing picture point obtainedMatrix number and external parameter matrix." camera internal parameter matrix " in text refers to the inside of microlens array type optical field cameraParameter, " camera external parameter matrix " refer to the external parameter of microlens array type optical field camera.
S5, the camera internal parameter matrix and external parameter matrix obtained by S4 demarcate microlens array type optical field phaseThe geometric parameter of machine, the spacing b and main lens to lenticule distance L of the geometric parameter microlens array and detector plane.
Just five steps of the invention are described in detail separately below.
In one embodiment, in S1, microlens array type optical field camera before the detector in traditional camera by puttingMicrolens array is set, the field information of scene is obtained.
The index path of microlens array type optical field camera is as shown in Figure 2.As shown in Fig. 2, microlens array is by multiple arraysThe structure of the lenticule composition of formula arrangement, such as the partial microlenses in microlens array shown in Figure 2, i.e. the first lenticuleL1, the second lenticule l2, third lenticule l3, the 4th lenticule l4.
In Fig. 2, sequence from left to right is successively are as follows: the straight dashed line P1 at the place picture point m ', which is illustrated that, to be focused into where picture pointPlane (being hereafter referred to as " being focused into image planes ").Focus picture point m1With focusing picture point m2The number line P2 at place is illustrated thatPlane (being hereafter referred to as " detector face ") where detector.First lenticule l1, the second lenticule l2, third lenticulePlane where l3, the 4th lenticule l4 while the number line P3 at place illustrate that microlens array (is hereafter referred to as " micro-Lens array face ").Number line P3 where main lens lm illustrates that main lens plane.Straight dashed line P4 signal where object point MBe plane (being hereafter referred to as " object plane ") where object point.
Wherein: the spacing of microlens array surface and detector face is b, it may also be said to be: microlens array and detector are flatThe spacing b in face, that is, one of the geometric parameter to be calibrated of the invention referred in S5 above.Main lens plane is to lenticule battle arrayThe spacing in column face is L, it may also be said to be: the distance L of main lens to lenticule, that is, the present invention referred in S5 above is wait markThe two of fixed geometric parameter.
The distance for being focused into image planes to microlens array surface is a, and the focal length of lenticule is f, from lens imaging formula1/a+1/b=1/f.The case where b < f is corresponded in Fig. 2, a < 0 at this time, the picture point that object point is formed through main lens is in microlens arrayThe other side is the virtual image.
The distance for being focused into image planes to main lens plane is bL, the distance of object plane to main lens plane is aL.A is diaphragm,The pupil diameter of main lens is D, and the diameter of lenticule image is d.
In S1, for the microlens array arrangement mode of above-mentioned microlens array type, the identical lenticule battle array of focal length is consideredColumn provide the direct problem modeling that picture point is generated by object point.Direct problem modeling includes establishing object point and focusing picture point about main lensMapping relations, and focus picture point and mapping relations of the detector picture point about lenticule.
Microlens array type focal imaging process shown in Fig. 2, the process is there are two conjugate relations, i.e. object point M and poly-Burnt picture point m ' is about two focusing picture point m on main lens planar conjugate, picture point m ' and detector1,m2About microlens array surfaceConjugation, respectively corresponds " the first conjugate relation " and " the second conjugate relation " mentioned above.
" mapping relations of object point and focusing picture point about main lens " in S1 specifically:
For object point M and mapping relations of the picture point m ' about main lens are focused, are described by the first conjugate relation.Object point M warpMain lens forms the process of picture point m ', can be converted by the coordinate between world coordinate system, camera coordinates system and image coordinate systemRelationship describes.
As shown in figure 3, setting: three-dimensional camera coordinate system is using the center of main lens as origin O, xc、ycAxis is parallel to detectorPlane, zcAxis is perpendicular to detector plane.Two dimensional image coordinate system is using detector plane center as origin Oc, u, v axis is parallel toxc、ycAxis.Three-dimensional world coordinate system is to demarcate the center of object plane as origin Ow, xw、ywAxis is parallel to calibration object plane, zwAxis hangs downDirectly in calibration object plane.The unit of three-dimensional camera coordinate system is pixel, and the unit of two dimensional image coordinate system is mm, three-dimensional worldThe unit of coordinate system is mm.The coordinate of object point M is (xw,yw,zw), the coordinate of picture point m ' is (u ', v '), and u ' corresponds to show in Fig. 2The u ' to anticipate out.
Object point M is transformed into the process of the picture point m ' under two dimensional image coordinate system, is represented by formula (1):
Sm '=A [R t] M (1)
In formula (1):
S is scale factor；
M is represented by M=[xw,yw,zw,1]T；
M ' is represented by m '=[u ', v ', 1]T；
The external parameter matrix of [R t] expression microlens array type optical field camera；
The inner parameter matrix of A expression microlens array type optical field camera.
The external parameter matrix [R t] of camera describes the transformational relation between world coordinate system and camera coordinates system, shouldTransformational relation is represented by formula (2) and formula (3):
In formula (2) and formula (3), q1、q2、q3Indicate three reference axis in camera coordinates system and world coordinate system conversion processRotation angle；tx、ty、tzIndicate origin OwRespectively along the distance of three reference axis translation.
The inner parameter matrix A of camera is represented by formula (4), formula (4) describe camera coordinates system and image coordinate system itBetween transformational relation:
In formula (4): dx × dy is the Pixel Dimensions of detector；u0、v0For seat of the center under image coordinate system of main lensMark, θ are the out of plumb inclination angle of camera coordinates system and world coordinate system.
In one embodiment, in S2, setting: picture point m1It is (u in the coordinate of image coordinate system1,v1), it is corresponding micro-The center of lens is (p in the coordinate of image coordinate system1,q1), u1、p1Correspond to the u illustrated in Fig. 21、p1；Picture point m2SchemingAs the coordinate of coordinate system is (u2,v2), corresponding lenticule center is (p in the coordinate of image coordinate system2,q2), u2、p2It is correspondingFor the u illustrated in Fig. 22、p2.It can be obtained by geometrical relationship of the lenticule in Fig. 3 under pin-hole model, main lens, which focuses, to be formedPicture point m ' and picture point m on detector1Coordinate transformation relation be formula (5):
m1=Tm ' (5)
In formula (5):
m1=[u1,v1,1]TFor picture point m1Coordinate vector；
T is picture point m ' and picture point m1Between transition matrix, be specifically represented by following formula (6) and formula (7):
δ=b/a=(u1-u2)/(p1-p2)-1 (7)
Formula (1) is substituted into formula (5), can obtain and focus picture point m on object point M and detector1Coordinate transformation relation be formula (8):
Sm '=sT-1m1=A [R t] M (8)
In one embodiment, it in S3, is grouped into first with the picture point detected on the detector face, it will be sameThe corresponding detector picture point of object point is as one group, and every group of detector picture point is according to T-1Or geometrical relationship obtains focusing picpointed coordinate.
According to the geometrical relationship in the focal imaging index path of microlens array type can formula (9):
I.e. in formula (8)
Two picture points are chosen in the detector picture point of the same object point of correspondence, by picpointed coordinate and the corresponding lenticule of picture pointCentre coordinate substitutes into formula (1) and calculates δ value, and then solves the coordinate (u ', v ') of the focusing picture point in formula (9).
In one embodiment, in S4, in order to solve the microlens array type optical field camera internal parameter matrix in formula (8)A and external parameter matrix [R t], the focusing picpointed coordinate obtained using S3 are used for reference Zhang Zhengyou calibration method and use iterative algorithmIt solves, its specific method includes:
S41, if peg model plane z in world coordinate systemw=0, it substitutes into formula (1).According to object point M and corresponding focusing3 × 3 homography matrix H can be obtained in picture point m ' coordinate, are the maximal possibility estimation of solution matrix H, poly- in iterative imageBurnt picture point seeks lsqnonlin.
S42, according to r1With r2Constraint condition, the r that will be obtained by matrix H1With r2Expression formula substitute into, then by symmetrical squareIn the equation that battle array B and its calculating formula are brought into, 2 × 6 matrix Vs are finally obtained, substitute into the homography matrix of three width imageshiCan solution matrix b, and then the unknown parameter in solution matrix A.
S43 has solved matrix A, is substituted into r1With r2Expression formula can solve r1、r2, and associate(d) matrix H can be askedSolve the unknown parameter in matrix [R t].
S44 demarcates n width image, and the focusing picture point of each image has m, the ginseng found out to above-mentioned method for solvingNumber iteration optimization, seeks lsqnonlin, carries out nonlinear optimization with the parameter found out to above-mentioned steps.
In one embodiment, " substituting into formula (1) " in S41 is specifically such as formula (11):
" 3 × 3 homography matrix H " in S41 is specifically such as formula (12):
H=[h1 h2 h3]=λ A [r1 r2 t] (12)
" the focusing picture point in iterative image seeks lsqnonlin " in S41 is specifically such as formula (13):
M ' in formula (13)iThe actual coordinate vector of picture point is focused for i-th in same piece image,It canWith according to formula (11), formula (12) by object point MiIt is calculated.
In one embodiment, " the r in S421With r2Constraint condition " specifically such as formula (14):
" the r that will be obtained by H in S421With r2Expression formula substitute into " specifically such as formula (15):
r1=λ A-1h1,r2=λ A-1h2 (15)
In formula (15), λ=1/ | | A-1h1| |=1/ | | A-1h2||。
After formula (15) are substituted into formula (14), formula (16) can be obtained:
" in the equation for bringing symmetrical matrix B and its calculating formula into " in S42 is specifically such as formula (17):
It enables
B is symmetrical matrix, can be calculated formula (18):
In formula (18):
B=[B11,B12,B22,B13,B23,B33]T (19)
vij=[hi1hj1,hi1hj2+hi2hj1,hi2hj2,hi3hj1+hi1hj3,hi3hj2+hi2hj3,hi3hj3]T (20)
" obtaining 2 × 6 matrix Vs " in S42 is specifically that formula (17) and formula (18) are substituted into formula (16), obtains formula(21):
There are formula (22):
In formula (22), V is 2 × 6 matrixes, contains 6 unknown parameters.
" unknown parameter in solution matrix A " in S42 specifically includes:
V is substituted into the homography matrix h of three width imagesiCan solution matrix b, and then obtain:
γ=- B12α2β/λ (26)
u0=γ v0/β-B13α2/λ (27)
So that it is determined that the inner parameter matrix A of microlens array type optical field camera.
In one embodiment, " r is substituted into S431With r2Expression formula can solve r1、r2, and can be solved in conjunction with HUnknown parameter in matrix [R t] " specifically includes:
Matrix A substitution formula (15) can be solved into r1、r2:
r3=r1×r2 (29)
Formula (30) can be obtained by formula (12):
T=λ A-1h3 (30)
In one embodiment, " the non-linear least square problem " in S44 is specifically such as formula (31):
In formula (31):
m′ijFor the focusing picpointed coordinate of i-th j-th of angle point of width image；
According to formula (8) by j-th of object point coordinate vector M of the i-th width imagejAnd Ai, Ri, tiMeterIt obtains；
The initial value of A and [R t] are calculated by S41 to S43.
In one embodiment, in S5, after the inside and outside parameter matrix for acquiring microlens array type optical field camera, due to parameterα, δ become known parameters, then the spacing b and main lens face for solving microlens array surface and detector face arrive microlens array surfaceDistance L is specific as follows:
By similar triangles relationship in Fig. 2 it is found that the diameter of main lens pupil diameter and lenticule image meets following formula(32):
D/L=d/b (32)
Wherein, D is main lens pupil diameter, and d is the diameter of lenticule image.Formula (33) can be obtained by formula (7):
A=b/ δ (33)
Formula (32) and formula (33) are substituted into α=(L-a)/(dx) in formula (4), formula (34) and formula (35) can be acquired:
L=a+ α dx (34)
B=α δ ddx/ (δ D-d) (35)
Finally it is noted that the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations.ThisThe those of ordinary skill in field is it is understood that be possible to modify the technical solutions described in the foregoing embodiments or rightPart of technical characteristic is equivalently replaced；These are modified or replaceed, and it does not separate the essence of the corresponding technical solution originallyInvent the spirit and scope of each embodiment technical solution.