The content of the invention：
It is an object of the invention to provide a kind of 3D panoramic parking assist systems, by the three-dimensional scape of motor vehicle environment situationAs being truly presented to driver, to improve the security during automobile parking.
To realize the purpose, the present invention takes following technical scheme：
A kind of 3D panoramic parking assist systems that the present invention is provided, the system includes：Camera input module,Image correction module, tripleplane's module, image mosaic module and image display.
1) the camera input module, the image for gathering peripheral vehicle situation in real time；
Four fish-eye cameras are respectively arranged in car body all around on four direction, front camera and rear camera difference positionIn the center after Chinese herbaceous peony, left and right camera is located at the side to the front at car both sides center respectively；
Because the size of all kinds of vehicles is different, therefore the angle that camera is installed should be also adjusted accordingly by vehicle body.Coordinate system is set, respectively using each video camera as origin, crossing origin, upward straight line is Z axis perpendicular to the ground, excessively formerThe straight line of the vertical vehicle body major axis center of point and vertical Z axle is X-axis, and the straight line for crossing the vertical X of origin and Z axis is set to Y-axis,For convenience of description, original state is set to camera optical axis and overlapped with Z axis, and the rotation of camera intrinsic Y-axis makes the visual fieldAt least 8 meters of coverage distance vehicle body, video camera rotates about the z axis enables the image around vehicle body to be presented on imageIntermediate region；
2) described image correction module, for correcting anamorphose and distortion；
Fish eye lens is a kind of special lens of ultra-wide angle, and its focal length is extremely short and visual angle can reach 180 degree,Fish eye lens can realize closely big visual field, but be also introduced into very serious anamorphose simultaneously.AnamorphoseMain cause be that fish-eye real imaging surface is not plane, but be similar to spherical curved surface.
In addition, also there is fish-eye distortion, the light sent by picture point is reflected through fish eye lens lensAfterwards, optical parallax existing between formed picture point on ideal image point and video camera actual imaging face.
Therefore, it need to be allowed to meet the imaging model of common camera by correcting deformed and distortion, could be applicableOperated in follow-up stereoscopic vision；
Fish eye lenses correction uses the most frequently used equidistant projection model, and distortion correction is used on coordinate offset influence mostBig radial distortion model.Through deriving, the coordinate (x with the coordinate that deforms and distortu, yu), distortion coordinate (xd, yd),It is as follows with (x, y) coordinate relation after correction：
Wherein, f is focal length, k1, k2, k3, k4For distortion factor.
To solve above parameter, (x, y) and 3 d space coordinate point after stereoscopic model is corrected must be set up(Xw, Yw, Zw) through deriving, stereoscopic model is as follows：
Wherein, s is scale factor, and [R T] is outer ginseng matrix, and R is 3x3 spin matrix, and T is 3x1 translationMatrix.
Simultaneous calibration model and stereoscopic model, can obtain the coordinate points (x with the coordinate that deforms and distortu, yu) and3 d space coordinate point (Xw, Yw, Zw) non-linear relation.
By shooting multigroup three-dimensional feature point, multigroup non-linear relation is obtained, three-dimensional re-projection error is set up mostOptimization method.
Wherein k is characterized a number,For the space three-dimensional point coordinates of ith feature point,For the estimation space three-dimensional point coordinate of ith feature point, i.e., tried to achieve by non-linear relation.
The Nonlinear Optimization Problem is solved using ant colony optimization algorithm, optimal solution f, k can be tried to achieve1, k2, k3, k4。
Thus, the deformation of calibration model recoverable fish eye lens and pattern distortion are passed through；
3) tripleplane's module, for image after the correction of four direction video cameras to be projected into three-dimensional song respectivelyOn face；
According to the setting angle of video camera and position, relation of the picture point to three dimensions after correction is calculated；Through pushing awayLead, relational expression is as follows：
Wherein s ' is scale factor, and r is that, around the X-axis anglec of rotation, b is that, around the Y-axis anglec of rotation, a is to rotate about the z axisAngle, h is camera setting height(from bottom), cr, sr, cb, sb, ca, and sa is respectively cos (r), sin (r), cos (b), sin (b),Cos's (a), sin (a) writes a Chinese character in simplified form.
It is only relevant with image scaling because s ' is scale factor, to simplify process, it can omit, by subsequently splicingProcess is operated.Then formula is changed into：
Wherein P is 3x4 projection matrix.
Due to vehicle body, nearby environment is generally ground, and remote environment is generally the object and building being above the ground level, and isScalloping is deformed small and different zones image draws the harmony stretched good, the class for setting curvature transition softThree-dimensional surface model like " barrel-shaped " is more reasonable；The vertical point to ground is origin centered on vehicle body,The curve for generating three-dimensional surface model can be divided into three sections, first paragraph is ground linearity sector, and second segment is transitionCurve area, the 3rd section is the curve area highly steeply risen, can be set according to below equation：
Wherein, f1, f2, f3Three sections of curvilinear functions of difference, R is the distance apart from origin, reg1, reg2For three songsLine region segmentation point, a2, b2, c2, a3, b3For function coefficients, to make transition round and smooth, set curve should be canLead curve, should meet：
Image after the correction of four direction video cameras is projected on three-dimension curved surface respectively using projection matrix；
4) described image concatenation module, for the image mosaic of four cameras collections in an image, to be given birth toInto 3D panorama sketch；
Grid line is laid on vehicle's surroundings ground, to the grid line of overlay region registration, wherein only comprising image scaling andRotation translation.Therefore registration can be carried out by asking for affine matrix.Formula is as follows：
Wherein, xi, yi, xj, yjFor the point of adjacent two images overlay region, h11, h12, h13, h21, h22, h23For affine squareBattle array parameter, H is affine matrix.
It is as follows by multiple overlapping simultaneous equations：
Indeterminate equation is solved by singular value decomposition method, affine matrix H can be tried to achieve.
Four affine matrixs can be obtained altogether using four direction camera reviews, if using one of image as reference,We only need to three affine matrixs, you can make all directions image mosaic in one image.It can select at randomWherein three are taken, or sets up two-dimentional re-projection error and minimizes function, choose wherein error minimum three imitatePenetrate matrix.
For four direction camera review brightness disproportionation weighing apparatus problems, three passages of all directions image can be made respectivelySolved with algorithm of histogram equalization, specific formula is as follows：
Wherein T (k) is the gray value corresponding to k-th of gray level, and k is the gray level between 0 to 255, and n is figureAs pixel sum, niThe number for being i for gray level in image.
For overlay region brightness it is inconsistent the problem of, can to it is overlapping differentiation block processing brightness problem, can be by following stepIt is rapid to solve：
The first step：The area size of overlay region is divided into some sub-piecemeals；
Second step：RGB image triple channel is averaged as luminance picture L, i.e.,：
3rd step：The gain coefficient of each sub-piecemeal is obtained, specific formula is as follows：
Wherein, GI, x, y, GJ, x, yThe gain coefficient of respectively adjacent image i, j overlay region sub-piecemeal pixel (x, y).LI, x, y, LJ, x, yThe mean flow rate of difference adjacent brightness image i, j overlay region sub-piecemeal；
4th step：The gain coefficient of the smooth overlay region sub-piecemeal of gaussian filtering, gain coefficient is after obtaining smoothlyHI, x, y, HJ, x, y。
5th step：The RGB triple channels pixel of each sub-piecemeal of overlay region after then merging is adjacent when not mergingImage respective pixel point is multiplied by gain coefficient HI, x, y, HJ, x, yAverage value afterwards.
The splicing mark problem existed to overlay region, from respective frontier distance ratio can be used as power according to pixelValue, splicing vestige is eliminated using weighted mean method, and specific formula is as follows：
Wherein, L (x, y) is the image coordinate of 3D panorama sketch overlay region, I1(x, y) and I2(x, y) is respectively neighbor mapThe image coordinate of the overlay region of picture,It is (x, y) pixel from overlay region frontier distance ratio.
After these processing, final 3D panorama sketch is generated；
5) described image display module, for showing 3D panorama sketch；By 3D panorama sketch it is properly scaling after, outputTo display device, display device can be conformable display or LCDs.
The beneficial effect that the technical scheme that the present invention is provided is brought is：
A kind of 3D panoramic parking assist systems that the present invention is provided, can essence using equidistant projection and radial distortion modelReally correcting deformed and distortion, makes generation 3D panorama sketch truer；Based on actual conditions, similar " barrel-shaped " is setThree-dimension curved surface projection, make curved surface connection it is soft, distorting for image in kind is small.Using histogram equalizationChange method makes all directions brightness of image unified, makes overlay region brightness with piecemeal brightness processed and weighted average methodUnanimously, transition is soft, realizes seamless connection.The overall good visual effect of the system generation 3D panorama sketch, veryReal reduction surrounding environment, reduces risk of parking.
Embodiment
To make purpose, the technical scheme of the embodiment of the present invention clearer, below in conjunction with the embodiment of the present inventionAccompanying drawing, the technical scheme in the embodiment of the present invention is clearly and completely described, it is clear that describedEmbodiment be a part of embodiment of the invention, rather than whole embodiment.Based on the implementation in the present inventionExample, the every other implementation that those of ordinary skill in the art are obtained under the premise of creative work is not madeExample, belongs to the scope of protection of the invention.
Fig. 1 is to be sequentially connected between the flow chart of the allomeric function module of the present invention, modules, figure thereinAs correction module, tripleplane's module, image mosaic module are all that its function is realized by software algorithm.Fig. 2Show the software flow block diagram of the system.
Fig. 3 is the installation site and camera coverage schematic diagram of four cameras of the invention in embodiment, fourIndividual fish-eye camera 110,120,130 and 140 be respectively arranged in before below logo, below trunk handle,Below two rearview mirrors, the region photographed is respectively 110a, 120a, 130a and 140a.All aroundAll around 8 meters of coverage at least coverage distance vehicle body.
Processor receives the view data of four camera acquisitions, and 3D is exported using a whole set of image processing techniquesPanorama sketch, driver is presented to using display device.
Specifically, the processing procedure that processor gathers image to four video cameras at least comprises the following steps.
Step one：Processor receives the view data of four camera acquisitions；
Step 2：Using equidistant projection and the correction four direction anamorphose of radial distortion model and distortion；
Step 3：According to the setting angle of video camera and position, the projection matrix of four direction image is calculated, ifThe three-dimensional surface model of curvature transition soft similar " barrel-shaped " is put, is projected image onto using projection matrixThree-dimension curved surface；
Step 4：By means of the grid line being laid on vehicle's surroundings ground, to the grid line registration of overlay region,Three affine matrixs are solved, make all directions image mosaic in one image；For all directions brightness of imageThree passages of all directions image are used algorithm of histogram equalization respectively, eliminate each side by unbalanced problemThere is the inconsistent phenomenon of brightness to image；The problem of inconsistent to the pixel brightness of overlapping region, useThe processing of piecemeal brightness processed method；The splicing mark problem existed to overlay region, according to pixel from respectiveFrontier distance ratio eliminates splicing vestige, so that it is complete to generate final 3D as weights using weighted mean methodJing Tu；
Step 5：3D panorama sketch is exported to display device after properly scaling.
Above-mentioned steps three, the three-dimensional surface model of similar " barrel-shaped " mentioned, its curve map is as shown in Figure 4.201 be first paragraph ground linearity sector, and 202 be second segment easement curve area, and 203 be that the 3rd section of height steeply risesCurve area.Due to vehicle body, nearby environment is generally ground, and remote environment is generally the object being above the ground level and builtThing is built, therefore such a harmony for setting small and different zones the image drawing of scalloping deformation to stretch is good.
3D panoramic parking assist systems can realize the Lothrus apterus co-ordination with original in-vehicle electronic system.So,When driver starts to park, display device can automatically switch to 3D panoramic parking assist systems.And driverThe 3D panorama sketch of display device only need to be observed, and correctly manipulates automobile, you can ensures safety of parking.
In summary, the present invention combines a variety of machine vision and the Processing Algorithm of Digital Image Processing, mergesThe multinomial advanced technology of field of machine vision, makes automobile parking process have higher security.
Although description of this invention combination embodiments above is carried out, it is familiar with the artPersonnel many replacements, modifications and variations can be carried out according to above-mentioned content, are obvious.Therefore,All such replacements, improvement and change are all regarded as not unconventional initiative spirit of the invention.