CN110782394A - Panoramic video rapid splicing method and system - Google Patents

Abstract

The invention discloses a panoramic video fast splicing method, which comprises the following steps: s1, performing off-line calibration on a camera set through a calibration object image, and acquiring and storing a projection transformation matrix of a camera; s2, acquiring images to be spliced acquired by a camera group aiming at a shooting area, performing off-line detection and registration after preprocessing, and processing the images through a projection transformation matrix of the camera to obtain and store an image mapping matrix of the images to be spliced; and S3, acquiring the current images to be spliced collected by the camera group in real time, projecting the images to be spliced into a standard coordinate system respectively, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to generate the panoramic spliced images. The invention also provides a system for realizing the method, so as to realize the nondestructive fusion of the splicing seams in the splicing process, the fusion algorithm has small calculated amount, high operation efficiency and better splicing effect, can well meet the requirements of panoramic monitoring in various civil and military fields, and is suitable for popularization.

Description

Panoramic video rapid splicing method and system

Technical Field

The invention relates to the technical field of video image synthesis, in particular to a method and a system for quickly splicing panoramic videos.

Background

The development and improvement of computer technology has led to the continuous progress of various video technologies. High-quality and high-definition images are gradually integrated into daily life of people, the video monitoring technology is no exception, and as the video monitoring technology is continuously enhanced, the requirements of people on video monitoring are higher and higher, on one hand, more and more information is required to be obtained, and on the other hand, the quality of the images is also required to be higher and higher. In the application scene of video monitoring, for example parking area, railway station, square, traffic crossing etc. place, because the scope of finding a view that single camera was shot is very little, managers often need to make a round trip to observe a plurality of control picture, and this can make the control personnel produce visual fatigue, is unfavorable for handling emergency. However, if a large range of scenes is to be photographed, the video camera is either a very expensive wide-angle camera or a fisheye lens camera at the cost of image distortion. Therefore, a large wide-angle video shooting technology which is cheap and has high quality picture quality is needed to meet the requirements of people. Therefore, a video stitching technique for synthesizing a wide-angle shot with a plurality of cameras is applied.

The video splicing technology is developed from an image splicing technology, wherein image splicing refers to a process of splicing two or more images with overlapped areas describing the same scene into a brand new image of a large scene through image registration and image fusion technologies. At present, the image stitching and synthesizing technology is widely applied to the fields of digital video processing, medical image analysis, remote sensing image processing and the like.

The traditional image splicing method generally extracts features such as ORB, Sift and the like from a picture, and then performs feature matching and fusion, for example, in the invention patent application with the publication number of CN103516995A, discloses a real-time panoramic video splicing method and device based on ORB features, wherein the method adopts an ORB feature extraction algorithm to perform feature point extraction on each path of image at the same moment and calculates ORB feature vectors of each feature point; splicing video frame scenes by adopting a nearest neighbor matching method, a RANSAC matching algorithm and the like; finally, the spliced video is output, a large amount of resources and time are consumed, and in some specific scenes such as battle test scenes and the like which mainly have high requirements on the panoramic splicing integrity, the splicing method is large in operation amount and poor in splicing effect, and cannot meet the requirements.

In view of the above, the present invention is particularly proposed.

Disclosure of Invention

Aiming at the defects in the prior art, the invention provides a novel panoramic video fast splicing method which is small in operation rate, high in operation efficiency and good in splicing effect. The invention also provides a system for implementing the method.

In order to achieve the purpose, the technical scheme of the invention is as follows:

a panoramic video fast splicing method comprises the following steps:

s1, performing off-line calibration on a camera set through a calibration object image, and acquiring and storing a projection transformation matrix of a camera;

s2, acquiring images to be spliced acquired by a camera group aiming at a shooting area, performing off-line detection and registration after preprocessing, and processing the images subjected to registration processing through a projection transformation matrix of the camera to obtain an image mapping matrix of the images to be spliced and storing the image mapping matrix;

and S3, acquiring the current images to be spliced collected by the camera group in real time, projecting the images to be spliced into a standard coordinate system respectively after preprocessing, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to generate the panoramic spliced images.

Further, in the panoramic video fast stitching method, the offline detection and registration in step s2 includes:

carrying out feature point detection and registration on the image obtained by preprocessing by using an SIFT feature registration algorithm;

and eliminating noise points in the images to be spliced by using a normalized cross-correlation algorithm, and purifying the geometry of the matching points.

Further, in the method for quickly splicing the panoramic video, step s3 includes:

respectively projecting the current images to be spliced to a standard coordinate system in a columnar projection mode, and then determining the overlapping area between the current images to be spliced according to the image mapping matrix;

performing exposure correction on two adjacent images to be spliced by adopting an HDR correction method, and fusing each image according to an overlapping area between the images and the global position of the image;

processing the splicing seams by adopting a weighted smoothing algorithm, and finally generating a panoramic spliced image; when the weighted smoothing algorithm is used for processing the splicing seams, the method comprises the following steps:

the gray value C of the pixel point in the image overlapping region is obtained by weighted average of the gray value A and the gray value B of the corresponding point in two adjacent images, namely C equals kA + (1-k) B, wherein k is an adjustable factor.

Further, in the method for quickly splicing the panoramic video, the step s1 includes:

s11, obtaining checkerboard images for camera calibration shot under a plurality of viewpoints;

s12, extracting corner points of the checkerboard calibration object image shot by the camera;

s13, calculating internal parameters, external parameters and distortion coefficients of the camera according to the acquired angular point information;

s14, optimizing external parameters of all cameras by using mixed errors;

and S15, calculating a projection transformation matrix corresponding to the camera according to the internal parameters and the external parameters.

Further, in the above method for quickly splicing panoramic video, s15 includes

Assuming that the average focal length of the current cameras is f,the rotation matrix is R, the translation vector is t, the image plane miscut parameter is s, and the focal point of the camera optical axis and the image plane is (u)₀,v₀)；

In the camera teaching model, a point P ═ X, Y, Z,1 in a three-dimensional space]^TPoint p ═ x, y,1 mapped onto two-dimensional plane of camera]^TThe following mathematical model was followed:

p＝MP,M＝K[RI-Rt]；

wherein, the matrix M is a camera matrix and is obtained by multiplying a camera internal reference matrix K and a camera external reference matrix [ RI-Rt ] matrix; the camera internal reference matrix K consists of several variables:

the camera group for video image splicing is set to meet the following conditions:

(u₀,v₀) (0,0), s ≈ 0, and the internal parameters of multiple cameras can be represented by the same internal parameter matrix:

t is approximately equal to 0 in the external reference matrix of the camera, and the external reference matrix is degenerated into a rotation matrix R;

then, the camera model is simplified to:

p＝MP＝KRP

let the camera matrix of two cameras be M respectively₁＝K₁R₁And M₂＝K₂R₂The same point P ═ X, Y, Z in the images taken by the two cameras]^TAnd the homogeneous coordinate of the point P in the two images respectively represents P₁＝[x₁,y₁,1]^TAnd p₂＝[x₂,y₂,1]^TThen the following formula holds:

the transformation of the corresponding points in the camera is represented by a homography matrix H of 3 x 3; the homography matrix H between the two images is represented in the form:

h is a non-singular matrix, for point p₁、p₂Using non-homogeneous coordinates instead of homogeneous coordinates, let p₁、p₂Is p as a non-homogeneous coordinate₁＝[x₁,y₁,1]And p₂＝[x₂,y₂,1]Then, the relationship of the two-dimensional projective transformation is expressed as:

let A be [ a ═ a₁₁,a₁₂,a₁₃,a₂₁,a₂₂,a₂₃,a₃₁,a₃₂,a₃₃]^TWhen n pairs of points (n > 4) are used:

and calculating a projection transformation matrix H by using a least square method, and storing for later steps to call.

On the other hand, the invention also provides a panoramic video fast splicing system, which comprises a processor and a memory, wherein the memory is stored with a program, and when the program is run by the processor, the following steps are executed:

acquiring a plurality of calibration object images in different directions shot by a camera group for off-line calibration, and acquiring and storing a projection transformation matrix of the camera;

acquiring images to be spliced acquired by a camera group aiming at a shooting area, preprocessing the images to be spliced, performing off-line detection and registration, processing the images subjected to registration processing through a projection transformation matrix of the camera to obtain an image mapping matrix of the images to be spliced and storing the image mapping matrix;

and acquiring the current images to be spliced collected by the camera group in real time, projecting the images to be spliced into a standard coordinate system respectively after preprocessing, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to obtain the panoramic image.

Further, in the above panoramic video fast mosaic system, when the program is run, the steps of the acquired images to be mosaic are executed: when performing off-line detection and registration, include

Carrying out feature point detection and registration on the image obtained by preprocessing by using an SIFT feature registration algorithm;

and eliminating noise points in the images to be spliced by using a normalized cross-correlation algorithm, and purifying the geometry of the matching points.

Further, in the above panoramic video fast mosaic system, when the program is run, the following is executed for the preprocessed current image to be mosaiced: respectively projecting the images to a standard coordinate system, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to obtain panoramic images, wherein the process comprises the steps of

Respectively projecting the current images to be spliced to a standard coordinate system in a columnar projection mode, and then determining the overlapping area between the current images to be spliced according to the image mapping matrix;

performing exposure correction on two adjacent images to be spliced by adopting an HDR correction method, and fusing each image according to an overlapping area between the images and the global position of the image;

processing the splicing seams by adopting a weighted smoothing algorithm, and finally generating a panoramic spliced image; when the weighted smoothing algorithm is used for processing the splicing seams, the method comprises the following steps:

the gray value C of the pixel point in the image overlapping region is obtained by weighted average of the gray value A and the gray value B of the corresponding point in two adjacent images, namely C equals kA + (1-k) B, wherein k is an adjustable factor.

Further, in the above panoramic video fast mosaic system, when the program is executed, the following is executed: the method comprises the following steps of obtaining calibration object images in a plurality of different directions shot by a camera group for off-line calibration, and obtaining and storing a projection transformation matrix of the camera, wherein the method comprises the following steps:

acquiring checkerboard images for camera calibration shot under a plurality of viewpoints;

carrying out corner extraction on a checkerboard calibration object image shot by a camera;

calculating internal parameters, external parameters and distortion coefficients of the camera according to the acquired angular point information;

optimizing external parameters of all cameras by using the mixed errors;

and calculating a projection transformation matrix corresponding to the camera according to the internal parameter data and the external parameter data.

Further, in the above panoramic video fast mosaic system, when the program is executed, the following is executed: when the projective transformation matrix corresponding to the camera is calculated according to the internal parameter data and the external parameter data, the method comprises the following steps:

let the current camera average focal length be f, the rotation matrix be R, the translation vector be t, the image plane miscut parameter be s, and the camera optical axis and the focus of the image plane be (u)₀,v₀)；

In the camera teaching model, a point P ═ X, Y, Z,1 in a three-dimensional space]^TPoint p ═ x, y,1 mapped onto two-dimensional plane of camera]^TThe following mathematical model was followed:

p＝MP,M＝K[RI-Rt]；

wherein, the matrix M is a camera matrix and is obtained by multiplying a camera internal reference matrix K and a camera external reference matrix [ RI-Rt ] matrix; the camera internal reference matrix K consists of several variables:

the camera group for video image splicing is set to meet the following conditions:

(u₀,v₀) (0,0), s ≈ 0, and the internal parameters of multiple cameras can be represented by the same internal parameter matrix:

t is approximately equal to 0 in the external reference matrix of the camera, and the external reference matrix is degenerated into a rotation matrix R;

then, the camera model is simplified to:

p＝MP＝KRP

let the camera matrix of two cameras be M respectively₁＝K₁R₁And M₂＝K₂R₂The same point P ═ X, Y, Z in the images taken by the two cameras]^TAnd the homogeneous coordinate of the point P in the two images respectively represents P₁＝[x₁,y₁,1]^TAnd p₂＝[x₂,y₂,1]^TThen the following formula holds:

the transformation of the corresponding points in the camera is represented by a homography matrix H of 3 x 3; the homography matrix H between the two images is represented in the form:

h is a non-singular matrix, for point p₁、p₂Replacing homogeneous coordinates with non-homogeneous coordinates, let p1, p₂Is p as a non-homogeneous coordinate₁＝[x₁,y₁,1]And p₂＝[x₂,y₂,1]Then, the relationship of the two-dimensional projective transformation is expressed as:

let A be [ a ═ a₁₁,a₁₂,a₁₃,a₂₁,a₂₂,a₂₃,a₃₁,a₃₂,a₃₃]^TWhen n pairs of points (n > 4) are used:

and calculating a projection transformation matrix H by using a least square method, and storing for later steps to call.

Compared with the prior art, the invention has the beneficial effects that:

the method solves the problems of poor splicing effect and low shooting frame frequency of the area array scanning camera of the traditional high-definition image by combining offline calibration and real-time splicing; the gradual-in and gradual-out algorithm based on morphology realizes the lossless fusion of the splicing seams, has small calculation amount, high operation efficiency and better splicing effect, can well meet the requirements of panoramic monitoring in various civil and military fields, and is suitable for popularization.

Drawings

In order to more clearly illustrate the detailed description of the invention or the technical solutions in the prior art, the drawings that are needed in the detailed description of the invention or the prior art will be briefly described below. Throughout the drawings, like elements or portions are generally identified by like reference numerals. In the drawings, elements or portions are not necessarily drawn to scale.

FIG. 1 is a schematic view; the panoramic video fast splicing method provides a calibration board used in the camera calibration step in a specific implementation;

FIG. 2 is a schematic view; the invention relates to a flow chart of steps executed when a program of a panoramic video quick splicing system is operated.

FIG. 3 is a diagram of: the method carries out weighting smoothing algorithm to process the schematic diagram of the splicing seam;

Detailed Description

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following examples are only for illustrating the technical solutions of the present invention more clearly, and therefore are only examples, and the protection scope of the present invention is not limited thereby.

It is to be noted that, unless otherwise specified, technical or scientific terms used herein shall have the ordinary meaning as understood by those skilled in the art to which the invention pertains.

A panoramic video fast splicing method comprises the following steps:

s1, performing off-line calibration on a camera set through a calibration object image, and acquiring and storing a projection transformation matrix of a camera;

s2, acquiring images to be spliced acquired by a camera group aiming at a shooting area, performing off-line detection and registration after preprocessing, and processing the images subjected to registration processing through a projection transformation matrix of the camera to obtain an image mapping matrix of the images to be spliced and storing the image mapping matrix;

and S3, acquiring the current images to be spliced collected by the camera group in real time, projecting the images to be spliced into a standard coordinate system respectively after preprocessing, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing.

In the method, the panoramic image splicing is realized by combining the off-line calibration and the real-time splicing. Adjusting the angle of the camera set to fix so that the acquired images have an overlapping area; calculating the mapping relation of the camera through off-line calibration, and storing the mapping relation; and then, the mapping matrix is loaded firstly when the real-time splicing of each video image is started, so that a large amount of early-stage work of feature extraction, matching and correction is saved.

In order to ensure the splicing effect, in the step s1, in the process of performing offline calibration on the camera set through the calibration object image, acquiring and storing the projection transformation matrix of the camera, a specific embodiment provided by the method specifically comprises the following steps:

s11, as shown in figure 1, a check series film calibration plate (the external dimension is 400mm multiplied by 400mm, the square side length is 30mm, and the precision is +/-0.02) is selected as the calibration object. Selecting a proper angle to fix the camera, shooting images of the checkerboard calibration objects, and shooting 15 photos in different directions by the camera through adjusting the direction of the calibration objects (experiments show that when the number of N is more than 10 and less than 20, N is the number of photos, and the calibration results are more accurate);

s12, extracting corners of the chessboard pattern calibration object images shot by the camera group, wherein the corners can be automatically extracted, and the users can be allowed to extract chessboard corners interactively;

s13, calculating internal parameters, external parameters and distortion coefficients of the camera according to the acquired angular point information; in this embodiment, rapid calculation is performed by an MATLAB calibration toolbox, which is a mature technique in the field and is not described again;

s14, optimizing external parameters of the camera by using the mixed errors, reducing the influence of objective factors on subsequent operation results, improving the calculation accuracy and further improving the splicing effect. The mixed error comprises a projection error and a correction error, the projection error refers to the projection position error of the same angular point in different views, the correction error is the difference of the same angular point in the y direction in left and right views after epipolar correction, and the mixed error is the weighted mixture of the two errors. Then

S141, projection error: for a pair of left and right cameras, defining the difference between the corner point detected in the right view and the position (projection position) in the right view transformed from the corresponding point detected in the left view;

s142, recording the projection error as a correction error: for a pair of left and right cameras, the difference in y direction (perpendicular to the epipolar line) in left and right views after epipolar line correction is defined for the same intersection point.

Let the projection error be Eproj and the correction error be Erect, then the mixed error Emix is:

Emix＝αEproj+(1-α)Erect

α is an adjustable weight, and the Levenberg-Marquardt algorithm for solving the nonlinear least squares is used as an optimization method to carry out the optimization solution of the camera external parameters.

S15, when a projective transformation matrix corresponding to the camera is calculated according to the internal parameter and the external parameter data, calculating the projective transformation matrix according to the internal parameter and the external parameter data optimized in the steps, and storing the projective transformation matrix in a configuration file for use in the subsequent steps; the method specifically comprises the following steps:

s151, setting the average focal length of all current cameras as f, setting the rotation matrix as R, setting the translation vector as t, setting the image plane miscut parameter as s, and setting the image plane miscut parameter to represent the degree that the light sensing plane of the cameras is not vertical to the optical axis of the lens; the focal point of the camera optical axis and the image plane is (u)₀,v₀)。

In the camera teaching model, a point P ═ X, Y, Z,1 in a three-dimensional space]^TPoint p ═ x, y,1 mapped onto two-dimensional plane of camera]^TThe following mathematical model was followed:

p＝MP,M＝K[RI-Rt]；

wherein, the matrix M is a camera matrix and is obtained by multiplying a camera internal reference matrix K and a camera external reference matrix [ RI-Rt ] matrix; the camera internal reference matrix K consists of several variables:

obtaining a camera matrix M by utilizing the internal parameters and the external parameters of the camera obtained in the step S1; and obtaining a corresponding projection transformation matrix by obtaining the camera matrix M.

In the method, a camera group used for video image splicing is set to meet the following conditions:

(1) the camera quality is high and the focal length between multiple cameras is similar, then (u)₀,v₀) (0,0), s ≈ 0, and the internal parameters of multiple cameras can be represented by the same internal parameter matrix:

(2) the depth of field of the scenery shot by the camera set is deep enough, so that the difference of positions among the camera sets can be ignored; then t ≈ 0 in the outlier matrix, so the outlier matrix may degenerate into a rotation matrix R.

After the above two conditions are satisfied, the camera model is simplified as follows:

p＝MP＝KRP

let the camera matrix of two cameras be M respectively₁＝K₁R₁And M₂＝K₂R₂The same point P ═ X, Y, Z in the images taken by the two cameras]^TAnd the homogeneous coordinate of the point P in the two images respectively represents P₁＝[x₁,y₁,1]^TAnd p₂＝[x₂,y₂,1]^TThen the following formula holds:

thus, the transformation of the corresponding point in the camera can be represented by a homography matrix H of 3 × 3; the homography matrix H between two images can be expressed in the form:

h is a non-singular matrix, and thus for point p₁、p₂Using non-homogeneous coordinates instead of homogeneous coordinates, let p₁、p₂Is p as a non-homogeneous coordinate₁＝[x₁,y₁,1]And p₂＝[x₂,y₂,1]Then the relationship of the two-dimensional projective transformation can be expressed as:

let A be [ a ═ a₁₁,a₁₂,a₁₃,a₂₁,a₂₂,a₂₃,a₃₁,a₃₂,a₃₃]^TSince a pair of matching points can be determined2 independent linear equations, under the projection transformation model, 4 pairs of points are theoretically needed to solve the projection transformation matrix H. For n point pairs (n > 4) in practical engineering:

and finally, calculating a projection transformation matrix H by using a least square method, and storing the projection transformation matrix H in a local storage area.

S2, acquiring images to be spliced acquired by a camera group aiming at a shooting area, performing off-line detection and registration after preprocessing, and processing the images subjected to registration processing through a projection transformation matrix of the camera to obtain an image mapping matrix of the images to be spliced and store the image mapping matrix:

the fixed camera group shoots images aiming at respective shooting areas, and the images are a group of images to be spliced, such as 6 images and the like, and after pretreatment, offline detection, registration and processing are carried out to obtain an image mapping matrix of the images to be spliced; the registration processing enables points corresponding to the same position in space in two adjacent images to be in one-to-one correspondence, and the overlapping area between the adjacent images in the image to be processed is determined through the camera projection transformation matrix, so that the subsequent splicing fusion is facilitated; specifically, comprise

S21, image drying: in the step, a group of original images to be spliced, which are shot by a camera set aiming at respective shooting areas, are obtained firstly, the original images are converted into gray-scale images, so that the contrast is enhanced, the subsequent characteristic points are conveniently extracted, and then the original images are subjected to drying treatment by using wiener filtering and converted back into color images;

s22, image distortion correction: and (3) establishing an image distortion correction model, and then carrying out geometric distortion removal treatment on each graph of the image to be spliced by using the camera internal reference data and the camera external reference data obtained in the step S1 through an MATLAB software tool for correction.

After the preprocessed pictures to be spliced are obtained, the characteristic point detection and the registration are carried out through the step S23, and then the matching point set is purified, wherein the method comprises the following steps:

s221, carrying out feature point detection and registration on the image obtained after the processing in the step S21 and/or the step S22 by using an SIFT feature registration algorithm;

s222, eliminating noise points in the images to be spliced by using a normalized cross-correlation (NCC algorithm) algorithm, and purifying the matching points.

The method adopts an off-line calibration method, and does not depend on the time consumed in the global registration stage of the multi-view camera, so the SIFT algorithm with the best effect is selected for the image registration method based on the feature point information in the embodiment to obtain the matching points, but the matching points obtained by the traditional SIFT algorithm have the condition of mismatching, therefore, further, the matching point set is purified by the normalized cross-correlation (NCC) technology of the gray information, the matching point pairs of the SIFT are purified by utilizing the gray information, for a correct SIFT matching point pair, the gray values on the templates with the same size on the image are very close, and therefore, the normalized cross-correlation can be used for checking whether the matching points are the mismatching points.

And (4) obtaining a mapping matrix of the image according to the obtained camera projection transformation matrix after the image to be processed is subjected to feature point extraction and matching. After the mapping matrix of the image is stored, the mapping matrix is loaded at the beginning of real-time online splicing of the video each time, so that a large amount of early-stage work such as feature extraction, matching, correction and the like is omitted, the image splicing and fusion steps are directly carried out, and the image splicing processing efficiency is improved.

And S3, acquiring the current images to be spliced collected by the camera group in real time, projecting the images to be spliced into a standard coordinate system respectively after preprocessing, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing.

Because the image sequence is two-dimensional projection of an entity scene under different coordinate systems, and the direct splicing of the shot images cannot meet the visual consistency, the current images to be spliced need to be respectively projected under a standard coordinate system, and then the image splicing is carried out. The transformation of the cylindrical coordinates has the advantages of simplicity and quickness in operation, the projection images obtained by performing cylindrical projection on the original image at different positions are the same, and the spliced cylindrical panoramic image has a better visual effect and can describe scene information in detail at 360 degrees in the horizontal direction, so that the current image to be spliced is processed by the cylindrical projection.

The cylindrical projection formula is as follows:

wherein, x 'and y' are the image coordinates after the cylindrical projection, x and y are the original coordinates of the image, W and H are the width and height of the original image, and f is the focal length of the camera. When projection is carried out, a blank image with the same size as the original image of the current image to be spliced is established for storing the projected image, each point of the blank image is used for obtaining a corresponding point of the blank image on the original image by using a back projection formula, and therefore the brightness value of a pixel point on the blank image is determined, and the image coordinate of the columnar projected image of the current image to be spliced can be obtained.

S31, the images to be spliced are respectively projected to a standard coordinate system by adopting the mode, and then the overlapping area between the images is determined according to the mapping matrix of the images.

And S32, carrying out exposure correction on two adjacent images to be spliced by adopting an HDR correction method, thereby reducing the brightness difference between the images. Then, fusing each image according to the overlapping area between the images and the global position of the image, wherein the step of fusing the images is mature technology in the field and is not described again;

and S33, processing the splicing seams by adopting a weighted smoothing algorithm, and finally generating a panoramic splicing image.

In order to obtain a larger exposure range, the array panoramic camera is set to an automatic exposure mode. However, this has a serious consequence that adjacent cameras may have a particularly large difference in exposure (up to 3 times) due to their different orientations. This not only causes the concatenation difficulty, still can bring very big uncomfortable sense to the 3D effect because left and right eyes exposure difference is too big. Therefore, before image fusion, the method adopts the HDR correction method to carry out exposure correction on two adjacent images to be spliced, thereby reducing the brightness difference between the images.

Secondly, in order to avoid obvious brightness change at two ends of the spliced image suture line, the suture line needs to be processed in the fusion process; the processing method which can be adopted comprises color interpolation, multi-resolution spline technology and the like, and the invention adopts a faster and simpler weighting smoothing algorithm to process the problem of the splicing seam: the gray value C of the pixel point in the image overlapping region is obtained by weighted averaging the gray values a and B of the corresponding points in the two images, that is, C ═ kA + (1-k) B, where k is an adjustable factor, as shown in fig. 3.

The method can utilize multiple cameras to locally shoot the target image, overcomes the difficulty of shooting large-amplitude images by using a single camera, and realizes the scanning-level panoramic stitching synthesis of multiple planar images with overlapping areas. Meanwhile, the problems of poor splicing effect of the traditional high-definition images and low shooting frame frequency of an area-array scanning camera are solved by a method combining offline calibration and real-time splicing; the invention realizes the lossless fusion of splicing seams based on the morphological gradual-in gradual-out algorithm (weighted smoothing algorithm), has small calculation amount of the fusion algorithm, high operation efficiency and better splicing effect, and can well meet the requirements of panoramic monitoring in various civil and military fields.

On the other hand, the invention also provides a system for quickly splicing the panoramic video, which is used for implementing the method. The system comprises a processor and a memory, wherein the memory stores a program, and when the program is executed by the processor, as shown in fig. 2, the following steps are executed:

acquiring a plurality of calibration object images in different directions shot by a camera group for off-line calibration, and acquiring and storing a projection transformation matrix of the camera;

acquiring images to be spliced acquired by a camera group aiming at a shooting area, preprocessing the images to be spliced, performing off-line detection and registration, processing the images subjected to registration processing through a projection transformation matrix of the camera to obtain an image mapping matrix of the images to be spliced and storing the image mapping matrix;

and acquiring the current images to be spliced collected by the camera group in real time, projecting the images to be spliced into a standard coordinate system respectively after preprocessing, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to acquire panoramic images.

In a specific embodiment of the system of the present invention, when performing offline calibration on a plurality of calibration object images in different directions captured by the capturing camera group, and obtaining and storing a projection transformation matrix of the camera, the method includes:

the checkerboard images for camera calibration shot under multiple viewpoints are obtained and used for calculating an internal parameter matrix K, external parameters (camera rotation R and camera translation T), distortion coefficients and the like of a camera.

Wherein the calibration object adopts a check series film calibration board (the external dimension is 400mm multiplied by 400 mm: the square side length is 30mm, the precision is +/-0.02), then a camera set is fixed, the image of the check calibration object is shot, and 15 photos in different directions are shot by the camera through adjusting the direction of the calibration object (experiments show that when N is more than 10 and less than 20, N is the number of photos, the calibration result is more accurate);

then, extracting the corner points of the checkerboard calibration object image shot by the camera, wherein the extraction can be automatic, and the extraction of the checkerboard corner points can be carried out by a user in an interactive mode;

calculating internal parameters, external parameters and distortion coefficients of the camera according to the acquired angular point information; in the embodiment, rapid calculation is performed through an MATLAB calibration tool box;

and the external parameters of all cameras are optimized by using the mixed errors, so that the influence of objective factors on subsequent operation results is reduced, the calculation accuracy is improved, and the splicing effect is further improved. The mixed error comprises a projection error and a correction error, the projection error refers to the projection position error of the same angular point in different views, the correction error is the difference of the same angular point in the y direction in left and right views after epipolar correction, and the mixed error is the weighted mixture of the two errors. Then

Let the projection error be Eproj and the correction error be Erect, then the mixed error Emix is:

Emix＝αEproj+(1-α)Erect

α is an adjustable weight, and the Levenberg-Marquardt algorithm for solving the nonlinear least squares is used as an optimization method to carry out the optimization solution of the camera external parameters.

The program further executes: calculating a projection transformation matrix corresponding to the camera according to the internal parameter data and the external parameter data; calculating a projective transformation matrix from the internal and external parameter data optimized in the steps, and storing the projective transformation matrix in a configuration file for use in the subsequent steps; the method specifically comprises the following steps:

setting the current focal length of the camera as f, the rotation matrix as R, the translation vector as t, the image plane miscut parameter as s, and the image plane miscut parameter representing the degree of the camera light sensing plane not perpendicular to the lens optical axis; the focal point of the camera optical axis and the image plane is (u)₀,v₀)。

In the camera teaching model, a point P ═ X, Y, Z,1 in a three-dimensional space]^TPoint p ═ x, y,1 mapped onto two-dimensional plane of camera]^TThe following mathematical model was followed:

p＝MP,M＝K[RI-Rt]；

wherein, the matrix M is a camera matrix and is obtained by multiplying a camera internal reference matrix K and a camera external reference matrix [ RI-Rt ] matrix; the camera internal reference matrix K consists of several variables:

the camera matrix M can be solved by using the obtained internal parameters and external parameters of the camera; and obtaining a corresponding projection transformation matrix by obtaining the camera matrix M.

In the process of solving the projection transformation matrix of the camera, the camera group for video image splicing is set to meet the following conditions:

(1) the camera quality is high and the focal length between multiple cameras is similar, then (u)₀,v₀) (0,0), s ≈ 0, and the internal parameters of multiple cameras can be represented by the same internal parameter matrix:

(2) the depth of field of the scenery shot by the camera set is deep enough, so that the difference of positions among the camera sets can be ignored; then t ≈ 0 in the outlier matrix, so the outlier matrix may degenerate into a rotation matrix R.

After the above two conditions are satisfied, the camera model is simplified as follows:

p＝MP＝KRP

let the camera matrix of two cameras be M respectively₁＝K₁R₁And M₂＝K₂R₂The same point P ═ X, Y, Z in the images taken by the two cameras]^TAnd the homogeneous coordinate of the point P in the two images respectively represents P₁＝[x₁,y₁,1]^TAnd p₂＝[x₂,y₂,1]^TThen the following formula holds:

thus, the transformation of the corresponding point in the camera can be represented by a homography matrix H of 3 × 3; the homography matrix H between two images can be expressed in the form:

h is a non-singular matrix, and thus for point p₁、p₂Using non-homogeneous coordinates instead of homogeneous coordinates, let p₁、p₂Is p as a non-homogeneous coordinate₁＝[x₁,y₁,1]And p₂＝[x₂,y₂,1]Then the relationship of the two-dimensional projective transformation can be expressed as:

let A be [ a ═ a₁₁,a₁₂,a₁₃,a₂₁,a₂₂,a₂₃,a₃₁,a₃₂,a₃₃]^TBecause 2 independent linear equations can be determined by one pair of matching points, 4 pairs of points are theoretically needed to solve the projective transformation matrix H under the projective transformation model. For n point pairs (n > 4) in practical engineering:

and finally, calculating a projection transformation matrix H by using a least square method, and storing the projection transformation matrix H in a local storage area.

When the program executes the image to be spliced acquired by the camera group aiming at the shooting area, the off-line registration is carried out after the preprocessing, the image after the registration processing is processed by the projection transformation matrix of the camera, and the image mapping matrix of the image to be spliced is obtained and stored, the method specifically comprises the following steps of

Image drying: firstly, acquiring a group of original images to be spliced, which are shot by a camera set aiming at respective shooting areas, converting the original images into gray-scale images so as to enhance contrast and facilitate the extraction of subsequent characteristic points, and then performing drying treatment on the original images by using wiener filtering and converting the original images back into color images;

and (3) image distortion correction: and establishing an image distortion correction model, and then performing geometric distortion removal treatment on each graph of the image to be spliced by using the camera internal reference data and the camera external reference data obtained in the previous steps through an MATLAB software tool to perform correction.

After the preprocessed pictures to be spliced are obtained, feature point detection and registration are carried out, and then a matching point set is stored, wherein the method comprises the following steps:

carrying out feature point detection and registration on the preprocessed image by using an SIFT feature registration algorithm;

and (3) eliminating noise points in the images to be spliced by using a normalized cross-correlation (NCC) algorithm so as to purify the matching quasi points.

In the embodiment, the SIFT algorithm with the best effect is selected to obtain the matching points, but the matching points obtained by the traditional SIFT algorithm have the condition of mismatching, so that the invention further utilizes the Normalized Cross Correlation (NCC) technology of gray information to purify the matching point set; the matching point set is purified through a Normalized Cross Correlation (NCC) technology of gray information, the matching point pairs of SIFT are purified through the gray information, for a correct SIFT matching point pair, the gray values on the templates with the same size on the image are very close, and therefore the normalized cross correlation can be used for checking whether the matching points are mismatching points.

And the image to be processed after the characteristic point extraction and matching can obtain a mapping matrix of the image through the previously obtained camera projection transformation matrix. The method is used for loading the mapping matrix when the online video splicing starts each time, so that a large amount of early-stage work such as feature extraction, matching, correction and the like is omitted, and the steps of image splicing and fusion are directly carried out.

Executing the program to acquire the current images to be spliced collected by the camera set in real time, projecting the images to be spliced into a standard coordinate system respectively after preprocessing, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to obtain a panoramic image:

the method comprises the steps of preprocessing the current images to be spliced, wherein the preprocessing mode is the same as that before offline registration, and then respectively projecting the images to be spliced to a standard coordinate system.

The cylindrical projection formula is as follows:

wherein, x 'and y' are the image coordinates after the cylindrical projection, x and y are the original coordinates of the image, W and H are the width and height of the original image, and f is the focal length of the camera. When projection is carried out, a blank image with the same size as the original image of the current image to be spliced is established for storing the projected image, each point of the blank image is used for obtaining a corresponding point of the blank image on the original image by using a back projection formula, and therefore the brightness value of a pixel point on the blank image is determined, and the image coordinate of the columnar projected image of the current image to be spliced can be obtained.

The images to be spliced are respectively projected to a standard coordinate system by adopting the mode, and then the overlapping area between the images is determined according to the mapping matrix of the images.

And then, carrying out exposure correction on two adjacent images to be spliced by adopting an HDR correction method so as to reduce the brightness difference between the images. Then, all the images are fused according to the overlapping area between the images and the global position of the images, the problem of splicing seams is processed by adopting a weighted smoothing algorithm, and finally, a panoramic spliced image is generated.

When the suture is processed, a quick and simple weighted smoothing algorithm is adopted: the gray value C of the pixel point in the image overlapping region is obtained by weighted average of the gray values a and B of the corresponding points in the two images, that is, C ═ kA + (1-k) B, where k is an adjustable factor.

The system is used for implementing the method, has high operation efficiency and good splicing effect, and can well meet the requirements of panoramic monitoring in various civil and military fields in the process of splicing the panoramic video images; the step principle executed when the program is operated in the system of the invention corresponds to the method of the invention.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; while the invention has been described in detail and with reference to the foregoing embodiments, it will be understood by those skilled in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some or all of the technical features may be equivalently replaced; such modifications and substitutions do not depart from the spirit and scope of the present invention, and they should be construed as being included in the following claims and description.

Claims (10)

1. A panoramic video fast splicing method is characterized by comprising the following steps:

s1, performing off-line calibration on a camera set through a calibration object image, and acquiring and storing a projection transformation matrix of a camera;

s2, acquiring images to be spliced acquired by a camera group aiming at a shooting area, performing off-line detection and registration after preprocessing, and processing the images subjected to registration processing through a projection transformation matrix of the camera to obtain an image mapping matrix of the images to be spliced and storing the image mapping matrix;

and S3, acquiring the current images to be spliced collected by the camera group in real time, projecting the images to be spliced into a standard coordinate system respectively after preprocessing, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to generate the panoramic spliced images.

2. The panoramic video fast stitching method according to claim 1, wherein the offline detection and registration in step s2 comprises:

carrying out feature point detection and registration on the image obtained by preprocessing by using an SIFT feature registration algorithm;

and eliminating noise points in the images to be spliced by using a normalized cross-correlation algorithm, and purifying the geometry of the matching points.

3. The panoramic video fast splicing method according to claim 2, wherein the step S3 comprises:

respectively projecting the current images to be spliced to a standard coordinate system in a columnar projection mode, and then determining the overlapping area between the current images to be spliced according to the image mapping matrix;

performing exposure correction on two adjacent images to be spliced by adopting an HDR correction method, and fusing each image according to an overlapping area between the images and the global position of the image;

processing the splicing seams by adopting a weighted smoothing algorithm, and finally generating a panoramic spliced image; when the weighted smoothing algorithm is used for processing the splicing seams, the method comprises the following steps:

the gray value C of the pixel point in the image overlapping region is obtained by weighted average of the gray value A and the gray value B of the corresponding point in two adjacent images, namely C equals kA + (1-k) B, wherein k is an adjustable factor.

4. The panoramic video fast splicing method according to any one of claims 1 to 3, wherein the step S1 comprises:

s11, obtaining checkerboard images for camera calibration shot under a plurality of viewpoints;

s12, extracting corner points of the checkerboard calibration object image shot by the camera;

s13, calculating internal parameters, external parameters and distortion coefficients of the camera according to the acquired angular point information;

s14, optimizing external parameters of all cameras by using mixed errors;

and S15, calculating a projection transformation matrix corresponding to the camera according to the internal parameters and the external parameters.

5. The panoramic video fast stitching method according to claim 4, wherein S15. comprises setting the average focal length of the current camera to be f, the rotation matrix to be R, the translation vector to be t, the image plane miscut parameter to be s, and the focal point of the camera optical axis and the image plane to be (u)₀,v₀)；

In the camera teaching model, a point P ═ X, Y, Z,1 in a three-dimensional space]^TPoint p ═ x, y,1 mapped onto two-dimensional plane of camera]^TThe following mathematical model was followed:

p＝MP,M＝K[RI-Rt]；

wherein, the matrix M is a camera matrix and is obtained by multiplying a camera internal reference matrix K and a camera external reference matrix [ RI-Rt ] matrix; the camera internal reference matrix K consists of several variables:

the camera group for video image splicing is set to meet the following conditions:

(u₀,v₀) (0,0), s ≈ 0, and the internal parameters of multiple cameras can be represented by the same internal parameter matrix:

t is approximately equal to 0 in the external reference matrix of the camera, and the external reference matrix is degenerated into a rotation matrix R;

then, the camera model is simplified to:

p＝MP＝KRP

let the camera matrix of two cameras be M respectively₁＝K₁R₁And M₂＝K₂R₂The same point P ═ X, Y, Z in the images taken by the two cameras]^TAnd the homogeneous coordinate of the point P in the two images respectively represents P₁＝[x₁,y₁,1]^TAnd p₂＝[x₂,y₂,1]^TThen the following formula holds:

the transformation of the corresponding points in the camera is represented by a homography matrix H of 3 x 3; the homography matrix H between the two images is represented in the form:

h is a non-singular matrix, for point p₁、p₂Using non-homogeneous coordinates instead of homogeneous coordinates, let p₁、p₂Is p as a non-homogeneous coordinate₁＝[x₁,y₁,1]And p₂＝[x₂,y₂,1]Then, the relationship of the two-dimensional projective transformation is expressed as:

let A be [ a ═ a₁₁,a₁₂,a₁₃,a₂₁,a₂₂,a₂₃,a₃₁,a₃₂,a₃₃]^TWhen n pairs of points (n > 4) are used:

and calculating a projection transformation matrix H by using a least square method, and storing for later steps to call.

6. A panoramic video fast splicing system is characterized by comprising a processor and a memory, wherein the memory stores a program, and when the program is executed by the processor, the following steps are executed:

acquiring a plurality of calibration object images in different directions shot by a camera group for off-line calibration, and acquiring and storing a projection transformation matrix of the camera;

acquiring images to be spliced acquired by a camera group aiming at a shooting area, preprocessing the images to be spliced, performing off-line detection and registration, processing the images subjected to registration processing through a projection transformation matrix of the camera to obtain an image mapping matrix of the images to be spliced and storing the image mapping matrix;

and acquiring the current images to be spliced collected by the camera group in real time, projecting the images to be spliced into a standard coordinate system respectively after preprocessing, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to obtain the panoramic image.

7. The panoramic video rapid stitching system according to claim 6, wherein when the program is executed, the steps of: when performing off-line detection and registration, include

Carrying out feature point detection and registration on the image obtained by preprocessing by using an SIFT feature registration algorithm;

and eliminating noise points in the images to be spliced by using a normalized cross-correlation algorithm, and purifying the geometry of the matching points.

8. The panoramic video rapid stitching system according to claim 7, wherein when the program is executed, for the preprocessed current image to be stitched, the following steps are performed: respectively projecting the images to a standard coordinate system, calling the image mapping matrix to determine the overlapping area of the current images to be spliced, and then carrying out splicing fusion processing to obtain panoramic images, wherein the process comprises the steps of

Respectively projecting the current images to be spliced to a standard coordinate system in a columnar projection mode, and then determining the overlapping area between the current images to be spliced according to the image mapping matrix;

performing exposure correction on two adjacent images to be spliced by adopting an HDR correction method, and fusing each image according to an overlapping area between the images and the global position of the image;

processing the splicing seams by adopting a weighted smoothing algorithm, and finally generating a panoramic spliced image; when the weighted smoothing algorithm is used for processing the splicing seams, the method comprises the following steps:

the gray value C of the pixel point in the image overlapping region is obtained by weighted average of the gray value A and the gray value B of the corresponding point in two adjacent images, namely C equals kA + (1-k) B, wherein k is an adjustable factor.

9. The scene video rapid mosaic system according to any one of claims 6-8, wherein said program, when executed, performs said: the method comprises the following steps of obtaining calibration object images in a plurality of different directions shot by a camera group for off-line calibration, and obtaining and storing a projection transformation matrix of the camera, wherein the method comprises the following steps:

acquiring checkerboard images for camera calibration shot under a plurality of viewpoints;

carrying out corner extraction on a checkerboard calibration object image shot by a camera;

calculating internal parameters, external parameters and distortion coefficients of the camera according to the acquired angular point information;

optimizing external parameters of all cameras by using the mixed errors;

and calculating a projection transformation matrix corresponding to the camera according to the internal parameter data and the external parameter data.

10. The panoramic video quick mosaic system of claim 9, wherein said program, when executed, performs said: when the projective transformation matrix corresponding to the camera is calculated according to the internal parameter data and the external parameter data, the method comprises the following steps:

let the current camera average focal length be f, the rotation matrix be R, the translation vector be t, the image plane miscut parameter be s, and the camera optical axis and the focus of the image plane be (u)₀,v₀)；

In the camera teaching model, a point P ═ X, Y, Z,1 in a three-dimensional space]^TPoint p ═ x, y,1 mapped onto two-dimensional plane of camera]^TThe following mathematical model was followed:

p＝MP,M＝K[RI-Rt]；

wherein, the matrix M is a camera matrix and is obtained by multiplying a camera internal reference matrix K and a camera external reference matrix [ RI-Rt ] matrix; the camera internal reference matrix K consists of several variables:

the camera group for video image splicing is set to meet the following conditions:

(u₀,v₀) (0,0), s ≈ 0, and the internal parameters of multiple cameras can be represented by the same internal parameter matrix:

t is approximately equal to 0 in the external reference matrix of the camera, and the external reference matrix is degenerated into a rotation matrix R;

then, the camera model is simplified to:

p＝MP＝KRP

let the camera matrix of two cameras be M respectively₁＝K₁R₁And M₂＝K₂R₂Through these two phasesThe same point P ═ X, Y, Z in the captured image]^TAnd the homogeneous coordinate of the point P in the two images respectively represents P₁＝[x₁,y₁,1]^TAnd p₂＝[x₂,y₂,1]^TThen the following formula holds:

the transformation of the corresponding points in the camera is represented by a homography matrix H of 3 x 3; the homography matrix H between the two images is represented in the form:

h is a non-singular matrix, for point p₁、p₂Replacing homogeneous coordinates with non-homogeneous coordinates, let p1, p₂Is p as a non-homogeneous coordinate₁＝[x₁,y₁,1]And p₂＝[x₂,y₂,1]Then, the relationship of the two-dimensional projective transformation is expressed as:

let A be [ a ═ a₁₁,a₁₂,a₁₃,a₂₁,a₂₂,a₂₃,a₃₁,a₃₂,a₃₃]^TWhen n pairs of points (n > 4) are used:

and calculating a projection transformation matrix H by using a least square method, and storing for later steps to call.

Images