Disclosure of Invention
The application aims to provide an image stitching method, an image stitching system and a readable storage medium. By designing a calibration plate consisting of a black marker and a plurality of small holes and a conversion relation calculation method, the extraction and one-to-one pairing of the small Kong Zhixin coordinates are respectively completed between two images, and the conversion matrix is directly calculated through multiple points, so that the accuracy and the speed of image splicing are improved. The first aspect of the application provides an image stitching method, which comprises the following steps:
Obtaining calibration plate images of a first view and a second view, wherein the inside of the calibration plate images is distributed with black rectangular marker shapes of a plurality of ordered round holes;
performing relation conversion on the calibration plate images of the first view and the second view to obtain a homography matrix between the images of the first view and the second view, and using the homography matrix as a conversion matrix between the two spliced images;
Acquiring spliced images of a first visual field and a second visual field, which are respectively the first spliced image and the second spliced image, from the same visual angle as the calibration plate image;
Converting the first spliced image and the second spliced image by using a conversion matrix, and splicing according to a conversion result to obtain a preliminary spliced image;
According to the preliminary stitched image, respectively calculating a homography matrix H1 from the first stitched image to the preliminary stitched image and a homography matrix H2 from the second stitched image to the preliminary stitched image;
And transforming the first spliced image and the second spliced image into the preliminary spliced image through a homography matrix H1 and a homography matrix H2 respectively, and carrying out pixel fusion on the transformed overlapping area to obtain a spliced image.
Optionally, the performing relationship conversion on the calibration plate images of the first field of view and the second field of view to obtain a homography matrix between the images of the first field of view and the second field of view specifically includes:
threshold segmentation is carried out on the calibration plate image to obtain a first brightness area and a second brightness area of the calibration plate image, wherein the brightness value of the second brightness area is higher than that of the first brightness area
Taking the connected domain of the first brightness area as a darkening connected domain, and filling holes in the darkening connected domain of the calibration plate image;
filtering the darkening connected domain according to the area, and reserving the darkening connected domain with the area larger than a set threshold value;
Screening out a connected domain with the highest rectangle degree from the darkened connected domain with the area larger than a set threshold value, namely the marker;
extracting the minimum bounding box of the marker, and obtaining two corner points of the upper edge of the minimum bounding box of the marker;
calculating a linear function of the upper edge according to the two corner points of the upper edge of the minimum bounding box;
Interference shielding is carried out on the second brightness area, and a luminous communication area inside the calibration plate image is obtained;
Screening the connected domains of the round holes according to the area, calculating the mass center of each connected domain of the round holes, and obtaining a round hole mass center Set1;
Calculating the vertical distance from the center of mass of each circular hole to the upper edge of the marker according to the linear function of the upper edge, and sequencing the center of mass Set1 of the circular holes from small to large according to the vertical distance to obtain a center of mass Set2 of the circular holes;
The round hole centroid Set2 is locally ordered from small to large according to the abscissa, and a final round hole centroid Set is obtained;
performing circular hole coordinate extraction on the calibration plate images of the first view and the second view, and respectively obtaining two circular hole centroid sets corresponding to each other in sequence;
And calculating and storing a homography matrix H between the calibration plate image of the first field of view and the calibration plate image of the second field of view through two circular hole centroid sets in one-to-one correspondence.
Optionally, the connected area of the first brightness area of the calibration plate image is a white area connected in the binary image of the calibration plate image.
Optionally, the calculation formula of the rectangle degree is:
,
wherein y is the rectangle degree,To be the minimum bounding box area,The area of the pixel is communicated with the area of the domain for the marker.
Optionally, the interference shielding is specifically that a marker binary image and a second brightness region binary image are extracted, the marker binary image is mapped into the second brightness region binary image, and the gray value of the region, corresponding to the black pixel in the marker binary image, in the second brightness region binary image is set to 255.
Optionally, the local sorting is performed on the round hole centroid Set2 from small to large according to the abscissa, and the local sorting rule is as follows:
judging whether the upper edge of the minimum bounding box is a long side or a short side, and determining the number of round holes in each row of the upper edge of the minimum bounding box;
and (3) setting the number of round holes in each row of the upper edge of the minimum bounding box as X, traversing the round hole centroid Set2 according to the number of the round holes, and sequencing every X holes in sequence according to the sequence from small to large.
Optionally, the step of respectively calculating a homography matrix H1 from the first stitched image to the preliminary stitched image and a homography matrix H2 from the second stitched image to the preliminary stitched image according to the preliminary stitched image specifically includes:
calculating corner points of the first spliced image and the second spliced image:
Let the height of the first stitched image beWidth ofThe height of the second spliced image isWidth of;
The four corner points of the first spliced image are respectively、、、;
The four corner points of the second spliced image are respectively、、、;
Transforming the angular point coordinates of the second spliced image into the coordinate system of the first spliced image through a homography matrix H, wherein the transformation formula is P ' =P ' ×H, and normalizing P ', so as to obtain new four angular point coordinates,,,;
Calculating the size of the preliminary stitched image after stitching through the four corner coordinates of the first stitched image and the four corner coordinates converted by the second stitched image;
The width of the preliminary mosaic image is:
;
the height of the preliminary stitched image is:
;
Calculating coordinates corresponding to the corner of the first stitched image and the corner of the second stitched image in the preliminary stitched image:
The corner of the first stitched image and the corner of the second stitched image are subtractedNamely, corresponding coordinates in the preliminary mosaic image;
Wherein,,
,
And respectively calculating a homography matrix H1 between the first spliced image and the preliminary spliced image and a homography matrix H2 between the second spliced image and the preliminary spliced image through converting corresponding image corner coordinates before and after conversion.
Optionally, the pixel fusion is performed on the transformed overlapping area, and the fusion mode includes:
average value fusion, namely obtaining an average value of corresponding pixels in the first spliced image and the second spliced image;
Maximum value fusion, namely taking the maximum value of the corresponding pixels in the first spliced image and the second spliced image;
The minimum value fusion, namely, the corresponding pixels in the first spliced image and the second spliced image take the minimum value;
tail-end fusion, namely directly covering the second spliced image by the overlapped area.
The second aspect of the present invention provides an image stitching system, comprising a memory and a processor, wherein the memory comprises an image stitching method program, and the image stitching method program, when executed by the processor, implements the steps of the image stitching method.
A third aspect of the present invention provides a computer-readable storage medium having an image stitching method program embodied therein, which when executed by a processor, implements the steps of the image stitching method.
As can be seen from the above, the present application provides an image stitching method, system and readable storage medium. The method has the characteristics of high precision, strong robustness, high speed and convenient use.
Aiming at the problems of precision and speed, the application designs a calibration plate consisting of a black marker and a plurality of small holes and a conversion relation calculating method. The small Kong Zhixin coordinates are extracted and matched one by one between the two visual field calibration plate images, the conversion matrix is directly calculated through multiple points, the more the number of the points is, the higher the accuracy is, and compared with the conventional calibration plate conversion method, the splicing can be completed through one-time conversion, and the accuracy and the speed are not lost in the middle.
Aiming at the convenience problem, the application provides the coordinate pairing method, which can automatically sort and pair the extracted circular hole centroids without manual matching of operators, and is simpler and more convenient than the conventional calibration plate conversion method.
Aiming at the robustness problem, the application adopts multiple modes to select and process the splicing overlapping area, can overcome the influence of illumination to a certain extent, and the splicing boundary is more natural.
Additional features and advantages of the application will be set forth in the description which follows, and in part will be apparent from the description, or may be learned by practice of the embodiments of the application. The objectives and other advantages of the application will be realized and attained by the structure particularly pointed out in the written description and claims thereof as well as the appended drawings.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. The components of the embodiments of the present application generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the application, as presented in the figures, is not intended to limit the scope of the application, as claimed, but is merely representative of selected embodiments of the application. All other embodiments, which can be made by a person skilled in the art without making any inventive effort, are intended to be within the scope of the present application.
It should be noted that like reference numerals and letters refer to like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures. Meanwhile, in the description of the present application, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance.
Referring to fig. 1, an image stitching method according to some embodiments of the present application includes:
S102, obtaining calibration plate images of a first view and a second view, wherein the inside of the calibration plate images is distributed with black rectangular marker shapes of a plurality of ordered round holes;
It should be noted that, the calibration plate is formed by a black rectangular marker shape with a plurality of orderly round holes distributed inside, the number of the round holes is customized by a user, the row-column distribution (the number of the holes in each row is equal, the number of the holes in each column is equal), and the more the number of the round holes, the higher the precision. The calibration plate is shown in fig. 2, where the black marker is located intermediate the first field of view and the second field of view, both of which require the black marker to be photographed.
It should be noted that both the first field of view and the second field of view need to contain the complete rectangular marker.
If the splice is lateral, the tag needs to be in the vertical direction (short side up and down, long side left and right). In the case of a vertical splice, the tag needs to be in the transverse direction (short side on the left and right, long side up and down).
S104, performing relation conversion on the calibration plate images of the first view and the second view to obtain a homography matrix between the images of the first view and the second view, and using the homography matrix as a conversion matrix between the two spliced images;
S106, acquiring spliced images of a first visual field and a second visual field, which are respectively the first spliced image and the second spliced image, from the same visual angle as the calibration plate image;
S108, converting the first spliced image and the second spliced image by using a conversion matrix, and splicing according to a conversion result to obtain a preliminary spliced image;
s110, respectively calculating a homography matrix H1 from the first jigsaw image to the preliminary mosaic image and a homography matrix H2 from the second jigsaw image to the preliminary mosaic image according to the preliminary mosaic image;
s112, converting the first spliced image and the second spliced image into the preliminary spliced image through a homography matrix H1 and a homography matrix H2 respectively, and carrying out pixel fusion on the overlapped area after conversion to obtain a spliced image.
According to an embodiment of the present invention, the performing relationship conversion on the calibration plate images of the first field of view and the second field of view to obtain a homography matrix between the images of the first field of view and the second field of view specifically includes:
It should be noted that, in the present application, the calibration plate image includes a first field of view and a second field of view, where the first calibration plate image is shown in fig. 3, and the second calibration plate image is shown in fig. 4.
Threshold segmentation is carried out on the calibration plate image to obtain a first brightness area and a second brightness area of the calibration plate image, wherein the brightness value of the second brightness area is higher than that of the first brightness area
And the connected area of the first brightness area of the calibration plate image is a white area connected in the binary image of the calibration plate image.
In the application, the binary processing is carried out on the calibration plate image to obtain a calibration plate binary image, and the white areas connected in the binary image are connected areas. As shown in fig. 5.
Taking the connected domain of the first brightness area as a darkening connected domain, and filling holes in the darkening connected domain of the calibration plate image;
In the present application, the hole filling is performed on the darkened communication area of the calibration plate image, as shown in fig. 6.
The darkened connected domain is filtered according to the area, and the darkened connected domain with the area larger than the set threshold value is reserved, as shown in fig. 7.
Screening out a connected domain with highest rectangle degree from dark connected domains with areas larger than a set threshold value, namely a marker (black rectangle connected domain);
extracting the minimum bounding box of the marker, and obtaining two corner points of the upper edge of the minimum bounding box of the marker;
It should be noted that, the application extracts the minimum bounding box of the rectangular marker, can obtain four corner points of the marker, obtain two corner points of the upper edge of the rectangular marker, and judge whether the upper edge is a short side or a long side according to the distance between the corner points. The number of round holes on one long side is different from that on one short side, so that a basis is provided for subsequent operation.
Calculating a linear function of the upper edge according to the two corner points of the upper edge of the minimum bounding box;
Interference shielding is carried out on the second brightness area, and a luminous communication area inside the calibration plate image is obtained;
It should be noted that, in the present application, the threshold of the calibration plate image is divided, that is, the marker is subjected to binarization processing, so as to obtain a marker binary image, so as to obtain a first brightness region and a second brightness region of the calibration plate image, the marker binary image is shown in fig. 8, and then the marker binary image is subjected to interference shielding, so as to obtain a lighting connected region inside the marker, that is, a second brightness region binary image, and the second brightness region binary image is shown in fig. 9.
The shielding mode is to map the marker binary image into a second brightness region binary image, and the gray value of the region, corresponding to the black pixel in the marker binary image, in the second brightness region binary image is set to 255. And obtaining a marker internal brightening binary image, and obtaining a marker internal brightening connected domain, wherein the marker internal brightening binary image is shown in fig. 10.
And screening the connected domains of the round holes according to the area, calculating the mass center of each connected domain of the round holes, and obtaining a round hole mass center Set1, wherein the schematic diagram of the mass center of the round holes is shown in figure 11.
Calculating the vertical distance from the center of mass of each circular hole to the upper edge of the marker according to the linear function of the upper edge, and sequencing the center of mass Set1 of the circular holes from small to large according to the vertical distance to obtain a center of mass Set2 of the circular holes;
the method calculates the vertical distance from the center of mass of each round hole to the upper edge of the marker, sorts the round hole center of mass Set1 according to the vertical distance from small to large, and obtains the round hole center of mass Set2. A schematic of the vertical distance of the centroid of the circular hole from the top edge of the tag is shown in fig. 12.
The round hole centroid Set2 is locally ordered from small to large according to the abscissa, and a final round hole centroid Set is obtained;
performing circular hole coordinate extraction on the calibration plate images of the first view and the second view, and respectively obtaining two circular hole centroid sets corresponding to each other in sequence;
It should be noted that, the first view field and the second view field are both extracted by the circular hole coordinates in the above steps, and two circular hole centroid sets corresponding to each other in sequence are respectively obtained, and are assumed to be Set' (reference image) and Set "(image to be spliced) respectively.
And calculating and storing a homography matrix H between the calibration plate image of the first field of view and the calibration plate image of the second field of view through two circular hole centroid sets in one-to-one correspondence.
According to the embodiment of the invention, the calculation formula of the rectangle degree is as follows:
,
wherein y is the rectangle degree,To be the minimum bounding box area,The area of the pixel is communicated with the area of the domain for the marker.
According to the embodiment of the invention, the interference shielding is specifically implemented by extracting a marker binary image and a second brightness region binary image, mapping the marker binary image into the second brightness region binary image, and fully blackening the region of the second brightness region binary image, which corresponds to the region of the marker binary image with black pixels.
According to the embodiment of the invention, the round hole centroid Set2 is locally ordered from small to large according to the abscissa, and the local ordering rule is as follows:
judging whether the upper edge of the minimum bounding box is a long side or a short side, and determining the number of round holes in each row of the upper edge of the minimum bounding box;
And (3) setting the number of round holes in each row of the upper edge of the minimum bounding box as X, traversing the round hole centroid Set2 according to the number of the round holes, and sequencing every X holes in sequence according to the sequence from small to large. For example, 4 holes per row, the round hole centroid Set2 is traversed, and every four holes are ordered from small to large.
According to the embodiment of the invention, according to the preliminary stitched image, a homography matrix H1 from the first stitched image to the preliminary stitched image and a homography matrix H2 from the second stitched image to the preliminary stitched image are respectively calculated, specifically:
calculating corner points of the first spliced image and the second spliced image:
it should be noted that, as a specific embodiment, the present application performs stitching using the first stitched image and the second stitched image.
Let the height of the first stitched image beWidth ofThe height of the second spliced image isWidth of;
The four corner points of the first spliced image are respectively、、、;
The four corner points of the second spliced image are respectively、、、;
Transforming the angular point coordinates of the second spliced image into the coordinate system of the first spliced image through a homography matrix H, wherein the transformation formula is P ' =P ' ×H, and normalizing P ', so as to obtain new four angular point coordinates,,,;
Calculating the size of the preliminary stitched image after stitching through the four corner coordinates of the first stitched image and the four corner coordinates converted by the second stitched image;
The width of the preliminary mosaic image is:
;
the height of the preliminary stitched image is:
;
Calculating coordinates corresponding to the corner of the first stitched image and the corner of the second stitched image in the preliminary stitched image:
The corner of the first stitched image and the corner of the second stitched image are subtractedNamely, corresponding coordinates in the preliminary mosaic image;
Wherein,,
,
And respectively calculating a homography matrix H1 between the first spliced image and the preliminary spliced image and a homography matrix H2 between the second spliced image and the preliminary spliced image through converting corresponding image corner coordinates before and after conversion.
According to an embodiment of the present invention, the pixel fusion is performed on the transformed overlapping area, and the fusion mode includes:
average value fusion, namely obtaining an average value of corresponding pixels in the first spliced image and the second spliced image;
Maximum value fusion, namely taking the maximum value of the corresponding pixels in the first spliced image and the second spliced image;
The minimum value fusion, namely, the corresponding pixels in the first spliced image and the second spliced image take the minimum value;
tail-end fusion, namely directly covering the second spliced image by the overlapped area.
The first stitched image and the second stitched image are respectively transformed into the preliminary stitched image through H1 and H2, and the transformed overlapping region needs to be specially processed. The special processing of the application is to perform pixel fusion on overlapping areas, and the fusion modes which can be selected by a user comprise mean value fusion, maximum value fusion, minimum value fusion, tail connection fusion and the like. And carrying out pixel fusion on the transformed overlapping area to obtain a spliced image.
As shown in fig. 13, the second aspect of the present invention provides an image stitching system, where the image stitching system 13 includes a memory 131 and a processor 132, and the memory 131 includes an image stitching method program, and the image stitching method program, when executed by the processor 132, implements the steps of the image stitching method.
A third aspect of the present invention provides a computer-readable storage medium having an image stitching method program embodied therein, which when executed by a processor, implements the steps of the image stitching method.
As can be seen from the above, the present application provides an image stitching method, system and readable storage medium. The method has the characteristics of high precision, strong robustness, high speed and convenient use.
Aiming at the problems of precision and speed, the application designs a calibration plate consisting of a black marker and a plurality of small holes and a conversion relation calculating method. The small Kong Zhixin coordinates are extracted and matched one by one between the calibration plate images of the first view and the second view respectively, the conversion matrix is directly calculated through multiple points, the more the number of the points, the higher the accuracy is, and compared with the conventional calibration plate conversion method, the splicing can be completed through one-time conversion, and the accuracy and the speed are not lost in the middle.
Aiming at the convenience problem, the application provides the coordinate pairing method, which can automatically sort and pair the extracted circular hole centroids without manual matching of operators, and is simpler and more convenient than the conventional calibration plate conversion method.
Aiming at the robustness problem, the application adopts multiple modes to select and process the splicing overlapping area, can overcome the influence of illumination to a certain extent, and the splicing boundary is more natural.
In the several embodiments provided by the present application, it should be understood that the disclosed apparatus and method may be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is merely a logical function division, and there may be additional divisions of actual implementation, such as multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. In addition, the various components shown or discussed may be coupled or directly coupled or communicatively coupled to each other via some interface, whether indirectly coupled or communicatively coupled to devices or units, whether electrically, mechanically, or otherwise.
The units described as separate components may or may not be physically separate, and components displayed as units may or may not be physical units, may be located in one place or distributed on a plurality of network units, and may select some or all of the units according to actual needs to achieve the purpose of the embodiment.
In addition, each functional unit in each embodiment of the present invention may be integrated in one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of hardware plus a form of software functional unit.
It will be appreciated by those of ordinary skill in the art that implementing all or part of the steps of the above method embodiments may be implemented by hardware associated with program instructions, where the above program may be stored in a readable storage medium, where the program when executed performs the steps comprising the above method embodiments, where the above storage medium includes various media that may store program code, such as a removable storage device, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk.
Or the above-described integrated units of the invention may be stored in a readable storage medium if implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solution of the embodiments of the present invention may be embodied in essence or a part contributing to the prior art in the form of a software product stored in a storage medium, including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in the embodiments of the present invention. The storage medium includes various media capable of storing program codes such as a removable storage device, a ROM, a RAM, a magnetic disk or an optical disk.