Detailed Description
In order to make the objects and advantages of the embodiments of the present invention easier to understand, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. It is apparent that the described embodiments are only some embodiments of the present invention, but not all embodiments, and the following detailed description of the embodiments of the present invention in the accompanying drawings does not limit the scope of the claimed invention, but only represent selected embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that, if no conflict arises, the technical features of the embodiments of the present invention described below may be combined with each other and all the features are within the protection scope of the present invention. In addition, while functional block division is performed in an apparatus or a structural diagram, a logical order is shown in a flowchart, in some cases, steps shown or described may be performed in a different order than block division in an apparatus or a different order than in a flowchart. Furthermore, the use of "first," "second," "third," and other similar expressions, herein do not limit the data and execution sequence, but merely facilitate description and distinguish between identical items or similar items that have substantially the same function and effect, without regard to indicating or implying a relative importance or implying a number of technical features.
Unless defined otherwise, technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. It should be understood that the term "and/or" as used in this specification includes any and all combinations of one or more of the associated listed items.
In the semiconductor manufacturing industry, automated optical inspection equipment (Automated Optical Inspection, AOI) is often required to detect and analyze problems with the appearance, quality, and performance of products. The optical machine and the camera are optical components forming the AOI equipment, and most of the current AOI equipment adopts a configuration of a top single camera to inspect objects to be detected on a bottom track. However, some scenes have failed to meet the detection requirements by using a top camera alone, for example, requiring a vertical or inwardly sloped sidewall surface for the object, and an obstruction above the object. In these cases, it is necessary to take a photograph from the side with another oblique camera to acquire the information of interest.
Since the imaging surface of the conventional lens (such as a perspective lens or a telecentric lens) is parallel to the photosensitive chip of the camera, and the optical axis of the side camera is not perpendicular to the surface of the object in consideration of the fact that the side camera is obliquely arranged, the side camera using the conventional lens can focus on only one line on the object, and the area beyond the line is virtual focus, so that the detection effect is greatly affected.
To solve this problem, the industry typically adds a sand lens (scheimpflug) to the camera on the basis of conventional lenses. The characteristic of the lens is that it can change the direction of the optical axis of the lens, so that the optical axis of the obliquely arranged camera is perpendicular to the surface of the object, and the whole picture is focused on. But this brings a new problem in that the pictures taken with the camera with the lens of the poloxamer are distorted. More precisely, the camera field of view (FOV), which is originally a rectangle, is now a trapezoid in the picture because of the sideways shooting. In consideration of machining errors, the original rectangular camera view may be distorted into an arbitrarily shaped quadrilateral, and thus the arbitrarily shaped quadrilateral camera view needs to be corrected back to the rectangular camera view.
In view of the above, an embodiment of the present invention provides a camera view correction method, shooting a calibration board picture by using a camera, preprocessing the calibration board picture to obtain a new calibration board picture, screening effective dots of an original rectangular-like dot in the new calibration board picture, obtaining dot information of the effective dots, establishing a dot coordinate system based on the identified dot information, rotating the original rectangular-like dot and finding out coordinates of each effective dot in the dot coordinate system according to the dot information, obtaining a maximum inscribed rectangular-like dot based on the dot coordinate system, generating an ideal rectangular dot based on the width and height of the maximum inscribed rectangular-like dot and the width and height of the calibration board picture, calculating a homography matrix based on the original rectangular-like dot and the ideal rectangular dot, and correcting an actual measurement photo according to the homography matrix, so as to correct the camera view.
Referring to fig. 1, fig. 1 schematically illustrates an application scenario of a camera view correction method according to some embodiments of the present invention.
As shown in fig. 1, the application scenario includes an electronic device 100. The electronic device 100 may be a desktop computer, a notebook computer, a tablet computer, or other suitable type of device having certain computing and control capabilities. It should be understood that, in the application scenario shown in fig. 1, the electronic device 100 is a notebook computer, but it does not impose any limitation on the structure, type, number, etc. of the electronic devices in other application scenarios. For example, in other application scenarios or embodiments, the electronic device may also be a desktop computer, tablet computer, or other suitable type of device, and the electronic device in other application scenarios or embodiments may also include more or fewer components than the notebook computer shown in the embodiment of fig. 1, or have a different configuration than the notebook computer shown in the embodiment of fig. 1.
It can be appreciated that, to complete the camera field of view correction, the electronic device 100 needs to work cooperatively with other software and hardware, including a camera or an image capturing device, a thermal imaging camera, image processing software, a display device, a storage device, etc., and those skilled in the art can add, delete or change the required software and hardware according to actual needs, and cooperate with the electronic device 100 to complete the correction of the camera field of view. It should be understood that these cooperating hardware and software may be configured on the electronic device 100 according to actual needs, or may exist independent of the electronic device 100.
Specifically, the electronic device 100 is used as a device for providing computing and control capabilities, acquires a calibration plate picture shot by a camera, and pre-processes the calibration plate picture shot by the camera into a new calibration plate picture, where the new calibration plate picture includes an original rectangular-like lattice, and the distribution of the original rectangular-like lattice corresponds to the distribution of calibration blocks of the calibration plate. Identifying and screening effective dots of the original rectangular-like dot matrix, and acquiring dot information of the effective dots, wherein the dot information comprises the mass centers of the dots. Based on the identified dot information, a dot coordinate system is established, the original rectangular-like dot is rotated and the coordinates of each effective dot in the dot coordinate system are found out according to the dot information, and the dot coordinate system comprises an X axis and a Y axis. Based on a lattice coordinate system, the mass centers of four vertexes of the maximum inscribed rectangular lattice are obtained, and the maximum inscribed rectangular lattice is found out from all effective dots based on the mass centers of the four vertexes. And generating an ideal rectangular lattice according to the width and the height of the maximum inscription rectangular lattice and the width and the height of the picture of the calibration plate. The homography matrix is calculated based on the original rectangular-like lattice and the ideal rectangular lattice. Correcting the measured photo according to the homography matrix to correct the camera vision, and obtaining the measured photo with accurate camera vision. And accurately restoring to obtain the surface information of the product structure according to the actual measurement photo after the camera vision correction, so that the accuracy of product structure defect detection is improved.
In order to facilitate understanding of the camera vision correction method provided by the embodiment of the present invention, first, an electronic device provided by the embodiment of the present invention is described in detail.
Referring to fig. 2, fig. 2 schematically illustrates a structural diagram of an electronic device 100 according to some embodiments of the present invention.
Illustratively, as shown in FIG. 2, the electronic device 100 includes at least one processor 110 and a memory 120 communicatively coupled, with one processor being illustrated in FIG. 2 as a bus system connection. Wherein the various components in the electronic device 100 are coupled together by a bus system 130, the bus system 130 being used to enable connection communications between the various components. It will be readily appreciated that the bus system 130 may include a power bus, a control bus, a status signal bus, and the like, in addition to the data bus. The various buses are labeled as bus system 130 in fig. 2 for clarity and conciseness. It will be appreciated that the configuration shown in the embodiment of fig. 2 is merely illustrative and is not intended to limit the configuration of the electronic device described above in any way. For example, the electronic device described above may also include more or fewer components than the structure shown in fig. 2, or have a different configuration than the structure shown in fig. 2.
Specifically, the processor 110 is configured to provide computing and control capabilities to control the electronic device 100 to perform corresponding tasks, for example, to control the electronic device 100 to perform any one of the camera view correction methods provided in the embodiments of the present invention, or to perform steps in any one of the possible implementation manners of any one of the camera view correction methods provided in the embodiments of the present invention. Those skilled in the art will appreciate that the processor 110 may be a general purpose processor including a central Processing unit (Central Processing Unit, CPU), a network processor (Network Processor, NP), etc., or may be a digital signal processor (DIGITAL SIGNAL Processing, DSP), an Application SPECIFIC INTEGRATED Circuit (ASIC), a Field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components.
The memory 120, which is a non-transitory computer readable storage medium, may be used to store non-transitory software programs, non-transitory computer executable programs, instructions, and modules, such as programs, instructions, and modules corresponding to the camera view correction method in the embodiment of the present invention. In some embodiments, the memory 120 may include a storage program area that may store an operating system, applications required for at least one function, and a storage data area that may store data created from the use of the processor 110, etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running non-transitory software programs, instructions and modules stored in the memory 120 to implement any of the camera field of view correction methods provided by the embodiments of the present invention, or to perform steps in any of the possible implementations of any of the camera field of view correction methods provided by the embodiments of the present invention. In some embodiments, memory 120 may include high-speed random access memory, and may also include non-transitory memory. Such as at least one disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, memory 120 may also include memory located remotely from processor 110, which may be connected to processor 110 through a communication network. It is understood that examples of the above-described communication networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
It may be appreciated from the foregoing that the implementation execution body of any one of the camera view correction methods provided in the embodiments of the present invention may be any suitable type of electronic device having a certain computing and control capability, and may be implemented and executed by the electronic device 100. In some possible implementations, any of the camera field of view correction methods provided by the embodiments of the present invention may be implemented by a processor invoking computer program instructions stored in a memory.
The camera vision correction method provided by the embodiment of the invention will be described in detail below in connection with exemplary applications and implementations of the electronic device provided by the embodiment of the invention.
Referring to fig. 3, fig. 3 schematically illustrates a flowchart of a camera view correction method according to some embodiments of the present invention.
It will be appreciated by those skilled in the art that the camera view correction method provided by the embodiment of the present invention may be applied to the above-described electronic apparatus (e.g., the electronic apparatus 100). Specifically, the execution subject of the camera vision correction method is one or at least two processors of the electronic device.
As shown in fig. 3, the camera view correction method includes, but is not limited to, the following steps S100-S700:
S100, preprocessing a calibration plate picture shot by a camera to obtain a new calibration plate picture, wherein the new calibration plate picture comprises an original rectangular-like lattice, and the distribution of the original rectangular-like lattice corresponds to the distribution of calibration blocks of the calibration plate.
The calibration plate is provided with a plurality of calibration blocks, the calibration blocks can be round, rectangular or square, and the like. The distribution shapes of the calibration blocks can be rectangular, square or round and the like, and also can be diamond, oval or triangular and the like, and the embodiment of the invention sets the distribution shapes of the calibration blocks to be rectangular, so that the subsequent calculation is convenient.
In some embodiments, a calibration plate is photographed using a camera to obtain a calibration plate picture, which includes a circular or near-circular shadow area formed by a number of calibration blocks.
Specifically, a calibration plate picture shot from a camera is obtained, color information of the calibration plate picture is removed by using a common image processing library (such as OpenCV, etc.), namely, the number of channels of the calibration plate picture is converted from three channels (such as RGB) to a single channel, an initial gray image is obtained, a threshold segmentation algorithm is adopted to carry out threshold segmentation on the initial gray image, the initial gray image is binarized, namely, the initial gray image is converted into a binary image only comprising black and white colors, and a final preprocessed new calibration plate image is obtained, wherein the new calibration plate image is in the form of a binary image. The preprocessed new calibration plate graph comprises an original rectangular-like lattice, wherein the original rectangular-like lattice comprises a plurality of dots, approximate dots or dots with other shapes, and the distribution of the original rectangular-like lattice corresponds to the distribution of calibration blocks of the calibration plate. Referring to fig. 4, fig. 4 schematically illustrates a new calibration plate diagram obtained by preprocessing a calibration plate picture shot by a camera according to some embodiments of the present invention, where a plurality of dots and dots with other shapes of the new calibration plate diagram illustrated in fig. 4 form an original rectangular-like lattice, and the new calibration plate diagram illustrated in fig. 4 is a new calibration plate diagram without camera field of view correction. Of course, fig. 4 only schematically illustrates the situation that the view of the camera is tilted up along the horizontal axis of the camera, and other types of deflection of the view of the camera are possible, for example, please refer to fig. 5 and 6, in which the view of the camera is simultaneously tilted up along the horizontal axis of the camera, flipped clockwise along the vertical axis of the camera, flipped left along the vertical axis of the camera, and flipped right along the vertical axis of the camera in the new calibration plate shown in fig. 6.
It is understood that the thresholding algorithm includes, but is not limited to, binary thresholding algorithm, OTSU algorithm, and TOZERO algorithm. In some embodiments, noise points in the binary image are removed by using Gaussian blur, the binary image is smoothed, and a new preprocessed calibration plate image with higher quality is obtained, so that features in the binary image are more obvious, and interference on feature extraction is reduced.
And 200, identifying and screening out effective dots of the original quasi-rectangular dot matrix, and acquiring dot information of the effective dots, wherein the dot information comprises the mass centers of the dots.
In some embodiments, a circular detection algorithm (e.g., a Hough transform algorithm) is used to detect dots of the original rectangular-like lattice in the pre-processed new calibration plate map, and the detected dots are extracted. The Hough transform algorithm is an image processing algorithm and can be used for identifying dots in an image and returning the center coordinates and the radius of the dots. In some embodiments, any other suitable algorithm may be used to detect dots in the original rectangular-like lattice, and embodiments of the present invention are not limited in this respect.
In some embodiments, for each extracted dot, determining whether the dot is valid according to a preset rule, if so, determining that the dot is a valid dot, screening the dot as a valid dot, if not, determining that the dot is an invalid dot, and removing or performing other processing. The preset rules include that the radius of the dots is within a preset range, the circle center position of the dots accords with the layout of the original rectangular-like dot matrix, and the like, and of course, a person skilled in the art can add, modify or delete the preset rules according to actual needs, and the embodiment of the invention is not limited in any way.
In some embodiments, after identifying and screening effective dots of the original rectangular-like dot matrix from the extracted dots, acquiring dot information of the effective dots by using a circular detection algorithm (e.g., a Hough transform algorithm, etc.), where the dot information includes a centroid of the dots and a radius of the dots, and the centroid of the dots is a center coordinate of the dots, and the Hough transform algorithm gives the center coordinate and the radius of each dot, so that the dot information of the effective dots can be acquired by the Hough transform algorithm.
And S300, establishing a dot coordinate system based on the identified dot information, rotating the original rectangular-like dot positively, and finding out the coordinates of each effective dot in the dot coordinate system according to the dot information, wherein the dot coordinate system comprises an X axis and a Y axis.
In the method, at least two effective dots positioned in the same horizontal or vertical direction are found out from all the identified effective dots, a first vector between the two effective dots is calculated as an X-axis direction, a second vector perpendicular to the first vector is calculated as a Y-axis direction, the centroid of any one effective dot of the at least two effective dots is taken as an origin, an X-axis is constructed in the X-axis direction, and a Y-axis is constructed in the Y-axis direction, so that a lattice coordinate system is established.
In some embodiments, after the lattice coordinate system is established, the original rectangular-like lattice is rotated to be positive, and the method comprises the steps of determining the current direction of the original rectangular-like lattice through geometric analysis (such as a least square method or Principal Component Analysis (PCA) and the like), calculating the rotation angle of the original rectangular-like lattice according to the current direction, and rotating the original rectangular-like lattice from the current direction to the positive direction according to the rotation angle to obtain the rotated original rectangular-like lattice. For example, the original rectangular-like lattice is rotated by the angle ofRotating the original rectangular-like lattice to a positive direction by using a rotation matrix formula to obtain the original rectangular-like lattice which rotates positively, wherein the rotation matrix is as follows:
The rotation matrix formula is:
Wherein, the method comprises the following steps of,) The coordinates of effective dots in the original rectangular dot matrix are the coordinates of the effective dots in the original rectangular dot matrix,) Is the coordinates of the effective dots in the original rectangular-like lattice that are rotated.
In some embodiments, after the original rectangular-like lattice is rotated, the position coordinates of each effective dot in the lattice coordinate system are determined according to the mass centers of the effective dots in the original rectangular-like lattice, specifically, the coordinates of the rest effective dots in the lattice coordinate system are calculated by taking the first effective dot in the rotated original rectangular-like lattice as the origin. Assuming that the origin is located at (x 0, y 0), the coordinates of the remaining valid dots (x, y) in the lattice coordinate system can be calculated by a simple translational transformation:
Wherein, the method comprises the following steps of,) The first effective dot in the original quasi-rectangular dot matrix is the coordinate of the first effective dot in the dot matrix coordinate system,) The coordinates of the rest effective dots in the lattice coordinate system in the original quasi-rectangular lattice which is the rotation positive.
S400, based on a lattice coordinate system, acquiring the mass centers of four vertexes of the largest inscribed rectangle lattice, and based on the mass centers of the four vertexes, finding the largest inscribed rectangle lattice in all effective dots.
Specifically, according to the coordinates of the effective dots in the original rectangular-like dot matrix in the dot matrix coordinate system, screening out the leftmost effective dots and the rightmost effective dots in each row of the original rectangular-like dot matrix, and according to the distance between the leftmost effective dots and the rightmost effective dots in two adjacent rows, taking the leftmost effective dots or the rightmost effective dots as one of the four vertexes of the largest inscription rectangular-like dot matrix until the four vertexes of the largest inscription rectangular-like dot matrix are found, and taking the center coordinates of the four vertexes of the largest inscription rectangular-like dot matrix as the mass centers of the four vertexes of the largest inscription rectangular-like dot matrix.
In some embodiments, after the centroids of the four vertices of the largest inscription rectangular lattice are obtained, starting from the first row where the top left corner vertex or the top right corner vertex is located, finding all the valid dots located in the same row as the top left corner vertex or the top right corner vertex until the last row where the bottom left corner vertex or the bottom right corner vertex is located is found, and finding all the valid dots located in the same row as the bottom left corner vertex or the bottom right corner vertex, then indicating that the valid dots located in each row of the largest inscription rectangular lattice are found, and forming the valid dots located in each row of the largest inscription rectangular lattice.
S500, generating an ideal rectangular lattice according to the width and the height of the maximum inscription rectangular lattice and the width and the height of the calibration plate picture.
In the embodiment of the invention, before the camera is used for shooting the picture of the calibration plate, the size (comprising the width and the height) of the picture shot by the camera is set, so that the width and the height of the picture of the calibration plate can be known after the picture of the calibration plate is shot by the camera.
Specifically, after a camera is used for shooting to obtain a calibration plate picture, the size of the calibration plate picture is read, and the width and the height of the calibration plate picture are obtained. And calculating the width and the height of the maximum inner-connection rectangular lattice according to the found maximum inner-connection rectangular lattice.
The ideal rectangular lattice is a regular matrix layout in which dots are distributed along regular rows and columns with equal distances between the rows and columns. According to the width and the height of the maximum inscribed rectangular dot matrix and the width and the height of the calibration plate picture, calculating to obtain the width and the height of the ideal rectangular dot matrix, determining that the number of rows and the number of columns of the ideal rectangular dot matrix are the same as those of the maximum inscribed rectangular dot matrix, converting a coordinate system, calculating to obtain the coordinate position of each dot, and generating to obtain the ideal rectangular dot matrix based on the coordinate position of each dot.
And S600, calculating a homography matrix based on the original rectangular-like lattice and the ideal rectangular lattice.
Wherein the homography matrixIs a 3x3 matrix that describes the perspective transformation between two planes, transforming points on one plane from the original coordinate system to the target coordinate system through a homography matrix.
Illustratively, at least 4 sets of corresponding dots are obtained from the original rectangular-like lattice and the ideal rectangular lattice, each set of dots including an effective dot in the original rectangular-like lattice and a dot corresponding to the effective dot in the ideal rectangular lattice.
In some embodiments, the coordinates of at least 4 groups of corresponding dots are normalized, a homography matrix is calculated using the cv2.findHomoprography () function of OpenCV, the coordinates of the normalized at least 4 groups of corresponding dots are input to the cv2.findHomoprography () function, a matrix multiplication equation is constructed from the coordinates of the normalized at least 4 groups of corresponding dots by the homography matrix, and the constructed matrix multiplication equation is calculated using the cv2.findHomopgraphy () function, thereby obtaining a homography matrix of 3x3。
For example, assume a homography matrixThe method comprises the following steps:
assuming that the coordinates (normalized) of a set of corresponding dots in the original rectangular-like lattice and the ideal rectangular lattice are:
,
The coordinates (normalized) of the corresponding set of dots are input to the cv2.findHomographic () function through a homography matrixConstructing a matrix multiplication equation, wherein the matrix multiplication equation is as follows:
from the above matrix multiplication equations, one matrix multiplication equation can only solve two unknowns (i.e.、、、、、、AndTwo of which are unknowns) homography matrixThere are 8 unknowns (i.e、、、、、、And) It is necessary to input the coordinates (normalized) of each set of corresponding dots to the cv2. FindHomographic () function by the homography matrix using the coordinates of at least 4 sets of corresponding dots in the original rectangular-like dot matrix and the ideal rectangular dot matrix, respectivelyConstructing a matrix multiplication equation, wherein two sides of the matrix multiplication equation are respectively provided with coordinates (homogenized) of at least four dots, then solving the matrix multiplication equation to calculate a homography matrixAll unknowns in (1) to obtain homography matrix。
For example, in some embodiments, the resulting matrix multiplication equation constructed using the coordinates (normalized) of four corresponding sets of dots in the original rectangular-like lattice and the ideal rectangular lattice is:
Illustratively, the homography matrix is calculated by solving the matrix multiplication equation described above using the method in the cv2.findHomograph () functionAll unknowns in (1) to obtain homography matrix。
It is easy to understand that the homography matrix is calculated by using the coordinates of at least 4 groups of corresponding dots in the original rectangular-like dot matrix and the ideal rectangular dot matrix, so when the calibration blocks are set on the calibration plate, the number of the calibration blocks should be at least 4, so that the original rectangular-like dot matrix contained in the new calibration plate diagram obtained by preprocessing comprises at least 4 effective dots, and the generated ideal rectangular dot matrix comprises at least 4 dots, so that the homography matrix can be calculated based on the original rectangular-like dot matrix and the ideal rectangular dot matrix.
It can be appreciated that the matrix multiplication equation is constructed by using the coordinates of the corresponding dots in the original rectangular-like dot matrix and the ideal rectangular dot matrix, and the number of the known numbers in the constructed matrix multiplication equation is far greater than the number of the unknown numbers, so that the matrix multiplication equation is an overdetermined equation, and the algorithm for calculating the overdetermined equation includes, but is not limited to, a RANSAC algorithm, an Lmed S algorithm and a least square method.
And S700, correcting the actual measurement photo according to the homography matrix so as to correct the camera field of view.
Specifically, an input actual measurement photo taken by a camera is obtained and loaded, perspective transformation is performed on the actual measurement photo through a homography matrix, namely, the homography matrix is applied to the actual measurement photo by using a cv2.warp Perselected () function to perform perspective transformation, each pixel in the actual measurement photo is multiplied by the homography matrix, the coordinate of each pixel in the camera view corrected picture is obtained, and the pixel value of each coordinate is filled into the corresponding coordinate position in the camera view corrected picture, so that the correction of the camera view is completed.
Referring to fig. 7, fig. 7 schematically illustrates a new calibration plate diagram obtained by preprocessing a calibration plate picture taken by a camera and correcting a camera field of view according to a homography matrix according to some embodiments of the present invention, where the new calibration plate diagram after preprocessing and correcting the camera field of view shown in fig. 7 corresponds to the new calibration plate diagram after preprocessing (without correcting the camera field of view) shown in fig. 4, 5 or 6.
It should be appreciated that correction of the measured photograph from the homography matrix may result in a corrected camera view, i.e., a corrected camera view from the skewed view shown in fig. 4, 5, and 6 or other type of skewed view to a forward view.
In some embodiments, a blank region (i.e., a coordinate location of an unfilled pixel value) in the camera view corrected picture uses an interpolation algorithm to calculate a pixel value for the corresponding coordinate location and fills the pixel value to the corresponding coordinate location. Those skilled in the art will appreciate that interpolation algorithms include, but are not limited to, INTER LINEAR and INTER NEARTEST algorithms.
According to the embodiment of the invention, a camera is utilized to shoot a calibration plate picture, the calibration plate picture is preprocessed to obtain a new calibration plate picture, effective dots of an original rectangular-like dot in the new calibration plate picture are screened out, dot information of the effective dots is obtained, a dot coordinate system is established based on the identified dot information, the original rectangular-like dot is rotated positive, the coordinates of each effective dot in the dot coordinate system are found out according to the dot information, the largest inscribed rectangular-like dot is obtained based on the dot coordinate system, an ideal rectangular dot is generated according to the width and the height of the largest inscribed rectangular-like dot and the width and the height of the calibration plate picture, a homography matrix is calculated based on the original rectangular-like dot and the ideal rectangular dot, and the actual measurement photo is corrected according to the homography matrix, so that the camera view is corrected.
Referring to fig. 8, fig. 8 is a schematic diagram schematically illustrating a sub-flowchart of step S200 in a camera view correction method according to some embodiments of the invention.
In some embodiments, identifying and screening effective dots of the original rectangular-like dot matrix, and obtaining dot information of the effective dots, including but not limited to the following steps S210-S230:
And S210, traversing all the connected domains to analyze the connected domains of the new calibration plate graph, and acquiring the connected domains and connected domain information in the new calibration plate graph, wherein the connected domain information comprises the total number of the connected domains, the centroid coordinates and the area, and the upper left corner coordinates, the width and the height of the circumscribed rectangle of each connected domain.
Specifically, a connected domain analysis algorithm is adopted to analyze the connected domain of the new preprocessed calibration plate graph, all the connected domains in the new preprocessed calibration plate graph are identified, and all the identified connected domains are traversed to obtain connected domain information. The connected domain information comprises the total number of connected domains, barycenter coordinates and areas, and the upper left corner coordinates, width and height of the circumscribed rectangle of each connected domain. It is understood that the connected domain analysis algorithm includes, but is not limited to, SPAGHETTI algorithm, SAUF algorithm, and BBDT algorithm.
Illustratively, in some embodiments, the area of the connected domain is the total number of pixels in the connected domain, and the total number of pixels in the connected domain is calculated using a cv2.Contourarea () function in the OpenCV library or other similar method, to obtain the area of the connected domain.
Illustratively, in some embodiments, the centroid coordinates of the connected domain are the geometric centers of the connected domain, and moments of the connected domain may be calculated using the cv2. Motion () function in the OpenCV library to obtain the centroid coordinates of the connected domain.
Illustratively, in some embodiments, the bounding rectangle of the connected domain is represented by the upper left corner coordinates (x, y) and width and height (w, h) of the rectangle, and the coordinates and dimensions (including width and height) of the bounding rectangle of the connected domain are obtained using the cv2.Boundingrect () function of the OpenCV library, resulting in the bounding rectangle of the connected domain.
S220, filtering out incomplete connected domains based on the left upper corner coordinates, the width and the height of the circumscribed rectangle of each connected domain, wherein the incomplete connected domains are invalid dots.
Specifically, for each connected domain, according to the upper left corner coordinate, the width and the height of the circumscribed rectangle of each connected domain, the upper right corner coordinate, the lower right corner coordinate and the lower left corner coordinate of the circumscribed rectangle of each connected domain are obtained through calculation, and the upper left corner coordinate, the upper right corner coordinate, the lower right corner coordinate and the lower left corner coordinate of the circumscribed rectangle of each connected domain are respectively compared with the edge of the new calibration plate graph. If any one of the upper left corner coordinate, the upper right corner coordinate, the lower right corner coordinate and the lower left corner coordinate of the circumscribed rectangle exceeds the edge of the new calibration plate diagram or is positioned on the edge of the new calibration plate diagram, determining that the connected domain is an incomplete connected domain, determining that the incomplete connected domain is an invalid dot, and filtering the incomplete connected domain.
It will be appreciated that the upper right-hand corner coordinate of the bounding rectangle is the same as the upper left-hand corner coordinate, so that when calculating the upper right-hand corner coordinate, the upper left-hand corner coordinate is added to the width of the bounding rectangle to obtain the upper right-hand corner coordinate, and the upper right-hand corner coordinate is obtained. Similarly, the lower left corner coordinate of the circumscribed rectangle is the same as the upper left corner coordinate, so that when the lower left corner coordinate is calculated, the ordinate of the upper left corner coordinate is subtracted from the height of the circumscribed rectangle to obtain the ordinate of the lower left corner coordinate, and the lower left corner coordinate is obtained. Similarly, after the upper right corner coordinate and/or the lower left corner coordinate are calculated, the lower right corner coordinate of the circumscribed rectangle is the same as the abscissa of the upper right corner coordinate, so that when the lower right corner coordinate is calculated, the ordinate of the upper right corner coordinate is subtracted from the height of the circumscribed rectangle to obtain the ordinate of the lower right corner coordinate, and the lower right corner coordinate is obtained. Or the lower right corner coordinate of the circumscribed rectangle is the same as the ordinate of the lower left corner coordinate, so that when the lower right corner coordinate is calculated, the abscissa of the lower left corner coordinate is added with the width of the circumscribed rectangle to obtain the abscissa of the lower right corner coordinate, and the lower right corner coordinate is obtained.
In some embodiments, the edges of the new calibration plate map are determined by detecting the edges of the new calibration plate map using an edge detection algorithm. It is understood that edge detection algorithms include, but are not limited to, sobel algorithm, canny algorithm, prewitt algorithm, and Laplacian algorithm.
S230, acquiring the median of the area according to the area of each connected domain, and setting an area difference threshold value based on the median.
Specifically, according to the area of each connected domain, the areas of all the connected domains are arranged in order from small to large, and an area arrangement sequence is obtained. And selecting the area value at the middle position from the area arrangement sequence as the median of the area, and if the area value at the middle position is two, taking the median of the area as the average value of the two area values at the middle position.
After the median of the area is obtained, an area difference threshold is set according to the median of the area. In some embodiments, the area difference threshold may be set according to a preset multiple of the median of the areas for distinguishing between valid and invalid connected domains. For example, the area difference threshold is 1.2 times, 1.5 times, or other suitable times of the median of the area, and those skilled in the art can set the area difference threshold according to actual needs, which is not limited in any way in the embodiments of the present invention.
And S240, filtering the connected domains based on the area difference threshold value to obtain qualified connected domains, wherein the connected domain information of the qualified connected domains is dot information of effective dots.
Specifically, each connected domain is traversed, the area of each connected domain is subjected to difference with the median of the area to obtain an area difference value, the area difference value is compared with an area difference threshold value, if the area difference value is smaller than or equal to the area difference threshold value, the connected domain is determined to be a qualified connected domain (namely an effective dot), and if the area difference value is larger than the area difference threshold value, the connected domain is determined to be a disqualified connected domain (namely an ineffective dot). And taking the qualified connected domain information of the connected domain obtained by filtering as dot information of the effective dots, thereby obtaining the dot information of the effective dots so as to filter the connected domain.
In some embodiments, a lattice coordinate system is established based on the identified dot information, including, but not limited to, the following steps S310-S320:
S310, calculating the real resolution of the camera according to the dot information and the actual size of the calibration block on the calibration plate, wherein the real resolution comprises X-axis resolution and Y-axis resolution.
In the embodiment of the invention, the calibration block on the calibration plate has a known actual size in an actual physical space, and the actual size comprises the actual diameter of the circular calibration block.
Specifically, the width (in pixel units) of the circumscribed rectangle of each effective dot is summed to obtain the sum of the widths of the circumscribed rectangles, the sum of the widths of the circumscribed rectangles is divided by the total number of the effective dots (namely, the total number of the connected domains), the diameter (in pixel units) of the effective dots in the X-axis direction is obtained, and the diameter of the effective dots in the X-axis direction is divided by the actual diameter of the circular calibration block to obtain the X-axis resolution of the camera.
And summing the heights (in units of pixels) of the external rectangles of each effective dot to obtain the sum of the heights of the external rectangles, dividing the sum of the heights of the external rectangles by the total number of the effective dots (namely the total number of the connected domains) to obtain the diameter (in units of pixels) of the effective dot in the Y-axis direction, and dividing the diameter of the effective dot in the Y-axis direction by the actual diameter of the circular calibration block to obtain the Y-axis resolution of the camera. And combining the X-axis resolution and the Y-axis resolution of the camera to obtain the real resolution of the camera.
And S320, establishing a lattice coordinate system according to the X-axis resolution and the Y-axis resolution, wherein the X-axis resolution corresponds to the X axis of the lattice coordinate system, and the Y-axis resolution corresponds to the Y axis of the lattice coordinate system.
Specifically, an actual lattice coordinate system is constructed by using the X-axis resolution and the Y-axis resolution, and the position of each effective dot is mapped to the lattice coordinate system, that is, the pixel coordinate of each effective dot is converted into an actual physical coordinate in the lattice coordinate system by the actual resolution. The coordinate system of the lattice coordinate system is set such that an X-axis corresponds to a horizontal direction of the actual physical space, a Y-axis corresponds to a vertical direction of the actual physical space, an X-axis resolution corresponds to an X-axis of the lattice coordinate system, the X-axis resolution represents a horizontal distance (for example, in millimeters/pixel) corresponding to each pixel in the image in the actual physical space, the Y-axis resolution corresponds to a Y-axis of the lattice coordinate system, and the Y-axis resolution represents a vertical distance (for example, in millimeters/pixel) corresponding to each pixel in the image in the actual physical space. In the lattice coordinate system, the X-axis and the Y-axis represent the horizontal coordinate and the vertical coordinate in the actual physical space, respectively, and each pixel in the image has a corresponding actual coordinate in the lattice coordinate system.
In some embodiments, a lattice coordinate system is established with the centroid of the active dot in the upper left corner as the origin of coordinates, the horizontal axis (horizontal direction) as the X axis, and the vertical axis (vertical direction) as the Y axis, wherein the lattice coordinate system includes the X axis and the Y axis, and the unit of the lattice coordinate system is the unit distance, that is, the unit distance 1 is the interval of the abscissa or the ordinate between two adjacent active dots in the lattice coordinate system.
In some embodiments, the original rectangular-like dot matrix is rotated and the coordinates of each effective dot in the dot matrix coordinate system are found according to the dot information, which specifically includes but is not limited to the following steps S330-S340:
S330, calculating the rotation angle of the original rectangular-like lattice, and rotating the original rectangular-like lattice to a required angle according to the rotation angle to obtain a positive lattice, wherein the positive lattice comprises an X axis and a Y axis.
Specifically, according to the width and the height of the calibration plate picture, calculating to obtain the coordinate (x, y) of the center point of the calibration plate picture, wherein the abscissa x of the center point is equal to the width W/2 of the calibration plate picture, and the abscissa y of the center point is equal to the width H/2 of the calibration plate picture. Traversing the centroids of all the effective dots, calculating Euclidean distances between the barycenter coordinates of all the effective dots and the coordinates of the central point of the calibration plate picture, and determining the barycenter of the effective dot closest to the central point of the calibration plate picture as the central barycenter. Next, the center centroid is taken as a target centroid, and two adjacent centroids closest to the center centroid are found, wherein the two adjacent centroids are respectively positioned at the left side and the right side of the target centroid. And fitting out a target straight line by using a linear fitting algorithm according to the target centroid and the two adjacent centroids, and calculating the rotation direction and the rotation angle of the original rectangular-like lattice based on the target straight line.
It is understood that the linear fitting algorithm includes, but is not limited to, taxicab-distance-based linear regression algorithm, euclidean distance-based linear regression algorithm, and Welsch-distance-based linear regression algorithm.
In some embodiments, a center point of a calibration plate picture is set as a rotation center, a rotation matrix is constructed based on the rotation center and the rotation angle, and an original rectangular-like lattice is rotated to be positive by using the rotation matrix, so that the original rectangular-like lattice is rotated to a required angle, and a positive lattice is obtained, wherein the positive lattice comprises an X axis and a Y axis.
In some embodiments, the original rectangular-like lattice is rotated positive using a rotation matrix, i.e., each effective dot is rotated by a rotation matrix formula:
Wherein, the method comprises the following steps of,) The coordinates of effective dots in the original rectangular dot matrix are the coordinates of the effective dots in the original rectangular dot matrix,) For the coordinates of the effective dots in the positive dot matrix obtained after rotating positive,Is the rotation angle.
And S340, traversing the mass centers of the effective dots to identify the relative positions of the effective dots on the Y axis based on the real point distance and the Y axis resolution of each effective dot on the calibration plate, and identifying the relative positions of the effective dots on the X axis of the same row based on the X axis resolution so as to find out the coordinates of each effective dot on a dot matrix coordinate system.
In some optional embodiments, the dot information of the obtained effective dot includes a real dot distance of the dot, which indicates a distance between the effective dot in the real physical space, that is, a distance between the calibration blocks in the calibration plate, where the distance between the calibration blocks refers to a distance between centroids of the calibration blocks, and a distance between the left and right calibration blocks or the upper and lower calibration blocks in the calibration plate is the same (that is, a distance between any two calibration blocks in the calibration plate is the same).
In some alternative embodiments, the Y-axis point distance is calculated from the real point distance of the effective dot and the Y-axis resolution, and the Y-axis point distance is equal to the real point distance of the effective dot multiplied by the Y-axis resolution. Multiplying the ordinate of the centroid of each effective dot by the Y-axis resolution to obtain the actual physical ordinate of each effective dot in the dot matrix coordinate system, sorting all the effective dots according to the actual physical ordinate of each effective dot in the dot matrix coordinate system, and determining the relative position of each effective dot on the Y-axis, thereby identifying the relative position of each effective dot on the Y-axis.
In some alternative embodiments, a line feed threshold is set, the centroids of the effective dots are traversed from the first line, a target difference value between the ordinate of the centroids of the next effective dots and the ordinate of the centroids of the current effective dots is calculated for each centroid of the effective dots, the target difference value is divided by the Y-axis point distance to obtain a first ratio, and if the first ratio is smaller than the line feed threshold, the next effective dot is divided into the line where the current effective dot is located. If the first ratio is greater than or equal to the line feed threshold, the next effective dot and the current effective dot are not located in the same line, and the next effective dot is divided into the next line of the line where the current effective dot is located.
After the line division is completed, multiplying the horizontal coordinate of the centroid of each effective dot by the X-axis resolution according to the centroid of each effective dot in each line to obtain the actual physical horizontal coordinate of each effective dot in the dot matrix coordinate system, sorting each effective dot in ascending order according to the actual physical horizontal coordinate of each effective dot in the dot matrix coordinate system to determine the relative position of the effective dot in the same line on the X-axis (namely the actual physical horizontal coordinate in the dot matrix coordinate system), and combining the relative position of the effective dot on the Y-axis (namely the actual physical vertical coordinate in the dot matrix coordinate system) to obtain the coordinate of each effective dot in the dot matrix coordinate system.
In some embodiments, based on the lattice coordinate system, the centroids of the four vertices of the maximum inscribed rectangular lattice are obtained, specifically including but not limited to the following steps S410-S430:
s410, identifying the leftmost dot centroid and the rightmost dot centroid of each row in all the valid dot centroids.
In some optional embodiments, the dot information of the obtained effective dot includes a real dot distance of the dot, which indicates a distance between the effective dot in the real physical space, that is, a distance between the calibration blocks in the calibration plate, where the distance between the calibration blocks refers to a distance between centroids of the calibration blocks, and a distance between the left and right calibration blocks or the upper and lower calibration blocks in the calibration plate is the same (that is, a distance between any two calibration blocks in the calibration plate is the same).
Specifically, in all the effective dots in the dot matrix coordinate system, for each row of effective dots, comparing the X-axis coordinates of each effective dot in the row, and determining whether a certain effective dot is the leftmost effective dot or the rightmost effective dot, thereby identifying the leftmost effective dot and the rightmost effective dot in each row, and determining the centroids of the leftmost effective dot and the rightmost effective dot as the leftmost dot centroid and the rightmost dot centroid respectively.
S420, calculating the X-axis point distance based on the X-axis resolution and the real point distance, wherein the X-axis resolution and the X-axis point distance are both in units of pixels.
Specifically, an X-axis point distance is calculated according to the real point distance of the effective dot and the X-axis resolution, wherein the X-axis point distance is equal to the multiplication of the real point distance of the effective dot and the X-axis resolution, and the X-axis resolution and the X-axis point distance are both in units of pixels.
S430, acquiring the rotation directions of the original rectangular-like lattice, traversing the largest internal rectangular-like lattice according to different rotation directions, and calculating and acquiring the centroids of four vertexes of the largest internal rectangular-like lattice, wherein the centroids of the four vertexes comprise an upper left corner centroid, an upper right corner centroid, a lower left corner centroid and a lower right corner centroid.
Specifically, before the original rectangular-like lattice is rotated, the rotation direction and rotation angle of the original rectangular-like lattice are calculated, and the rotation direction of the original rectangular-like lattice is obtained from the calculation result. And calculating and obtaining the mass centers of four vertexes of the maximum inscribed rectangle according to the rotation direction and the effective dot positions, and determining the positions of the mass centers of the four vertexes, wherein the mass centers of the four vertexes comprise an upper left corner mass center, an upper right corner mass center, a lower left corner mass center and a lower right corner mass center.
In some embodiments, if the rotation direction of the original rectangular-like lattice is clockwise rotation, firstly calculating and obtaining the right upper horny center of the largest internal rectangular-like lattice, starting from the leftmost centroid of the first row of the original rectangular-like lattice (i.e. the normal lattice) after rotation, looking down row by row along the left side of the normal lattice, calculating the difference value between the abscissa of the leftmost centroid of the current row and the abscissa of the leftmost centroid of the next row, and comparing the difference value between the abscissa of the leftmost centroid of the current row and the abscissa of the leftmost centroid of the next row with the X-axis point distance.
And if the difference value between the abscissa of the leftmost centroid of the current row and the abscissa of the leftmost centroid of the next row is smaller than or equal to the X-axis point distance, determining that the rightmost centroid of the current row is the upper right horny center of the maximum inscribed rectangular lattice.
If the difference between the abscissa of the leftmost centroid of the current row and the abscissa of the leftmost centroid of the next row is larger than the X-axis point distance, updating the next row into the current row, and continuously comparing the difference between the abscissa of the leftmost centroid of the current row and the abscissa of the leftmost centroid of the next row with the X-axis point distance until the upper right cutin center of the maximum inscription rectangular dot matrix is found.
Then, calculating to obtain the left lower corner centroid of the maximum inscription rectangular dot matrix, starting from the rightmost centroid of the last line of the original rectangular dot matrix (namely the positive dot matrix) after rotation, searching upwards line by line along the right side of the positive dot matrix, calculating the difference value between the abscissa of the rightmost centroid of the current line and the abscissa of the rightmost centroid of the last line, and comparing the difference value between the abscissa of the rightmost centroid of the current line and the abscissa of the rightmost centroid of the last line with the X-axis dot distance.
And if the difference value between the abscissa of the rightmost centroid of the current row and the abscissa of the rightmost centroid of the previous row is smaller than or equal to the X-axis point distance, determining that the leftmost centroid of the current row is the left lower corner centroid of the maximum inscribed rectangular dot matrix.
If the difference between the abscissa of the right-most centroid of the current row and the abscissa of the right-most centroid of the previous row is larger than the X-axis point distance, updating the previous row into the current row, and continuously comparing the difference between the abscissa of the right-most centroid of the current row and the abscissa of the right-most centroid of the previous row with the X-axis point distance until the left lower corner centroid of the maximum inscribed rectangular lattice is found.
Then, calculating to obtain the right lower cutin center of the maximum inscription rectangular lattice, starting from the determined right upper cutin center, searching downwards row by row, determining the right upper corner centroid as an anchor point, determining the current row where the anchor point is located, determining the target effective dot centroid closest to the Euclidean distance between the anchor points in the effective dot centroids of the next row, updating the anchor point as the target effective dot centroid, and determining whether the updated anchor point (namely the target effective dot centroid) and the left lower cutin center are located in the same row.
If the updated anchor point (i.e., the target effective dot centroid) is in the same row as the lower left centroid, then the updated anchor point (i.e., the target effective dot centroid) is determined to be the lower right corner centroid of the largest inscribed rectangular-like lattice. If the updated anchor point (i.e., the target effective dot centroid) and the left lower corner centroid are not located in the same row, continuing to determine the current row in which the updated anchor point is located, determining the target effective dot centroid closest to the Euclidean distance between the updated anchor points in the effective dot centroid of the next row, updating the anchor point into the target effective dot centroid, and determining whether the updated anchor point (i.e., the target effective dot centroid) and the left lower corner centroid are located in the same row until the right lower corner centroid of the largest inscribed rectangular dot matrix is found.
And finally, calculating to obtain the left upper cutin center of the maximum inscription rectangular dot matrix, starting from the determined left lower cutin center, searching upwards row by row, determining the left lower corner centroid as an anchor point, determining the current row in which the anchor point is located, determining the target effective dot centroid closest to the Euclidean distance between the anchor points in the effective dot centroid of the last row, updating the anchor point as the target effective dot centroid, and determining whether the updated anchor point (namely the target effective dot centroid) and the right upper cutin center are located in the same row.
If the updated anchor point (i.e., the target effective dot centroid) is in the same row as the upper right centroid, then the updated anchor point (i.e., the target effective dot centroid) is determined to be the upper left corner centroid of the largest inscribed rectangular-like lattice. If the updated anchor point (i.e., the target effective dot centroid) and the upper right centroid are not located in the same row, continuing to determine the current row in which the updated anchor point is located, determining the target effective dot centroid closest to the euclidean distance between the updated anchor points in the effective dot centroid of the previous row, updating the anchor point to the target effective dot centroid, and determining whether the updated anchor point (i.e., the target effective dot centroid) and the upper right centroid are located in the same row until the upper left corner centroid of the maximum inscribed rectangular dot matrix is found.
So far, calculating and obtaining the right upper corner centroid, the left lower corner centroid, the right lower corner centroid and the left upper corner centroid of the maximum inscribed rectangle, and taking the right upper corner centroid, the left lower corner centroid, the right lower corner centroid and the left upper corner centroid as centroids of four vertexes of the maximum inscribed rectangle lattice.
In some embodiments, if the rotation direction of the original rectangular-like lattice is counterclockwise, first, calculating and obtaining the upper left horny center of the largest inscribed rectangular-like lattice, starting from the rightmost centroid of the first row of the original rectangular-like lattice (i.e., the normal lattice) after rotation, looking down row by row along the right side of the normal lattice, calculating the difference between the abscissa of the rightmost centroid of the current row and the abscissa of the rightmost centroid of the next row, and comparing the difference between the abscissa of the rightmost centroid of the current row and the abscissa of the rightmost centroid of the next row with the X-axis point distance.
And if the difference value between the abscissa of the rightmost centroid of the current row and the abscissa of the rightmost centroid of the next row is smaller than or equal to the X-axis point distance, determining that the leftmost centroid of the current row is the left upper corner centroid of the maximum inscribed rectangular dot matrix.
If the difference between the abscissa of the right-most centroid of the current row and the abscissa of the right-most centroid of the next row is larger than the X-axis point distance, updating the next row into the current row, and continuously comparing the difference between the abscissa of the right-most centroid of the current row and the abscissa of the right-most centroid of the next row with the X-axis point distance until the upper left corner centroid of the maximum inscribed rectangular lattice is found.
Then, calculating to obtain the right lower cuticle of the maximum inscription rectangular dot matrix, starting from the leftmost centroid of the last line of the original rectangular dot matrix (i.e. the positive dot matrix) after rotation, searching upwards line by line along the left side of the positive dot matrix, calculating the difference value between the abscissa of the leftmost centroid of the current line and the abscissa of the leftmost centroid of the last line, and comparing the difference value between the abscissa of the leftmost centroid of the current line and the abscissa of the leftmost centroid of the last line with the X-axis dot distance.
And if the difference value between the abscissa of the leftmost centroid of the current row and the abscissa of the leftmost centroid of the previous row is smaller than or equal to the X-axis point distance, determining that the rightmost centroid of the current row is the right lower corner centroid of the maximum inscribed rectangular dot matrix.
If the difference between the abscissa of the leftmost centroid of the current row and the abscissa of the leftmost centroid of the previous row is larger than the X-axis point distance, updating the previous row into the current row, and continuously comparing the difference between the abscissa of the leftmost centroid of the current row and the abscissa of the leftmost centroid of the previous row with the X-axis point distance until the right lower corner centroid of the maximum inscribed rectangular lattice is found.
Then, calculating to obtain the left lower cutin center of the maximum inscription rectangular lattice, starting from the determined left upper cutin center, searching downwards row by row, determining the left upper corner centroid as an anchor point, determining the current row where the anchor point is located, determining the target effective dot centroid closest to the Euclidean distance between the anchor points in the effective dot centroids of the next row, updating the anchor point as the target effective dot centroid, and determining whether the updated anchor point (namely the target effective dot centroid) and the right lower cutin center are located in the same row.
If the updated anchor point (i.e., the target effective dot centroid) is in the same row as the lower right centroid, then the updated anchor point (i.e., the target effective dot centroid) is determined to be the lower left corner centroid of the largest inscribed rectangular-like lattice. If the updated anchor point (i.e., the target effective dot centroid) and the right lower corner centroid are not located in the same row, continuing to determine the current row in which the updated anchor point is located, determining the target effective dot centroid closest to the Euclidean distance between the updated anchor points in the effective dot centroid of the next row, updating the anchor point into the target effective dot centroid, and determining whether the updated anchor point (i.e., the target effective dot centroid) and the right lower corner centroid are located in the same row until the left lower corner centroid of the maximum inscribed rectangular dot matrix is found.
And finally, calculating to obtain the right upper cutin center of the maximum inscription rectangular dot matrix, starting from the determined right lower cutin center, searching upwards row by row, determining the right lower corner centroid as an anchor point, determining the current row in which the anchor point is located, determining the target effective dot centroid closest to the Euclidean distance between the anchor points in the effective dot centroid of the last row, updating the anchor point as the target effective dot centroid, and determining whether the updated anchor point (namely the target effective dot centroid) and the left upper cutin center are located in the same row.
If the updated anchor point (i.e., the target effective dot centroid) is in the same row as the upper left centroid, then the updated anchor point (i.e., the target effective dot centroid) is determined to be the upper right centroid of the largest inscribed rectangular-like lattice. If the updated anchor point (i.e., the target effective dot centroid) and the upper left centroid are not located in the same row, continuing to determine the current row in which the updated anchor point is located, determining the target effective dot centroid closest to the euclidean distance between the updated anchor points in the effective dot centroid of the previous row, updating the anchor point into the target effective dot centroid, and determining whether the updated anchor point (i.e., the target effective dot centroid) and the upper left centroid are located in the same row until the upper right centroid of the largest inscribed rectangular lattice is found.
So far, calculating and obtaining the right upper corner centroid, the left lower corner centroid, the right lower corner centroid and the left upper corner centroid of the maximum inscribed rectangle, and taking the right upper corner centroid, the left lower corner centroid, the right lower corner centroid and the left upper corner centroid as centroids of four vertexes of the maximum inscribed rectangle lattice.
In some embodiments, if the direction of rotation of the original rectangular-like lattice is not significantly clockwise or counter-clockwise, the centroid of the leftmost effective dot centroid of the last row is determined to be the lower left corner centroid and the centroid of the rightmost effective dot centroid of the last row is determined to be the lower right corner centroid. It is understood that no significant clockwise or counter-clockwise rotation means that the rotation angle of the original rectangular-like lattice is less than or equal to the preset angle threshold, which may be set by one skilled in the art according to actual needs, e.g., in some embodiments, the preset angle threshold is set to 3 °,5 °, or other angle values.
Then, calculating and obtaining the right upper cutin center of the maximum inscription rectangular dot matrix, starting from the determined right lower cutin center, searching upwards row by row, determining the right lower corner centroid as an anchor point, determining the current row where the anchor point is located, determining the target effective dot centroid closest to the Euclidean distance between the effective dot centroids in the upper row, updating the anchor point as the target effective dot centroid, and determining whether the updated anchor point (namely the target effective dot centroid) is located in the first row.
Wherein, if the updated anchor point (i.e., the target effective dot centroid) is located in the first row, the updated anchor point (i.e., the target effective dot centroid) is determined to be the upper right centroid of the largest inscribed rectangular lattice. If the updated anchor point (i.e., the target effective dot centroid) is not located in the first row, continuing to determine the current row in which the updated anchor point is located, determining the target effective dot centroid closest to the euclidean distance between the updated anchor points in the effective dot centroid of the last row, updating the anchor point to the target effective dot centroid, and determining whether the updated anchor point (i.e., the target effective dot centroid) is located in the first row until the upper right cuticle of the maximum inscribed rectangular dot matrix is found.
Then, calculating to obtain the upper left cutin center of the maximum inscription rectangular dot matrix, starting from the determined lower left cutin center, searching upwards row by row, determining the lower left corner centroid as an anchor point, determining the current row in which the anchor point is located, determining the target effective dot centroid closest to the Euclidean distance between the effective dot centroids in the upper row, updating the anchor point as the target effective dot centroid, and determining whether the updated anchor point (namely the target effective dot centroid) is located in the first row.
Wherein if the updated anchor point (i.e., the target effective dot centroid) is located in the first row, the updated anchor point (i.e., the target effective dot centroid) is determined to be the lower left corner centroid of the largest inscribed rectangular dot matrix. If the updated anchor point (i.e., the target effective dot centroid) is not located in the first row, continuing to determine the current row in which the updated anchor point is located, determining the target effective dot centroid closest to the euclidean distance between the updated anchor points in the effective dot centroid of the last row, updating the anchor point to the target effective dot centroid, and determining whether the updated anchor point (i.e., the target effective dot centroid) is located in the first row until the upper left corner centroid of the maximum inscribed rectangular dot matrix is found.
So far, calculating and obtaining the right upper corner centroid, the left lower corner centroid, the right lower corner centroid and the left upper corner centroid of the maximum inscribed rectangle, and taking the right upper corner centroid, the left lower corner centroid, the right lower corner centroid and the left upper corner centroid as centroids of four vertexes of the maximum inscribed rectangle lattice.
In some embodiments, the maximum inscribed rectangular lattice is found among all valid dots based on the centroids of the four vertices, including, but not limited to, the following steps S440-S470:
S440, finding out the mass centers of all effective dots in the first row of the maximum inscription rectangular dot matrix according to the coordinates of the mass centers of the upper left corners and the coordinates of the upper right horny centers.
Specifically, the effective dots of the largest inscription rectangular dot matrix are arranged according to rows and columns, the effective dots which are positioned in the same row with the left upper corner centroid or the right upper corner centroid in the largest inscription rectangular dot matrix are determined according to the ordinate of the left upper corner centroid or the ordinate of the right upper corner centroid, and the effective dots which are positioned in the same row with the left upper corner centroid or the right upper corner centroid in the effective dots which are positioned in the same row with the left upper corner centroid or the right upper corner centroid are determined according to the abscissa of the left upper corner centroid or the right upper corner centroid, and the effective dots which are positioned between the abscissa of the left upper corner centroid and the abscissa of the right upper corner centroid are taken as all the effective dots in the first row of the largest inscription rectangular dot matrix, so that the centroids of all the effective dots in the first row of the largest inscription rectangular dot matrix are found.
S450, starting from each centroid of the first row, setting a j-th row of anchor points, searching centroids closest to the Euclidean distance of the anchor points in the j+1-th row, and judging whether the centroids closest to the anchor points are in the same row with the left lower cuticle or the right lower cuticle.
S460, when not in the same line, setting the centroid as a new anchor point.
Illustratively, starting from each centroid of the first row in the maximum inscription rectangular lattice, looking down row by row, setting the centroids of the effective dots of the j-th row (j=1, the first row at this time) as anchor points, looking up the effective dot centroid of the target closest to the Euclidean distance of the anchor point in the next row (j+1-th row, the second row at this time) for each anchor point in the j-th row, and judging whether the effective dot centroid of the target is in the same row as the left lower corner centroid or the right lower corner centroid.
If the effective dot centroid of the target is not in the same row with the left lower corner centroid or the right lower corner centroid, setting the effective dot centroid of the target as a new anchor point, continuously searching the effective dot centroid of the target closest to the new anchor point in the next row, judging whether the effective dot centroid of the target is in the same row with the left lower corner centroid or the right lower corner centroid until the effective dot centroid of the target is in the same row with the left lower corner centroid or the right lower corner centroid, ending the searching, and recording all the effective dot centroids of the target which are not in the same row with the left lower corner centroid or the right lower corner centroid, thereby obtaining all centroids in the maximum inscription rectangle.
And S470, finishing searching when the two adjacent lines are in the same line, recording the current anchor point, finding all centroids in the largest inscribed rectangle, marking all the found centroids as rectangle-like centroids, and marking a lattice formed by effective dots with the rectangle-like centroids as the largest inscribed rectangle-like lattice.
If the target effective dot centroid is in the same line as the lower left centroid or the lower right centroid, the search is ended, and all the target effective dot centroids (i.e., anchor points in each line) which are not in the same line as the lower left corner centroid or the lower right corner centroid are recorded, and at this time, all centroids in the largest inscribed rectangle have been found. All centroids found in the largest inscribed rectangle are marked as rectangle-like centroids, and a dot matrix formed by the effective dots of the rectangle-like centroids and the four vertex centroids (namely, the left upper corner centroid, the left lower corner centroid, the right upper corner centroid and the right lower corner centroid) is marked as the largest inscribed rectangle-like dot matrix.
In some embodiments, the maximum inscribed rectangular lattice is found among all valid dots based on the centroids of the four vertices, including, but not limited to, the following steps S480-S495:
and S480, finding out the mass centers of all the effective dots in the last row of the maximum inscription rectangular dot matrix according to the coordinates of the left lower corner mass center and the coordinates of the right lower corner mass center.
Specifically, the effective dots of the largest inscription rectangular dot matrix are arranged according to rows and columns, the effective dots which are positioned in the same row with the left lower corner centroid or the right lower corner centroid in the largest inscription rectangular dot matrix are determined according to the ordinate of the left lower corner centroid or the ordinate of the right lower corner centroid, and the effective dots which are positioned in the same row with the left lower corner centroid or the right lower corner centroid in the effective dots which are positioned in the same row with the left lower corner centroid or the right lower corner centroid are determined according to the abscissa of the left lower corner centroid or the abscissa of the right lower corner centroid, and the effective dots which are positioned between the abscissa of the left lower corner centroid and the abscissa of the right lower corner centroid are taken as all the effective dots in the last row of the largest inscription rectangular dot matrix, so that the centroids of all the effective dots in the last row of the largest inscription rectangular dot matrix are found.
S485, starting from each centroid of the last row, setting a j-th row of anchor points, searching the centroid closest to the Euclidean distance of the anchor points in the j-1-th row, and judging whether the centroid closest to the anchor points is in the same row with the upper left cuticle or the upper right cuticle.
S490, when not in the same line, sets the centroid as a new anchor point.
Illustratively, starting from each centroid of the first row in the maximum inscription rectangular lattice, looking down row by row, setting the centroids of the effective dots of the j-th row (j=1, the last row is this time) as anchor points, for each anchor point in the j-th row, looking up the target effective dot centroid closest to the Euclidean distance of the anchor point in the last row (j-1-th row, the second last row is this time), and judging whether the target effective dot centroid and the upper left corner centroid or the upper right corner centroid are in the same row.
If the effective dot centroid of the target is not in the same row with the upper left corner centroid or the upper right corner centroid, setting the effective dot centroid of the target as a new anchor point, continuously searching the effective dot centroid of the target closest to the new anchor point in the previous row, judging whether the effective dot centroid of the target is in the same row with the upper left corner centroid or the upper right corner centroid until the effective dot centroid of the target is in the same row with the upper left corner centroid or the upper right corner centroid, ending the searching, and recording all the effective dot centroids of the target which are not in the same row with the upper left corner centroid or the upper right corner centroid, thereby obtaining all centroids in the maximum inscription rectangle.
S495, when the two adjacent points are in the same line, the search is ended, the current anchor point is recorded, at the moment, all centroids in the largest inscribed rectangle are found, all the found centroids are marked as rectangle-like centroids, and the lattice formed by the effective dots with the rectangle-like centroids is marked as the largest inscribed rectangle-like lattice.
If the target effective dot centroid is in the same line as the upper left centroid or the upper right centroid, the search is ended, and all the target effective dot centroids (i.e., anchor points in each line) found that are not in the same line as the upper left corner centroid or the upper right corner centroid are recorded, at which point all centroids in the largest inscribed rectangle have been found. All the found centroids positioned in the largest inscribed rectangle are marked as rectangle-like centroids, and the dot matrix formed by the effective dots of the rectangle-like centroids and the four vertex centroids (namely the left upper corner centroid, the left lower corner centroid, the right upper corner centroid and the right lower corner centroid) is marked as the largest inscribed rectangle-like dot matrix.
In some embodiments, the ideal rectangular lattice is generated according to the width and the height of the maximum inscription rectangular lattice and the width and the height of the calibration plate picture, and the method specifically comprises, but is not limited to, the following steps S510-S540:
s510, acquiring the width and the height of the picture of the calibration plate.
In the embodiment of the invention, before the camera is used for shooting the picture of the calibration plate, the size (comprising the width and the height) of the picture shot by the camera is set, so that the width and the height of the picture of the calibration plate can be known after the picture of the calibration plate is shot by the camera.
Specifically, after a camera is used for shooting to obtain a calibration plate picture, the size of the calibration plate picture is read, and the width and the height of the calibration plate picture are obtained.
S520, determining the width and the height of the ideal rectangular lattice according to the width and the height of the maximum inscription rectangular lattice.
Specifically, according to the found maximum inscription rectangular lattice, the width and the height of the maximum inscription rectangular lattice are calculated. And calculating to obtain the width and the height of the ideal rectangular lattice according to the width and the height of the maximum inscription rectangular lattice and the width and the height of the picture of the calibration plate.
According to the width and the height of the maximum inscription rectangular lattice and the width and the height of the calibration plate picture, the width and the height of the ideal rectangular lattice are calculated, and the method comprises the following steps:
And taking the width of the calibration plate picture as the width of the ideal rectangular lattice, enabling the height of the maximum inscription rectangular lattice to be equal to a first ratio to the width of the maximum inscription rectangular lattice, and multiplying the first ratio by the width of the calibration plate picture to obtain the height of the ideal rectangular lattice.
In some embodiments, calculating the width and the height of the ideal rectangular lattice according to the width and the height of the maximum inscription rectangular lattice and the width and the height of the calibration plate picture includes:
And taking the height of the calibration plate picture as the height of an ideal rectangular lattice, enabling the width of the maximum inscribed rectangular lattice and the height of the maximum inscribed rectangular lattice to be equal to a second ratio, and multiplying the second ratio by the height of the calibration plate picture to obtain the height of the ideal rectangular lattice.
S530, converting a coordinate system according to the width and the height of the maximum inscription rectangular lattice and the width and the height of the ideal rectangular lattice, and obtaining all coordinates of the ideal rectangular lattice.
Specifically, according to the width and height of the maximum inscribed rectangular dot matrix and the width and height of the ideal rectangular dot matrix, converting the coordinates of the effective dots of the maximum inscribed rectangular dot matrix under the dot matrix coordinate system into the coordinates of the effective dots under the ideal rectangular dot matrix, and completing the coordinate system conversion to obtain the coordinates of all the effective dots in the ideal rectangular dot matrix.
In some embodiments, the specific process of coordinate system conversion is that the abscissa of the effective dot under the lattice coordinate system is divided by the width of the maximum inscription rectangular lattice to obtain a first ratio, and the first ratio is multiplied by the width of the ideal rectangular lattice to obtain the abscissa of the effective dot under the ideal rectangular lattice. Dividing the ordinate of the effective dot under the dot matrix coordinate system by the height of the maximum inscription rectangular dot matrix to obtain a second ratio, and multiplying the second ratio by the height of the ideal rectangular dot matrix to obtain the ordinate of the effective dot under the ideal rectangular dot matrix.
S540, establishing an ideal rectangular lattice according to all coordinates of the obtained ideal rectangular lattice.
Specifically, after the coordinates of all the effective dots in the ideal rectangular dot matrix are obtained, the coordinates of all the effective dots are ordered according to rows and columns, the coordinates of the effective dots in each row in the X-axis direction are increased, the coordinates in the Y-axis direction are unchanged, the coordinates of the effective dots in each column in the Y-axis direction are increased, and the coordinates in the X-axis direction are unchanged. The coordinates of the effective dots are inserted into a suitable container (such as a two-dimensional array, a list or a set of dot objects, etc.), and the effective dots are sequentially filled into each row and each column of the ideal rectangular dot matrix according to the abscissa of each effective dot, so that the position correspondence of each effective dot is ensured to be correct, and the ideal rectangular dot matrix is established.
In some embodiments, the homography matrix is calculated based on the original rectangular-like lattice and the ideal rectangular lattice, including, but not limited to, the following steps S610-S620:
S610, acquiring a rotation angle and a rotation center of an original rectangular-like lattice, constructing a rotation matrix according to the rotation angle, and rotating the corrected lattice based on the rotation matrix to enable the maximum internal rectangular-like lattice to rotate back to the original angle.
Specifically, the rotation angle of the original rectangular-like lattice may be obtained by calculating the principal direction of the original rectangular-like lattice (e.g., fitting a straight line using a least squares method or by Principal Component Analysis (PCA), etc.).
In some embodiments, obtaining the rotation angle of the original rectangular-like lattice comprises calculating coordinates (x, y) of a center point of the calibration plate picture according to the width and the height of the calibration plate picture, wherein the abscissa x of the center point is equal to the width W/2 of the calibration plate picture, and the abscissa y of the center point is equal to the width H/2 of the calibration plate picture. Traversing the centroids of all the effective dots, calculating Euclidean distances between the barycenter coordinates of all the effective dots and the coordinates of the central point of the calibration plate picture, and determining the barycenter of the effective dot closest to the central point of the calibration plate picture as the central barycenter. Next, the center centroid is taken as a target centroid, and two adjacent centroids closest to the center centroid are found, wherein the two adjacent centroids are respectively positioned at the left side and the right side of the target centroid. And fitting out a target straight line by using a linear fitting algorithm according to the target centroid and the two adjacent centroids, and calculating the rotation direction and the rotation angle of the original rectangular-like lattice based on the target straight line.
It is understood that the linear fitting algorithm includes, but is not limited to, taxicab-distance-based linear regression algorithm, euclidean distance-based linear regression algorithm, and Welsch-distance-based linear regression algorithm.
In some embodiments, a center point of the calibration plate picture is set as a rotation center, a rotation matrix is constructed based on the rotation center and the rotation angle, and the original rectangular-like lattice after the rotation is rotated back to the original angle (i.e., the angle before the original rectangular-like lattice) by using the rotation matrix, so that the maximum inscribed rectangular-like lattice is rotated back to the original angle, and the maximum inscribed rectangular-like lattice rotated back to the original angle is obtained. Obviously, the maximum inscribed rectangular dot matrix is a dot matrix formed by partial effective dots in the original rectangular dot matrix, so that the original rectangular dot matrix after being rotated forward is rotated back to the original angle (namely, the maximum inscribed rectangular dot matrix is rotated back to the original angle in a following way), and the maximum inscribed rectangular dot matrix rotated back to the original angle can be obtained.
In some embodiments, the original rectangular-like lattice after being rotated is rotated back to the original angle by using a rotation matrix, i.e. each effective dot is rotated back to the original angle by a rotation matrix formula:
Wherein, the method comprises the following steps of,) The coordinates of effective dots in the original rectangular-like dot matrix after rotation are calculated,) To rotate back to the original angular coordinates of the effective dots in the original rectangular-like lattice,Is a rotation angle and is a negative value.
S620, calculating a homography matrix based on the ideal rectangular lattice and the rotated maximum inscription rectangular lattice.
Wherein the homography matrixIs a 3x3 matrix that describes the perspective transformation between two planes, transforming points on one plane from the original coordinate system to the target coordinate system through a homography matrix.
Specifically, at least 4 groups of corresponding dots are obtained from the maximum inscribed rectangular lattice rotated back to the original angle and the ideal rectangular lattice, and each group of dots comprises an effective dot in the maximum inscribed rectangular lattice rotated back to the original angle and a dot corresponding to the effective dot in the ideal rectangular lattice.
In some embodiments, the coordinates of at least 4 groups of corresponding dots are leveled, a homography matrix is calculated using the cv2.findHomoprography () function of OpenCV, the coordinates of the leveled at least 4 groups of corresponding dots are input to the cv2.findHomoprography () function, a matrix multiplication equation is constructed by the homography matrix from the coordinates of the leveled at least 4 groups of corresponding dots, and the constructed matrix multiplication equation is calculated using the cv2.findHomoprography () function to obtain a homography matrix of 3x3。
It can be appreciated that the matrix multiplication equation is constructed by using the coordinates of the maximum inscribed rectangular lattice rotated back to the original angle and the corresponding dots in the ideal rectangular lattice, the number of known numbers in the constructed matrix multiplication equation is far greater than the number of unknown numbers, the matrix multiplication equation is an overdetermined equation, and the algorithm for calculating the overdetermined equation includes but is not limited to a RANSAC algorithm, an LmedS algorithm and a least square method.
According to the embodiment of the invention, a camera is utilized to shoot a calibration plate picture, the calibration plate picture is preprocessed to obtain a new calibration plate picture, effective dots of an original rectangular-like dot in the new calibration plate picture are screened out, dot information of the effective dots is obtained, a dot coordinate system is established based on the identified dot information, the original rectangular-like dot is rotated positive, the coordinates of each effective dot in the dot coordinate system are found out according to the dot information, the largest inscribed rectangular-like dot is obtained based on the dot coordinate system, an ideal rectangular dot is generated according to the width and the height of the largest inscribed rectangular-like dot and the width and the height of the calibration plate picture, a homography matrix is calculated based on the original rectangular-like dot and the ideal rectangular dot, and the actual measurement photo is corrected according to the homography matrix, so that the camera view is corrected.
An embodiment of the present invention provides a computer readable storage medium, on which computer program instructions executable by a processor are stored, which when called by the processor, cause the processor to perform any one of the camera field of view correction methods provided by the embodiment of the present invention, or perform steps in any one of possible implementation manners of any one of the camera field of view correction methods provided by the embodiment of the present invention.
Those skilled in the art will appreciate from the foregoing description of the embodiments that the processes implemented in whole or in part in the methods of the embodiments described above can be implemented by means of computer programs, which can be stored in a computer readable storage medium, and which when executed can include processes as described in the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a register, a Read-Only memory (ROM), a flash memory, a hard disk, a magnetic surface memory, a removable magnetic disk, a CD-ROM, or a random access memory (Random Access memory, RAM), etc.
The foregoing disclosure is merely illustrative of the preferred embodiments of the present application and is not intended to limit the scope of the application, which is defined by the appended claims.