Detailed Description
The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
First, terms of the embodiments of the present application are explained.
Multiplying power splicing: the single lens can not give consideration to both short focus and long focus, and the video splicing of one short focus lens and one long focus lens is realized by using a double-lens or multi-lens splicing technology, so that the aim of expanding the focal length of the lens is achieved.
Zoom motor: the parts in the lens control the zooming of the lens.
Focus motor: the parts in the lens control the focusing of the lens to make the image clear.
In order to reduce abnormal image jump when switching output images of different lenses, an embodiment of the present application provides a method for processing an image of a camera, which is applied to a camera, where the camera includes at least two zoom lenses, and equivalent focal lengths of the zoom lenses are different in range, with reference to fig. 1, the method includes:
s101, acquiring a first image acquired by a target zoom lens under a first focal length and a reference image acquired by a reference zoom lens under a second focal length, wherein an overlapping interval exists between equivalent focal lengths of the target zoom lens and the reference zoom lens, and the equivalent focal lengths of the first focal length and the second focal length are the same and within the overlapping interval.
The camera picture processing method is applied to the camera, and therefore the camera picture processing method can be specifically realized through the camera. The camera includes at least two zoom lenses, because the parameters of the optical elements of each lens are different, for convenience of calculation and description, the focal length of each lens is converted into a focal length under the same parameters, i.e. an equivalent focal length, for example, the focal length of each lens is converted into a focal length of a lens corresponding to the same imaging angle of view on a 135-degree camera (a camera specification), and this converted focal length is the equivalent focal length.
For example, in a video camera, two zoom lenses are included: in the case of the a lens and the B lens, the equivalent focal lengths of the a lens and the B lens may be as shown in fig. 2A, where the equivalent focal length of the a lens is Fa1-Fa2, the equivalent focal length of the B lens is Fb1-Fb2, and the overlapping interval of the equivalent focal lengths of the a lens and the B lens is [ Fb1, Fa2 ]. Or the camera comprises three zoom lenses: when the lens elements a, B, and C are used, the equivalent focal lengths of the lens elements a, B, and C can be as shown in fig. 2B, wherein the equivalent focal length of the lens element a is Fa1-Fa2, the equivalent focal length of the lens element B is Fb1-Fb2, the equivalent focal length of the lens element C is Fc1-Fc2, the overlapping regions of the equivalent focal lengths of the lens elements a and B are [ Fb1, Fa2], and the overlapping regions of the equivalent focal lengths of the lens element B and C are [ Fc1, Fb2 ].
The reference zoom lens may be set according to actual conditions, for example, when the camera includes two zoom lenses, the telephoto lens is set as the reference zoom lens, and of course, the short-focus lens may also be set as the reference zoom lens; or when the camera includes three zoom lenses, the zoom lens at the center of the equivalent focal length range is set as the reference zoom lens, etc., for example, lens B in fig. 2B is set as the reference zoom lens.
The first focal length is the current focal length of the target zoom lens. The first focal length and the second focal length have the same equivalent focal length and are within the overlapping interval. For example, as shown in fig. 2A, the equivalent focal length of the first focal length and the second focal length is in the overlapping region [ Fb1, Fa2], such as Fab; alternatively, as shown in fig. 2B, when the reference zoom lens is a B lens and the target zoom lens is a C lens, the equivalent focal length of the first focal length and the second focal length may be Fbc or the like in the overlap interval [ Fc1, Fb2 ].
And S102, determining the feature points and the features which are commonly contained in the first image and the reference image.
The camera determines feature points and features that are included in both the first image and the reference image using computer vision techniques, such as convolutional neural networks. In order to further reduce the degree of the image jump, when the feature point is selected, a point with obvious features is selected as the feature point near the center of the image, the length and the height of the feature are as large as possible, but the feature must be completely contained in the first image and the reference image.
And S103, determining the reference coordinates of the feature points in the reference image, and determining the reference length and the reference height of the feature in the reference image.
And the camera determines the coordinates of the characteristic points in the two-dimensional coordinate system of the reference image to obtain reference coordinates. And the length (reference length) and height (reference height) of the feature in the reference image are determined according to the two-dimensional coordinate system of the reference image.
And S104, determining a first coordinate of the characteristic point in the first image, and determining a first length and a first height of the characteristic object in the first image.
The camera determines the coordinates of the feature points in the two-dimensional coordinate system of the first image to obtain first coordinates. And determining the length (first length) and height (first height) of the feature in the first image according to the two-dimensional coordinate system of the first image. In the embodiment of the present application, the sequence of S103 and S104 is not limited, and S103 may be executed first, and then S104 may be executed; or executing S104 first and then executing S103; or perform S103 and S104 simultaneously.
And S105, determining the area to be output of the first image according to the reference coordinate, the reference length, the reference height, the first coordinate, the first length and the first height.
The camera aligns the first image with the reference image according to the reference coordinate, the reference length, the reference height, the first coordinate, the first length and the first height by taking the reference image as a reference, and determines a region to be output of the first image.
And S106, outputting the area to be output according to the output resolution of the camera.
When the current image output by the camera is the image collected by the target zoom lens, the camera outputs the area to be output of the first image according to the output resolution. Therefore, when the image output by the camera is converted into the image collected by the reference zoom lens, the degree of picture jump is small. Or when the image output by the camera is the image collected by the reference zoom lens and the image output by the camera is converted into the image collected by the target zoom lens, the camera outputs the area to be output of the first image, so that the degree of image jump is reduced.
According to the camera picture processing method, for cameras with two or more zoom lenses, when the magnification splicing is applied, the area to be output of the first image is determined according to the reference coordinate, the reference length, the reference height, the first coordinate, the first length and the first height, and the picture jumping degree is reduced.
Optionally, the acquiring a first image acquired by the target zoom lens at the first focal length and a reference image acquired by the reference zoom lens at the second focal length includes:
step 1, acquiring a first image collected by a target zoom lens under a first focal length.
And 2, determining a second focal length of the reference zoom lens according to the first focal length, wherein the equivalent focal length of the first focal length is the same as that of the second focal length.
And converting the first focal length into a focal length under the reference zoom lens according to the optical parameters of the target zoom lens and the reference zoom lens to obtain a second focal length of the reference zoom lens.
And 3, setting the focal length of the reference zoom lens as the second focal length, and acquiring a reference image acquired by the reference zoom lens under the second focal length.
The reference image and the first image may be as shown in fig. 3A and 3B, with the camera output image having a width FwThe height of the output image of the camera is Fh. Optionally, the determining a region to be output of the first image according to the reference coordinate, the reference length, the reference height, the first coordinate, the first length, and the first height includes:
step one, according to the first coordinate, the reference coordinate, the first length and the reference length
Determining the Left cutting length, wherein Left is the Left cutting length X
pX-coordinate, W, of said first coordinate
pIs the first length, X
qX-coordinate, W, of the above-mentioned reference coordinate
qIs the above reference length.
Step two, according to the first coordinate, the reference coordinate, the first length and the reference length
Determining the Right cutting length, wherein Right is the Right cutting length, F
wThe output resolution for the camera is wide.
Step three, according to the first coordinate, the reference coordinate, the first height and the reference height
Determining an upper clipping length, wherein U
pCutting the length, Y, for the above
pY-coordinate, H, being the first coordinate
pIs the above first height, Y
qIs the Y coordinate of the above-mentioned reference coordinate, H
qThe reference height is set;
step four, according to the first coordinate, the reference coordinate, the first height and the reference height
Determining a lower cutting length, wherein Down is the lower cutting length; f
hThe output resolution of the camera is high.
And step five, cutting the first image according to the left cutting length, the right cutting length, the upper cutting length and the lower cutting length to obtain a region to be output of the first image.
In the embodiment of the application, the position of the central point of the picture can be ensured to be unchanged when the lens is switched, and the width and the height of the object of the picture are consistent, that is, the picture is not deformed after the switching. The switching parameters are calculated in real time, the cutting smooth transition is guaranteed, the two pictures do not need to be fused, and only one lens outputs the shot picture at the same time, so that the requirements on an image processing chip are greatly reduced, and the cost is reduced.
When the focal length of the target zoom lens is not within the overlap interval, optionally, the method for processing the camera image according to the embodiment of the present application further includes:
step one, when the equivalent focal length of the current focal length of the target zoom lens is not in the overlap interval, acquiring a target left-side cutting length, a target right-side cutting length, a target upper-side cutting length and a target lower-side cutting length when the equivalent focal length is an overlap extremum focal length, wherein the overlap extremum focal length is an extremum in the overlap interval.
The overlapping extremum focal length is an extremum in the overlapping interval and is not an extremum of the target zoom lens. For example, as shown in fig. 2A, when the target zoom lens is lens a and the reference zoom lens is lens B, the overlapped extremum focal length is Fb 1; when the target zoom lens is lens B and the reference zoom lens is lens a, the superimposed extremum focal length is Fa 2.
And obtaining the left side cutting length, the right side cutting length, the upper cutting length and the lower cutting length when the equivalent focal lengths of the first focal length and the second focal length are the overlapped extremum focal lengths, and obtaining the target left side cutting length, the target right side cutting length, the target upper cutting length and the target lower cutting length.
And step two, cutting the image collected by the target zoom lens according to the target left cutting length, the target right cutting length, the target upper cutting length and the target lower cutting length to obtain a target area to be output.
And step three, outputting the target area to be output according to the output resolution of the camera.
When the focal length of the target zoom lens is not within the overlap interval, optionally, the method for processing the camera image according to the embodiment of the present application further includes:
step one, acquiring a second image acquired by the target zoom lens at a third focal length and a third image acquired by the target zoom lens at an overlapping extremum focal length, wherein the overlapping extremum focal length is an extremum of the overlapping interval, and the third focal length is not within the overlapping interval.
The overlapping extremum focal length is an extremum in the overlapping interval and is not an extremum of the target zoom lens. For example, as shown in fig. 2A, when the target zoom lens is lens a and the reference zoom lens is lens B, the overlapped extremum focal length is Fb 1; when the target zoom lens is lens B and the reference zoom lens is lens a, the superimposed extremum focal length is Fa 2.
And step two, determining the target feature point and the target feature object which are commonly contained in the second image and the third image.
And step three, determining a second coordinate of the target feature point in the second image, and determining a second length and a second height of the target feature in the second image.
And step four, determining a third coordinate of the target feature point in the third image, and determining a third length and a third height of the target feature in the third image.
And step five, calculating a target left cutting length, a target right cutting length, a target upper cutting length and a target lower cutting length according to the second coordinate, the second length, the second height, the third coordinate, the third length and the third height.
The calculation methods of the target left-side cutting length, the target right-side cutting length, the target upper cutting length and the target lower cutting length are shown in the above formulas (1) - (4), and are not described herein again.
And sixthly, calculating the unit left side cutting length, the unit right side cutting length, the unit upper cutting length and the unit lower cutting length which need to be cut in the third focal length from the overlapped extremum focal length to the target zoom lens according to the target left side cutting length, the target right side cutting length, the target upper cutting length, the target lower cutting length and the unit adjusting focal length of the target zoom lens, wherein the unit left side cutting length, the unit right side cutting length, the unit upper cutting length and the unit lower cutting length need to be cut when one unit adjusting focal length is adjusted.
The unit adjustment focal length of the target Zoom lens may be determined according to a minimum adjustment unit of the Zoom motor of the camera target Zoom lens. In the following description, the unit left-side cut length is calculated as an example, the focal length of the target zoom lens is adjusted from the third focal length to the overlapped extremum focal length, and N unit adjusted focal lengths are required in total, and the unit left-side cut length is equal to the target left-side cut length/N. The calculation methods of the unit right cutting length, the unit upper cutting length and the unit lower cutting length are similar, and are not described herein again.
And seventhly, determining a target area to be output of the image acquired by the target zoom lens according to the unit left cutting length, the unit right cutting length, the unit upper cutting length and the unit lower cutting length.
And determining a target area to be output according to the current focal length of the target zoom lens. For example, in the process of adjusting from the third focal distance to the overlapped extremal focal distance, when the total M unit adjusted focal distances are adjusted from the third focal distance, it is necessary to cut out M unit left-side cut lengths on the left side of the image currently captured by the target zoom lens, cut out M unit right-side cut lengths on the right side, cut out M unit upper cut lengths on the upper side, and cut out M unit lower cut lengths on the lower side, so as to obtain the target area to be output.
And step eight, outputting the target area to be output according to the output resolution of the camera.
The embodiment of the application provides a camera with at least 2 zoom lenses, and the camera comprises: the main control circuit and at least 2 zoom lenses; the at least 2 variable power lenses include a first variable power lens having a first variable power continuous section and a second variable power lens having a second variable power continuous section, the first variable power continuous section and the second variable power continuous section having partially overlapping continuous sections;
the master control circuit is configured to:
when the current magnification corresponding to the output image of the camera is in the overlapped continuous interval, acquiring a reference image acquired by the first zoom lens under the current magnification; performing feature recognition on the reference image to obtain the length and height of a feature in the reference image and the position of a feature point in the reference image;
acquiring a first image acquired by the second zoom lens under the current magnification; performing feature recognition on the first image to obtain the length and height of the feature in the first image and the position of the feature point in the first image;
determining a region to be output of the first image based on the length and height of the feature in the reference image, the position of the feature, and the length and height of the feature in the first image, and the position of the feature when the image output by the camera is switched from the image captured by the first variable magnification lens to the image captured by the second variable magnification lens at the current magnification;
and outputting the area to be output according to the output resolution of the camera.
The first and second ones of the first and second variable magnification lenses are only for convenience of description of the two variable magnification lenses. For example, as shown in fig. 2A, the lens a may be a first zoom lens, and the lens B may be a second zoom lens; of course, the lens B may be a first zoom lens, and the lens a may be a second zoom lens.
According to the camera picture processing method, for cameras with two or more zoom lenses, when the magnification splicing is applied, the to-be-output area of the reference image is determined according to the reference coordinate, the reference length, the reference height, the first coordinate, the first length and the first height, and the picture jumping degree is reduced.
Optionally, the determining the region to be output of the first image according to the length and height of the feature in the reference image, the position of the feature, the length and height of the feature in the first image, and the position of the feature includes:
step 1 of acquiring an output area of the reference image, wherein the feature is included in the output area of the reference image, and the feature point is included in the output area of the reference image;
step 2, calculating the ratio of the length of the feature in the reference image to the length of the feature in the first image, and the ratio of the height of the feature in the reference image to the height of the feature in the first image to obtain a target zoom ratio;
step 3, zooming the output area of the reference image according to the target zoom ratio to obtain a target zoomed image;
and 4, determining a mapping area of the target scaling image in the first image by taking the characteristic points in the target scaling image and the characteristic points in the first image as aligned reference points, and obtaining an area to be output of the first image.
Optionally, the determining the region to be output of the first image according to the length and height of the feature in the reference image, the position of the feature, the length and height of the feature in the first image, and the position of the feature includes:
step one, according to the coordinates of the characteristic points in the first image and the characteristic points in the first imageThe coordinates in the reference image, the length of the feature in the first image, and the length of the feature in the reference image are determined in accordance with
Determining the Left cutting length, wherein Left is the Left cutting length X
pIs the X coordinate, W, of the feature point in the first image
pIs the length, X, of the feature in the first image
qFor the X coordinate, W, of the feature point in the reference image
qThe length of the characteristic object in the reference image;
secondly, according to the coordinates of the feature point in the first image, the coordinates of the feature point in the reference image, the length of the feature in the first image and the length of the feature in the reference image
Determining the Right cutting length, wherein Right is the Right cutting length, F
wA width of an output resolution for the camera;
thirdly, according to the coordinates of the feature points in the first image, the coordinates of the feature points in the reference image, the heights of the features in the first image and the reference image
Determining an upper clipping length, wherein U
pCutting the length, Y, for the above
pIs the Y coordinate, H, of the feature point in the first image
pIs the height of the feature in the first image, Y
qFor the Y coordinate, H, of the feature point in the reference image
qThe height of the characteristic object in the reference image;
fourthly, according to the coordinates of the characteristic points in the first image, the coordinates of the characteristic points in the reference image and the coordinates of the characteristic objects in the reference imageThe height of the feature in the first image and the height of the feature in the reference image are determined according to
Determining a lower cutting length, wherein Down is the lower cutting length; f
hThe output resolution of the camera is high;
and step five, cutting the reference image according to the left cutting length, the right cutting length, the upper cutting length and the lower cutting length to obtain a region to be output of the reference image.
A schematic diagram of the reference image may be as shown in fig. 3A, and a schematic diagram of the first image may be as shown in fig. 3B.
Optionally, the main control circuit is further configured to:
when the current magnification of the second zoom lens is not in the overlapped continuous interval, acquiring a target left cutting length, a target right cutting length, a target upper cutting length and a target lower cutting length when the magnification is an extreme value, wherein the magnification is an extreme value of the overlapped continuous interval;
cutting the image collected by the target zoom lens according to the target left cutting length, the target right cutting length, the target upper cutting length and the target lower cutting length to obtain a target area to be output;
and outputting the target area to be output according to the output resolution of the camera.
Optionally, the main control circuit is further configured to:
when the current magnification of the second zoom lens is not in the overlapped continuous interval, acquiring a third image acquired by the second zoom lens under the current magnification and a fourth image acquired by the first zoom lens under a magnification extreme value, wherein the magnification extreme value is an extreme value of the overlapped continuous interval, and the magnification extreme value is an extreme value of the magnification of the first zoom lens;
performing feature recognition on the third image to obtain the length and height of a feature in the third image and the coordinates of a feature point in the third image; performing feature recognition on the fourth image to obtain the length and the height of the feature in the fourth image and the coordinates of the feature point in the fourth image;
calculating a target left-side cut length, a target right-side cut length, a target upper cut length and a target lower cut length according to the length and height of the feature in the third image, the position of the feature point, the length and height of the feature in the fourth image and the position of the feature point;
calculating a unit left clipping length, a unit right clipping length, a unit upper clipping length, and a unit lower clipping length to be clipped from the magnification extremum to the current magnification according to the target left clipping length, the target right clipping length, the target upper clipping length, the target lower clipping length, and the unit adjustment magnification of the second zoom lens, each time one unit adjustment magnification is adjusted;
determining a target area to be output of the image acquired by the second zoom lens according to the unit left side cutting length, the unit right side cutting length, the unit upper cutting length and the unit lower cutting length;
and outputting the target area to be output according to the output resolution of the camera.
The calculation methods of the target left-side cutting length, the target right-side cutting length, the target upper-side cutting length and the target lower-side cutting length are shown in the above formula (5) - (8), and are not described herein again.
Optionally, the main control circuit is further configured to:
features and feature points commonly included in the reference image and the first image are identified by computer vision techniques.
Alternatively, a camera including two Zoom lenses may be as shown in fig. 4, in which a Sensor is an image Sensor, Focus is a Focus motor, and Zoom is a Zoom motor.
Optionally, the main control circuit is a processor, and the camera further includes a storage medium, a communication interface and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus.
The communication bus mentioned in the above-mentioned video camera may be a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, or the like. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is shown, but this does not mean that there is only one bus or one type of bus.
The communication interface is used for communication between the camera and other devices.
The Memory may include a Random Access Memory (RAM) or a Non-Volatile Memory (NVM), such as at least one disk Memory. Optionally, the memory may also be at least one memory device located remotely from the processor.
The Processor may be a general-purpose Processor, including a Central Processing Unit (CPU), a Network Processor (NP), and the like; but also Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field Programmable Gate Arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components.
An embodiment of the present application further provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the computer program implements any one of the above-mentioned camera image processing methods.
It is noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.
All the embodiments in the present specification are described in a related manner, and the same and similar parts among the embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, as for the embodiments of the camera and the storage medium, since they are substantially similar to the method embodiments, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiments.
The above description is only for the preferred embodiment of the present application, and is not intended to limit the scope of the present application. Any modification, equivalent replacement, improvement and the like made within the spirit and principle of the present application are included in the protection scope of the present application.