Summary of the invention
To overcome the problems in correlation technique, the application provide a kind of image processing method, device, electronic equipment andStorage medium.
According to the embodiment of the present application in a first aspect, providing a kind of image processing method, which comprises
Obtain the depth information of each pixel of image to be processed；
According to the two-dimensional position of the depth information and the pixel in image coordinate system, the pixel is obtained in imagePixel three-dimensional position in acquisition device coordinate system；
Obtain the focusing three-dimensional position of view parameter and focus point；Wherein, the view parameter be with it is described to be processedThe parameter at the different visual angle in the corresponding Orientation observation visual angle of image；The focus point is to change to observe the image midfield to be processedThe point when visual angle of scape, as rotary shaft；
According to the focusing three-dimensional position, the view parameter and the pixel three-dimensional position, the pixel is obtainedThree-dimensional position after offset；Wherein, the three-dimensional position after the offset is that observation is in the pixel under the view parameterWhen the scene of three-dimensional position, the three-dimensional position for the scene observed；
Respectively according to the three-dimensional position after the offset of each pixel, by each pixel projection to it is described toIn the two-dimensional coordinate system for handling image, target image is obtained.
Optionally, the step of depth information of each pixel for obtaining image to be processed, comprising:
The image to be processed is inputted into preset neural network model, obtains the depth information；Wherein, described defaultNeural network model be to advance with multiple sample images and the depth information label training of the multiple sample image obtainsModel；Scene in the sample image is identical as the type of scene in the image to be processed；The type of the sceneFor the type divided according to the distributional difference of the depth of scene.
Optionally, the preset neural network is obtained using following steps training:
The multiple sample image is inputted initial neural network model respectively to be trained, obtains each sample imagePredetermined depth information；
According to predetermined depth information, the depth information label, first-loss function, the second loss function andWhether three loss functions, judgement restrain in the neural network model of current training stage；Wherein, the first-loss function isFor calculating the loss function of the global error of predetermined depth information and the depth information label；The second loss letterNumber is the loss function for calculating the error of predetermined depth information and the depth information label in gradient direction；It is describedThird loss function is for calculating predetermined depth information and the depth information label in the error in normal vector directionLoss function；
If convergence, the neural network model in the current training stage is determined as the preset neural network mouldType；
If do not restrained, stochastic gradient descent algorithm is utilized, adjustment is in the neural network model of current training stageModel parameter, the neural network model after being adjusted；
The multiple sample image is inputted into the neural network model adjusted respectively, and repeats above-mentioned be trainedThe step of with the adjustment model parameter, until neural network model adjusted is restrained.
Optionally, described the multiple sample image is inputted into initial neural network model to be respectively trained, it obtainsThe step of predetermined depth information of each sample image, comprising:
According to the type of scene in each sample image, the multiple sample image is divided into the type with the sceneCorresponding image collection；
Count sample image in the first sum and each described image set of the multiple sample image second is totalNumber；
By the ratio of first sum and second sum of described image set, as adopting for described image setSample weight；
The sample image in described image set with the sample weight corresponding number is chosen, initial nerve net is inputtedNetwork model is trained, and obtains predetermined depth information of the sample image.
Optionally, the two-dimensional position according to the depth information and the pixel in the two-dimensional coordinate system of image,The step of obtaining pixel three-dimensional position of the pixel in image collecting device coordinate system, comprising:
The two-dimensional position of the pixel is converted into homogeneous coordinates；
Using the depth information of the pixel as the Z coordinate of the homogeneous coordinates of the pixel, the picture is obtainedPixel three-dimensional position of the element in image collecting device coordinate system.
Optionally, described according to the focusing three-dimensional position, the view parameter and the pixel three-dimensional position, it obtainsThe step of three-dimensional position after the offset of the pixel, comprising:
According to the view parameter, obtain by the pixel from the three-dimensional position be offset to the offset after three-dimensional positionThe offset vector set；
Calculate offset of the pixel three-dimensional position relative to the focusing three-dimensional position；
The offset of the pixel is multiplied with the offset vector, it is inclined from the three-dimensional position to obtain the pixelThe offset distance of three-dimensional position after moving to the offset；
The pixel three-dimensional position is added with the offset distance of the pixel, after obtaining the offset of the pixelThree-dimensional position.
Optionally, the acquisition view parameter and the step of the focusing three-dimensional position of focus point, comprising:
It obtains in the electronic equipment for showing the image to be processed, the angle of the electronic equipment of angular motion sensor acquisitionKinematic parameter, and using the angular movement parameter as the view parameter；
Focusing three-dimensional position by the three-dimensional position of specified point in the image to be processed, as the focus point.
According to the second aspect of the embodiment of the present application, a kind of image processing apparatus is provided, described device includes:
Depth information acquistion module is configured as obtaining the depth information of each pixel of image to be processed；
Pixel three-dimensional position acquisition module is configured as according to the depth information and the pixel in image coordinate systemTwo-dimensional position, obtain pixel three-dimensional position of the pixel in image collecting device coordinate system；
Parameter acquisition module is configured as obtaining the focusing three-dimensional position of view parameter and focus point；Wherein, describedView parameter is the parameter at the different visual angle in Orientation observation visual angle corresponding from the image to be processed；The focus point is to changePoint when observing the visual angle of scene in the image to be processed, as rotary shaft；
Three-dimensional position after offset obtains module, be configured as according to the focusing three-dimensional position, the view parameter withAnd the pixel three-dimensional position, the three-dimensional position after obtaining the offset of the pixel；Wherein, the three-dimensional position after the offset isWhen described image acquisition device observes the scene in the pixel three-dimensional position under the view parameter, that observes is describedThe three-dimensional position of scene；
Target image acquisition module, the three-dimensional position after being configured to the offset according to each pixel,By in the two-dimensional coordinate system of each pixel projection to the image to be processed, target image is obtained.
Optionally, the depth information acquistion module, is configured as:
The image to be processed is inputted into preset neural network model, obtains the depth information；Wherein, described defaultNeural network model be to advance with multiple sample images and the depth information label training of the multiple sample image obtainsModel；Scene in the sample image is identical as the type of scene in the image to be processed；The type of the sceneFor the type divided according to the distributional difference of the depth of scene.
Optionally, the preset neural network is obtained using following steps training:
The multiple sample image is inputted initial neural network model respectively to be trained, obtains each sample imagePredetermined depth information；
According to predetermined depth information, the depth information label, first-loss function, the second loss function andWhether three loss functions, judgement restrain in the neural network model of current training stage；Wherein, the first-loss function isFor calculating the loss function of the global error of predetermined depth information and the depth information label；The second loss letterNumber is the loss function for calculating the error of predetermined depth information and the depth information label in gradient direction；It is describedThird loss function is for calculating predetermined depth information and the depth information label in the error in normal vector directionLoss function；
If convergence, the neural network model in the current training stage is determined as the preset neural network mouldType；
If do not restrained, stochastic gradient descent algorithm is utilized, adjustment is in the neural network model of current training stageModel parameter, the neural network model after being adjusted；
The multiple sample image is inputted into the neural network model adjusted respectively, and repeats above-mentioned be trainedThe step of with the adjustment model parameter, until neural network model adjusted is restrained.
Optionally, described the multiple sample image is inputted into initial neural network model to be respectively trained, it obtainsThe step of predetermined depth information of each sample image, comprising:
According to the type of scene in each sample image, the multiple sample image is divided into the type with the sceneCorresponding image collection；
Count sample image in the first sum and each described image set of the multiple sample image second is totalNumber；
By the ratio of first sum and second sum of described image set, as adopting for described image setSample weight；
The sample image in described image set with the sample weight corresponding number is chosen, initial nerve net is inputtedNetwork model is trained, and obtains predetermined depth information of the sample image.
Optionally, the pixel three-dimensional position acquisition module, is configured as:
The two-dimensional position of the pixel is converted into homogeneous coordinates；
Using the depth information of the pixel as the Z coordinate of the homogeneous coordinates of the pixel, the picture is obtainedThree-dimensional position of the element in image collecting device coordinate system.
Optionally, the three-dimensional position after the offset obtains module, is configured as:
According to the view parameter, obtain by the pixel from the three-dimensional position be offset to the offset after three-dimensional positionThe offset vector set；
Calculate offset of the pixel three-dimensional position relative to the focusing three-dimensional position；
The offset of the pixel is multiplied with the offset vector, it is inclined from the three-dimensional position to obtain the pixelThe offset distance of three-dimensional position after moving to the offset；
The pixel three-dimensional position is added with the offset distance of the pixel, after obtaining the offset of the pixelThree-dimensional position.
Optionally, the parameter acquisition module, is configured as:
It obtains in the electronic equipment for showing the image to be processed, the angle of the electronic equipment of angular motion sensor acquisitionKinematic parameter, and using the angular movement parameter as the view parameter；
Focusing three-dimensional position by the three-dimensional position of specified point in the image to be processed, as the focus point.
According to the third aspect of the embodiment of the present application, a kind of electronic equipment is provided, which includes:
Processor；
Memory for storage processor executable instruction；
Wherein, the processor when being configured as executing the executable instruction stored on the memory, is realized above-mentionedThe step of image processing method described in first aspect.
According to the fourth aspect of the embodiment of the present application, a kind of non-transitorycomputer readable storage medium is provided, when describedWhen instruction in storage medium is executed by the processor of electronic equipment, enable the processor to execute above-mentioned first aspect instituteThe step of image processing method stated.
According to the 5th of the embodiment of the present application the aspect, a kind of computer program product is provided, when it is transported on an electronic deviceWhen row, so that the step of electronic equipment executes image processing method described in above-mentioned first aspect.
The technical solution that embodiments herein provides can include the following benefits: the depth information of image to be processedThe scene of each pixel representative of image to be processed be can reflect at a distance from real world between image capture device, andAnd each pixel can reflect the position between the different scenes that each pixel respectively represents in the two-dimensional position in image coordinate systemSet relationship.Therefore, the two-dimensional position according to the depth information of pixel and pixel in image coordinate system can obtain pixel and existThree-dimensional position in image collecting device coordinate system, the three-dimensional position are able to reflect the scene in image to be processed in Image AcquisitionThree-dimensional structure in device coordinate system.On this basis, view parameter be Orientation observation corresponding with image to be processed visual angle notThe parameter at same visual angle, also, focus point is the point changed when observing the visual angle of scene in image to be processed as rotary shaft.CauseThis, can obtain the offset of the pixel according to the focusing three-dimensional position, view parameter and pixel three-dimensional position of focus pointThree-dimensional position afterwards.Also, the three-dimensional position after the offset is that image collecting device is observed under view parameter in pixel threeWhen tieing up the scene of position, the three-dimensional position for the scene observed.Therefore, respectively according to the three-dimensional position after the offset of each pixelIt sets, by the two-dimensional coordinate system of each pixel projection to image to be processed, obtained target image is that image collecting device is regardingWhen being acquired under angular dimensions to the scene, the image of the scene collected.The bandwagon effect of target image just has as a result,There are different bandwagon effects corresponding to the different view parameter in Orientation observation visual angle corresponding from image to be processed.As it can be seen that passing throughThis programme, which can be realized the scene in image to be processed, to be had when being observed in real world, from different observation visual angles pairThe different bandwagon effects answered.
It should be understood that above general description and following detailed description be only it is exemplary and explanatory, notThe application can be limited.
Specific embodiment
Example embodiments are described in detail here, and the example is illustrated in the accompanying drawings.Following description is related toWhen attached drawing, unless otherwise indicated, the same numbers in different drawings indicate the same or similar elements.Following exemplary embodimentDescribed in embodiment do not represent all embodiments consistented with the present invention.On the contrary, they be only with it is such as appendedThe example of device and method being described in detail in claims, some aspects of the invention are consistent.
The executing subject of image processing method provided by the embodiments of the present application can be electronic equipment, and the electronic equipment is specificIt can be image capture device, alternatively, can be image presentation device.For example, when the electronic equipment sets for Image AcquisitionWhen standby, the desktop computer for being equipped with image collecting device, intelligent mobile terminal, laptop and wearable can beIntelligent terminal etc..When the electronic equipment is image presentation device, desktop computer, internet television, intelligent sliding can beDynamic terminal, laptop and wearable intelligent terminal etc..Any electronic equipment that can carry out image displayingFor the present invention, this is not restricted.
Fig. 1 is a kind of flow chart of image processing method shown according to an exemplary embodiment, as shown in Figure 1, a kind ofImage processing method, this method may comprise steps of:
Step S101 obtains the depth information of each pixel of image to be processed.
Wherein, the depth information of each pixel can reflect between the corresponding scene of the pixel and image collecting device away fromFrom therefore, the depth information that can use pixel obtains the pixel three-dimensional position of pixel in subsequent step S102.Specifically answeringIn, the acquisition modes of the depth information of each pixel of image to be processed can be a variety of.Illustratively, it can use doubleVisually feel that depth estimation method obtains: shooting the scene in image to be processed using double cameras, obtain two images；According to twoThe parallax for opening image, the depth information of each pixel of image to be processed is calculated using triangulation and solid geometry.OrImage to be processed illustratively can be inputted preset neural network model, obtain the depth information of each pixel by person；ItsIn, preset neural network model is the depth information label training for advancing with multiple sample images and multiple sample imagesThe model arrived.In order to facilitate understanding and rational deployment, subsequent to be obtained in Fig. 2 embodiment to using preset neural network modelThe mode of the depth information of each pixel of image to be processed, is specifically described.Any each pixel that can obtain imageDepth information mode, be used equally for the present invention, the present embodiment to this with no restriction.
Step S102 obtains pixel and adopts in image according to the two-dimensional position of depth information and pixel in image coordinate systemPixel three-dimensional position in acquisition means coordinate system.
Wherein, the image coordinate system of image to be processed is two-dimensional coordinate system, and pixel is in the Two-dimensional Position in image coordinate systemThe positional relationship that can reflect the corresponding each scene of each pixel in real world is set, for example, scene S1 is in scene S2The left side.Also, Image Acquisition coordinate system are as follows: using collect image to be processed image collecting device optical center as origin,Using the optical axis of image collecting device as Z axis, using the axis parallel with the X-axis of image to be processed as X-axis and with image to be processedThe parallel axis of Y-axis is Y-axis, and the rectangular coordinate system of foundation, which is the three-dimensional system of coordinate that can reflect real world.HereinOn the basis of, pixel three-dimensional position of each pixel in image collecting device coordinate system in order to obtain, in subsequent step S104Three-dimensional position after the middle offset for being obtained pixel using the three-dimensional position, is needed on the basis of the two-dimensional position of pixel, is obtainedPixel is at a distance from image collecting device.Also, the depth information of image to be processed can reflect each picture of image to be processedThe corresponding scene of element is at a distance from real world between image capture device.Therefore, it can be directed to each pixel, according to thisThe depth information of pixel and the two-dimensional position of the pixel, obtain pixel three-dimensional position of the pixel in image collecting device coordinate systemIt sets.In order to facilitate understanding and rational deployment, subsequent to be existed in the form of alternative embodiment to each pixel for obtaining image to be processedThe mode of pixel three-dimensional position in image collecting device coordinate system, is specifically described.
In addition, image collecting device can be it is a variety of.Illustratively, when executing subject of the invention is that can carry out figureWhen the electronic equipment of picture acquisition, which is image collecting device.For example, intelligent mobile terminal, tablet computer etc..Alternatively, the executing subject when image processing method provided in an embodiment of the present invention is the electronic equipment that can not carry out Image AcquisitionWhen, which is set to the device for collecting image to be processed.For example, being different from the camera shooting of executing subject of the present inventionMachine, intelligent mobile terminal etc..
Step S103 obtains the focusing three-dimensional position of view parameter and focus point.Wherein, view parameter be with wait locateManage the different visual angle in the corresponding Orientation observation visual angle of image；When focus point is that the visual angle of scene in image to be processed is observed in change,Point as rotary shaft.
In a particular application, the acquisition modes of view parameter and focus point can be a variety of.Below with alternative embodimentMode be described.
In an alternative embodiment, step S103 may include: from the multiple visual angles prestored, select one with toThe parameter for handling the different visual angle in the corresponding visual angle of image, as view parameter.From the multiple two-dimensional positions prestored, one is selectedA two-dimensional position corresponding with view parameter, and using the corresponding three-dimensional position of the two-dimensional position as the focusing three-dimensional position of focus pointIt sets.
Wherein, for view parameter, illustratively, user selects one from the option that displaying has multiple view parameters,As view parameter, alternatively, electronic equipment selects one from the multiple view parameters prestored automatically, as view parameter.ExampleSuch as, LOOK LEFT, LOOK RIGHT or upward angle of visibility etc. be can choose.In addition, image to be processed is two-dimensional, the three-dimensional position of pixelIt is that two-dimensional position based on pixel in image to be processed obtains.Therefore, for it is any collect image to be processed when figureAs the corresponding visual angle of scene in the observation visual angle of acquisition device, that is, any image to be processed, it is considered as figure to be processedAs intrinsic visual angle, not to be related to the Orientation observation visual angle of depth of field variation, the displaying under different perspectives is imitated in order to obtain as a result,Fruit, view parameter Orientation observation visual angle corresponding from image to be processed are different.
The focusing three-dimensional position of focus point can be a variety of.Illustratively, can be the multiple two-dimensional positions prestored canTo be three-dimensional position of the central point of image to be processed in image collecting device coordinate system, alternatively, image to be processed is divided equallyBehind four parts, the three-dimensional position of the center of upper left and the central point of bottom left section in image collecting device coordinate systemSet etc..Alternatively, it is illustrative, it can be using the three-dimensional position of any point in image to be processed as the focusing three-dimensional position of focus pointIt sets.In a particular application, the three-dimensional dimension position of focusing can be corresponding with any view parameter, alternatively, can be in upper leftHeart position is corresponding with upper left visual angle, and the center of bottom left section is corresponding with lower-left visual angle etc..Also, since focus point is to changeWhen becoming the visual angle of scene in observation image to be processed, as the point of rotary shaft, therefore, focus point can be used as visual angle change processThe central point of the corresponding different images of middle different perspectives.
This alternative embodiment has prestored view parameter and focusing three-dimensional position, therefore, to as executing subject of the present inventionThe hsrdware requirements of electronic equipment are in contrast less, install outside angular motion sensor, touch screen, and/or person without electronic equipmentThe human-computer interaction devices such as mouse are connect, the three-dimensional position of view parameter and focus point can be obtained.
In an alternative embodiment, above-mentioned steps S103 can specifically include following steps:
It obtains in the electronic equipment for showing image to be processed, the angular movement ginseng of the electronic equipment of angular motion sensor acquisitionNumber, and using angular movement parameter as view parameter；
Focusing three-dimensional position by the three-dimensional position of specified point in image to be processed, as focus point.
In a particular application, user can select view parameter by the interaction posture with electronic equipment, when user is to hand overWhen mutual posture is interacted with electronic equipment, electronic equipment is moved, at this point, the angular motion sensor of electronic equipment can acquire electricityThe angular movement parameter of sub- equipment, and using angular movement parameter as view parameter.This alternative embodiment can pass through user and electronicsThe posture interaction of equipment, obtains view parameter and focus point, improves the interaction sense and interest of user during image is shown, makesIt is more true lively to obtain image bandwagon effect.
Wherein, illustratively, angular motion sensor specifically can be gyroscope, correspondingly, angular movement parameter may include:Roll angle and pitch angle.For example, when electronic equipment level, pitch angle and roll angle are 0 degree, are bowed when electronic equipment is verticalThe elevation angle be 90 degree, electronic equipment side downward when roll angle be 90 degree.Also, the interaction posture of user and electronic equipment can beBe tilted to the left electronic equipment, and be tilted to the right electronics, upward tilted electron equipment and tilts down electronic equipment etc..Citing andSpeech, user be tilted to the left electronic equipment when angular movement parameter, can be used as view parameter LOOK LEFT, user is tilted to the right electronicsAngular movement parameter when equipment, can be used as view parameter LOOK RIGHT etc..Also, as the finger of focus point in image to be processedFixed point can be a variety of.Illustratively, specified point can be user and move on the touchscreen, and/or, when rotating image,The point of the position of Fingertip touch, alternatively, point of the user using interactive devices positions selected in image to be processed such as mouses.It is any to obtain the mode for obtaining the three-dimensional position of view parameter and focus point, it is used equally for the present invention, the present embodiment pairThis is with no restriction.
Step S104, according to focusing three-dimensional position, view parameter and pixel three-dimensional position, after the offset for obtaining pixelThree-dimensional position.Wherein, the three-dimensional position after offset is that image collecting device is observed under view parameter in pixel three-dimensional positionScene when, the three-dimensional position for the scene observed.
In real world, when being observed with different visual angles Same Scene, two visual angles observed are distinguishedIn corresponding image, the pixel arrangement position of scene will appear offset.Therefore, observation is in pixel under view parameter in order to obtainWhen the scene of three-dimensional position, obtained target image needs to obtain the offset of the three-dimensional position of each pixel in image to be processedSituation, and then the three-dimensional position after each pixel-shift is determined according to the drift condition.
When on this basis, in order to guarantee to change visual angle, the positional relationship between each scene will not change, can be withDetermine that focus point, pixel three-dimensional position change are equivalent to focus point the direction and distance carried out under view parameter for fixed pointOffset.It therefore, can be according to focusing three-dimensional position, view parameter and pixel three-dimensional position, after the offset for obtaining pixelThree-dimensional position.In order to facilitate understanding and rational deployment, it is subsequent in the form of alternative embodiment to the three-dimensional position after the offset of pixelThe acquisition modes set are specifically described.
Step S105, respectively according to the three-dimensional position after the offset of each pixel, by each pixel projection to figure to be processedIn the two-dimensional coordinate system of picture, target image is obtained.
Illustratively, according to the three-dimensional position after the offset of each pixel, by each pixel projection to image to be processedIn two-dimensional coordinate system, target image is obtained, can specifically include: by the three-dimensional position projection after the offset of each pixel wait locateIn the two-dimensional coordinate system for managing image, the two-dimensional position after obtaining the offset of each pixel；According to two after the offset of each pixelPosition is tieed up, the pixel in image to be processed is arranged, target image is obtained.
For image to be processed, present after changing to the observation visual angle of scene in image to be processed, the sight of the sceneExamine effect, be equivalent under view parameter observe real world in the scene obtain target image.Therefore, it is necessary to pass through stepS105, according to the three-dimensional position after the offset of each pixel, by the two-dimensional coordinate system of each pixel projection to image to be processed,Obtain target image.
Illustratively, as shown in Fig. 2 (a), image 201 to be processed is the LOOK LEFT image focused nearby, target image 202To focus LOOK RIGHT image nearby.It focuses nearby, selects the point in image scene to be processed nearby as focus point, at this point,The Scene Blur of distant place.After the LOOK RIGHT of target image is changed into from the LOOK LEFT of image to be processed in visual angle, image to be processedCompared with same scene in target image 202, the part that can be observed tails off scene 2011 in 201,202 midfield of target imageFor scape 2021 compared with same scene in image 201 to be processed, the part that can be observed becomes more.And, it is understood that InWhen observation visual angle changes into the LOOK RIGHT of focusing nearby from the LOOK LEFT focused nearby, the image observed is the mesh in Fig. 2 (a)Logo image 202.It can be seen that the scene in image to be processed may be implemented with corresponding to different observation visual angles in the embodiment of the present inventionDifferent bandwagon effects.
Similar, as shown in Fig. 2 (b), image 203 to be processed is the upward angle of visibility image for focusing distant place, and target image 204 isFocus the downwards angle of visibility image of distant place.It focuses at a distance, selects the point of distant place in image scene to be processed as focus point, at this point, closelyThe Scene Blur at place.After the downwards angle of visibility of target image is changed into from the upward angle of visibility of image to be processed in visual angle, image 203 to be processedMiddle scene 2031 is covered in image 203 to be processed by greenbelt originally compared with same scene 2041 in target image 204Part can be observed in target image 204.And, it is understood that observation visual angle from focus distant place upward angle of visibilityWhen changing into the downwards angle of visibility for focusing distant place, the image observed is the target image 204 in Fig. 2 (b).It can be seen that the embodiment of the present inventionScene in image to be processed, which may be implemented, to be had when being observed in real world, it is corresponding from different observation visual angles notSame bandwagon effect.
The technical solution that embodiments herein provides can include the following benefits: the depth information of image to be processedThe scene of each pixel representative of image to be processed be can reflect at a distance from real world between image capture device, andAnd each pixel can reflect the position between the different scenes that each pixel respectively represents in the two-dimensional position in image coordinate systemSet relationship.Therefore, the two-dimensional position according to the depth information of pixel and pixel in image coordinate system can obtain pixel and existThree-dimensional position in image collecting device coordinate system, the three-dimensional position are able to reflect the scene in image to be processed in Image AcquisitionThree-dimensional structure in device coordinate system.On this basis, view parameter be Orientation observation corresponding with image to be processed visual angle notThe parameter at same visual angle, also, focus point is the point changed when observing the visual angle of scene in image to be processed as rotary shaft.CauseThis, can obtain the offset of the pixel according to the focusing three-dimensional position, view parameter and pixel three-dimensional position of focus pointThree-dimensional position afterwards.Also, the three-dimensional position after the offset is that image collecting device is observed under view parameter in pixel threeWhen tieing up the scene of position, the three-dimensional position for the scene observed.Therefore, respectively according to the three-dimensional position after the offset of each pixelIt sets, by the two-dimensional coordinate system of each pixel projection to image to be processed, obtained target image is that image collecting device is regardingWhen being acquired under angular dimensions to the scene, the image of the scene collected.The bandwagon effect of target image just has as a result,There are different bandwagon effects corresponding to the different view parameter in Orientation observation visual angle corresponding from image to be processed.As it can be seen that passing throughThis programme, which can be realized the scene in image to be processed, to be had when being observed in real world, from different observation visual angles pairThe different bandwagon effects answered.
Optionally, above-mentioned steps S102: according to the two-dimensional position of depth information and pixel in the two-dimensional coordinate system of image,Pixel three-dimensional position of the pixel in image collecting device coordinate system is obtained, can specifically include following steps:
The two-dimensional position of pixel is converted into homogeneous coordinates；
Using the depth information of pixel as the Z coordinate of the homogeneous coordinates of the pixel, the pixel is obtained in image collecting devicePixel three-dimensional position in coordinate system.
Wherein, homogeneous coordinates include: the vector that n dimension is indicated with a n+1 dimensional vector.Therefore, pixel in order to obtainThree-dimensional position in image collecting device coordinate system can be directed to each pixel of image to be processed, by the two dimension of the pixelPosition is converted to homogeneous coordinates.For example, the two-dimensional position of a certain pixel be (X, Y), then the homogeneous coordinates of the pixel be (X,Y, 1).Also, the depth information of the pixel can reflect the distance between scene and image collecting device representated by the pixel,Therefore, it can be adopted to obtain the pixel in image using the depth information of the pixel as the Z coordinate of the homogeneous coordinates of the pixelPixel three-dimensional position in acquisition means coordinate system.For example, the depth information of the pixel is Z, then the pixel is in Image AcquisitionPixel three-dimensional position in device coordinate system is (X, Y, Z).
Optionally, above-mentioned steps S104: position is tieed up according to focusing three-dimensional position, view parameter and pixel, obtains pixelOffset after three-dimensional position, can specifically include following steps:
According to the view parameter of pixel, obtain pixel from the inclined of the three-dimensional position after pixel three-dimensional positional shift to offsetMove vector；
Calculate offset of the pixel three-dimensional position relative to focusing three-dimensional position；
The offset of pixel is multiplied with offset vector, obtains pixel from the three-dimensional after pixel three-dimensional positional shift to offsetThe offset distance of position；
The three-dimensional position of pixel is added with the offset distance of pixel, the three-dimensional position after obtaining the offset of pixel.
Wherein, offset vector is for showing a certain pixel from the three-dimensional position after pixel three-dimensional positional shift to offsetOffset direction.The acquisition modes of offset vector can be a variety of.Illustratively, when view parameter is in the multiple visual angles prestoredOne when, corresponding offset vector can be calculated for the visual angle that each prestores, to obtain preset offset vector and viewThe corresponding relationship at angle.Therefore, it is corresponding partially that view parameter can be searched from the corresponding relationship at preset offset vector and visual angleMove vector.Alternatively, illustrative, the angular motion sensor in the electronic equipment that view parameter is for showing image to be processedWhen the angular movement parameter of acquisition, angular movement parameter is the angle of the situation of change of plane angle with horizontal plane where reflecting electronic equipmentParameter is spent, for example, pitch angle and roll angle.Also, plane where electronic equipment is equivalent to the plane where image to be processed, thisWhen view parameter when can reflect the three-dimensional position by image shift to be processed to after deviating, image to be processed and horizontal plane press from both sidesThe situation of change at angle.It therefore, can be by view parameter, that is, angular movement parameter, for example, pitch angle and roll angle are converted toOffset vector.
In a particular application, for each pixel, the pixel three-dimensional position of the pixel is calculated relative to focusing three-dimensional positionOffset, may include: offset △ d=a × (Zi-Z0).Wherein, focusing three-dimensional position is (X0, Y0, Z0), pixel i'sPixel three-dimensional position is (Xi, Yi, Zi), and a is constant, and i is the serial number of pixel.If offset vector is (x, y), pixel i is from pictureThe offset distance d that plain three-dimensional position is offset to the three-dimensional position after offset may include: d=[a × (Zi-Z0) × x, a × (Zi-Z0) y, a × (Zi-Z0)].Correspondingly, the three-dimensional position after the offset of pixel i is [Xi+a × (Xi-X0) × x, Yi+a × (Yi-Y0) y, Zi+a × (Zi-Z0)].
Illustratively, the range of the depth information of image to be processed is 0~1, and focus point is the central point of image to be processed(L/2, W/2,0.5), wherein L is the length of image to be processed, and W is the width of image to be processed.The a certain pixel depth in the upper left corner is0, the pixel three-dimensional position of the pixel is (0,0,0).Offset vector under view parameter is (x, y), by the pixel-shift to viewAfter the three-dimensional position after offset under angular dimensions, the new three-dimensional position of the pixel, that is, three-dimensional position after the offset of the pixelIt is set to [0+a × (0-0.5) × x, 0+a × (0-0.5) y, 0].
Fig. 3 is a kind of flow chart of the image processing method shown according to another exemplary embodiment, as shown in figure 3, oneThe determination method of kind individualized content, this method may comprise steps of:
Image to be processed is inputted preset neural network model, obtains each pixel in image to be processed by step S301Depth information.Wherein, preset neural network model is the depth for advancing with multiple sample images and multiple sample imagesThe model that information labels training obtains；Scene in sample image is identical as the type of scene in image to be processed；SceneType is the type divided according to the distributional difference of the depth of scene.
In a particular application, the distribution of the depth of different scenes is different, therefore, in order to guarantee to train obtained preset mindDiversified depth distribution can be coped with through network model, the class of scene can be divided according to the distributional difference of the depth of sceneType, and guarantee that the scene in sample image is identical as the type of scene in image to be processed.Illustratively, scene type can be withIncluding indoor scene, outdoor scene and there are the scenes of personage.
Illustratively, as shown in figure 4, being implemented in a kind of image processing method exemplified according to Fig. 3, preset nerve netThe structural schematic diagram of network, the preset neural network may include four parts: Base Model (basic model), Multi-Scale Model (multiple dimensioned model), Feature Fuse layer (Fusion Features layer) and Prediction layer are (pre-Survey layer).In any portion, illustratively, the specific structure of convolutional layer can be depthwise-pointwise (it is longitudinal-byPoint) structure, have the characteristics that parameter is few, the small level-one accuracy rate of model loses few, the volume of the structure compared with the convolutional coding structure of partLamination can use different convolution kernels and extract feature to the different channel (channel) of image 401 to be processed；And it is possible toFor each pixel extraction feature of image 401 to be processed.
Base Model is used to carry out image 401 to be processed from picture bottom to high-rise feature extraction, for forMulti-Scale Model provides feature.Wherein, low-level image feature can be right for some angle points, edge and turning etc. basisAs.Middle level features higher than low-level image feature can be geometry, such as triangle, circle, square etc. shape objects, highMore complicated in the high-level characteristic of middle level features, representing this feature position is the subjects such as people, cup, automobile.ThusPreset neural network is understood that scene information different in picture, to carry out depth calculation for neural network further partIn contrast more sufficient, clearly data are provided.
Multi-Scale Model is used for the characteristic pattern to the Base Model feature extraction different scale provided.Specifically, each location of pixels of characteristic pattern has recorded this in the receptive field in image to be processed on whole picture image to be processedRelativeness.Therefore, the characteristic pattern of different scale can reflect the local feature and global characteristics in image to be processed respectively, fromAnd local feature and global characteristics can be provided for Feature Fuse layer and Prediction layer.
Feature Fuse layer is used to restore image resolution ratio and reduction port number, and to bottom to high-rise spySign is merged, to provide the Global Information of each scene in image to be processed to Prediction layer.PredictionLayer is used to calculate the depth information of each pixel in image to be processed, and to obtained depth using the feature receivedInformation is exported in the form of image 402.
Step S302 obtains pixel and sits in image collecting device according to the two-dimensional position of the depth information of pixel and pixelPixel three-dimensional position in mark system.
Step S303 obtains the focusing three-dimensional position of view parameter and focus point.
Step S304, according to focusing three-dimensional position, view parameter and pixel three-dimensional position, after the offset for obtaining pixelThree-dimensional position.
Step S305, respectively according to the three-dimensional position after the offset of each pixel, by each pixel projection to figure to be processedIn the two-dimensional coordinate system of picture, target image is obtained.
Above-mentioned steps S302 to step S305 is identical to step S105 as the step S102 of Fig. 1 embodiment of the present invention, hereinIt repeats no more, is detailed in the description of Fig. 1 embodiment and alternative embodiment of the present invention.
In above-mentioned Fig. 2 embodiment, the depth information of image to be processed is obtained using preset neural network, can be improvedThe efficiency of Depth Information Acquistion reduces the hardware cost of Depth Information Acquistion and obtains convenience.Realizing figure to be processed as a result,Scene as in has when being observed in real world, different bandwagon effects corresponding from different observation visual angles it is sameWhen, take into account the improved efficiency and convenience of image displaying.
Optionally, above-mentioned preset neural network can specifically be obtained using following steps training:
Multiple sample images are inputted initial neural network model respectively to be trained, obtain the pre- of each sample imageSurvey depth information；
According to predetermined depth information of sample image, depth information label, first-loss function, the second loss function andWhether third loss function, judgement restrain in the neural network model of current training stage；Wherein, first-loss function is to useIn the loss function for the global error for calculating predetermined depth information and depth information label；Second loss function is pre- for calculatingDepth information and depth information label are surveyed in the loss function of the error of gradient direction；Third loss function is to predict for calculatingThe loss function of depth information and depth information label in the error in normal vector direction；
If convergence, the neural network model in the current training stage is determined as preset neural network model；
If do not restrained, stochastic gradient descent algorithm is utilized, adjustment is in the neural network model of current training stageModel parameter, the neural network model after being adjusted；
Multiple sample images are inputted into the neural network model adjusted respectively, and repeat to be trained and adjust mouldThe step of shape parameter, until neural network model adjusted is restrained.
In a particular application, in multiple sample images, the simple sample image of scene depth, such as simple flat surface imageThe prediction difficulty of depth information, the often higher than sample image of scene depth complexity, as there is the image of large stretch of flowers, plants and treesThe prediction difficulty of depth information.It is complicated to edge such as object boundary and statue boundary etc. also, in a certain sample imageThe prediction difficulty of the depth information of sample characteristics, the often higher than pixel in plane such as road surface and desktop etc. simple sample featureDepth information prediction difficulty.Therefore, different loss functions can be set, different prediction difficulty is taken not with realizingThe differentiation training of same training degree.
Specifically, the global error that first-loss function calculates predetermined depth information Yu depth information label can be set,To realize specific aim error calculation on the whole from sample image.Illustratively, first-loss function can lose letter for HuBerNumber, which can reduce the training degree of prediction difficulty in contrast low sample image, and it is opposite to increase prediction difficultyFor high sample image training degree, to realize the sample image and the simple sample of scene depth to scene depth complexityThe differentiation training of this image.And it is possible to be arranged, the second loss function calculates predetermined depth information and depth information label existsThe loss function of the error of gradient direction.Wherein, gradient direction includes horizontal direction and vertical direction, and the depth of gradient directionInformation is spent in edge in contrast than more significant, and therefore, increasing by the second loss function can be improved boundary sample characteristicsTraining degree, to improve the prediction accuracy of the depth information of boundary pixel.Third loss function calculates predetermined depth letterThe error of breath and depth information label in normal vector direction.Wherein, normal vector direction represents the direction of plane, therefore, increasesThird loss function can guarantee the training differentiation of sample at plan-position, to protect when using the second loss functionDemonstrate,prove the forecasting accuracy of the depth information of pixel at plan-position.It is realized as a result, using first-loss function to complex samples figureThe training of the differentiation of picture and simple sample image, is realized using the second loss function and third loss function to complex samples featureWith the differentiation training of simple sample feature.
Wherein, the error of each loss function output is the smaller the better, when the neural network for being in the current training stageWhen model is restrained, show in the neural network model by training, in the current training stage, the mistake of each loss function outputDifference reaches aspiration level, it may be assumed that the predicted value of the depth information of the global feature of image to be processed reaches aspiration level, marginal positionThe depth information at place and the predicted value of the depth information at plan-position also reach aspiration level.Also, in the training process, withThe model parameter of convolutional neural networks model of the machine gradient descent algorithm adjustment in the current training stage, so that convolutional Neural netAfter model parameter adjusts, testing result is improved network model, the difference between reduction and the classification information marked in advance,To reach convergence.Correspondingly, above-mentioned training and tune can be repeated before the model convergence in the current training stageThe step of mould preparation shape parameter, until neural network model adjusted is restrained.Certainly, training has adjusted both for newest every timeThe convolutional neural networks model of model parameter.
Furthermore it is possible to utilize multiple test images and multiple test charts after training obtains preset neural network modelThe prediction effect of the depth information label Verification model of picture.Wherein, scene in the type of scene and sample image in test imageType it is identical.It can specifically include: multiple test images being inputted into preset neural network model respectively, obtain each testPredetermined depth information of image；According to predetermined depth information of test image, depth information label and the 4th loss function, meterCalculate predetermined depth information of test image and the error of depth information label；When error meets aspiration level, test passes through, shouldPreset neural network model can be used for obtaining the depth information of each pixel of image to be processed, otherwise, can more varyThis image re-starts training.Wherein, the 4th loss function is specifically as follows average relative error function or root-mean-square errorFunction etc. loss function.
Optionally, it multiple sample images is inputted into initial neural network model is respectively trained above-mentioned, obtain everyBefore the step of predetermined depth information of a sample image, image presentation method provided by the embodiments of the present application can also include:
Using sample image and preset random perturbation rule, enhanced sample image is obtained；Preset random perturbationRule is that can adjust the rule of the specified characteristics of image of sample image；
Increase enhanced sample image in multiple sample images, obtains preset neural network model for training.
In a particular application, preset random perturbation rule can be a variety of.Illustratively, preset random perturbation ruleIt then can be picture superposition, image rotates left and right, and image random cropping and/or image pixel disturbance etc. are specifiedThe adjustment of characteristics of image.Enhanced sample image is equivalent to the figure carried out in preset random perturbation rule to sample imageAs the new sample image after Character adjustment, obtained.
This alternative embodiment is the pretreatment carried out before being trained to sample image, and a sample image is pre- by thisAfter processing, the diversity of sample image can be increased.After increasing enhanced sample image in multiple sample images, it is used forThe sample image that training obtains preset neural network model just includes multiple sample images and enhanced sample image,Guarantee that model can learn the sample image comprising a variety of situations with this.It is thus possible to improve preset neural network modelIt is in contrast smaller by the interference effect of extraneous factor to make preset neural network model for robustness, for example, for there are illuminationIt changes, the image of contrast variation etc. interference can calculate depth information.
Optionally, above-mentioned multiple sample images are inputted into initial neural network model to be respectively trained, it obtains eachIt the step of predetermined depth information of sample image, can specifically include:
According to the type of scene in each sample image, multiple sample images are divided into figure corresponding with the type of sceneImage set closes；
Count the second sum of sample image in the first sum and each image collection of multiple sample images；
Sample weight by the ratio of the first sum and the second sum of image collection, as image collection；
Choose the sample image in image collection with sample weight corresponding number, input initial neural network model intoRow training, obtains predetermined depth information of sample image.
In a particular application, the quantity of the sample image of different scenes type has certain difference, if according to tradition sideMethod is therefrom chosen a sample image at random and is trained, it may appear that deep by the unbalanced caused model over-fitting of sample sizeSpend the problem of information inaccuracy.Therefore, in order to reduce the over-fitting of preset neural network model, the accurate of depth information is improvedDegree can be directed to the corresponding image collection of type of different scenes, corresponding sample weight is arranged.The sample of different images setDifferent sample weights is arranged in image, to guarantee the equilibrium of each type of sample size, reduces model over-fitting.
Illustratively, according to the type of scene in each sample image, divide obtain outdoor scene image collection ag1,The image collection ag2 of indoor scene and image collection ag3 there are the scene of who object.Count each multiple sample imagesThe second sum of first sum K, image collection ag1 is k1, and the second sum of image collection ag2 is k2 and image collection ag3Second sum be k3.The sample weight of each image collection is K/ki: the sample weight of image collection ag1 is K/k1, imageThe sample weight of set ag2 is K/k2, and the sample weight of image collection ag3 is K/k3.Wherein, the more sample image of quantity is adoptedSample weight is smaller, and the fewer sample image sample weight of quantity is bigger, guarantees the different scenes class in network model training in this wayThe equal number of the sample image of type, prevents model training from deviation occur.
Corresponding to above method embodiment, the application also provides a kind of image processing apparatus.
Fig. 5 is a kind of image processing apparatus block diagram shown according to an exemplary embodiment.As shown in figure 5, a kind of imageProcessing unit, the apparatus may include: depth information acquistion module 501, three-dimensional position obtain module 502, parameter acquisition module503, the three-dimensional position after deviating obtains module 504, target image obtains module 505 and target image display module 506,In:
Depth information acquistion module 501 is configured as obtaining the depth information of each pixel of image to be processed；
Pixel three-dimensional position acquisition module 502 is configured as according to the depth information and the pixel in image coordinateTwo-dimensional position in system obtains pixel three-dimensional position of the pixel in image collecting device coordinate system；
Parameter acquisition module 503 is configured as obtaining the focusing three-dimensional position of view parameter and focus point；Wherein, instituteStating view parameter is the different visual angle in Orientation observation visual angle corresponding from the image to be processed；The focus point is to change observationPoint in the image to be processed when the visual angle of scene, as rotary shaft；
Three-dimensional position after offset obtains module 504, is configured as according to the focusing three-dimensional position, the view parameterAnd the pixel three-dimensional position, the three-dimensional position after obtaining the offset of the pixel；Wherein, the three-dimensional position after the offsetWhen the scene in the pixel three-dimensional position is observed under the view parameter for described image acquisition device, the institute that observesState the three-dimensional position of scene；
Target image obtains module 505, the three-dimensional position after being configured to the offset according to each pixelIt sets, by the two-dimensional coordinate system of each pixel projection to the image to be processed, obtains target image.
The technical solution that embodiments herein provides can include the following benefits: the depth information of image to be processedThe scene of each pixel representative of image to be processed be can reflect at a distance from real world between image capture device, andAnd each pixel can reflect the position between the different scenes that each pixel respectively represents in the two-dimensional position in image coordinate systemSet relationship.Therefore, the two-dimensional position according to the depth information of pixel and pixel in image coordinate system can obtain pixel and existThree-dimensional position in image collecting device coordinate system, the three-dimensional position are able to reflect the scene in image to be processed in Image AcquisitionThree-dimensional structure in device coordinate system.On this basis, view parameter be Orientation observation corresponding with image to be processed visual angle notThe parameter at same visual angle, also, focus point is the point changed when observing the visual angle of scene in image to be processed as rotary shaft.CauseThis, can obtain the offset of the pixel according to the focusing three-dimensional position, view parameter and pixel three-dimensional position of focus pointThree-dimensional position afterwards.Also, the three-dimensional position after the offset is that image collecting device is observed under view parameter in pixel threeWhen tieing up the scene of position, the three-dimensional position for the scene observed.Therefore, respectively according to the three-dimensional position after the offset of each pixelIt sets, by the two-dimensional coordinate system of each pixel projection to image to be processed, obtained target image is that image collecting device is regardingWhen being acquired under angular dimensions to the scene, the image of the scene collected.The bandwagon effect of target image just has as a result,There are different bandwagon effects corresponding to the different view parameter in Orientation observation visual angle corresponding from image to be processed.As it can be seen that passing throughThis programme, which can be realized the scene in image to be processed, to be had when being observed in real world, from different observation visual angles pairThe different bandwagon effects answered.
Optionally, the depth information acquistion module 501, is configured as:
The image to be processed is inputted into preset neural network model, obtains the depth information；Wherein, described defaultNeural network model be to advance with multiple sample images and the depth information label training of the multiple sample image obtainsModel；Scene in the sample image is identical as the type of scene in the image to be processed；The type of the sceneFor the type divided according to the distributional difference of the depth of scene.
Optionally, the preset neural network is obtained using following steps training:
The multiple sample image is inputted initial neural network model respectively to be trained, obtains each sample imagePredetermined depth information；
According to predetermined depth information, the depth information label, first-loss function, the second loss function andWhether three loss functions, judgement restrain in the neural network model of current training stage；Wherein, the first-loss function isFor calculating the loss function of the global error of predetermined depth information and the depth information label；The second loss letterNumber is the loss function for calculating the error of predetermined depth information and the depth information label in gradient direction；It is describedThird loss function is for calculating predetermined depth information and the depth information label in the error in normal vector directionLoss function；
If convergence, the neural network model in the current training stage is determined as the preset neural network mouldType；
If do not restrained, stochastic gradient descent algorithm is utilized, adjustment is in the neural network model of current training stageModel parameter, the neural network model after being adjusted；
The multiple sample image is inputted into the neural network model adjusted respectively, and repeats above-mentioned be trainedThe step of with the adjustment model parameter, until neural network model adjusted is restrained.
Optionally, described the multiple sample image is inputted into initial neural network model to be respectively trained, it obtainsThe step of predetermined depth information of each sample image, comprising:
According to the type of scene in each sample image, the multiple sample image is divided into the type with the sceneCorresponding image collection；
Count sample image in the first sum and each described image set of the multiple sample image second is totalNumber；
By the ratio of first sum and second sum of described image set, as adopting for described image setSample weight；
The sample image in described image set with the sample weight corresponding number is chosen, initial nerve net is inputtedNetwork model is trained, and obtains predetermined depth information of the sample image.
Optionally, the pixel three-dimensional position acquisition module 502, is configured as:
The two-dimensional position of the pixel is converted into homogeneous coordinates；
Using the depth information of the pixel as the Z coordinate of the homogeneous coordinates of the pixel, the picture is obtainedThree-dimensional position of the element in image collecting device coordinate system.
Optionally, the three-dimensional position after the offset obtains module 504, is configured as:
According to the view parameter, obtain by the pixel from the three-dimensional position be offset to the offset after three-dimensional positionThe offset vector set；
Calculate offset of the pixel three-dimensional position relative to the focusing three-dimensional position；
The offset of the pixel is multiplied with the offset vector, it is inclined from the three-dimensional position to obtain the pixelThe offset distance of three-dimensional position after moving to the offset；
The three-dimensional position of the pixel is added with the offset distance of the pixel, obtains the inclined of the pixelThree-dimensional position after shifting.
Optionally, the parameter acquisition module 503, is configured as:
It obtains in the electronic equipment for showing the image to be processed, the angle of the electronic equipment of angular motion sensor acquisitionKinematic parameter, and using the angular movement parameter as the view parameter；
Focusing three-dimensional position by the three-dimensional position of specified point in the image to be processed, as the focus point.
Corresponding to above method embodiment, the application also provides a kind of electronic equipment.
Fig. 6 is a kind of electronic equipment shown according to an exemplary embodiment.Referring to Fig. 6, which may include:
Processor 601；
Memory 602 for storage processor executable instruction；
Wherein, the application when being configured as executing the executable instruction stored on memory 602, is realized in processor 601Provided by embodiment the step of any image processing method.
The technical solution that embodiments herein provides can include the following benefits: the depth information of image to be processedThe scene of each pixel representative of image to be processed be can reflect at a distance from real world between image capture device, andAnd each pixel can reflect the position between the different scenes that each pixel respectively represents in the two-dimensional position in image coordinate systemSet relationship.Therefore, the two-dimensional position according to the depth information of pixel and pixel in image coordinate system can obtain pixel and existThree-dimensional position in image collecting device coordinate system, the three-dimensional position are able to reflect the scene in image to be processed in Image AcquisitionThree-dimensional structure in device coordinate system.On this basis, view parameter be Orientation observation corresponding with image to be processed visual angle notThe parameter at same visual angle, also, focus point is the point changed when observing the visual angle of scene in image to be processed as rotary shaft.CauseThis, can obtain the offset of the pixel according to the focusing three-dimensional position, view parameter and pixel three-dimensional position of focus pointThree-dimensional position afterwards.Also, the three-dimensional position after the offset is that image collecting device is observed under view parameter in pixel threeWhen tieing up the scene of position, the three-dimensional position for the scene observed.Therefore, respectively according to the three-dimensional position after the offset of each pixelIt sets, by the two-dimensional coordinate system of each pixel projection to image to be processed, obtained target image is that image collecting device is regardingWhen being acquired under angular dimensions to the scene, the image of the scene collected.The bandwagon effect of target image just has as a result,There are different bandwagon effects corresponding to the different view parameter in Orientation observation visual angle corresponding from image to be processed.As it can be seen that passing throughThis programme, which can be realized the scene in image to be processed, to be had when being observed in real world, from different observation visual angles pairThe different bandwagon effects answered.
Fig. 7 is the block diagram of the electronic equipment 700 shown according to another exemplary embodiment.For example, electronic equipment 700 can be withIt is mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, body-building equipment, individualDigital assistants etc..
Referring to Fig. 7, electronic equipment 700 may include following one or more components: processing component 702, memory 704,The interface 710 of power supply module 706, multimedia component 708 and input/output (I/O).
The integrated operation of the usual controlling electronic devices 700 of processing component 702, such as with display, call, data are logicalLetter, camera operation and record operate associated operation.Processing component 702 may include one or more processors 720 to holdRow instruction, to perform all or part of the steps of the methods described above.In addition, processing component 702 may include one or more mouldsBlock, convenient for the interaction between processing component 702 and other assemblies.For example, processing component 702 may include multi-media module, withFacilitate the interaction between multimedia component 708 and processing component 702.
Memory 704 is configured as storing various types of data to support the operation in equipment 700.These data are shownExample includes the instruction of any application or method for operating on electronic equipment 700, contact data, telephone directory numberAccording to, message, picture, video etc..Memory 704 can by any kind of volatibility or non-volatile memory device or theyCombination realize, such as SRAM (Static Random Access Memory, static random access memory), EEPROM(Electrically Erasable Programmable Read Only Memory, the read-only storage of electrically erasableDevice), EPROM (Erasable Programmable Read-Only Memory, Erasable Programmable Read Only Memory EPROM), PROM(Programmable Read-Only Memory, programmable read only memory), ROM, magnetic memory, flash memory, diskOr CD.
Power supply module 706 provides electric power for the various assemblies of device 700.Power supply module 706 may include that power management is setIt is standby, one or more power supplys and other with for device 700 generate, manage, and distribute the associated component of electric power.
Multimedia component 708 includes the screen of one output interface of offer between the equipment 700 and user.OneIn a little embodiments, screen may include liquid crystal display (LCD) and touch panel (TP).If screen includes touch panel, screenCurtain may be implemented as touch screen, to receive input signal from the user.Touch panel includes one or more touch sensingsDevice is to sense the gesture on touch, slide, and touch panel.The touch sensor can not only sense touch or sliding actionBoundary, but also detect duration and pressure associated with the touch or slide operation.In some embodiments, more matchmakersBody component 708 includes a front camera and/or rear camera.When equipment 700 is in operation mode, such as screening-mode orWhen video mode, front camera and/or rear camera can receive external multi-medium data.Each front camera andRear camera can be a fixed optical lens equipment or have focusing and optical zoom capabilities.
I/O interface 710 provides interface between processing component 702 and peripheral interface module, and above-mentioned peripheral interface module canTo be keyboard, click wheel, button etc..These buttons may include, but are not limited to: home button, volume button, start button and lockDetermine button.
In the exemplary embodiment, electronic equipment 700 can be by one or more application ASIC (ApplicationSpecific Integrated Circuit, specific integrated circuit), DSP (Digital Signal Processor, number letterNumber processor), DSPD (Digital Signal Processing Equipment, digital signal processing appts), PLD(Programmable Logic Devices, programmable logic device), FPGA (Field Programmable GateArray, field programmable gate array), controller, microcontroller, microprocessor or other electronic components realize, for executingState image processing method.
In addition, being contained in electronic equipment present invention also provides a kind of non-transitorycomputer readable storage medium, work as instituteWhen stating instruction in storage medium and being executed by the processor of electronic equipment, so that electronic equipment is able to carry out in the embodiment of the present applicationThe step of any described image processing method.
In the exemplary embodiment, a kind of non-transitorycomputer readable storage medium including instruction, for example including fingerThe memory 402 of order, above-metioned instruction can be executed by processor 401 to complete the above method；Alternatively, including the memory of instruction704, above-metioned instruction can be executed by the processing component device 702 of electronic equipment 700 to complete the image that any of the above-described embodiment providesProcessing method.For example, the non-transitorycomputer readable storage medium can be ROM (Read-Only Memory, it is read-only to depositReservoir), RAM (Random Access Memory, random access memory), CD-ROM (Compact Disc Read-OnlyMemory, compact disc read-only memory), tape, floppy disk and optical data storage devices etc..
The technical solution that embodiments herein provides can include the following benefits: the depth information of image to be processedThe scene of each pixel representative of image to be processed be can reflect at a distance from real world between image capture device, andAnd each pixel can reflect the position between the different scenes that each pixel respectively represents in the two-dimensional position in image coordinate systemSet relationship.Therefore, the two-dimensional position according to the depth information of pixel and pixel in image coordinate system can obtain pixel and existThree-dimensional position in image collecting device coordinate system, the three-dimensional position are able to reflect the scene in image to be processed in Image AcquisitionThree-dimensional structure in device coordinate system.On this basis, view parameter be Orientation observation corresponding with image to be processed visual angle notThe parameter at same visual angle, also, focus point is the point changed when observing the visual angle of scene in image to be processed as rotary shaft.CauseThis, can obtain the offset of the pixel according to the focusing three-dimensional position, view parameter and pixel three-dimensional position of focus pointThree-dimensional position afterwards.Also, the three-dimensional position after the offset is that image collecting device is observed under view parameter in pixel threeWhen tieing up the scene of position, the three-dimensional position for the scene observed.Therefore, respectively according to the three-dimensional position after the offset of each pixelIt sets, by the two-dimensional coordinate system of each pixel projection to image to be processed, obtained target image is that image collecting device is regardingWhen being acquired under angular dimensions to the scene, the image of the scene collected.The bandwagon effect of target image just has as a result,There are different bandwagon effects corresponding to the different view parameter in Orientation observation visual angle corresponding from image to be processed.As it can be seen that passing throughThis programme, which can be realized the scene in image to be processed, to be had when being observed in real world, from different observation visual angles pairThe different bandwagon effects answered.
In another embodiment provided by the present application, a kind of computer program product comprising instruction is additionally provided, when itWhen running on an electronic device, so that electronic equipment executes any described image processing method in above-described embodiment.
In the above-described embodiments, can come wholly or partly by software, hardware, firmware or any combination thereof realIt is existing.When implemented in software, it can entirely or partly realize in the form of a computer program product.The computer programProduct includes one or more computer instructions.When loading on computers and executing the computer program instructions, all orIt partly generates according to process or function described in the embodiment of the present application.The computer can be general purpose computer, dedicated meterCalculation machine, computer network or other programmable devices.The computer instruction can store in computer readable storage mediumIn, or from a computer readable storage medium to the transmission of another computer readable storage medium, for example, the computerInstruction can from a web-site, computer, server or data center by wired, such as: coaxial cable, optical fiber, DSL(Digital Subscriber Line, Digital Subscriber Line；Or it is wireless, such as: the modes such as infrared ray, radio, microwave are to anotherOne web-site, computer, server or data center are transmitted.The computer readable storage medium can be calculatingAny usable medium that machine can access either includes the numbers such as one or more usable mediums integrated server, data centerAccording to storage equipment.The usable medium can be magnetic medium, such as: floppy disk, hard disk, tape；Optical medium, such as: DVD(Digital Versatile Disc, digital versatile disc)；Or semiconductor medium, such as: SSD (Solid StateDisk, solid state hard disk) etc..
Those skilled in the art after considering the specification and implementing the invention disclosed here, will readily occur to its of the applicationIts embodiment.This application is intended to cover any variations, uses, or adaptations of the application, these modifications, purposes orPerson's adaptive change follows the general principle of the application and including the undocumented common knowledge in the art of the applicationOr conventional techniques.The description and examples are only to be considered as illustrative, and the true scope and spirit of the application are by followingClaim is pointed out.
Herein, relational terms such as first and second and the like be used merely to by an entity or operation with it is anotherOne entity or operation distinguish, and without necessarily requiring or implying between these entities or operation, there are any this realityRelationship or sequence.Moreover, the terms "include", "comprise" or its any other variant are intended to the packet of nonexcludabilityContain, so that the process, method, article or equipment for including a series of elements not only includes those elements, but also includingOther elements that are not explicitly listed, or further include for elements inherent to such a process, method, article, or device.In the absence of more restrictions, the element limited by sentence "including a ...", it is not excluded that including the elementProcess, method, article or equipment in there is also other identical elements.
Each embodiment in this specification is all made of relevant mode and describes, identical and similar between each embodimentPart may refer to each other, each embodiment focuses on the differences from other embodiments.Especially for dressSet with for apparatus embodiments, since it is substantially similar to the method embodiment, so be described relatively simple, related place referring toThe part of embodiment of the method illustrates.
It should be understood that the application is not limited to the precise structure that has been described above and shown in the drawings, andAnd various modifications and changes may be made without departing from the scope thereof.Scope of the present application is only limited by the accompanying claims.