Grid generation method, grid generation device, electronic equipment and computer readable storage mediumTechnical neighborhood
The present disclosure relates to the field of image processing technologies, and in particular, to a grid generating method, a grid generating device, an electronic device, and a computer readable storage medium.
Background
Augmented reality is an important topic in the field of three-dimensional perception, and the use of dense grids in augmented reality applications can describe three-dimensional scene information more fully. Based on the known dense grids, some common 3D scene interactions or special effects can be achieved, such as occlusion of objects, collisions, shadow mapping, etc. The traditional dense grid generation algorithm sequentially fuses depth maps of different visual angles into the structured data according to the internal and external parameters of the camera, and extracts corresponding dense grids.
The traditional dense grid generation algorithm extracts grids for all structured data after fusing a depth image into the structured data, and has the disadvantages of large calculation amount, long time consumption and poor efficiency.
Disclosure of Invention
The embodiment of the disclosure at least provides a grid generation method and device.
In a first aspect, an embodiment of the present disclosure provides a grid generating method, including:
Acquiring a current depth image and structural data corresponding to a front depth image for grid generation before the current depth image;
Determining a voxel corresponding to the pixel in a preset voxel space based on the image coordinates and the depth values of the pixel in the current depth image;
Based on the neighborhood voxels of the voxels and the structured data corresponding to the front depth image, determining structured data increment information corresponding to the neighborhood voxels of the voxels;
And generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels.
In the aspect, based on the structured data increment information corresponding to the voxels, the grids are extracted, the target grids are generated instead of extracting grids for all structured data, the calculated amount of grid extraction can be reduced, the time consumption of grid extraction is reduced, and the grid extraction efficiency is effectively improved.
In a possible implementation manner, the determining, based on the neighborhood voxels of the voxels and the structured data corresponding to the previous depth image, structured data increment information corresponding to the neighborhood voxels of the voxels includes:
determining bias information corresponding to the neighborhood voxels based on voxel coordinates of the neighborhood voxels in the preset voxel space;
determining a key value of the neighborhood voxel based on the offset information of the neighborhood voxel;
and determining structured data increment information corresponding to the neighborhood voxels of the voxels based on the key value of the neighborhood voxels and the structured data corresponding to the front depth image.
In this embodiment, generating the voxel key value using the offset information can avoid a hash collision when a voxel outside the voxel space is queried. In addition, by comparing the key value of the voxel with the structured data corresponding to the front depth image, more accurate structured data increment information can be determined.
In one possible implementation, the structured data corresponding to the front depth image includes key values of voxels corresponding to the front depth image;
the determining the structured data increment information corresponding to the neighborhood voxel of the voxel based on the key value of the neighborhood voxel and the structured data corresponding to the front depth image comprises the following steps:
Projecting the neighborhood voxel onto the current depth image in response to the key value of the neighborhood voxel being included in the structured data;
In response to the neighborhood voxel being projected onto the current depth image, and if voxel depth information obtained by the neighborhood voxel being projected onto the current depth image is within a preset threshold, determining structured data increment information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel and pixel depth information of pixels corresponding to the neighborhood voxel in the current depth image, or
And determining structured data increment information corresponding to the neighborhood voxel based on voxel depth information corresponding to the neighborhood voxel in response to the fact that the key value of the neighborhood voxel is not included in the structured data.
In this embodiment, the structured data incremental information can be determined more accurately by using the result of the projection of the neighboring voxels and the voxel depth information and/or the pixel depth information corresponding to the voxels.
In one possible embodiment, the structured data increment information includes state information of a corresponding voxel and a truncated symbol distance function increment value of the corresponding voxel;
the determining the structured data increment information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image includes:
Taking the difference value of the voxel depth information corresponding to the neighborhood voxels and the pixel depth information corresponding to the neighborhood voxels as a truncated symbol distance function increment value corresponding to the neighborhood voxels;
and setting the state information of the neighborhood voxels to an updated state.
In one possible embodiment, the structured data increment information includes state information of a corresponding voxel and a truncated symbol distance function increment value of the corresponding voxel;
the determining the structured data increment information corresponding to the neighborhood voxels based on the voxel depth information corresponding to the neighborhood voxels comprises the following steps:
taking the voxel depth information as a truncated symbol distance function increment value corresponding to the neighborhood voxel;
And setting the state information of the neighborhood voxels to be an addition state.
In this embodiment, the structured data incremental information can be determined more accurately by using voxel depth information and/or pixel depth information corresponding to the voxels.
In a possible implementation manner, the determining the structured data increment information corresponding to the neighborhood voxel of the voxel based on the key value of the neighborhood voxel and the structured data corresponding to the front depth image further includes:
And setting the state information of the neighborhood voxels to be in a deleted state in response to the fact that the neighborhood voxels are not projected onto the current depth image or the voxel depth information obtained by the fact that the neighborhood voxels are projected onto the current depth image is not within the preset threshold range.
In this embodiment, voxels that cannot be projected onto the current depth image or voxels whose voxel depth information is not within the preset threshold range are set to the deleted state, so that voxels with low accuracy can be deleted, which is beneficial to improving the quality of the generated grid.
In one possible embodiment, the offset information includes first offset information and second offset information of the corresponding voxels;
the determining offset information corresponding to the neighborhood voxel based on the voxel coordinates of the neighborhood voxel in the preset voxel space comprises the following steps:
screening a coordinate item with the largest absolute value from the voxel coordinates;
determining the first offset information based on the coordinate item with the maximum absolute value and the number of unit voxels included in the preset voxel space;
the second bias information is determined based on the first bias information and the number of unit voxels included in the preset voxel space.
In this embodiment, offset information can be generated relatively accurately based on voxel coordinates.
In one possible implementation manner, the generating the target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels includes:
Determining target structured data for the voxel based on the structured data delta information for the voxel and the structured data;
And generating a target grid corresponding to the current depth image based on the target structured data of the voxels.
According to the embodiment, the structured data increment information is added into the original structured data to generate the target structured data, and the target structured data can be used for generating a more accurate target grid.
In one possible embodiment, the structured data increment information includes state information of a corresponding voxel and a truncated symbol distance function increment value of the corresponding voxel;
the generating a target grid corresponding to the current depth image based on the target structured data of the voxels comprises:
Responding to the state information of the voxels of the neighborhood voxels of the voxels to be in a preset state, and generating an increment grid corresponding to the voxels based on the target structural data of the voxels under the condition that the number of times that the state information of the voxels of the neighborhood voxels is in the preset state is determined to be greater than the preset number of times;
and generating a target grid corresponding to the current depth image based on the incremental grid corresponding to the voxel and the grid corresponding to the front depth image.
According to the embodiment, when the state information is updated in the preset state for more than the preset times, the grids are extracted from the corresponding target structured data, repeated grid extraction is avoided, the calculated amount of grid extraction is reduced, the calculation resources are saved, and the grid extraction efficiency is improved and the grid extraction noise is reduced.
In one possible implementation manner, the grid generating method further includes:
Acquiring an optimized pose parameter corresponding to shooting equipment for shooting the depth image and a pose parameter when the shooting equipment shoots each depth image;
determining a pose change value based on the pose parameter corresponding to each depth image and the optimized pose parameter;
In response to the situation that the pose change value is larger than a preset pose change threshold, removing the structured data increment information of the voxels corresponding to the depth image from the target structured data, and determining the final structured data increment information of the voxels corresponding to the depth image based on the optimized pose parameters;
based on the final structured data increment information of the voxels corresponding to the depth image, the structured data corresponding to the depth image is determined.
According to the embodiment, when the pose of the shooting equipment is changed greatly, the structured data increment information generated according to the non-optimized pose parameters is removed from the target structured data, new and more accurate structured data increment information is generated based on the optimized pose parameter information, and the more accurate structured data can be generated based on the structured data increment information, so that grid layering dislocation phenomenon is reduced.
In a possible implementation manner, the structured data increment information comprises state information of corresponding voxels, truncated symbol distance function increment values of the corresponding voxels;
the grid generation method further comprises the following steps:
And setting the state information of the voxels corresponding to the depth image to a deleted state in response to the situation that the truncated symbol distance function value of the voxels corresponding to the depth image is smaller than zero after the structured data increment information of the voxels corresponding to the depth image is removed from the target structured data.
In the embodiment, the voxels with the truncated symbol distance function value smaller than zero are set to be in the deleted state, so that voxels with low accuracy can be deleted, the quality of the generated grid is improved, and the grid layering dislocation phenomenon is reduced.
In one possible implementation manner, before the generating the target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels, the method further includes:
screening target voxels with empty structured data increment information from voxels corresponding to the current depth image;
projecting the target voxel onto the current depth image;
And in response to the fact that the target voxel is projected onto the current depth image, and in the case that voxel depth information obtained by the fact that the target voxel is projected onto the current depth image is within a preset threshold range, determining structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and pixel depth information of pixels corresponding to the target voxel in the current depth image.
According to the embodiment, the moving object is determined by using the structural data and the depth image which are not updated, and the corresponding structural data increment information is updated, so that the accuracy of grid extraction can be improved.
In one possible embodiment, the structured data increment information includes state information of a corresponding voxel and a truncated symbol distance function increment value of the corresponding voxel;
The determining the structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image includes:
Responding to the situation that the difference value obtained by subtracting the pixel depth information corresponding to the target voxel from the voxel depth information corresponding to the target voxel is smaller than zero, and taking the difference value obtained by subtracting the pixel depth information from the voxel depth information as a truncated symbol distance function increment value corresponding to the target voxel;
And setting the state information of the target voxel to be an updated state.
In this embodiment, the structured data incremental information of the voxels with the difference value smaller than zero between the voxel depth information and the corresponding pixel depth information is updated, which is the structured data incremental information of the moving object is substantially updated, and the accuracy of grid extraction can be improved.
In a possible implementation manner, before the determining, based on the image coordinates of the pixel in the current depth image, a corresponding voxel of the pixel in a preset voxel space, the method further includes:
for each pixel in the current depth image, calculating a pixel depth difference value between the pixel depth of the pixel and the pixel depth of a neighborhood pixel of the pixel;
and setting the pixel as invalid in response to the pixel depth difference value corresponding to the pixel not being within the preset depth difference value range.
In the embodiment, the neighborhood pixel depth consistency check is carried out on the pixels in the current depth image, noise in the current depth image is removed, and the accuracy of grid extraction is improved.
In a second aspect, an embodiment of the present disclosure further provides a grid generating apparatus, including:
The data acquisition module is used for acquiring a current depth image and structured data corresponding to a front depth image used for grid generation before the current depth image;
the voxel determining module is used for determining voxels corresponding to the pixels in a preset voxel space based on the image coordinates and the depth values of the pixels in the current depth image;
the incremental information determining module is used for determining the structured data incremental information corresponding to the neighborhood voxels of the voxels based on the neighborhood voxels of the voxels and the structured data corresponding to the front depth image;
And the grid generation module is used for generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels.
In a third aspect, the disclosed embodiments also provide an electronic device comprising a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, the processor and the memory in communication via the bus when the electronic device is in operation, the machine-readable instructions when executed by the processor performing the steps of the first aspect, or any of the possible implementations of the first aspect.
In a fourth aspect, the presently disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the first aspect, or any of the possible implementations of the first aspect.
The description of the effects of the above grid generating apparatus, the electronic device, and the computer-readable storage medium is referred to the description of the above grid generating method, and is not repeated here.
The foregoing objects, features and advantages of the disclosure will be more readily apparent from the following detailed description of the preferred embodiments taken in conjunction with the accompanying drawings.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for the embodiments are briefly described below, which are incorporated in and constitute a part of the specification, these drawings showing embodiments consistent with the present disclosure and together with the description serve to illustrate the technical solutions of the present disclosure. It is to be understood that the following drawings illustrate only certain embodiments of the present disclosure and are therefore not to be considered limiting of its scope, for the person of ordinary skill in the art may admit to other equally relevant drawings without inventive effort.
FIG. 1 illustrates a flow chart of a grid generation method provided by an embodiment of the present disclosure;
FIG. 2 shows a schematic diagram of a neighborhood voxel in an embodiment of the present disclosure;
FIG. 3 shows a schematic diagram of a preset voxel space in an embodiment of the present disclosure;
FIG. 4 shows a schematic diagram of an extracted grid in an embodiment of the present disclosure;
FIG. 5 shows a schematic diagram of a mesh generation apparatus provided by an embodiment of the present disclosure;
Fig. 6 shows a schematic diagram of an electronic device provided by an embodiment of the disclosure.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is apparent that the described embodiments are only some embodiments of the present disclosure, but not all embodiments. The components of the embodiments of the present disclosure, which are generally described and illustrated in the figures herein, may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the disclosure, as claimed, but is merely representative of selected embodiments of the disclosure. All other embodiments that may be made by those of ordinary skill in the art, based on the embodiments of the present disclosure, without the exercise of inventive faculty, are intended to be within the scope of the present disclosure.
It should be noted that like reference numerals and letters refer to like items in the following figures, and thus once an item is defined in one figure, no further definition or explanation thereof is necessary in the following figures.
The term "and/or" is used herein to describe only one relationship, and means that three relationships may exist, for example, A and/or B, and that three cases exist, A alone, A and B together, and B alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, C, may mean including any one or more elements selected from the group consisting of A, B and C.
Aiming at the prior art, when grid extraction is carried out, grids are extracted for all structured data after each depth image is fused into the structured data, so that the calculation amount is large, the time consumption is long and the efficiency is poor. In view of the defect, the embodiment of the disclosure provides a grid generation method, a device, an electronic device and a computer readable storage medium, and the embodiment of the disclosure extracts grids based on structured data increment information corresponding to voxels to generate a target grid instead of extracting grids from all structured data, so that the calculation amount of grid extraction can be reduced, the time consumption of grid extraction is reduced, and the grid extraction efficiency is effectively improved.
The grid generation method, the grid generation device, the electronic equipment and the storage medium disclosed by the disclosure are described below through specific embodiments.
As shown in fig. 1, an embodiment of the present disclosure discloses a grid generation method, which may be applied to a device such as a server that needs to generate a grid, for generating a target grid based on a depth image. Specifically, the mesh generation method may include the steps of:
S110, acquiring a current depth image and structuring data corresponding to a front depth image used for grid generation before the current depth image.
The depth image may be captured by a depth camera and transmitted to a device performing the grid generating method. The front depth image may include one or more depth images, which are images that generate structured data before the current depth image and perform mesh extraction based on the extracted structured data.
The embodiment is described with respect to how the current depth image extracts the grid, and the method for extracting the grid from the previous depth image may be the same as the method for extracting the grid from the current depth image.
The "meshes" in the mesh extraction, generation target mesh in this embodiment each refer to a dense mesh, and the extracted or generated dense mesh may be used to generate an augmented reality AR image or AR special effect.
When generating or extracting the grid, firstly fusing the structured data corresponding to the depth image into the structured data obtained after the previous depth image fusion, and then performing grid extraction operation on the fused structured data to obtain the grid. Therefore, when the grid is generated by the front depth image, corresponding structured data and fused structured data are generated first.
S120, determining a voxel corresponding to the pixel in a preset voxel space based on the image coordinates of the pixel in the current depth image.
Here, as shown in fig. 3, the preset voxel space may be flexibly set according to the actual application scene, and for example, the preset voxel space may be set to a voxel space composed of N3 unit voxels, n=500. The voxel coordinate system range of the voxel space is [ -N/2, N/2], the corresponding actual size of the unit voxel is ζ, ζ=0.06 meter. The actual size of the preset voxel space is N x ζ.
In fig. 3, a voxel located in a preset voxel space is a voxel 301, and a voxel located outside the preset voxel space is a voxel 302.
In determining the voxels of a pixel in particular, this can be achieved by the following steps:
In determining the voxel coordinates of a pixel, the depth value of the pixel is required to be combined, specifically, the pixel coordinates p (i, j) of the pixel are acquired, the depth is d (i, j), and the pixel is projected into the world coordinate system by utilizing the internal and external parameters of a shooting device for shooting the current depth image to obtain the three-dimensional coordinateAnd then the three-dimensional coordinates are obtainedProjection to voxel coordinatesVoxel coordinates (x, y, z) are obtained.
In order to save the memory, we only need to manage the transformed voxels, and not maintain the whole preset voxel space.
S130, based on the neighborhood voxels of the voxels and the structured data corresponding to the front depth image, determining structured data increment information corresponding to the neighborhood voxels of the voxels.
As shown in fig. 2, the neighborhood voxels of a certain voxel may include voxels 201 located on 8 vertices of the cube corresponding to the unit voxel where the voxel is located.
In a particular embodiment, the structured data delta information may include state information for the corresponding voxel and a truncated symbol distance function tsdf delta value for the corresponding voxel. The state information may include an added state, an updated state, and a deleted state. The truncated symbol distance function increment value may be a difference between voxel depth information obtained by projecting a certain voxel onto the current depth image and pixel depth information of a pixel corresponding to the voxel.
When the structured data increment information is determined, the key value of the neighborhood voxel can be calculated first, then whether the calculated key value exists in the structured data obtained by fusion of the previous depth image is judged, if not, the structured data of the neighborhood voxel is used as the structured data increment information, and if so, the structured increment information is determined based on the voxel coordinates favorable for the voxel and the corresponding pixel coordinates.
And S140, generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels.
Specifically, the structured data increment information of a certain voxel may be fused with the structured data corresponding to the voxel in the structured data obtained by fusing the previous depth image, so as to obtain the target structured data of the voxel, and then the target grid is generated based on the grid obtained by extracting the grid from the target structured data.
In specific implementation, the structured data increment information can be added to the structured data corresponding to the corresponding voxels in a weighted summation mode to obtain the target structured data.
The structured data increment information is added into the original structured data to generate target structured data, and the target structured data can be used for generating more accurate target grids.
In some embodiments, the determining the structured data increment information corresponding to the neighborhood voxel of the voxel based on the neighborhood voxel of the voxel and the structured data corresponding to the previous depth image may specifically be implemented by using the following steps:
and 11, determining offset information corresponding to the neighborhood voxels based on voxel coordinates of the neighborhood voxels in the preset voxel space.
In an implementation, the offset information may include first offset information and second offset information of the neighborhood voxels. The method can be realized by the following substeps:
and step 111, selecting the coordinate item with the largest absolute value from the voxel coordinates of the neighborhood voxels.
The coordinate term with the largest absolute value can be screened by using the following formula:
The coordinate item with the largest absolute value is selected.
And a substep 112 of determining the first offset information based on the coordinate item with the largest absolute value and the number of unit voxels included in the preset voxel space.
At the position ofWhen the first offset information can be calculated using the following formula:
Where sl denotes the first offset information.
When the first offset information can be calculated using the following formula:
A substep 113 of determining the second offset information based on the first offset information and the number of unit voxels comprised by the preset voxel space.
The second offset information may be calculated here specifically using the following formula:
Where sG denotes the second bias information and k denotes the index value of the voxel.
In this embodiment, offset information can be generated relatively accurately based on voxel coordinates.
And step 12, determining the key value of the neighborhood voxel based on the offset information of the neighborhood voxel.
In implementations, the key value of a neighborhood voxel may be calculated using the following formula:
Wherein,
In the formula,A key representing a neighborhood voxel.
The key value calculation mode can avoid hash collision when inquiring voxels outside the preset voxel space, the inquiry range is limited in the preset voxel space, and the voxels obtained by converting any depth image can be obtained, so that grid generation is not limited by the set generation range.
And step 13, based on the key value of the neighborhood voxel and the structured data corresponding to the front depth image, determining structured data increment information corresponding to the neighborhood voxel of the voxel.
Here, the structured data includes key values of voxels corresponding to the front depth image. After obtaining the key value of the neighborhood voxel corresponding to the current depth image, judging whether the key value of the neighborhood voxel is included in the structured data, and projecting the neighborhood voxel onto the current depth image when the key value of the neighborhood voxel is included in the structured data. And if the neighborhood voxel is projected onto the current depth image and the voxel depth information obtained by projecting the neighborhood voxel onto the current depth image is within a preset threshold range, determining structured data increment information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image.
Specifically, the difference value between the voxel depth information corresponding to the neighborhood voxel and the pixel depth information corresponding to the neighborhood voxel is used as a truncated sign distance function increment value corresponding to the neighborhood voxel, and the state information of the neighborhood voxel is set to be an updated state.
The preset threshold range can be flexibly set according to specific application scenarios, and can be set as [0.4m,2.5m ].
And if the key value of the neighborhood voxel is not included in the structured data, determining structured data increment information corresponding to the neighborhood voxel based on voxel depth information corresponding to the neighborhood voxel.
The voxel depth information is used as a truncated symbol distance function increment value corresponding to the neighborhood voxels, and the state information of the neighborhood voxels is set to be an addition state.
The structured data increment information can be accurately determined by utilizing voxel depth information and/or pixel depth information corresponding to the voxels.
And setting the state information of the neighborhood voxels to be in a deleted state in response to the fact that the neighborhood voxels are not projected onto the current depth image or the voxel depth information obtained by the fact that the neighborhood voxels are projected onto the current depth image is not within the preset threshold range.
Voxels with low accuracy can be deleted by setting voxels which cannot be projected onto the current depth image or voxels with voxel depth information which is not within a preset threshold range to be in a deleted state, which is beneficial to improving the quality of the generated grid.
The above embodiment can determine the structured data increment information more accurately by using the result of the projection of the neighborhood voxels and the voxel depth information and/or the pixel depth information corresponding to the voxels.
Generating the key value of the voxel by using the offset information can avoid hash collision when the voxel outside the voxel space is queried. In addition, by comparing the key value of the voxel with the structured data corresponding to the front depth image, more accurate structured data increment information can be determined.
After the above steps are performed, a counter may be used to record the update condition of the state information of each neighborhood voxel, specifically, if the state information of a certain neighborhood voxel is in an added state, the counter of the neighborhood voxel is set to 1, if the state information of a certain neighborhood voxel is in an updated state, the counter of the neighborhood voxel is added to 1 on the basis of the original value, and if the state information of a certain neighborhood voxel is in a deleted state, the counter of the neighborhood voxel is set to 0.
Based on the value of the neighborhood voxel recorded by the counter, determining whether to carry out grid extraction on the neighborhood voxel, which can be realized by the following steps:
and step 21, responding to the state information of the voxels of the neighborhood voxels of the voxels to be in a preset state, such as an addition state or an update state, and generating an increment grid corresponding to the voxels based on the target structured data of the voxels when the number of times of determining the state information of the voxels of the neighborhood voxels to be in the preset state is larger than the preset number of times, namely, when the value in the counter corresponding to the neighborhood voxels is larger than the preset number of times.
The preset times can be flexibly set according to specific application scenes, and for example, can be set to 3.
The method can be specifically implemented by utilizing Marching Cubes algorithm when extracting the grid from the target structured data.
After the mesh is extracted for a certain voxel, the state information of the voxel is set to a normal state.
And 22, generating a target grid corresponding to the current depth image based on the incremental grid corresponding to the voxel and the grid corresponding to the front depth image.
Specifically, the incremental grid generated by each voxel is added to the grid generated by the front depth image, so as to obtain a target grid.
As shown in fig. 4, incremental grid 401 is added to existing grid 402, with the area of the grid increasing.
And when the state information is updated in the preset state for more than the preset times, extracting the grids from the corresponding target structured data, thereby avoiding repeated grid extraction, reducing the calculated amount of grid extraction, saving the calculation resources, and being beneficial to improving the grid extraction efficiency and reducing the grid extraction noise.
If the state information of one voxel in the neighborhood voxels of the voxel is in a deleted state, the grid is not extracted for the voxel.
The pose of the shooting device for shooting the depth image may change, if the pose of the shooting device is optimized and the pose change of the shooting device is larger than that of the pose used for extracting the grid before, the structural data generated before needs to be deleted or updated, and the method can be realized by the following steps:
And step 31, acquiring the optimized pose parameters corresponding to the shooting equipment for shooting the depth images and the pose parameters when the shooting equipment shoots each depth image.
In a specific implementation, an external parameter of a photographing device, for example, a camera, may be denoted as Tit, where the external parameter includes six parameters, the first three parameters are translation parameters of the camera, the second three parameters are rotation parameters of the camera, i represents a frame number of a depth image, and T represents a time sequence number of the depth image.
Step 32, determining a pose change value according to the pose parameter corresponding to each depth image and the optimized pose parameter.
After the optimized pose parameters are received, the pose change value is calculated according to the pose parameters corresponding to each frame of depth image, and specifically, the pose change value can be calculated according to the following formula:
ΔTit′＝||Tit′-Tit||2
Wherein Tit′ represents the optimized pose parameter, ||2 denotes an L2 norm.
And step 33, eliminating the structured data increment information of the voxels corresponding to the depth image from the target structured data in response to the situation that the pose change value is larger than the preset pose change threshold, and determining the final structured data increment information of the voxels corresponding to the depth image based on the optimized pose parameters.
The preset pose change threshold value can be flexibly set according to a specific application scene, for example, can be set to be 0.01.
When the structured data incremental information is removed, the pose parameter corresponding to the depth image can be utilized to determine the voxel coordinates of each voxel in the depth image, then the structured data incremental information corresponding to each voxel is calculated by the same method as the embodiment, and the calculated structured data incremental information is deleted from the target structured data. The structured data increment information corresponding to the stored depth image can also be directly utilized to delete the structured data increment information from the target structured data.
After the structured data increment information of the voxel corresponding to the depth image is removed from the target structured data, if the truncated symbol distance function value of the voxel corresponding to the depth image is smaller than zero, setting the state information of the voxel corresponding to the depth image to be in a deleted state.
Voxels with the truncated symbol distance function value smaller than zero are set to be in a deleted state, voxels with low accuracy can be deleted, the quality of the generated grid is improved, and the grid layering dislocation phenomenon is reduced.
And then, determining the voxel coordinates of each voxel in the depth image by using the optimized pose parameters, and calculating the structured data increment information corresponding to each voxel by using the same method as the embodiment.
And step 34, determining the structured data corresponding to the depth image based on the final structured data increment information of the voxels corresponding to the depth image.
Here, the structured data incremental information may be added to the structured data corresponding to the corresponding voxel by means of weighted summation.
When the pose of the shooting equipment is greatly changed, the embodiment eliminates the structured data increment information generated according to the non-optimized pose parameters from the target structured data, generates new and more accurate structured data increment information based on the optimized pose parameters, and can generate more accurate structured data based on the structured data increment information, thereby being beneficial to reducing grid layering dislocation phenomenon.
The grid generation technology in the prior art does not carry out special treatment on dynamic objects in a scene, can not update the dynamic objects in the scene in real time, and the generated grid is not accurate enough. The embodiment of the disclosure can specifically realize the processing of the dynamic object by using the following steps:
Before generating a target grid corresponding to a current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels, screening target voxels with empty structured data increment information from the voxels corresponding to the current depth image, and then projecting the target voxels onto the current depth image. And under the condition that the target voxel is projected onto the current depth image and voxel depth information obtained by the projection of the target voxel onto the current depth image is within a preset threshold range, determining structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and pixel depth information of pixels corresponding to the target voxel in the current depth image.
The projection of the target voxels to the current depth image may be achieved in particular by means of pose parameters of the camera device.
The preset threshold range can be flexibly set according to specific application scenes, and can be set as 0.4m and 2.5 m.
The determining the structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image may specifically be:
And setting the state information of the target voxel as an updated state, wherein the difference value obtained by subtracting the pixel depth information from the voxel depth information is used as a truncated symbol distance function increment value corresponding to the target voxel under the condition that the difference value obtained by subtracting the pixel depth information corresponding to the target voxel from the voxel depth information corresponding to the target voxel is smaller than zero.
The structural data increment information of the voxels with the difference value smaller than zero between the voxel depth information and the corresponding pixel depth information is updated, so that the structural data increment information of the moving object is substantially updated, and the grid extraction accuracy can be improved.
In processing a dynamic object, it is noted that, although the state information of a voxel is updated, the value in the counter corresponding to the voxel is not modified.
The grid generation technology in the prior art is used for processing a high-precision depth image acquired by a depth sensor, and for a depth image with strong noise and poor quality, the generated grid noise is more, and the embodiment of the disclosure can firstly perform noise reduction treatment before processing the depth image so as to improve the accuracy of the generated grid, and can be realized by the following steps:
Step 41, before determining the corresponding voxel of the pixel in the preset voxel space based on the image coordinates of the pixel in the current depth image, calculating, for each pixel in the current depth image, a pixel depth difference value between the pixel depth of the pixel and the pixel depth of the neighborhood pixel of the pixel.
The neighborhood pixels may include pixels within a certain range adjacent to the pixel, for example, a pixel may have coordinates p (i, j), and the neighborhood pixel of the pixel may be a pixel having coordinates within { i+Δi, j+Δj }. Where Δi e (-R, R), Δj e (-R, R), the neighborhood radius can be set to 3 pixels.
And step 42, setting the pixel as invalid in response to the pixel depth difference value corresponding to the pixel not being within the preset depth difference value range.
The preset depth difference may be set to a fixed value according to a specific scene, or the preset depth difference may be determined according to the depth information of the pixel, for example, the preset depth difference is set to 15% of the depth value corresponding to the pixel.
And performing neighborhood pixel depth consistency check on pixels in the current depth image, removing noise in the current depth image, and improving the accuracy of grid extraction.
According to the grid generation method in the embodiment, grids are extracted from corresponding target structured data only when the accumulated update times of the state information in each depth image reach a certain value and are combined into the previous grids, so that repeated access to all structured data is avoided, and generated grid noise is also greatly reduced. The embodiment reduces the amount of structured data to be maintained, is beneficial to grid extraction, does not generate voxel access conflict, can be arbitrarily expanded to be outside a preset voxel space, and does not need to set a grid generation range in advance. Furthermore, the embodiment solves the problem of high memory occupation in the traditional method, and even the mobile phone at the middle and low ends can also run the grid generation method in real time. Furthermore, whether a dynamic object exists in the depth image is judged in the process of generating the structured data, and if the dynamic object exists, the corresponding structured data increment information is updated and removed, so that the accuracy of grid generation can be improved. Furthermore, before the depth image is processed, the embodiment performs consistency check to remove noise in the depth image, so as to reduce noise of the generated grid. Furthermore, the above embodiment re-fuses the frame with the larger change of the optimized pose parameter, namely deletes the structured data increment information fused by the old pose parameter of the frame, uses the pose parameter after the frame is optimized, and re-fuses the structured data increment information by the optimized pose parameter, thereby being beneficial to reducing the grid layering phenomenon.
Corresponding to the grid generation method, the disclosure further discloses a grid generation device, where each module in the device can implement each step in the grid generation method of each embodiment and can obtain the same beneficial effects, so that the description of the same parts is omitted here. Specifically, as shown in fig. 5, the mesh generation apparatus includes:
The data obtaining module 510 is configured to obtain a current depth image, and a structured number corresponding to a previous depth image used for grid generation before the current depth image.
The voxel determining module 520 is configured to determine a voxel corresponding to the pixel in a preset voxel space based on the image coordinate and the depth value of the pixel in the current depth image.
The incremental information determining module 530 is configured to determine, based on the neighborhood voxels of the voxel and the structured data corresponding to the previous depth image, structured data incremental information corresponding to the neighborhood voxels of the voxel.
The grid generating module 540 is configured to generate a target grid corresponding to the current depth image based on the structured data increment information corresponding to the neighborhood voxels of the voxels.
In some embodiments, delta information determination module 530, when determining structured data delta information corresponding to a neighborhood voxel of the voxel based on the neighborhood voxel of the voxel and the structured data corresponding to the front depth image, is to:
determining bias information corresponding to the neighborhood voxels based on voxel coordinates of the neighborhood voxels in the preset voxel space;
determining a key value of the neighborhood voxel based on the offset information of the neighborhood voxel;
and determining structured data increment information corresponding to the neighborhood voxels of the voxels based on the key value of the neighborhood voxels and the structured data corresponding to the front depth image.
In some embodiments, the structured data includes key values for voxels corresponding to the front depth image;
the incremental information determining module 530 is configured to, when determining the structured data incremental information corresponding to the neighborhood voxel of the voxel based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image:
Projecting the neighborhood voxel onto the current depth image in response to the key value of the neighborhood voxel being included in the structured data;
In response to the neighborhood voxel being projected onto the current depth image, and if voxel depth information obtained by the neighborhood voxel being projected onto the current depth image is within a preset threshold, determining structured data increment information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel and pixel depth information of pixels corresponding to the neighborhood voxel in the current depth image, or
And determining structured data increment information corresponding to the neighborhood voxel based on voxel depth information corresponding to the neighborhood voxel in response to the fact that the key value of the neighborhood voxel is not included in the structured data.
In some embodiments, the structured data delta information comprises state information of a corresponding voxel and a truncated symbol distance function delta value of the corresponding voxel;
the incremental information determining module 530 is configured to, when determining the structured data incremental information corresponding to the neighborhood voxel based on the voxel depth information corresponding to the neighborhood voxel and the pixel depth information of the pixel corresponding to the neighborhood voxel in the current depth image:
Taking the difference value of the voxel depth information corresponding to the neighborhood voxels and the pixel depth information corresponding to the neighborhood voxels as a truncated symbol distance function increment value corresponding to the neighborhood voxels;
setting the state information of the neighborhood voxels to an updated state;
the determining the structured data increment information corresponding to the neighborhood voxels based on the voxel depth information corresponding to the neighborhood voxels comprises the following steps:
taking the voxel depth information as a truncated symbol distance function increment value corresponding to the neighborhood voxel;
And setting the state information of the neighborhood voxels to be an addition state.
In some embodiments, the incremental information determining module 530 is configured to, when determining the structured data incremental information corresponding to the neighborhood voxel of the voxel based on the key value of the neighborhood voxel and the structured data corresponding to the previous depth image:
And setting the state information of the neighborhood voxels to be in a deleted state in response to the fact that the neighborhood voxels are not projected onto the current depth image or the voxel depth information obtained by the fact that the neighborhood voxels are projected onto the current depth image is not within the preset threshold range.
In some embodiments, the bias information includes first bias information and second bias information for corresponding voxels;
The incremental information determining module 530 is configured to, when determining offset information corresponding to the neighborhood voxel based on voxel coordinates of the neighborhood voxel in the preset voxel space:
screening a coordinate item with the largest absolute value from the voxel coordinates;
determining the first offset information based on the coordinate item with the maximum absolute value and the number of unit voxels included in the preset voxel space;
the second bias information is determined based on the first bias information and the number of unit voxels included in the preset voxel space.
In some embodiments, the grid generation module 540 is configured to, when generating the target grid corresponding to the current depth image based on the structured data delta information corresponding to the neighborhood voxels of the voxel:
Determining target structured data for the voxel based on the structured data delta information for the voxel and the structured data;
And generating a target grid corresponding to the current depth image based on the target structured data of the voxels.
In some embodiments, the structured data delta information comprises state information of a corresponding voxel and a truncated symbol distance function delta value of the corresponding voxel;
the grid generation module 540 is configured to, when generating a target grid corresponding to the current depth image based on the target structured data of the voxels:
Responding to the state information of the voxels of the neighborhood voxels of the voxels to be in a preset state, and generating an increment grid corresponding to the voxels based on the target structural data of the voxels under the condition that the number of times that the state information of the voxels of the neighborhood voxels is in the preset state is determined to be greater than the preset number of times;
and generating a target grid corresponding to the current depth image based on the incremental grid corresponding to the voxel and the grid corresponding to the front depth image.
In some embodiments, the incremental information determination module 530 is further configured to:
Acquiring an optimized pose parameter corresponding to shooting equipment for shooting the depth image and a pose parameter when the shooting equipment shoots each depth image;
determining a pose change value based on the pose parameter corresponding to each depth image and the optimized pose parameter;
In response to the situation that the pose change value is larger than a preset pose change threshold, removing the structured data increment information of the voxels corresponding to the depth image from the target structured data, and determining the final structured data increment information of the voxels corresponding to the depth image based on the optimized pose parameters;
based on the final structured data increment information of the voxels corresponding to the depth image, the structured data corresponding to the depth image is determined.
In some embodiments, the structured data delta information comprises state information of the corresponding voxel, truncated symbol distance function delta values of the corresponding voxel;
The incremental information determination module 530 is further configured to:
And setting the state information of the voxels corresponding to the depth image to a deleted state in response to the situation that the truncated symbol distance function value of the voxels corresponding to the depth image is smaller than zero after the structured data increment information of the voxels corresponding to the depth image is removed from the target structured data.
In some embodiments, the incremental information determining module 530 is further configured to, before generating the target grid corresponding to the current depth image based on the structured data incremental information corresponding to the neighborhood voxels of the voxels:
screening target voxels with empty structured data increment information from voxels corresponding to the current depth image;
projecting the target voxel onto the current depth image;
And in response to the fact that the target voxel is projected onto the current depth image, and in the case that voxel depth information obtained by the fact that the target voxel is projected onto the current depth image is within a preset threshold range, determining structured data increment information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and pixel depth information of pixels corresponding to the target voxel in the current depth image.
In some embodiments, the structured data delta information comprises state information of a corresponding voxel and a truncated symbol distance function delta value of the corresponding voxel;
The incremental information determining module 530 is configured to, when determining the structured data incremental information corresponding to the target voxel based on the voxel depth information corresponding to the target voxel and the pixel depth information of the pixel corresponding to the target voxel in the current depth image:
Responding to the situation that the difference value obtained by subtracting the pixel depth information corresponding to the target voxel from the voxel depth information corresponding to the target voxel is smaller than zero, and taking the difference value obtained by subtracting the pixel depth information from the voxel depth information as a truncated symbol distance function increment value corresponding to the target voxel;
And setting the state information of the target voxel to be an updated state.
In some embodiments, before the determining, based on the image coordinates of the pixel in the current depth image, that the pixel is in the preset voxel space, the voxel determining module 520 is further configured to:
for each pixel in the current depth image, calculating a pixel depth difference value between the pixel depth of the pixel and the pixel depth of a neighborhood pixel of the pixel;
and setting the pixel as invalid in response to the pixel depth difference value corresponding to the pixel not being within the preset depth difference value range.
Corresponding to the grid generation method, the embodiment of the present disclosure further provides an electronic device 600, as shown in fig. 6, which is a schematic structural diagram of the electronic device 600 provided in the embodiment of the present disclosure, including:
the processor 61, the memory 62, and the bus 63, the memory 62 is used for storing execution instructions, including a memory 621 and an external memory 622, the memory 621 is also called an internal memory, and is used for temporarily storing operation data in the processor 61 and data exchanged with the external memory 622 such as a hard disk, the processor 61 exchanges data with the external memory 622 through the memory 621, and when the electronic device 600 operates, the processor 61 and the memory 62 communicate through the bus 63, so that the processor 61 executes the following instructions:
The method comprises the steps of obtaining a current depth image and structured data corresponding to a front depth image used for grid generation before the current depth image, determining voxels corresponding to pixels in a preset voxel space based on image coordinates of the pixels in the current depth image, determining structured data increment information corresponding to neighbor voxels of the voxels based on neighbor voxels of the voxels and the structured data corresponding to the front depth image, and generating a target grid corresponding to the current depth image based on the structured data increment information corresponding to neighbor voxels of the voxels.
The disclosed embodiments also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the grid generation method described in the method embodiments described above. Wherein the storage medium may be a volatile or nonvolatile computer readable storage medium.
The disclosed embodiments also provide a computer program product comprising a computer readable storage medium storing a program code comprising instructions operable to perform the steps of the grid generating method described in the above method embodiments, in particular, the computer program product may be implemented in particular by means of hardware, software or a combination thereof. In an alternative embodiment, the computer program product is embodied as a computer storage medium, and in another alternative embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), or the like.
The present disclosure relates to the field of augmented reality, and more particularly, to the field of augmented reality, in which, by acquiring image information of a target object in a real environment, detection or identification processing of relevant features, states and attributes of the target object is further implemented by means of various visual correlation algorithms, so as to obtain an AR effect combining virtual and reality matching with a specific application. By way of example, the target object may relate to a face, limb, gesture, action, etc. associated with a human body, or a marker, a marker associated with an object, or a sand table, display area, or display item associated with a venue or location, etc. Vision related algorithms may involve vision localization, SLAM, three-dimensional reconstruction, image registration, background segmentation, key point extraction and tracking of objects, pose or depth detection of objects, and so forth. The specific application not only can relate to interactive scenes such as navigation, explanation, reconstruction, virtual effect superposition display and the like related to real scenes or articles, but also can relate to interactive scenes such as makeup beautification, limb beautification, special effect display, virtual model display and the like related to people.
The detection or identification processing of the relevant characteristics, states and attributes of the target object can be realized through a convolutional neural network. The convolutional neural network is a network model obtained by performing model training based on a deep learning framework.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described system and apparatus may refer to corresponding procedures in the foregoing method embodiments, which are not described herein again. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a non-volatile computer readable storage medium executable by a processor. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or a part of the technical solution, or in the form of a software product stored in a storage medium, including several instructions for causing an electronic device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiments of the present disclosure. The storage medium includes a U disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a magnetic disk, an optical disk, or other various media capable of storing program codes.
It should be noted that the foregoing embodiments are merely specific implementations of the disclosure, and are not intended to limit the scope of the disclosure, and although the disclosure has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any modification, variation or substitution of some of the technical features described in the foregoing embodiments may be made or equivalents may be substituted for those within the scope of the disclosure without departing from the spirit and scope of the technical aspects of the embodiments of the disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.