Summary of the invention
At least one to solve the above-mentioned problems, first aspect present invention is provided in a kind of vision based on depth filterJourney meter map updating method, comprising:
Judge whether to need to add according to the depth information characteristic value of the fisrt feature of the current frame image of acquisition point crucialFrame image,
If not needing to add, institute is updated according to the fisrt feature point of the current frame image using existing depth filterState the seed of existing depth filter；
If desired it adds, then establishes and initialize new depth filter, it is special that second is extracted from the current frame imageSeed of the sign point as the new depth filter, determines key frame images and using the new depth filter according to instituteState the seed that key frame images update the new depth filter；
The probability distribution for calculating and updating the seed, when the seed estimation of Depth convergence when be added the map withUpdate the map.
Further, the method also includes
It obtains in real time and stores frame image；
Multiple fisrt feature points of current frame image are extracted, the depth information of each fisrt feature point is calculated and determine all theThe depth information characteristic value of one characteristic point depth information.
Further, the depth information characteristic value of the fisrt feature point of the current frame image according to acquisition judges whetherNeeding to add key frame images further comprises:
The depth information characteristic value is compared with the eigenvalue threshold pre-seted, if the depth information characteristic valueGreater than eigenvalue threshold, then need to add key frame images.
Further, described if desired to add, then new depth filter is established and initializes, from the current frame imageThe middle seed for extracting second feature point as the new depth filter determines key frame images and uses the new depthFilter further comprises according to the seed that the key frame images update the new depth filter:
New depth filter is established and initialized, second feature point is extracted from the current frame image as described newDepth filter seed；
The frame image of preset quantity is selected as key frame images according to the time from the frame image of the storage；
The new depth is updated according to the key frame images of the preset quantity using the new depth filter to filterThe depth information of seed in wave device.
Further, the frame image of preset quantity is selected as pass according to the time in the frame image from the storageKey frame image further comprises:
Key point is selected from the second feature point of the current frame image, multiple frame images are chosen according to the time and is passed throughThe key point judges each frame image and the current frame image with the presence or absence of overlapping region respectively, and if it exists, then determinesThe frame image is key frame images, otherwise gives up the frame image.
Further, if described do not need to add, using existing depth filter according to the of the current frame imageThe seed that one characteristic point updates the existing depth filter further comprises:
The fisrt feature point of the current frame image is subjected to characteristic matching with point map corresponding in the map to obtainObtain the depth information of the point map；
The existing depth filter updates the existing depth filtering according to the fisrt feature point of the current frame imageCorresponding seed in device.
Further, the probability distribution calculated and update the seed, when the convergence of the estimation of Depth of the seedThe map, which is added, to update the map further comprises:
The probability distribution of the seed is updated using bayesian probability model；
The seed estimation of Depth convergence is then determined when the probability of the seed is greater than the probability threshold value pre-seted, then by itThe characteristic point of corresponding picture frame is converted to point map and is added to the map to update the map.
Second aspect of the present invention provides a kind of visual odometry map updating system based on depth filter, comprising:
Key frame extraction device, for judging whether system needs to add key frame images；
Key frame depth updating device is described new for establishing and initializing new depth filter and determine seedDepth filter updates the new depth filter according to the key frame images with overlapped view that the system storesSeed；
It is existing deep according to current frame image update to have depth filter for the system for depth filter updating deviceSpend the seed of filter；
Probability distribution updating device, for calculating and updating the probability distribution of the seed, when the depth of the seed is estimatedIt collects and the map is added when holding back to update the map.
Third aspect present invention provides a kind of computer readable storage medium, is stored thereon with computer program, the programMethod as described in relation to the first aspect is realized when being executed by processor.
Fourth aspect present invention provides a kind of computer equipment, including memory, processor and storage are on a memory simultaneouslyThe computer program that can be run on a processor, the processor realize side as described in relation to the first aspect when executing described programMethod.
Beneficial effects of the present invention are as follows:
The present invention formulates one for the problem that existing depth filter convergence time is slow, map rejuvenation is slow at presentVisual odometry map updating method and system of the kind based on depth filter, extract second feature while adding key frameIt puts and carries out depth update, to make full use of present frame and there is the overlapping frame of overlapped view with present frame, dramatically speed up depthIt spends the convergence rate of filter, improve the efficiency that depth updates, to compensate for the problem of existing in the prior art, effectively promote viewFeel the robustness of odometer map rejuvenation.
Specific embodiment
In order to illustrate more clearly of the present invention, the present invention is done further below with reference to preferred embodiments and drawingsIt is bright.Similar component is indicated in attached drawing with identical appended drawing reference.It will be appreciated by those skilled in the art that institute is specific belowThe content of description is illustrative and be not restrictive, and should not be limited the scope of the invention with this.
In the prior art, during building map in real time, depth update method is generally passed through using depth filterThe image obtained to camera is handled and is updated map, is specifically updated using existing dichotomy method, crucial in additionNew characteristic point is extracted when frame as seed, carries out the operation of depth update until next frame normal frames arrive, however this mistakeJourney also includes the processing such as pose estimation and the optimization of camera, and time loss is excessive for depth update, is easy to causeThe case where tracking failure, it often cannot achieve the real-time building of map.
Based on the above situation, An embodiment provides a kind of visual odometry based on depth filterFigure update method, comprising: needs are judged whether according to the depth information characteristic value of the fisrt feature of the current frame image of acquisition pointKey frame images are added, if not needing to add, using existing depth filter according to the fisrt feature of the current frame imagePoint updates the seed of the existing depth filter；If desired it adds, then establishes and initialize new depth filter, from describedSeed of the second feature point as the new depth filter is extracted in current frame image, determines key frame images and uses instituteState the seed that new depth filter updates the new depth filter according to the key frame images；It calculates and updates describedThe map is added when the convergence of the estimation of Depth of the seed to update the map in the probability distribution of seed.
In a specific example, as shown in Figure 1, acquisition is schemed in real time during actual motion using monocular camPicture simultaneously constructs map, and the monocular cam obtains according to the time interval pre-seted and store frame image, such as 1 second acquiresImage is handled and is stored sequentially in time by 30 frame images, the application to using monocular or more mesh cameras with no restrictions.It willTime immediate current frame image is described in acquisition image:
Firstly, extracting the fisrt feature point of the current frame image, and the depth information of each characteristic point is calculated separately, determinedThe depth information characteristic value of all fisrt feature point depth informations.Depth information is determined i.e. from the depth information of multiple characteristic pointsCharacteristic value, the characteristic value can be the statistical natures such as extreme value, depth information mean value and the depth information intermediate value of depth informationValue is used as judgment basis, and the present embodiment is using the depth information intermediate value of characteristic point as judgment basis.It is worth noting that, about featureThere are the gray differences of 16 pixels around various ways, such as one pixel of detection for the extraction of point to extract characteristic point, rightWith no restrictions, those skilled in the art can be according to the requirement extract characteristic point of true resolution and sensitivity by this application.
Secondly, judging whether to need to add key frame images according to the depth information characteristic value.It specifically includes: will be describedDepth information characteristic value is compared with the eigenvalue threshold pre-seted, if the depth information characteristic value is greater than characteristic value thresholdValue, then need to add key frame images.The depth information intermediate value is carried out with the intermediate value threshold value pre-seted in the present embodimentIt compares, thinks to need to add key frame images if the depth information intermediate value is greater than intermediate value threshold value.The intermediate value threshold value is rootAccording to the determining threshold value of the acquisition of actual frame image, processing and analysis, such as fisrt feature point and the map according to present frameIn the corresponding picture frame of newest point map with the fisrt feature point at a distance from corresponding characteristic point, when in the distanceValue is more than that pre-determined distance then thinks that camera has moved farther out, then needs to select tool from the frame image stored before present frameThere is the frame image of overlapping region to be added as key frame images, it can be according to described in the case where only considering distance featureThe intermediate value threshold value is determined apart from intermediate value；The frame of storage can also be for example calculated again on the basis of being compared using distance againThe ratio of coordinate of the image under camera coordinates system and the depth intermediate value, if the ratio meets the ratio range pre-setedThink not needing addition key frame images, key frame images is otherwise added, to further increase visual odometry map rejuvenationSensitivity.
If desired key frame images are added, then new depth filter are established and initialize, from the current frame imageSeed of the second feature point as the new depth filter is extracted, determine key frame images and is filtered using the new depthWave device updates the seed of the new depth filter according to the key frame images.As shown in Fig. 2, specifically including:
New depth filter is established and initialized, second feature point is extracted from the current frame image as described newDepth filter seed.A new depth filter is re-established, and the depth filter is initialized, except normalIt further include being arranged in the depth filter for calculating the parameter of probability distribution other than rule initialization；Then again to currentFrame image carry out the extraction of second feature point, the second feature point be different from fisrt feature point, using second feature point as newlyThe seed of depth filter, i.e., each second feature point correspond to a seed of new depth filter, and each seed storage is stillThe character pair point of depth information is not obtained for subsequent carry out estimation of Depth.It is worth noting that the application is to the ginsengSeveral settings calculate probability distribution as design criteria with no restrictions, can be realized.The design parameter ginseng being arranged in the present embodimentSee and " is proposed in Vogiatzis G, Hernandez C.Video-based, real-time multi-view stereo [J] "Bayesian probability model.
The frame image of preset quantity is selected as key frame images according to the time from the frame image of the storage.Due toPresent frame with point map newest in the map at a distance from corresponding picture frame farther out, i.e., camera moved farther out, becauseThis needs chooses key frame images from the frame image stored before the present frame, in the present embodiment from the frame image of storageThe time immediate frame image for choosing preset quantity is selected, i.e., choosing from the frame image of storage has weight with present frameThe frame image in folded region (i.e. overlapped view).It specifically includes and selects key point from the characteristic point of the current frame image, lead toIt crosses the key point and judges that the frame image and the current frame image with the presence or absence of overlapping region, then determine the frame figure if it existsAs being key frame images, otherwise give up the frame image.Wherein, the preset quantity can be a frame image, or moreA frame image, the application are without limitation.
As shown in figure 3,5 points are chosen in the present embodiment from the characteristic point that present frame extracts as key point, 5 passesKey o'clock is made of 1 central point and 4 boundary points, and central point is the point nearest with image central distance in all characteristic points, and 4Boundary point is respectively point nearest with four angular distance of image in all characteristic points.It can be from storage according to above-mentioned 5 key pointsThe frame image with current frame image there are overlapping region is picked out in frame image, and the frame image is determined as key frame images.Since the key frame images and current frame image are apart from close, then the seed maximum probability in new depth filter is selectedKey frame images observed, therefore with these key frame images more new seed, to make full use of current frame image and addThe convergence rate of the estimation of Depth of seed in fast new depth filter, to improve the robust of visual odometry map rejuvenationProperty.It is worth noting that the application to the selection of the key point with no restrictions, those skilled in the art should be according to practical needSeek selection key point.
The new depth is updated according to the key frame images of the preset quantity using the new depth filter to filterThe depth information of seed in wave device.The i.e. described new depth filter is directly updated described in acquisition stored seedThe depth information of seed does not need to wait a normal frames image arrival just seed according to the normal frames image update againDepth information, thus effectively reduce seed update consumed by the time.
If not needing addition key frame images, using existing depth filter according to the first spy of the current frame imageSign point updates the seed of the existing depth filter.When not needing addition key frame images, the current frame image is as generalLogical frame image, is updated the seed stored in existing depth filter.As shown in figure 4, specifically including:
First, the fisrt feature point of the present frame is subjected to characteristic matching with point map corresponding in the map to obtainObtain the depth information of the point map.Specifically include: the existing depth filter is special according to the first of the current frame imageSign point updates the depth information for being stored in seed in the existing depth filter, i.e., by point map corresponding to each seedOn projection to present frame, the point after projection is located at camera rear or skips these seeds if not projecting on present frame, becauseThe depth information of these seeds can not be updated and be had an impact for present frame.Calculate the corresponding picture frame of each seed and present frameBetween affine transformation matrix, that is, be stored in each seed in the depth filter come it is affine between source frame and present frameTransformation matrix.The determinant for calculating affine transformation matrix, being found in image pyramid by the value of the determinant has mostThe horizontal pyramidal layer of good matching.
Second, the existing depth filter updates the existing depth according to the fisrt feature point of the current frame imageCorresponding seed in filter.The i.e. described existing depth filter is believed using the depth that present frame assesses the stored seedBreath, the existing depth filter use bilinear interpolation, the affine transformation matrix and pyramidal layer by current frame imageImage block, which transforms to, obtains transformed image block to obtain polar curve in reference frame image, the interval sampling on grade line, and calculatesDifference and Jacobian matrix between the image block of present frame after the corresponding image block of sampled result and variation, thus accuratelyThe feature locations of prediction and matching obtain accurate estimation of Depth value using triangulation.Wherein, the reference frame image isThe previous normal frames image of the current frame image；Described image block is characterized a pixel for surrounding preset quantity, described pre-If quantity is configured according to practical application request.Then the depth for calculating each seed using trigonometric sine, the cosine law is not trueIt is qualitative, method used in the present embodiment referring specifically to " Pizzoli M, Forster C, Scaramuzza D.REMODE:Probablistic, Monocular dense reconstruction in real time [C] ", details are not described herein.
Finally, the probability distribution of the seed is calculated and updates, when the convergence of the estimation of Depth of the seed described in additionMap is to update the map.New depth filter and new at this is either established when needing to add key frame imagesIncrease seed in depth filter and utilize key frame images more new seed, or present frame is used to deposit as normal frames updateThe seed in existing depth filter is stored up, updates the probability point of each seed using bayesian probability model after seed updateCloth.It specifically includes: updating the probability distribution of each seed using bayesian probability model；When the probability of each seed is greater thanThe probability threshold value pre-seted then determines the seed estimation of Depth convergence, then is converted to ground for the characteristic point of its corresponding picture frameFigure point is added to the map to update the map.I.e. when the seed estimation of Depth is restrained by its corresponding world coordinatesThe update of completing map is added in map as point map.Bayesian probability model used in the present embodiment referring specifically to" Vogiatzis G, Hernandez C.Video-based, real-time multi-view stereo [J] ", specific tableIt is now that a series of histogram distribution of measurements of the depth of each seed using Gaussian Profile and is uniformly distributed and carrys out association listShow, the solution that the solution of the depth of seed can be converted to Gaussian Profile and Beta distribution by introducing latent variable, i.e., by repeatedlyFor formula after new observed quantity is added, the Posterior probability distribution of more new seed, when the seed probability in Posterior probability distribution is bigWhen the probability threshold value pre-seted, then it is assumed that the convergence of seed estimation of Depth, so that its corresponding point map is added in map, withRealize the update of cartographic information.
The present embodiment directly carries out the depth that assessment obtains seed to stored seed using the new depth filterDegree estimation does not need that a normal frames image is waited to arrive just according to normal frames image progress pose estimation again, effectively subtractsLack the time consumed by seed estimation of Depth, to accelerate the convergence rate of seed estimation of Depth, improves the effect that depth updatesRate effectively promotes the robustness of visual odometry map rejuvenation to compensate for problems of the prior art.
Corresponding with map updating method provided by the above embodiment, one embodiment of the application also provides one kind and is based onThe visual odometry map updating system of depth filter, due to map updating system provided by the embodiments of the present application with it is above-mentioned severalThe map updating method that kind of embodiment provides is corresponding, therefore is also applied for that the present embodiment provides maps more in aforementioned embodimentsNew system is not described in detail in the present embodiment.
As shown in figure 5, one embodiment of the application also provides a kind of visual odometry map based on depth filterMore new system, comprising: key frame extraction device, for judging whether system needs to add key frame images；Key frame depth is moreNew equipment, for establishing and initializing new depth filter and determine seed, the new depth filter is according to the systemThe key frame images with overlapped view of system storage update the seed of the new depth filter；Depth filter more new clothesIt sets, has depth filter for the system and update the seed for having depth filter according to current frame image；Probability distributionUpdating device, for calculating and updating the probability distribution of the seed, when the convergence of the estimation of Depth of the seed described in additionMap is to update the map.
In a specific example, key frame extraction device is according to the multiple characteristic points extracted from current frame imageDepth information characteristic value judges whether to need to add key frame images, that is, judges whether the camera has moved farther out, beNo needs select key frame images from the frame image stored before current frame image；If desired it adds and then uses key frame depthNew depth filter is established and initialized to updating device, using the characteristic point extracted in current frame image as new depth filteringThe seed of device, and the more new seed of the key frame images with overlapped view stored using system；If not needing addition key frameDepth filter updating device is then used, using the existing depth filter of the system according to the feature of the current frame imagePoint updates the seed of the existing depth filter storage；Finally, the seed that updates of either new depth filter is stillThe seed for having depth filter to update, the probability distribution of the seed is calculated using probability distribution updating device, when the seedProbability distribution be greater than probability threshold value when think seed estimation of Depth restrain, then by the seed be added map, to realize groundThe update of figure.
The system also includes feature point extraction devices and point map updating device, and wherein feature point extraction device, is used forExtract multiple characteristic points of current frame image；Point map updating device, when do not need addition key frame when, depth filter moreBefore the seed that new equipment is updated storage using existing depth filter according to current frame image, for estimating the map roughlyThe depth information of point map, i.e., when the described current frame image is normal frames image, characteristic point that will be extracted from the present frameCharacteristic matching is carried out with point map corresponding in the map to obtain the depth information of the point map.
The map updating system updates depth filter by selecting key frame images from stored frame imageSeed to complete the estimation of Depth of seed using a small amount of existing information does not need that a normal frames image is waited to arrive just againPose estimation is carried out according to the normal frames image, the time consumed by seed estimation of Depth is effectively reduced, to accelerate to plantThe convergence rate of sub- estimation of Depth improves the efficiency that depth updates, to compensate for problems of the prior art, have compared withStrong anti-interference ability, significantly improve system operation robustness, can be widely applied for VR, AR equipment pose calculate,Figure building and scene interactivity.
Another embodiment of the present invention provides a kind of computer readable storage mediums, are stored thereon with computer journeySequence, realization when which is executed by processor: according to the depth information characteristic value of the fisrt feature of the current frame image of acquisition pointJudge whether to need to add key frame images, if not needing to add, using existing depth filter according to the present frame figureThe fisrt feature point of picture updates the seed of the existing depth filter；If desired it adds, then establish and initializes new depthFilter extracts seed of the second feature point as the new depth filter from the current frame image, determines crucialFrame image and the seed for updating the new depth filter according to the key frame images using the new depth filter；The probability distribution for calculating and updating the seed, it is described to update when the convergence of the estimation of Depth of the seed map to be addedMap.
In practical applications, the computer readable storage medium can be using one or more computer-readable mediaAny combination.Computer-readable medium can be computer-readable signal media or computer readable storage medium.It calculatesMachine readable storage medium storing program for executing can for example be but not limited to system, device or the device of electricity, magnetic, optical, electromagnetic, infrared ray or semiconductorPart, or any above combination.The more specific example (non exhaustive list) of computer readable storage medium includes: to haveThe electrical connection of one or more conducting wires, portable computer diskette, hard disk, random access memory (RAM), read-only memory(ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, portable compact disc read-only memory (CD-ROM), light storage device, magnetic memory device or above-mentioned any appropriate combination.It is computer-readable to deposit in this in real time exampleStorage media can be any tangible medium for including or store program, which can be commanded execution system, device or devicePart use or in connection.
Computer-readable signal media may include in a base band or as carrier wave a part propagate data-signal,Wherein carry computer-readable program code.The data-signal of this propagation can take various forms, including but unlimitedIn electromagnetic signal, optical signal or above-mentioned any appropriate combination.Computer-readable signal media can also be that computer canAny computer-readable medium other than storage medium is read, which can send, propagates or transmit and be used forBy the use of instruction execution system, device or device or program in connection.
The program code for including on computer-readable medium can transmit with any suitable medium, including but not limited to withoutLine, electric wire, optical cable, RF etc. or above-mentioned any appropriate combination.
The computer for executing operation of the present invention can be write with one or more programming languages or combinations thereofProgram code, described program design language include object oriented program language-such as Java, Smalltalk, C++,It further include conventional procedural programming language-such as " C " language or similar programming language.Program code can be withIt fully executes, partly execute on the user computer on the user computer, being executed as an independent software package, portionDivide and partially executes or executed on a remote computer or server completely on the remote computer on the user computer.?Be related in the situation of remote computer, remote computer can pass through the network of any kind --- including local area network (LAN) orWide area network (WAN)-be connected to subscriber computer, or, it may be connected to outer computer (such as mentioned using Internet serviceIt is connected for quotient by internet).
As shown in fig. 6, another embodiment of the present invention provides a kind of computer equipment structural schematic diagram.Fig. 6 is aobviousThe computer equipment 12 shown is only an example, should not function to the embodiment of the present invention and use scope bring any limitSystem.
As shown in fig. 6, computer equipment 12 is showed in the form of universal computing device.The component of computer equipment 12 can be withIncluding but not limited to: one or more processor or processing unit 16, system storage 28 connect different system componentsThe bus 18 of (including system storage 28 and processing unit 16).
Bus 18 indicates one of a few class bus structures or a variety of, including memory bus or Memory Controller,Peripheral bus, graphics acceleration port, processor or the local bus using any bus structures in a variety of bus structures.It liftsFor example, these architectures include but is not limited to industry standard architecture (ISA) bus, microchannel architecture (MAC)Bus, enhanced isa bus, Video Electronics Standards Association (VESA) local bus and peripheral component interconnection (PCI) bus.
Computer equipment 12 typically comprises a variety of computer system readable media.These media can be it is any can be byThe usable medium that computer equipment 12 accesses, including volatile and non-volatile media, moveable and immovable medium.
System storage 28 may include the computer system readable media of form of volatile memory, such as arbitrary accessMemory (RAM) 30 and/or cache memory 32.Computer equipment 12 may further include it is other it is removable/can notMobile, volatile/non-volatile computer system storage medium.Only as an example, storage system 34 can be used for reading and writing notMovably, non-volatile magnetic media (Fig. 6 do not show, commonly referred to as " hard disk drive ").It, can be with although being not shown in Fig. 6The disc driver for reading and writing to removable non-volatile magnetic disk (such as " floppy disk ") is provided, and non-volatile to movingThe CD drive of CD (such as CD-ROM, DVD-ROM or other optical mediums) read-write.In these cases, each drivingDevice can be connected by one or more data media interfaces with bus 18.Memory 28 may include that at least one program producesProduct, the program product have one group of (for example, at least one) program module, these program modules are configured to perform of the invention eachThe function of embodiment.
Program/utility 40 with one group of (at least one) program module 42 can store in such as memory 28In, such program module 42 include but is not limited to operating system, one or more application program, other program modules andIt may include the realization of network environment in program data, each of these examples or certain combination.Program module 42 is usualExecute the function and/or method in embodiment described in the invention.
Computer equipment 12 can also be with one or more external equipments 14 (such as keyboard, sensing equipment, display 24Deng) communication, can also be enabled a user to one or more equipment interact with the computer equipment 12 communicate, and/or with makeThe computer equipment 12 any equipment (such as network interface card, the modulatedemodulate that can be communicated with one or more of the other calculating equipmentAdjust device etc.) communication.This communication can be carried out by input/output (I/O) interface 22.Also, computer equipment 12 may be used alsoTo pass through network adapter 20 and one or more network (such as local area network (LAN), wide area network (WAN) and/or public networkNetwork, such as internet) communication.As shown in fig. 6, network adapter 20 is logical by other modules of bus 18 and computer equipment 12Letter.It should be understood that other hardware and/or software module, packet can be used in conjunction with computer equipment 12 although being not shown in Fig. 6It includes but is not limited to: microcode, device driver, redundant processing unit, external disk drive array, RAID system, magnetic tape driveDevice and data backup storage system etc..
Processor unit 16 by the program that is stored in system storage 28 of operation, thereby executing various function application withAnd data processing, such as realize a kind of visual odometry map rejuvenation based on depth filter provided by the embodiment of the present inventionMethod.
The present invention formulates one for the problem that existing depth filter convergence time is slow, map rejuvenation is slow at presentVisual odometry map updating method and system of the kind based on depth filter, extract second feature while adding key frameIt puts and carries out depth update, to make full use of present frame and there is the overlapping frame of overlapped view with present frame, dramatically speed up depthIt spends the convergence rate of filter, improve the efficiency that depth updates, to compensate for the problem of existing in the prior art, effectively promote viewFeel the robustness of odometer map rejuvenation.
Obviously, the above embodiment of the present invention be only to clearly illustrate example of the present invention, and not be pairThe restriction of embodiments of the present invention may be used also on the basis of the above description for those of ordinary skill in the artTo make other variations or changes in different ways, all embodiments can not be exhaustive here, it is all to belong to this hairThe obvious changes or variations that bright technical solution is extended out are still in the scope of protection of the present invention.