Background technology
Augmented reality system has vast potential for future development, is subjected to people's attention day by day, and it has a wide range of applications in medical research, dissection training, exact instrument manufacturing, assembling and fields such as maintenance, military training, engineering design and tele-robotic.Owing to will realize virtual and real object perfect adaptation, dummy object must be merged in the real world position accurately, must detect the position of observer in scene, the angle of observation in real time in the augmented reality system, or even travel direction, so that be used for which kind of dummy object of help system decision demonstration, and rebuild coordinate system according to observer's visual field, promptly three-dimensional registration technology is the basis of augmented reality technology.Three-dimensional registration technology essence is exactly to calculate the mapping relations of the coordinate system of real world and virtual world, represents with matrix.And different matrixes has also determined the difference that three-dimensional registration technology method is chosen, and wherein matrix table is shown with homography matrix, fundamental matrix and essential matrix.
The homography matrix method is the simplest three-dimensional register method, because this method only need utilize 4 pairs of known points just can calculate the corresponding relation of the coordinate system of real world and virtual world, but the point that this method necessarily requires to choose is on the same plane and is the point of non-colinear, traditional like this homography matrix computing method can only be at the angular field of view of video camera in same plane or seek the situation of the special point on the same plane, for example use the angle point that detects on the gridiron pattern (referring to Z.Zhang, " A Flexible New Technique for Camera Calibration; " IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.22, no.11,2000, pp.1330-1334).In addition, based on the calculating of homography matrix, for multilevel three-dimensional registration, main now employing identification shape and texture carry out multilevel identification and registration.Method for the identification shape, often when shape is by partial occlusion, understand registration failure (referring to http://www.hitl.washington.edu/artoolkit/), method for the identification texture, more complicated needs study (referring to Jacky Baltes.Camera Calibration Using Rectangular Textures.Springer Berlin Publish.2001) in advance.
Summary of the invention
The objective of the invention is to overcome the deficiencies in the prior art, provide a kind of many planes dynamic augmented reality process registration based on homography matrix, utilizing recognition feature point auto-initiation on the basis of colouring information, avoided general register method initialization to need manual shortcoming, utilize shape information to carry out reorientation simultaneously and realize multilevel augmented reality drafting, and utilize tracking technique real-time update homography matrix result of calculation, reach dynamic augmented reality effect.
The technical solution used in the present invention: the dynamic augmented reality process registration in a kind of many planes based on homography matrix, its characteristics are that step is as follows: at first carry out the detection and the identification of scene characteristic, the unique point of utilizing colouring information to identify is carried out the initialization of homography matrix and is calculated; According to the real-time calculating homography matrix of the tracking of unique point, utilize the result who calculates to judge the mapping relations of real world and virtual world, thereby carry out the drafting of dummy model then; Identification according to shape information at last comes trigger re-positioning.
The detection of described scene characteristic and the method for identification are as follows:
(1) color space conversion: each two field picture of storage is become hsv color space form by RGB color space format conversion, solve the variation of the color codomain that RGB color image storage format causes the susceptibility of illumination;
(2) feature point detection: the method for utilizing the Harris Corner Detection detects angle point feature in the image according to the first order difference of gray scale and filtering;
(3) unique point identification: for the feature angle point that detects, calculate color value in its neighborhood, and utilize sort algorithm to find out wherein specifically defined 4 maximum points of color value in the neighborhood, utilize this 4 points to calculate initial homography matrixes.
Described real-time calculating homography matrix method is as follows: utilize the conplane unique point of the real-time tracking of KLT tracking, and according to the result of calculation and the formula of initial homography matrix
Wherein
Be the i frame homography matrix result who calculates according to the 0th frame result, last real-time renewal homography matrix result, thereby the real-time coordinate Mapping relation that obtains real world and virtual world.
Described identification according to the shape information on the special sign plate comes the method for trigger re-positioning as follows: every two field picture all can detect the shape information that the present invention has defined, switch when detecting special shape information, the capital triggering system is carried out reorientation, promptly reinitialize, re-register, realize the coordinate Mapping relation of real world and virtual world.
The conplane unique point method of described real-time follow-up is as follows: in order to reduce calculated amount, adopt the method compressed picture of pyramidal compression, dwindle the hunting zone; At the unique point after following the tracks of owing to produce the mistake that unique point on error and the on-plane surface all can cause homography matrix thereafter to calculate in the motion, so adopt the RANSAC method to reject frontier point and Characteristics of Fault point.
The present invention's beneficial effect compared with prior art is:
(1) auto-initiation calculating is finished in detection and the identification that utilizes colouring information to carry out unique point, avoids manual initialization; Color space selects the HSV space to alleviate the susceptibility of color for illumination; At the influence of change of scale, adopt neighborhood to calculate and sort algorithm eliminating noise, eliminate the influence of change of scale at last.
(2) for dynamic augmented reality system, KLT track algorithm and RANSAC algorithm are used to real-time tracking and choose the real-time update that conplane unique point is carried out homography matrix, realize that with the mapping relations of the coordinate system that obtains real world and virtual world correct augmented reality draws effect.
(3) can only register on single plane for homography matrix, simple colouring information is used to differentiate different planes.And the color codomain may be because the influence of illumination change and cause recognition failures, and employing shape information (character shape among simple shape or the ARToo1kit) is further differentiated positional information, comes trigger re-positioning, carries out multilevel registration effect.
Embodiment
Below in conjunction with accompanying drawing and embodiment the present invention is described in further detail:
The invention process process comprises three key steps: auto-initiation, real-time update homography matrix, reorientation.As shown in Figure 1.
Step 1 is an auto-initiation, mainly is divided into three phases:
The present invention is in order to overcome the problem of traditional three-dimensional register initialization, the mapping relations of the coordinate system that obtains initialized real world and virtual world have been, must in real world, find some features physical coordinates and with these selected corresponding image coordinate of feature, set up the initialized relation of hinting obliquely at of the coordinate system of real world and virtual world.General traditional way is the physical coordinates of some features in the self-defined real world, manual then on initialized two field picture, choose with real world in the corresponding point of feature chosen, obtaining its image coordinate, opening relationships, the initialization matrix that obtains like this is more accurate; The method that also has is that the image coordinate of utilizing shape or texture information to obtain some special features is carried out initialization, and initialization does not at this time need manually, but its shortcoming is also arranged, pre-service more complicated, and the feature out of true on the special texture.The present invention has taked to carry out auto-initiation with the information of color, and utilizes probability and sort algorithm to solve the problem of change of scale, and concrete step is as follows:
First stage: memory image
The present invention has used industrial camera to improve frame per second and resolution, after this industrial camera needs hardware initialization, with itself data structure storage is the IplImage data structure of opencv, and the color channel number is 3, and every two field picture is stored as the presentation format R (k) of RGB color space.
Second stage: converted image form
Color space is responsive for light application ratio, particularly traditional RGB color space.RGB image R (k) with storage is converted to hsv color spatial image V (k) thus.
Three phases: feature point detection identification
The image of the hsv color space representation after obtaining changing carries out the Harris Corner Detection to image.Single order differential transform according to gray scale detects the angle point in the image.Because angle point is the point that two dimensional image brightness changes curvature maximum value on violent point or the image border curve, promptly is the point of crossing in strong variations zone, so codomain is inaccurate under the color.The present invention takes to detect the color of the neighborhood point around the angle point, determines that a threshold value discerns needed initialized known point.
Determine that neighborhood is the core of looking for known point.Detected point under different yardsticks is different but dimensional variation has caused identical neighborhood, and then has caused a lot of noises occurring.As shown in Figure 2, be red color in the abcd square frame, getting neighborhood is 20 pixels, the neighborhood scope is approximatelycircle 1 when distance is for 1.2m; Under same vicinity, the neighborhood scope is approximatelycircle 2 when distance is 1m; This has just caused occurring removing a, b, and c outside the d point, also has a lot of noises.
Use method for normalizing to solve the problem of dimensional variation.Method for normalizing is exactly to determine total amount and component.Total amount is exactly the maximal value of sum that detects the point of designated color in all angle points in the neighborhood, and component is exactly the sum that detects the point of designated color in the angle point neighborhood.The maximal value of taking out the sum detect with sort algorithm is as total amount value, obtains value after the normalization with component value divided by total amount value then.What the present invention used is the mapping relations that homography matrix removes to calculate the coordinate system between real world and the virtual world, only needs 4 pairs of match points to come update calculation homography matrix result.4 points that identify are exactly preceding 4 points maximum after the normalization.As Fig. 2, shown in Figure 3, be respectively multi-form identification plate with designated color, can find out 4 points accurately.
Step 2: real-time update homography matrix.
Calculating homography matrix, in fact is exactly the process of asking for camera interior and exterior parameter.The video camera confidential reference items are about physical parameters such as focal length of camera, are to demarcate before the three-dimensional registration; The video camera external parameter mainly is rotation and translation variable, has mainly represented the mapping relations of real world coordinates system and virtual world coordinate system.In the process that video camera moves, external parameter is a real-time change, if obtain correct three-dimensional registration relation, obtains real actual situation and merges the drafting effect, and homography matrix just needs real-time renewal.
Utilize the 4 pairs of points that identify in thestep 1, can calculate initial homography matrix, obtain the external parameter of video camera under the stationary state.Along with moving of video camera, known unique point may shift out outside the visual angle, and new unique point occurs.The present invention adopts the KLT tracking, follows the tracks of the unique point of initial frame, the result of renewal homography matrix that just can be real-time.But traditional KLT tracking has its defective, 1) be too fast when moving, tracking will be failed; 2) be the unique point that KLT just follows the tracks of previous frame, and the homography matrix method requires to use the point on the same plane could guarantee to calculatecorrect.At problem 1, the present invention has carried out error handling processing, when counting of tracing into is less than 20, just detects angle point again, serves as the point of following the tracks of former frame with present point.Because the unique point of tracking origin is exactly an angle point, though mobile camera moving comparatively fast causes following the tracks of failure, change can not differ greatly between two frames, still have common angle point, be rational so detect the angle point tracking again.Atproblem 2, use the incorrect point that the RANSAC algorithm is rejected the point on some on-plane surfaces and the method in 1 of dealing with problems obtains.
Step 3: reorientation is respectively two kinds of methods.
Though the logarithm of the match point that homography matrix needs is minimum, but to satisfy strict constraint, can only use the feature on the same plane exactly.When many planes actual situation merges drafting, using homography matrix to calculate will lead to the failure.At this problem, the present invention has used method for relocating, (special shape or the color that define is arranged) on this plane when another plane is shifted at the visual angle, will carry out reorientation.
First method: only use color.
Suppose to draw dummy object respectively on two different planes, the present invention can use two kinds of different colors, and is for example red and green.When the change color that occurs in the visual angle, will enter the reorientation module, the initialization of carrying out in thestep 1 is demarcated, and starts different drafting schemes, carries out the drafting of actual situation fusion.
Second method: add the identification of shape.
Because the visual angle turns to different planes, may cause when camera angle changes greatly under the varying environment, lamplight brightness changes also under the big situation, for the color codomain, may cause bigger variation, what cause detecting is inaccurate, so the present invention has added the label detection module of ARToolkit on the basis of color attribute information.When different signs occurring, will the trigger re-positioning module, carry out the initialization homography matrix according to the 4 pairs of special match points that identify again and calculate, starts different drafting schemes, carry out the drafting of actual situation fusion.Fig. 4, the drafting effect that Fig. 5 merges for actual situation, Fig. 5 background are the signs that increases shape, are red in the middle big square frame.