Summary of the invention
The object of this invention is to provide a kind of augmented reality method based on Quick Response Code, system and mobile terminal, when there is no sample image, the Quick Response Code front elevation consistent with the Quick Response Code code system in scene image is generated by Quick Response Code and recompile in decoding scene image, and corresponding Corner Detection is carried out in the positioning module region of image in 2 D code in the scene image of to catch positioning module region and the photographing module of Quick Response Code front elevation, calculate homography matrix, realize the certain deviation position in real scene Quick Response Code position or Quick Response Code, present the video that Quick Response Code is relevant, image, text, the multimedia messages of 3D model.
In order to realize foregoing invention object, the invention provides a kind of augmented reality method based on Quick Response Code, comprising:
Photographing module catches the real scene image containing Quick Response Code;
Quick Response Code in scanning scene image, and Quick Response Code is decoded, obtain the coded message of Quick Response Code, described coded message comprises: code system and resource information;
According to the coded message of Quick Response Code, encode by code system identical with it, generate the Quick Response Code front elevation consistent with the Quick Response Code code system in scene image; Resolve described resource information, to obtain virtual information corresponding to Quick Response Code simultaneously;
Corner Detection is carried out in the positioning module region of the image in 2 D code in the scene image of catching positioning module region and the photographing module of Quick Response Code front elevation respectively, and by same sequence, the angle point that the two extracts is sorted respectively, make the two upper angle point one_to_one corresponding extracted;
According to the coordinate figure of the two upper corresponding angle point extracted described, calculate homography matrix;
According to homography matrix, the certain deviation position of Quick Response Code position or Quick Response Code in real scene, plays up and virtual information corresponding with Quick Response Code described in output display.
Preferably, when resolve described resource information obtain be content of text time, content of text is played up as texture; Be resource URI when what resolve that described resource information obtains, then access this URI and obtain virtual information, and load according to virtual information type predetermined manner.
Preferably, described virtual information type comprises: one or more in video, image, text, 3D model.
Preferably, the Quick Response Code in described real scene image is the one in conventional QR Quick Response Code, conventional DataMatrix Quick Response Code, customization QR Quick Response Code, customization DataMatrix Quick Response Code; Wherein, customization QR Quick Response Code or the resource information customized in DataMatrix Quick Response Code comprise one or more in resource identifier, resource type, resource loading interface dimensions, rendering position drift rate.
Wherein, the positioning module region of the image in 2 D code in the scene image of catch positioning module region and the photographing module of Quick Response Code front elevation also comprises: according to the positioning module region of the image in 2 D code in the scene image that positioning module region and the photographing module of code system determination Quick Response Code front elevation are caught before carrying out Corner Detection.
Accordingly, present invention also offers a kind of augmented reality system based on Quick Response Code, comprising:
Photographing module, for catching the real scene image containing Quick Response Code;
Quick Response Code decoder module, for scanning the Quick Response Code in scene image, and decodes to Quick Response Code, and obtain the coded message of Quick Response Code, described coded message comprises: code system, version and resource information;
Quick Response Code coding module, for the coded message parsed according to described Quick Response Code decoder module, encodes by code system identical with it, generates the Quick Response Code front elevation consistent with the Quick Response Code code system in scene image;
Source obtaining module, for resolving the resource information in described coded message, to obtain virtual information corresponding to Quick Response Code;
Angle point grid module, Corner Detection is carried out in positioning module region for the image in 2 D code in the scene image of catching positioning module region and the photographing module of Quick Response Code front elevation, and by same sequence, the angle point that the two extracts is sorted respectively, make the two upper angle point one_to_one corresponding extracted;
Homography matrix computing module, the coordinate figure of the corresponding angle point of the said two devices for extracting according to angle point grid module, calculates homography matrix;
Play up display module, for according to homography matrix, the certain deviation position of Quick Response Code position or Quick Response Code in real scene, plays up and virtual information corresponding with Quick Response Code described in output display.
Preferably, described system also comprises: locating module, for the positioning module region of the image in 2 D code in the scene image that the positioning module region of code system determination Quick Response Code front elevation that parses according to Quick Response Code decoder module and photographing module are caught.
Preferably, the Quick Response Code in described real scene image is the one in conventional QR Quick Response Code, conventional DataMatrix Quick Response Code, customization QR Quick Response Code, customization DataMatrix Quick Response Code; Wherein, customization QR Quick Response Code or the resource information customized in DataMatrix Quick Response Code comprise one or more in resource identifier, resource type, resource loading interface dimensions, rendering position drift rate.
Preferably, when described source obtaining module resolve that described resource information obtains be content of text time, content of text is played up as texture; And be resource URI when what resolve that described resource information obtains, then access this URI and obtain virtual information, and load according to virtual information type predetermined manner, described virtual information type comprises: one or more in video, image, text, 3D model.
Accordingly, present invention also offers a kind of mobile terminal, described mobile terminal comprises the above-mentioned augmented reality system based on Quick Response Code.
Compared with prior art, the present invention has following beneficial effect:
1, the present invention is directly by the Quick Response Code decoding in scene image and re-encoding, regenerate the Quick Response Code front elevation consistent with Quick Response Code in scene image, the Quick Response Code positioning module region generated due to same code system is identical, Corner Detection is carried out in the positioning module region of the image in 2 D code in the scene image of therefore catching positioning module region and the photographing module of Quick Response Code front elevation respectively, homography matrix can be calculated, do not need to store Quick Response Code sample image in database, all applicable to random two-dimensional code, break in traditional augmented reality application, the limitation that corresponding sample image just can carry out tracking and matching must be prestored in a database.
2, the present invention is when calculating homography matrix, carry out Corner Detection by means of only to positioning module region, fault-tolerant ability is strong, such as, during Quick Response Code re-encoding generation data encoding mistake, do not affect the calculating of homography matrix, have the situation of small icon can be suitable in the middle of Quick Response Code in addition yet.
3, the present invention is not owing to needing sample image, therefore avoid from the inquiry of remote server with mate step, the system response delay because network transmission problems causes can be reduced, save the network traffic of user.
4, the degree of depth of the present invention excavates Quick Response Code as the application potential of information entry, makes the relevant information of Quick Response Code, resource present to user with more lively form.
Embodiment
Below in conjunction with the accompanying drawing in the embodiment of the present invention, be clearly and completely described the technical scheme in the embodiment of the present invention, obviously, described embodiment is only the present invention's part embodiment, instead of whole embodiments.Based on the embodiment in the present invention, those of ordinary skill in the art, not making the every other embodiment obtained under creative work prerequisite, belong to the scope of protection of the invention.
Those skilled in the art should know: in augmented reality process, are superimposed upon accurately on target object by virtual information, need to calculate the attitude of video camera, determine that the coordinate from target object is tied to the homography matrix of image coordinate system.
Present inventor finds, existing augmented reality is all generally carry out registration by real scene image and sample image, obtains homography matrix.This mode requires the image to carrying out augmented reality, its sample image for registration must be stored at mobile terminal or server end, if mobile terminal or augmented reality server end do not store the sample image of certain real scene or the characteristic point information of this sample image, then cannot realize the fusion of virtual information and this real scene, the popularization of augmented reality limit by sample image.In addition sample image be stored in mobile terminal can occupied terminal storage space, and can not the storage of satisfying magnanimity marker; And if sample image is stored in remote server, the retrieval of sample image with download and delay system can respond and waste the network traffic of user.
Present inventor finds, Quick Response Code has ease for use and popularization, and Quick Response Code is as information entry, can be generated by any information, it can carry the URI of any one section of text or any cost, and the most important is: can generate the Quick Response Code front elevation consistent with Quick Response Code code system before according to identical code system recompile again after Quick Response Code decoding.Feature point tracking can be carried out with the Quick Response Code front elevation generated after decoding re-encoding as sample image and the Quick Response Code in scene to mate, calculate homography matrix.
But in actual life, we can find Quick Response Code as shown in Figure 5, a small icon is had to block the central area of Quick Response Code in the middle of it, although Quick Response Code has error correcting capability, can be correctly decoded when having small icon blocks certain area, and then coding generates the Quick Response Code front elevation comprising data consistent with former Quick Response Code, but can not exist in this front elevation and block small icon in former Quick Response Code, if now carry out full figure feature point tracking with the Quick Response Code front elevation generated with the Quick Response Code in scene to mate, calculate homography matrix, error may be produced, make virtual information can not on the Quick Response Code be added in scene image of entirely accurate.
Therefore, technical solution of the present invention is not carried out feature point tracking to the Quick Response Code front elevation generated with the Quick Response Code in scene and is mated, and Corner Detection is carried out to calculate homography matrix in the positioning module region that have chosen the image in 2 D code in the scene image of catching at positioning module region and the photographing module of Quick Response Code front elevation.The positioning module of described Quick Response Code, figure is resembled also referred to as seeking, such as, in QR Quick Response Code, seek and resemble figure and comprise three identical position sensing figures, lay respectively at the upper left corner of symbol, the upper right corner and the lower left corner, as shown in Fig. 3 QR Quick Response Code, each position sensing figure can be regarded as and is made up of 3 overlapping concentric squares, and they are respectively 7 × 7 dark modules, 5 × 5 shallow modules and 3 × 3 dark modules.
Introduce the present invention in detail below in conjunction with accompanying drawing and introduce scheme.
See Fig. 1, Fig. 2, for the embodiment of the present invention is based on the augmented reality method flow schematic diagram of Quick Response Code, comprise following S101 ~ S106 step:
S101: photographing module catches the real scene image containing Quick Response Code, described Quick Response Code can be conventional Quick Response Code, also can be customization of two-dimension code, conventional Quick Response Code refers to the network Quick Response Code comprising a text field or URI link in resource information, customization of two-dimension code refers to that its resource information comprises resource identifier, resource type, resource load in some other configuration information such as interface dimensions, rendering position drift rate one or more;
S102: the Quick Response Code in scanning scene image, and Quick Response Code is decoded, obtain the coded message of Quick Response Code, described coded message comprises: code system and resource information, and described resource information refers to the relevant information that scanning Quick Response Code obtains;
S103: according to the coded message of Quick Response Code, encodes with code system identical with it and version, generates the Quick Response Code front elevation consistent with the Quick Response Code code system in scene image; Resolve described resource information, to obtain virtual information corresponding to Quick Response Code, the resource information that Quick Response Code is corresponding may be text message, may be also resource URI simultaneously, and when for resource URI, access UIR address, obtains the virtual information content that URI is corresponding;
S104: Corner Detection is carried out in the positioning module region of the image in 2 D code in the scene image of catching positioning module region and the photographing module of Quick Response Code front elevation respectively, and by same sequence, the angle point that the two extracts is sorted respectively, make the two upper angle point one_to_one corresponding extracted, at least extract 4 groups of corresponding angle points when calculating homography matrix, get a little more, result of calculation is more accurate;
Wherein, the determination in the positioning module region of Quick Response Code is determined according to code system, positioning module is all identical for the Quick Response Code of code system different editions different pieces of information of the same race, using the positioning module of image in 2 D code as region of interest ROI, carry out FAST, Harris, Shi-Thomas isocenter and detect.Because positioning module position in image in 2 D code of Quick Response Code is fixed, the positioning module of Quick Response Code can be utilized to calculate the principal direction of image in 2 D code, the angle point race that ROI region detects is sorted, to arrange all angle points according to clockwise or sequence counter-clockwise from principal direction.Ensure when calculating homography matrix, image in 2 D code and each angle point one_to_one corresponding of its front elevation.The module area schematic of QR Quick Response Code, DataMatrix Quick Response Code is respectively see Fig. 3, Fig. 4, Fig. 4 shows how by the principal direction of module region decision image in 2 D code, the angle point pointwise that ROI region detects is sorted, to arrange all angle points according to clockwise or sequence counter-clockwise from principal direction, to ensure when calculating homography matrix, image in 2 D code and each angle point one_to_one corresponding of its front elevation.
S105: according to the coordinate figure of the two upper corresponding angle point extracted described, calculate homography matrix;
Quick Response Code in real scene is planar object, that define a world coordinate system, the Quick Response Code front elevation generated through decoding re-encoding belongs to image coordinate system, 4 angle points are extracted in the positioning module region of the image in 2 D code in the scene image of catching for positioning module region and the photographing module respectively at Quick Response Code front elevation respectively, the coordinate that the coordinate of these four angle points on world coordinate system and image coordinate are fastened can be set up following corresponding relation:
Wherein, (u, v) represents the image coordinate of angle point, (xm, ym, zm) represent the coordinate of angle point on world coordinate system, be known parameters, H represents homography matrix to be calculated.
S106: according to homography matrix, the certain deviation position of Quick Response Code position or Quick Response Code in real scene, plays up and virtual information corresponding with Quick Response Code described in output display.
In this step, if Quick Response Code is conventional Quick Response Code, be directly added to virtual information Quick Response Code position; And if Quick Response Code is customization of two-dimension code, sets resource in its resource information and load interface dimensions, rendering position drift rate etc., be then presented at folded for virtual information apart from setting deviation post place, target location according to the size set and drift rate.
In this step, when resolve described resource information obtain be content of text time, content of text is played up as texture; Resource URI when what resolve that described resource information obtains, then access this URI and obtain virtual information, and load according to virtual information type predetermined manner, described virtual information type comprises: one or more in video, image, text, 3D model, such as: when virtual information is text message, content of text is directly played up as texture; When virtual information is 3D model, need first to resolve 3D model; When virtual information is video information, needs first to decode to video, using each for video two field picture as texture, be mapped on described 3D model by sequence frame by frame, carry out video loading and play up.
Embodiment of the present invention method, because positioning module is all identical for the Quick Response Code of code system different editions different pieces of information of the same race, therefore the embodiment of the present invention only carries out Corner Detection to positioning module region, fault-tolerant ability is strong, such as during Quick Response Code re-encoding generation data encoding mistake, do not affect the calculating of homography matrix, have the situation of small icon also can be suitable in the middle of Quick Response Code in addition, there is the Quick Response Code of small icon this centre of such as Fig. 5.
The invention allows for a kind of augmented reality system based on Quick Response Code, comprising: photographing module, Quick Response Code decoder module, Quick Response Code coding module, source obtaining module, angle point grid module, homography matrix computing module and play up display module, wherein:
Described photographing module, for catching the real scene image containing Quick Response Code;
Described Quick Response Code decoder module, for scanning the Quick Response Code in scene image, and decodes to Quick Response Code, and obtain the coded message of Quick Response Code, described coded message comprises: code system, version and resource information;
Described Quick Response Code coding module, for the coded message parsed according to described Quick Response Code decoder module, encodes with code system identical with it and version, generates the Quick Response Code front elevation consistent with the Quick Response Code code system in scene image;
Described source obtaining module, for resolving the resource information in described coded message, to obtain virtual information corresponding to Quick Response Code;
Described angle point grid module, Corner Detection is carried out in positioning module region for the image in 2 D code in the scene image of catching positioning module region and the photographing module of Quick Response Code front elevation, and by same sequence, the angle point that the two extracts is sorted respectively, make the two upper angle point one_to_one corresponding extracted;
Described homography matrix computing module, the coordinate figure of the corresponding angle point of the said two devices for extracting according to angle point grid module, calculates homography matrix;
Describedly play up display module, for according to homography matrix, the certain deviation position of Quick Response Code position or Quick Response Code in real scene, play up and virtual information corresponding with Quick Response Code described in output display, when described source obtaining module resolve that described resource information obtains be content of text time, content of text is played up as texture; And be resource URI when what resolve that described resource information obtains, then access this URI and obtain virtual information, and load according to virtual information type predetermined manner, described virtual information type comprises: one or more in video, image, text, 3D model.
Wherein, described system also comprises locating module, for the positioning module region of the image in 2 D code in the scene image that the positioning module region of code system determination Quick Response Code front elevation that parses according to Quick Response Code decoder module and photographing module are caught, so that described angle point grid module carries out angle point grid accurately.
Because the present embodiment is based on the augmented reality system of Quick Response Code, be and execute the routine system corresponding based on the augmented reality method of Quick Response Code above, therefore do not do too detailed repeating to it, detail feature is see describing above.
Method and system in the embodiment of the present invention, to QR Quick Response Code, that DataMatrix Quick Response Code carries out augmented reality effect is better, Quick Response Code can be conventional QR Quick Response Code, conventional DataMatrix Quick Response Code in addition, also can be customization QR Quick Response Code, customization DataMatrix Quick Response Code.
So-called conventional QR Quick Response Code, conventional DataMatrix Quick Response Code refer to that Quick Response Code derives from internet, and the resource information comprised in it is generally a character string, such as, be the URI of one section of text or Quick Response Code related resource.
So-called customization QR Quick Response Code or customization DataMatrix refer to, according to the resource identification that certain consolidation form is formed, the resource information namely in Quick Response Code comprises resource identifier, resource type, resource load in some other configuration information such as interface dimensions, rendering position drift rate one or more.Such as, ID:xxx_URI:xxx_TYPE:xxx_WIDTH:xxx_HEIGHT:xxx_OFFSET:xxx, (this address may at remote server to determine resource address by ID and URI, also may be local in client), can be pre-set the load mode (load mode of picture, text, audio frequency, video and 3D model is different) of related resource by resource type, other configuration informations include resource and load the skew etc. relative to Quick Response Code of the size at interface and rendering position.
The invention allows for a kind of mobile terminal, described mobile terminal comprises the above-mentioned augmented reality system based on Quick Response Code.
The present invention is using Quick Response Code as the identification in augmented reality system and positioning mark, Quick Response Code front elevation is directly generated by the decoding of Quick Response Code and re-encoding, from the corresponding angle point of the positioning module extracted region of the two, for the calculating of homography matrix, both broken in traditional augmented reality application, the limitation that corresponding sample image just can carry out tracking and matching must be prestored in a database, turn avoid use legacy identification thing time to remote server template query with mate step, the system response delay because network transmission problems causes can be reduced, save the network traffic of user.
All features disclosed in this instructions, or the step in disclosed all methods or process, except mutually exclusive feature and/or step, all can combine by any way.
Arbitrary feature disclosed in this instructions (comprising any accessory claim, summary and accompanying drawing), unless specifically stated otherwise, all can be replaced by other equivalences or the alternative features with similar object.That is, unless specifically stated otherwise, each feature is an example in a series of equivalence or similar characteristics.
The present invention is not limited to aforesaid embodiment.The present invention expands to any new feature of disclosing in this manual or any combination newly, and the step of the arbitrary new method disclosed or process or any combination newly.