The content of the invention
The technical problems to be solved by the invention are in view of the shortcomings of the prior art, there is provided one kind is based on mixed reality equipmentAssembling bootstrap technique and system.
The technical scheme that the present invention solves above-mentioned technical problem is as follows：
A kind of assembling bootstrap technique based on mixed reality equipment, including：
Build Virtual assemble scene；
Merged by mixed reality equipment with reality assembling scene；
Assembling stage in the reality assembling scene and assembly are identified；
Assembling guidance information is obtained according to the assembling stage identified and the assembly identified；
The assembling guidance information is shown by the mixed reality equipment, assembling process is guided.
The beneficial effects of the invention are as follows：A kind of assembling bootstrap technique based on mixed reality equipment provided by the invention, lead toCross mixing real world devices to be merged the Virtual assemble scene and reality scene of structure, and the assembling in scene is assembled to realityStage and assembly are identified, and realize and carry out assembling guiding for specific installation step, enter with real assembling sceneRow real-time matching, each stage of assembling can be flexibly identified in real time and provides corresponding assembling guiding, passes through mixingDisplay device reality assembling guidance information has the advantages of directly perceived, has feeling of immersion, improves the efficiency of assembling guiding, reduces assemblingDuring error rate.
On the basis of above-mentioned technical proposal, the present invention can also do following improvement.
Further, the assembling stage and assembly in the reality assembling scene is identified and specifically included：
Obtain the image of each assembly in the reality assembling scene；
Marked according to the identification being set in advance in all on the assembly, each assembly in described image is knownNot；
It is determined that the hierarchical relationship between each assembly identified；
According to each assembly identified and the hierarchical relationship, it is determined that current assembling stage.
It is using the above-mentioned further beneficial effect of scheme：By according to the identification mark pair being set in advance on assemblyAssembly is identified, and judges the hierarchical relationship between assembly, level according to the assembly identified and each otherRelation, current assembling stage can be identified exactly, improve efficiency and the degree of accuracy of identification.
Further, in addition to：
RGB image, depth image, exercise data, temperature data, the stress of each assembly identified are obtained respectivelyData and strain data；
According to default multisource data fusion algorithm, respectively in data level, feature level and decision level to each institute for identifyingRGB image, depth image, exercise data, temperature data, stress data and the strain data for stating assembly merge at decision-makingReason, the real time status information for each assembly being identified out.
It is using the above-mentioned further beneficial effect of scheme：By the way that multi-source data processing method is integrated into augmented reality fieldScape substantially increases the utilization rate of data and the practicality of augmented reality, handled especially by multi-source data on identifying and being interactiveMethod carries out fusion treatment, Neng Gouyou to the RGB image of assembly, depth image, exercise data, stress data and strain dataThe real-time status of assembly is identified to effect, is easy to accurately guide assembling process, and assembling bootup process is carried outAdjustment in real time and monitoring.
Further, in addition to：
Supervised according to the assembling process of each assembly of the real time status information to identifying with assembling resultControl, when assembling successfully, records the assembling result；When assembling occurs wrong, mistake is sent to the mixed reality equipmentPrompting message.
It is using the above-mentioned further beneficial effect of scheme：By the real time status information of assembly that identifies to assemblingProcess is monitored, and assembling process can be modified in real time, and the guiding for assembling process, can with more ageingThe operating mistake in assembling process is reduced, and assembling result is monitored by real time status information, can more accurately be unitedAssembling result is counted out, is easy to search and assembles unsuccessful assembly.
Further, the structure Virtual assemble scene specifically includes：
Build virtual scene；
The attribute information of assemble flow and whole assemblies is obtained, and it is more according to the assemble flow and attribute information generationIndividual assembling stage；
Hierarchical relationship and sequential relationship between each assembly is determined according to the attribute information of each assembly；
The assembling that each assembling stage is generated according to the hierarchical relationship between each assembly and sequential relationship guidesInformation；
By the attribute information of the assembly, the assembling stage, the assembling guidance information of each assembling stage and eachHierarchical relationship and sequential relationship between the assembly are added in the virtual scene, obtain Virtual assemble scene.
It is using the above-mentioned further beneficial effect of scheme：Built by the attribute information of assemble flow and whole assembliesVirtual assemble scene, so as to get assembling scene include the attribute information of assemble flow and assembly, be easy to identify different dressesWith the stage, there is provided comprehensively assemble guidance information.
Further, the assembling guidance information specifically includes：Operation information and guidance information, wherein：
The operation information includes word, image and the voice that prompting user is operated；
The guiding message includes the essential information of assembly.
The another technical solution that the present invention solves above-mentioned technical problem is as follows：
A kind of assembling guiding system based on mixed reality equipment, including：
Model building device, for building Virtual assemble scene；
Mixed reality equipment, for the Virtual assemble scene to be merged with reality assembling scene；
Identification device, for the assembling stage in the reality assembling scene and assembly to be identified；
The mixed reality equipment is additionally operable to be obtained according to the assembling stage identified and the assembly identifiedAssembling guidance information is taken, and shows the assembling guidance information, assembling process is guided.
A kind of assembling guiding system based on mixed reality equipment provided by the invention, will be built by mixed reality equipmentVirtual assemble scene and reality scene merged, and to reality assemble scene in assembling stage and assembly knowNot, realize and carry out assembling guiding for specific installation step, real-time matching is carried out with real assembling scene, can be flexibleEach stage of assembling is identified in real time and provides corresponding assembling guiding, passes through the reality assembling guiding of mixed display equipmentInformation has the advantages of directly perceived, has feeling of immersion, improves the efficiency of assembling guiding, reduces the error rate in assembling process.
Further, the identification device includes：
Unit is imaged, for obtaining the image of assembly in the reality assembling scene；
Assembly recognition unit, for being marked according to the identification being set in advance in all on the assembly, to the figureEach assembly as in is identified；
Hierarchical relationship recognition unit, for determining the hierarchical relationship between each assembly for identifying；
Assembling stage recognition unit, each assembly identified for basis and the hierarchical relationship, it is determined that currentlyAssembling stage.
Further, the identification device also includes：
Sensor group, for obtain respectively the RGB image of each assembly identified, depth image, exercise data,Temperature data, stress data and strain data；
Data processing unit, for according to default multisource data fusion algorithm, respectively in data level, feature level and decision-makingLevel is to RGB image, depth image, exercise data, temperature data, stress data and the dependent variable of each assembly identifiedAccording to fusion decision-making treatment is carried out, the real time status information for each assembly being identified out.
Further, in addition to：
Supervising device, for according to the real time status information to the assembling process and dress of each assembly identifiedIt is monitored with result, when assembling successfully, records the assembling result；When assembling occurs wrong, to the mixed realityEquipment sends miscue message.
Further, the model building device includes：
Modeling unit, for building virtual scene；
Assembling stage generation unit, for obtaining the attribute information of assemble flow and whole assemblies, and according to the dressMultiple assembling stages are generated with flow and attribute information；
Assembly relation recognition unit, for being determined according to the attribute information of each assembly between each assemblyHierarchical relationship and sequential relationship；
Guidance information generation unit, for generating each institute according to the hierarchical relationship between each assembly and sequential relationshipState the assembling guidance information of assembling stage；
Scene modeling unit is assembled, for by the attribute information of the assembly, the assembling stage, each assembling rankHierarchical relationship and sequential relationship between the assembling guidance information and each assembly of section are added in the virtual scene, obtainTo Virtual assemble scene.
The advantages of aspect that the present invention adds, will be set forth in part in the description, and will partly become from the following descriptionObtain substantially, or recognized by present invention practice.
Embodiment
The principle and feature of the present invention are described below in conjunction with accompanying drawing, the given examples are served only to explain the present invention, andIt is non-to be used to limit the scope of the present invention.
As shown in figure 1, it is a kind of stream of the assembling bootstrap technique based on mixed reality equipment provided in an embodiment of the present inventionJourney schematic diagram, this method comprise the following steps：
S1, builds Virtual assemble scene, and Virtual assemble scene is referred in traditional model buildings and virtual 3D scenes structureOn the basis of building, the dummy model of the assembly informations such as assemble flow, assembly attribute information is included.
In the building process of Virtual assemble scene, with the LOD technologies in development of games, 3D display performance can be optimizedAnd loading velocity, lift the usage experience of user.
S2, merged by mixed reality equipment with reality assembling scene.
For example, Hololens, or the mixed reality equipment of the brand such as grand base, Asus, Dell, or traditional calculations can be passed throughMachine and its display device etc. are realized.
S3, the assembling stage in scene is assembled to reality and assembly is identified, and can be obtained by multiple sensorsThe information such as image, the speed of assembly are taken, assembly is identified by the identification point being set in advance on assembly or characteristic value etc.,Can determines assembling stage according to the hierarchical relationship between default assembly after identification assembly.
S4, assembling guidance information is obtained according to the assembling stage identified and the assembly identified, assembles guidance informationIt is stored in advance in Virtual assemble scene, assembling guidance information is obtained by trigger condition.For example, mixing can be passed throughEye movement of the final function of eyeball of real world devices to user detects, when a certain assembly of the watching of user whenBetween reach preset time after, it is possible to automatically mixed reality equipment display interface ejection assembling guidance information.It can also lead toThe trigger conditions such as voice, gesture are crossed to obtain assembling guidance information.
It should be noted that assembling guidance information specifically includes：Operation information and guidance information, wherein：
Operation information includes word, image and voice that prompting user is operated etc., and the content of prompting is included currentlyAssembly, current assembling stage, the assembly that user is just being look at, interface message to be assembled, installation time, assemble sequence,Points for attention are assembled, assemble safety operation guide, the correct information of installation step, menu operation etc..
Guiding message includes the essential information of assembly, and the content of prompting includes：It is text panel, assembly information, to be installedWith information, specification and parameter information etc..
S5, assembling guidance information is shown by mixed reality equipment, assembling process is guided, when existing with mixingAfter the assembler of real equipment sees the assembling guidance information virtually shown, it is possible to according to virtual assembling guidance information to existingPieces of equipment in reality etc. is assembled, it is possible to increase the accuracy and efficiency of assembling, greatlys save time and labor cost.
It is appreciated that said process should also contain the conventional steps such as actual situation registration and visualization display.The present embodiment providesA kind of assembling bootstrap technique based on mixed reality equipment, by mixed reality equipment by the Virtual assemble scene of structure and existingReal field scape is merged, and the assembling stage in reality assembling scene and assembly are identified, and is realized for specificInstallation step carries out assembling guiding, carries out real-time matching with real assembling scene, can flexibly identify assembling in real timeEach stage and provide corresponding assembling guiding, have by mixed display equipment reality assembling guidance information intuitively excellentPoint, there is feeling of immersion, improve the efficiency of assembling guiding, reduce the error rate in assembling process.
A kind of as shown in Fig. 2 assembling bootstrap technique based on mixed reality equipment provided for another embodiment of the present inventionSchematic flow sheet, this method comprises the following steps：
S1, build Virtual assemble scene.
Preferably, as shown in figure 3, step S1 can be subdivided into following steps：
S11, virtual scene is built, virtual scene here is the three-dimensional scenic based on traditional 3D model buildings.
S12, the attribute information of assemble flow and whole assemblies is obtained, and it is more according to assemble flow and attribute information generationIndividual assembling stage.
It should be noted that the attribute information of assembly refers to the information such as the shape of assembly, material, assembling stageDivision can be according to actual setting, for example, for automobile this assembling process, can be by the different portion of assembling motor vehicleObstructed assembling stage is divided into, the assemble flow and assembly in each stage are different, for example, in the wheel of automobileAssembling stage, assembly are exactly wheel, and assemble flow is exactly related procedure for assembling bus or train route etc..
S13, hierarchical relationship and sequential relationship between each assembly are determined according to the attribute information of each assembly, hereHierarchical relationship and sequential relationship refer to the assemble sequence between assembly and the structural relation of assembling.
S14, the assembling guidance information of each assembling stage is generated according to the hierarchical relationship between each assembly and sequential relationship,The structural relation of assemble sequence and assembling between each assembly is obtained, has been known that the assembly method of each assembly, justThe assembling guidance information for being guided to assembling process can be generated.
S15, by the attribute information of assembly, assembling stage, the assembling guidance information of each assembling stage and each assembly itBetween hierarchical relationship and sequential relationship be added in virtual scene, obtain Virtual assemble scene.
Pass through the attribute information of assemble flow and whole assemblies build Virtual assemble scene, so as to get assembling scene bagAttribute information containing assemble flow and assembly, it is easy to identify different assembling stages, there is provided comprehensively assemble guidance information.
S2, merged by mixed reality equipment with reality assembling scene, ripe virtual reality device, increase at presentReal world devices and mixed reality equipment etc. can realize merging virtual scene and reality scene, will not be repeated here.
S3, the assembling stage in scene is assembled to reality and assembly is identified.
Preferably, as shown in figure 4, step S3 can be subdivided into following steps：
S31, the image of each assembly in reality assembling scene is obtained, for example, each dress can be obtained by depth cameraThe image of part, to determine the position of the image of each assembly.
S32, marked according to the identification being set in advance on whole assemblies, each assembly in image is identified.
It should be noted that the identification and tracking to assembly can be realized by technologies such as characteristic matching, binocular trackings.Specifically, can according to the two-dimensional signal and depth information of the assembly that sensor obtains, with reference to RGB cameras identification and chase afterTrack, particular location of the practical set body in virtual scene and the relative position in actual scene are determined, and carry out actual situationRegistration.
S33, it is determined that the hierarchical relationship between each assembly identified, because the hierarchical relationship of each assembling is to prestoreIn Virtual assemble scene, therefore, after identifying current assembly, it is possible to drawn according to current assembly between itHierarchical relationship.
S34, according to each assembly and hierarchical relationship identified, it is determined that current assembling stage, because assembling stage is pre-First set according to assemble flow and assembly, therefore, when identifying current assembly and hierarchical relationship each otherAfterwards, it is possible to it is determined that current assembling stage.
In step S3 refinement step, by being carried out according to the identification mark being set in advance on assembly to assemblyIdentification, and judge the hierarchical relationship between assembly, hierarchical relationship according to the assembly identified and each other, Neng GouzhunCurrent assembling stage really is identified, improves efficiency and the degree of accuracy of identification.
S4, assembling guidance information is obtained according to the assembling stage identified and the assembly identified.
S5, assembling guidance information is shown by mixed reality equipment, assembling process is guided.
S6, RGB image, depth image, exercise data, temperature data, the stress of each assembly identified are obtained respectivelyData and strain data.
For example, can by RGB video camera, depth camera, velocity sensor, acceleration transducer, temperature sensor,Nine axle sensors, strain transducer and strain gauge etc. obtain these data.
S7, it is each to what is identified in data level, feature level and decision level respectively according to default multisource data fusion algorithmRGB image, depth image, exercise data, temperature data, stress data and the strain data of assembly merge at decision-makingReason, the real time status information for each assembly being identified out., can be to assembling after data fusion decision-making treatment is carried outAssembly carry out actual situation registration, record assembly state and 3D scenes, for later stage system maintenance and manage basis is provided.
For example, RGB image, depth image, exercise data, temperature data, stress data and strain data can be carried outFeature extraction, respectively carries out the feature of extraction the data fusion of feature level, then by the data after fusion and Virtual assemble sceneMerged, show the real time status information of each assembly in real time.
By the way that multi-source data processing method is integrated into augmented reality scene Recognition and interaction, data are substantially increasedThe practicality of utilization rate and augmented reality, especially by multi-source data processing method to the RGB image of assembly, depth image,Exercise data, temperature data, stress data and strain data carry out fusion treatment, can efficiently identify out the real-time of assemblyState, it is easy to accurately guide assembling process, and assembling bootup process adjust and monitor in real time.
S8, it is monitored according to the assembling process of each assembly of the real time status information to identifying with assembling result, whenWhen assembling successfully, record assembling result；When assembling occurs wrong, miscue message is sent to mixed reality equipment, with sideAssembler is helped to correct loading error.
For example, the position of each assembly, state can be determined by detecting the real time status information of assembly in real timeDeng, contrasted with the default successful model of assembling, to determine whether to assemble successfully, or loading error.
Assembling process is monitored by the real time status information of the assembly identified, can be in real time to assemblingJourney is modified, and the guiding for assembling process can reduce the operating mistake in assembling process, and pass through with more ageingReal time status information is monitored to assembling result, can more accurately count assembling result, is easy to lookup assembling unsuccessfulAssembly.
As shown in figure 5, guide system for a kind of assembling based on mixed reality equipment 2 that another embodiment of the present invention providesStructural framing figure, the system includes：Model building device 1, mixed reality equipment 2, identification device 3 and supervising device 4, separately belowThis 4 devices are described in detail.
Model building device 1 is used to build Virtual assemble scene, and its function can be realized by computer, including：
Modeling unit 11, for building virtual scene.
Assembling stage generation unit 12, for obtaining the attribute information of assemble flow and whole assemblies, and according to assemblingFlow and attribute information generate multiple assembling stages.
Assembly relation recognition unit 13, for determining that the level between each assembly is closed according to the attribute information of each assemblySystem and sequential relationship.
Guidance information generation unit 14, for generating each assembling according to the hierarchical relationship between each assembly and sequential relationshipThe assembling guidance information in stage.
Scene modeling unit 15 is assembled, for the assembling of the attribute information of assembly, assembling stage, each assembling stage to be drawnLead the hierarchical relationship between information and each assembly and sequential relationship is added in virtual scene, obtain Virtual assemble scene.
Mixed reality equipment 2 is used to be merged Virtual assemble scene with reality assembling scene, is additionally operable to according to identificationThe assembling stage gone out and the assembly identified obtain assembling guidance information, and show assembling guidance information, and assembling process is enteredRow guiding.
For example, Hololens, or the mixed reality equipment 2 of the brand such as grand base, Asus, Dell, or tradition meter can be passed throughCalculation machine and its display device etc. are realized.
The assembling stage and assembly that identification device 3 is used to assemble reality in scene are identified, including：
Unit 31 is imaged, for obtaining the image of assembly in reality assembling scene.
Preferably, shooting unit 31 can include RGB video camera and depth camera.
Assembly recognition unit 32, for being marked according to the identification being set in advance on whole assemblies, in imageEach assembly is identified.
Hierarchical relationship recognition unit 33, for determining the hierarchical relationship between each assembly for identifying.
Assembling stage recognition unit 34, each assembly identified for basis and hierarchical relationship, it is determined that currently assembling rankSection.
Sensor group 35, for obtaining the RGB image of each assembly identified, depth image, exercise data, temperature respectivelyDegrees of data, stress data and strain data.
Preferably, sensor group 35 can include：RGB video camera, depth camera, velocity sensor, acceleration sensingDevice, nine axle sensors, temperature data, strain transducer and strain gauge etc..
Data processing unit 36, for according to default multisource data fusion algorithm, respectively in data level, feature level and certainlyPlan level is to RGB image, depth image, exercise data, temperature data, stress data and the strain data of each assembly identifiedFusion decision-making treatment is carried out, the real time status information for each assembly being identified out.
Supervising device 4 is used to assembling result be entered according to the assembling process of each assembly of the real time status information to identifyingRow monitoring, when assembling successfully, record assembling result；When assembling occurs wrong, miscue is sent to mixed reality equipment 2Message.
It should be noted that supervising device 4 can be integrated in mixed reality equipment 2, can also be as independent physicsIn the presence of monitoring assembling process.
Reader should be understood that in the description of this specification, reference term " one embodiment ", " some embodiments ", " showThe description of example ", " specific example " or " some examples " etc. mean to combine the specific features of the embodiment or example description, structure,Material or feature are contained at least one embodiment or example of the present invention.In this manual, above-mentioned term is shownThe statement of meaning property need not be directed to identical embodiment or example.Moreover, specific features, structure, material or the feature of descriptionIt can be combined in an appropriate manner in any one or more embodiments or example.In addition, in the case of not conflicting, thisThe technical staff in field can be by the different embodiments or example described in this specification and the spy of different embodiments or exampleSign is combined and combined.
It is apparent to those skilled in the art that for convenience of description and succinctly, the dress of foregoing descriptionThe specific work process with unit is put, the corresponding process in preceding method embodiment is may be referred to, will not be repeated here.
In several embodiments provided herein, it should be understood that disclosed apparatus and method, it can be passed throughIts mode is realized.For example, device embodiment described above is only schematical, for example, the division of unit, is onlyA kind of division of logic function, can there is an other dividing mode when actually realizing, for example, multiple units or component can combine orPerson is desirably integrated into another system, or some features can be ignored, or does not perform.
The unit illustrated as separating component can be or may not be physically separate, be shown as unitPart can be or may not be physical location, you can with positioned at a place, or can also be distributed to multiple networksOn unit.Some or all of unit therein can be selected to realize the mesh of scheme of the embodiment of the present invention according to the actual needs's.
In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, can alsoIt is that unit is individually physically present or two or more units are integrated in a unit.It is above-mentioned integratedUnit can both be realized in the form of hardware, can also be realized in the form of SFU software functional unit.
If integrated unit is realized in the form of SFU software functional unit and is used as independent production marketing or in use, canTo be stored in a computer read/write memory medium.Based on such understanding, technical scheme substantially orSay that the part to be contributed to prior art, or all or part of the technical scheme can be embodied in the form of software productOut, the computer software product is stored in a storage medium, including some instructions are causing a computer equipment(can be personal computer, server, virtual reality device, mixed reality equipment, or network equipment etc.) performs the present inventionThe all or part of step of each embodiment method.Foregoing storage medium includes but are not limited in the following manner：It is locally stored(RAM/ROM), mobile storage (USB flash disk, mobile hard disk etc.), webserver storage, cloud storage etc. are various can be with storage program generationThe medium of code.
More than, it is only embodiment of the invention, but protection scope of the present invention is not limited thereto, and it is any to be familiar withThose skilled in the art the invention discloses technical scope in, various equivalent modifications or substitutions can be readily occurred in,These modifications or substitutions should be all included within the scope of the present invention.Therefore, protection scope of the present invention should be wanted with rightThe protection domain asked is defined.