Summary of the invention
To solve the above-mentioned problems, the embodiment of the present invention provides a kind of mixed reality operation method for registering and device.
In a first aspect, the embodiment of the present invention provides a kind of mixed reality operation method for registering, comprising: obtain visual coordinate systemThe predetermined position of operating area is arranged in the space coordinate of lower visual indicia, the visual indicia；According to the visual indiciaSpace coordinate and the visual indicia and the operating area relative pose relationship, obtain operating area vision sitPose in mark system；It will include courses of action and components placement position according to pose of the operating area in visual coordinate systemVirtual three-dimensional model is transformed into visual coordinate system and is imaged, to realize operation registration.
Further, described according to the space coordinate of the visual indicia and the visual indicia and the operating spaceThe relative pose relationship in domain obtains operating area before the pose in visual coordinate system, further includes: by calibration, obtains instituteState the relative pose of operating area and the visual indicia.
Further, described by the virtual three-dimensional model including courses of action and components placement position, it is transformed into vision seatBefore mark system is imaged, further includes: according to the 3 d scan data of operating area, carry out three-dimensional reconstruction；After reconstructionThreedimensional model obtains the courses of action and the components placement position, to generate the virtual three-dimensional model.
Further, the visual indicia is arranged on guide plate matrix, and the guide plate is arranged in a manner of being bonded to be operatedAt region.
Further, described to obtain under visual coordinate system before the space coordinate of visual indicia, further includes: according to the behaviourMake path and the components placement position, the guide plate including visual indicia is generated by 3D printing.
Further, the guide plate waits for that observation window and/or pilot hole are arranged at operation；The observation window, for observing behaviourMake the position of instrument；The pilot hole, for guiding operation instrument.
Further, the space coordinate for obtaining visual indicia under visual coordinate system, the acquisition operating area are regardingFeel the pose and described by the virtual three-dimensional model including courses of action and components placement position in coordinate system, is transformed into viewFeel that coordinate system is imaged, is servo-actuated in real time carry out.
Second aspect, the embodiment of the present invention provide a kind of mixed reality operation registration apparatus, comprising: coordinate obtaining module,For obtaining the space coordinate of visual indicia under visual coordinate system, the predeterminated position of operating area is arranged in the visual indiciaPlace；Coordinate transferring, according to the space coordinate of the visual indicia and the phase of the visual indicia and the operating areaTo position orientation relation, pose of the operating area in visual coordinate system is obtained；Imaging module, for being regarded according to operating areaFeel that the virtual three-dimensional model including courses of action and components placement position is transformed into visual coordinate system by the pose in coordinate systemIt is imaged, to realize operation registration.
The third aspect, the embodiment of the present invention provides a kind of electronic equipment, including memory, processor and is stored in memoryComputer program that is upper and can running on a processor, processor realize first aspect present invention mixed reality behaviour when executing programThe step of making method for registering.
Fourth aspect, the embodiment of the present invention provide a kind of non-transient computer readable storage medium, are stored thereon with calculatingMachine program realizes the step of first aspect present invention mixed reality operation method for registering when the computer program is executed by processorSuddenly.
Mixed reality operation method for registering and device provided in an embodiment of the present invention, obtain visual indicia under visual coordinate systemSpace coordinate, the coordinate of visual indicia need to be only obtained in implementation process, without obtaining the threedimensional model in whole operation region, fromAnd calculation amount is smaller.By the virtual three-dimensional model including courses of action and components placement position, it is transformed into the progress of visual coordinate systemImaging is rebuild by three-dimensional data after obtaining the relative position of operating area and visual coordinate system, determines courses of action and portionPart placement location so as to accurately determine region to be operated, and then is advantageously implemented accurately operation registration.
Specific embodiment
In order to make the object, technical scheme and advantages of the embodiment of the invention clearer, below in conjunction with the embodiment of the present inventionIn attached drawing, technical scheme in the embodiment of the invention is clearly and completely described, it is clear that described embodiment isA part of the embodiment of the present invention, instead of all the embodiments.Based on the embodiments of the present invention, those of ordinary skill in the artEvery other embodiment obtained without creative efforts, shall fall within the protection scope of the present invention.
Fig. 1 is that mixed reality provided in an embodiment of the present invention operates method for registering flow chart, and Fig. 2 mentions for the embodiment of the present inventionThe mixed reality of confession operates method for registering application scenario diagram, and as shown in Figure 1, 2, the embodiment of the present invention provides a kind of mixed reality behaviourMake method for registering, comprising:
101, the space coordinate of visual indicia under visual coordinate system is obtained, the pre- of operating area is arranged in the visual indiciaIf at position.
The embodiment of the present invention realized by the wearable MR glasses such as mixed reality glasses, for example, HoloLens orMagicLeap mixed reality glasses.MR glasses 203 perceive the movement of user's sight by eyeball tracking sensor, and then automaticVirtual three-dimensional model is adjusted, and is projected in user's eye by light.In addition to this depth camera is also equipped on MR glasses 203It perceives user to carry out SLAM (positioning and map structuring immediately) calculating with inertial navigation sensors such as gyroscopes and is closingPosition and posture in scene, and then determine position of the dummy model relative to user.
In 101, visual coordinate system is the coordinate system of the depth camera in MR glasses 203, in the default position of operating areaThe place of setting is provided with visual indicia 201.Visual indicia 201 can be used QR code (one kind of two-dimensional bar code) or have height right by a series ofSpecific pattern than degree forms, and the angle point that chequered with black and white region is constituted in pattern can be used as characteristic point convenient for image procossing and visionDetection.Visual indicia constitutes a space coordinates.Visual indicia is obtained in vision by the depth camera module of MR glasses 203The space coordinate of coordinate system.
102, according to the space coordinate of the visual indicia and the opposite position of the visual indicia and the operating areaAppearance relationship obtains pose of the operating area in visual coordinate system.
In 102, pose (position and posture) relationship of visual indicia 201 and operating area has obtained in advance, according toThe position of visual indicia 201 can calculate position and posture of the operating area in 203 visual coordinate system of MR glasses indirectly.
103, according to pose of the operating area in visual coordinate system, by the void including courses of action and components placement positionQuasi-3-dimensional model is transformed into visual coordinate system and is imaged, to realize operation registration.
In 103, virtual three-dimensional model is the threedimensional model for including courses of action, components placement position and operating area,Component is the object for needing to be mounted on operating area, before operation and in operating process, need to be projected the threedimensional model by lightInto user's eye.The threedimensional model image of operating area and the true picture of operating area are overlapped in user's eye after projection, then realThe registration of operation is showed.At this point, the three-dimensional model diagram of courses of action, components placement position seem it is accurate and with actual position oneIt causes.Operating area is after the pose in visual coordinate system determines, then pose of the threedimensional model in visual coordinate system is accordingly trueIt is fixed, the predetermined virtual three-dimensional model including courses of action and components placement position, it is transformed into the progress of visual coordinate systemIt is imaged and carries out imaging projection, i.e. completion dummy model 205 and actual scene 204 is registrated, to realize the registration of operation.It can implement corresponding operation by operating instrument 202.
Mixed reality provided in this embodiment operates method for registering, and the space for obtaining visual indicia under visual coordinate system is satIt marks, the coordinate of visual indicia need to be only obtained in implementation process, without obtaining the threedimensional model in whole operation region, thus calculation amountIt is smaller.By the virtual three-dimensional model including courses of action and components placement position, it is transformed into visual coordinate system and is imaged, obtainedIt behind the relative position for obtaining operating area and visual coordinate system, is rebuild by three-dimensional data, determines that courses of action and component place positionIt sets, so as to accurately determine region to be operated, and then is advantageously implemented accurately operation registration.
Content based on the above embodiment, as a kind of alternative embodiment, according to the space coordinate of the visual indicia, withAnd the relative pose relationship of the visual indicia and the operating area, obtain pose of the operating area in visual coordinate system itBefore, further includes: by calibration, obtain the relative pose of the operating area and the visual indicia.I.e. visual indicia relative toThe relative pose relationship of operating area can be obtained by calibration in advance before the use.As alternative embodiment, the relative poseRelationship can also be measured in Three-dimensional Design Software and be obtained.By calibration, the phase of the operating area and the visual indicia is obtainedTo pose, be conducive to the calculating speed for improving subsequent coordinates conversion.
Content based on the above embodiment will include courses of action and components placement position as a kind of alternative embodimentVirtual three-dimensional model, be transformed into before visual coordinate system is imaged, further includes: according to the 3-D scanning number of operating areaAccording to progress three-dimensional reconstruction；According to the threedimensional model after reconstruction, the courses of action and the components placement position are obtained, with lifeAt the virtual three-dimensional model.
Before implementing operation, three-dimensional reconstruction is carried out according to the 3 d scan data of operating area, obtains operating area after reconstructionThreedimensional model.For example, scanning three-dimensional data by 3D point cloud, CT and CBCT (conical beam CT) etc. carries out three-dimensional reconstruction.In weightOperation planning, design operation path and ideal components placement position are carried out in threedimensional model after building, such as pass through design softwareThe planning in path and placement location is carried out, components placement position may also include the angle of component simultaneously and place the information such as depth.Threedimensional model after being determined according to courses of action and the components placement position, virtual three-dimensional model in generation.
Method provided in an embodiment of the present invention, is rebuild by 3 d scan data, ultimately generates the virtual three-dimensional model,Region and courses of action to be operated can be accurately determined, to be advantageously implemented accurate operation before operation is implementedRegistration.
Content based on the above embodiment, as a kind of alternative embodiment, visual indicia is arranged on guide plate matrix, guide plateIt is arranged at operating area by the way of fitting.
Fig. 3 is the guide plate structure schematic diagram provided in an embodiment of the present invention with visual indicia, as shown in figure 3, in order to guaranteeThe relative position of visual indicia and operating area is not susceptible to change, and in the embodiment of the present invention, visual indicia 303 is arranged in guide plateMatrix 301 on.Guide plate is arranged on operating area by the way of fitting, i.e., is set according to the contoured surface shape of operating areaFamily planning allows to tightly be fitted in operating area face, provides good immobilizing foundation at the inner surface of guide plate.
As an alternative embodiment, visual indicia 303 can be connect by connecting rod 302 with the matrix 301 of guide plate, thusOutside extended operating area, consequently facilitating depth camera obtains visual indicia, it will not be because of the screening of operating area in operating processIt keeps off and is affected.
Content based on the above embodiment, as a kind of alternative embodiment, visual indicia under the acquisition visual coordinate systemSpace coordinate before, further includes: according to the courses of action and the components placement position, by 3D printing generate described in leadPlate.
In order to guarantee that the relative position of visual indicia and operating area is not susceptible to change in operation, the present invention is realIt applies example visual indicia is arranged on guide plate, before implementing operation, integrally be printed by rapid shaping techniques such as 3D printingsThe above-mentioned guide plate equipped with visual indicia.In view of guide plate is for implementing operation, convenient for placing component, the specific shape of guide plateShape need to specifically be determined according to courses of action and components placement position.As alternative embodiment, courses of action and components placement positionIt can be obtained according to virtual three-dimensional model in above-described embodiment.The embodiment of the present invention is put according to the courses of action and the componentSeated position generates the plantation guide plate by 3D printing, so as to reduce the Production Time of guide plate, and can generate and be conducive toOperate the guide plate implemented.
Content based on the above embodiment, as a kind of alternative embodiment, the guide plate wait at operation be arranged observation window and/Or pilot hole；The observation window, for observing the position of operation instrument；The pilot hole, for guiding operation instrument.Such as Fig. 3,Guide plate waits for that pilot hole 304 is arranged at operation.In order in operation, further increase the precision of registration, can wait grasping in guide plateAt work, that is, the key position of operation is embodied, observation window is set, the observation of operating process is easy to implement.Meanwhile it may also set upPilot hole, for guiding the position of operation instrument mobile.
Content based on the above embodiment, as a kind of alternative embodiment, visual indicia under the acquisition visual coordinate systemSpace coordinate, it is described to obtain pose of the operating area in visual coordinate system, and will include that courses of action and component are placedThe virtual three-dimensional model of position is transformed into visual coordinate system and is imaged, and is servo-actuated in real time carry out.
Due to operator or a certain range of movement can occur, operating area also due to the reason of application scenarios generateIt is mobile.In the embodiment of the present invention, to the spatial position transformation of the acquisition of the coordinate of visual indicia, three dimensional virtual models and virtual threeThe imaging of dimension module is servo-actuated in real time carries out.Real-time perfoming refers to the acquisition of coordinate and conversion is to execute in real time, non-delay.WithWhen change in location occurs for dynamic finger operating area or operator, the coordinate of visual indicia is obtained again accordingly, carry out coordinate conversionAnd imaging generates.In this situation, three dimensional virtual models can follow in real scene operating area or operator together withIt is dynamic, to give operator's accurately visual feedback, avoid the generation of operating accident.
Fig. 4 is that mixed reality provided in an embodiment of the present invention operates registration apparatus structure chart, mixed reality behaviour as shown in Figure 4It include: coordinate obtaining module 401, coordinate transferring 402 and imaging module 403 as registration apparatus.Wherein, coordinate obtainsModule 401 is used to obtain the space coordinate of visual indicia under visual coordinate system, and the pre- of operating area is arranged in the visual indiciaIf at position；Coordinate transferring 402 is according to the space coordinate of the visual indicia and the visual indicia and the operationThe relative pose relationship in region obtains pose of the operating area in visual coordinate system；Imaging module 403 is used for according to behaviourMake pose of the region in visual coordinate system to be transformed into the virtual three-dimensional model including courses of action and components placement positionVisual coordinate system is imaged, to realize operation registration.
The predetermined position of operating area is provided with visual indicia, visual indicia constitutes a space coordinates.Pass throughCoordinate obtaining module 401 obtains visual indicia in the space coordinate of visual coordinate system.
The position orientation relation of visual indicia and operating area has obtained in advance, and coordinate transferring 402 is according to visual indiciaPosition can calculate position and posture of the operating area in visual coordinate system under true environment indirectly.
Virtual three-dimensional model is the threedimensional model for including courses of action, components placement position and operating area, operation before andIn operating process, which need to be projected in user's eye by light.After projection user eye in operating area three-dimensionalThe true picture of model image and operating area is overlapped, then realizes the registration of operation.At this point, courses of action, component place positionThe three-dimensional model diagram set seems accurate and consistent with actual position.Pose of the operating area in visual coordinate system determinesAfterwards, pose of the threedimensional model in visual coordinate system accordingly determines, imaging module 403 is predetermined including operation roadThe virtual three-dimensional model of diameter and components placement position is transformed into visual coordinate system and is imaged and is carried out imaging projection, that is, completesDummy model is registrated with actual scene, to realize the registration of operation.
Installation practice provided in an embodiment of the present invention is the detailed process and in detail in order to realize above-mentioned each method embodimentThin content please refers to above method embodiment, and details are not described herein again.
Mixed reality provided in an embodiment of the present invention operates registration apparatus, and coordinate obtaining module obtains to be regarded under visual coordinate systemFeel the space coordinate of label, the coordinate of visual indicia need to be only obtained in implementation process, without obtaining the three-dimensional in whole operation regionModel, so that calculation amount is smaller.Imaging module turns the virtual three-dimensional model including courses of action and components placement positionIt changes to visual coordinate system to be imaged, after obtaining the relative position of operating area and visual coordinate system, passes through three-dimensional data weightIt builds, determines courses of action and components placement position, so as to accurately determine region to be operated, and then be advantageously implemented standardTrue operation registration.
Fig. 5 is the entity structure schematic diagram of a kind of electronic equipment provided in an embodiment of the present invention, as shown in figure 5, the electronicsEquipment may include: processor (processor) 501, communication interface (Communications Interface) 502, storageDevice (memory) 503 and bus 504, wherein processor 501, communication interface 502, memory 503 complete phase by bus 504Communication between mutually.Communication interface 502 can be used for the information transmission of electronic equipment.Processor 501 can call in memory 503Logical order, with execute include following method: obtain visual coordinate system under visual indicia space coordinate, the vision markThe predetermined position of operating area is arranged in note；According to the space coordinate of the visual indicia and the visual indicia and instituteThe relative pose relationship of operating area is stated, pose of the operating area in visual coordinate system is obtained；According to operating area in visionPose in coordinate system, by the virtual three-dimensional model including courses of action and components placement position, be transformed into visual coordinate system intoRow imaging, to realize operation registration.
In addition, the logical order in above-mentioned memory 503 can be realized by way of SFU software functional unit and conductIndependent product when selling or using, can store in a computer readable storage medium.Based on this understanding, originallySubstantially the part of the part that contributes to existing technology or the technical solution can be in other words for the technical solution of inventionThe form of software product embodies, which is stored in a storage medium, including some instructions toSo that a computer equipment (can be personal computer, server or the network equipment etc.) executes the above-mentioned each side of the present inventionThe all or part of the steps of method embodiment.And storage medium above-mentioned include: USB flash disk, mobile hard disk, read-only memory (ROM,Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic or disk etc. it is variousIt can store the medium of program code.
On the other hand, the embodiment of the present invention also provides a kind of non-transient computer readable storage medium, is stored thereon with meterCalculation machine program, the computer program are implemented to carry out the transmission method of the various embodiments described above offer when being executed by processor, such asIt include: the space coordinate for obtaining visual indicia under visual coordinate system, the predeterminated position of operating area is arranged in the visual indiciaPlace；According to the space coordinate of the visual indicia and the relative pose relationship of the visual indicia and the operating area, obtainPose of the extract operation region in visual coordinate system；It will include operation road according to pose of the operating area in visual coordinate systemThe virtual three-dimensional model of diameter and components placement position is transformed into visual coordinate system and is imaged, to realize operation registration.
The apparatus embodiments described above are merely exemplary, wherein described, unit can as illustrated by the separation memberIt is physically separated with being or may not be, component shown as a unit may or may not be physics listMember, it can it is in one place, or may be distributed over multiple network units.It can be selected according to the actual needsIn some or all of the modules achieve the purpose of the solution of this embodiment.Those of ordinary skill in the art are not paying creativenessLabour in the case where, it can understand and implement.
Through the above description of the embodiments, those skilled in the art can be understood that each embodiment canIt realizes by means of software and necessary general hardware platform, naturally it is also possible to pass through hardware.Based on this understanding, onStating technical solution, substantially the part that contributes to existing technology can be embodied in the form of software products in other words, shouldComputer software product may be stored in a computer readable storage medium, such as ROM/RAM, magnetic disk, CD, including several fingersIt enables and using so that a computer equipment (can be personal computer, server or the network equipment etc.) executes each implementationMethod described in certain parts of example or embodiment.
Finally, it should be noted that the above embodiments are merely illustrative of the technical solutions of the present invention, rather than its limitations；AlthoughPresent invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that: it still may be usedTo modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features；And these are modified or replaceed, technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution spirit andRange.