US20140168264A1 - System, method and computer program product for real-time alignment of an augmented reality device - Google Patents

Abstract

A system including a structure attachable to a surface in a real world environment, the structure establishing a known location and orientation of the structure, a docking element, as part of the structure, to secure an augmented reality device in a stationary position for alignment of the augmented reality device with the real world environment and with a parallel virtual environment, and a processor operable to perform the alignment by resetting the inertial navigation system of the augmented reality device to the known location when docked in the docking element and aligning the location and orientation of the virtual representation of the augmented reality device in the parallel virtual environment so that the parallel virtual environment in the augmented reality device overlaps the real world environment. A method and computer software produce are also disclosed.

Description

BACKGROUND
• Embodiments relate to aligning an inertial navigation system with a real-world environment and a parallel virtual-world environment. Motion calibration of the inertial navigation system of a head mounted display within these environments is also provided,
• A head-mounted display, such as a helmet-mounted display or eyeglass-mounted display (abbreviated herein as an “HMD” and “HMDs” when referring to a plurality) is a display device worn on the head of an individual that has one or more small display devices located near one eye or, more commonly, both eyes of the individual, user, or wearer. HMDs may be monocular (where one eye has a view screen), biocular (where both eyes see the same scene on the same screen), or binocular (where each eye has an individual view screen).
• Some HMDs display only simulated (computer-generated) images, as opposed to real-world images and, accordingly are often referred to as “virtual reality” or immersive HMDs. Other HMDs superimpose (combine) a simulated image upon a non-simulated, real-world image. The combination of non-simulated and simulated images allows the HMD user to view the world through, by way of non-limiting example, a visor or eyepiece on which additional data relevant to the task to be performed may be superimposed onto the forward field of view (FOV) of the user. This superposition is sometimes referred to as “augmented reality” or “mixed reality.”
• Combining a non-simulated, real-world view with a simulated image can be achieved using a partially-reflective/partially-transmissive optical surface (a “beam splitter”) where the surface's reflectivity may be used to display the simulated image as a virtual image (in the optical sense) and the surface's transmissivity may be used to allow the user to view the real world directly (referred to as an “optical see-through system”). Combining a real-world view with a. simulated image can also be done electronically by accepting video of a real world view from a camera and mixing it electronically with a simulated image using a combiner (referred to as a “video see-through system”). The combined image can then be presented to the user as a virtual image (in the optical sense) by means of a reflective optical surface, which in this case need not have transmissive properties.
• As a part of the HMD, an inertial navigation system (“INS”), or sometimes referenced as an inertial measurement unit (“IMU”), may be included to provide detailed, accurate position and orientation information. The inertial navigation system usually possesses rapid processing capabilities while requiring a minimum amount of power. Furthermore, the INS does not typically require constant satellite visibility because internal components such as, but not limited to, at least one gyroscope and at least one accelerometer (or other motion-sensing devices) in communication with a processor, are used to continuously calculate position, orientation, and velocity (including direction and speed of movement).
• The INS of the HMD usually gets its velocity calibration from another source such as, but not limited to, a human operator, pre-set factory (or laboratory) specifications, a satellite receiver, etc. Whereas location information has to be provided by a human operator or other means when the HMD is powered up. By way of non-limiting example, when the HMD is turned on, or powered up, information may be entered regarding where the HMD is located and its orientation, such as through a graphic user interface connected to a computer which processes INS data. Whereas, when the HMD is turned off it may not update and integrate its current location, and thus needs to be told where it is and in what direction it is pointing when it is turned on again. Henceforth, the accelerometers and gyroscope information provide rates of change of position or orientation, and these rates must be integrated, with the given initial condition specified in order to estimate the current position and orientation of the HMD.
• However, the gyroscopes can drift and the accelerometer can lose accuracy, resulting in errors with respect to position, velocity, and orientation. Though the errors may at first be small, depending on the use of the HMD, such errors could have significant consequences. Furthermore, the errors may expand over time. Thus the INS may require periodic aligning, refreshing, and/or updating with accurate location and orientation information.
• Furthermore, each user has a unique view through the HMD due to each user's inter-pupillary distance and the depth of eyes in each user's head. Additionally, the height of the user may also affect the user's ability to properly register to an external environment viewable through the HMD. If multiple users utilize the same HMD, the device may only be calibrated for one of the users, thus making the HMD not effective for the other users.
• Currently, most updating of the inertial navigation system of the HMD is accomplished in a laboratory, or a sterile facility. Ample tools and equipment are available at such locations to a user to calibrate the inertial navigation system of the HMD and to adjust the HMD to best accommodate a particular user. However, the same is not true when an HMD is being used away from the sterile facility, more specifically when the HMD is being used in the field, or in an operational setting. When in the operational setting, as an error may occur and possibly expand, or multiple users may need to use the HMD, but without an ability to align, refresh or adjust the HMD in real-time, the user may simply cease using the HMD. Thus, manufacturers and users of HMDs would benefit from being able to perform real-time alignments, refreshments and/or adjustments of the HMD when the HMD is being used in an operational setting.
BRIEF DESCRIPTION
• Embodiments relate to a system, method, and computer software code for alignment of an augmented reality inertial navigation system. An embodiment of the system may comprise a structure attachable to a surface in a real world environment, the structure establishing a. known location and orientation of the structure, and a docking element, as part of the structure, to secure an augmented reality device in a stationary position for alignment of the augmented reality device with the real world environment and with a parallel virtual environment. The system may also comprise a processor operable to perform the alignment by resetting the inertial navigation system of the augmented reality device to the known location when docked in the docking element and aligning the location and orientation of the virtual representation of the augmented reality device in the parallel virtual environment so that the parallel virtual environment in the augmented reality device overlaps the real world environment.
• An embodiment of the method may comprise establishing a known location and orientation of a calibration location post having a docking element, and providing an augmented reality device with the known location and orientation when docked in the docking element. The method may also comprise calibrating the augmented reality device to align a virtual environment with the real world environment of the augmented reality device with respect to a current location and orientation of the post.
• The computer software code is a part of a non-transitory processor readable storage medium, providing an executable computer program product, the executable computer program product may comprise a computer software code that, when executed on a processor, initiate acquisition of information comprising a current location and/or orientation of an electronic calibration and/or location post, and detect when an augmented reality device is attached to the post. The computer software code may further, when executed on a processor, calibrate the augmented reality device to align a virtual environment with a real world environment of the augmented reality device with respect to a current location and/or orientation of the post.
BRIEF DESCRIPTION OF THE DRAWINGS
• A more particular description briefly stated above will be rendered by reference to specific embodiments thereof that are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments and are not therefore to be considered to be limiting of its scope, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
• FIG. 1 shows an embodiment of a location post in accordance with an embodiment;
• FIG. 1ashows an example of an HMD alignment with a real world environment simultaneously with a parallel virtual world environment in accordance with an embodiment;
• FIG. 2 shows a location post in use in an operational environment accordance with an embodiment;
• FIG. 3 shows the an advanced electronic post with a horizontal linear motion calibration system in accordance with an embodiment;
• FIG. 4 shows a diagram illustrating a calibration technique in accordance with an embodiment;
• FIG. 5 shows a diagram illustrating waveform examples used in calibration;
• FIG. 6 shows a block, diagram illustrating the electronic post in accordance with an embodiment;
• FIG. 7 shows a diagram illustrating a side view of a head-mounted device in accordance with an embodiment;
• FIG. 8 shows a diagram illustrating a frontal view of a head-mounted device in accordance with an embodiment;
• FIG. 9 shows a block diagram illustrating elements of the head-mounted device in accordance with an embodiment;
• FIG. 10 shows a flowchart of a method of an embodiment;
• FIG. 11 shows a flowchart of another method of an embodiment;
• FIG. 12ashows the rotational motion calibration fixture in accordance with an embodiment; and
• FIG. 12bshows an HMD installed in the rotational motion calibration fixture in accordance with an embodiment.
DETAILED DESCRIPTION
• Reference will be made below in detail to embodiments which are illustrated in the accompanying drawings. Wherever possible, the same reference numerals used throughout the drawings refer to the same or like parts. Broadly speaking, a technical effect may be to align an. augmented reality inertial navigation system for both a single user and multiple users of a single head-mounted device or display (“HMD”). To facilitate an understanding of embodiments, reference to specific implementations thereof is provided herein.
• Throughout this specification the terms “align,” “refresh,” “calibrate” and/or “update,” including verbs of these terms, may be utilized interchangeably. Each term is associated with correcting an error that may have developed with an inertial navigation system (“INS”) of the HMD or establishing a location of the HMD. The INS may also be referred to as an inertial measurement unit. The intent of the INS may be a navigation aid that uses a processor, motion sensors (namely at least one accelerometer), and rotation sensors (namely at least one gyroscope). Whereas the term “adjust” may be used with respect to changing or correcting a physical feature on the HMD; however, it may also be used with respect to the above discussed terms. The term “adjust” primarily may be considered to consist of an electronic adjustment internal to the augmented reality device, such that it applies correction data to the current location and orientation information that the AR device is representing internal as the AR device location and orientation. Other internal AR Device electronic adjustments include shifting the image left, right, up, or down in order to align the internal virtual environment imagery that the user is presented with to each eye such that it matches calibration imagery that exists in the real world, and that it is aligned to adjust to the positioning of the user's eyes. Another internal AR device electronic adjustment is that of changing the rates and internally expected directions of motion with respect to linear and rotational velocities and accelerations, when the AR device is calibrated using the motor-driven linear and rotation calibration devices.
• In an embodiment, the location post may be passive where the post may contain no electronics, and configured to provide an ID number and a dock for the HMD. When placed in the dock, the HMD may automatically perform an internal calibration, resetting all offsets to location and rotation to zero, and using the current location and orientation of the location post to which the HMD is physical placed as the new initial condition for subsequent estimation of the location and orientation through the use of the INS.
• In an embodiment, the location post is an electronic post where the location post provides a digital communication system to communicate location and orientation information to the attached HMD. In an embodiment, the electronic post may provide additional information such as user biometric calibration data stored on the post or in a remote location. After the biometric calibration data is provided, then the HMD may perform the same reset of the INS offsets and accept the current location and orientation as the starting point for subsequent location estimation based upon INS processing.
• In an embodiment, the location post may be an INS-calibrating electronic post. In this embodiment, the INS-calibrating electronic post includes motor-driven calibration capabilities that allow the translational and rotational velocity and acceleration sensor calibration to be aligned and adjusted.
• In an embodiment, the location post may be built into a building. The post may be part of a permanent fixture, and not intended to be transitory or moveable. The location and orientation of the location post may have been determined by a surveyor team, and this information may be programmed into the post, printed on it, or stored in a remote database for network access to the user of the post. In this manner, the post may be provided as a public service by businesses in a shopping center to facilitate the use of augmented reality devices in the area by allowing the user to calibrate their own augmented reality devices as they enter the area. These permanent location posts can be placed in a building in the same manner that Automated Teller Machines are provided. They may be permanently calibrated and installed at military training sites where users can calibrate their AR devices with respect to the location, and/or recalibrate the AR device periodically due to the drift that occurs in the INS.
• Though embodiments are disclosed herein with reference to the HMD, the embodiments are applicable with other devices. As a non-limiting example, the embodiments may be utilized with other augmented reality implements or devices. Thus the descriptions regarding the embodiments provided herein which specifically discuss being used with an HMD shall not be considered limiting as the term augmented reality device may be used in place of HMD.
• Referring now to the drawings, embodiments will be described. Embodiments can be implemented in numerous ways, including as a system (including a. computer processing system), a method (including a computerized method), an apparatus, and/or with a non-transitory processor readable storage medium. Several embodiments are discussed below.
• FIG. 1 shows an embodiment of a location post. In the embodiment ofFIG. 1, the location post is an electronic post.FIG. 6 shows a block diagram illustrating the electronic post. The electronic post will be described in relation toFIGS. 1 and 6. Theelectronic post10 may be a self-contained unit. Theelectronic post10 may be secured to asupport surface12 at a specific location which will result in theelectronic post10 remaining stabilized (immobilized) during an alignment process. The specific location may be a plurality of places including, but not limited to, a vehicle (such as to a roof of a ground transportation vehicle), location upon the ground, and/or any other location which is remote from a laboratory or facility where anHMD18 is manufactured and is able to provide theelectronic post10 stationary and/or stabilized placement. Theelectronic post10 may comprise a securingelement14, or device, as illustrated inFIG. 6, to connect theelectronic post10 to thesupport surface12 at a location. In an embodiment, the securingelement14 may provide a temporary connection or a permanent connection to a vehicle, such as a hood or roof of the vehicle.
• In an embodiment, a permanent attachment to a building or location also may be used and this would allow the post to be calibrated once. In an embodiment where the electronic post is attached to vehicle, the electronic post may require recalibration after the vehicle moves to avoid miscalibrations that may arise from vehicle motion.
• To provide the temporary connection, the securingelement14 may comprise a magnet or another easily removable connector. When connecting theelectronic post10 to the ground, the securingelement14 may comprise a tripod support stand or another support having a pointed end, or insertion element, which may be inserted into the ground. By using the securingelement14 temporarily, theelectronic post10 may be easily moved from one location to another.
• Theelectronic post10 may further comprise theprocessor24 which controls many of the other elements that make up theelectronic post10. Theprocessor24 may be used to activate thedocking element22 while also delivering information to thedocking element22, which in turn provides the information to theHMD18. Thedocking element22 may be motorized, hence amotor32 is provided. Information from theHMD18 may also be delivered from theHMD18, through thedocking element22 to theprocessor24. Likewise, theprocessor24 may be used to activate thelinear translation unit40, and transmit information and receive information from theHMD18 by way of thelinear translation unit40, and may also be used to activate the rotational motion unit300 (FIG. 12A) and rotational unit motor330 (FIG. 12A) and transmit information and receive information from the HMD by way of therotational motion unit300.
• The post may include amagnetometer26. Information may be provided to and from themagnetometer26, by way of communications with the processor24 (such as, but not limited to, magnetic declination data) where information from the magnetometer, or frommemory storage65, or remote storage, if the magnetic declination has already been determined, may eventually be provided to themagnetometer28 of theHMD18.
• Theelectronic post10 may include adata storage device65 which is also in communication with theprocessor24. Thedata storage device65 may be used to store, or maintain, calibration data, biometric data specific to a user, information collected from theHMD18, and/or a processor-specific application. Another element that is in communication with theprocessor24 is theindicator30, which may notify the user. In an embodiment, the user may be notified that a particular calibration fail to calibrate theHMD18. Theindicator30 may be a computer graphics display capable of displaying text, another visual indicator (such as an illumination), and/or an audible indicator. Also disclosed is a physical component of theelectronic post10. The securingelement14 is used to connect theelectronic post10 to a desired location.
• Theelectronic post10 may include adata entry port67. Thedata entry port67 may comprise a plurality of configurations. In one embodiment thedata entry port67 may comprise a biometric reader, by way of non-limiting example, a thumb print reader or retinal scanner. The purpose of this form of data entry port is to access biometric information from a particular user based on a biometric characteristic unique to the user. The type of biometric information to be accessed includes, but is not limited to, inter-pupillary distance and depth of eyes in the head70 (FIG. 7) of the user, height of the user, fingerprints of the user, etc. Such information is needed because each user has a unique view through theHMD18 due to each user's inter-pupillary distance and depth of the user's eyes in the head, and height (of the eyes from the ground). Thus, alignment of the geometry of theHMD18 with thehead70 of the user is needed. As described further below, this alignment may be accomplished by adjusting aframe72 of theHMD18 or by adjusting vision components that are supported by the frame of theHMD18. Inter-pupillary adjustment may comprise bringing binocular views of the lens system83 (such as is shown inFIGS. 7 and 8) closer together or further apart horizontally. By doing so, the images within theHMD18 are moved with respect to where they would have been placed in the real world. In another embodiment, the user may use a manual adjuster, such as a dial, or even a computer graphical user interface (“GUI”) interface to adjust what the user sees through theHMD18 to the real world.
• Another form of thedata entry port67 may comprise a keypad Which the user may use to enter information including, but not limited to, biometric information, external environmental conditions, etc. Though disclosed as two distinct embodiments, thedata entry port67 may comprise both the biometric reader and the keypad. As explained briefly above, the types of data which may be entered with the data entry port may also be transmitted from a remote location to theelectronic post10.
• In one embodiment, theelectronic post10 may comprise avisual detection device52 capable of detecting images in both a real environment and the virtual environment.
• Theprocessor24 has access to calibration data. The post includes a receive16 andtransmitter17. Additional calibration data may be collected through thereceiver16. Thereceiver16 may collect additional(data specific to where theelectronic post10 is placed with respect to a coordinate system. Thedocking element22 may be controlled by theprocessor24 and is also configured to secure theHMD18 to thedocking element22 during movement of thedocking element22. The movement of thedocking element22 may occur with respect to a coordinate system as described above to determine measured data that may be associated with at least one component of theHMD18. Theprocessor24 may compare the measured data with the calibration data, and calibrate theHMD18. More specifically, the at least one component that may he outside of an acceptable tolerance is calibrated with respect to a measurement from theHMD18.
• In another embodiment, the location post may be passive. The post may be located and positioned at a known location and orientation in the real world environment, having a known and overlapping location in a parallel virtual-reality. In a non-limiting example, when used in an operational field or theater, the location post may be located at a pre-selected waymarker or waypoint that the user will pass by or transgress when in the field. Based on its location, location and orientation data about the location post may be stored in a memory device that is not a part of the post, such as at a remote location. The post may include an identifier such as, but not limited to, some form of an identification determinator (a bar code or another form of identification) which identifies the specific location post. The identification determinator may be readable by theHMD18, such as when theHMD18 is connected to or docked in thedocking element22 of thelocation post10.
• In another embodiment, the identification determinator of the location post may be entered into theHMD18, such as through at least onecamera90a,90b(FIGS. 7,8, and9) on the HMD using optical recognition. The at least onecamera90a,90bmay be configured to capture an identification (ID) fixedly placed on the post. The HMD may be configured to optically recognize the identification of the instant location post based on the captured ID. By way of a non-limiting example, the post includes a code, such as but not limited to a post barcode. TheHMD18 may searches its own internal database to find the location and orientation data associated with the barcode of the post. Alternately, the HMD may communicate through a communications network to search a remote database for the location and orientation data associated with the barcode of the post. Location data may include coordinate information.
• Once theHMD18 is installed in thedocking element22 of the post (passive or electronic), an internal calibration may be performed on theHMD18 based on location and orientation data regarding the post. An internal calibration may be accomplished based on theHMD18 accessing a location and orientation data associated with the post (such as, but not limited to a localized database) or by theHMD18 communicating to a remote location that maintains a database with location and orientation data specific to the post, where the information is associated with the post ID. Theprocessor24 on theelectronic post10 or theprocessor224 on theHMD18 may be used to perform the calibration.
• The passive post may align the location and orientation data of the HMD with both the real world environment and the parallel virtual world. Furthermore, the passive post may allow magnetic declination data to be aligned or updated using external data.
• In another embodiment, the user may input the identification determinator into theHMD18, such as through thedata entry port67 of theelectronic post10 or alternately, an user input mechanism associated with the HMD.
• In a non-limiting example, in operation, location and orientation data of theelectronic post10 may be stored in a memory device that is not a part of the post (remote from the electronic post10).
• The HMD may then be placed on the docking element of theelectronic post10. Provided that theelectronic post10 is physically oriented and positioned in accordance with the data regarding its orientation and location stored remotely from theelectronic post10, using itsinternal processor224, theHMD18 is internally calibrated so that its internal representation of orientation and position correspond with the remotely stored data associated with the orientation and position of theelectronic post10 and hence the real-world.FIG. 1ashows an example400 ofHMD18 alignment with areal world environment410 simultaneously with a parallelvirtual world environment420. InFIG. 1 a, the location post is used to align the parallelvirtual world420 and theHMD18 with the real world location and orientation of the location post. The orientation may be controlled through the rigid docking fixture of the location post.
• The parallelvirtual world environment420 is expected to closely match the geometry of thereal world environment410. However, since the parallelvirtual world environment420 is maintained in a computer that drives theHMD18, it may not have been lined up exactly with the real world such that the parallel virtual world precisely overlays the real world. Alternately the parallelvirtual world environment420 may not be aligned or nearly aligned with thereal world environment410 if theHMD18 was just powered on and needs to be programmed with its current location and orientation.
• There is a location post setup in thereal world410 and indicated by an “X” denoted by thereference numeral440 on astructure415. TheX440 in thereal world410 should match the location of the “X” denoted byreference numeral450 in the parallelvirtual world420, as would thestructure425 in the parallel virtual world. TheHMD18 also has an denoted by thereference numeral460 that should overlay the real world'sX440 and the parallel virtual world'sX450 when the HMD is placed at thelocation440 in the real world. TheHMD18 may not precisely indicate its position in the real orvirtual worlds410,420 due to calibration and alignment errors. in an embodiment, the calibration and alignment errors may have formed in theINS38 of theHMD18.
• The alignment process is configured to align theHMD18 with thereal world410 and the parallel virtual world simultaneously. To perform the alignment of theHMD18 with thereal world410 and the parallel virtual world simultaneously, theHMD18 is placed in thedocking element22 at the location post denoted byX440 in the real world so that theHMD18 points in an expected direction. Then, the virtual world location and orientation are set/adjusted/updated by rotating and translating the virtual world electronically in the 3D image generation system (3DIGS)77 that drives the virtual imagery displayed on theHMD18. The virtual world may be adjusted to be at precisely the coordinates and rotational state established by thedocking element22 on the location post denoted byX440 in thereal world410. Hence, theHMD18 is currently located at a known location and the virtual world is aligned precisely with the HMD and the real world. At this time, the offsets produced by theINS38 may all be reset to zero for both rotation and translation offsets.
• After theHMD18 is removed from thedocking element22 theINS38 may start integrating differential motion indicated by theaccelerometers34 andgyroscopes36 in theINS38 to estimate the current location of theHMD18 and the direction in which theHMD18 is pointing.
• After some period of time and amount of motion, the location and orientation reported by theINS38 may drift. The drift may cause theINS38 to no longer synchronize thereal world410 with thevirtual world420 and with theHMD18. Thus, theHMD18 needs to be aligned again by repeating the alignment process. To repeat the alignment process, the user may place theHMD18 into an available location post and the alignment process may commence thereafter.
• In an embodiment, the alignment of theHMD18 in the virtual and real worlds is not accomplished by the rotation/motion operations. Instead, theHMD18 is secured to thedocking element22 of the post. A docked HMD is in a known location and orientation in the real world for which the virtual world in theHMD18 can be aligned.
• Acalibration unit19 may be used to calibrate theelectronic post10. Thecalibration unit19 may comprise a global positioning system receiver, a compass, and other equipment to ensure accurate localizing and coordination of the real environment and a virtual environment, usually a parallel virtual environment, with the real environment forHMD18. In this way, the virtual environment may also be calibrated to bring it into alignment with the real environment for theHMD18 using theelectronic post10 as a real/virtual shared landmark. Thus, as disclosed further herein, once theelectronic post10 is calibrated and/or located, theHMD18, and/or several HMDs, may be rapidly aligned by physical attachment to theelectronic post10, where theelectronic post10 communicates with theHMD18 to provide calibration and location information to theHMD18.
• Calibration of the post may also be performed manually. In an embodiment, a manual calibration is adapted to use surveying and metrology techniques using the location and orientation data entered into the post or stored in an external database.
• In an embodiment theseparate receiver16 and theseparate receiver17, may be a single unit or transceiver. Thus the term “receiver/transmitter” may be used to describe the receiver and transmitter as separate units or as a single unit or transceiver. The receiver/transmitter16,17 may be used to communicate, usually digitally, with a head-mounted display (“HMD”)18 to obtain information from theHMD18 when the HMD is placed in a certain pose, and/or moved around, or to transmit to theHMD18 how to adjust its internal calibration tables when a calibration function is performed. In another embodiment an additional, or second, receiver/transmitter may be provided. The first receiver/transmitter16,17 may be dedicated to allowing theHMD18 to communicate with theelectronic post10. The second receiver/transmitter may be used to form a communication network with allHMDs18 which are calibrated by theelectronic post10. The network may be a wireless network. Similarly the connection for the first receiver/transmitter16,17 may be in a wireless network. Though the receiver/transmitter16,17 is disclosed, it may not be needed in order to calibrate and provide location information to theHMD18. In another embodiment, a wired or physical connection between aprocessor24 and theHMD18 is provided. Location and calibration information is communicated directly from theprocessor24 to theHMD18. As a non-limiting example, the wired connection may be a direct wired Ethernet RJ45 connection. The wired connection may be galvanic. In another embodiment, the connection may be an optical connection. Also shown inFIG. 1 is adata entry port67. Further detail regarding thedata entry port67 is provided during a discussion ofFIG. 6.
• Theelectronic post10 may include a plurality of attachment elements ordocking elements22 to which theHMD18 may be connected, docked, or secured. Though only onedocking element22 is illustrated, multiple docking elements may be provided where the docking elements may be provided for different types ofHMDs18. Thedocking element22 may also provide a connection through which location and/or orientation information may be transmitted to theHMD18, with accuracy benefiting theHMD18 from being at an exact same location that is associated with the information being obtained with respect to theelectronic post10. Once the communication of information from theelectronic post10 to theHMD18 is complete, theHMD18 will have an accurate initialization from which to base its subsequent movement calculations which in turn will assist with ensuring the correct position and orientation in both a virtual environment and an actual environment. More specifically, by securing theHMD18 to a particular known location based on location coordinates which are communicated to theHMD18, theHMD18 is able to be aligned to the same location as theelectronic post10 as well as pointing in a known direction as identified by theelectronic post10.
• Theelectronic post10 may also provide local magnetic declination information to theHMD18. Magnetic declination varies both from place to place and with the passage of time. By way of non-limiting example, in the United States, the magnetic declination varies from 20 degrees west in Maine to 0 degrees in Florida, to 10 degrees east in Texas. This means that a compass adjusted at the beginning of a journey to each of these states would have a true north error of over 30 degrees if not adjusted for the changing magnetic declination. Theelectronic post10 may be used to program an identified magnetic declination into theHMD18. This may be accomplished with a table or database which contains declination adjustment values based on locations. Based on theelectronic post10 determining its location, a declination adjustment value may also be ascertained and provided to amagnetometer28 of theHMD18. Themagnetometer28 of theHMD18 may be a self-calibrating magnetometer, based on the magnetic flux and geometrical position of a sensor in theHMD18, which may determine an approximation of a direction of true north. In another embodiment, theelectronic post10 has amagnetometer26 which is first adjusted based on the magnetic declination information, and the setting from thismagnetometer26 is communicated to theHMD18 to adjust itsmagnetometer28.
• In an embodiment, themagnetometer26 may be optional in the electronic post. However, the electronic post should be provided with a current magnetic declination for the location of the post and the date. Magnetic declination may slowly change over years. The magnetic declination information may be communicated to the HMD for use in internally calibrating the HMD's magnetometer. Amagnetometer26 in theelectronic post10 may assist in calibrating the post in the case of a moveable post being set up in the field.
• Thedocking element22 may he motorized to twist, rotate, or translate the HMD in certain directions, or angles, at certain rates of accelerations. In essence, movement of theHMD18 is performed to adjust sensitivity to motion of theHMD18. Because of such movement, thedocking element22 is rigidly connected to theHMD18. A rigid connection is also provided where milliradian and millimeter resolution in geophysical alignment may he realized. The twisting, rotating, and translating using thedocking element22 does not perform its functions with respect to a fixed location or orientation, but instead operates to address dynamic information such as, but not limited to, rates of change. In a non-limiting example, this is done to ensure that if theHMD18 twists at X degrees/second, the INS may report an accurate X degrees/second and then theHMD processor224 may use the information to move the portrayed virtual environment by X degrees/second with respect to the real world viewable through theHMD18.
• These movements may be provided to calibrate anaccelerometer34 and/or a gyro rate tracker, or gyroscope,36 of theINS38 which is a part of theHMD18 since accelerometers and gyroscopes in particular may drift over time. With respect to thegyroscope36, thedocking element22 would turn theHMD18 at a known constant or acceleration rate to ensure that thegyroscope36 of theHMD18 correctly reflects that rate. if thegyroscope36 does not reflect the correct rate, information from theelectronic post10 may be provided to theHMD18 to correct thegyroscope36. in an embodiment, if thedocking element22 turns theHMD18 twenty-five (25) degrees, this information is processed by theprocessor24. If thegyroscope36 reports to theprocessor224 or24 that it was turned thirty-two (32) degrees, then theprocessor224 or24 will conclude that thegyroscope36 requires calibration, and will provide information to theHMD18 to complete the calibration.
• Location is determined by integrating accelerometer information, with information about a base starting point, such as location of theelectronic post10. Theelectronic post10 coordinates theHMD18 with respect to a certain coordinate system. Such a coordinate system used may be the geocentric coordinate system. The geocentric coordinate system is a three-dimensional, earth-centered. reference system in which locations are identified by their x-, y-, and z-values. The x-axis is in the equatorial plane and intersects the prime meridian (Greenwich). The y-axis is also in the equatorial plane; it lies at right angles to the x-axis and intersects the 90-degree meridian. The z-axis coincides with the polar axis and is positive toward the North Pole. The origin is located at the center of the sphere or spheroid. Theelectronic post10 is then able to determine whether theHMD18 requires calibration along the x-axis, y-axis, z-axis, as well as angles of rotation in three dimensions about the center of thegyroscope36 in the HMD, known as roll, pitch and yaw.
• In an embodiment, a rotation calibration process for the HMD may be accomplished. Thus, thedocking element22 may comprise arotational motion unit300 as shown inFIG. 12A with a HMDrotational attachment point310 for attaching or docking of an HMD.FIG. 12B illustrates an HMD attached to therotational motion unit300. In an embodiment, the HMDrotational attachment point310 is a cradle for seating an augmented reality device. The HMDrotational attachment point310 is attached and/or secured to a rotationalfixture securing device340 configured to rotate therotational attachment point310 bymotor330. When theHMD18 is docked in the HMDrotational attachment point310, theHMD18 is aligned with the location and orientation of the real world and the virtual world. The attachment slugs320 on the HMDrotation attachment point310 mate with matching locations on theHMD18 to hold theHMD18 securely in place when docked. Themotor330 attached to the rotationalfixture securing device340 is commanded by themotor controller350 to twist the axis of therotational attachment point310 that is currently attached to the rotationalfixture securing device340.
• The HMDrotational attachment point310 may be removed from the rotational fixture securing device. In an embodiment, the HMDrotation attachment point310 includes a plurality ofaxis connectors360,370 and380, each axis connector corresponding to a different axis of rotation. In the illustration,axis connector360 when connected to the rotationalfixture securing device340 would rotate the HMD and the HMD rotation attachment point301 in afirst direction360aor the reverse ofdirection360a. When theaxis connector370 is connected to the rotationalfixture securing device340, the HMD and the HMD rotation attachment point301 may be rotated in the asecond direction370aor the reverse ofdirection370a. When theaxis connector380 is connected to the rotationalfixture securing device340, the HMD and the HMD rotation attachment point301 may be rotated in the athird direction380aor the reverse ofdirection380a.
• In view of the above, rotational information associated with theHMD18 may be calibrated about three axes.
• In an embodiment, the rotation calibration process may cause rotations about a plurality of central locations or axes in relation to the HMD. During the rotation calibration process, waveforms may be recorded. The waveforms may be compared to the waveforms generated during calibration accordingly. In an embodiment, the rotation calibration process may use thegyroscope36. The rotation calibration process may include turning or rotating thedocking element22 holding theHMD18 at various angles where instantaneous rotational velocity and intermediate positions, at pre-defined stops, are recorded and then used to calibrate theHMD18 with actual physical rotations. In an embodiment, thedocking element22 may be a rotational motion unit, or at least comprise a rotational motion unit which may provide for thedocking element22 to operate as described.
• In another embodiment, the location of theHMD18 may be updated periodically by the user by performing an optical sighting and triangulation of fiducial marks in the real environment.
• In another embodiment, thedocking element22 may also move theHMD18 up and down, front to back, and/or side to side. Doing so will allow theelectronic post10 to provide for a static calibration mode (determining actual location and orientation of the HMD18), a steady velocity mode, and a steady acceleration, where the steady velocity mode and the steady acceleration modes may be performed for both translation (change in position as opposed to orientation) and rotation of the HMD, thus calibrating theaccelerometer34.
• Alinear translation unit40 shown inFIG. 1 may provide a straight-line motion up and down to thedocking element22 and, thus, the HMD docked in thedocking element22. Thedocking element22 may be re-inserted to get three different axes of linear motion, as described in relation toFIGS. 12A and 12B. Thelinear translation unit40 may comprise an electronic motor, a piston, a damped solenoid or a spring-loaded shaft without a motor. The spring-loaded shaft may allow the HMD to bounce or move up and down. Amotion detection unit45 may be mounted on thedocking element22. Themotion detection unit45 may be pre-calibrated to respond accurately to the HMD motion in translation and rotation, and may serve to generate the calibratedwaveform520Wave 1 ofFIG. 5.
• In an embodiment, thedocking element22 may be configured to spin in three different axis orientations automatically.
• If theaccelerometer34 orgyroscope36 cannot be aligned, either a user interface orindicator element30 on theelectronic post10 or a visual feedback element provided within theHMD18 will notify the user of this fact, and/or a status of the calibration. Possible reasons why theaccelerometer34 may not calibrate include, but are not limited to, lower power, component damage, etc. TheHMD18 may also comprise anindicator230, such as on an outer surface of aframe72 of the HMD18 (illustrated inFIGS. 7 and 9). Theelectronic post10 may also include anindicator30. Notification by theindicator230 may be accomplished audibly and/or visually. When audible, a sound may emit from a speaker during the calibration immediately when done. When visual, a text message may be displayed or another visual indication is provided within theHMD18 regarding a state of calibration. Additionally, a printout of an error may be provided in another embodiment.
• FIG. 2 shows the location post in use in an operational environment. As illustrated the electronic (location) post10 is attached to a roof of avehicle44. A global positioning system (“GPS”)satellite20 is also disclosed. When theelectronic post10 is ready to be used, such as with the vehicle stationary or theelectronic post10 secured directly to the ground, thereceiver16 may collect information to establish an accurate location and/or orientation reading with respect to the Earth or some other coordinate system. Such accuracy is useful because when theHMD18 is attached to theelectronic post10, the location of theHMD18 is in a position and orientation that is directly traceable to the location of theelectronic post10. This then allows for calibration with nearly as much accuracy as possible, such as within millimeters, as the orientation ofelectronic post10 is known. The information collected may be based on information received through thereceiver16 from theGPS20. Theelectronic post10 may also be calibrated using an optical approach by relying on known or pre-established visual landmarks or Real Fiducial Marker (RFM)48 to triangulate the location of theelectronic post10, as is explained in further detail below. Once the location information is collected by theelectronic post10, the location information is either processed by theelectronic post10 to update calibration charts with respect to location or the location information is communicated directly to theHMD18, which in turn applies the location information to perform its own updates.
• In another embodiment, three-dimensional orthogonal implementation of differential GPS may be used to obtain orientation and location information to an accurate degree. Though not disclosed herein, those skilled in the art will readily recognize that other techniques of obtaining such location information may be used wherein thereceiver16 is configured to receive such information based on the technique used. In an embodiment, another technique may comprise the use of GPS while in view of its satellite(s), or another technique which allows for measuring orientation and location with respect to the Earth. Using differential GPS information may result in the location being known within ten (10) centimeters.
• FIG. 3 shows an advanced electronic post with a horizontal linearmotion calibration system210. This is another potential way to induce the linear translation besides that of thelinear translation unit40. The advanced electronic post may be an INS-calibrating electronic post. The horizontal linearmotion calibration system210 may be used to further calibrate theaccelerometer34 of theHMD18. Though a vertical or near vertical linear motion calibration system may be utilized, the horizontal linear motion calibration system removes or minimizes effects associated with gravity during a calibration process. The advancedelectronic post270 may have anextendable arm230 which transitions, or extends, from a vertical position (not shown) where it is parallel with the advancedelectronic post270 to a horizontal position, as shown. Ahinge240 is illustrated as providing a connection point to allow the extendable arm to be transitioned from its vertical position to its horizontal position. Though not necessary required, asupport rod260 is provided to ensure stability of theextendable arm230 when in the horizontal position. Aleveling device250 may be used to determine that the extendable arm is at an acceptable horizontal position. The leveling device may be aspirit level250, as shown inFIG. 3. Thetranslation unit40 is provided to move an attachedHMD18 sideways. An attachment point or dockingelement22 is provided to connect theHMD18 to the translation unit. In an embodiment, thetranslation unit40 may comprise abelt drive system220 to move theHMD18 in the sideways motion. The belt drive could be replaced by any suitable linear motion device. Amotor42 is also provided to actuate components of the translation unit where theattachment point22 may moves in a sideways motion.
• Based on the above, movement-based dynamic calibrations are all relative where they define a first or second derivative of position and/or orientation. The stationary measurement, specifically location and orientation information, is used to define the actual position and orientation.
• TheHMD18 may be used in a multi-user augmented reality world. When used in such an instance, Distributed Interactive Simulation (“DIS”) dead reckoning information, or parameters, may be used from theHMD18 to provide accurate location and directional information to other entities in the multi-user augmented reality world. More specifically, the calibration of theHMD18 output may be extended to include determining the accuracy of translational and rotational dead-reckoning parameters and the position parameters that theHMD18 may transmit to describe its position and trajectory. Then other entities in the simulation may see the location and direction of heading of theHMD18, in order to re-create a view of the ongoing simulation, but at a remote location. For a given acceleration and velocity, a certain numerical value may be placed into a “DIS packet” for data transmission across a network to other simulation entities. The calibration aspect of theelectronic post10 can ensure that these values are correct for a given acceleration, position, and velocity. This may include various derivatives of orientation and position, where theHMD18 provides an as-close-as-possible-to-ground-truth network transmission packet to the other entities for incorporation in a distributed Live-Virtual-Constructive exercise.
• FIG. 4 shows a diagram illustrating a calibration technique utilizing a Real Fiducial Marker (RFM) and a Virtual Fiducial Marker (VFM). In one embodiment, theelectronic post10 may comprise a visual detection device52 (illustrated inFIG. 6) capable of detecting images in both a real environment and the virtual environment. With theHMD18 secured to theelectronic post10 by thedocking element22, theHMD18 is positioned where a viewpoint of the lenses74 (as illustrated inFIG. 7) on theHMD18 is directed towards theRFM48. Thevisual detection device52 detects both theRFM48 and theVFM50. As a non-limiting example, if theRFM48 andVFM50 are not aligned, or do not appear as being overlapped, theelectronic post10 will adjust aspects (explained in further detail below) of theHMD18 until alignment is achieved. The distance to themarker48 is also a known distance. TheRFM48 has a known height and distance from theelectronic post10. Though theRFM48 is illustrated as some form of a stand inFIG. 2, theRFM48 has a plurality of other configurations. TheRFM48 may simply be a mark on an object (such as a wall), a mark on the ground, a natural object in an area (such as a rock), etc. Though not required, theRFM48 may be specially marked, such as with a particular color. The intent is that theRFM48 is distinguishable from objects visible in the real environment. TheRFM48 is used to align aVFM50 with theRFM48. In another embodiment, the user would wear theHMD18 and may then perform the calibration of theVFM50 to theRFM48 by way of an adjuster which is a part of theHMD18 or through an electronic or computer-based interface. The adjuster, or a variation, could also actuate adjustment interfaces, ordisplacement elements89 and189 (as further illustrated inFIG. 7 andFIG. 8), to adjust aspects of theHMD18 until acceptable alignment from the user's perspective is achieved. Generally, adjustments to the virtual world to overlap with the real world may be performed electronically using the3DIGS77. Physical adjustments to the HMD may be needed by the user to properly situate their gaze in the HMD system.
• FIG. 5 shows a diagram illustrating waveforms. These waveforms may correspond to a measured output of theINS38 when theHMD18 is moved in a controlled manner. Peak accelerations may be displayed in these types of waveforms such as, but not limited to, an output value corresponding to acceleration or velocity when theHMD18 is physically moved in a controlled manner that is supposed to produce an expected acceleration and/or velocity when moved in a periodic, sinusoidal, back-and-forth manner (or motion), or twisting manner for rotational calibration.
• As illustrated, theelectronic post10 may have already been provided or calculated an expected resultant, or waveform,Wave 1520. This wave can also be read off themotion detection unit45 as the HMD is moved during calibration. When theHMD18 is secured to theelectronic post10 for calibration, theHMD18 is moved certain directions by the calibration motors of theelectronic post10, and a new or actual measurement is made,Wave 2530. Because of the amplitudes of the new wave,Wave 2530, this wave may differ fromWave 1520 as a result of any number of factors such as, but not limited to, a low battery, malfunctioning components, a need for calibration, etc. Once the basis for the difference is addressed such as, but not limited to, calibration being performed, another measurement may be made to determine ifWave 2530 is within an acceptable range, or tolerance ofWave 1520. If calibration does not changeWave 2530 or fails to bringWave 2530 within tolerance ofWave 1520, theindicator230 on either theHMD18 or theindicator30 on theelectronic post10 or some other notification means may notify the user that calibration was not successful, or that another issue is affecting theHMD18. The calibration may require a factor, or some other adjustment, to be applied to an aspect of theHMD18 whereWave 2530 is within an acceptable range, or tolerance. A goal of the calibration is to make the measured waveforms align with the expected waveforms for the given motion. The waveforms correspond to the expected and actual output of the HMD's INS when it is moved in a controlled manner.
• FIG. 7 shows a side view representation of a head-mounted device (HMD) andFIG. 8 shows a front view representation of the head-mounted device (HMD). Though two particular HMD devices are disclosed herein in detail, theelectronic post10 may be applicable for use with a variety of HMD devices and other virtual reality devices. In one embodiment, theHMD18 may be an optical see-through, augmented reality, binocular viewer. Because an optical see-through, augmented reality, binocular viewer is typically the most complex form of theHMD18, explanation of theHMD18 herein will primarily discuss embodiments of this type. In these discussions, it is understood that the principles discussed herein are equally applicable to optical see-through systems, augmented reality systems, monocular viewer systems, video see-through systems, augmented reality systems, binocular systems, biocular systems, and monocular viewer systems, wherein each system is an “augmented reality system.”
• As shown inFIGS. 7 and 8, theHMD18 may further comprise aframe72 adapted to be worn by the user and supported by the user's nose and ears in a manner similar to that in which eyeglasses are worn. TheHMD18 may have a variety of configurations and can, in various embodiments, resemble conventional goggles, glasses, helmets, and the like. In sonic embodiments, a strap may be used to hold the HMD'sframe72 in a fixed position with respect to the eyes of the user. In general terms, the outside surface of theHMD18 may assume any form that holds the optical system in the required orientation with respect to the HMD's display(s) and the user's eyes.
• The HMD may comprise at least oneimage display system76 and at least oneoptical system78 that further comprises a reflectiveoptical surface80. Theoptical surface80 may be a free space, ultra-wide angle, reflective optical surface (“FS/UWA/RO surface”), which by necessity is curved. In some embodiments, the FS/UWA/RO surface80 may be the entireoptical system78. Thesurface80 may be purely reflective or may have both reflective and transmissive properties, in which case, it can be thought of as a type of “beam splitter.”
• Thesurface80 may completely surround one or both eyes, as well as the at least oneimage display system76. In particular, thesurface80 may curve around the sides of the eyes and toward the sides of the face so as to expand the available horizontal FOV. In one embodiment, the FS/UWA/RO surface80 may extend up to 180 degrees or more (e.g., more than 200 degrees). As best illustrated inFIG. 8, theHMD18 may include two separate FS/UWA/RO surfaces80 for the user's two eyes which are separately supported by theframe72 and/or anasal ridge piece82 of theframe72. Alternately, theHMD18 may employ a single FS/UWA/RO surface80 that serves both eyes with a single structure, some portions of which are viewed by both eyes and other portions of which are viewed by only one eye.
• Anoptional lens system83 is also provided having at least onelens74, such as without limitation, a Fresnel Lens. Thelens system83 is supported by theframe72 and is located between theimage display system76 and the reflectiveoptical surface80. Thelens system83 may be provided to modify diopter characteristics of the light emanating from a surface of thedisplay system76.
• Thenasal ridge piece82 of theframe72 may be a vertical bar or wall which provides a separation between two FS/UWA/RO surfaces80, one for each of the user's eye. Thenasal ridge piece82 may also provide a separation between the fields of view of the user's two eyes. In this way, the user's right eye can be shown a first representation of three-dimensional physical reality in the environment by displaying a first image to the right eye via a first image display device and a first FS/UWA/RO surface80R, while the user's left eye is shown a second representation of three-dimensional physical reality in the environment by displaying a second image to the left eye via a second image display device and a second FS/UWA/RO surface80L. A separate display device/reflective surface combination thus services each eye of the user, with each eye seeing the correct image for its location relative to the three-dimensional physical reality in the environment. By separating the user's two eyes, theridge piece82 of theframe72 allows the image applied to each eye to be optimized independently of the other eye. In one embodiment, the nasal ridge piece's vertical wall may include two reflectors, one on each side, to allow the user to see imagery as he/she turns his/her eyes nasally, either to the left or to the right.
• Theframe72 may further comprise at least oneadjuster87 to modify a physical position of a component of theframe72. The physical position of a component of theframe72 may be changed or be adjusted based on inter-pupillary information known about the user or a known height of the user (relative to the height of the eyes from the ground). This may be done to ensure that theHMD18 is in accordance with a viewpoint of the user. The adjustment may occur at a joint, extension, or hinged element, ordisplacement element89, located at specific locations to provide a desired adjustment. In an embodiment, depending on the inter-pupillary distance apart and depth of eyes in thehead70 of the user, thenasal ridge piece82 may require adjusting to allow for the HMD to best accommodate a particular user. Theadjuster87 may be activated to expand or contract a width of the frames measured from side to side of the user, where the frame has a section, thedisplacement element89, which extends along an inner track on the nasal ridge piece.
• In another embodiment, theadjuster87 may be activated to change the spacing between the lenses on the face of the user based on inter-pupillary information about the user. The adjuster may also be configured to allow for manual adjustment. Adjustment may be accomplished with thedisplacement element89 having hinged tendencies also associated with thenasal ridge piece82. The intent of both of these adjustments is to bring the binocular views closer together or further apart. By doing so, this action moves where virtual images are supposed to point in the outside world as viewed by the user with the user's unique view. Other approaches may include, but not limited to, providing displacement element(s) to move placement of theimage display system76, angling of the reflectiveoptical surface80, and/or positioning of the lens system83 (more particularly at least onelens74 of the system as both are illustrated inFIG. 9). The adjustments may also be made electronically in the HMD by moving the imagery left, right, up, or down to accommodate the locations of the user's eyes in their head.
• Theadjuster87, or multiple adjusters, may also be provided to adjust theimage display system76 with respect to theframe72, the reflectiveoptical surface80 with respect to theframe72, and/or the lens system83 (or lens74) with respect to theframe72. Theadjuster87 may be an analog adjustment knob, digital adjustment knob, and/or a combination of these means to adjust theHMD18. Theadjuster87 is also ridged enough where, when not being used, it will not slip or inadvertently make an unwanted adjustment. In one embodiment, alocking mechanism91 is provided to prohibit such errors from happening.
• Theprocessor224, which may be a part of an electronics package, is attached to theframe72 of theHMD18, and may also be used to operate theadjuster87. In one embodiment, thedocking element22 is configured to secure theHMD18 at attachment points that provide for communication with theadjuster87 and allows for free movement of the parts, thedisplacement element89 or189 of theHMD18 which may be moved during an adjustment. When docked, a particular user's biometric information that resides within theelectronic post10 or on a remote networked database may be accessed and communicated to theprocessor224 ofHMD18. TheHMD processor24 may then actuate theadjuster87 to configure theHMD18 to best accommodate the particular user. Theadjuster87 may also be manually operated. Thus, instead of, or in combination with, relying on theelectronic post10 to make adjustments, the user may wear theHMD18 and may perform a manual adjustment.
• By having manual control, the user may perform some calibration techniques manually. In an embodiment, with respect to the calibration disclosed above with respect toFIG. 4 regarding the use of theRFM48 and theVFM50, this calibration may be completed manually. Wearing theHMD18, the user can see theRFM48. With theVFM50 also virtually displayed, the user may utilize theadjuster87 to align (such as by overlapping) theRFM48 andVFM50 by moving theVFM50 to overlap theRFM48. Once manually calibrated, thelocking mechanism91 is used to prevent theadjuster87 from inadvertently making a change to the calibration. This calibration will more likely be performed electronically as the user moves the location of the VFM, visible to them on the HMD screen, to overlay the RFM in the outside world. The user can control the electronic location of the VFM through computer interfaces including knobs, keyboards, and computer mouse devices.
• Additionally, the height of the user, more specifically the distance of the user's eyes from the ground, may be communicated to theHMD18 to adjust theHMD18 to further assist in generating the virtual image in theHMD18 at a correct location, such as by assisting in establishing a viewpoint that the user looks through into the virtual reality world.
• FIG. 9 shows a block diagram illustrating elements that may be a part of an augmented reality device. TheHMD18 may comprise aprocessor224, astorage device65,display system76, the reflectiveoptical surface80, theINS38 which may comprise theaccelerometer34 and thegyroscope36, anindicator230, acalibration module93, and themagnetometer26. Depending on an embodiment of theHMD18, atleast lens74 orlens system83 may or may not be provided. Though many of these components may not be supported by theframe72, at least theoptical surface80 and orlens system83, at least oneaccelerometer34 and at least onegyroscope36 are supported by theframe72. With respect to at least theaccelerometer34 and the at least onegyroscope36, as these are what are being calibrated, with respect to theframe72, as the frame moves these components are calibrated correctly to electronically indicate the location, motion, and orientation of theframe72. When theHMD18 is in use, these components will move with respect to movement of the frame, thus continuing to indicate the location, motion, and orientation of theframe72.
• Also, a part of theframe72 may comprise theadjuster87,locking mechanism91, anddisplacement element89. Thecalibration module93 may perform geo-location calibration of theHMD18. It may also be configured to perform orientation calibration of theHMD18. Thecalibration module93 also may perform dynamic INS calibration. Thecalibration module93 may also perform a calibration based on a user's biometric data. Furthermore, the embodiment disclosed with respect toFIG. 4 may also be accomplished likely with the user manually performing the calibration to ensure that the final calibration is specific to the user. Thecalibration module93 may also be capable of adjusting aspects of theHMD18, such as angle or placement oflens74 on theHMD18, by techniques disclosed herein. Thecalibration module93 may be a part of theprocessor224 of theHMD18, or a separate processor.
• Based on the embodiments disclosed herein and with further reference toFIG. 1, a system is provided which may include theHMD18 and theelectronic post10, which may also be referred to as a portable calibration unit. The term “portable” is used because as explained, theelectronic post10 is configured to be placed wherever the user desires to set up theelectronic post10. TheHMD18 has aframe72 andaugmented reality components76,80,83 which are supported by theframe72. The augmented reality components include theimage display system76, the reflectiveoptical surface80, and/or thelens system83, all of which are disclosed above. TheHMD18 may have at least onedisplacement element89 and/or189 as described above. Various types of calibration which may be performed on the HMD when connected to theelectronic post10 may be generally considered as geo-location calibration, orientation calibration, dynamic inertial navigation system calibration, and biometric-based calibration. Theoptional displacement element89 or189 may change a physical placement of theframe72 and/or at least one of theaugmented reality components76,80,83 with respect to theframe72 based on a calibration result, most specifically the biometric-based calibration. The system may further comprise thelocking mechanism91 to prevent theoptional adjuster87 from performing an unwanted adjustment.
• FIG. 10 shows a flowchart illustrating a method of an embodiment. Theflowchart100 illustrating themethod102 comprises attaching an electronic calibration and/or location post to a surface, at104. The surface may be within a field of operation. The method may further comprise connecting an augmented reality device to a docking element of the location post, at108. The location post may be a passive post, an electronic post, or an advanced electronic post. The method may also comprise calibrating the augmented reality device to align a virtual environment with the real world environment of the augmented reality device with respect to a current location and/or orientation of the post, at110. The method may also comprise obtaining information about a location and/or orientation specific to a current location and/or orientation of the post, at106. The information may be used during calibrating.
• The calibrating, at110 may further comprise adjusting location information and orientation data of the augmented reality device with respect to a location and orientation of the post, at112. Additionally, calibrating, at110, may further comprise adjusting at least one structural component of the augmented reality device in response to calibrating. The above steps of themethod100 may be performed in the order shown, a different order or one or more steps may be performed contemporaneously.
• FIG. 11 shows a flowchart illustrating another method of an embodiment. Theflowchart120 illustrating themethod122 may comprise attaching an electronic calibration and/or location post to a surface, at124. Themethod122 may further comprise connecting an augmented reality device to a docking element which is a part of the post and facilitates controlled movement of the augmented reality device in a plurality of directions responsive when at least one aspect of a calibration is performed, at126. Themethod122 may further comprise obtaining information for use during calibrating, the information collected comprising dynamic data, at128.
• The calibrating, at130, may further comprise moving the augmented reality device while secured to the docking element in defined directions and/or at defined rates and accelerations about a three-dimensional coordinate system and/or rotating angularly with respect to each axis of the three-dimensional coordinate system as needed to determine a measurement of a same kind as the collected information, comparing the measurement to the collected information of the same kind to determine an alignment factor for each particular capability being measured, and specifically calibrating at least one component of the augmented reality device with the alignment factor to produce measurements within a tolerance to the collected information. The calibrating, at130 may also comprise adjusting motion detection sensitivity in the augmented reality device.
• Persons skilled in the art will recognize that an apparatus, such as a data processing system, including a CPU, memory, I/O, program storage, a connecting bus, and other appropriate components, could be programmed or otherwise designed to facilitate the practice of embodiments of the method. Such a system may include appropriate program. means for executing the method. Also, an article of manufacture, such as a pre-recorded disk, computer readable media, or other similar computer program product, for use with a data processing system, may include a storage medium and program means recorded thereon for directing the data processing system to facilitate the practice of the method.
• Embodiments may also be described in the general context of computer-executable instructions, such as program modules, being executed by any device such as, but not limited to, a computer, designed to accept data, perform prescribed mathematical and/or logical operations usually at high speed, where results of such operations may or may not be displayed. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. In an embodiment, the software programs that underlie embodiments can be coded in different programming languages, for use with different devices, or platforms. It will be appreciated, however, that the principles that underlie the embodiments can be implemented with other types of computer software technologies.
• Moreover, those skilled in the art will appreciate that the embodiments may be practiced with other computer system configurations, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. Embodiments may also be practiced in distributed computing environments where tasks are performed by processing devices located at different locations on board of a vehicle or stationary device, that are linked through at least one communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media which may include memory storage devices.
• In view of the above, a non-transitory processor readable storage medium is provided. The storage medium may comprise an executable computer program product which further comprises a computer software code that, when executed on aprocessor24, causes the processor to initiate acquisition of information comprising a current location and/or orientation of an electronic calibration and/or location post, detect when an augmented reality device is attached to the post, and calibrate the augmented reality device to align a virtual environment with a real world environment of the augmented reality device with respect to the current location and/or orientation of the post.
• Thus, based on the embodiments disclosed herein, users and manufacturers of HMDs may be provided with a portable infrastructure support system, namely theelectronic post10 described above, to allow for greater accuracy of real-world and virtual images viewable through theHMD18 when theHMD18 is used operationally, more specifically within a field of operation. Theelectronic post10 may be portable enough to be mounted, either permanently or temporarily on a vehicle or inserted into the ground, or it may be permanently installed in a building or foundation intended to be used for many years. Theelectronic post10 may be adapted to align multiple HMDs18 (especially since even if a same brand eachHMD18 may have unique characteristics requiring its own calibration and/or alignment) while also integrating biometric information about respective users into the HMD to provide for tailored alignment specific to a particular user. Information residing in on theHMD18 may also be downloaded and stored on theelectronic post10 for later retrieval, or for rapid, real-time, delivery to a remote location, and information on the post may be transmitted into the HMD. Theelectronic post10 also provides for an alignment approach which may be automated.
• The motor-driven translational and rotational calibration may be performed without the need to include them on a location post. They could be completely separate, in a separate facility.
• While embodiments have been described with reference to various embodiments, it will be understood by those skilled in the art that various changes, omissions and/or additions may be made and equivalents may be substituted for elements thereof without departing from the spirit and scope of the embodiments. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the embodiments without departing from the scope thereof. Therefore, it is intended that the embodiments not be limited to the particular embodiment disclosed as the best mode contemplated, but that all embodiments falling within the scope of the appended claims are considered. Moreover, unless specifically stated, any use of the terms first, second, etc., does not denote any order or importance, but rather the terms first, second, etc., are used to distinguish one element from another.

Claims (20)

What is claimed is:

1. A system comprising:

a structure attachable to a surface in a real world environment, the structure establishing a known location and orientation of the structure;

a docking element, as part of the structure, to secure an augmented reality device in a stationary position for alignment of the augmented reality device with the real world environment and with a parallel virtual environment; and

a processor operable to perform the alignment by resetting the inertial navigation system of the augmented reality device to the known location when docked in the docking element and aligning the location and orientation of the virtual representation of the augmented reality device in the parallel virtual environment so that the parallel virtual environment in the augmented reality device overlaps the real world environment.

2. The system according toclaim 1, wherein the processor has access to calibration data or data specific to a location and orientation of the structure at a time of the calibration of the augmented reality device.

3. The system according toclaim 1, further comprising a transceiver to communicate information from a remote location for use by the processor.

4. The system according toclaim 1, further comprising a transmitter to communicate information used by the processor to a remote location.

5. The system according toclaim 1, wherein information specific to a user of the augmented reality device is communicated to the augmented reality device during calibration.

6. The system according toclaim 1, wherein magnetic declination information specific to a location of the structure is communicated to the augmented reality device during calibration.

7. The system according toclaim 1, wherein the augmented reality device comprises a head-mounted device and wherein the calibration is used to adjust at least one structural component of the head-mounted device.

8. The system according toclaim 1, wherein the processor is a part of the augmented reality device and information about the current location and orientation of the structure is communicated from a database remote from the structure.

9. The system according toclaim 8, wherein the structure further comprises an identification code that is associated with the current location and orientation of the structure maintained at the database for use by the augmented reality device when retrieving the information from the database specific to the structure.

10. A method comprising the steps of

establishing a known location and orientation of a calibration location post having a docking element;

providing an augmented reality device with the known location and orientation when docked in the docking element; and

calibrating the augmented reality device to align a virtual environment with the real world environment of the augmented reality device with respect to a current location and orientation of the post.

11. The method according toclaim 10, further comprising obtaining information for use during calibrating wherein the information obtained comprises location and orientation data specific to the current location and orientation of the post.

12. The method according toclaim 10, wherein calibrating further comprises adjusting location information and orientation data of the augmented reality device with respect to a location and orientation of the post.

13. The method according toclaim 10, further comprising adjusting at least one structural component of the augmented reality device in response to calibrating.

14. The method according toclaim 10, further comprising adjusting at least one electronic setting of the augmented reality device in response to calibrating.

15. The method according toclaim 10, wherein calibrating further comprises:

moving the augmented reality device while secured to the docking element in defined directions and at defined rates about a three-dimensional coordinate system or rotating angularly with respect to each axis of the three-dimensional coordinate system as required to determine a measurement of a same kind as the collected information;

comparing the measurement to the collected information of the same kind to determine an alignment factor for each particular existing measurement; and

specifically calibrating at least one component of the augmented reality device with the alignment factor to provide the measurement within a tolerance to the collected information.

16. The method according toclaim 10, Wherein calibrating further comprises adjusting motion detection sensitivity in the augmented reality device, or wherein calibrating further comprises adjusting rotation detection sensitivity in the augmented reality device.

17. A non-transitory processor readable storage medium, providing an executable computer program product, the executable computer program product comprising a computer software code that, when executed on a processor, causes the processor to:

initiate acquisition of information comprising a current location and/or orientation of an electronic calibration and/or location post;

detect when an augmented reality device is attached to the post; and

calibrate the augmented reality device to align a virtual environment with a real world environment of the augmented reality device with respect to a current location and/or orientation. of the post.

18. The processor readable storage medium according to claim21, wherein causing the processor to calibrate further comprises the processor to adjust a location and orientation of the augmented reality device with respect to a location or orientation of the post.

19. The processor readable storage medium according to claim21, wherein causing the processor to calibrate further comprises causing the processor to adjust at least one structural component of the augmented reality device in response to calibrating.

20. The processor readable storage medium according to claim21, wherein causing the processor to calibrate further comprises the processor to adjust at least one electronic setting of the augmented reality device in response to calibrating.

Images