US8786953B2 - User interface - Google Patents

Abstract

A head-mounted display (HMD) may include an eye-tracking system, an HMD-tracking system and a display configured to display virtual images. The virtual images may present an augmented reality to a wearer of the HMD and the virtual images may adjust dynamically based on HMD-tracking data. However, position and orientation sensor errors may introduce drift into the displayed virtual images. By incorporating eye-tracking data, the drift of virtual images may be reduced. In one embodiment, the eye-tracking data could be used to determine a gaze axis and a target object in the displayed virtual images. The HMD may then move the target object towards a central axis. The HMD may also record data based on the gaze axis, central axis and target object to determine a user interface preference. The user interface preference could be used to adjust similar interactions with the HMD.

Description

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of U.S. patent application Ser. No. 13/302,916, filed on Nov. 22, 2011 now U.S. Pat. No. 8,611,015 B2, which is herein incorporated by reference as if fully set forth in this description.

BACKGROUND

Wearable systems can integrate various elements, such as miniaturized computers, input devices, sensors, detectors, image displays, wireless communication devices as well as image and audio processors, into a device that can be worn by a user. Such devices provide a mobile and lightweight solution to communicating, computing and interacting with one's environment. With the advance of technologies associated with wearable systems and miniaturized optical elements, it has become possible to consider wearable compact optical displays that augment the wearer's experience of the real world.

By placing an image display element close to the wearer's eye(s), an artificial image can be made to overlay the wearer's view of the real world. Such image display elements are incorporated into systems also referred to as “near-eye displays”, “head-mounted displays” (HMDs) or “heads-up displays” (HUDs). Depending upon the size of the display element and the distance to the wearer's eye, the artificial image may fill or nearly fill the wearer's field of view.

SUMMARY

In a first aspect, a method is provided. The method includes displaying images on a display, the display having a central axis, determining a gaze axis with respect to the central axis, and determining a target object in the displayed images based on the gaze axis. The method further includes adjusting the displayed images on the display to move the target object towards the central axis.

In a second aspect, a method is provided. The method includes displaying images on a display, the display having a central axis. The method further includes determining a gaze axis with respect to the central axis, and determining a target object in the displayed images based on the gaze axis. The method further includes recording data based on the central axis, the gaze axis, the target object, and the displayed images. The method further includes adjusting the displayed images on the display based on the recorded data.

In a third aspect, a head-mounted display (HMD) is provided. The HMD includes a head-mounted support and an optical system attached to the head-mounted support. The optical system includes a display having a central axis and the display is configured to display images that are viewable from a viewing location. The HMD further includes an infrared light source configured to illuminate the viewing location with infrared light such that infrared light is reflected from the viewing location as reflected infrared light. The HMD further includes a camera configured to image the viewing location by collecting the reflected infrared light and a sensor configured to generate sensor data that relates to the motion of the HMD. The HMD further includes a computer configured to determine a gaze axis based on one or more images of the viewing location obtained by the camera, control the display to display images based on the sensor data, determine a target object in the displayed images based on the gaze axis, and control the display to move the target object towards the central axis.

In a fourth aspect, a non-transitory computer readable medium having stored instructions is provided. The instructions are executable by a computing device to cause the computing device to perform functions. The functions include: (i) controlling a display to display images, the display having a central axis; (ii) determining a gaze axis with respect to the central axis; (iii) determining a target object in the displayed images based on the gaze axis; and (iv) controlling the display to adjust the displayed images so as to move the target object towards the central axis.

In a fifth aspect, a method is provided. The method includes displaying images on a display of a head-mounted display (HMD). The displayed images are viewable at a viewing location and the display includes a central axis. The method further includes acquiring sensor data related to the motion of the HMD and controlling the display to display images based on the sensor data. The method also includes determining a gaze axis based on one or more images of the viewing location obtained by a camera and determining a target object in the displayed images based on the gaze axis. The method additionally includes controlling the display to move the target object towards the central axis.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a wearable computing device, in accordance with an example embodiment.

FIG. 2 is a top view of an optical system, in accordance with an example embodiment.

FIG. 3A is a front view of a head-mounted display, in accordance with an example embodiment.

FIG. 3B is a top view of the head-mounted display ofFIG. 3A, in accordance with an example embodiment.

FIG. 3C is a side view of the head-mounted display ofFIG. 3A andFIG. 3B, in accordance with an example embodiment.

FIG. 4A is a side view of a head-mounted display with a forward gaze axis, in accordance with an example embodiment.

FIG. 4B is a side view of the head-mounted display ofFIG. 4A with an upward gaze axis, in accordance with an example embodiment.

FIG. 5 is a flowchart of a method, in accordance with an example embodiment.

FIG. 6 is a flowchart of a method, in accordance with an example embodiment.

FIG. 7A shows a field of view of an HMD, in accordance with an example embodiment.

FIG. 7B shows two fields of view of an HMD, in accordance with an example embodiment.

FIG. 8 is a flowchart of a method, in accordance with an example embodiment.

DETAILED DESCRIPTION

In the following detailed description, reference is made to the accompanying figures, which form a part thereof. In the figures, similar symbols typically identify similar components, unless context dictates otherwise. The illustrative embodiments described in the detailed description and figures are not meant to be limiting. Other embodiments may be utilized, and other changes may be made, without departing from the spirit or scope of the subject matter presented herein. It will be readily understood that the aspects of the present disclosure, as generally described herein, and illustrated in the figures, can be arranged, substituted, combined, separated, and designed in a wide variety of different configurations, all of which are contemplated herein.

1. Overview

A head-mounted display (“HMD”) may enable its wearer to observe the wearer's real-world surroundings and also view a displayed image, such as a computer-generated image or virtual image. In some cases, the displayed image may overlay a portion of the wearer's field of view of the real world. Thus, while the wearer of the HMD is going about his or her daily activities, such as walking, driving, exercising, etc., the wearer may be able to see a displayed image generated by the HMD at the same time that the wearer is looking out at his or her real-world surroundings.

The displayed image might include, for example, graphics, text, and/or video. The content of the displayed image could relate to any number of contexts, including but not limited to the wearer's current environment, an activity in which the wearer is currently engaged, the biometric status of the wearer, and any audio, video, or textual communications that have been directed to the wearer. The images displayed by the HMD may also be part of an interactive user interface. For example, the HMD could be part of a wearable computing device. Thus, the images displayed by the HMD could include menus, selection boxes, navigation icons, or other user interface features that enable the wearer to invoke functions of the wearable computing device or otherwise interact with the wearable computing device.

The images displayed by the HMD could appear anywhere in the wearer's field of view. For example, the displayed image might occur at or near the center of the wearer's field of view, or the displayed image might be confined to the top, bottom, or a corner of the wearer's field of view. Alternatively, the displayed image might be at the periphery of or entirely outside of the wearer's normal field of view. For example, the displayed image might be positioned such that it is not visible when the wearer looks straight ahead but is visible when the wearer looks in a specific direction, such as up, down, or to one side. In addition, the displayed image might overlay only a small portion of the wearer's field of view, or the displayed image might fill most or all of the wearer's field of view. The displayed image could be displayed continuously or only at certain times (e.g., only when the wearer is engaged in certain activities).

The virtual images could be displayed based on the position and orientation of the HMD. For example, the HMD may include position and orientation sensors so that when the user moves his or her head, data regarding the position and orientation of the HMD can be received by a processor. The HMD may additionally include a display controllable by the processor, so when the user moves his or her head, the processor may adjust the displayed image on the display. In particular, the displayed image may move in the opposite direction of head movement to create the sensation of looking around a world with superimposed virtual images. In other words, the virtual images could be displayed to create the illusion that the graphical images are part of the real world.

However, HMD position and orientation sensors are unable to process information infinitely fast and are limited at least by the speed of the sensor hardware. Thus, when the HMD is in motion, some sensory data may be lost. Additionally, because of imperfections in the hardware and other factors, drifts, offsets, and other errors may be introduced into the sensor data. As a result, the processor may receive erroneous sensor data. For instance, when the sensor is at rest, the data it returns may cause the sensor appear as if it is in motion, at least slightly. Additional errors can occur when the sensor is in motion.

Sensor drift problems can be corrected for by, for example, software algorithms that may round or average data to reduce sensor error. These techniques can help resolve the drift problems when the sensor is at rest. However, these algorithms may not correct the problem when the sensor is in motion.

For instance, consider a HMD wearer traveling by train who is seated facing a side wall of the train. The HMD wearer may experience both a lateral shift while the train is in motion and also a gradual rotation change while the train rounds a corner. The position and orientation sensors of the HMD may measure these movements and may cause images to be displayed inappropriately. Drift errors may not be properly controlled in this situation by rounding or data averaging.

The below described methods, non-transitory computer readable media and apparatus may serve to reduce the effect of drift errors and unintentional HMD movement by utilizing eye-tracking information. The eye-tracking information could allow the determination of a gaze axis, which could be related to a target object in the images displayed by the HMD. Subsequently, the target object could be moved towards a central axis of the HMD.

In other words, the HMD may generally use HMD motion data to adjust virtual images on the display. For instance, as the HMD user walks around the real-world environment, virtual images could be presented to the wearer based upon the orientation of the HMD. However, when the system detects a target object selection from the eye-tracking system (e.g. the HMD wearer gazes at a specific object for a sufficient period of time), the HMD may act to move the target object to the central axis. In this manner, the determination of a gaze axis could act to override the HMD motion data and move or keep the target object near the central axis. Thus, errors due to imperfect sensor data and/or unintentional HMD movement could be reduced or eliminated.

Certain illustrative examples of a system and method for correcting sensor drift based on current and recorded eye gaze information are described below. It is to be understood, however, that other embodiments are possible and are implicitly considered within the context of the following example embodiments.

2. A Head-Mounted Display Apparatus with Eye Tracking Functionality

FIG. 1 is a schematic diagram of a wearable computing device or a head-mounted display (HMD)100 that may include several different components and subsystems. Components of theHMD100 may include an eye-trackingsystem102, a HMD-trackingsystem104, anoptical system106,peripherals108, apower supply110, aprocessor112, amemory114, and auser interface115. The eye-trackingsystem102 may include hardware such as aninfrared camera116 and at least one infraredlight source118. The HMD-trackingsystem104 may include agyroscope120, a global positioning system (GPS)122, and anaccelerometer124. Theoptical system106 may include, in one embodiment, adisplay panel126, a displaylight source128, andoptics130. Theperipherals108 may include, for example, awireless communication interface134, atouchpad136, amicrophone138, acamera140, and aspeaker142.

In an example embodiment,HMD100 includes a see-through display. Thus, the wearer ofHMD100 may observe a portion of the real-world environment, i.e., in a particular field of view provided by theoptical system106. In the example embodiment,HMD100 is operable to display virtual images that are superimposed on the field of view, for example, to provide an “augmented reality” experience. Some of the virtual images displayed byHMD100 may be superimposed over particular objects in the field of view.HMD100 may also display images that appear to hover within the field of view instead of being associated with particular objects in the field of view.

Components of theHMD100 may be configured to work in an interconnected fashion with other components within or outside their respective systems. For instance, in an example embodiment, theinfrared camera116 may image one or both of the HMD wearer's eyes. Theinfrared camera116 may deliver image information to theprocessor112, which may access thememory114 and make a determination regarding the direction of the HMD wearer's gaze, also termed a gaze axis. Theprocessor112 may further accept input from theGPS unit122, thegyroscope120, and/or theaccelerometer124 to determine the location and orientation of theHMD100. Subsequently, theprocessor112 may control theuser interface115 and thedisplay panel126 to display virtual images to the HMD wearer that may include context-specific information based on the HMD location and orientation as well as the HMD wearer's gaze axis.

HMD100 could be configured as, for example, eyeglasses, goggles, a helmet, a hat, a visor, a headband, or in some other form that can be supported on or from the wearer's head. Further,HMD100 may be configured to display images to both of the wearer's eyes, for example, using two see-through displays. Alternatively,HMD100 may include only a single see-through display and may display images to only one of the wearer's eyes, either the left eye or the right eye. TheHMD100 may also represent an opaque display configured to display images to one or both of the wearer's eyes without a view of the real-world environment. Further, theHMD100 could provide an opaque display for a first eye of the wearer as well as provide a view of the real-world environment for a second eye of the wearer.

Apower supply110 may provide power to various HMD components and could represent, for example, a rechargeable lithium-ion battery. Various other power supply materials and types known in the art are possible.

The function of theHMD100 may be controlled by aprocessor112 that executes instructions stored in a non-transitory computer readable medium, such as thememory114. Thus, theprocessor112 in combination with instructions stored in thememory114 may function as a controller ofHMD100. As such, theprocessor112 may control theuser interface115 to adjust the images displayed byHMD100. Theprocessor112 may also control thewireless communication interface134 and various other components of theHMD100. Theprocessor112 may additionally represent a plurality of computing devices that may serve to control individual components or subsystems of theHMD100 in a distributed fashion.

In addition to instructions that may be executed by theprocessor112, thememory114 may store data that may include a set of calibrated wearer eye pupil positions and a collection of past eye pupil positions. Thus, thememory114 may function as a database of information related to gaze direction. Such information may be used byHMD100 to anticipate where the user will look and determine what images are to be displayed to the wearer. Calibrated wearer eye pupil positions may include, for instance, information regarding the extents or range of the wearer's eye pupil movement (right/left and upwards/downwards) as well as wearer eye pupil positions that may relate to various reference axes.

Reference axes could represent, for example, an axis extending from a viewing location and through a target object or the apparent center of a field of view (i.e. a central axis that may project through a center point of the apparent display panel of the HMD). Other possibilities for reference axes exist. Thus, a reference axis may further represent a basis for determining dynamic gaze direction.

In addition, information may be stored in thememory114 regarding possible control instructions that may be enacted using eye movements. For instance, two consecutive wearer eye blinks may represent a control instruction directing theHMD100 to capture an image with aperipheral camera140. Another possible embodiment may include a configuration such that specific eye movements may represent a control instruction. For example, a HMD wearer may unlock theuser interface115 with a series of predetermined eye movements.

Control instructions could be based on dwell-based selection of a target object. For instance, if a wearer fixates visually upon a particular virtual image or real-world object for longer than a predetermined time period, a control instruction may be generated to select the virtual image or real-world object as a target object. Many other control instructions are possible.

In addition to the aforementioned features,memory114 could store various recorded data from previous HMD/user interactions. For instance, multiple images of a HMD wearer's eye(s) could be averaged to obtain an averaged eye gaze axis. This could lessen the effect of saccadic eye movements or saccades, in which the eye moves in a rapid and somewhat random manner around an eye gaze axis. These saccades help humans build up a mental image of a field of view with better resolution than if the eye remained static, and by averaging a number of eye images within a particular time period, an average gaze axis could be determined with less saccadic ‘noise’.

Additionally,memory114 could store recorded data regarding recent eye gaze axes for various application-based functions. For instance, the recent variance of the eye gaze axis could be coupled to scrolling images generated by theHMD100. In this embodiment, if recent eye gaze axis variance is high, the images (e.g. text or other images) could scroll faster. If the eye gaze axis variance is low, the images may scroll slower or stop altogether. In this context, a lower variance in eye gaze axis could indicate the HMD wearer is concentrating on one particular gaze location, whereas a higher eye gaze axis variance means the opposite—the HMD wearer may be quickly scanning a document and desire a faster scrolling speed.

Depending on the content that is presented on the HMD display, the variance may differ depending on the axis along which it is measured. For example, the horizontal variance of a HMD wearer's eye gaze may be high while the vertical variance may be relatively low. This could indicate to theHMD100 that the wearer is reading text. Accordingly, text scrolling/tracking could be adjusted in a different or more controlled fashion compared to ‘non-reading’ scrolling/panning/pagination situations.

TheHMD100 may include auser interface115 for providing information to the wearer or receiving input from the wearer. Theuser interface115 could be associated with, for example, the displayed virtual images, a touchpad, a keypad, buttons, a microphone, and/or other peripheral input devices. Theprocessor112 may control the functioning of theHMD100 based on inputs received through theuser interface115. For example, theprocessor112 may utilize user input from theuser interface115 to control how theHMD100 displays images within a field of view or to determine what images theHMD100 displays.

An eye-trackingsystem102 may be included in theHMD100. In an example embodiment, an eye-trackingsystem102 may deliver information to theprocessor112 regarding the eye position of a wearer of theHMD100. The eye-tracking data could be used, for instance, to correct for sensor drift errors introduced by gaps in sensor data and/or sensor noise. In particular, theprocessor112 could determine a target object in the displayed images based on information from the eye-trackingsystem102. Theprocessor112 may then control theuser interface115 and thedisplay panel126 to adjust the target object and/or other displayed images in various ways. For instance, the target object could be held static on thedisplay panel126. Alternatively, the target object could be moved towards a central axis of thedisplay panel126.

Aninfrared camera116 may be utilized by the eye-trackingsystem102 to capture images of a viewing location associated with theHMD100. Thus, theinfrared camera116 may image the eye of a HMD wearer that may be located at the viewing location. The viewing location may be illuminated by an infraredlight source118. The images could be either video images or still images. The images obtained by theinfrared camera116 regarding the HMD wearer's eye may help determine where the wearer is looking within the HMD field of view, for instance by allowing theprocessor112 to ascertain the location of the HMD wearer's eye pupil. Analysis of the images obtained by theinfrared camera116 could be performed by theprocessor112 in conjunction with thememory114.

The imaging of the viewing location could occur continuously or at discrete times depending upon, for instance, user interactions with theuser interface115. Theinfrared camera116 could be integrated into theoptical system106 or mounted on theHMD100. Alternatively, the infrared camera could be positioned apart from theHMD100 altogether. Furthermore, theinfrared camera116 could additionally represent a conventional visible light camera with sensing capabilities in the infrared wavelengths.

The infraredlight source118 could represent one or more infrared light-emitting diodes (LEDs) or infrared laser diodes that may illuminate a viewing location. One or both eyes of a wearer of theHMD100 may be illuminated by the infraredlight source118. The infraredlight source118 may be positioned along an optical axis common to theinfrared camera116 and/or the infraredlight source118 may be positioned elsewhere. The infraredlight source118 may illuminate the viewing location continuously or may be turned on at discrete times. Additionally, when illuminated, the infraredlight source118 may be modulated at a particular frequency. Other types of modulation of the infraredlight source118 are possible.

The HMD-trackingsystem104 could be configured to provide a HMD position and a HMD orientation to theprocessor112. This position and orientation data may help determine a central axis to which a gaze axis is compared. For instance, the central axis may correspond to the orientation of the HMD.

Thegyroscope120 could be a microelectromechanical system (MEMS) gyroscope, a fiber optic gyroscope, or another type of gyroscope known in the art. Thegyroscope120 may be configured to provide orientation information to theprocessor112. TheGPS unit122 could be a receiver that obtains clock and other signals from GPS satellites and may be configured to provide real-time location information to theprocessor112. The HMD-trackingsystem104 could further include anaccelerometer124 configured to provide motion input data to theprocessor112.

Theoptical system106 could represent components configured to provide virtual images to a viewing location. An example ofoptical system106 is described in detail below with respect toFIG. 2.

Variousperipheral devices108 may be included in theHMD100 and may serve to provide information to and from a wearer of theHMD100. In one example, theHMD100 may include awireless communication interface134 for wirelessly communicating with one or more devices directly or via a communication network. For example,wireless communication interface134 could use 3G cellular communication, such as CDMA, EVDO, GSM/GPRS, or 4G cellular communication, such as WiMAX or LTE. Alternatively,wireless communication interface134 could communicate with a wireless local area network (WLAN), for example, using WiFi. In some embodiments,wireless communication interface134 could communicate directly with a device, for example, using an infrared link, Bluetooth, or ZigBee.

AlthoughFIG. 1 shows various components of the HMD100 (i.e.,wireless communication interface134,processor112,memory114,infrared camera116,display panel126,GPS122, and user interface115) as being integrated intoHMD100, one or more of these components could be physically separate fromHMD100. For example, theinfrared camera116 could be mounted on the wearer separate fromHMD100. Thus, theHMD100 could be part of a wearable computing device in the form of separate devices that can be worn on or carried by the wearer. The separate components that make up the wearable computing device could be communicatively coupled together in either a wired or wireless fashion.

FIG. 2 illustrates a top view of anoptical system200 that is configured to display a virtual image superimposed upon a real-world scene viewable along aviewing axis204. For clarity, adistal portion232 and aproximal portion234 represent optically-coupled portions of theoptical system200 that may or may not be physically separated. An example embodiment includes adisplay panel206 that may be illuminated by alight source208. Light emitted from the visiblelight source208 is incident upon thedistal beam splitter210. The visiblelight source208 may include one or more light-emitting diodes (LEDs) and/or laser diodes. The visiblelight source208 may further include a linear polarizer that acts to pass one particular polarization to the rest of the optical system.

In an example embodiment, thedistal beam splitter210 is a polarizing beam splitter that reflects light depending upon the polarization of light incident upon the beam splitter. To illustrate, s-polarized light from the visiblelight source208 may be preferentially reflected by a distal beam-splittinginterface212 towards thedisplay panel206. Thedisplay panel206 in the example embodiment is a liquid crystal-on-silicon (LCOS) display. In an alternate embodiment in which the beam splitter coating atinterface212 is not a polarizing beam splitter, the display could be a digital light projector (DLP) micro-mirror display, or other type of reflective display panel. In either embodiment, thedisplay panel206 acts to spatially-modulate the incident light to generate a light pattern at an object plane in the display. Alternatively, thedisplay panel206 may be an emissive-type display such as an organic light-emitting diode (OLED) display, and in such a case, thebeam splitter cube210 is not needed.

In the example in which thedisplay panel206 is a LCOS display panel, thedisplay panel206 generates a light pattern with a polarization perpendicular to the polarization of light initially incident upon the panel. In this example embodiment, thedisplay panel206 converts incident s-polarized light into a light pattern with p-polarization. The generated light pattern from thedisplay panel206 is directed towards thedistal beam splitter210. The p-polarized light pattern passes through thedistal beam splitter210 and is directed along anoptical axis214 towards the proximal region of theoptical system200. In an example embodiment, theproximal beam splitter216 is also a polarizing beam splitter. The light pattern is at least partially transmitted through theproximal beam splitter216 to the image former218. In an example embodiment, image former218 includes aconcave mirror230 and a proximal quarter-wave plate228. The light pattern passes through the proximal quarter-wave plate228 and is reflected by theconcave mirror230.

The reflected light pattern passes back through proximal quarter-wave plate228. Through the interactions with the proximal quarter-wave plate228 and theconcave mirror230, the light patterns are converted to the s-polarization and are formed into a viewable image. This viewable image is incident upon theproximal beam splitter216 and the viewable image is reflected from proximalbeam splitting interface220 towards aviewing location222 along aviewing axis204. A real-world scene is viewable through aviewing window224. Theviewing window224 may include a linear polarizer in order to reduce stray light within the optical system. Light from theviewing window224 is at least partially transmitted through theproximal beam splitter216. Thus, both a virtual image and a real-world image are viewable to theviewing location222 through theproximal beam splitter216.

AlthoughFIG. 2 depicts thedistal portion232 of the optical system housing as to the left of theproximal portion234 of the optical system housing when viewed from above, it is understood that other embodiments are possible to physically realize theoptical system200, including thedistal portion232 being configured to be to the right, below and above with respect to theproximal portion234. Further, although an example embodiment describes an image former218 as comprising aconcave mirror230, it is understood by those skilled in the art that the image former218 may comprise a different optical element, such as an optical lens or a diffractive optic element.

In one embodiment, theproximal beam splitter216, thedistal beam splitter210, and other components ofoptical system200 are made of glass. Alternatively, some or all of such optical components may be partially or entirely plastic, which can also serve to reduce the weight ofoptical system200. A suitable plastic material is Zeonex® E48R cyclo olefin optical grade polymer which is available from Zeon Chemicals L.P., Louisville, Ky. Another suitable plastic material is polymethyl methacrylate (PMMA).

An example embodiment may include an infraredlight source226 that is configured to illuminate theviewing location222. AlthoughFIG. 2 depicts the infraredlight source226 as adjacent toviewing window224, those skilled in the art will understand that the infraredlight source226 could be located elsewhere, such as on the side of theproximal beam splitter216 that is adjacent to theviewing location222 or in thedistal portion232 of theoptical system200. The infraredlight source226 may represent, for example, one or more infrared light-emitting diodes (LEDs). Infrared LEDs with a small size may be implemented, such as the Vishay Technology TSML 1000 product.

Those skilled in the art will understand that, for best eye-tracking accuracy, it may be advantageous to obtain infrared images of the eye pupil using light sources that illuminate the eye from positions off-axis and/or on-axis with respect to theviewing axis204. Therefore, the infraredlight source226 may include one or more LEDs located at different locations in, and/or separate from, theoptical system200.

Infrared light generated from the infraredlight source226 is configured to be incident upon theviewing location222. Thus, the wearer's eye pupil may be illuminated with the infrared light. The infrared light may be reflected from the wearer's eye back along theviewing axis204 towards theproximal beam splitter216. A portion of the reflected infrared light may be reflected from thebeam splitting interface220 towards the image former218.

In order to transmit infrared light to aninfrared camera202, the image former218 may include a dichroic thin film configured to selectively reflect or transmit incident light depending upon the wavelength of the incident light. For instance, the dichroic thin film may be configured to pass infrared light while reflecting visible light. In an example embodiment, the visible light pattern generated by thedisplay panel206 may be reflected by theconcave mirror230 and the visible light pattern may be formed into a viewable image. The infrared light may thus be preferably transmitted through theconcave mirror230 to theinfrared camera202. Dichroic thin film coatings are available commercially from companies such as JML Optical Industries and Precision Glass & Optics (PG&O) and comprise multiple layers of dielectric and/or metal films. These dichroic coatings are also called ‘cold mirrors’.

In an example embodiment, a small aperture may be introduced into the image former218, which may be realized by a pinhole in the center of theconcave mirror230. In this example embodiment, most of the visible and infrared light is reflected off of and is formed by the image former218 into an image viewable by the HMD wearer. Some of the visible and infrared light passes through the aperture and is incident upon theinfrared camera202. Theinfrared camera202 may selectively filter and detect the infrared light from the combination of visible and infrared light to obtain information regarding the wearer's eye pupil location. Alternatively and/or additionally, the infraredlight source226 may be modulated with respect to a clock signal of a lock-in amplifier or phase-locked loop in order that the infrared light signal is transduced efficiently. Also, the visiblelight source208 may be modulated and infrared light detection could be performed when the visiblelight source208 is off, for example. Reflected infrared light may also be collected from off-axis angles, and thus the infrared camera may also be located elsewhere onoptical system200 or located separately fromoptical system200. Those with skill in the art will understand that there are other variations of transducing an infrared light signal mixed with a visible light signal with an infrared camera and that those variations are included implicitly in this specification.

FIG. 3A presents a front view of a head-mounted display (HMD)300 in an example embodiment that includes a head-mountedsupport309.FIGS. 3B and 3C present the top and side views, respectively, of the HMD inFIG. 3A. Although this example embodiment is provided in an eyeglasses format, it will be understood that wearable systems and HMDs may take other forms, such as hats, goggles, masks, headbands and helmets. The head-mountedsupport309 includes lens frames314 and316, acenter frame support318,lens elements310 and312, and extending side-arms320 and322. Thecenter frame support318 and side-arms320 and322 are configured to secure the head-mountedsupport309 to the wearer's head via the wearer's nose and ears, respectively. Each of theframe elements314,316, and318 and the extending side-arms320 and322 may be formed of a solid structure of plastic or metal, or may be formed of a hollow structure of similar material so as to allow wiring and component interconnects to be internally routed through the head-mountedsupport309. Alternatively or additionally, head-mountedsupport309 may support external wiring.Lens elements310 and312 are at least partially transparent so as to allow the wearer to look through them. In particular, the wearer'sleft eye308 may look throughleft lens312 and the wearer'sright eye306 may look throughright lens310.Optical systems302 and304, which may be configured as shown inFIG. 2, may be positioned in front oflenses310 and312, respectively, as shown inFIGS. 3A,3B, and3C.Optical systems302 and304 may be attached to the head-mountedsupport309 using support mounts324 and326, respectively. Furthermore,optical systems302 and304 may be integrated partially or completely intolens elements310 and312, respectively.

Although this example includes an optical system for each of the wearer's eyes, it is to be understood that a HMD might include an optical system for only one of the wearer's eyes (eitherleft eye308 or right eye306). As described inFIG. 2, the HMD wearer may simultaneously observe fromoptical systems302 and304 a real-world image with an overlaid virtual image. TheHMD300 may include various elements such as aprocessor340, atouchpad342, amicrophone344, and abutton346. Thecomputer340 may use data from, among other sources, various sensors and cameras to determine the virtual image that should be displayed to the user. In an example embodiment, as described earlier, an infrared light source or sources may illuminate the viewing position(s)308 and306, i.e. the wearer's eye(s), and the reflected infrared light may be collected with an infrared camera.

Those skilled in the art would understand that other user input devices, user output devices, wireless communication devices, sensors, and cameras may be reasonably included in such a wearable computing system.

FIGS. 4A and 4B depict side and front views of an eye as well as schematic drawings of pupil location information under different conditions. One way to determine a gaze axis of an eye is to ascertain the position of an eye pupil with respect to a reference point. In an example embodiment that tracks eye pupil position, infrared light is reflected off of a person's eye. The reflected light may be imaged with an infrared camera. Upon imaging of the eye, image processing can be conducted with aprocessor112 in order to determine, for instance, the extents and centroid location of the person's pupil. Other known means and methods of eye-tracking, including the use of visible light illumination and/or other imaging techniques are possible.

In anembodiment400, a person may be looking directly forward as depicted inFIG. 4A. Theeye412 is open and thepupil418 is located along acentral axis410. After image processing, which may include edge detection, the position of the pupil may be determined to be atpupil location422. In this example, theprocessor112 may subsequently determine that the gaze axis, based on thepupil location422, coincides with acentral axis410. Virtual image display position and movement may be adjusted due to the determinedpupil location422. For instance, theprocessor112 may adjust a tracking rate to zero when a gaze axis and the central axis are equivalent or nearly equivalent. This may allow a user to slowly read critical text or closely examine a virtual image, for example.

In anexample embodiment424, as illustrated inFIG. 4B, a person may be looking upwards with respect to acentral axis428. Theeye434 is open and the pupil location is generally higher than areference point440. In this situation, imaging the person'spupil438 with infrared light may result in adetermined pupil position442. Theprocessor112 may determine that thegaze axis430 is inclined above thecentral axis428. Theangle difference432 may represent the absolute difference in angle between thecentral axis428 and thegaze axis430. Theprocessor112 may calculate theangle difference432 and, based on theangle difference432, adjust a tracking rate. For instance, alarge angle difference432 could represent an adjustment in tracking rate such that the tracking rate is higher, for instance to scroll a virtual image across a field of view at a faster rate. In other embodiments, theprocessor112 may calculate theangle difference432 and move a target object in the displayed images in order to minimize theangle difference432. In other words, HMD display may be controlled such that the target object is moved toward the center of the HMD display.

Other embodiments could include the use of different eye gaze determination techniques. For instance, instead of using the eye pupil to determine gaze axis, it is possible to track eye motions using the boundary between the sclera and iris (the boundary is labelled as416 and436 inFIGS. 4A and 4B, respectively). For the purposes of determining an eye gaze axis, finding the centroid of the sclera/iris boundary (also called the limbus) may be equivalent to finding the centroid of a pupil.

Other possibilities for eye-tracking exist that may determine different reference points on the eye and may be implemented within the context of this invention. For instance, instead of ascertaining the pupil centroid to determine the gaze axis, position data from multiple glint reflections on the eye may be used in addition to or in lieu of information about the pupil position to determine the gaze axis.

3. Method for Adjusting Virtual Images on a Display Based on a Gaze Axis and a Central Axis of the Display.

Amethod500 is provided for adjusting displayed images on a display to move the target object closer to the central axis.Method500 could be performed using a HMD that could be configured as shown in any ofFIGS. 1 through 3C, or configured in some other way.FIG. 5 illustrates the steps in an example method, however, it is understood that in other embodiments, the steps may appear in different order and steps may be added or subtracted.

Step502 includes displaying images on a display. The display may have a central axis that could be considered a line passing through the center of the display, normal to the display itself. In alternate embodiments, the central axis may pass through other locations on the display that may not be at the center of the display, for instance if the display is not symmetric.

As discussed above, the display may present a wide variety of virtual images to a user including text, shapes, pictures, etc. These virtual images may allow the HMD wearer to experience an augmented reality.

Step504 includes determining a gaze axis with respect to the central axis. The determination of a gaze axis could be performed through eye-tracking. For example, an eye-trackingsystem102 could be used to image the eye of an HMD wearer to determine the position of his or her eye pupil and therefore determine a gaze axis, such as illustrated inFIGS. 4A and 4B.

Step506 includes determining a target object in the displayed images based on the gaze axis. After determining a gaze axis, theprocessor112 could determine that the wearer is gazing at a specific target object among the set of displayed images. This target object could be any virtual image from the images being displayed on the display. For example, a wearer may be viewing lines of virtual text and may gaze fixedly at a particular word or phrase. In turn, the word or phrase could be determined to be a target object.

Step508 includes adjusting the displayed images on the display to move the target object towards the central axis. This step could include moving a whole set of displayed images (including the target object) toward the central axis. For instance, following the previous example embodiment where a specific word or phrase is the target object, the entire formatted body of text may be moved towards the central axis.

Alternatively, only the target object may move towards the central axis. For instance, if a virtual image representing a specific icon from a set of icons is determined to be the target object, theprocessor112 may move only the target object icon toward the central axis of the display. Other movements are possible as well, including dynamic reformatting of text and other virtual images while moving a target object towards the central axis.

The rate at which the target object is moved, or the tracking rate, may be based on the angle between the gaze axis and the target axis, among other factors. For instance, if the HMD wearer is gazing at a target object near the edge of the field of view of the HMD (thus, the angle difference is large), the target object may be moved at a tracking rate that is relatively fast. Conversely, if the target object is only slightly separated from the central axis, the tracking rate may be slow.

An example ofmethod500 is illustrated inFIGS. 7A and 7B.FIG. 7A shows a HMD field ofview700 in whichvirtual text704 is being presented to the HMD wearer. An eye-trackingsystem102 could determine that an eye gaze axis may pass through thedisplay702 at a central eye gaze point706. The target object could be determined to be the word, ‘of.’ Since the target object is near the center of the display, the gaze axis and central axis are aligned or nearly so. Thus, theprocessor112 may determine that the wearer is gazing at the center of the screen and no virtual image movement may occur.

InFIG. 7B, a related field ofview708 may be presented to the HMD wearer. The HMD wearer is determined to have a gaze point710 within a set ofvirtual text704. In this example, the eye-trackingsystem102 andprocessor112 may determine that the wearer is gazing at the word, ‘hope’ and determine that word to be the target object. Thus, the virtual images may be adjusted so that the word ‘hope’ is moved toward the central axis. Another field ofview712 may be presented to the HMD wearer where the set ofvirtual text704 has been moved to the right and up such that the target object (in this example, the word ‘hope’) is located along the central axis of the display.

4. Method for Recording Data Based on a Gaze Axis, a Target Object and a Central Axis of a Display and Adjusting the Displayed Images on the Display Based on the Recorded Data

Amethod600 is provided for recording data based on the central axis, gaze axis, and target object and adjusting displayed images on a display to move the target object closer to the central axis.Method600 could be performed using a HMD that could be configured as shown in any ofFIGS. 1 through 3C, or configured in some other way.FIG. 6 illustrates the steps in an example method, however, it is understood that in other embodiments, the steps may appear in different order and steps may be added or subtracted.

In afirst step602, images are displayed on a display. The display may have a central axis that could be represented by a line that passes through the display at a normal incidence. The central axis may pass through other locations on the display that may not be at the center of the display, for instance if the display is not symmetric.

In asecond step604, a gaze axis may be determined with respect to the central axis. For example, a means for eye-tracking, such as eye-trackingsystem102, may be used to acquire images of an HMD wearer's eye. The images could be used byprocessor112 in order to determine information regarding the wearer's eye position, for instance, a centroid of a wearer's pupil and from this information further determine a gaze axis.

In athird step606, a target object may be determined based on the gaze axis. For example, aprocessor112 may determine that a particular gaze axis may be associated with a target object from among the displayed virtual images on the display. In one embodiment, the gaze axis may pass through a displayed virtual image that may be determined to be the target object. In other embodiments, target objects may be determined from the gaze axis even if the gaze axis does not intersect a corresponding virtual image. For instance, an HMD wearer may gaze fixedly at a particular virtual image on the display. Correspondingly, theprocessor112 may determine a target object not spatially related to the virtual image of interest.

In afourth step608, data may be recorded based on the central axis, the gaze axis, the target object, and the displayed images. For instance, data regarding the central axis, the gaze axis of an HMD wearer, the target object, and the displayed images may be stored inmemory114. The data could be recorded in a continuous fashion, or only be recorded when specific tasks are being performed. Alternatively or additionally, data could be recorded upon user request through theuser interface115 or during other HMD operations.

In afifth step610, the displayed images may be adjusted on the display based on the recorded data. In one example embodiment, data may be recorded during normal wearer interaction with the HMD. The data may include information, for instance regarding the HMD wearer's average reading speed, that could thereafter be used to adjust the virtual images displayed on the screen when the HMD wearer is reading text. For example, the rate of automated text scrolling could be controlled by prerecorded data regarding the wearer's average reading speed. The HMD could obtain data to determine user preferences that may encompass a broad range of various HMD user interactions and functions.

Other examples of how such recorded data could be used are possible. For example, the recorded data could be used for HMD calibration. Due to variations in interocular spacing and facial features between users, information from the eye-tracking system may vary slightly from one HMD wearer to the next, even when the users are performing the same actions. Thus, eye gaze determinations may also vary between individuals. Aprocessor112 could analyze the data stored in thememory114 to match a user eye gaze input to an intended user input. Displayed images on the display could depend at least on the results of the recorded data and corresponding user customizations. By recording data regarding eye-tracking interactions with the HMD, the need for calibration and customization for individual users may be reduced.

5. Non-Transitory Computer Readable Medium for Sensor Drift Correction

Some or all of the functions described above and illustrated inFIGS. 5,6,7A and7B may be performed by a computing device in response to the execution of instructions stored in a non-transitory computer readable medium. The non-transitory computer readable medium could be, for example, a random access memory (RAM), a read-only memory (ROM), a flash memory, a cache memory, one or more magnetically encoded discs, one or more optically encoded discs, or any other form of non-transitory data storage. The non-transitory computer readable medium could also be distributed among multiple data storage elements, which could be remotely located from each other. The computing device that executes the stored instructions could be a wearable computing device, such as awearable computing device100 illustrated inFIG. 1. Alternatively, the computing device that executes the stored instructions could be another computing device, such as a server in a server network.

With reference toFIG. 1, the non-transitory computer readable medium, which may correspond tomemory114, may store instructions executable by theprocessor112 to perform various functions. For instance, upon receiving gaze axis information from the eye-trackingsystem102, theprocessor112 may be instructed to control thedisplay panel126 to adjust the displayed images based on, for instance, the gaze axis and the central axis. In another embodiment, data could be recorded based on the gaze axis, central axis and target object in order to adjust the target object on the display or determine a user interface preference. The user interface preference could be used to adjust similar interactions with the HMD in the future. Those skilled in the art will understand that other sub-functions or functions may be reasonably included to instruct a processor to adjust a virtual image on a display based upon eye-tracking and other sensor data.

6. Method for Displaying Images in a HMD Based on the Movement of the HMD and Eye-Tracking Data

Amethod800 is provided for displaying images on a display of a head-mounted display (HMD) based on the movement of the HMD and eye-tracking data of the HMD wearer.Method800 could be performed using a HMD that could be configured as shown in any ofFIGS. 1 through 3C, or configured in some other way.FIG. 8 illustrates the steps in an example method, however, it is understood that in other embodiments, the steps may appear in different order and steps may be added or subtracted.

Method step802 includes displaying images on a display of a HMD. In this embodiment, the displayed images are viewable at a viewing location and the HMD display has a central axis. The viewing location could be, for instance, the location of an eye (or both eyes) of the HMD wearer. Similar to aforementioned embodiments, the central axis could be an axis protruding normally from the center of the HMD display.

Method step804 includes acquiring sensor data related to the motion of the HMD. The sensor data could be generated by, for instance, theaccelerometer124 and/orgyroscope120. The sensor data could include information such as the angle and azimuth of the HMD, which may correspond to the angle and azimuth of the central axis. Further, the sensor data could include information regarding the position of the HMD, such as may be acquired by theGPS122. Other types of sensor data, such as HMD velocity and acceleration information, are possible and may be reasonably applied using the example embodiment.

Method step806 includes controlling the display to display images based on the sensor data. In the example embodiment, the HMD could provide a graphical user interface in response to a movement of the HMD. The graphical user interface could be configured to be displayed in response to any HMD movement, or alternately, in response to a predetermined HMD movement. Thus,processor112 could be configured to acquire sensor data. In response to an appropriate sensor data input, theprocessor112 could control theoptical system106 to display images associated with a graphical user interface. For instance, if the HMD user moves his or her head upwards, the graphical user interface could be provided.

Further, while the graphical user interface is provided by the HMD, the user may view virtual images that could appear overlaid upon the real world environment. These virtual images could be substantially anchored to real world objects and/or reference points. Alternatively or additionally, some or all of the virtual images could stay substantially fixed with respect to, in one example, the azimuth (rotation angle within the horizontal plane) of the HMD. Thus, the HMD wearer could view different aspects of the graphical user interface by rotating his or her head and/or body.

Themethod step808 includes determining a gaze axis based on one or more images of the viewing location using a camera. In this step, as described in above embodiments, an infrared camera could work together with an infrared light source or sources to image the HMD wearer's eye(s). The images may be captured in an effort to ascertain a gaze direction or gaze axis of the HMD wearer. The gaze direction/axis may relate to the direction in which the user is looking.

Inmethod step810, a target object may be determined within the displayed images based on the gaze axis. Thus, using information obtained frommethod step808, theprocessor112 could be used to determine a particular target object from the set of currently displayed images. For example, a target object may be determined based on the point of intersection of the gaze direction and thedisplay panel126.

Method step812 includes controlling the display to move the target object towards the central axis. Within the context of this method, the user may be interacting within an established graphical user interface that could overlay his or her view of the real world environment. By gazing at a particular element within the graphical user interface could select it from the set of currently displayed images. The particular element could thus be selected to become the target object. Target object selection could be performed by gazing at the element for a predetermined period of time. The target object may then be moved towards the central axis of the HMD. This movement of the target object could allow the HMD wearer to, for instance, read a passage of text more easily or concentrate on a particular area of interest on an image. Additionally, as the centering action inmethod step812 could be independent of HMD movement, the method could be useful to reduce the effect of noise on drift in the sensor data, as well as unintentional movement of the HMD. In effect, commands to move the displayed images based on the HMD motion data can be ‘overridden’ when eye-tracking data may indicate a target object. In this manner, display of images could be essentially decoupled from HMD motion, which may reduce or eliminate the effect of unwanted motion sensor drift and HMD movement.

Those skilled in the art will understand that there are other possible ways of using HMD movement data and eye-tracking data to center target objects on a display of a HMD, and the aforementioned example embodiment is not meant to preclude any other such examples.

CONCLUSION

The above detailed description describes various features and functions of the disclosed systems, devices, and methods with reference to the accompanying figures. While various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims (14)

What is claimed is:

1. A method comprising:

displaying images on a display, the display having a central axis;

determining a gaze axis with respect to the central axis;

determining a target object in the displayed images based on the gaze axis; and

adjusting the displayed images on the display to move the target object towards the central axis.

2. The method ofclaim 1, wherein determining the gaze axis comprises:

obtaining an eye pupil image;

determining the gaze axis from the eye pupil image.

3. The method ofclaim 1, wherein determining the gaze axis comprises:

obtaining a plurality of eye pupil images; and

determining the gaze axis from the plurality of eye pupil images.

4. The method ofclaim 1, wherein adjusting the displayed images on the display comprises:

moving the target object at a tracking rate.

5. The method ofclaim 4, wherein adjusting the displayed images on the display further comprises:

adjusting the tracking rate based on a difference between the gaze axis and the central axis.

6. A method comprising:

displaying images on a display, the display having a central axis;

determining a gaze axis with respect to the central axis;

determining a target object in the displayed images based on the gaze axis;

recording data based on the central axis, the gaze axis, the target object, and the displayed images; and

adjusting the displayed images on the display based on the recorded data, wherein adjusting the displayed images on the display based on the recorded data comprises controlling movement of the displayed images on the display.

7. The method ofclaim 6, further comprising:

determining a user interface preference based on the recorded data;

adjusting the displayed images on the display based on the user interface preference.

8. The method ofclaim 6, wherein controlling movement of the displayed images on the display comprises controlling a rate of text scrolling.

9. A non-transitory computer readable medium having stored therein instructions executable by a computing device to cause the computing device to perform functions, the functions comprising:

controlling a display to display images, the display having a central axis;

determining a gaze axis with respect to the central axis;

determining a target object in the displayed images based on the gaze axis; and

controlling the display to adjust the displayed images so as to move the target object towards the central axis.

10. The non-transitory computer readable medium ofclaim 9, wherein determining a gaze axis comprises:

obtaining an eye pupil image;

determining the gaze axis from the eye pupil image.

11. The non-transitory computer readable medium ofclaim 9, wherein determining the gaze axis comprises:

obtaining a plurality of eye pupil images; and

determining a gaze axis from the plurality of eye pupil images.

12. The non-transitory computer readable medium ofclaim 9, wherein controlling the display to adjust the displayed images comprises moving the target object at a tracking rate.

13. The non-transitory computer readable medium ofclaim 12, wherein controlling the display to adjust the displayed images further comprises adjusting the tracking rate based on a difference between the gaze axis and the central axis.

14. The non-transitory computer readable medium ofclaim 9, wherein the functions further comprise:

recording data based on the central axis, the gaze axis, the target object, and the displayed images, and adjusting the displayed images on the display based on the recorded data.

Images