US20210406529A1 - Gesture-based casting and manipulation of virtual content in artificial-reality environments - Google Patents

Abstract

Embodiments are directed to a near eye display (NED) system for displaying artificial reality content to a user and to manipulate displayed content items based upon gestures performed by users of the NED system. A user of the NED system may perform a gesture simulating the throwing of an object to “cast” a content item to a target location in an artificial reality (AR) environment displayed by the NED system. The gesture may comprise a first portion in which the user's hand “grabs” or “pinches” a virtual object corresponding to the content item and moves backwards relative to their body, and a second portion in which the user's hand moves forwards relative to their body and releases the virtual object. The target location may be identified based upon a trajectory associated with the backwards motion of the first portion of the gesture.

Description

CROSS REFERENCE TO RELATED APPLICATIONS
• This application is a continuation of co-pending U.S. application Ser. No. 16/819,938, filed Mar. 16, 2020, which is a continuation of U.S. application Ser. No. 16/020,407, filed Jun. 27, 2018, now U.S. Pat. No. 10,635,895, each of which is incorporated by reference in its entirety.
BACKGROUND
• The present disclosure generally relates to a gesture and eye tracking system, and specifically gesture-based casting and manipulation of virtual content in artificial-reality environments.
• Augmented reality systems typically rely on wearable devices that have smaller form factors than classical virtual reality (VR) head mounted devices. The use of augmented reality systems presents new challenges in user interaction. Previous methods of user interaction with the local area may not be sufficient or optimal in an augmented reality system. For example, without the use of augmented reality, a user may need to interact physically with a device in a local area in order to enable a change in that device. However, with the user of augmented reality, both the device and the user experience may be upgraded to allow the user to cause a change in the device using methods other than simply physical interaction. However, such changes in user experience should be intuitive for the user to understand and should be technically feasible. Current method of user interaction in augmented reality are not readily intuitive and do not exploit the technical capabilities of an augmented reality system, and thus are not optimal for use.
SUMMARY
• Embodiments are directed to a near eye display (NED) system for displaying artificial reality content to a user and to manipulate displayed content items based upon gestures performed by users of the NED system. A user of the NED system may perform a gesture simulating the throwing of an object to “cast” a content item to a target location in an artificial reality (AR) environment displayed by the NED system. The gesture may comprise a first portion in which the user's hand “grabs” or “pinches” a virtual object corresponding to the content item and moves backwards relative to their body, and a second portion in which the user's hand moves forwards relative to their body and releases the virtual object. The target location may be identified based upon a trajectory associated with the backwards motion of the first portion of the gesture.
• In some embodiments, the system comprises a NED that includes an optical assembly having an electronic display that is configured to display images in accordance with display instructions and an imaging device configured to capture one or more images of a portions of a local area of the NED. The system further comprises a controller configured to determine a position of a hand of a user of the NED located within the local area using the captured one or more images, and based upon the determined hand positions, identify a first portion of a gesture performed by the hand of the user based upon one or more determined positions of the hand, the gesture corresponding to a predetermined sequence of one or more hand movements or hand positions and associated with a content item. The controller may further determine a position within the local area corresponding to an extension of a trajectory associated with at least the first portion of the identified gesture, and, in response to identifying a second portion of the gesture performed by the hand of the user based upon the one or more determined positions of the hand, update the display instructions to cause the electronic display to display a representation of the content item at the determined position.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 is a diagram of an eyewear device, in accordance with one or more embodiments.
• FIG. 2 is a cross section of the eyewear device ofFIG. 1, in accordance with one or more embodiments.
• FIG. 3 is a block diagram of a NED system with an eye tracker, in accordance with one or more embodiments.
• FIG. 4 illustrates an example of a virtual menu that may be displayed to a user by an eyewear device, in accordance with one or more embodiments.
• FIG. 5 is a flowchart of an example process for manipulating an artificial reality environment based upon identified gestures, in accordance with one or more embodiments.
• FIG. 6A illustrates the user performing a first part of the dartboard gesture to prepare to cast a virtual object onto a cast target, in accordance with one or more embodiments.
• FIG. 6B illustrates the user performing a second part of the dartboard gesture to cast the virtual object onto the cast target, in accordance with one or more embodiments.
• FIG. 7 illustrates a flowchart of an example process for casting a virtual object corresponding to a content item using a dartboard gesture, in accordance with one or more embodiments.
• FIG. 8A illustrates a virtual interface elements in AR environment where the user is “holding” a virtual object corresponding to a content item, in accordance with one or more embodiments.
• FIG. 8B illustrates the AR environment where the user manipulates the held virtual object using the virtual interface elements, in accordance with one or more embodiments.
• FIG. 9A illustrates the user performing a “slice” gesture to cause a menu to be displayed in the AR environment, in accordance with some embodiments.
• FIG. 9B illustrates a menu containing interface virtual interface elements being displayed to the user following performance of the “slice” gesture, in accordance with some embodiments.
• The figures depict embodiments of the present disclosure for purposes of illustration only. One skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles, or benefits touted, of the disclosure described herein.
DETAILED DESCRIPTION
• Embodiments of the invention may include or be implemented in conjunction with an artificial reality system. Artificial reality is a form of reality that has been adjusted in some manner before presentation to a user, which may include, e.g., a virtual reality (VR), an augmented reality (AR), a mixed reality (MR), a hybrid reality, or some combination and/or derivatives thereof. Artificial reality content may include completely generated content or generated content combined with captured (e.g., real-world) content. The artificial reality content may include video, audio, haptic feedback, or some combination thereof, and any of which may be presented in a single channel or in multiple channels (such as stereo video that produces a three-dimensional effect to the viewer). Additionally, in some embodiments, artificial reality may also be associated with applications, products, accessories, services, or some combination thereof, that are used to, e.g., create content in an artificial reality and/or are otherwise used in (e.g., perform activities in) an artificial reality. The artificial reality system that provides the artificial reality content may be implemented on various platforms, including a head-mounted display (HMD) connected to a host computer system, a standalone HMD, a mobile device or computing system, or any other hardware platform capable of providing artificial reality content to one or more viewers.
• In some embodiments, the near eye display (NED) system is configured to project artificial reality content to a user of the NED system. The NED system further includes a hand pose and gesture tracking system to determine hand positions and/or gestures performed by the user of the NED system and/or by other individuals with a local area of the NED system. The NED system is able to perform manipulations on displayed content items based upon the identified gestures. For example, in some embodiments, a user of the NED system may “cast” a content item to be displayed at a particular location in the AR environment (e.g., at a “cast target”) by performing a gesture that simulates that throwing of a content item towards the desired cast target location. Such types of gestures may be easily understood by users, allowing the user to display various content items at predetermined virtual cast target locations or on surfaces of objects in a local area in an intuitive way. In some embodiments, the user may further manipulate a displayed content item by interacting with one or more virtual interface elements, such as buttons displayed in the AR environment. By moving their hands to locations associated with the displayed virtual interface elements, various operations can be performed on the displayed content items.
• As used herein, the local area of the NED system may refer to an area surrounding a NED of the NED system. In some embodiments, the local area of the NED system may refer to a 360° area surrounding a location of a NED of the NED system that would be potentially visible to a user of the NED at the location. In addition, while the following discussion refers primarily to AR content, it is understood that in other embodiments the NED system may display to the user VR content in a VR environment.
Near Eye Display System Overview
• FIG. 1 is a diagram of aneyewear device100, in accordance with one or more embodiments. In some embodiments, theeyewear device100 is a near eye display (NED) for presenting media to a user. Examples of media presented by theeyewear device100 include one or more images, text, video, audio, or some combination thereof. In some embodiments, audio is presented via an external device (e.g., speakers and/or headphones) that receives audio information from theeyewear device100, a console (not shown), or both, and presents audio data based on the audio information. Theeyewear device100 can be configured to operate as an artificial reality NED. In some embodiments, theeyewear device100 may augment views of a physical, real-world environment with computer-generated elements (e.g., images, video, sound, etc.).
• Theeyewear device100 shown inFIG. 1 includes aframe105 and anoptical assembly110, which is surrounded by a rim115. Theoptical element110 is substantially transparent (e.g., allows a percentage transmittance) in the visible spectrum and may also include a substantially transparent electronic display. Theframe105 is coupled to one or more optical elements. In some embodiments, theframe105 may represent a frame of eye-wear glasses. Theoptical assembly110 may be configured for users to see content presented by theeyewear device100. For example, theeyewear device110 can include at least one waveguide display assembly (not shown) for directing one or more image light to an eye of the user. A waveguide display assembly includes, e.g., a waveguide display, a stacked waveguide display, a stacked waveguide and powered optical elements, a varifocal waveguide display, or some combination thereof. For example, the waveguide display may be monochromatic and include a single waveguide. In some embodiments, the waveguide display may be polychromatic and include a single waveguide. In yet other embodiments, the waveguide display is polychromatic and includes a stacked array of monochromatic waveguides that are each associated with a different band of light, i.e., are each sources are of different colors. A varifocal waveguide display is a display that can adjust a focal position of image light emitted from the waveguide display. In some embodiments, a waveguide display assembly may include a combination of one or more monochromatic waveguide displays (i.e., a monochromatic waveguide display or a stacked, polychromatic waveguide display) and a varifocal waveguide display. Waveguide displays are described in detail in U.S. patent application Ser. No. 15/495,373, incorporated herein by references in its entirety.
• In some embodiments, theoptical assembly110 may include one or more lenses or other layers, such as lenses for filtering ultraviolet light (i.e., sunglass lenses), polarizing lenses, corrective or prescription lenses, safety lenses, 3D lenses, tinted lenses (e.g., yellow tinted glasses), reciprocal focal-plane lenses, or clear lenses that do not alter a user's view. Theoptical assembly110 may include one or more additional layers or coatings, such as protective coatings, or coatings for providing any of the aforementioned lens functions. In some embodiments, theoptical assembly110 may include a combination of one or more waveguide display assemblies, one or more lenses, and/or one or more other layers or coatings.
• FIG. 2 is across-section200 of theeyewear device100 illustrated inFIG. 1, in accordance with one or more embodiments. Theoptical assembly110 is housed in theframe105, which is shaded in the section surrounding theoptical assembly110. A user's eye220 is shown, with dotted lines leading out of the pupil of the eye220 and extending outward to show the eye's field of vision. An eyebox230 shows a location where the eye220 is positioned if the user wears theeyewear device100. Theeyewear device100 includes an eye tracking system.
• The eye tracking system determines eye tracking information for the user's eye220. The determined eye tracking information may include information about a position of the user's eye220 in an eyebox230, e.g., information about an angle of an eye-gaze. An eyebox represents a three-dimensional volume at an output of a display in which the user's eye is located to receive image light.
• In one embodiment, the eye tracking system includes one or more light sources to illuminate the eye at a particular wavelength or within a particular band of wavelengths (e.g., infrared). The light sources may be placed on theframe105 such that the illumination from the light sources are directed to the user's eye (e.g., the location of the eyebox230). The light sources may be any device capable of producing visible or infrared light, such as a light emitting diode. The illumination of the user's eye by the light sources may assist theeye tracker240 in capturing images of the user's eye with more detail. Theeye tracker240 receives light that is emitted from the light sources and reflected off of the eye220. Theeye tracker240 captures images of the user's eye, and theeye tracker240 or an external controller can analyze the captured images to measure a point of gaze of the user (i.e., an eye position), motion of the eye220 of the user (i.e., eye movement), or both. Theeye tracker240 may be a camera or other imaging device (e.g., a digital camera) located on theframe105 at a position that is capable of capturing an unobstructed image of the user's eye220 (or eyes).
• In one embodiment, the eye tracking system determines depth information for the eye220 based in part on locations of reflections of the light sources. Additional discussion regarding how theeye tracker240 determines depth information is found in, e.g., U.S. application Ser. No. 15/456,383 and U.S. application Ser. No. 15/335,634, both of which are hereby incorporated by reference. In another embodiment, theeye tracker240 does not include light sources, but instead captures images of the user's eye220 without additional illumination.
• Theeye tracker240 can be embedded in an upper portion of theframe105, but may be located at any portion of the frame at which it can capture images of the user's eye. While only oneeye tracker240 is shown inFIG. 2, theeyewear device100 may includemultiple eye trackers240 per eye220.
• By tracking the user's gaze, theeye tracker240 can be used to determine where a user is looking at. This can be combined with a system, described below, which determines the gesture of the same user's hands. The combination of both a gaze and a particular gesture may be detected by the system and, in response, the system may perform some action based on the combination of gesture and gaze.
• FIG. 3 is a block diagram of aNED system300, in accordance with one or more embodiments-. TheNED system300 shown byFIG. 3 comprises aNED305 coupled to acontroller310, with thecontroller310 coupled to animaging device315. WhileFIG. 3 shows anexample NED system300 including oneNED305 and oneimaging device315, in other embodiments any number of these components may be included in theNED system300. In alternative configurations, different and/or additional components may be included in theNED system300. Similarly, functionality of one or more of the components can be distributed among the components in a different manner than is described here. For example, some or all of the functionality of thecontroller310 may be contained within theNED305. TheNED system300 may operate in an artificial reality environment.
• TheNED305 presents content to a user. In some embodiments, theNED305 is theeyewear device100. Examples of content presented by theNED305 include one or more images, video, audio, text, or some combination thereof In some embodiments, audio is presented via an external device (e.g., speakers and/or headphones) that receives audio information from theNED305, thecontroller310, or both, and presents audio data based on the audio information. In some embodiments, theNED305 operates as an artificial reality NED. In some embodiments, theNED305 may augment views of a physical, real-world environment with computer-generated elements (e.g., images, video, sound, etc.).
• TheNED305 includes anoptical assembly320 for each eye, aneye tracker325, an inertial measurement unit (IMU)330, one ormore position sensors335, and a depth camera array (DCA)340. Some embodiments of theNED305 have different components than those described here. Similarly, the functions can be distributed among other components in theNED system300 in a different manner than is described here. In some embodiments, theoptical assembly320 displays images to the user in accordance with data received from thecontroller310. In one embodiment, theoptical assembly320 is substantially transparent (e.g., by a degree of transmittance) to electromagnetic radiation in the visible spectrum.
• Theeye tracker325 tracks a user's eye movement. Theeye tracker325 includes a camera for capturing images of the user's eye. An example of the placement of the eye tracker is shown ineye tracker240 as described with respect toFIG. 2. Based on the detected eye movement, theeye tracker325 may communicate with thecontroller310 for further processing.
• In some embodiments, theeye tracker325 allows a user to interact with content presented to the user by thecontroller310 based on the detected eye movement. Example interactions by the user with presented content include: selecting a portion of content presented by the controller310 (e.g., selecting an object presented to the user), movement of a cursor or a pointer presented by thecontroller310, navigating through content presented by thecontroller310, presenting content to the user based on a gaze location of the user, or any other suitable interaction with content presented to the user.
• In some embodiments,NED305, alone or conjunction with thecontroller310 or another device, can be configured to utilize the eye tracking information obtained from theeye tracker325 for a variety of display and interaction applications. The various applications include, but are not limited to, providing user interfaces (e.g., gaze-based selection), attention estimation (e.g., for user safety), gaze-contingent display modes, metric scaling for depth and parallax correction, etc. In some embodiments, based on information about position and orientation of the user's eye received from the eye tracking unit, a controller (e.g., the controller310) determines resolution of the content provided to theNED305 for presentation to the user on theoptical assembly320. Theoptical assembly320 may provide the content in a foveal region of the user's gaze (and may provide it at a higher quality or resolution at this region).
• In another embodiment, the eye tracking information obtained from theeye tracker325 may be used to determine the location of the user's gaze in the local area. This may be used in conjunction with a gesture detection system to allow the system to detect various combinations of user gesture and gazes. As described in further detail below, different combinations of user gaze and gestures, upon detection by thecontroller310, may cause thecontroller310 to transmit further instructions to devices or other objects in the local area, or execute additional instructions in response to these different combinations.
• In some embodiments, theeye tracker325 includes a light source that is used to project light onto a user's eye or a portion of the user's eye. The light source is a source of the light that is reflected off of the eye and captured by theeye tracker325.
• TheIMU330 is an electronic device that generates IMU tracking data based on measurement signals received from one or more of theposition sensors335. Aposition sensor335 generates one or more measurement signals in response to motion of theNED305. Examples ofposition sensors335 include: one or more accelerometers, one or more gyroscopes, one or more magnetometers, another suitable type of sensor that detects motion, a type of sensor used for error correction of theIMU330, or some combination thereof. Theposition sensors335 may be located external to theIMU330, internal to theIMU330, or some combination thereof.
• Based on the one or more measurement signals from one ormore position sensors335, theIMU330 generates IMU tracking data indicating an estimated position of theNED305 relative to an initial position of theNED305. For example, theposition sensors335 include multiple accelerometers to measure translational motion (forward/back, up/down, left/right) and multiple gyroscopes to measure rotational motion (e.g., pitch, yaw, and roll). In some embodiments, theIMU330 rapidly samples the measurement signals and calculates the estimated position of theNED305 from the sampled data. For example, theIMU330 integrates the measurement signals received from the accelerometers over time to estimate a velocity vector and integrates the velocity vector over time to determine an estimated position of a reference point on theNED305. Alternatively, theIMU330 provides the sampled measurement signals to thecontroller310, which determines the IMU tracking data. The reference point is a point that may be used to describe the position of theNED305. While the reference point may generally be defined as a point in space; however, in practice the reference point is defined as a point within the NED305 (e.g., a center of the IMU330).
• The depth camera assembly (DCA)340 captures data describing depth information of a local area surrounding some or all of theNED305. The data that may be captured may include information captured from a structured light pattern projected on the local area, stereo images, time of flight data, or depth information captured using other depth measurement techniques. TheDCA340 can compute the depth information using the data (e.g., based on a captured portion of a structured light pattern), or theDCA340 can send this information to another device such as the controller710 that can determine the depth information using the data from theDCA340.
• TheDCA340 includes a light generator, an imaging device and a controller. The light generator of theDCA340 is configured to illuminate the local area with illumination light in accordance with emission instructions. The imaging device of theDCA340 includes a lens assembly, and a detector. The lens assembly is configured to receive light from a local area surrounding the imaging device and to direct at least a portion of the received light to the detector. The controller of theDCA340 generates the emission instructions and provides the emission instructions to the light generator. The controller of theDCA340 further determines depth information for the one or more objects based in part on the captured one or more images.
• Theimaging device315 may be used to capture a representation of the user's hands over time for use in tracking the user's hands (e.g., by capturing multiple images per second of the user's hand). To achieve a more accurate capture, theimaging device315 may be able to capture depth data of the local area or environment. This may be achieved by various means, such as by the use of computer vision algorithms that generate3D data via detection of movement in the scene, by the emission of a grid pattern (e.g., via emission of an infrared laser grid) and detection of depth from the variations in the reflection from the grid pattern, from computation of time-of-flight of reflected radiation (e.g., emitted infrared radiation that is reflected), and/or from the user of multiple cameras (e.g., binocular vision, stereophotogrammetry). Theimaging device315 may be positioned to capture a large spatial area, such that all hand movements within the spatial area are captured. In one embodiment, more than oneimaging device315 is used to capture images of the user's hands. As described in further detail below, the captured images of the user's hands may be used to identify various gestures for the user. Upon detecting these gestures in combination with the satisfaction of other conditions, a controller may execute certain associated actions.
• In another embodiment, theimaging device315 may also capture images of one or more objects in the local area, and in particular the area encompassing the field of view of a user wearing an eyewear device that includes theNED305. Theimaging device315 may also capture depth data of these one or more objects in the local area according to any of the methods described above.
• Although theimaging device315 is illustrated inFIG. 3 as being separate from theNED305, in some embodiments the imaging device is attached to theNED305, e.g., attached to theframe105, and may also be part of theDCA340.
• Theimaging device315 may include one or more cameras, imaging sensors, one or more video cameras, any other device capable of capturing images, or some combination thereof. Additionally, theimaging device315 may include one or more hardware and software filters (e.g., used to increase signal to noise ratio). Image tracking data is communicated from theimaging device315 to thecontroller310, and theimaging device315 receives one or more calibration parameters from thecontroller310 to adjust one or more imaging parameters (e.g., focal length, focus, frame rate, ISO, sensor temperature, shutter speed, aperture, etc.).
• In some embodiments, theNED305 may further compriseaudio sensors345. Theaudio sensors345 may comprise a microphone or other type of sensor configured to capture audio data from the local area surrounding theNED305. The audio data may include speech by the user of theNED system300 or by other individuals within the local area, other types of noises within the local area, and/or the like. In some embodiments, theaudio sensors345 may be located separately from theNED305.
• Thecontroller310 provides content to theNED305 for presentation to the user in accordance with information received from theimaging device315 or theNED305. In the example shown inFIG. 3, thecontroller310 includes aninput interface350, anapplication store355, atracking module360, a gesture identification (ID)module365, aspeech recognition module370, and acommunication module375, auser profile store380, and anexecution engine385. Some embodiments of thecontroller310 have different modules than those described herein. Similarly, the functions further described below may be distributed among components of thecontroller310 in a different manner than is described herein. In one embodiment, thecontroller310 is a component within theNED305.
• In one embodiment, thecontroller310 includes aninput interface350 to receive additional external input. These external inputs may be action requests. An action request is a request to perform a particular action. For example, an action request may be to start or end an application or to perform a particular action within the application. Theinput interface350 may receive input from one or more input devices. Example input devices include: a keyboard, a mouse, a game controller, or any other suitable device for receiving action requests. In another embodiment, theinput interface350 receives input from one or more radio frequency (RF) signal receivers. These may be used to receive radio signals from RF identifiers in the local area, and in some cases to determine a distance (based on signal strength) and position (based on triangulation or other method) of the RF identifier. After receiving an action request, thecontroller310 performs an action corresponding to the action request. In some embodiments, the action performed by thecontroller310 may include haptic feedback, which may be transmitted via theinput interface350 to haptic feedback devices. In some embodiments, theinput interface350 may comprise a separate component in communication with thecontroller310, such as a mobile device able to communicate wirelessly with thecontroller310.
• Theapplication store355 stores one or more applications for execution by thecontroller310. An application is a group of instructions, that when executed by a processor, generates content for presentation to the user. Content generated by an application may be in response to inputs received from the user via movement of theNED305, theinput interface350, theeye tracker325, and/or theaudio sensors345. Examples of applications include: gaming applications, conferencing applications, video playback application, or other suitable applications. In some embodiments, theapplication store355 may further store content generated by the one or more applications. In other embodiments, generated content may be stored in a separate content data store (not shown) implemented as part of thecontroller310 or as an external device in communication with thecontroller310.
• Thetracking module360 tracks movements of theNED305 and the hands of the user wearing theNED305. To track the movement of theNED305, thetracking module360 uses information from theDCA340, theimaging device315, the one ormore position sensors335, theIMU330, or some combination thereof. For example, thetracking module360 determines a position of a reference point of theNED305 in a mapping of a local area based on information from theNED305. Thetracking module360 may also determine positions of the reference point of theNED305 using data indicating a position of theNED305 from theIMU330. Additionally, in some embodiments, thetracking module360 may use portions of data indicating a position or theNED305 from theIMU330 as well as representations of the local area from theDCA340 to predict a future location of theNED305. Thetracking module360 may provide the estimated or predicted future position of theNED305 to theexecution engine385.
• As noted, thetracking module360 also tracks the user's hands, and the digits of the user's hands, in order to recognize various poses for the user's hand. Each pose indicates a position of a user's hand. By detecting a combination of multiple poses over time, thetracking module360 is able to determine a gesture for the user's hand. These gestures may in turn translate into various inputs to the system. For example, a movement using a single digit in one direction may translate into a button press input in the system.
• In one embodiment, thetracking module360 uses a deep learning model to determine the poses of the user's hands. The deep learning model may be a neural network, such as a convolutional neural network, or a residual neural network. The neural network may take as input feature data extracted from raw data from theimaging device315 of the hand, e.g., depth information of the user's hand, or data regarding the location of locators on any input device worn on the user's hands. The neural network may output the most likely pose that the user's hands are in. Alternatively, the neural network may output an indication of the most likely positions of the joints of the user's hands. The joints are positions of the user's hand, and may correspond to the actual physical joints in the user's hand, as well as other points on the user's hand that may be needed to sufficiently reproduce the motion of the user's hand in a simulation.
• If the neural network outputs the positions of joints, thetracking module360 additionally converts the joint data into a pose, e.g., using inverse kinematics principles. For example, the position of various joints of a user's hand, along with the natural and known restrictions (e.g., angular, length, etc.) of joint and bone positions of the user's hand allow thetracking module360 to use inverse kinematics to determine a most likely pose of the user's hand based on the joint information. The pose data may also include an approximate structure of the user's hand, e.g., in the form of a skeleton, point mesh, or other format.
• The neural network is trained using training data. In one embodiment, the training data is generated from a multiple camera array, such asmultiple imaging devices315, that captures hand movements in different poses with different hands from different users, and/or the locators on input devices worn by the different hands. The ground truth for this training data indicates joint positions and/or poses for the hands, and may be generated using human verification.
• In one embodiment, thetracking module360 is used to track movement of the digits of the user's hands and the hands themselves in order to recognize various gestures and poses for the user's hand. Each pose indicates a position of a user's hand. By detecting a combination of multiple poses over time, thetracking module360 is able to determine a gesture for the user's hand. These gestures may in turn translate into various inputs to the system. For example, a movement using a single digit in one direction may translate into a button press input in the system.
• An additional neural network may be used by thetracking module360 to determine a gesture from a particular series of poses. Such a neural network may be similarly trained, using as input data computed poses (or joints) and with output data indicating the most likely gesture. Other methods may be used by thetracking module360 to determine the gesture from the pose, such as a measurement of the distances and positions between the digits of the hand and the positions of a series of poses in3D space. If these distances and positions of each pose fall within certain thresholds, thetracking module360 may indicate that a particular gesture is present.
• Using such a method, thetracking module360 is able to determine the likely poses of a user's hands, and with the determination of the poses, thetracking module360 may be able to match the movement of the user's hands with predefined gestures. These gestures may be used to indicate various actions in an augmented reality environment.
• Thegesture ID module365 identifies the gestures of a user's hand based on the poses determined by thetracking module360. Thegesture ID module365 may utilize a neural network to determine a gesture from a particular series of poses. Such a neural network may be trained using as input data computed poses (or joints) and with output data indicating the most likely gesture. Other methods may be used by thegesture ID module365 to determine the gesture from the pose, such as a measurement of the distances and positions between the digits of the hand and the positions of a series of poses in 3D space. If these distances and positions of each pose fall within certain thresholds, thegesture ID module365 may indicate that a particular gesture is present. In one embodiment, thegesture ID module365 identifies a pinch gesture of the user. A pinch gesture is formed when distal ends of a user's index finger and thumb on one of the user's hands move within a threshold distance of each other. When the series of poses which form this gesture are recognized by thegesture ID module365, thegesture ID module365 determines that a pinch gesture is formed. As described in further detail below, thecontroller310 may execute certain actions when an identified gesture is detected along with other conditions, such as a particular gaze direction of the user's eyes as determined by theeye tracker325, audio data captured by theaudio sensors345, and/or the like.
• Using such a method, thetracking module360 is able to determine the likely poses of a user's hands, and with the determination of the poses, thegesture ID module365 matches the movement of the user's hands with predefined gestures. These gestures may be used to indicate various actions in an artificial reality environment.
• Additional details regarding the tracking and determination of hand positions using imaging devices and input devices are described in U.S. application Ser. No. 15/288,453, filed Oct. 7, 2016, and U.S. application Ser. No. 15/668,418, filed Aug. 3, 2017, both of which are incorporated by reference in their entirety.
• In another embodiment, thetracking module360 is also configured to recognize objects in images captured by theimaging device315. To perform this function, thetracking module360 may first be trained on a large corpus of labeled object data, or be coupled to a pre-trained image recognition system, which may be on an online system. In the former case, thetracking module360 includes a machine learning model (e.g., a convolutional neural network) and is trained on a standard image-object library (e.g., ImageNet), or on a large set of user-provided images from an online system. These user-provided images may include a large number of images of objects, as well as a labeling of these objects (e.g., using captions, etc.). Alternatively, in the latter case, the online system itself already includes a machine learning model trained on the aforementioned user-provided and labeled images. For example, the online system may already have an object recognition system which receives images and outputs a label for each. The model on the online system is used instead of any model on thecontroller310 to perform the object recognition in this case. After recognizing an object, thetracking module360 may be able to track the location of the object in the field of view provided by theNED305 to the user. This may be achieved by continuously recognizing users in each frame captured by theimaging device315. Once an object is recognized, thetracking module360 can indicate the location of the object, and the boundaries of the object (e.g., the pixels corresponding to the recognized object) in the captured image. This can be translated to a location of the object in the user's field of view provided by theNED305 through theoptical assembly310.
• In some embodiments, thecontroller310 comprises aspeech recognition module370 configured to transcribe audio data received via one or moreaudio sensors345. In some embodiments, theaudio sensors345 are able to record audio data corresponding to speech by the user of theNED system300 or other individuals within the local area of theNED305. Thespeech recognition module370 uses one or more audio transcription algorithms to parse to received audio data and transcribe a transcription of the detected speech. In some embodiments, the transcription may be used to create or manipulate one or more content items, to display one or more virtual objects to the user of theNED305, mapped to an executable command, and/or the like.
• In some embodiments, thecontroller310 comprises acommunication module375 for communication with other systems, such as other NED systems, allowing theNED system300 to transmit and receive commands and/or content items with other systems through thecommunication module375. For example, different users of different NED systems may be able to access and view different content items in their respective AR environments. Thecommunication module375 may allow for the first user of afirst NED system300 to transmit and share one or more content items with a second user of a second NED system (which may have a structure similar to that of the first NED system300), allowing both users to view the same content. In some embodiments, thecommunication module375 may transmit one or more commands to an external system based upon inputs received from a user (e.g., through theinput interface350, or through the performance of one or more gestures).
• In some embodiments, theNED system300 may be used by a plurality of different users. Each user of theNED system300 may be associated with an account or user profile stored in auser profile store385. Theuser profile store385 stores user information corresponding to each user, such as user attributes, user preferences, content associated with the user, and/or the like. In some embodiments, theuser profile store385 stores permission information indicating content that may be viewed and/or modified by different users of theNED system300. As such, different users of theNED system300 may be able to view different content through theNED305. For example, a first user of theNED system300 may be able to view through an AR environment projected by theNED305 virtual objects depicting images associated with a user profile of the first user, which would not be visible to a second user of theNED system300 associated with a different user profile.
• In some embodiments, instead of user profile information being stored in theuser profile store380, theNED system300 may, using thecommunication module375, access an online system such as a social network platform in order to access user profile information associated with different users. In some embodiments, theNED system300 accesses the social network platform to identify social network connections between different users of theNED system300. Based upon the social network connections, theNED system300 may determine which types of content can be displayed to which users (e.g., content associated with a first user may be viewed by a second user having a “friend” relationship with the first user, but not by a third user having no social network connection to the first user).
• In some embodiments, different users may concurrently use theNED system300 viadifferent NEDs305. For example, thecontroller310 may transmit display instructions to afirst NED305 to display a first AR environment to a first user, and to a second NED to display a second AR environment to a second user. As such, each of the first and second users may be able to view different virtual objects through their respective AR environments. In some embodiments, theNED system300 may cause different content to be displayed in the first or second AR environments in response to detecting one or more predetermined gestures performed by the first and/or second users. For example, as discussed in greater detail below, theNED system300 may, in response to detecting a particular gesture by the first user, display to the second user a visual flair emphasizing the gesture in the second AR environment. In some embodiments, theNED system300 may, in response to a gesture by the first user, change the permissions of a content item such that the content item can be viewed by the second user in the second AR environment.
• In one embodiment, thecontroller310 additionally includes anexecution engine385. Theexecution engine385 executes applications within theNED system300 and receives position information, acceleration information, velocity information, predicted future positions, or some combination thereof, from theNED305,input interface350, and/or thetracking module360. Based on the received information, theexecution engine385 determines content to provide to theNED305 for presentation/display to the user. For example, if the received information indicates that the user has looked to the left, theexecution engine385 generates content for theNED305 that is based off the user's movement in the artificial reality environment. Similarly, if information received from thetracking module360 indicates the user's hand makes a particular gesture, theexecution engine385 generates content based on the identified gesture. In addition, if the information received from theNED305 indicates a particular gaze of the user, theexecution engine385 may generate content based on that gaze. This content may include an update to theoptical assembly320 in theNED305, such that content displayed to a user wearing theNED305 changes.
• Theexecution engine385 may also perform an action within an application executing on thecontroller310 in response to an action request received from theinput interface350 and provides feedback to the user that the action was performed. The provided feedback may be visual or audible feedback via theNED305. For example, theexecution engine385 may receive an action from theinput interface350 to open an application, and in response, theexecution engine385 opens the application and presents content from the application to the user via theNED305.
• In one embodiment, theexecution engine370 determines the current pose of the user's hand based on raw joint data received from thetracking module360. In another embodiment, theexecution engine370 detects the gestures of the user's hand based on the pose data. The methods in which theexecution engine370 determines poses and gestures are similar to the methods described above for thetracking module360. In one embodiment, theexecution engine370 performs object recognition similar to the method described above for thetracking module360.
• In addition to determining the current pose of the user's hand(s), theexecution engine385 may also provide output to theoptical assembly320 in accordance with a set of display instructions (e.g., pixel data, vector data, etc.). This output to the electronic display of theoptical assembly320 may include a virtual recreation (using computer graphics) of the user's hands, as well as other objects (virtual or otherwise), such as outlines of objects in the local area, text, graphics, other elements that coincide with objects within a field of view of a user wearing theNED305, and so on. In some embodiments, theexecution engine385 provides output to theoptical assembly320 to cause theoptical assembly320 to display one or more virtual objects to be displayed to user at locations corresponding to the position of the user's hands. For example, if a user performs a “pinch” gesture with their hand, theexecution engine385 may provide instructions to theoptical assembly320 to display a particular virtual object such that a corner of the virtual object remains between the thumb and index finger of the user's hand, even as the user moves their hand.
• Theexecution engine385 may receive from thetracking module360 an indication of a tracked object. Thetracking module360 may be previously configured to recognize and track certain objects. These objects may provide some control functionality or may be associated with additional details or information. Upon receiving the indication of the tracked object, theexecution engine385 transmits display instructions to theoptical assembly320 to cause theoptical assembly320 to display various elements, such as contextual menus, control user interface elements, informational menus, and so on, to the user. These displayed elements may be shown at a threshold distance from the tracked object as viewed by the user in the augmented or artificial reality environment presented by theNED305.
• In one embodiment, theexecution engine385 may first recognize the recognizable objects in a local area as captured by theimaging device315. An object may be recognized according to a pre-programmed recognition pattern. A recognition pattern may include a unique identifier of the object as generated by the object recognition system of thetracking module360. The recognition pattern may include the values of the output parameters generated by the object recognition system that caused thetracking module360 to recognize the object (e.g., the confidence weights generated by the object recognition system). In another embodiment, the recognition pattern may be some other fingerprint, pattern, identifier, or other data that is able to be used to recognize the object again under different orientation and lighting. When the object is encountered, the object recognition system of thetracking module360 may generate another identifier based on the characteristics of the object. This identifier is compared to the stored recognition pattern for the object, and if a match occurs, the object is recognized as the object associated with the stored recognition pattern.
• Additionally, in some embodiments, theexecution engine385 further utilizes additional tracking indicators in the local area to assist in the recognition of objects. As noted above, the objects in the environment may have RF identifiers, which may be received by theinput interface350 via one or more RF receivers. Theexecution engine385, via the signals received from the RF receivers, and through various signal source locating mechanisms (e.g., triangulation, time-of-flight, Doppler shift), may determine the position of an object that has an RF identifier using the RF signals from the object. This information may be used to augment (e.g., adjust for error) the image based object recognition system, or may be used in place of the image based object recognition system (e.g., in the case where the image based object recognition system fails or has high error/uncertainty). Other tracking indicators, such as retroreflectors (which may respond to a non-visible light signal from the eyewear device100), high contrast locators, QR codes, barcodes, identifying image patterns, and so on, may also be used by theexecution engine385 to assist in recognizing the object, and this information may be stored in the recognition pattern for the object. Upon recognition of the enhanced object, theexecution engine385 may update the display instructions of theoptical assembly320 to present additional simulated or virtual elements related to the enhanced object in the artificial reality environment presented by theNED system300. The virtual elements may be positioned in the artificial reality environment at a threshold distance (e.g., 1 cm) of the enhanced object. Theexecution engine385 may compute the position of the enhanced object in 3D space and project the virtual elements on the display such that they appear to be within the 3D space and near to the enhanced object (within the threshold distance). Upon detection of movement of the enhanced object, theexecution engine385 may submit updated display instructions to move the virtual elements based on the movement of the enhanced object.
• In some embodiments, the one or more tracked objects may correspond to other individuals within the local area, such as users of other NEDs. Theexecution engine385 may, as part of tracking the other individuals, identify gestures performed by the other individuals, and update the display instructions of theoptical assembly320 based upon the identified gestures. For example, theexecution engine385 may instruct theoptical assembly320 to display visual flair corresponding to the identified gestures. In some embodiments where the other individuals are users of other NED systems, theexecution engine385 may transmit or receive content with the other NED systems, based upon identified gestures performed by the user of theNED system300 or by the users of the other NED systems.
• Theexecution engine385, in response to receiving from thetracking module360 an indication of certain gestures, may execute certain instructions related to one or more of the recognized objects. This may allow for the user to select and manipulate various virtual objects in the AR environment intuitively through the use of gestures. For example, the user may select a virtual object corresponding to a content item (e.g., a representation of a content item) within the AR environment using a pinching or grabbing gesture, cast the virtual object onto different locations of the AR environment, share the content associated with the virtual object with other users, and/or manipulate the virtual object using one or more virtual interface elements. In some embodiments, the instructions may be further based upon an eye-gaze direction of the user (as determined by the eye tracker325), transcribed speech of the user or other individual in the local area (as determined by the speech recognition module370), and/or some combination thereof. Additional examples of such gestures are described with further detail below with reference toFIGS. 4-8B.FIGS. 4-5 describe gestures for selecting virtual interface objects on a displayed virtual menu.FIGS. 6A-6B and 7 describe a “dartboard” gesture used to display virtual objects at cast targets within the local area.FIGS. 8A-8B describe gestures for manipulating a displayed virtual object via virtual interface elements.
• Although the description below is made primarily with reference to an artificial reality (e.g., an augmented reality) environment, the method described herein can also be applied to a virtual reality environment as well. In the virtual reality environment, the user wears a head mounted device that has an electronic display that does not allow light from the local area to strike the user's eyes. Instead, the user's view includes only what is displayed by the electronic display. In such a case, the eye gaze of the user may be towards a virtual object rather than an object in the local area, and the user may see a virtual representation of his or her hands in the virtual reality environment instead of his or her actual hands.
Virtual Interface in AR
• In some embodiments, a near eye display (NED) system (e.g., NED system300) having object recognition and gesture tracking capabilities may allow theNED system300 to perform manipulations of the AR environment based upon detected user hand gestures. In some embodiments, theNED305 presents a user of the NED system with a virtual interface in the AR environment, which may be manipulated by the user using hand gestures.
• FIG. 4 illustrates an example of a virtual menu that may be displayed to a user by aneyewear device410, according to an embodiment. The view inFIG. 4 is of theeyewear device410 and the AR environment. In one embodiment, theeyewear device410 includes theNED system300 described with reference toFIG. 3, and thus includes thecontroller310,optical assembly365, and so on. In other embodiments, theeyewear device410 corresponds to theNED305 illustrated inFIG. 3, and does not include thecontroller310. Theoptical assembly365 of theeyewear device410 may display the virtual menu430 in the artificial environment to the user. The virtual menu430 is represented here by a ring of circular objects, with each circle representing a virtual menu option. However, in other embodiments, they may take on other shapes and features. For example, they could be spheres, cubes, text, or shaped (e.g., skeuomorphically) like real-world objects such as buttons, etc.
• Thecontroller310 may further detect that the user's hand440 is performing a touch gesture with one of the virtual menu options. A touch gesture is detected when thecontroller310 detects that the distal end of one of the user's fingers is within a threshold distance of one of the virtual menu options (e.g., 2 mm) in the artificial reality environment. Once the touch gesture is detected, thecontroller310 may execute an option corresponding to the selected or touched virtual menu option.
• In some embodiments, thecontroller310 is configured to be able to identify different types of gestures that may be performed by the user. As used herein, a “gesture” may refer to any series of one or more movements or positions of the user's hand440 able to be identified or categorized by thecontroller310. In some embodiments, a gesture comprises a sequence of multiple motions of the user's hand440. In some embodiments, a gesture also corresponds to a particular position or orientation of the user's hand440. Thecontroller310 may, in response to an identified gesture, execute one or more operations manipulating the artificial reality environment, based upon the specific gesture identified. As used herein, motions or positions of the user's hand may encompass particular motions or positions of the fingers/digits of the user's hand.
• In some embodiments, thecontroller310 detects a pinch-pull gesture with one of the virtual menu options. The pinch-pull gesture is detected when the distal ends of the user's index finger and thumb are within a threshold distance of each other (e.g., 5 mm) and they surround one of the virtual menu options. Furthermore, the user's hand moves in a direction towards the eyewear device410 (i.e., that reduces the distance between the user's hand and the eyewear device). This type of gesture may be used for a virtual menu option that indicates a status indicator for the user. Upon detecting such a gesture, thecontroller310 may apply the status indicator indicated by the selected virtual menu option to the user. For example, the virtual menu option may indicate a status indicator of “busy.” If the user pinches this option and pulls it towards the user, thecontroller310 may apply the “busy” status indicator to the user, and transmit this status to an online system. In some embodiments, other users with eyewear devices can subsequently see this updated status (which may be updated in real time) for the user. In other embodiments, the virtual menu options are used to manipulate the AR environment displayed to the user, such as by displaying a content item corresponding to the virtual menu option, perform an operation corresponding to the virtual menu option on a displayed content item, and/or the like.
• FIG. 5 is a flowchart of an example process for manipulating an artificial reality environment based upon identified gestures, in accordance with one or more embodiments. As discussed above, in some embodiments, a NED system (e.g., the NED system300) may comprise an imaging device (e.g., imaging device315) or other type of sensor able to capture a representation of the user's hands over time for use in tracking the user's hands. A controller (e.g., the controller310) may perform one or more manipulations of the artificial reality environment displayed to the user through the eyewear device, based upon the tracked position and movement of the user's hands (e.g., using the tracking module360).
• The controller determines510 a position of a hand of the user associated with the NED system. In some embodiments, the position of the hand determined by the controller may comprise a position of the user's hand relative to the user's body, a position of the user's hand relative to their other hand, a position of the fingers of the user's hand(s), or any combination thereof.
• The controller identifies520 a first gesture, based upon a first set of determined hand positions. In some embodiments the controller tracks the position of the user's hand over time to determine a movement of the user's hand. The first gesture may correspond to a sequence of one or more determined movements of the user's hand. In some embodiments, the controller identifies a pulling gesture or a pushing gesture when it detects movement of the user's hand on a path towards or away from the user's body, respectively.
• The first gesture may be identified based upon a determined position or pose of the user's hand(s). In an embodiment, the controller, in response to determining that the user's hands are on either side of their body, with their palms turned upwards, identifies a shrug gesture.
• In some embodiments, identifying the first gesture comprises identifying that the user's hand moves to a particular location relative to a real-world object (e.g., the user's body or some other object within the local area) or a virtual object displayed in the artificial reality environment during a particular portion of the gesture, and/or achieves a particular pose at the particular location. In some embodiments, the controller, as part of identifying a pulling gesture, determines that the position of the user's hand at the start of the pulling gesture corresponds to a particular displayed virtual object.
• The controller performs530 a manipulation of the artificial reality environment, based upon the identified gesture. In some embodiments, the controller causes the eyewear device to display a new virtual object as part of the artificial reality environment, change an existing virtual object (e.g., how the virtual object is displayed, and/or a setting of the virtual object), and/or the like. In some embodiments, the controller, in response to the identified gesture, changes a setting associated with the NED system300 (e.g., a status of the user, a setting of a piece of data associated with the user, and/or the like), communicate with another system (e.g., through the communication module375) such as a controller associated with a real-world object in the local area or anotherNED system300, and/or the like. In some embodiments, the manipulation is based upon a position of the user's hand a particular point in time during the gesture (e.g., in relation to a virtual object, a real-world object, and/or the like). Examples of specific gestures and their effects are described in greater detail below.
Casting Virtual Content Items Dartboard Gestures
• In some embodiments, different types of gestures may be used to display and manipulate virtual objects displayed to a user in the AR environment. For example, in some embodiments, the user of a NED system (e.g., NED system300) may perform a dartboard gesture to place or “cast” a displayed virtual object in the AR environment onto a target location.
• FIGS. 6A and 6B illustrates an example of a user of a NED system being able to pick and place virtual objects using a dartboard gesture, in accordance with some embodiments.FIG. 6A illustrates the user performing a first part of the dartboard gesture to prepare to cast a virtual object onto a cast target. As used herein, a dartboard gesture may refer to a gesture comprising a first motion where the user's hand moves in a backwards (i.e., posterior) direction relative to the user's body at shoulder height or above, followed by a second motion where the user's hand moves in a forward (i.e., anterior) direction relative to the user's body at shoulder height or above. In some embodiments, the dartboard gesture may require that the fingers of the user's hand be in a certain position (e.g., as if pinching or grabbing a virtual object during the backwards motion, and spreading apart as if releasing the virtual object during the forwards motion). In addition, the user's hand may be required to have the palm facing upwards and elbow facing downwards during the backwards motion, while having the palm facing away from the user's body during the forward motion. In some embodiments, the dartboard gesture requires that the first or second motion have at least a threshold speed or range of movement. In some embodiments, the first or second motions may reach at certain position relative to the user's body, such as behind the user's ear.
• The user of the NED system may be able to access, view, and manipulate different virtual objects corresponding to content items (e.g., images, videos, and/or the like). In order to organize the virtual objects displayed in the AR environment, the user may wish to be able to place virtual objects corresponding to different content items onto different locations in the AR environment. For example, as illustrated inFIG. 6A, the AR environment may contain casttargets605 onto which the user may place various virtual objects, allowing the user to organize the virtual objects and view multiple virtual objects corresponding to different content items at once. Eachcast target605 corresponds to a boundary defining an area within the AR environment viewed by the user through theNED305, and may be marked by a virtual boundary (e.g., a dotted line) displayed in the AR environment enclosing the area corresponding to thecast target605. In other embodiments, the NED system identifies surfaces in the local area (e.g., walls, tables, etc.) onto which the user may “cast” one or more content items.
• In some embodiments, a content item may be initially displayed as avirtual object610 at a first location within the AR environment. The user may perform a “pinching” gesture in order to “pinch” the virtual object corresponding to content item, in which the user may move their hand near the displayedvirtual object610 and close their fingers over a portion of the virtual object610 (e.g., a corner of the virtual object610) such that the distal ends of their index finger and thumb are within a threshold distance and on opposite sides of a portion of thevirtual object610. In response to detecting the performed gesture, the NED system displays thevirtual object610 at a particular location relative the user's hand (e.g., such that a portion of thevirtual object610 appears between the user's index finger and thumb, to appear as being “pinched” by the user), and moves thevirtual object610 in accordance with the movements of the user's hand so long as the user continues to “pinch” thevirtual object610. Thus, thevirtual object610 is displayed to remain in the same location relative to the user's hand615 (e.g., between the distal ends of the user's index finger and thumb) as the user'shand615 moves as if travelling with the user'shand615.
• InFIG. 6A, the user has performed a “pinching” gesture with thevirtual object610 using herhand615, and has performed a first motion with herhand615, corresponding to a firstbackwards motion620 relative to the body of the first user (e.g., towards the dorsal side of the first user). Because thevirtual object610 has been pinched by the user, thevirtual object610 may move with the user'shand615 during performance of thebackwards motion620. TheNED system300, in response to identifying the first backwards motion, may identify atrajectory625 associated with themotion620, and acast target630 of the plurality of cast targets605 corresponding to thetrajectory625. The identifiedtrajectory625 may be based on a portion of the path travelled by thehand615 over the course of the motion620 (e.g., in a direction opposite of the path of the motion620). In some embodiments, thetrajectory625 may instead be based upon a determined gaze direction of the user, as determined by the eye tracker325 (e.g., aligned with the gaze direction), and/or a head direction or head gaze of the user, as determined by a head tracking sensor (not shown). The identified casttarget630 corresponds to a cast target displayed over an area of the AR environment that intersects thetrajectory625 or an extension of thetrajectory625. TheNED system300 may change how the identifiedcast target630 is displayed in the AR environment (e.g., illustrated inFIG. 6A with a thicker dotted-line border) in order to indicate that thecast target630 corresponds to the identifiedtrajectory625 and has been “selected.”
• In some embodiments, instead of selecting thecast target630 from a plurality of virtual cast targets605, theNED system300 may identify thecast target630 corresponding to thetrajectory625 based upon one or more objects in the local area (e.g., a location where the identifiedtrajectory625 intersects a wall or other surface) and displays a visual indicator (e.g., a target symbol) at the identified location in the AR environment. Thecast target630 may correspond to a location on a surface of an object in the local area, such as a table or wall, which intersects thetrajectory625 or an extension of thetrajectory625. In some embodiments, thecast target630 may be identified based upon objects of the local area if there are no cast targets605 in the AR environment that intersect the trajectory625 (or its extension).
• FIG. 6B illustrates the user performing a second part of the dartboard gesture to cast the virtual object onto the target location (e.g., the identified cast target), in accordance with some embodiments. As illustrated inFIG. 6B, the user performs asecond motion635, which corresponds to moving the user'shand615 in a forward direction relative to the user's body. In some embodiments, thesecond motion635 travels along a path that is within a threshold angle from the opposite direction of themotion620. In addition, the user, as part of thesecond motion635, may change the positions of their fingers to signify “releasing” or “letting go” of thevirtual object610, in which the palm of the user'shand615 faces in a direction radiating away from the user's body, and the distal ends of the user's thumb and index fingers are separated by at least a threshold distance. In some embodiments, when the user releases thevirtual object610, thevirtual object610 may be displayed moving away from the user's hand, as if thrown by the user.
• TheNED system300 may, in response to detecting thesecond motion630, display thevirtual object610 in the AR environment at the identifiedcast target630. For example, theNED system300 may display thevirtual object610 on thecast target630, thevirtual object610 sized to fill the area ofcast target630 in at least one dimension. In embodiments where theNED system300 identifies thecast target630 on a surface within the local area, the NED system displays thevirtual object610 at the location of thecast target630 on the surface within the AR environment with a predetermined size, or a size based upon a size of the surface. In some embodiments, the user may be able to cast multiplevirtual objects610 onto the same surface within the local area. As the virtual objects are cast onto the surface, the layout of the virtual objects cast onto the surface may change in order to accommodate the virtual objects displayed on the surface. In some embodiments, one or more cast targets630 may be dynamically displayed on the surface in the local area, indicating the layout of how thevirtual object610 would be displayed if cast onto the surface. The cast targets630 on the surface may be automatically adjusted as additional virtual objects are cast onto the surface, allowing for the surface to fit multiple casted virtual objects.
• As such, through the performance of dartboard gestures, the user can cast various virtual objects corresponding to content items onto different locations within the AR environment, allowing the user to view and organize multiple content items. In some embodiments, once the user has cast one or more virtual objects, the user may perform one or more manipulations on the virtual objects (e.g., pick up and move the virtual objects between cast targets, resize the displayed virtual objects, delete one or more virtual objects, and/or the like). In some embodiments, when thevirtual object610 is displayed on thecast target605, theNED system300 may display one ormore interface elements640 on or near thevirtual object610, allowing for the user to perform manipulations on thevirtual object610 using theinterface elements640.
• In some embodiments, the user, after casting one or more virtual objects onto locations in the AR environment, may remove a casted virtual object from its cast target using one or more gestures. For example, the user may select a casted virtual object based upon a pointing direction (e.g., by pointing with their hand towards the intended virtual object), a gaze direction (e.g., as determined by the eye tracker325), a head direction (e.g., as determined by a head tracking sensor), or some combination thereof. Once the virtual object has been selected, the user may remove the virtual object from its cast target by performing a “pinch” gesture on the virtual object and moving their hand in a backwards (i.e. posterior) direction while pinching the virtual object. In some embodiments, the user may perform the “pinch” gesture to select and pinch the virtual object without having to first separately select the virtual object.
• FIG. 7 illustrates a flowchart of an example process for casting a virtual objects using a dartboard gesture, in accordance with some embodiments. TheNED system300 determines710 a set of positions of a hand of the user of theNED305, with the set of positions corresponding to a first backwards motion that may be part of a dartboard gesture as described above. The first motion may also comprise the user “pinching” a particular virtual object displayed by theNED305 in an AR environment.
• TheNED system300 determines720 a location in the AR environment based upon the determined hand positions of the user. In some embodiments, the location is determined based upon a trajectory of the first motion performed by the user, and may correspond to a cast target or a location on a surface within the local area.
• TheNED system300 identifies730 a second motion performed by the user, based upon the determined hand positions of the user. The second motion may correspond to the forward motion of a dartboard gesture as described above.
• TheNED system300 manipulates740 the AR environment at the determined location, based upon the identified first and second gestures (which collectively form a dartboard gesture). In some embodiments, theNED system300, in response to identifying a dartboard gesture, displays a virtual objects at the determined location (e.g., within a cast target).
Other Gestures
• FIGS. 8A and 8B illustrate an example of other types of gestures that may be performed by the user of an NED system to manipulate content items in the AR environment displayed by the NED system, in accordance with some embodiments.FIG. 8A illustrates an AR environment where the user is “holding” a content item, in accordance with some embodiments. As illustrated inFIG. 8A, the user may “hold” acontent item815 by performing a “pinch” gesture with herhands805 or810 to hold the content item815 (e.g., a virtual object). When the user holds thecontent item815, the NED system displays thecontent item815 at a location corresponding to the fingers of the user'shands805 or810 such that a portion of thecontent item815 is displayed between the distal ends of the user's thumb and index finger, indicating that thecontent item815 is being held by the user. In some embodiments, the user performs a “pinch” gesture with both hands, whereupon thecontent item815 is displayed such that a respective portion of thecontent item815 is located between the thumb and index fingers of each of the user's hands. If the user stops performing the “pinch” gesture with one hand, thecontent item815 may be displayed to continue to be held by the user's other hand, and displayed such that its position relative to the user's other hand remains unchanged relative to the user's other hand moves, as long as the user maintains the “pinch” gesture with their other hand.
• TheNED system300 displays one or morevirtual interface elements820 within the AR environment that may be used by the user to manipulate thecontent item815. In some embodiments, thevirtual interface elements820 are displayed on a dorsal surface of thewrist825 of the user, and may comprise elements that, when interacted with by the user's other hand (e.g., hand810), delete thecontent item815, display additional data relating to thecontent item815, and/or perform other functions on thecontent item815. In some embodiments, the user may use theirhand810 to interact with one of thevirtual interface elements820 by performing a “push” gesture, in which the user extends an index finger of theirhand810 along a path that moves the distal end of the user's index finger to within a threshold distance to avirtual interface element820, with the index finger remaining within the threshold distance for at least a threshold period of time. In some embodiments, thevirtual interface elements820 are displayed on the surfaces of the wrists of both hands of the user, allowing the user to be able to manipulate thevirtual interface elements820 with either hand.
• In some embodiments, thevirtual interface elements820 are displayed only when the user is “holding” a content item as described above, such that the operations associated with thevirtual interface elements820 will, when selected by the user, be performed on thecontent item815 currently being held by the user. Thevirtual interface elements820 may comprise a first virtual interface element, shown as an “X” in the figure, that when interacted with by the user using a gesture, causes theNED system300 to delete thecontent item815 from the AR environment, and a second virtual interface element, shown as the label “Detail”, in the figure, that when interacted with by the user using a gesture, causes theNED system300 to display supplemental data associated with thecontent item815.
• FIG. 8B illustrates the AR environment where the user manipulates the heldcontent item815 using thevirtual interface elements820, according to one or more embodiments. As discussed above, the user may continue holding thecontent item815 by continuing to maintain the “pinch” gesture with the onehand805, while theirother hand810 releases thecontent item815. The user may move herother hand810 to a location in the AR environment corresponding to one of thevirtual interface elements820. TheNED system300 may, in response to detecting that a distal end of the user's extended index finger of theirhand810 correspond to one of thevirtual interface elements820, perform the action corresponding to thevirtual interface element820 on the heldcontent item815. For example, the user may activate the virtual interface element labelled “Details,” causing theNED system300 to display a content item830 (e.g., a virtual object) adjacent to the heldcontent item815 that comprises an image displaying data associated with thecontent item815. As illustrated inFIG. 8B, thecontent item830 may be displayed on a common plane as thecontent item815.
• As such, the user is able to select and manipulate various content items in the AR environment intuitively through the use of gestures. For example, the user may select a virtual object corresponding to a content item by pinching or grabbing it within the AR environment, cast the virtual object onto different locations of the AR environment, share the content associated with the virtual object with other users, and/or manipulate the virtual object using one or more virtual interface elements.
• FIGS. 9A and 9B illustrate an example of a user performing a slice gesture in order to access a menu in the AR environment, in accordance with some embodiments. In some embodiments, a menu containing interactive virtual interface elements is not initially displayed in the AR environment, but can be displayed in response to one or more predetermined gestures performed by the user. For example, a menu associated with a virtual object is not initially displayed with the virtual object in order to reduce clutter around the virtual object in the AR environment. In some embodiments, a menu may not be associated with any particular virtual objects. For example, a menu may comprise an album of content items from which the user may select and cast into the AR environment as virtual objects. The menu may initially not be displayed in the AR environment, and instead is displayed in response to the user performing a predetermined gesture, allowing for the user to control when the menu is displayed in the AR environment. As used herein, a “menu” may refer to a set of interactive virtual interface elements arranged in a preconfigured format, the interactive virtual interface elements usable by the user to perform operations or manipulations on a virtual object within the AR environment.
• FIG. 9A illustrates the user performing a “slice” gesture to cause a menu to be displayed in the AR environment, in accordance with some embodiments. A “slice” gesture may comprise the user'shand905 moving over a substantially linear path910 extending in a particular direction (e.g., a downward direction relative to the user's body, as illustrated in FIG.9A) over at least a threshold distance and at least a threshold rate. For example, a slice gesture may require the user's hand to move over a downwards trajectory spanning a distance of at least9 inches over no more than half a second.
• In some embodiments, in order to perform the “slice” gesture, the fingers of the user's hand may remain in a particular position throughout the performance of the gesture. For example, the user may be required to extend at least two fingers915 (e.g., the index and middle fingers) of thehand905, where thefingers915 are held together by the user throughout the performance of the gesture. In some embodiments, the path910 is defined by a movement along a point at the tip of the user'sextended fingers915.
• In some embodiments, the user may first select a virtual object displayed in the AR environment (e.g., based upon a gaze direction or a previous gesture) before performing the “slice” gesture, whereupon the NED system displays a menu corresponding to the selected virtual object responsive to performance of the gesture. In embodiments where the menu is not associated with a displayed virtual object, the user may first select the menu (e.g., using a mobile device, through interactions with a different menu, and/or the like) before performing the “slice” gesture, which causes the selected menu to be displayed in the AR environment.
• FIG. 9B illustrates a menu containing interface virtual interface elements being displayed to the user following performance of the “slice” gesture, in accordance with some embodiments. Once the user has performed the “slice” gesture, theNED system300 displays themenu920 at a location in the AR environment associated with the performed gesture. For example, themenu920 may be displayed such that a center line of themenu920 substantially coincides with the path910 of the performed “slice” gesture.
• Themenu920 comprises a plurality of interactivevirtual elements925. The interactivevirtual elements925 may comprise buttons, dials, and or other elements that may be manipulated by the user in the AR environment. For example, as discussed above, the user may interact with a displayed button element by moving a hand moving to a location associated with the button element to press the button.
• In some embodiments, the interactivevirtual elements925 may comprise one or more content items. For example, as illustrated inFIG. 9B, themenu920 may comprise an album of content items that the user may view, interact with, and/or cast into the AR environment. For example, the user may use a “pinch” gesture to select a particular content item, which the user may then cast onto a cast target as discussed above. In other embodiments, the user may perform a “pinch” gesture to select a particular content item and move the content item within the AR environment. When the user stops performing the “pinch” gesture, the content item may be displayed as a virtual object at the location it was in when the user stopped performing the “pinch” gesture, at a location of a nearest cast target or surface, and/or the like.
Additional Configuration Information
• The foregoing description of the embodiments of the disclosure has been presented for the purpose of illustration; it is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Persons skilled in the relevant art can appreciate that many modifications and variations are possible in light of the above disclosure.
• Some portions of this description describe the embodiments of the disclosure in terms of algorithms and symbolic representations of operations on information. These algorithmic descriptions and representations are commonly used by those skilled in the data processing arts to convey the substance of their work effectively to others skilled in the art. These operations, while described functionally, computationally, or logically, are understood to be implemented by computer programs or equivalent electrical circuits, microcode, or the like. Furthermore, it has also proven convenient at times, to refer to these arrangements of operations as modules, without loss of generality. The described operations and their associated modules may be embodied in software, firmware, hardware, or any combinations thereof.
• Any of the steps, operations, or processes described herein may be performed or implemented with one or more hardware or software modules, alone or in combination with other devices. In one embodiment, a software module is implemented with a computer program product comprising a computer-readable medium containing computer program code, which can be executed by a computer processor for performing any or all of the steps, operations, or processes described.
• Embodiments of the disclosure may also relate to an apparatus for performing the operations herein. This apparatus may be specially constructed for the required purposes, and/or it may comprise a general-purpose computing device selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a non-transitory, tangible computer readable storage medium, or any type of media suitable for storing electronic instructions, which may be coupled to a computer system bus. Furthermore, any computing systems referred to in the specification may include a single processor or may be architectures employing multiple processor designs for increased computing capability.
• Embodiments of the disclosure may also relate to a product that is produced by a computing process described herein. Such a product may comprise information resulting from a computing process, where the information is stored on a non-transitory, tangible computer readable storage medium and may include any embodiment of a computer program product or other data combination described herein.
• Finally, the language used in the specification has been principally selected for readability and instructional purposes, and it may not have been selected to delineate or circumscribe the inventive subject matter. It is therefore intended that the scope of the disclosure be limited not by this detailed description, but rather by any claims that issue on an application based hereon. Accordingly, the disclosure of the embodiments is intended to be illustrative, but not limiting, of the scope of the disclosure, which is set forth in the following claims.

Claims (20)

What is claimed is:

1. A system comprising:

a near eye display (NED) comprising:

an optical assembly having an electronic display that is configured to display images in accordance with display instructions;

an imaging device configured to capture one or more images of a portions of a local area of the NED; and

a controller configured to:

determine that a first hand of a user of the NED located within the local area is performing a slice gesture, using the captured one or more images;

responsive to the slice gesture,

display a first virtual menu at a position along a trajectory of the slice gesture, using the electronic display of the NED.

2. The system ofclaim 1, wherein the trajectory is a linear path that extends in a particular direction.

3. The system ofclaim 2, wherein the particular direction is a downward direction relative to a body of the user.

4. The system ofclaim 2, wherein the linear path spans a distance of at least nine inches and the first hand moved through the linear path in at most a half a second.

5. The system ofclaim 1, wherein the slice gestured corresponds to one or more fingers of the first hand remaining in a particular position as the first hand moves through the trajectory of the slice gesture.

6. The system ofclaim 5, wherein the particular position corresponds to at least two fingers of the first hand being extended and held together.

7. The system ofclaim 6, wherein the at least two fingers are an index and middle finger.

8. The system ofclaim 1, wherein a centerline of the first virtual menu coincides with the trajectory of the slice gesture.

9. A method comprising:

capturing, via an imaging device, one or more images of a portions of a local area of a near eye display (NED);

determining that a first hand of a user of the NED located within the local area is performing a slice gesture, using the captured one or more images; and

responsive to the slice gesture, displaying using the electronic display of the NED a first virtual menu at a position along a trajectory of the slice gesture.

10. The method ofclaim 9, wherein the trajectory is a linear path that extends in a particular direction.

11. The method ofclaim 10, wherein the particular direction is a downward direction relative to a body of the user.

12. The method ofclaim 10, wherein the linear path spans a distance of at least nine inches and the first hand moved through the linear path in at most a half a second.

13. The method ofclaim 9, wherein the slice gestured corresponds to one or more fingers of the first hand remaining in a particular position as the first hand moves through the trajectory of the slice gesture.

14. The method ofclaim 13, wherein the particular position corresponds to at least two fingers of the first hand being extended and held together.

15. The method ofclaim 14, wherein the at least two fingers are an index and middle finger.

16. The method ofclaim 9, wherein a centerline of the first virtual menu coincides with the trajectory of the slice gesture.

17. A non-transitory computer readable medium storing instructions that, when executed by a processor of a near-eye display (NED), cause the NED to perform steps comprising:

capturing, via an imaging device, one or more images of a portions of a local area of the NED;

determining that a first hand of a user of the NED located within the local area is performing a slice gesture, using the captured one or more images; and

responsive to the slice gesture, displaying using the electronic display of the NED a first virtual menu at a position along a trajectory of the slice gesture.

18. The computer readable medium ofclaim 17, wherein the slice gestured corresponds to one or more fingers of the first hand remaining in a particular position as the first hand moves through the trajectory of the slice gesture.

19. The computer readable medium ofclaim 18, wherein the particular position corresponds to at least two fingers of the first hand being extended and held together.

20. The computer readable medium ofclaim 17, wherein a centerline of the first virtual menu coincides with the trajectory of the slice gesture.

Images