US20190391391A1 - Methods and apparatuses for providing input for head-worn image display devices - Google Patents

Abstract

An apparatus for use with an image display device configured for head-worn by a user, includes: a screen; and a processing unit configured to assign a first area of the screen to sense finger-action of the user; wherein the processing unit is configured to generate an electronic signal to cause a change in a content displayed by the display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus.

Description

RELATED APPLICATION DATA
• The present application claims benefit under 35 U.S.C. § 119 to U.S. Provisional Patent Application Ser. No. 62/688,108 filed on Jun. 21, 2018, entitled “METHODS AND APPARATUSES FOR PROVIDING INPUT FOR HEAD-WORN IMAGE DISPLAY DEVICES,” which is hereby incorporated by reference into the present application in its entirety.
INCORPORATION BY REFERENCE
• The following applications are expressly incorporated by reference in their entireties:
• U.S. patent application Ser. No. 15/968,673 filed on May 1, 2018, published on Nov. 1, 2018 as U.S. Patent Application Publication No. 2018/0315248,
  U.S. patent application Ser. No. 15/965,702 filed on Apr. 27, 2018, published on Nov. 1, 2018 as U.S. Patent Application Publication No. 2018/0314406,
  U.S. Provisional Patent Application No. 62/610,101 filed on Dec. 22, 2017, U.S. patent application Ser. No. 16/224,719 filed on Dec. 18, 2018,
  “DISPLAY PANEL OR PORTION THEREOF WITH A TRANSITIONAL GRAPHICAL USER INTERFACE” having attorney-docket No. ML-0678USDES1, filed concurrently, and
  “DISPLAY PANEL OR PORTION THEREOF WITH A GRAPHICAL USER INTERFACE” having attorney-docket No. ML-0678USDES2, filed concurrently.
FIELD
• The present disclosure relates to head-worn image display devices, and methods and apparatus for providing input for such image display devices.
BACKGROUND
• Modern computing and display technologies have facilitated the development of “mixed reality” (MR) systems for so called “virtual reality” (VR) or “augmented reality” (AR) experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A VR scenario typically involves presentation of digital or virtual image information without transparency to actual real-world visual input. An AR scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the real world around the user (i.e., transparency to real-world visual input). Accordingly, AR scenarios involve presentation of digital or virtual image information with transparency to the real-world visual input.
• MR systems may generate and display color data, which increases the realism of MR scenarios. Many of these MR systems display color data by sequentially projecting sub-images in different (e.g., primary) colors or “fields” (e.g., Red, Green, and Blue) corresponding to a color image in rapid succession. Projecting color sub-images at sufficiently high rates (e.g., 60 Hz, 120 Hz, etc.) may deliver a smooth color MR scenario in a user's mind.
• Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility patent application Ser. No. 14/555,585 filed on Nov. 27, 2014 (attorney docket number ML.20011.00), the contents of which are hereby expressly and fully incorporated by reference in their entirety, as though set forth in full.
• MR systems may employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the user's head moves. If the user's head motions are detected by the display device, the data being displayed can be updated (e.g., “warped”) to take the change in head pose (i.e., the orientation and/or location of user's head) into account.
• As an example, if a user wearing a head-worn display device views a virtual representation of a virtual object on the display and walks around an area where the virtual object appears, the virtual object can be rendered for each viewpoint, giving the user the perception that they are walking around an object that occupies real space. If the head-worn display device is used to present multiple virtual objects, measurements of head pose can be used to render the scene to match the user's dynamically changing head pose and provide an increased sense of immersion.
• Head-worn display devices that enable AR provide concurrent viewing of both real and virtual objects. With an “optical see-through” display, a user can see through transparent (or semi-transparent) elements in a display system to view directly the light from real objects in an environment. The transparent element, often referred to as a “combiner,” superimposes light from the display over the user's view of the real world, where light from by the display projects an image of virtual content over the see-through view of the real objects in the environment. A camera may be mounted onto the head-worn display device to capture images or videos of the scene being viewed by the user.
• Current optical systems, such as those in MR systems, optically render virtual content. Content is “virtual” in that it does not correspond to real physical objects located in respective positions in space. Instead, virtual content only exist in the brains (e.g., the optical centers) of a user of the head-worn display device when stimulated by light beams directed to the eyes of the user.
• Methods and apparatuses for providing input for head-worn image display devices (e.g., MR devices, AR devices, VR devices, etc.) are disclosed herein.
SUMMARY
• An apparatus for use with an image display device configured for head-worn by a user, includes: a screen; and a processing unit configured to assign a first area of the screen to sense finger-action of the user; wherein the processing unit is configured to generate an electronic signal to cause a change in a content displayed by the image display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus. As used in this specification, the term “finger-action” may include action performed by one or more fingers, and may include actions such as swiping, pinching, un-pinching, tapping, pressing, holding, twisting, turning, etc.
• Optionally, the screen has a touch-sensitive region, and wherein the assigned first area is less than a total area of the touch-sensitive region.
• Optionally, the assigned first area has a shape that corresponds with a shape of a screen of the image display device.
• Optionally, the assigned first area has a dimension based on a brand and/or a model of the apparatus.
• Optionally, the assigned first area has a dimension that is based on a feature of the apparatus and a feature of the image display device.
• Optionally, the processing unit is configured to ignore input generated by the user using a portion of the touch-sensitive region that is not a part of the assigned first area, and that is not a part of an assigned button.
• Optionally, the processing unit is also configured to operate a feedback component in response to the finger-action of the user.
• Optionally, the first area has a boundary, and wherein the finger-action of the user comprises a finger of the user crossing, reaching, or moving to a location that is within a prescribed distance from, the boundary.
• Optionally, the first area has one or more boundaries that at least partially surround a reference location, and wherein the processing unit is configured to operate the feedback component in response to a finger of the user reaching a prescribed distance from the reference location.
• Optionally, the processing unit is configured to operate the feedback component to generate different types of feedback based on different respective spatial relationships between one or more finger(s) of the user with respect to the first area.
• Optionally, the different types of feedback comprise a first haptic impulse with a first amplitude, and a second haptic impulse with a second amplitude that is different from the first amplitude.
• Optionally, the different types of feedback comprise a first number of haptic impulse(s), and a second number of haptic impulse(s) that is different from the first number.
• Optionally, the different types of feedback comprise a first series of haptic impulses with a first frequency, and a second series of haptic impulses with a second frequency that is different from the first frequency.
• Optionally, the different respective spatial relationships comprise different distances between (1) one of the one or more finger(s), or a point that is between two fingers, of the user and (2) a reference location within the assigned first area.
• Optionally, the reference location comprises a center of the assigned first area.
• Optionally, the different distances exceed a threshold.
• Optionally, the different respective spatial relationships comprise one of the one or more finger(s), or a point that is between two fingers, reaching different respective distances from a boundary of the assigned first area.
• Optionally, the assigned first area has a first boundary and a second boundary; wherein the different types of the feedback comprise at least a first type of feedback and a second type of feedback; wherein the processing unit is configured to operate the feedback component to generate the first type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the first boundary; and wherein the processing unit is configured to operate the feedback component to generate the second type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the second boundary.
• Optionally, the first boundary comprises a left or right boundary, and the second boundary comprises a top or bottom boundary, of the assigned first area.
• Optionally, the processing unit is configured to operate the feedback component based on a swiping direction.
• Optionally, the processing unit is configured to obtain an input signal associated with a pinching or un-pinching action performed on the assigned first area of the screen.
• Optionally, the processing unit is configured to generate the electronic signal to cause a size of the content displayed by the image display device to change based on the input signal that is associated with the pinching or un-pinching action.
• Optionally, the apparatus further includes an orientation sensor for sensing an orientation of the apparatus, wherein the processing unit is also configured to generate the electronic signal to cause the content displayed by the image display device to change based on the input signal associated with the pinching or un-pinching action and the sensed orientation of the apparatus.
• Optionally, the processing unit is configured to generate the electronic signal to cause the content to contract or expand in a first plane if the pinching or un-pinching action is sensed by the apparatus while the apparatus is at a first orientation; and wherein the processing unit is configured to generate the electronic signal to cause the content to contract or expand in a second plane if the pinching or un-pinching action is sensed by the apparatus while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane.
• Optionally, the apparatus has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°.
• Optionally, the apparatus has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.
• Optionally, the first plane comprises a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• Optionally, the first plane and the second plane are with respect to a virtual three-dimensional environment.
• Optionally, the first plane is perpendicular to the second plane.
• Optionally, the processing unit is configured to obtain an input signal associated with a swiping action performed on the assigned first area of the screen.
• Optionally, the processing unit is configured to generate the electronic signal to cause the content to change by moving the content in response to the input signal associated with the swiping action.
• Optionally, the apparatus further includes an orientation sensor for sensing an orientation of the apparatus, wherein the processing unit is configured to generate the electronic signal to cause the content displayed by the image display device to change based on the input signal associated with the swiping action and the sensed orientation of the apparatus.
• Optionally, the processing unit is configured to generate the electronic signal to cause the content to move in a first plane if the swiping action is sensed by the apparatus while the apparatus is at a first orientation; and wherein the processing unit is configured to generate the electronic signal to cause the content to move in a second plane if the swiping action is sensed by the apparatus while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane.
• Optionally, the apparatus has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°.
• Optionally, the apparatus has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.
• Optionally, the first plane comprises a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• Optionally, the first plane and the second plane are with respect to a virtual three-dimensional environment.
• Optionally, the first plane is perpendicular to the second plane.
• Optionally, the content is in a virtual three-dimensional environment, and wherein the processing unit is configured to generate the electronic signal to cause the content displayed by the image display device to change by moving the content closer to or further from the user when the swiping action is sensed by the apparatus while the orientation of the apparatus is approximately parallel to a horizontal plane.
• Optionally, the content is in a virtual three-dimensional environment, and wherein the processing unit is configured to generate the electronic signal to cause the content displayed by the image display device to move in a vertical plane in the three-dimensional environment when the swiping action is sensed by the apparatus while the orientation of the apparatus is approximately perpendicular to a horizontal plane.
• Optionally, the apparatus further includes an orientation sensor for sensing an orientation of the apparatus, wherein the processing unit is configured to generate the electronic signal based on the sensed orientation of the apparatus.
• Optionally, the processing unit is configured to generate the electronic signal to cause the content to expand in one or more directions based on the sensed orientation of the apparatus.
• Optionally, the processing unit is configured to generate the electronic signal to cause the content to rotate based on the sensed orientation of the apparatus.
• Optionally, the processing unit is configured to generate the electronic signal to cause the content to move based on the sensed orientation of the apparatus.
• Optionally, the apparatus further includes a movement sensor for sensing a movement of the apparatus, wherein the processing unit is configured to generate the electronic signal to cause the content displayed by the image display device to change based on the sensed movement of the apparatus.
• Optionally, the processing unit is configured to generate the electronic signal to cause the content to change by moving the content based on the sensed movement of the apparatus.
• Optionally, the apparatus is a handheld apparatus.
• Optionally, the handheld apparatus comprises a cell phone, a smart phone, a personal-digital-assistant (PDA), or a tablet.
• Optionally, the assigned first area of the screen has no displayed object while the assigned first area of the screen is sensing the finger-action of the user.
• Optionally, the processing unit is configured to operate the screen to display a grid of dots in the assigned first area of the screen.
• Optionally, the processing unit is also configured to change a feature of one or more of the dots in response to the user touching a part of the assigned first area of the screen where the one or more of the dots are displayed.
• Optionally, the processing unit is also configured to assign a second area of the screen as a first button.
• Optionally, the first button is a “HOME” button.
• Optionally, the processing unit is also configured to assign a third area of the screen as a second button.
• Optionally, the second button is a “TOGGLE” button.
• Optionally, the processing unit is also configured to assign a fourth area of the screen as a third button.
• Optionally, the third button is a “BUMPER” button.
• Optionally, the processing unit is also configured to assign a second area of the screen as a keyboard activation button, and wherein the processing unit is configured to operate the screen to display a keyboard in response to the user touching the assigned second area of the screen.
• Optionally, the apparatus further includes a wireless receiver for communication with the image display device.
• Optionally, the apparatus further includes a connector for communication with the image display device via a cable.
• Optionally, the apparatus further includes a non-transitory medium storing a set of instruction, an execution of which will cause the processing unit to assign the first area of the screen for sensing finger-action of the user.
• Optionally, the change in the content comprises a change in a size of the content, a change in a position of the content, a change in a shape of the content, a change in a color of the content, a replacement of information in the content, an increase or decrease in a quantity of information in the content, or any combination of the foregoing.
• A method includes: assigning a first area of a screen of an apparatus to sense finger-action of a user of an image display device, wherein the image display device is configured for head-worn by the user, and wherein the apparatus is different from the image display device; generating an electronic signal to cause a change in a content displayed by the image display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus.
• Optionally, the screen has a touch-sensitive region, and wherein the assigned first area is less than a total area of the touch-sensitive region.
• Optionally, the assigned first area has a shape that corresponds with a shape of a screen of the image display device.
• Optionally, the assigned first area has a dimension based on a brand and/or a model of the apparatus.
• Optionally, the assigned first area has a dimension that is based on a feature of the apparatus and a feature of the image display device.
• Optionally, the method further includes ignoring input generated by the user using a portion of the touch-sensitive region that is not a part of the assigned first area, and that is not a part of an assigned button.
• Optionally, the method further includes generating a control signal to operate a feedback component in response to the finger-action of the user.
• Optionally, the first area has a boundary, and wherein the finger-action of the user comprises a finger of the user crossing, reaching, or moving to a location that is within a prescribed distance from, the boundary.
• Optionally, the first area has one or more boundaries that at least partially surround a reference location, and wherein the control signal is for operating the feedback component in response to a finger of the user reaching a prescribed distance from the reference location.
• Optionally, the method further includes generating different control signals to operate a feedback component to generate different respective types of feedback based on different respective spatial relationships between one or more finger(s) of the user with respect to the first area.
• Optionally, the different types of feedback comprise a first haptic impulse with a first amplitude, and a second haptic impulse with a second amplitude that is different from the first amplitude.
• Optionally, the different types of feedback comprise a first number of haptic impulse(s), and a second number of haptic impulse(s) that is different from the first number.
• Optionally, the different types of feedback comprise a first series of haptic impulses with a first frequency, and a second series of haptic impulses with a second frequency that is different from the first frequency.
• Optionally, the different respective spatial relationships comprise different distances between (1) one of the one or more finger(s), or a point that is between two fingers, of the user and (2) a reference location within the assigned first area.
• Optionally, the reference location comprises a center of the assigned first area.
• Optionally, the different distances exceed a threshold.
• Optionally, the different respective spatial relationships comprise one of the one or more finger(s), or a point that is between two fingers, reaching different respective distances from a boundary of the assigned first area.
• Optionally, the assigned first area has a first boundary and a second boundary; wherein the different types of the feedback comprise at least a first type of feedback and a second type of feedback; wherein the feedback component is operated to generate the first type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the first boundary; and wherein the feedback component is operated to generate the second type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the second boundary.
• Optionally, the first boundary comprises a left or right boundary, and the second boundary comprises a top or bottom boundary, of the assigned first area.
• Optionally, the control signal for operating the feedback component is based on a swiping direction.
• Optionally, the method further includes receiving an input signal associated with a pinching or un-pinching action performed by the user on the assigned first area of the screen.
• Optionally, the electronic signal is for changing a size of the content displayed by the image display device in response to the input signal that is associated with the pinching or un-pinching action.
• Optionally, the method further includes obtaining an orientation of the apparatus from an orientation sensor, wherein the electronic signal is for changing the content displayed by the image display device based on the input signal that is associated with the pinching or un-pinching action and the orientation of the apparatus.
• Optionally, the content is changed by contracting or expanding the content in a first plane if the pinching or un-pinching action is sensed by the assigned first area while the apparatus is at a first orientation; and wherein the content is changed by contracting or expanding the content in a second plane if the pinching or un-pinching action is sensed by the assigned first area while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane.
• Optionally, the apparatus has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°.
• Optionally, the apparatus has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.
• Optionally, the first plane comprises a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• Optionally, the first plane and the second plane are with respect to a virtual three-dimensional environment.
• Optionally, the first plane is perpendicular to the second plane.
• Optionally, the method further includes receiving an input signal associated with a swiping action performed by the user on the assigned first area of the screen.
• Optionally, the electronic signal is for moving the content displayed by the image display device in response to the sensed swiping action.
• Optionally, the method further includes obtaining an orientation of the apparatus from an orientation sensor, wherein the electronic signal is for changing the content displayed by the image display device based on the input signal that is associated with the swiping action and the orientation of the apparatus.
• Optionally, the content is changed by moving the content in a first plane if the swiping action is sensed by the assigned first area while the apparatus is at a first orientation; and wherein the content is changed by moving the content in a second plane if the swiping action is sensed by the assigned first area while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane.
• Optionally, the apparatus has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°.
• Optionally, the apparatus has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.
• Optionally, the first plane comprises a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• Optionally, the first plane and the second plane are with respect to a virtual three-dimensional environment.
• Optionally, the first plane is perpendicular to the second plane.
• Optionally, the content is in a virtual three-dimensional environment, and wherein the electronic signal is for causing the content displayed by the image display device to move closer to or further from the user when the swiping action is sensed by the assigned first area while the orientation of the apparatus is approximately parallel to a horizontal plane.
• Optionally, the content is in a virtual three-dimensional environment, and wherein the electronic signal is for causing the content displayed by the image display device to move in a vertical plane in the three-dimensional environment when the swiping action is sensed by the assigned first area while the orientation of the apparatus is approximately perpendicular to a horizontal plane.
• Optionally, the method further includes obtaining a sensor input indicating a sensed orientation of the apparatus, wherein the electronic signal is for changing the content displayed by the image display device based on the sensor input indicating the sensed orientation of the apparatus.
• Optionally, the electronic signal is for changing the content by expanding the content in one or more directions based on the sensor input indicating the sensed orientation of the apparatus.
• Optionally, the electronic signal is for changing the content by rotating the content based on the sensor input indicating the sensed orientation of the apparatus.
• Optionally, the electronic signal is for changing the content by moving the content based on the sensor input indicating the sensed orientation of the apparatus.
• Optionally, the method further includes obtaining a sensor input indicating a sensed movement of the apparatus, wherein the electronic signal is for changing the content displayed by the image display device based on the sensor input indicating the sensed movement of the apparatus.
• Optionally, the electronic signal is for changing the content by moving the content based on the sensor input indicating the sensed movement of the apparatus.
• Optionally, the apparatus is a handheld apparatus.
• Optionally, the handheld apparatus comprises a cell phone, a smart phone, a personal-digital-assistant (PDA), or a tablet.
• Optionally, the assigned first area of the screen has no displayed object while the assigned first area of the screen is sensing the finger-action of the user.
• Optionally, the method further includes operating the screen to display a grid of dots in the assigned first area of the screen.
• Optionally, the method further includes changing a feature of one or more of the dots in response to the user touching a part of the assigned first area of the screen where the one or more of the dots are displayed.
• Optionally, the method further includes assigning a second area of the screen as a first button.
• Optionally, the first button is a “HOME” button.
• Optionally, the method further includes assigning a third area of the screen as a second button.
• Optionally, the second button is a “TOGGLE” button.
• Optionally, the method further includes assigning a fourth area of the screen as a third button.
• Optionally, the third button is a “BUMPER” button.
• Optionally, the method further includes: assigning a second area of the screen as a keyboard activation button, and operating the screen to display a keyboard in response to the user touching the assigned second area of the screen.
• Optionally, the method further includes wirelessly communicating with the image display device.
• Optionally, the method further includes communicating with the image display device via a cable.
• Optionally, the apparatus comprises a non-transitory medium storing an instruction, and wherein the act of assigning the first area of the screen for sensing finger-action of the user is performed based on the instruction.
• Optionally, the change in the content comprises a change in a size of the content, a change in a position of the content, a change in a shape of the content, a change in a color of the content, a replacement of information in the content, an increase or decrease in a quantity of information in the content, or any combination of the foregoing.
• A product includes a non-transitory medium storing a set of instructions, an execution of which will cause a method to be performed, the method comprising: assigning a first area of a screen of an apparatus to sense finger-action of a user of an image display device, wherein the image display device is configured for head-worn by the user, and wherein the apparatus is different from the image display device; generating an electronic signal to cause a change in a content displayed by the image display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus.
• Optionally, the screen has a touch-sensitive region, and wherein the assigned first area is less than a total area of the touch-sensitive region.
• Optionally, the assigned first area has a shape that corresponds with a shape of a screen of the image display device.
• Optionally, the assigned first area has a dimension based on a brand and/or a model of the apparatus.
• Optionally, the assigned first area has a dimension that is based on a feature of the apparatus and a feature of the image display device.
• Optionally, the method further comprises ignoring input generated by the user using a portion of the touch-sensitive region that is not a part of the assigned first area, and that is not a part of an assigned button.
• Optionally, the method further comprises generating a control signal to operate a feedback component in response to the finger-action of the user.
• Optionally, the first area has a boundary, and wherein the finger-action of the user comprises a finger of the user crossing, reaching, or moving to a location that is within a prescribed distance from, the boundary.
• Optionally, the first area has one or more boundaries that at least partially surround a reference location, and wherein the control signal is for operating the feedback component in response to a finger of the user reaching a prescribed distance from the reference location.
• Optionally, the method further comprises generating different control signals to operate a feedback component to generate different respective types of feedback based on different respective spatial relationships between one or more finger(s) of the user with respect to the first area.
• Optionally, the different types of feedback comprise a first haptic impulse with a first amplitude, and a second haptic impulse with a second amplitude that is different from the first amplitude.
• Optionally, the different types of feedback comprise a first number of haptic impulse(s), and a second number of haptic impulse(s) that is different from the first number.
• Optionally, the different types of feedback comprise a first series of haptic impulses with a first frequency, and a second series of haptic impulses with a second frequency that is different from the first frequency.
• Optionally, the different respective spatial relationships comprise different distances between (1) one of the one or more finger(s), or a point that is between two fingers, of the user and (2) a reference location within the assigned first area.
• Optionally, the reference location comprises a center of the assigned first area.
• Optionally, the different distances exceed a threshold.
• Optionally, the different respective spatial relationships comprise one of the one or more finger(s), or a point that is between two fingers, reaching different respective distances from a boundary of the assigned first area.
• Optionally, the assigned first area has a first boundary and a second boundary; wherein the different types of the feedback comprise at least a first type of feedback and a second type of feedback; wherein the feedback component is operated to generate the first type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the first boundary; and wherein the feedback component is operated to generate the second type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the second boundary.
• Optionally, the first boundary comprises a left or right boundary, and the second boundary comprises a top or bottom boundary, of the assigned first area.
• Optionally, the control signal for operating the feedback component is based on a swiping direction.
• Optionally, the method further comprises receiving an input signal associated with a pinching or un-pinching action performed by the user on the assigned first area of the screen.
• Optionally, the electronic signal is for changing a size of the content displayed by the image display device in response to the input signal that is associated with the pinching or un-pinching action.
• Optionally, the method further comprises obtaining an orientation of the apparatus from an orientation sensor, wherein the electronic signal is for changing the content displayed by the image display device based on the input signal that is associated with the pinching or un-pinching action and the orientation of the apparatus.
• Optionally, the content is changed by contracting or expanding the content in a first plane if the pinching or un-pinching action is sensed by the assigned first area while the apparatus is at a first orientation; and wherein the content is changed by contracting or expanding the content in a second plane if the pinching or un-pinching action is sensed by the assigned first area while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane.
• Optionally, the apparatus has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°.
• Optionally, the apparatus has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.
• Optionally, the first plane comprises a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• Optionally, the first plane and the second plane are with respect to a virtual three-dimensional environment.
• Optionally, the first plane is perpendicular to the second plane.
• Optionally, the method further comprises receiving an input signal associated with a swiping action performed by the user on the assigned first area of the screen.
• Optionally, the electronic signal is for moving the content displayed by the image display device in response to the sensed swiping action.
• Optionally, the method further comprises obtaining an orientation of the apparatus from an orientation sensor, wherein the electronic signal is for changing the content displayed by the image display device based on the input signal that is associated with the swiping action and the orientation of the apparatus.
• Optionally, the content is changed by moving the content in a first plane if the swiping action is sensed by the assigned first area while the apparatus is at a first orientation; and wherein the content is changed by moving the content in a second plane if the swiping action is sensed by the assigned first area while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane.
• Optionally, the apparatus has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°.
• Optionally, the apparatus has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.
• Optionally, the first plane comprises a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• Optionally, the first plane and the second plane are with respect to a virtual three-dimensional environment.
• Optionally, the first plane is perpendicular to the second plane.
• Optionally, the content is in a virtual three-dimensional environment, and wherein the electronic signal is for causing the content displayed by the image display device to move closer to or further from the user when the swiping action is sensed by the assigned first area while the orientation of the apparatus is approximately parallel to a horizontal plane.
• Optionally, the content is in a virtual three-dimensional environment, and wherein the electronic signal is for causing the content displayed by the image display device to move in a vertical plane in the three-dimensional environment when the swiping action is sensed by the assigned first area while the orientation of the apparatus is approximately perpendicular to a horizontal plane.
• Optionally, the method further comprises obtaining a sensor input indicating a sensed orientation of the apparatus, wherein the electronic signal is for changing the content displayed by the image display device based on the sensor input indicating the sensed orientation of the apparatus.
• Optionally, the electronic signal is for changing the content by expanding the content in one or more directions based on the sensor input indicating the sensed orientation of the apparatus.
• Optionally, the electronic signal is for changing the content by rotating the content based on the sensor input indicating the sensed orientation of the apparatus.
• Optionally, the electronic signal is for changing the content by moving the content based on the sensor input indicating the sensed orientation of the apparatus.
• Optionally, the method further comprises obtaining a sensor input indicating a sensed movement of the apparatus, wherein the electronic signal is for changing the content displayed by the image display device based on the sensor input indicating the sensed movement of the apparatus.
• Optionally, the electronic signal is for changing the content by moving the content based on the sensor input indicating the sensed movement of the apparatus.
• Optionally, the apparatus is a handheld apparatus.
• Optionally, the handheld apparatus comprises a cell phone, a smart phone, a personal-digital-assistant (PDA), or a tablet.
• Optionally, the assigned first area of the screen has no displayed object while the assigned first area of the screen is sensing the finger-action of the user.
• Optionally, the method further comprises operating the screen to display a grid of dots in the assigned first area of the screen.
• Optionally, the method further comprises changing a feature of one or more of the dots in response to the user touching a part of the assigned first area of the screen where the one or more of the dots are displayed.
• Optionally, the method further comprises assigning a second area of the screen as a first button.
• Optionally, the first button is a “HOME” button.
• Optionally, the method further comprises assigning a third area of the screen as a second button.
• Optionally, the second button is a “TOGGLE” button.
• Optionally, the method further comprises assigning a fourth area of the screen as a third button.
• Optionally, the third button is a “BUMPER” button.
• Optionally, the method further comprises: assigning a second area of the screen as a keyboard activation button, and operating the screen to display a keyboard in response to the user touching the assigned second area of the screen.
• Optionally, the method further comprises wirelessly communicating with the image display device.
• Optionally, the method further comprises communicating with the image display device via a cable.
• Optionally, the product includes instruction for assigning the first area of the screen for sensing finger-action of the user.
• Optionally, the change in the content comprises a change in a size of the content, a change in a position of the content, a change in a shape of the content, a change in a color of the content, a replacement of information in the content, an increase or decrease in a quantity of information in the content, or any combination of the foregoing.
• A computing device includes: a proximity-sensitive display; and one or more processors that are operatively coupled to the proximity-sensitive display and are communicatively coupled to a wearable computing system, the one or more processors configured to: receive, from the wearable computing system, data indicating whether the proximity-sensitive display is visible to a user of the wearable computing system; monitor the data that is received from the wearable computing system for changes in the user's visibility of the proximity-sensitive display; in response to a detection of a change in the user's visibility of the proximity-sensitive display, switch between: (i) a first presentation mode in which the one or more processors are configured to operate the proximity-sensitive display in a manner such that a first load is placed on a power supply of the computing device, and (ii) a second presentation mode in which the one or more processors are configured to operate the proximity-sensitive display in a manner such that a second load is placed on the power supply of the computing device, the second load being less than the first load.
• Optionally, the first presentation mode is one in which the one or more processors are configured to present a graphical user interface on the proximity-sensitive display.
• Optionally, the second presentation mode is one in which the one or more processors are configured to present a limited version of the graphical user interface on the proximity-sensitive display.
• Optionally, the first presentation mode is one in which the one or more processors are configured to operate the proximity-sensitive display such that content is presented at a first level of brightness, and the second presentation mode is one in which the one or more processors are configured to operate the proximity-sensitive display such that content is presented at a second level of brightness that is lower than the first level of brightness.
• Optionally, the computing device is configured to operate in a same or similar manner as one or more of the computing devices and/or apparatuses described herein.
• Optionally, the computing device is configured to perform one or more of the operations described in one or both of the methods described immediately below.
• A computer-implemented method includes: receiving, from a wearable computing system, a first set of data indicating whether a proximity-sensitive display is visible to a user of the wearable computing system at a first point in time; receiving, from the wearable computing system, a second set of data indicating whether the proximity-sensitive display is visible to the user of the wearable computing system at a second, later point in time; determining, based on receiving the first and second sets of data from the wearable computing system, that a change in the user's visibility of the proximity-sensitive display has occurred; and in response to determining that the change in the user's visibility of the proximity-sensitive display has occurred, switching between (i) a first presentation mode in which a graphical user interface is presented on the proximity-sensitive display, and (ii) a second presentation mode in which the proximity-sensitive display consumes less power than it does in the first presentation mode.
• Optionally, determining, based on receiving the first and second sets of data from the wearable computing system, that a change in the user's visibility of the proximity-sensitive display has occurred comprises: determining, based on receiving the first and second sets of data from the wearable computing system, that the user has lost visibility of the proximity-sensitive display.
• Optionally, in response to determining that the user has lost visibility of the proximity-sensitive display, the method comprises switching from the first presentation mode to the second presentation mode.
• Optionally, determining, based on receiving the first and second sets of data from the wearable computing system, that a change in the user's visibility of the proximity-sensitive display has occurred comprises: determining, based on receiving the first and second sets of data from the wearable computing system, that the user has regained visibility of the proximity-sensitive display.
• Optionally, in response to determining that the user has lost visibility of the proximity-sensitive display, the method comprises switching from the second presentation mode to the first presentation mode.
• Optionally, the method may be performed by the computing device described immediately above and/or the computing device described immediately below.
• A computing device includes: a proximity-sensitive display; a feedback component; and one or more processors that are operatively coupled to the proximity-sensitive display and the feedback component and are communicatively coupled to a wearable computing system, the one or more processors configured to: receive, from the wearable computing system, data indicating whether the proximity-sensitive display is visible to a user of the wearable computing system; monitor the data that is received from the wearable computing system for changes in the user's visibility of the proximity-sensitive display; in response to a detection of a change in the user's visibility of the proximity-sensitive display, switch between: (i) a first mode in which the one or more processors are configured to present a user interface on the proximity-sensitive display, and (ii) a second mode in the one or more processors are configured to convey one or more portions of the user interface to the user through use of the feedback component.
• Optionally, the feedback component is a haptic actuator.
• Optionally, the feedback component is a speaker.
• Optionally, the user interface is a graphical user interface for controlling one or more functions of the wearable computing system.
• Optionally, the graphical user interface a plurality of graphical control elements.
• Optionally, the second mode is one in which the one or more processors are configured to refrain from presenting one or more of the plurality of graphical control elements on the proximity-sensitive display.
• Optionally, the second mode is one in which the one or more processors are configured to present a limited version of the graphical user interface on the proximity-sensitive display.
• Optionally, the limited version of the graphical user interface on the proximity-sensitive display is one in which one or more of the plurality of graphical control elements are not shown, one in which brightness levels of one or more of the plurality of graphical control elements are reduced, or a combination thereof.
• Optionally, the second mode is one in which the one or more processors are configured to convey one or more locations on the proximity-sensitive display at which one or more of the plurality of graphical control elements as presented in the first mode.
• Optionally, the one or more locations on the proximity-sensitive display that are conveyed in the second mode correspond to one or more outer boundaries of the plurality of graphical control elements as presented in the first mode.
• Optionally, the plurality of graphical control elements correspond to a plurality of physical user input components of a dedicated input controller that is associated with the wearable computing system.
• Optionally, the computing device is configured to operate in a same or similar manner as one or more of the computing devices and/or apparatuses described herein.
• Optionally, the computing device is configured to perform one or more of the operations described in the method described immediately above and/or the method described immediately below.
• A computer-implemented method includes: receiving, from a wearable computing system, a first set of data indicating whether a proximity-sensitive display is visible to a user of the wearable computing system at a first point in time; receiving, from the wearable computing system, a second set of data indicating whether the proximity-sensitive display is visible to the user of the wearable computing system at a second, later point in time; determining, based on receiving the first and second sets of data from the wearable computing system, that a change in the user's visibility of the proximity-sensitive display has occurred; and in response to determining that the change in the user's visibility of the proximity-sensitive display has occurred, switching between (i) a first mode in which a user interface is presented on the proximity-sensitive display, and (ii) a second mode in which one or more portions of the user interface are conveyed to the user of the wearable computing system through use of a feedback component.
• Optionally, determining, based on receiving the first and second sets of data from the wearable computing system, that a change in the user's visibility of the proximity-sensitive display has occurred comprises: determining, based on receiving the first and second sets of data from the wearable computing system, that the user has lost visibility of the proximity-sensitive display.
• Optionally, in response to determining that the user has lost visibility of the proximity-sensitive display, the method comprises switching from the first mode to the second mode.
• Optionally, determining, based on receiving the first and second sets of data from the wearable computing system, that a change in the user's visibility of the proximity-sensitive display has occurred comprises: determining, based on receiving the first and second sets of data from the wearable computing system, that the user has regained visibility of the proximity-sensitive display.
• Optionally, in response to determining that the user has lost visibility of the proximity-sensitive display, the method comprises switching from the second mode to the first mode.
• Optionally, the method may be performed by one or both of the computing devices described immediately above.
• A wearable computing system includes: a head-mounted display configured to be worn on a head of a user; one or more sensing devices configured to monitor an environment of head-mounted display; and one or more processors that are operatively coupled to the head-mounted display and the one or more sensing devices and are communicatively coupled to a computing device, the one or more processors configured to: use data obtained from the one or more sensing devices to determine whether a display of the computing device is visible to the user of the wearable computing system; generate one or more messages based at least in part on whether the display of the computing device is determined to be visible to the user of the wearable computing system; and transmit the one or more messages to the computing device.
• Optionally, the one or more sensing devices include one or more cameras.
• Optionally, the one or more cameras include one or more forward-facing cameras that are configured to capture images of an environment of the user of the wearable computing system.
• Optionally, the one or more cameras include one or more inward-facing cameras that are configured to capture images of one or both of the user's eyes, one or more portions of the user's head or face, or a combination thereof.
• Optionally, the one or more sensing devices include one or more proximity sensors.
• Optionally, the wearable computing system is configured to operate in a same or similar manner as one or more of the wearable computing systems, the wearable display systems, and/or image display devices described herein.
• Optionally, the wearable computing system is configured to perform one or more of the operations in the method described immediately below.
• A computer-implemented method includes: obtaining data from one or more sensing devices of a wearable computing system; determining, based on receiving the data from the one or more sensing devices of the wearable computing system, whether a display of a computing device is visible to the user of the wearable computing system; generating one or more messages based on whether the display of the computing device is determined to be visible to the user of the wearable computing system; and transmitting the one or more messages to the computing device.
• Optionally, obtaining data from one or more sensing devices of a wearable computing system comprises: obtaining data from one or more forward-facing cameras that are configured to capture images of an environment of the user of the wearable computing system.
• Optionally, determining, based on receiving the data from the one or more sensing devices of the wearable computing system, whether a display of a computing device is visible to the user of the wearable computing system comprises: determining whether the display of the computing device is shown in one or more images captured by the one or more forward-facing cameras.
• Optionally, obtaining data from one or more sensing devices of a wearable computing system comprises: obtaining data from one or more inward-facing cameras that are configured to capture images of one or both of the user's eyes, one or more portions of the user's head or face, or a combination thereof.
• Optionally, determining, based on receiving the data from the one or more sensing devices of the wearable computing system, whether a display of a computing device is visible to the user of the wearable computing system comprises: determining, based on one or more images captured by the one or more inward-facing cameras, whether the wearable computing system is being worn by the user on the user's head.
• Optionally, determining, based on one or more images captured by the one or more inward-facing cameras, whether the wearable computing system is being worn by the user on the user's head comprises: determining whether the user is shown in one or more images captured by the one or more inward-facing cameras.
• Optionally, obtaining data from one or more sensing devices of a wearable computing system comprises: obtaining data from one or more proximity sensors.
• Optionally, determining, based on receiving the data from the one or more sensing devices of the wearable computing system, whether a display of a computing device is visible to the user of the wearable computing system comprises: determining whether the data received from the one or more proximity sensors indicate that the wearable computing system is being worn by the user on the user's head.
• Optionally, the method may be performed by the wearable computing system described immediately above.
• A computing device includes: a proximity-sensitive display; a feedback component; and one or more processors that are operatively coupled to the proximity-sensitive display and the feedback component and are communicatively coupled to a wearable computing system, the one or more processors configured to: receive data indicating a first touch input received at a first location on the proximity-sensitive display; receive data indicating a second touch input received at a second location on the proximity-sensitive display immediately subsequent the first touch input; determine whether each of the first and second locations fall within a particular region of the proximity-sensitive display; in response to a determination that one or both of the first and second locations fall within the particular region of the proximity-sensitive display: generate one or more messages based at least in part on one or both of the first and second touch inputs; and transmit the one or more messages generated based at least in part on one or both of the first and second touch inputs to the wearable computing system; and in response to (i) a determination that the first location falls within the particular region of the proximity-sensitive display, and (ii) a determination that the second location falls outside of the particular region of the proximity-sensitive display: cause the feedback component to generate feedback.
• Optionally, the particular region of the proximity-sensitive display is associated with a particular control element.
• Optionally, the particular control element is a graphical control element, and the one or more processors are further configured to display the particular control element at a location on the proximity-sensitive display coinciding with the particular region.
• Optionally, the feedback component is a haptic actuator.
• Optionally, the feedback component is a speaker.
• Optionally, the computing device is configured to operate in a same or similar manner as one or more of the computing devices and/or apparatuses described herein.
• Optionally, the computing device is configured to perform one or more of the operations in the method described immediately below.
• A computer-implemented method includes: receiving data indicating a first touch input received at a first location on a proximity-sensitive display; receiving data indicating a second touch input received at a second location on the proximity-sensitive display immediately subsequent the first touch input; determining whether each of the first and second locations fall within a particular region of the proximity-sensitive display; in response to determining that one or both of the first and second locations fall within the particular region of the proximity-sensitive display: generating one or more messages based at least in part on one or both of the first and second touch inputs; and transmitting the one or more messages generated based at least in part on one or both of the first and second touch inputs to a wearable computing system; and in response to (i) determining that the first location falls within the particular region of the proximity-sensitive display, and (ii) determining that the second location falls outside of the particular region of the proximity-sensitive display: providing feedback for output through a feedback component.
• Optionally, the method may be performed by the computing device described immediately above.
• A computing device includes: a proximity-sensitive display; an orientation sensor configured to sense an orientation of the computing device; and one or more processors that are operatively coupled to the proximity-sensitive display and the orientation sensor and are communicatively coupled to a wearable computing system, the one or more processors configured to: monitor for a sequence of touch inputs received through the proximity-sensitive display corresponding to any one of a plurality of different predefined gestures; in response to a detection of any one of the plurality of different predefined gestures: select, from among a plurality of different types of transformations that are associated with the plurality of different predefined gestures, respectively, a particular type of transformation that is associated with the detected gesture; select, from among multiple different axes, a particular axis based on data obtained from the orientation sensor; generate a command to apply the particular type of transformation to a virtual object relative to the particular axis; and transmit the command to the wearable computing system. Optionally, the computing device is configured to operate in a same or similar manner as one or more of the computing devices and/or apparatuses described herein.
• Optionally, the computing device is configured to perform one or more of the operations in the method described immediately below.
• A computer-implemented method includes: obtaining data indicating an orientation of a computing device; receiving data indicating a sequence of touch inputs received through a proximity-sensitive display of the computing device; determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to a particular gesture; and in response to determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to the particular gesture: selecting, from among a plurality of different types of transformations that are associated with a plurality of different predefined gestures, respectively, a particular type of transformation that is associated with the particular gesture; and selecting, from among multiple different axes, a particular axis based on the orientation of the device; and generating a command to apply the particular type of transformation to a virtual object relative to the particular axis.
• Optionally, the plurality of different types of transformations includes one or more of: rotation, translation, and resizing.
• Optionally, the method may be performed by the computing device described immediately above.
• A computing device includes: a proximity-sensitive display; and one or more processors that are operatively coupled to the proximity-sensitive display and are communicatively coupled to a wearable display system, the one or more processors configured to: present a particular piece of content on the proximity-sensitive display; monitor for a sequence of touch inputs received through the proximity-sensitive display corresponding to any one of a plurality of different predefined gestures during the presentation of the particular piece of content on the proximity-sensitive display; in response to a detection of any one of the plurality of different predefined gestures during the presentation of the particular piece of content on the proximity-sensitive display: generate one or more messages indicating that presentation of the particular piece of content is to be handed off to the wearable display system; transmit the one or more messages to the wearable display system; and discontinue presentation of the particular piece of content on the proximity-sensitive display.
• Optionally, the computing device is configured to operate in a same or similar manner as one or more of the computing devices and/or apparatuses described herein.
• Optionally, the computing device is configured to perform one or more of the operations in the method described immediately below.
• A computer-implemented method includes: presenting a particular piece of content on a proximity-sensitive display; while presenting the particular piece of content on the proximity-sensitive display, receiving data indicating a sequence of touch inputs received through the proximity-sensitive display; determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to a particular gesture; in response to determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to the particular gesture: generating one or more messages indicating that presentation of the particular piece of content is to be handed off to a wearable display system; transmitting the one or more messages to the wearable display system; and discontinuing presentation of the particular piece of content on the proximity-sensitive display.
• Optionally, determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to the particular gesture comprises: determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to a gesture in which the user has dragged one or more of their fingers from a location on the proximity-sensitive display that is associated with the particular piece of content to an edge of the proximity-sensitive display.
• Optionally, determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to the particular gesture comprises: determining that the sequence of touch inputs received through the proximity-sensitive display corresponds to a gesture in which the user has pinched two or more of their fingers together at a particular location on the proximity-sensitive display that is associated with the particular piece of content and subsequently moved their two or more pinched-together fingers away from the particular location.
• Optionally, a gesture in which the user has pinched two or more of their fingers together at a particular location on the proximity-sensitive display that is associated with the particular piece of content and subsequently moved their two or more pinched-together fingers away from the particular location comprises: a gesture in which the user has pinched two or more of their fingers together at a particular location on the proximity-sensitive display that is associated with the particular piece of content and subsequently moved their two or more pinched-together fingers to an edge of the proximity-sensitive display.
• Optionally, a gesture in which the user has pinched two or more of their fingers together at a particular location on the proximity-sensitive display that is associated with the particular piece of content and subsequently moved their two or more pinched-together fingers away from the particular location comprises: a gesture in which the user has pinched two or more of their fingers together at a particular location on the proximity-sensitive display that is associated with the particular piece of content and subsequently lifted their two or more pinched-together fingers off of the proximity-sensitive display.
• Optionally, presenting the particular piece of content on the proximity-sensitive display comprises: presenting a scene that includes the particular piece of content on the proximity-sensitive display.
• Optionally, discontinuing presentation of the particular piece of content on the proximity-sensitive display comprises: moving the particular piece of content out of the scene.
• Optionally, the method further comprises: while presenting the particular piece of content on the proximity-sensitive display, generating identification information for the particular piece of content, and transmitting the identification information for the particular piece of content to the wearable display system.
• Optionally, the method further comprises: while presenting the particular piece of content on the proximity-sensitive display, generating data indicating a current location of the particular piece of content relative to the proximity-sensitive display, and transmitting the data indicating the current location of the particular piece of content to the wearable display system.
• Optionally, the method may be performed by the computing device described immediately above.
• A wearable computing system includes: a head-mounted display configured to be worn on a head of a user; one or more cameras configured to capture images of an environment in front of the user; and one or more processors that are operatively coupled to the head-mounted display and the one or more cameras and are communicatively coupled to a computing device, the one or more processors configured to: identify a particular piece of content that is being presented on a display of the computing device; receive one or more messages from the computing device indicating that presentation of the particular piece of content is to be handed off from the display of the computing device to the head-mounted display; in response to the one or more messages received from the computing device: use one or more images captured by the one or more cameras to determine a location in the environment in front of the user; and present the particular piece of content on the head-mounted display in a manner so as to be perceived by the user as being positioned at the determined location in the environment in front of the user.
• Optionally, the wearable computing system is configured to operate in a same or similar manner as one or more of the wearable computing systems, the wearable display systems, and/or image display devices described herein.
• Optionally, the wearable computing system is configured to perform one or more of the operations in the method described immediately below.
• A computer-implemented method includes: identifying a particular piece of content that is being presented on a display of a computing device; receiving one or more messages from the computing device indicating that presentation of the particular piece of content is to be handed off from the display of the computing device to a head-mounted display; and in response to receiving the one or more messages received from the computing device: obtaining one or more images of an environment in front of a user of the head-mounted display; identifying a location in the environment in front of the user based on the one or more obtained images; and presenting the particular piece of content on the head-mounted display in a manner so as to be perceived by the user as being positioned at the determined location in the environment in front of the user.
• Optionally, the display of the computing device on which the particular piece of content is presented is a two-dimensional screen.
• Optionally, presenting the particular piece of content on the head-mounted display in a manner so as to be perceived by the user as being positioned at the determined location in the environment in front of the user comprises: presenting a three-dimensional representation of the particular piece of content on the head-mounted display in a manner so as to be perceived by the user as being positioned at the determined location in the environment in front of the user.
• Optionally, the method further comprises receiving identification information for the particular piece of content from the computing device.
• Optionally, identifying the particular piece of content that is being presented on the display of the computing device comprises: identifying the particular piece of content that is being presented on the display of the computing device based on the identification information received from the computing device.
• Optionally, the method further comprises receiving data indicating a current location of the particular piece of content from the computing device.
• Optionally, the current location of the particular piece of content corresponds to a current location of the particular piece of content relative to the display of the computing device.
• Optionally, the method may be performed by the wearable computing system described immediately above.
• A system includes: a computing device that is configured to operate as a stand-in for a dedicated input controller that is associated with a wearable computing system, wherein the computing device includes: a proximity-sensitive display; and one or more processors that are operatively coupled to the proximity-sensitive display and are communicatively coupled to the wearable computing system, the one or more processors configured to: present a plurality of graphical control elements at a plurality of locations on the proximity-sensitive display, respectively, wherein the plurality of graphical control elements correspond to a plurality of physical user input components of the dedicated input controller, respectively; monitor for touch input received at any one of the plurality of locations on the proximity-sensitive display; and in response to a detection of touch input received at any one of the plurality of locations on the proximity-sensitive display: identify, from among the plurality of graphical control elements that are associated with the plurality of locations on the proximity-sensitive display, a particular one of the plurality of graphical control elements that is associated with a particular one of the plurality of locations on the proximity-sensitive display at which the detected touch input was received; generate one or more messages based on the particular graphical control element; and transmit the one or more messages to the wearable computing system.
• Optionally, the plurality of physical user input components of the dedicated input controller to which plurality of graphical control elements correspond include one or more of: a “Home” button, a “Trigger” button, a “Bumper” button, and a touchpad.
• Optionally, the computing device is configured to operate in a same or similar manner as one or more of the computing devices and/or apparatuses described herein.
• Optionally, the computing device is configured to perform one or more of the operations in the method described immediately below.
• A computer-implemented method includes: providing a graphical user interface of an application that is running on a computing device for output on a proximity-sensitive display of the computing device; presenting a plurality of graphical control elements through the graphical user interface that correspond to a plurality of physical user input components of a dedicated input controller that is associated with a wearable computing system, respectively; receiving data indicating touch input received at a particular location on the proximity-sensitive display; determining that the particular location at which touch input was received is associated with a particular one of the plurality of graphical control elements; and in response to determining that the particular location at which touch input was received is associated with the particular graphical control element: generating one or more messages based on the particular graphical control element; and transmitting the one or more messages to the wearable computing system.
• Optionally, the method may be performed by the computing device described immediately above.
• Additional and other objects, features, and advantages of the disclosure are described in the detail description, figures and claims.
BRIEF DESCRIPTION OF THE DRAWINGS
• The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the figures. In order to better appreciate how to obtain the above-recited and other advantages and objects of various embodiments of the disclosure, a more detailed description of the present disclosures briefly described above will be rendered by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. Understanding that these drawings depict only typical embodiments of the disclosure and are not therefore to be considered limiting of its scope, the disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings in which:
• FIG. 1 illustrates another image display system having an image display device in accordance with some embodiments.
• FIG. 2 illustrates another image display system having an image display device in accordance with other embodiments.
• FIG. 3 illustrates another image display system having an image display device in accordance with other embodiments.
• FIG. 4 illustrates another image display system having an image display device in accordance with other embodiments.
• FIG. 5 illustrates an image display device displaying frames in multiple depth planes.
• FIG. 6 illustrates an apparatus having a touch-sensitive screen with an assigned area for allowing a user to provide input for an image display device.
• FIGS. 7A-7F illustrate examples of different alternative assigned areas of the screen of the apparatus ofFIG. 6.
• FIG. 8A illustrates an example of visual feedback generated in response to a user touching a part of the assigned area of the screen of the apparatus ofFIG. 6 with one finger.
• FIG. 8B illustrates an example of visual feedback generated in response to a user touching a part of the assigned area of the screen of the apparatus ofFIG. 6 with two fingers.
• FIG. 9A illustrates an example of a keyboard that may be displayed by the apparatus ofFIG. 6.
• FIG. 9B illustrates another example of a keyboard that may be displayed by the apparatus ofFIG. 6.
• FIG. 10A illustrates an example of a one finger swipe performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 10B illustrates another example of a one finger swipe performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 11A illustrates an example of a one finger touch move performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 11B illustrates an example of a two fingers touch move performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 12A illustrates an example of a two fingers pinch performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 12B illustrates an example of a two fingers un-pinch performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 13 illustrates an example of a two fingers radial move performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 14 illustrates an example of a long touch-hold performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 15 illustrates an example of a tap performed on an assigned area of a screen of the apparatus ofFIG. 6, and detectable by the apparatus as a finger-action input for an image display device.
• FIG. 16A illustrates examples of finger-actions performed on an area of a screen of the apparatus ofFIG. 6 that has been assigned as a button, and detectable by the apparatus as input for an image display device
• FIG. 16B illustrates an example of a combination of different finger-action inputs detectable by the apparatus ofFIG. 6 as a combined input for an image display device.
• FIG. 17 illustrates a feedback feature of the apparatus ofFIG. 6.
• FIG. 18 illustrates an algorithm or a method for providing feedback.
• FIG. 19A illustrates examples of finger(s)-action performed on the apparatus ofFIG. 6 while apparatus is in a face-up orientation, and examples of the corresponding effects on a displayed content.
• FIG. 19B illustrates examples of finger(s)-action performed on the apparatus ofFIG. 6 while apparatus is in an up-right orientation, and examples of the corresponding effects on a displayed content.
• FIG. 20 illustrates a method in accordance with some embodiments.
• FIG. 21 illustrates the apparatus ofFIG. 6 downloading an application in the form of a set of instruction from server(s).
• FIG. 22 illustrates a specialized processing system in accordance with some embodiments.
DETAILED DESCRIPTION
• Various embodiments of the disclosure are directed to methods, apparatuses, and articles of manufacture for providing input for head-worn video image devices. Other objects, features, and advantages of the disclosure are described in the detailed description, figures, and claims.
• Various embodiments are described hereinafter with reference to the figures. It should be noted that the figures are not drawn to scale and that elements of similar structures or functions are represented by like reference numerals throughout the figures. It should also be noted that the figures are only intended to facilitate the description of the embodiments. They are not intended as an exhaustive description of the invention or as a limitation on the scope of the invention. In addition, an illustrated embodiment needs not have all the aspects or advantages shown. An aspect or an advantage described in conjunction with a particular embodiment is not necessarily limited to that embodiment and can be practiced in any other embodiments even if not so illustrated, or if not so explicitly described.
• The description that follows pertains to an illustrative VR, AR, and/or MR system with which embodiments described herein may be practiced. However, it is to be understood that the embodiments also lends themselves to applications in other types of display systems (including other types of VR, AR, and/or MR systems), and therefore the embodiments are not to be limited to only the illustrative examples disclosed herein.
Summary of Problems and Solutions
• In some cases, a head-worn image display device may have a dedicated input controller for allowing a user of the head-worn image display device to provide input. The dedicated input controller may be specific for the particular model of the head-worn image display device, and may be unique for each brand and model of the image display device. Such dedicated input controller may allow the user of the head-worn image display device to enter text by displaying a keyboard on the screen of the head-worn image display device, and the user may enter text by using the dedicated input controller to interact with the keyboard as displayed on the screen of the head-worn image display device. Such technique of entering text may be cumbersome and may not be ideal. Also, the dedicated input controller may have limited voice and gesture control. In addition, if the dedicated input controller for a particular head-worn image display device is lost, if the dedicated input controller breaks, or if the user forgets to bring the dedicated input controller, then there is no backup controller for the user to use. Furthermore, a dedicated input controller for head-worn image display device may not be able to transfer digital images, photos, and other media content between a user's media storage device and the head-worn image display device. A dedicated input controller also may not be capable of accessing application store and content, and may not be able to receive notifications from the head-worn image display device or from a network when the head-worn image display device is not being worn by the user.
• In accordance with one or more embodiments described herein, an apparatus having a touch-screen is provided as an input device for allowing a user of a head-worn image display device to enter input for the image display device, which is sometimes referred to herein as the “wearable computing system” or “wearable display system.” The apparatus, which is sometimes referred to herein more simply as the “computing device,” may serve as a backup input device so that if a dedicated input controller for a head-worn image display device is unavailable, the apparatus may be used instead by the user to enter input. The apparatus may alternatively be also used as the main or sole input device for the image display device. The apparatus may be used as the main device for inputting text, or alternatively, be used as an alternative to a dedicated input controller for inputting text. Entering text through a touch screen of the apparatus while the user is viewing the screen of the apparatus through a transparent part of the head-worn image display device is more convenient and efficient than displaying a keyboard on a viewing screen of the image display device while the user enters text using the dedicated input controller. Also, in the embodiment in which the apparatus is implemented using the user's cell phone, the apparatus may also facilitate transfer of digital images and photos, and other media content, between the user's phone and the image display device. In addition, in some embodiments, the apparatus described herein allows a user of the head-worn image display device to access application store and content. Furthermore, in some embodiments, the apparatus described herein may receive notifications from the head-worn image display device or from a network when the head-worn image display device is not being worn by the user.
• FIGS. 1-4 illustrate various components of animage display system100 in various embodiments. Theimage display system100 includes animage display device101, and anapparatus200 for providing input for theimage display device101. Theapparatus200 will be described in further detail below. Theimage display device101 may be a VR device, an AR device, a MR device, or any of other types of display devices. Theimage display device101 includes aframe structure102 worn by anend user50, adisplay subsystem110 carried by theframe structure102, such that thedisplay subsystem110 is positioned in front of the eyes of theend user50, and aspeaker106 carried by theframe structure102, such that thespeaker106 is positioned adjacent the ear canal of the end user50 (optionally, another speaker (not shown) is positioned adjacent the other ear canal of theend user50 to provide for stereo/shapeable sound control). Thedisplay subsystem110 is designed to present the eyes of theend user50 with light patterns that can be comfortably perceived as augmentations to physical reality, with high-levels of image quality and three-dimensional perception, as well as being capable of presenting two-dimensional content. Thedisplay subsystem110 presents a sequence of frames at high frequency that provides the perception of a single coherent scene.
• In the illustrated embodiments, thedisplay subsystem110 employs “optical see-through” display through which the user can directly view light from real objects via transparent (or semi-transparent) elements. The transparent element, often referred to as a “combiner,” superimposes light from the display over the user's view of the real world. To this end, thedisplay subsystem110 comprises a partially transparent display. The display is positioned in the end user's50 field of view between the eyes of theend user50 and an ambient environment, such that direct light from the ambient environment is transmitted through the display to the eyes of theend user50.
• In the illustrated embodiments, an image projection assembly provides light to the partially transparent display, thereby combining with the direct light from the ambient environment, and being transmitted from the display to the eyes of theuser50. The projection subsystem may be an optical fiber scan-based projection device, and the display may be a waveguide-based display into which the scanned light from the projection subsystem is injected to produce, e.g., images at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple, discrete optical viewing distances or focal planes, and/or image layers stacked at multiple viewing distances or focal planes to represent volumetric 3D objects. These layers in the light field may be stacked closely enough together to appear continuous to the human visual subsystem (i.e., one layer is within the cone of confusion of an adjacent layer). Additionally or alternatively, picture elements may be blended across two or more layers to increase perceived continuity of transition between layers in the light field, even if those layers are more sparsely stacked (i.e., one layer is outside the cone of confusion of an adjacent layer). Thedisplay subsystem110 may be monocular or binocular.
• Theimage display device101 may also include one or more sensors (not shown) mounted to theframe structure102 for detecting the position and movement of thehead54 of theend user50 and/or the eye position and inter-ocular distance of theend user50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, radio devices, and/or gyros), or any combination of the foregoing. Many of these sensors operate on the assumption that theframe102 on which they are affixed is in turn substantially fixed to the user's head, eyes, and ears.
• Theimage display device101 may also include a user orientation detection module. The user orientation module detects the instantaneous position of thehead54 of the end user50 (e.g., via sensors coupled to the frame102) and may predict the position of thehead54 of theend user50 based on position data received from the sensors. Detecting the instantaneous position of thehead54 of theend user50 facilitates determination of the specific actual object that theend user50 is looking at, thereby providing an indication of the specific virtual object to be generated in relation to that actual object and further providing an indication of the position in which the virtual object is to be displayed. The user orientation module may also track the eyes of theend user50 based on the tracking data received from the sensors.
• Theimage display device101 may also include a control subsystem that may take any of a large variety of forms. The control subsystem includes a number of controllers, for instance one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers, such as application specific integrated circuits (ASICs), programmable gate arrays (PGAs), for instance field PGAs (FPGAs), and/or programmable logic controllers (PLUs).
• The control subsystem of theimage display device101 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a three-dimensional data base for storing three-dimensional scene data. The CPU may control overall operation, while the GPU may render frames (i.e., translating a three-dimensional scene into a two-dimensional image) from the three-dimensional data stored in the three-dimensional data base and store these frames in the frame buffers. One or more additional integrated circuits may control the reading into and/or reading out of frames from the frame buffers and operation of the image projection assembly of thedisplay subsystem110.
• The various processing components of theimage display device101 may be physically contained in a distributed subsystem. For example, as illustrated inFIGS. 1-4, theimage display device101 may include a local processing anddata module130 operatively coupled, such as by a wired lead orwireless connectivity136, to thedisplay subsystem110 and sensors. The local processing anddata module130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure102 (FIG. 1), fixedly attached to a helmet or hat56 (FIG. 2), removably attached to thetorso58 of the end user50 (FIG. 3), or removably attached to thehip60 of theend user50 in a belt-coupling style configuration (FIG. 4). Theimage display device101 may also include aremote processing module132 andremote data repository134 operatively coupled, such as by a wired lead orwireless connectivity138,140, to the local processing anddata module130, such that theseremote modules132,134 are operatively coupled to each other and available as resources to the local processing anddata module130.
• The local processing anddata module130 may comprise a power-efficient processor or controller, as well as digital memory, such as flash memory, both of which may be utilized to assist in the processing, caching, and storage of data captured from the sensors and/or acquired and/or processed using theremote processing module132 and/orremote data repository134, possibly for passage to thedisplay subsystem110 after such processing or retrieval. Theremote processing module132 may comprise one or more relatively powerful processors or controllers configured to analyze and process data and/or image information. Theremote data repository134 may comprise a relatively large-scale digital data storage facility, which may be available through the internet or other networking configuration in a “cloud” resource configuration. In some embodiments, all data is stored and all computation is performed in the local processing anddata module130, allowing fully autonomous use from any remote modules.
• Thecouplings136,138,140 between the various components described above may include one or more wired interfaces or ports for providing wires or optical communications, or one or more wireless interfaces or ports, such as via RF, microwave, and IR for providing wireless communications. In some implementations, all communications may be wired, while in other implementations all communications may be wireless. In still further implementations, the choice of wired and wireless communications may be different from that illustrated inFIGS. 1-4. Thus, the particular choice of wired or wireless communications should not be considered limiting.
• In some embodiments, the user orientation module is contained in the local processing anddata module130, while CPU and GPU are contained in the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be contained in the local processing anddata module130. The 3D database can be associated with theremote data repository134 or disposed locally.
• Some image display systems (e.g., VR system, AR system, MR system, etc.) use a plurality of volume phase holograms, surface-relief holograms, or light guiding optical elements that are embedded with depth plane information to generate images that appear to originate from respective depth planes. In other words, a diffraction pattern, or diffractive optical element (“DOE”) may be embedded within or imprinted/embossed upon a light guiding optical element (“LOE”; e.g., a planar waveguide) such that as collimated light (light beams with substantially planar wavefronts) is substantially totally internally reflected along the LOE, it intersects the diffraction pattern at multiple locations and exits toward the user's eye. The DOEs are configured so that light exiting therethrough from an LOE are verged so that they appear to originate from a particular depth plane. The collimated light may be generated using an optical condensing lens (a “condenser”).
• For example, a first LOE may be configured to deliver collimated light to the eye that appears to originate from the optical infinity depth plane (0 diopters). Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (½ diopter). Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter). By using a stacked LOE assembly, it can be appreciated that multiple depth planes may be created, with each LOE configured to display images that appear to originate from a particular depth plane. It should be appreciated that the stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Further, N,2N or3N stacked LOEs may be used to generate RGB colored images at N depth planes.
• In order to present 3-D virtual content to the user, the image display system100 (e.g., VR system, AR system, MR system, etc.) projects images of the virtual content into the user's eye so that they appear to originate from various depth planes in the Z direction (i.e., orthogonally away from the user's eye). In other words, the virtual content may not only change in the X and Y directions (i.e., in a 2D plane orthogonal to a central visual axis of the user's eye), but it may also appear to change in the Z direction such that the user may perceive an object to be very close or at an infinite distance or any distance in between. In other embodiments, the user may perceive multiple objects simultaneously at different depth planes. For example, the user may see a virtual dragon appear from infinity and run towards the user. Alternatively, the user may simultaneously see a virtual bird at a distance of 3 meters away from the user and a virtual coffee cup at arm's length (about 1 meter) from the user.
• Multiple-plane focus systems create a perception of variable depth by projecting images on some or all of a plurality of depth planes located at respective fixed distances in the Z direction from the user's eye. Referring now toFIG. 5, it should be appreciated that multiple-plane focus systems may display frames at fixed depth planes150 (e.g., the sixdepth planes150 shown inFIG. 5). Although MR systems can include any number ofdepth planes150, one exemplary multiple-plane focus system has six fixeddepth planes150 in the Z direction. In generating virtual content one or more of the sixdepth planes150, 3-D perception is created such that the user perceives one or more virtual objects at varying distances from the user's eye. Given that the human eye is more sensitive to objects that are closer in distance than objects that appear to be far away,more depth planes150 are generated closer to the eye, as shown inFIG. 5. In other embodiments, the depth planes150 may be placed at equal distances away from each other.
• Depth plane positions150 may be measured in diopters, which is a unit of optical power equal to the inverse of the focal length measured in meters. For example, in some embodiments,depth plane1 may be ⅓ diopters away,depth plane2 may be 0.3 diopters away,depth plane3 may be 0.2 diopters away,depth plane4 may be 0.15 diopters away,depth plane5 may be 0.1 diopters away, anddepth plane6 may represent infinity (i.e., 0 diopters away). It should be appreciated that other embodiments may generatedepth planes150 at other distances/diopters. Thus, in generating virtual content at strategically placeddepth planes150, the user is able to perceive virtual objects in three dimensions. For example, the user may perceive a first virtual object as being close to him when displayed indepth plane1, while another virtual object appears at infinity atdepth plane6. Alternatively, the virtual object may first be displayed atdepth plane6, thendepth plane5, and so on until the virtual object appears very close to the user. It should be appreciated that the above examples are significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be concentrated on a particular focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup half a meter away from the user, all six depth planes could be generated at various cross-sections of the coffee cup, giving the user a highly granulated 3-D view of the coffee cup.
• In some embodiments, the image display system100 (e.g., VR system, AR system, MR system, etc.) may work as a multiple-plane focus system. In other words, all six LOEs may be illuminated simultaneously, such that images appearing to originate from six fixed depth planes are generated in rapid succession with the light sources rapidly conveying image information toLOE1, thenLOE2, thenLOE3 and so on. For example, a portion of the desired image, comprising an image of the sky at optical infinity may be injected attime1 and the LOE retaining collimation of light (e.g.,depth plane6 fromFIG. 5) may be utilized. Then an image of a closer tree branch may be injected attime2 and an LOE configured to create an image appearing to originate from adepth plane 10 meters away (e.g.,depth plane5 fromFIG. 5) may be utilized; then an image of a pen may be injected attime3 and an LOE configured to create an image appearing to originate from adepth plane 1 meter away may be utilized. This type of paradigm can be repeated in rapid time sequential (e.g., at 360 Hz) fashion such that the user's eye and brain (e.g., visual cortex) perceives the input to be all part of the same image.
• Theimage display system100 may project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) to generate images for a 3-D experience/scenario. As used in this application, light beams include, but are not limited to, directional projections of light energy (including visible and invisible light energy) radiating from a light source. Generating images that appear to originate from various depth planes conforms the vergence and accommodation of the user's eye for that image, and minimizes or eliminates vergence-accommodation conflict.
• As mentioned and shown inFIGS. 1-4, theimage display system100 includes theapparatus200 for providing input for theimage display device101. Theapparatus200 allows theuser50 of theimage display device101 to enter user input while theuser50 is viewing content displayed by theimage display device101. In one implementation, theapparatus200 may be a handheld device (such as a cell phone (e.g., smart phone), a tablet, an IPAD, a mini-pad, etc.) configured to interact with theimage display device101. In the examples shown inFIGS. 1-4, theapparatus200 interacts with theimage display device101 by communicating with theprocessing module130 of theimage display device101. The communication between theapparatus200 and theprocessing module130 of theimage display device101 may be achieved via a wireless connection or a wired connection. In other embodiments, theapparatus200 may also communicate with theremote processing module132 and/or theremote data repository134 via a wireless connection.
• FIG. 6 illustrates an example of theapparatus200. Theapparatus200 is for use with theimage display device101 configured for head-worn by theuser50. In particular, theapparatus200 is configured as an input device/controller for allowing theuser50 to provide user input using his/her finger(s) while theuser50 is viewing images displayed by thedisplay device101. In some cases, theapparatus200 may be considered to be a part of theimage display system100. The apparatus includes ascreen202, and aprocessing unit204. Theprocessing unit204 is configured to assign afirst area206 of thescreen202 to sense finger-action of theuser50. Theprocessing unit204 is configured to detect different finger(s)-actions performed on the assignedfirst area206, thereby allowing the assignedfirst area206 to function like a touchpad. Theprocessing unit204 is also configured to generate an electronic signal to cause a change in a content displayed by theimage display device101 based on the finger-action of theuser50 sensed by the assigned first area of thescreen202 of theapparatus200. The electronic signal may represent an identification of a detected finger(s)-action, and/or may represent a command generated by theprocessing unit204 based on a detected finger(s)-action.
• In the illustrated embodiment, theapparatus200 is a handheld apparatus. By means of non-limiting examples, the handheld apparatus may be a cell phone, a smart phone, a personal-digital-assistant (PDA), or a tablet.
• As shown inFIG. 6, thescreen202 of theapparatus200 has a touch-sensitive region210, and the assignedfirst area206 is less than a total area of the touch-sensitive region210. Accordingly, theprocessing unit204 is configured to ignore input generated by theuser50 using a portion of the touch-sensitive region210 that is not a part of the assignedfirst area206.
• The size and shape of thefirst area206 may be pre-defined, or may be selectively configured by theuser50. In other embodiments, thefirst area206 may have other shapes instead of the square shape shown inFIG. 6.FIGS. 7A-7F illustrate examples of differentfirst area206 that may be assigned by theprocessing unit204.FIG. 7A illustrates thefirst area206 having a rectangular shape with a long side of thefirst area206 being parallel to a short side of theapparatus200, and the short side of thefirst area206 being parallel to a long side of theapparatus200. In other embodiments, like that shown inFIG. 7B, thefirst area206 may have a rectangular shape with a long side of thefirst area206 being parallel to a long side of theapparatus200, and the short side of thefirst area206 being parallel to a short side of theapparatus200. In further embodiments, the size of thefirst area206 may be smaller (or larger) that the examples illustrated. For example, in other embodiments, thefirst area206 may have a small size occupying only a discrete area of the screen (FIG. 7C). In further embodiments, thefirst area206 may have a circular shape (FIG. 7D), a hexagon shape (FIG. 7E), or any of other polygonal shapes.
• In some embodiments, thefirst area206 may have a shape that corresponds with an area of the display screen of theimage display device101. For example, if the screen of theimage display device101 has an aspect ratio of 4:3, then thefirst area206 assigned by theprocessing unit204 may also have an aspect ratio of 4:3 (FIG. 7F). In one implementation, theprocessing unit204 of theapparatus200 is configured to determine a shape of the display screen area of theimage display device101, and determine the shape of thefirst area206 based on the shape of the display screen area of theimage display device101. For example, theprocessing unit204 may determine the shape of thefirst area206 to be the same as the shape of the display screen area of theimage display device101.
• In some embodiments, theprocessing unit204 may be configured to determine a size of the touch-sensitive region210, and determine a size and/or shape for thefirst area206 being assigned to sense finger-action of theuser50. For example, theprocessing unit204 may determine that the entire touch-sensitive region210 of theapparatus200 is 2.5 inches in width and 5.5 inches in length. In such cases, theprocessing unit204 may then determine the width of thefirst area206 to be a constant k1 (e.g., 0.9) times the width of the touch-sensitive region210 (which would be 0.9×2.5=2.25 inches in the above example). Similarly, theprocessing unit204 may determine the length of thefirst area206 to be a constant k2 (e.g., 0.5) times the length of the touch-sensitive region210 (which would be 0.5×5.5=2.75 inches in the above example).
• In other embodiments, theprocessing unit204 may determine a brand and a model of theapparatus200, and determine the size and/or shape of thefirst area206 to be assigned to sense finger-action of theuser50. For example, a brand and a model of theapparatus200 having a relatively larger screen size may be assigned a relatively largerfirst area206 compared to another brand and another model of theapparatus200 having a relatively smaller screen size.
• After thefirst area206 has been assigned to sense finger-action of theuser50, theuser50 may then use one or more fingers to enter input for theimage display system100 using the assignedfirst area206 of theapparatus200. For example, theuser50 may move a finger within the assignedfirst area206 to move an object being displayed by theimage display device101. The object may be any object, such as a cursor, a text, an image, a photo, a window, a frame, an application page, etc. As another example, theuser50 may move two fingers in a same swiping direction within the assignedfirst area206 to move an object being displayed by theimage display device101. As a further example, theuser50 may perform a pinching or un-pinching action within the assignedfirst area206 to change a size (e.g., reducing a size by pinching, increasing a size by un-pinching) of an object. As a further example, theuser50 may perform a tapping action (e.g., single tapping, double tapping, etc.) within the assignedfirst area206 to select an object, un-select an object, grab an object, un-grab an object, etc. Thus, theapparatus200 is configured as an input device/controller for allowing theuser50 to provide input for interacting with content being displayed by theimage display device101.
• In a first mode of operation, theapparatus200 may provide no visual display on thescreen202 while theuser50 is using theapparatus200 to generate input for theimage display system100. The assignedfirst area206 of the screen has no displayed object while the assignedfirst area206 of thescreen202 is sensing the finger-action of theuser50. In such cases, theapparatus200 is configured like a touchpad. However, unlike a touchpad, theapparatus200 itself may be a mobile or handheld device. In some implementations, the presentation of content on thescreen202 by theapparatus200 may be at least partially degraded or disabled when in the first mode of operation. More specifically, when in the first mode of operation, theapparatus200 may, in such implementations, provide less content for visual display on thescreen202, provide content of reduced brightness and/or contrast for visual display on thescreen202, reduce the brightness level of thescreen202, or take one or more other actions to reduce display functionality to conserve power. In some examples, when in the first mode of operation, theapparatus200 may convey the boundaries of the touchpad and/or other portions of the user interface to which the touchpad belongs through use of one or more feedback components (e.g., haptic feedback components, speakers, etc.). In these examples, when in first mode of operation, theapparatus200 may use one or more feedback components to convey one or more portions of such a user interface instead of or in addition to using thescreen202 to visually display the user interface. In some embodiments, a representation of such a user interface or a portion thereof may be displayed by theimage display device101 while theapparatus200 is operating in the first mode. In at least some of these embodiments, the relative locations on thescreen202 to which touch input has been provided may also be visually represented for the user through theimage display device101. In this way, the user may be able to effectively interact with thescreen202 while looking elsewhere. In some of these examples, when in first mode of operation, a representation of such a user interface or a portion thereof may be displayed using theimage display device101 instead of or in addition to using thescreen202 of theapparatus200 to visually display the user interface or a limited version thereof (e.g., an at least partially degraded or disabled version of the user interface).
• In a second mode of operation, theprocessing unit204 may be configured to operate thescreen202 to display content while allowing thescreen202 to receive finger(s)-action input from theuser50. In some implementations, the quantity, brightness, and/or contrast of the content that is provided by theapparatus200 for visual display on thescreen202 when in the second mode of operation may be greater than the quantity, brightness, and/or contrast of the content that is provided by theapparatus200 for visual display on thescreen202 when in the first mode of operation. Similarly, in some examples, theapparatus200 may operate thescreen202 at a higher brightness level in the second mode of operation than it does in the first mode of operation. For example, theprocessing unit204 may operate thescreen202 to display a grid ofdots220 in the assignedfirst area206 of thescreen202, like that shown inFIG. 6. The grid ofdots220 allows theuser50 to see where the assignedfirst area206 is located, so that theuser50 can more accurately position his/her finger(s). In some cases, theprocessing unit204 may also be configured to change a feature of one or more of thedots220 in response to theuser50 touching a part of the assignedfirst area206 of thescreen202 where the one or more of thedots206 are displayed (FIGS. 8A-8B). In particular,FIG. 8A illustrates an example ofvisual feedback230 generated by theprocessing unit204 in response to theuser50 touching a part of the assignedarea206 of thescreen202 of theapparatus200 with one finger.FIG. 8B illustrates an example ofvisual feedback230 generated by theprocessing unit204 in response to theuser50 touching a part of the assignedarea206 of thescreen202 of theapparatus200 with two fingers. This allows theuser50 to see that his/her finger action generated input is successfully received by theprocessing unit204. In the above examples, thevisual feedback230 includes a circle and a change in a spacing of thedots220 in proximity to the touched location. In other embodiments, thevisual feedback230 may include just the circle without the change in the spacing of thedots220. In further embodiments, instead of a circular shape, thevisual feedback230 may include an object (of any shape) generated by theprocessing unit204 positioned at the location of thescreen202 where it is touched by theuser50. In still further embodiments, thevisual feedback230 may not include any object overlay with thedots220. In such cases, thevisual feedback230 may be a change in the color of the dot(s), and/or a change in the shape and/or size of the dot(s), where theuser50 touches thescreen202.
• In some embodiments, the horizontal spacing of thedots220 may be the same as the vertical spacing of thedots220. In other embodiments, the horizontal spacing of thedots220 may be different from the vertical spacing of thedots220. For examples, the horizontal spacing of thedots202 may be larger than, or less than, the vertical spacing of thedots220.
• It should be noted that in the second mode, the screen of theapparatus200 is not limited to displaying thedots220, and may display other content in other embodiments. For example, in other embodiments, when in the second mode, the screen of theapparatus200 may display a number of objects (e.g., photos, videos, icons, etc.) for theuser50 to select using theapparatus200, a text box for theuser50 to enter text, a drawing “pad” for allowing theuser50 to draw things, tabs for theuser50 to select, application graphics for allowing theuser50 to interact with, etc. In some implementations, a polygonal shape may be displayed in the assignedfirst area206 of thescreen202. For instance, in these implementations, the apparatus may display a solid polygonal shape, an outline of a polygonal shape, or both, in the assignedfirst area206 of thescreen202. In some examples, the geometry of such a polygonal shape may correspond to the geometry of the assignedfirst area206, such that one or more boundaries of the polygonal shape may coincide with one or more boundaries of the assignedfirst area206, respectively.
• In some embodiments, theapparatus200 may switch between the first and second modes. For example, theapparatus200 may switch between the first and second modes based on commands received from theimage display device101. In some implementations, theimage display device101 may analyze data output from one or more of its sensing devices to determine whether theapparatus200 is within the user's field of view. In some examples, such one or more sensing devices of theimage display device101 may correspond to one or more forward-facing cameras. For instance, theimage display device101 may analyze images captured by its camera(s) to determine whether theapparatus200 is shown in any of the images (and thus located within the user's field of view). If theapparatus200 is not detected by any of the camera(s), in response, theimage display device101 may then instruct theapparatus200 to operate in the first mode. On the other hand, if theapparatus200 is detected by a camera of theimage display device101, in response, theimage display device101 may then instruct theapparatus200 to operate in the second mode. This feature is advantageous because it may provide power-saving benefits at times when theuser50 is not looking at the screen of theapparatus200. So if theuser50 is not looking at theapparatus200 through the screen of theimage display device101, the screen of theapparatus200 may operate in the first mode (displaying no content or a degraded set of content), and if theuser50 is looking at theapparatus200 through the screen of theimage display device101, the screen of theapparatus200 may operate in the second mode (displaying content in its entirety). In some examples, theimage display device101 may leverage one or more of sensing devices other than one or more of its forward-facing cameras to determine whether theapparatus200 is within the user's field of view. For instance, theimage display device101 may determine whether theapparatus200 is within the user's field of view based on data output from one or more proximity sensors, inward-facing eye tracking cameras, or other sensing devices of theimage display device101 that are capable of outputting data indicative of whether the user is wearing theimage display device101. It may be particularly beneficial for theimage display device101 to instruct theapparatus200 to switch between the first and second modes of operation based on data output from one or more such sensing devices in implementations where theimage display device101 is a virtual reality (VR) headset or other wearable computing device that physically occludes some portion of the user's view of the environment when worn by the user.
• Also, in some embodiments, each unit length Lh of a horizontal movement input performed at thefirst area206 may cause a corresponding amount of horizontal movement Mh of an object displayed by theimage display device101. Similarly, each unit length Lv of a vertical movement input performed at thefirst area206 may cause a corresponding amount of vertical movement My of the object displayed by theimage display device101. A horizontal movement efficiency may be represented by the ratio Mh/Lh, and a vertical movement efficiency may be represented by the ratio Mv/Lv. If the ratio Mh/Lh is 1.0, that means one until of horizontal movement input at the assignedfirst area206 will cause the same unit of horizontal movement to occur for an object displayed by theimage display device101. If the ratio Mh/Lh is larger than 1.0, that means one unit of horizontal movement input at the assignedfirst area206 will cause more than one unit of horizontal movement to occur for an object displayed by theimage display device101. If the ratio Mh/Lh is less than 1.0, that means one unit of horizontal movement input at the assignedfirst area206 will cause less than one unit of horizontal movement to occur for an object displayed by theimage display device101. Similar concepts apply for the vertical direction. In some embodiments, the screen of theapparatus200 may be smaller in size compared to the screen of theimage display device101. In such cases, it may be desirable to configure the assignedfirst area206 so that Mh/Lh>1, and so that Mv/Lv>1. This allows theuser50 to operate a relatively smallerfirst area206 on theapparatus200 to cover a larger screen area in theimage display device101. In other embodiments, the screen of theapparatus200 may be larger in size compared to the screen of theimage display device101. In such cases, it may be desirable to configure the assignedfirst area206 so that Mh/Lh<1, and so that Mv/Lv<1. This allows theuser50 to operate a relatively largerfirst area206 on theapparatus200 to cover a smaller screen area in theimage display device101. In other embodiments, regardless of the relative screen sizes of theapparatus200 and theimage display device101, the movement efficiencies (Mh/Lh and Mv/Lv) of the assignedfirst area206 may be arbitrary set to any values.
• In some embodiments, the ratio Mh/Lh may be equal to the ratio Mv/Lv. This means that a horizontal movement efficiency is the same as a vertical movement efficiency for input entered using the assignedfirst area206. In other embodiments, the ratio Mh/Lh may be higher than the ratio Mv/Lv. This means that a horizontal movement efficiency is higher than a vertical movement efficiency for input entered using the assignedfirst area206. This allows theuser50 to cover more horizontal space in the displayed screen of theimage display device101 by moving through less horizontal space in the assigned first area206 (compared to that for the vertical space). In further embodiments, the ratio Mh/Lh may be less than the ratio Mv/Lv. This means that a horizontal movement efficiency is lower than a vertical movement efficiency for input entered using the assignedfirst area206. This allows theuser50 to cover more vertical space in the displayed screen of theimage display device101 by moving through less vertical space in the assigned first area206 (compared to that for the horizontal space).
• In some embodiments, Mh/Lh may be based on a width of the screen of theimage display device101, and Mv/Lv may be based on a height of the screen of theimage display device101. Thus, Mh/Lh and Mv/Lv may be based on an aspect ratio or shape of the screen of theimage display device101. For example, if the screen of theimage display device101 has an aspect ratio (width to height ratio) of 4:3, then Mh/Lh may be 1.33 times Mv/Lv. When these movement efficiencies are applied for an assignedfirst area206 having a square shape, for example, a unit of horizontal movement input by the user will cause an object displayed in theimage display device101 to move 1.33 times more compared to that for a unit of vertical movement. Accordingly, even though the assigned first area206 (with a square shape in the example) may not correspond with the shape (having an aspect ratio of 4:3 in the example) of the screen of theimage display device101, the movement efficiencies of the assignedfirst area206 may be selected or determined so that theuser50 can cover all of the space horizontally and vertically using the assignedfirst area206.
• In further embodiments, the horizontal movement efficiency and/or the vertical movement efficiency for the assignedfirst area206 may be variable. In particular, the horizontal movement efficiency and/or the vertical movement efficiency for the assignedfirst area206 may be a function of an input location within the assignedfirst area206, and/or a function of a cursor/pointer position in the screen of theimage display device101. For example, the horizontal movement efficiency may have a first value when an input generated at thefirst area206 corresponds to a center of a field of view or a center of thefirst area206, and may have a second value (higher than the first value) when the input generated at thefirst area206 corresponds with a location that is at a periphery of the field of view or at a periphery of thefirst area206. This feature may be desirable as it allows theuser50 to scroll or move through more content as his/her finger(s) approaches a left/right edge of thefirst area206 or the field of view. The scroll speed may appear to be faster because the cursor is “moving more” through the content. In other embodiments, the second value may be lower than the first value.
• In one or more embodiments described herein, the horizontal movement efficiency and/or the vertical movement efficiency for thefirst area206 may be predetermined, may be determined by theprocessing unit204, and/or may be configurable by theuser50 of theapparatus200.
• Returning toFIG. 6, in some embodiments, theprocessing unit204 may also optionally be configured to assign asecond area240 of thescreen202 as afirst button242. In the illustrated example, thefirst button242 is a “HOME” button. In other embodiments, thefirst button242 may be other types of button for performing other types of functions.
• Also, theprocessing unit204 may optionally be configured to assign athird area250 of the screen as asecond button252. In the illustrated example, thesecond button252 is a “TOGGLE” button. The TOGGLE button may allow the user to perform a select function, and/or to change a function of a cursor or pointer. In other embodiments, thesecond button252 may be other types of button for performing other types of functions.
• In addition, theprocessing unit204 may optionally be configured to assign afourth area260 of the screen as athird button262. In the illustrated example, thethird button262 is a “BUMPER” button. The BUMPER button is a multifunction button depending on the software/operating system of theapparatus200. In some cases, the BUMPER button may allow theuser50 to control a volume, turn a page, change a magnification, etc. In other embodiments, thethird button262 may be other types of button for performing other types of functions.
• In other embodiments, theprocessing unit204 may assign more than three areas of the screen as respective input areas (e.g., pad area(s) for receiving finger movement input, buttons, controls, etc.).
• Furthermore, in some embodiments, any control of theapparatus200 may be operable to generate input for theimage display device101. For example, in some embodiments, a volume control of theapparatus200 may be utilized to control speaker(s) of theimage display device101. In one implementation, theprocessing unit204 is configured to detect an actuation of the volume control at theapparatus200. In response to the detected actuation, theprocessing unit204 then provides a corresponding control signal (e.g., volume up signal, volume down signal, mute, etc.) for transmission to theimage display device101. The transmission of the control signal may be achieved wirelessly or through a cable. Theimage display device101 then operates its speaker(s) in accordance with the control signal provided by theapparatus200.
• Also, as shown inFIG. 6, in some embodiments, theprocessing unit204 may provide an “Image Capture”button274 for allowing theuser50 to capture an image presented by the screen of theimage display device101. The image may include content displayed by the screen of theimage display device101, and/or content in the surrounding environment as viewed through the screen of theimage display device101. In one implementation, theprocessing unit204 is configured to detect a pressing of the “Image Capture”button274, and in response, generates a control signal (for transmission to the image display device101) to cause theimage display device101 to perform a screen-shot function to capture a screen-shot of content displayed by theimage display device101. Alternatively, or additionally, theprocessing unit204 may generate a control signal to cause theimage display device101 to operate its camera to take a picture of the surrounding as viewed by the camera. The image of the displayed content and the image of the surrounding may be combined by theprocessing unit204 and/or theprocessing unit130 to form a composite image. In some cases, the “Image Capture”button274 may also be pressed to capture an image of content displayed by theapparatus200. The image displayed by theapparatus200 may be artificially generated graphics and/or camera image captured by the camera of theapparatus200. For example, theprocessing unit204 may detect a pressing of the “Image Capture”button274, and in response, performs a screen-shot function to capture a screen-shot of content displayed by theapparatus200. Also, in other embodiments, the “Image Capture”button274 may be used to capture a video. For example, the “Image Capture”button274 may be pressed and hold for a certain duration. Theprocessing unit204 detects that there is a long hold for the “Image Capture”button274, and operates the camera of theapparatus200 to start recording a video. Theuser50 may long hold the “Image Capture”button274 again to stop the recording of the video. Thus, as soon as theprocessing unit204 detects that there is a second long hold for the “Image Capture”button274, theprocessing unit204 then operates the camera to stop the recording of the video. It should be noted that other techniques for generating a video using theapparatus200 may be employed in other embodiments, and that the “Image Capture”button274 is just an example of the control that may be used to generate a video.
• In one or more embodiments, an image or a video captured by theapparatus200 may be sent by theapparatus200 to a network (e.g., a Cloud network) for storage. In such cases, theimage display device101 may retrieve the stored image or video from the network (e.g., the Cloud network) for display on its screen. Alternatively, an image or a video captured by theapparatus200 may be sent by a short distance network (e.g., Bluetooth network, Wi-Fi, etc.) to theimage display device101. Also, in some embodiments, theapparatus200 may be configured to obtain media content (e.g., photos, pictures, videos, etc.) from the network. For example, another user may upload a media content to a Cloud network, and theuser50 may obtain such media content by accessing the Cloud network via an account that is associated with theapparatus200 and/or theimage display device101.
• In some embodiments, when theuser50 is using the assigned first area206 (the simulated touchpad region) of thescreen202, thebuttons242,252,262 are disabled. This may have the benefit of preventing theuser50 from inadvertently touching and activating any of these buttons. In other embodiments, when theuser50 is using the assignedfirst area206 of thescreen202, one or more of thebuttons242,252,262 are not disabled, thereby allowing theuser50 to perform simultaneous finger(s)-actions using both the first assigned area206 (the simulated touchpad region) and any of thebuttons242,252,262.
• As shown inFIG. 6, theprocessing unit204 may also optionally be configured to assign anotherarea270 of thescreen202 as akeyboard activation button272, and wherein theprocessing unit204 is configured to operate thescreen202 to display a keyboard274 (FIG. 9A) in response to theuser50 touching the assignedsecond area270 of thescreen202 where thekeyboard activation button272 is located.FIG. 9B illustrates another example of akeyboard274 that may be displayed by thescreen202. The displayedkeyboard274 allows theuser50 to enter text, numbers, symbols, expressions, messages, etc.
• Returning toFIG. 6, theprocessing unit204 is configured to perform gesture recognition to identify a finger-action(s) performed on any or a combination of the assignedareas206,240,250,260. In one implementation, theapparatus200 includes a buffer, and as theuser50 enters input using finger-action(s) performed on the assigned area(s), theapparatus200 generates the touch input data corresponding to the touched positions on thescreen202. Theprocessing unit204 is configured to add the touch input data to the buffer, and performs gesture recognition using the input data stored in the buffer. In some embodiments, the input data may include positional data representing respective touched locations in the assignedfirst area206. The input data may also include respective timing data representing respective times at which the respective locations were touched by theuser50. If theuser50 touches any of thebuttons242,252,262, the corresponding input data generated may include the identification of the button touched, and timing of the touch action, and also a duration for which the button was touched.
• In the illustrated embodiments, theprocessing unit204 is configured to perform gesture recognition using the input data to identify a corresponding command for the performed gesture. For example, if theprocessing unit204 determines from the input data that theuser50 is making a swiping action, theprocessing unit204 may then identify “movement” as the command that corresponds with the user's finger-action. As another example, if theprocessing unit204 determines from the input data that theuser50 is making a pinching action, theprocessing unit204 may then identify “size reduction” as the command that corresponds with the finger-action.
• It should be noted that theprocessing unit204 is not limited to detecting the above gestures or finger-actions, and that theprocessing unit204 can detect other finger-actions of theuser50. By means of non-limiting examples, theprocessing unit204 may be configured to detect one finger swipe (e.g., up/down swipe (FIG. 10A), left/right swipe (FIG. 10B)), two fingers swipe, one finger touch move (FIG. 11A), two fingers touch move (FIG. 11B), two fingers pinch (FIG. 12A), two fingers un-pinch (FIG. 12B), two fingers radial move (FIG. 13), long touch-hold (FIG. 14), tap (FIG. 15), double tap, etc. In one implementation, theprocessing unit204 is configured to distinguish between a long hold action and a tap action based on a duration for which the user's finger is in contact with thescreen202. For example, if the touch duration is 0.7 second or less, then theprocessing unit204 may determine that the finger(s)-action is a tap action. On the other hand, if the touch duration is 1 second or more, then theprocessing unit204 may determine that the finger(s)-action is a long hold action. Also, theprocessing unit204 may detect finger action(s) performed using other assigned areas of thescreen202, such as the area where the “Home” button is located, the area where the “Trigger” button is located, and the area where the “Bumper” button is located. In some implementations, the functionality of each of one or more of the “Home,” “Trigger,” and “Bumper” buttons described herein may correspond to the functionality of each of one or more of the “Home,” “Trigger,” and “Bumper” buttons as described in U.S. patent application Ser. No. 15/965,702, which is incorporated by reference herein in its entirety. Similarly, in some examples, the functionality of the touchpad described herein may correspond to that which is described in U.S. patent application Ser. No. 15/965,702, the entirety of which is incorporated herein by reference. As such, theapparatus200 may function as an adequate stand-in for a dedicated input controller.FIG. 16A shows examples of finger-actions (e.g., long-hold, tap, etc.) that may be performed on the “Trigger” button and is detectable by theprocessing unit204. Furthermore, theprocessing unit204 may be configured to detect a combination of finger-actions performed on different assigned areas of thescreen202. For example, as shown inFIG. 16B, theuser50 may long hold the “Trigger” button, and while doing so, may also perform touch moves using the first assignedarea206 of thescreen202. Theprocessing unit204 is configured to detect both of these finger-actions simultaneously, and may consider them as a combined input for theimage display device101. For example, theprocessing unit204 may determine that the above combination of finger-actions is to create a command to move a selected object. Theprocessing unit204 may then transmits such command signal to the image display device to move a selected object. Theprocessing unit204 is configured to generate corresponding commands for the above different finger-actions, or combination of finger-actions, performed on one or more assigned areas of thescreen202.
• In some embodiments, while theuser50 is performing a certain finger(s)-action on the assigned area(s) of thescreen202, the user's50 finger(s) may unintentionally leave thescreen202. To address this situation, theprocessing unit204 may be configured to determine that a certain finger(s)-action (e.g., swiping action, etc.) has ended after receiving more than a predetermined threshold quantity of consecutive samples indicating that theuser50 is not touching any location on thescreen202. For example, theprocessing unit204 may wait until it has received three (or other number of) consecutive null samples before it determines that such a swiping action has ended. In this example, theprocessing unit204 may receive a series of touch input samples, followed by two consecutive null samples (e.g., due to the user's finger unintentionally not touching the screen202), followed by a series of touch input samples. In this situation, theprocessing unit204 would still treat this entire sequence of samples as though it is part of one continuous swiping action. This allows theuser50 to correct his/her action and to finish the finger(s)-action without requiring the user to re-perform the previous performed action. The above features are advantageous because they help filter out noise and increase system robustness to measurement or sensing errors.
• In some embodiments, theprocessing unit204 is configured to send the identified command to the image display device101 (e.g., to the processing module130). Theprocessing module130 then determines what content to display for theuser50 for viewing based on the received command determined by theprocessing unit204. In some cases, the same command may result in different effects on the content being displayed by theimage display device101, depending on the particular application being used by theuser50, and/or the particular content being viewed by theuser50. For example, if theuser50 is viewing a page that allows theuser50 to browse different pages of objects, then the “movement” command may cause a scrolling of the different pages of objects. On the other hand, if theuser50 is in an application that allows an object to be moved, then the “movement” command may cause the object to move from one location to another location.
• As illustrated in the above embodiments, theapparatus200 is advantageous and improves the technological field of head-worn image display devices in several respects. First, theapparatus200 may serve as a backup input device (for a head-worn image display device) so that if a dedicated input controller for a head-worn image display device is unavailable (e.g., if the dedicated input controller is lost, breaks, runs out of battery, etc.), theapparatus200 may be used instead by theuser50 to enter input for the head-worn image display device. Many users of image display devices already carry smartphones whenever they go. So implementing theapparatus200 as an input controller using a smartphone would increase the chance that the user has an input controller for the image display device. The apparatus may also be also used as the main or sole input device for the image display device, so that the image display device may not need to come with a dedicated input controller.
• As mentioned, many users of image display devices already carry smartphones or other types of mobile devices whenever they go. So implementing theapparatus200 as an input controller using a smartphone or any other types of mobile devices would improve portability and mobility for the image display device because the user may use theimage display device101 anywhere (even without a dedicated input controller for the image display device101) as long as the user has the mobile device. Backup and supplemental input support may also be achieved through theapparatus200 because most users already carry a mobile device whenever they go.
• Theapparatus200 may be used as the main device for inputting text, or alternatively, be used as an alternative to a dedicated input controller for inputting text. Entering text through a touch screen of the apparatus while the user is viewing the screen of the apparatus through a transparent part of the head-worn image display device is more convenient and efficient than displaying a keyboard on a viewing screen of the image display device while the user enters text using the dedicated input controller. While smartphones have been known to implement keyboards, the concept of using a smartphone to implement keyboard in the context of providing input for a head-worn image display device is believed to be novel and non-intuitive. This is because such solution would require theuser50 of the head-wornimage display device101 to view through a display screen of theimage display device101 to view the screen202 (where the keyboard is displayed) of theapparatus200. Accordingly, this solution would require the user to shift focus from the display screen of theimage display device101 to thescreen202 of the apparatus200 (which is visible through the display screen of the image display device101). However, it is believed that despite the shift in visual focus, the touchscreen keyboard implemented on theapparatus200 as input device for theimage display device101 may be more comfortable to some users in some instances.
• Also, in the embodiment in which theapparatus200 is implemented using the user's cell phone (e.g., smartphone), theapparatus200 may also facilitate transfer of digital images and photos, and other media content, between the user's phone and theimage display device101. For example, pictures taken by the image display device's101 camera may be transferred to the user'sapparatus200, and pictures taken by the apparatus's200 camera may be transferred to theimage display device101. An apparatus that provides both content and control input by a user for a head-worn image display device is believed to be unique.
• In addition, in some embodiments, theapparatus200 described herein allows a user of the head-worn image display device to access application store and content through a network, such as the Internet, Bluetooth network, etc. As most smartphones already have multiple network interfaces for multiple types of network connections (e.g., Wi-Fi connection, Bluetooth connection, connection to cellular towers, etc.), implementing theapparatus200 using smartphone (or other types of portable network devices) will have the benefit of allowing the user to obtain media content and other information from multiples sources through different types of connections. For example, theapparatus200 implemented as the input device for theimage display device101 may also access application store for obtaining applications useable with theimage display device101. As another example, theapparatus200 implemented as the input device for theimage display device101 may also obtain media content from the Web, from another mobile device (e.g., through email, texting, airdrop, etc.). In some embodiments, theapparatus200 may also allow theuser50 to control account settings for theimage display device101, and to manage theimage display device101 through a user interface displayed on thescreen202 of theapparatus200.
• Furthermore, in some embodiments, theapparatus200 described herein may receive notifications (for the image display device101) from a network (e.g., the Internet) even when the head-wornimage display device101 is not being worn by theuser50. For example, theimage display device101 may push different notifications to theapparatus200. The pushing of the notifications may be performed directly by theimage display device101, or indirectly through another network device (such as a component in a Cloud network). By means of non-limiting examples, notifications received by theapparatus200 may be calendar notification, advertisement notification, social media notification, operational notification (e.g., battery status, storage level, update notification, etc.) regarding an operation of theimage display device101, etc., or any combination of the foregoing. Also, in some embodiments, a third party in communication with theimage display device101 may provide notification to theimage display device101. In such cases, theimage display device101 may forward the notification to theapparatus200 so that theuser50 can see the notification even if theuser50 is not wearing theimage display device101. The notification feature is advantageous because it allows functions such as calendar management to be made more practical, and timely reminders and real-time updates for theimage display device101 may be received by the user via theapparatus200.
• Sensory Feedback
• In some embodiments, theprocessing unit204 may optionally be configured to operate a feedback component in theapparatus200 in response to the finger-action of theuser50. Such feature is advantageous in that it allows theuser50 to know that his/her finger(s) is reaching or crossing a boundary of thefirst area206. In particular, when theuser50 is viewing content displayed by theimage display device101, theuser50 may not be viewing theapparatus200. As such, theuser50 may not visually notice that his/her finger has moved out of a boundary of the first area206 (that has been assigned to receive finger-action input). The feedback feature solves this technical problem, and provides a technical improvement for theimage display system100 by informing theuser50 of theimage display system100 via feedback that his/her is about to cross, has reached, or has crossed, the boundary of the assignedfirst area206. The feedback may be in a form of a vibration (haptic feedback), which may include one or more mechanical pulses. Alternatively, or additionally, the feedback may include audio feedback.
• As shown inFIG. 17, in some embodiments, thefirst area206 may have aboundary280, and theprocessing unit204 may be configured to operate the feedback component when a finger of theuser50 is crossing, reaching, or moving to a location that is within aprescribed distance282 from, theboundary280. In one implementation, the first assignedarea206 has a square or rectangular shape with fourboundaries280. In such cases, theprocessing unit204 may be configured to operate the feedback component when the finger of theuser50 is crossing, reaching, or moving to a location that is within a prescribed distance from any of the fourboundaries280 of the first assignedarea206. The feedback component may include a haptic feedback component, a speaker, any of other types of component that is capable of generating feedback signal (such as a visual signal for display by the image display device101), or any combination of the foregoing. Thus, as used in this specification, the term “feedback component” may include one or more components for providing one or more sensory feedback. Similarly, as used in this specification, the term “feedback’ may include one or more different types of feedback.
• As another example, thefirst area206 may have one ormore boundaries280 that at least partially surround a reference location. For example, thefirst area206 may have a square or rectangular shape surrounding a reference location (e.g., center), or may have a circular shape surrounding a reference location (e.g., center). In such cases, theprocessing unit204 may be configured to operate the feedback component in theapparatus200 in response to a finger of theuser50 reaching a prescribed distance from the reference location.
• Also, in some embodiments, theprocessing unit204 may be configured to operate the feedback component in theapparatus200 to generate different types of feedback based on different respective spatial relationships between one or more finger(s) of theuser50 with respect to thefirst area206. This allows theuser50 to know the degree to which his/her finger(s) is reaching one ormore boundaries280 of the assignedfirst area206. In one example, the different respective spatial relationships may be one of the one or more finger(s), or a point that is between two fingers, reaching differentrespective distances282 from aboundary280 of the assignedfirst area206. In another example, the different respective spatial relationships may be different distances (exceeding a threshold) that is between (1) one of the one or more finger(s), or a point that is between two fingers, of the user and (2) a reference location within the assignedfirst area206.
• In one example, the different types of feedback may comprise a first haptic impulse with a first amplitude, and a second haptic impulse with a second amplitude that is different from the first amplitude. In such cases, as the user's finger moves closer to aboundary280 of the assigned first area206 (or moves further away from a reference location surrounded by one or more boundaries280), the amplitude of the haptic impulse will increase.
• In another example, the different types of feedback comprise a first number of haptic impulse(s), and a second number of haptic impulse(s) that is different from the first number. In such cases, as the user's finger moves closer to aboundary280 of the assigned first area206 (or moves further away from a reference location surrounded by one or more boundaries280), the number of the haptic impulse(s) will increase.
• In further example, the different types of feedback comprise a first series of haptic impulses with a first frequency, and a second series of haptic impulses with a second frequency that is different from the first frequency. In such cases, as the user's finger moves closer to aboundary280 of the assigned first area206 (or moves further away from a reference location surrounded by one or more boundaries280), the frequency of the haptic impulse will increase.
• In another example, the different types of feedback may comprise a first audio signal with a first tone, and a second audio signal with a second tone.
• In still another example, the different types of feedback may comprise a first number of audio signal, and a second number of audio signal, wherein the first number is different from the second number.
• In a further example, the different types of feedback may comprise a first audio message, and a second audio message that is different from the first audio message.
• In other embodiments, theprocessing unit204 may operate the feedback component to generate different types of feedback based on the user's finger(s) reachingdifferent boundaries280 of the assignedfirst area206. For example, the assignedfirst area206 may have a first boundary280 (e.g., a left boundary or a right boundary) and a second boundary280 (e.g., a top boundary or a bottom boundary). In such cases, theprocessing unit204 may be configured to operate the feedback component to generate a first type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the first boundary; and may be configured to operate the feedback component to generate a second type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the second boundary. Such feature may be desirable because it allows theuser50 to know whichboundary280 his/her finger(s) is approaching without looking at theapparatus200.
• Also, in other embodiments, theprocessing unit204 may be configured to operate the feedback component based on a swiping direction of the user's finger(s). For example, assuming the assignedfirst area206 has a rectangular shape with top boundary, bottom boundary, left boundary, and left boundary. If theuser50 is swiping his/her finger(s) upward, then theboundary280 that is likely to be reached would be the top boundary. In such cases, theprocessing unit204 may then operate the feedback component to generate a type of feedback to inform theuser50 that the user's finger(s) is reaching the top boundary.
• In some embodiments, it may be desirable to ensure that feedback is provided only in situations in which a touching of a location outside (or close to) theboundary280 of thefirst area206 is resulted from theuser50 using the assignedfirst area206. For example, if theuser50 touches a location outside (or within a prescribed distance from) the assignedfirst area206, and the touch-action is not a part of a continuous swiping action that starts from within thefirst area206, then theprocessing unit204 may not operate the feedback component to provide any feedback. On the other hand, if the touching of the location outside (or within the prescribed distance from) the assignedfirst area206 is a part of a swiping action that begins from within the assignedfirst area206, then theprocessing unit204 may operate the feedback component to provide feedback.
• FIG. 18 illustrates an algorithm ormethod300 for providing feedback that considers the above two scenarios. First, theprocessing unit204 determines whether there is an input signal indicating a receipt of touch input by the user50 (item302). The input signal may be generated as a result of theuser50 touching any location in the touch-sensitive region of theapparatus200. If there is no such input signal, theprocessing unit204 then clears a flag (item304). In the illustrated embodiments, the flag is used to keep track whether theuser50 has initiated a finger-action by touching a location inside the assignedfirst area206. If the user has initiated such action, then the flag will be set. If not, the flag will be cleared. After the flag is cleared, the method loops back toitem302 so that theprocessing unit204 continues to determine whether theuser50 has touched any part of the touch-sensitive region (e.g., area outside and inside the assigned first area206) of theapparatus200.
• If theprocessing unit204 determines that there is an input signal indicating a receipt of a touch input by theuser50, theprocessing unit204 then determines whether the touched location is within the assignedfirst area206 of the screen202 (item306). If the touched location is inside the assignedfirst area206, then theprocessing unit204 sets the flag to indicate that a finger-action has occurred that involves theuser50 touching a location in the assigned first area206 (item308). Theprocessing unit204 also adds the touch input data to a buffer (item310), and performs gesture recognition using the input data stored in the buffer (item312). As discussed, the buffer is configured to store input data generated by theuser50 performing a finger-action on the assignedfirst area206 and/or other assigned area(s). In some embodiments, initem312, theprocessing unit204 performs gesture recognition using the input data to identify a corresponding command for the performed gesture (finger-action(s)). Theprocessing unit204 also generates or identifies corresponding command for the detected finger-action(s) performed on one or more assigned areas of thescreen202. In some embodiments, theprocessing unit204 is configured to send the command to the image display device101 (e.g., to the processing module130). Theprocessing module130 then determines what content to display for theuser50 for viewing based on the received command provided by theapparatus200.
• Returning toFIG. 18, if, on the other hand, theprocessing unit204 determines initem306 that the touched location is outside the assignedfirst area206, theprocessing unit204 then determines whether a flag was previously set (item316). If no flag was previously set, that means the currently touched location is not a part of a finger-action initiated from a location that is within the assigned first area206 (e.g., the currently touched location may have been resulted from theuser50 tapping, grabbing, etc., a location that is outside the assigned first area206). In such cases, theprocessing unit204 does not operate the feedback component, and no feedback is provided to theuser50. The method then loops back toitem302, so that theprocessing unit204 can continue to determine whether theuser50 has touched any part of the touch-sensitive region (e.g., area outside and inside the assigned first area206) of theapparatus200.
• On the other hand, if theprocessing unit204 determines initem316 that a flag was previously set, that means the currently touched location (which is outside the assignedfirst area206 or is within aprescribed distance282 from aboundary280 of the assigned first area206) is a part of a trajectory of a finger-action that initiated from within thefirst area206. In such case, theprocessing unit204 then operates the feedback component to provide feedback (item318). Once feedback has been provided, the processing unit may then flush the buffer that stores input data generated from theuser50 touching the assigned first area206 (item314). The flushing of the buffer means that the previous input data resulted from theuser50 touching the assignedfirst area206 is deleted, and no command will be identified and generated because the user's finger has reached or exceeded theboundary280 of the assignedfirst area206.
• Change in content based on finger(s)-generated input signal and/or sensor signal
• As described above, the assigned first area206 (which functions like a touchpad) of thescreen202, and other assigned areas (likeareas240,250,260), allow theuser50 of thevideo display device101 to provide a variety of finger(s)-generated inputs for thevideo display device101. Theprocessing unit204 of theapparatus200 is configured to generate an electronic signal (resulted from the finger(s)-generated input) for causing a change in the content displayed by theimage display device101. In some cases, the electronic signal may represent an identification of a finger(s)-action. Alternatively or additionally, the electronic signal may represent a command or an identification of a command determined by theprocessing unit204 based on a recognition of a finger-action input generated using the assignedareas206,240,250,260 (like those described with reference toFIGS. 10-16, anditems312 and314 ofFIG. 18). In some embodiments, theapparatus200 is configured to transmit such electronic signal to theimage display device101 via a wired connection. In other embodiments, theapparatus200 may convert the electronic signal using a wireless communication module into a wireless form for wireless transmission to theimage display device101. Theprocessing module130 of thevideo display device101 receives the signal from theapparatus200, and change the content for display by thedisplay subsystem110 based on the signal.
• By means of non-limiting examples, the change in the content may be a change in a size of the content, a change in a position of the content, a change in a shape of the content, a change in a color of the content, a replacement of information in the content, an increase or decrease in a quantity of information in the content, or any combination of the foregoing. In some cases, a change in the content caused by the finger(s)-generated input (generated using the assigned first area206) may be a movement of a cursor or a pointer. In other cases, a change in the content caused by the finger(s)-generated input may be a change in a selection of an item to another selection of another item displayed by theimage display device101. In still other cases, a change in the content caused by the finger(s)-generated input may be a movement of an object (e.g., a photo, a computer-generated image, a cartoon, etc.) across a display of theimage display device101, wherein the movement may occur within a viewing plane of the user50 (e.g., in the X-Y plane), or along a viewing depth (e.g., along a Z-axis) of theuser50. Also, in some cases, a change in the size of content may be a change in the size of an object (e.g., a photo, a computer-generated image, a cartoon, etc.) being displayed by theimage display device101.
• It should be noted that the finger(s)-actions detected by theprocessing unit204 may be used by theprocessing unit204 to determine a variety of different commands (an example of electronic signals), and theprocessing unit204 may transmit such commands (wirelessly or through a wired connection) to theimage display device101 for allowing theimage display device101 to change the content being displayed based on the commands. By means of non-limiting examples, a command determined from a detected touch move (like that shown inFIG. 11A) may be processed by theimage display device101 to target elements, to control movement of a cursor, to navigate within the plane of the viewing screen of theimage display device101, etc.; a command determined from a detected upward or downward swipe (like that shown inFIG. 10A) may be processed by theimage display device101 for scrolling content in simple grid and list layouts; a command determined from a detected left or right swipe (like that shown inFIG. 10B) may be processed by theimage display device101 to move a next page, to move a different section, etc. to the current viewing frame for display by theimage display device101; a command determined from a two fingers touch and scroll action (like that shown inFIG. 11B) may be processed by theimage display device101 to provide inertial scroll of content (i.e., if the two-fingers touch is moved slowly, the page scrolls slowly and precisely, and if the two-fingers touch is moved with a quick swipe, the page flings) displayed by theimage display device101; a command determined from a pinching action (like that shown inFIG. 12A) or un-pinching action (like that shown inFIG. 12B) may be processed by theimage display device101 to change a size of an object displayed by theimage display device101; a command determined from a detected long hold in the assigned first area206 (like that shown inFIG. 14) may be processed by theimage display device101 to open a menu, to open a browser, to open an application, etc., for display by theimage display device101; a commend determined from a detected tap action in the assigned first area206 (like that shown inFIG. 15) or from a tap action on the “Trigger” button (like that shown inFIG. 16B) may be processed by theimage display device101 to open an additional function menu for display by theimage display device101; etc. The above functions caused by the commands generated by theprocessing unit204 are only examples. In other embodiments, the same detected finger(s)-actions described above may be used to generate other different commands for performing other functions different from the examples described.
• Also, in some embodiments, a sequence of commands determined from a sequence of detected finger(s)-action may be transmitted from theprocessing unit204 to theimage display device101 for allowing theuser50 to perform a variety of tasks on the content displayed by theimage display device101. For example, a sequence of commands may be provided by theprocessing unit204 for 3D placement of object and content extraction. In one implementation, theuser50 long holds the “Trigger button” to grab an object displayed by theimage display device101, then performs touch move using the first assignedarea206 to move the grabbed object, and then releases the “Trigger button” to place the object at a desired location. In some embodiments, the object is grabbed as long as the “Trigger” button is pressed, and is dropped as soon as the “Trigger” button is released. In such cases, theuser50 may use one hand to press the “Trigger” button, while the other hand is used to move the grabbed object. Alternatively, after an object is grabbed, the user can release the “Trigger” button, and the object will still remained grabbed. In such cases, the user can use the same hand to move the grabbed object. After the grabbed object is desirably placed, the user can then tap or long-hold the “Trigger” button to release the grabbed object. The 3D placement feature may allow theuser50 to move any object displayed by theimage display device101. For example, theuser50 may use the above features for dragging and dropping an image in an application for composing slide shows, for moving a web slider, etc. In some embodiments, while the object is grabbed, theuser50 can use the assignedfirst area206 and/or head pose to move the grabbed object, can use the assignedfirst area206 to rotate the grabbed object (using two fingers radial move like that shown inFIG. 13), and can use the assignedfirst area206 to scale the grabbed object (using pinch or un-pinch action like that shown inFIG. 12A/12B).
• As another example, theprocessing unit204 may detect a certain swiping action by theuser50, and in response, provide a text-box for allowing theuser50 to enter text (e.g., English letters, letters or characters in other languages, numbers, punctuation, special characters, symbols, emoji, text graphics, etc.). This swipe-to-type feature may provide a fast and convenient way for theuser50 to enter text and other information.
• In some embodiments, in addition to the input signal generated using finger action performed on the assignedarea206, theprocessing unit204 of theapparatus200 may also obtain sensor signal that is associated with an orientation of theapparatus200, and use the combination of the input signal and the sensor signal to cause a change of the content being displayed by theimage display device101. In particular, theprocessing unit204 may determine a command based on the combination of the input signal and the sensor signal, and may transmit such command to theimage display device101. Theimage display device101 then processes the command and changes the content based on the command. Alternatively, theprocessing unit204 may determine a first command for the input signal, and a second command for the sensor signal, and may transmit both commands to theimage display device101. Theimage display device101 then processes the commands and changes the content based on both commands.
• For example, in some embodiments, theapparatus200 further includes an orientation sensor for sensing an orientation of theapparatus200. The orientation sensor may be an inertial measurement unit (IMU), or any of other types of orientation sensor. In such cases, theprocessing unit204 may be configured to generate the electronic signal to cause the content displayed by theimage display device101 to change based on both (1) the finger(s)-generated input signal and (2) the sensed orientation of the apparatus.
• In some cases, the finger(s)-generated input signal may be associated with a pinching or un-pinching action of theuser50. In such cases, the apparatus theprocessing unit204 may be configured to generate the electronic signal to cause the content displayed by theimage display device101 to change based on (1) the input signal associated with the pinching or un-pinching action and (2) the sensed orientation of theapparatus200. For example, theprocessing unit204 may be configured to (1) generate the electronic signal to cause the content to contract or expand in a first plane if the pinching or un-pinching action is sensed by theapparatus200 while theapparatus200 is at a first orientation, and (2) generate the electronic signal to cause the content to contract or expand in a second plane if the pinching or un-pinching action is sensed by theapparatus200 while theapparatus200 is at a second orientation different from the first orientation, the second plane being different from the first plane. In some embodiments, theapparatus200 may be considered as having the first orientation when a major axis of theapparatus200 forms an angle with a horizontal plane that is less than 45°, or more preferably less than 30°. Also, theapparatus200 may be considered as having the second orientation when a major axis of theapparatus200 forms an angle with a vertical plane that is less than 45°, or more preferably less than 30°.
• In addition, the first plane in which the content contracts or expands due to apparatus being at the first orientation may be perpendicular to the second plane in which the contact contracts or expands due to apparatus being at the second orientation. For example, the first plane may be a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• In other embodiments, instead of a pinching or un-pinching action of theuser50, the input signal may be resulted from other types of finger(s)-generated actions. For example, the input signal may be resulted from a swiping action performed by theuser50. In such cases, the apparatus theprocessing unit204 may be configured to generate the electronic signal to cause the content displayed by theimage display device101 to change based on (1) the input signal associated with the swiping action and (2) the sensed orientation of theapparatus200.
• For example, theprocessing unit204 may be configured to (1) generate the electronic signal to cause the content to move in a first plane if the pinching or un-pinching action is sensed by theapparatus200 while theapparatus200 is at a first orientation, and (2) generate the electronic signal to cause the content to move in a second plane if the pinching or un-pinching action is sensed by theapparatus200 while theapparatus200 is at a second orientation different from the first orientation, the second plane being different from the first plane. In some embodiments, theapparatus200 may be considered as having the first orientation when a major axis of theapparatus200 forms an angle with a horizontal plane that is less than 45°, or more preferably less than 30°. Also, theapparatus200 may be considered as having the second orientation when a major axis of theapparatus200 forms an angle with a vertical plane that is less than 45°, or more preferably less than 30°.
• In addition, the first plane in which the content moves due to apparatus being at the first orientation may be perpendicular to the second plane in which the contact moves due to apparatus being at the second orientation. For example, the first plane may be a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• In some embodiments, the content is in a virtual three-dimensional environment, and theprocessing unit204 is configured to generate the electronic signal to cause the content displayed by theimage display device101 to change by moving the content closer to or further from theuser50 when the swiping action is sensed by theapparatus200 while the orientation of theapparatus200 is approximately parallel to a horizontal plane (e.g., forming an angle that is ±30° from the horizontal plane). Theprocessing unit204 is also configured to generate the electronic signal to cause the content displayed by theimage display device101 to move in a vertical plane in the three-dimensional environment when the swiping action is sensed by theapparatus200 while the orientation of theapparatus200 is approximately perpendicular to a horizontal plane (e.g., forming an angle that is ±30° from the a vertical plane).
• In some embodiments, theprocessing unit204 is configured to generate the electronic signal to cause the content to expand in one or more directions based on the sensed orientation of theapparatus200. For example, if theapparatus200 is in a first orientation (e.g., upright orientation), the electronic signal may be a command to expand the content in a first direction that corresponds with the first orientation of theapparatus200. If theapparatus200 is in a second orientation (e.g., a face-up orientation), the electronic signal may be a command to expand the content in a second direction that corresponds with the second orientation of theapparatus200.
• Also, in some embodiments, theprocessing unit204 is configured to generate the electronic signal to cause the content to move or rotate based on the sensed orientation of theapparatus200.
• Furthermore, in some embodiments, theapparatus200 further includes a movement sensor for sensing a movement of theapparatus200, wherein theprocessing unit204 is configured to generate the electronic signal to cause the content displayed by theimage display device101 to change (e.g., to move) based on the sensed movement of theapparatus200. The movement sensor may include an accelerometer, or any of other components that can detect movement and/or direction of movement.
• FIGS. 19A and 19B illustrate examples of some of the above described features. In particular,FIG. 19A illustrates examples of finger(s)-action performed on the apparatus ofFIG. 6 while apparatus is in a face-up orientation, and examples of the corresponding effects on a displayed content.FIG. 19B illustrates examples of finger(s)-action performed on the apparatus ofFIG. 6 while apparatus is in an up-right orientation, and examples of the corresponding effects on a displayed content. As shown inFIG. 19A, when the apparatus is held by theuser50 in a face-up orientation, theuser50 may perform (1) a two-fingersradial move400aon the assignedfirst area206 of thescreen202 of theapparatus200 to cause anobject402 displayed by theimage display device101 to rotate about a Y-axis, (2) avertical swipe move400bon the assignedfirst area206 of thescreen202 of theapparatus200 to cause theobject402 to move further away along a Z-axis, and (3) anun-pinch move400cto on the assignedfirst area206 of thescreen202 of theapparatus200 to cause theobject402 to increase in size along the Z-axis. On the other hand, as shown inFIG. 19B, when the apparatus is held by theuser50 in an upright orientation, theuser50 may perform (1) a two-fingersradial move410aon the assignedfirst area206 of thescreen202 of theapparatus200 to cause theobject402 displayed by theimage display device101 to rotate about the Z-axis, (2) avertical swipe move410bon the assignedfirst area206 of thescreen202 of theapparatus200 to cause theobject402 to move up (or down) along a Y-axis, and (3) anun-pinch move410cto on the assignedfirst area206 of thescreen202 of theapparatus200 to cause theobject402 to increase in size along the Y-axis. Accordingly, depending on the orientation of theapparatus200, the same finger(s)-action would achieve different effects on the content displayed by theimage display device101.
• In some embodiments, theprocessing unit204 may be configured to detect six different scenarios of finger(s)-action input(s) and/or orientation sensor signal, and generate a corresponding control signal to cause an object displayed by theimage display device101 to move in six different degrees of freedom, respectively. For example, based on the finger(s)-action input and/or orientation sensor signal, an object displayed by theimage display device101 may be translated along an X-axis, translated along an Y-axis, translated along an Z-axis, rotated about the X-axis, rotated about the Y-axis, and rotated about the Z-axis.
• In some embodiments, the electronic signal generated by theprocessing unit204 of theapparatus200 may also cause theimage display device101 to display touch hint, so that theuser50 viewing the screen of theimage display device101 can see what finger(s)-action is being detected by theprocessing unit204. For example, if theuser50 is performing a two-fingers touch move on the first assignedarea206 of thescreen202 to inertial move an object being displayed by theimage display device101, the electronic signal generated by theprocessing unit204 based on a detection of the two-fingers touch move may be transmitted to theimage display device101, which then displays a graphic (touch hint) to inform theuser50 that he/she is performing a two-fingers touch move. The same feature may be applied for other types of detected finger(s)-actions. Thus, theimage display device101 may display different finger(s)-action indicators for informing theuser50 that different respective finger(s)-actions have been detected by theapparatus200.
• It should be noted that in one or more embodiments, any feature described herein may be performed by theprocessing unit204 of theapparatus200, and/or theprocessing unit130 of theimage display device101. Accordingly, one or more features described herein as being performed by theprocessing unit204 of theapparatus200 may alternatively be performed by theprocessing unit130 of theimage display device101, or by the combination of theprocessing unit204 of theapparatus200 and theprocessing unit130 of theimage display device101. Similarly, one or more features described herein as being performed by theprocessing unit130 of theimage display device101 may alternatively be performed by theprocessing unit204 of theapparatus200, or by the combination of theprocessing unit204 of theapparatus200 and theprocessing unit130 of theimage display device101.
• Gesture Handoff
• In some embodiments, theimage display device101 may include a camera for detecting gestures of the hand(s) of theuser50, and theprocessing unit130 of theimage display device101 may interpret the detected gestures to generate corresponding control signals for operating theimage display device101. In some embodiments, a gesture performed by theuser50 on the assignedfirst area206 may be combined with a gesture detected by the camera of theimage display device101 to achieve a desired operation. For example, in some embodiments, theapparatus200 may display content on its screen, and theuser50 may perform a pinching action on the screen of theapparatus200 to extract the displayed content. Theprocessing unit204 detects the pinching action performed using theapparatus200, and interprets it as a control signal to extract the content. Theuser50 may then perform an un-pinching action in the field of view of the camera of theimage display device101. Theimage display device101 detects the un-pinching action, and interprets it as a control signal to place the content in a certain location presented by (or viewable through) the screen of theimage display device101. Accordingly, gesture detection performed by theapparatus200 may be combined with gesture detection performed by theimage display device101 to operate on content displayed by theapparatus200 and/or content displayed by theimage display device101.
• In another example, theimage display device101 may display content on its screen, and theuser50 may perform a pinching action in front of the camera of theimage display device101 to extract the displayed content. Theprocessing unit130 of theimage display device101 detects the pinching action as captured by the camera, and interprets it as a control signal to extract the content. Theuser50 may then perform an un-pinching action on theapparatus200. Theapparatus200 detects the un-pinching action, and interprets it as a control signal to place the content in a certain location presented by the screen of theapparatus200.
• Also, in some embodiments, content displayed by theapparatus200 may be moved to the “environment” of theimage display device101 by selecting the content at theapparatus200, and moving (e.g., by touch move, swiping, etc.) the content to an edge of the display area of theapparatus200. When theprocessing unit204 detects that the content is moved to the edge of the display, theprocessing unit204 then determines that the content is to be moved to the display screen of theimage display device101. Theapparatus200 then transmits a control signal to cause the content to be displayed by theimage display device101. After the content has been “moved” from the screen of theapparatus200 to the screen of theimage display device101, thenuser50 may then perform further operations on the content by using the apparatus200 (e.g., operating on the assignedfirst area206 to move the content, resize the content, etc.) and/or using hand gesture for detection by the camera of theimage display device101.
• In some embodiments, the “movement” of content from the screen of theapparatus200 to the screen of the image display device101 (or vice versa) may be executed to provide temporal continuity. For example, as soon as the content disappears on the screen of theapparatus200, theprocessing unit130 of theimage display device101 may immediately generate and provide an image of the content for display by the screen of theimage display device101. In other embodiments, the “movement” of content from the screen of theapparatus200 to the screen of the image display device101 (or vice versa) may be executed without any temporal continuity. For example, after the content disappears on the screen of theapparatus200, there may be a time lag for providing an image of the content for display by the screen of theimage display device101.
• Also, in some embodiments, the “movement” of content from the screen of theapparatus200 to the screen of the image display device101 (or vice versa) may be executed to provide spatial continuity. For example, if content on the screen of theapparatus200 is moved to the right edge of the display for moving the content to the screen of theimage display device101, then as soon as the content is moved out of the screen of theapparatus200, theimage display device101 may immediately display the content next to the left edge of the screen of theimage display device101. Similarly, as another example, if content on the screen of theimage display device101 is moved to the right edge of the display for moving the content to the screen of theapparatus200, then as soon as the content is moved out of the screen of theimage display device101, theapparatus200 may immediately display the content next to the left edge of the screen of theapparatus200.
• In other embodiments, the “movement” of content from the screen of theapparatus200 to the screen of the image display device101 (or vice versa) may be executed without any spatial continuity. For example, after the content on the screen of theapparatus200 is moved out of the screen, the content may always appear at a predetermined location (e.g., center) on the screen of theimage display device101.
• In one example of use, the above features may be implemented to allow theuser50 to deal cards in a poker game. For example, theuser50 may perform a swiping action using theapparatus200 to dispatch a poker card. The poker card may move out of the screen of theapparatus200, and may appear on the screen of the image display device101 (or on the screen of another image display device another user is using, wherein the other image display device may be communicating with theimage display device101 via a network, such as the Internet, a Wi-Fi, etc.).
• Also, in some embodiments, the camera of theimage display device101 for detecting user's gesture may be used to view the user's hand while theuser50 is performing finger(s)-action input on theapparatus200. In such cases, theprocessing unit130 of theimage display device101 may detect the finger(s)-action by theuser50, and verify a detection of the finger(s) gesture detected by theapparatus200.
• In addition, in some embodiments, theimage display device101 is configured to detect whether theuser50 is using theapparatus200. If theuser50 is using theapparatus200 for entering input, theimage display device101 may disable its gesture-detection camera. Alternatively, in some examples, if theuser50 is using theapparatus200 for entering input, theimage display device101 may keep its camera enabled and simply refrain from performing one or more gesture detection processes on images that are captured by the camera. As soon as theuser50 lifts his/her fingers away from the screen of theapparatus200, theimage display device101 may then activate the gesture-detection camera to allow the camera to take over the gesture-detection function. In one implementation, theprocessing unit130 of theimage display device101 is configured to disable the gesture-detection camera of theimage display device101 as long as theapparatus200 is receiving input generated by the finger(s)-actions of theuser50. Theprocessing unit130 may be configured to receive a signal from theapparatus200 as soon as theuser50 lifts his/her finger(s) away from the screen of theapparatus200. In response to such signal, theprocessing unit130 then activates the gesture-detection camera of theimage display device101 to allow gestures detected by the camera to be used as input for theimage display device101.
• Method Performed by the Processing Unit and/or Application in the Processing Unit
• FIG. 20 illustrates amethod500 in accordance with some embodiments. Themethod500 may be performed by theprocessing unit204 and/or an application in theprocessing unit204 of theapparatus200. Themethod500 includes: assigning afirst area206 of ascreen202 of theapparatus200 to sense finger-action of theuser50 of theimage display device101, wherein theimage display device101 is configured for head-worn by the user, and theapparatus200 is different from the image display device101 (item502). Theimage display device101 may be any of the ones shown inFIGS. 1-4. Themethod500 also includes: generating an electronic signal to cause a change in a content displayed by theimage display device101 based on the finger-action of the user sensed by the assignedfirst area206 of thescreen202 of the apparatus101 (item504).
• In some embodiments, the electronic signal may be a command determined by theprocessing unit204 or by the application in theprocessing unit204 based on gesture recognition. The electronic signal may be transmitted by theapparatus200 to theprocessing unit130 of theimage display device101, which then changes the content based on the electronic signal. Alternatively, if theapparatus200 participates in providing the content for display by theimage display device101, then theprocessing unit204 may change the content being displayed based on the electronic signal that it provides. Thus, the electronic signal may directly or indirectly cause the change in the content displayed by theimage display device101.
• Optionally, in themethod500, thescreen202 has a touch-sensitive region210, and wherein the assignedfirst area206 is less than a total area of the touch-sensitive region210.
• Optionally, themethod500 may further include ignoring input generated by theuser50 using a portion of the touch-sensitive region210 that is not a part of the assignedfirst area206, and that is not a part of an assigned button.
• Optionally, themethod500 may further include generating a control signal to operate a feedback component in response to the finger-action of theuser50.
• Optionally, in themethod500, the first area has a boundary, and wherein the finger-action of theuser50 comprises a finger of the user crossing, reaching, or moving to a location that is within a prescribed distance from, the boundary.
• Optionally, in themethod500, thefirst area206 has one or more boundaries that at least partially surround a reference location, and wherein the control signal is for operating the feedback component in response to a finger of theuser50 reaching a prescribed distance from the reference location.
• Optionally, themethod500 further includes generating different control signals to operate a feedback component to generate different respective types of feedback based on different respective spatial relationships between one or more finger(s) of theuser50 with respect to thefirst area206.
• Optionally, in themethod500, the different types of feedback comprise a first haptic impulse with a first amplitude, and a second haptic impulse with a second amplitude that is different from the first amplitude.
• Optionally, in themethod500, the different types of feedback comprise a first number of haptic impulse(s), and a second number of haptic impulse(s) that is different from the first number.
• Optionally, in themethod500, the different types of feedback comprise a first series of haptic impulses with a first frequency, and a second series of haptic impulses with a second frequency that is different from the first frequency.
• Optionally, in themethod500, the different respective spatial relationships comprise different distances between (1) one of the one or more finger(s), or a point that is between two fingers, of the user and (2) a reference location within the assignedfirst area206.
• Optionally, in themethod500, the reference location comprises a center of the assignedfirst area206.
• Optionally, in themethod500, the different distances exceed a threshold.
• Optionally, in themethod500, the different respective spatial relationships comprise one of the one or more finger(s), or a point that is between two fingers, reaching different respective distances from a boundary of the assignedfirst area206.
• Optionally, in themethod500, the assignedfirst area206 has a first boundary and a second boundary; wherein the different types of the feedback comprise at least a first type of feedback and a second type of feedback; wherein the feedback component is operated to generate the first type of feedback when one or more finger(s) of theuser50 crosses, reaches, or moves to a location that is within a prescribed distance from, the first boundary; and wherein the feedback component is operated to generate the second type of feedback when one or more finger(s) of theuser50 crosses, reaches, or moves to a location that is within a prescribed distance from, the second boundary.
• Optionally, in themethod500, the first boundary comprises a left or right boundary, and the second boundary comprises a top or bottom boundary, of the assignedfirst area206.
• Optionally, in themethod500, the control signal for operating the feedback component is based on a swiping direction.
• Optionally, themethod500 further includes receiving an input signal associated with a pinching or un-pinching action performed by theuser50 on the assignedfirst area210 of thescreen202.
• Optionally, in themethod500, the electronic signal is for changing a size of the content displayed by theimage display device101 in response to the input signal that is associated with the pinching or un-pinching action.
• Optionally, themethod500 further includes obtaining an orientation of theapparatus200 from an orientation sensor, wherein the electronic signal is for changing the content displayed by theimage display device101 based on the input signal that is associated with the pinching or un-pinching action and the orientation of theapparatus200.
• Optionally, in themethod500, the content is changed by contracting or expanding the content in a first plane if the pinching or un-pinching action is sensed by the assignedfirst area206 while theapparatus200 is at a first orientation; and wherein the content is changed by contracting or expanding the content in a second plane if the pinching or un-pinching action is sensed by the assignedfirst area206 while theapparatus200 is at a second orientation different from the first orientation, the second plane being different from the first plane.
• Optionally, in themethod500, theapparatus200 has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°.
• Optionally, in themethod500, theapparatus200 has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.
• Optionally, in themethod500, the first plane comprises a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• Optionally, in themethod500, the first plane and the second plane are with respect to a virtual three-dimensional environment.
• Optionally, in themethod500, the first plane is perpendicular to the second plane.
• Optionally, themethod500 further includes receiving an input signal associated with a swiping action performed by theuser50 on the assignedfirst area206 of thescreen202.
• Optionally, in themethod500, the electronic signal is for moving the content displayed by theimage display device101 in response to the sensed swiping action.
• Optionally, themethod500 further includes obtaining an orientation of theapparatus200 from an orientation sensor, wherein the electronic signal is for changing the content displayed by theimage display device101 based on the input signal that is associated with the swiping action and the orientation of theapparatus200.
• Optionally, in themethod500, the content is changed by moving the content in a first plane if the swiping action is sensed by the assignedfirst area206 while theapparatus200 is at a first orientation; and wherein the content is changed by moving the content in a second plane if the swiping action is sensed by the assignedfirst area206 while theapparatus200 is at a second orientation different from the first orientation, the second plane being different from the first plane.
• Optionally, in themethod500, theapparatus200 has the first orientation when a major axis of theapparatus200 forms an angle with a horizontal plane that is less than 45°.
• Optionally, in themethod500, theapparatus200 has the second orientation when a major axis of theapparatus200 forms an angle with a vertical plane that is less than 45°.
• Optionally, in themethod500, the first plane comprises a Y-Z plane in a virtual three-dimensional environment, and the second plane comprises a X-Y plane in the virtual three-dimensional environment.
• Optionally, in themethod500, the first plane and the second plane are with respect to a virtual three-dimensional environment.
• Optionally, in themethod500, the first plane is perpendicular to the second plane.
• Optionally, in themethod500, the content is in a virtual three-dimensional environment, and wherein the control signal is for causing the content displayed by theimage display device101 to move closer to or further from the user when the swiping action is sensed by the assignedfirst area206 while the orientation of theapparatus200 is approximately parallel to a horizontal plane.
• Optionally, in themethod500, the content is in a virtual three-dimensional environment, and wherein the control signal is for causing the content displayed by theimage display device101 to move in a vertical plane in the three-dimensional environment when the swiping action is sensed by the assignedfirst area206 while the orientation of theapparatus200 is approximately perpendicular to a horizontal plane.
• Optionally, themethod500 further includes obtaining a sensor input indicating a sensed orientation of theapparatus200, wherein the electronic signal is for changing the content displayed by theimage display device101 based on the sensor input indicating the sensed orientation of theapparatus200.
• Optionally, in themethod500, the control signal is for changing the content by expanding the content in one or more directions based on the sensor input indicating the sensed orientation of theapparatus200.
• Optionally, in themethod500, the control signal is for changing the content by rotating the content based on the sensor input indicating the sensed orientation of theapparatus200.
• Optionally, in themethod500, the control signal is for changing the content by moving the content based on the sensor input indicating the sensed orientation of theapparatus200.
• Optionally, themethod500 further includes obtaining a sensor input indicating a sensed movement of theapparatus200, wherein the control signal is for changing the content displayed by theimage display device101 based on the sensor input indicating the sensed movement of theapparatus200.
• Optionally, in themethod500, the electronic signal is for changing the content by moving the content based on the sensor input indicating the sensed movement of theapparatus200.
• Optionally, in themethod500, theapparatus200 is a handheld apparatus.
• Optionally, the handheld apparatus comprises a cell phone, a smart phone, a personal-digital-assistant (PDA), or a tablet.
• Optionally, in themethod500, the assignedfirst area206 of thescreen202 has no displayed object while the assignedfirst area206 of thescreen202 is sensing the finger-action of theuser50.
• Optionally, themethod500 further includes operating thescreen202 to display a grid of dots in the assignedfirst area206 of thescreen202.
• Optionally, themethod500 further includes changing a feature of one or more of the dots in response to theuser50 touching a part of the assignedfirst area206 of thescreen202 where the one or more of the dots are displayed.
• Optionally, themethod500 further includes assigning a second area of thescreen202 as a first button.
• Optionally, in themethod500, the first button is a “HOME” button.
• Optionally, themethod500 further includes assigning a third area of thescreen202 as a second button.
• Optionally, in themethod500, the second button is a “TOGGLE” button.
• Optionally, themethod500 further includes assigning a fourth area of thescreen202 as a third button.
• Optionally, in themethod500, the third button is a “BUMPER” button.
• Optionally, themethod500 further includes: assigning a second area of thescreen202 as a keyboard activation button, and operating thescreen202 to display a keyboard in response to theuser50 touching the assigned second area of the screen.
• Optionally, themethod500 further includes wirelessly communicating with theimage display device101.
• Optionally, themethod500 further includes communicating with theimage display device101 via a cable.
• Optionally, in themethod500, theapparatus200 comprises a non-transitory medium storing an instruction, and wherein the act of assigning thefirst area206 of thescreen202 for sensing finger-action of theuser50 is performed based on the instruction.
• Optionally, in themethod500, the change in the content comprises a change in a size of the content, a change in a position of the content, a change in a shape of the content, a change in a color of the content, a replacement of information in the content, an increase or decrease in a quantity of information in the content, or any combination of the foregoing.
• In some embodiments, themethod500 may further include: detecting the gestures or finger-actions, such as one finger touch move, two fingers touch move, one finger swipe, two fingers swipe, two fingers pinch, two fingers un-pinch, two fingers radial move, tap, double tap, or any combination of the foregoing; determining a command for a corresponding detected gesture or finger-action; and transmitting the command to an image display device.
• Also, one or more embodiments, any feature (e.g., function, item, step, etc.) in themethod500 may be performed by theprocessing unit204 of theapparatus200, and/or theprocessing unit130 of theimage display device101.
• Specialized Processing System
• In some embodiments, themethod500 described herein may be performed by theprocessing unit204 executing an application, or by the application. The application may contain a set of instruction. In one implementation, a specialized processing system having a non-transitory medium storing the set of instruction for the application may be provided. The execution of the instruction by theprocessing unit204 of theapparatus200 will cause theprocessing unit204 to perform the features described herein. As shown inFIG. 21, in some embodiments, the specialized processing system may include one or more server(s)600. The non-transitory medium storing theinstruction610 for the application may be implemented in the server(s)600 configured to provide the set ofinstruction610 for download to theapparatus200 through a network, such as through the Internet. The server(s)600 may be configured to obtain a request from theapparatus200 to download the set ofinstruction610, process the request, and provide the set ofinstruction610 based on the request and/or the processing of the request.
• In some embodiments, the processing of the request by the server(s)600 may include: verifying theapparatus200, verifying theuser50 of theapparatus200, determining the brand of theapparatus200, determining the model of theapparatus200, or any combination of the foregoing.
• It should be noted that the server(s)600 is a specialized processing system in that it contains instruction for execution by a processing unit of an apparatus to provide unique tangible effects in a real world. The features provided by the server(s)600 provide improvements in the technology of image display devices and systems, as described herein.
• By means of non-limiting examples, the instruction610 provided by the server(s)600 may include instruction for: assigning the first area206 of the screen202 of the apparatus200 for sensing finger-action; assigning the second area240 of the screen202 of the apparatus200 as a first button; assigning the third area250 of the screen202 as a second button; assigning the fourth area260 of the screen202 as a third button; assigning another area270 of the screen202 as a keyboard activation button; generating an electronic signal to cause a change in a content displayed by the image display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus; ignoring input generated by the user using a portion of the touch-sensitive region that is not a part of the assigned first area, and that is not a part of an assigned button; generating a control signal to operate a feedback component in response to the finger-action of the user; operating the feedback component in response to a finger of the user reaching a prescribed distance from the reference location; generating different control signals to operate a feedback component to generate different respective types of feedback based on different respective spatial relationships between one or more finger(s) of the user with respect to the first area; operating a feedback component to generate the first type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the first boundary; operating the feedback component to generate the second type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the second boundary; operating a feedback component based on a swiping direction; generating an electronic signal for changing a size of the content displayed by the image display device in response to an input signal that is associated with a pinching or un-pinching action; obtaining an orientation of the apparatus from an orientation sensor; generating an electronic signal for changing the content displayed by the image display device based on the input signal that is associated with a pinching or un-pinching action and the orientation of the apparatus; generating an electronic signal for moving the content displayed by the image display device in response to a sensed swiping action; generating an electronic signal for changing the content displayed by the image display device based on the input signal that is associated with the swiping action and an orientation of the apparatus; generating an electronic signal to move the content in a first plane if the swiping action is sensed by the assigned first area while the apparatus is at a first orientation; generating an electronic signal to move the content in a second plane if the swiping action is sensed by the assigned first area while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane; generating an electronic signal for causing the content displayed by the image display device to move closer to or further from the user when the swiping action is sensed by the assigned first area while the orientation of the apparatus is approximately parallel to a horizontal plane; generating an electronic signal for causing the content displayed by the image display device to move in a vertical plane in the three-dimensional environment when the swiping action is sensed by the assigned first area while the orientation of the apparatus is approximately perpendicular to a horizontal plane; generating an electronic signal for changing the content displayed by the image display device based on a sensor input indicating a sensed orientation of the apparatus; generating an electronic signal for changing the content by expanding the content in one or more directions based on a sensor input indicating a sensed orientation of the apparatus; generating an electronic signal for changing the content by rotating the content based on a sensor input indicating a sensed orientation of the apparatus; generating an electronic signal for changing the content by moving the content based on a sensor input indicating a sensed orientation of the apparatus; obtaining a sensor input indicating a sensed movement of the apparatus; generating an electronic signal for changing the content displayed by the image display device based on a sensor input indicating a sensed movement of the apparatus; generating an electronic signal is changing the content by moving the content based on a sensor input indicating a sensed movement of the apparatus; turning off pixels of the assigned first area of the screen so that the assigned first area has no displayed object while the assigned first area of the screen is sensing the finger-action of the user; operating the screen of the apparatus to display a grid of dots in the assigned first area of the screen; changing a feature of one or more of the dots in response to the user touching a part of the assigned first area of the screen where the one or more of the dots are displayed, or any combination of any of the foregoing.
• Also, theinstruction610 provided by the server(s)600 may include instruction for: detecting the gestures or finger-actions, such as one finger touch move, two fingers touch move, one finger swipe, two fingers swipe, two fingers pinch, two fingers un-pinch, two fingers radial move, tap, double tap, or any combination of the foregoing; determining a command for a corresponding detected gesture or finger-action; transmitting the command to an image display device.
• In other embodiments, theapparatus200 may also be considered as a specialized processing system. In particular, theapparatus200 is a specialized processing system in that it contains instruction stored in its non-transitory medium for execution by theprocessing unit204 to provide unique tangible effects in a real world. The features provided by the apparatus200 (as a result of theprocessing unit204 executing the instruction) provide improvements in the technology of image display devices and systems, as described herein.
• FIG. 22 is a block diagram illustrating an embodiment of aspecialized processing system1600 that can be used to implement various features described herein. For example, in some embodiments, theprocessing system1600 may be used to implement one or more of the server(s)600. In other embodiments, theprocessing system1600 may be used to implement theapparatus200.Processing system1600 includes abus1602 or other communication mechanism for communicating information, and aprocessor1604 coupled with thebus1602 for processing information. Theprocessor system1600 also includes amain memory1606, such as a random access memory (RAM) or other dynamic storage device, coupled to thebus1602 for storing information and instructions to be executed by theprocessor1604. Themain memory1606 also may be used for storing temporary variables or other intermediate information during execution of instructions to be executed by theprocessor1604. Theprocessor system1600 further includes a read only memory (ROM)1608 or other static storage device coupled to thebus1602 for storing static information and instructions for theprocessor1604. Adata storage device1610, such as a magnetic disk, solid state disk, or optical disk, is provided and coupled to thebus1602 for storing information and instructions.
• Theprocessor system1600 may be coupled via thebus1602 to adisplay1612, such as a screen, for displaying information to a user. In some cases, if theprocessing system1600 is part of the apparatus that includes a touch-screen, thedisplay1612 may be the touch-screen. Aninput device1614, including alphanumeric and other keys, is coupled to thebus1602 for communicating information and command selections toprocessor1604. Another type of user input device iscursor control1616, such as a mouse, a trackball, or cursor direction keys for communicating direction information and command selections toprocessor1604 and for controlling cursor movement ondisplay1612. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), that allows the device to specify positions in a plane. In some cases, if theprocessing system1600 is part of the apparatus that includes a touch-screen, theinput device1614 and the curser control may be the touch-screen.
• In some embodiments, theprocessor system1600 can be used to perform various functions described herein. According to some embodiments, such use is provided byprocessor system1600 in response toprocessor1604 executing one or more sequences of one or more instructions contained in themain memory1606. Those skilled in the art will know how to prepare such instructions based on the functions and methods described herein. Such instructions may be read into themain memory1606 from another processor-readable medium, such asstorage device1610. Execution of the sequences of instructions contained in themain memory1606 causes theprocessor1604 to perform the process steps described herein. One or more processors in a multi-processing arrangement may also be employed to execute the sequences of instructions contained in themain memory1606. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the various embodiments described herein. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.
• The term “processor-readable medium” as used herein refers to any medium that participates in providing instructions to theprocessor1604 for execution. Such a medium may take many forms, including but not limited to, non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical, solid state or magnetic disks, such as thestorage device1610. A non-volatile medium may be considered an example of non-transitory medium. Volatile media includes dynamic memory, such as themain memory1606. A volatile medium may be considered an example of non-transitory medium. Transmission media includes coaxial cables, copper wire and fiber optics, including the wires that comprise thebus1602. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.
• Common forms of processor-readable media include, for example, a flexible disk, hard disk, magnetic tape, or any other magnetic medium, a CD-ROM, any other optical medium, any other physical medium with patterns of holes, a RAM, a PROM, and EPROM, a FLASH-EPROM, solid state disks any other memory chip or cartridge, a carrier wave as described hereinafter, or any other medium from which a processor can read.
• Various forms of processor-readable media may be involved in carrying one or more sequences of one or more instructions to theprocessor1604 for execution. For example, the instructions may initially be carried on a magnetic disk or solid state disk of a remote computer. The remote computer can load the instructions into its dynamic memory and send the instructions over a network, such as the Internet. Theprocessing system1600 can receive the data on a network line. Thebus1602 carries the data to themain memory1606, from which theprocessor1604 retrieves and executes the instructions. The instructions received by themain memory1606 may optionally be stored on thestorage device1610 either before or after execution by theprocessor1604.
• Theprocessing system1600 also includes acommunication interface1618 coupled to thebus1602. Thecommunication interface1618 provides a two-way data communication coupling to anetwork link1620 that is connected to alocal network1622. For example, thecommunication interface1618 may be a local area network (LAN) card to provide a data communication connection to a compatible LAN. Wireless links may also be implemented. In any such implementation, thecommunication interface1618 sends and receives electrical, electromagnetic or optical signals that carry data streams representing various types of information.
• Thenetwork link1620 typically provides data communication through one or more networks to other devices. For example, thenetwork link1620 may provide a connection throughlocal network1622 to ahost computer1624 or toequipment1626. The data streams transported over thenetwork link1620 can comprise electrical, electromagnetic or optical signals. The signals through the various networks and the signals on thenetwork link1620 and through thecommunication interface1618, which carry data to and from theprocessing system1600, are exemplary forms of carrier waves transporting the information. Theprocessing system1600 can send messages and receive data, including program code, through the network(s), thenetwork link1620, and thecommunication interface1618.
• It should be noted that theapparatus200 described herein is not limited to having the functionalities and features described, and that theapparatus200 may be configured to provide other features for use with theimage display device101. And, as mentioned above, in some examples, some or all of such functionalities and features may be provided at least in part by way of an application (also referred to herein as an “app”) that is running on theapparatus200. By means of non-limiting examples, the apparatus200 may be configured to allow the user50 to: sign-up an account associated with the use of the image display device101; log into an application using credentials set up on the account; retrieve or re-set credentials of the account; switch between user profiles; automatically sync user account data between devices; manage the account that is associated with the use of the image display device101; manage configuration and use of the image display device101; obtain supplemental learning and support functionalities for the image display device101; obtain help for troubleshooting of the image display device101 and/or the apparatus200; access privacy, security, and data policies associated with the account; access app store to obtain applications for the image display device101 and/or the apparatus200; perform search queries for apps; view app detail pages; view order or purchase history; receive push notifications (e.g., application updates, etc.); perform media extraction from local device; open media files (e.g., photos, videos, etc.); select individual media files to send to the image display device101, select individual medial files to send from the image display device101 to the apparatus200; receive notifications regarding the use of image display device101 (e.g., notifications from applications for the image display device101); receive notifications from third-party developers concerning use of the image display device101; manage smartphone notification settings for the different apps for the image display device101; find the image display device101 or other image display device(s); obtain battery status of the image display device101; mirror the image display device101 on the apparatus200 (wherein content displayed on the screen of the image display device101, and/or environment surrounding the user50 as viewed through the screen and captured by camera(s) of the image display device101, may be sent to the apparatus200 for display by the apparatus200); or any combination of the foregoing. Theprocessing unit204 of theapparatus200 may be configured to execute instructions to cause theapparatus200 to provide one or more of the above features.
• Exemplary aspects of the disclosure, together with details regarding material selection and manufacture have been set forth above. As for other details of the present disclosure, these may be appreciated in connection with the above-referenced patents and publications as well as generally known or appreciated by those with skill in the art. The same may hold true with respect to method-based aspects of the disclosure in terms of additional acts as commonly or logically employed.
• In addition, though the disclosure has been described in reference to several examples optionally incorporating various features, the disclosure is not to be limited to that which is described or indicated as contemplated with respect to each variation of the disclosure. Various changes may be made to the disclosure described and equivalents (whether recited herein or not included for the sake of some brevity) may be substituted without departing from the true spirit and scope of the disclosure. In addition, where a range of values is provided, it is understood that every intervening value, between the upper and lower limit of that range and any other stated or intervening value in that stated range, is encompassed within the disclosure.
• Also, it is contemplated that any optional feature of the inventive variations described may be set forth and claimed independently, or in combination with any one or more of the features described herein. Reference to a singular item, includes the possibility that there are plural of the same items present. More specifically, as used herein and in claims associated hereto, the singular forms “a,” “an,” “said,” and “the” include plural referents unless the specifically stated otherwise. It is further noted that any claim may be drafted to exclude any optional element. As such, this statement is intended to serve as antecedent basis for use of such exclusive terminology as “solely,” “only” and the like in connection with the recitation of claim elements, or use of a “negative” limitation.
• The breadth of the present disclosure is not to be limited to the examples provided and/or the subject specification, but rather only by the scope of claim language associated with this disclosure.
• In the foregoing specification, the disclosure has been described with reference to specific embodiments thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the disclosure. For example, the above-described process flows are described with reference to a particular ordering of process actions. However, the ordering of many of the described process actions may be changed without affecting the scope or operation of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than restrictive sense.

Claims (26)

1. An apparatus for use with an image display device configured for head-worn by a user, the apparatus comprising:

a screen; and

a processing unit configured to assign a first area of the screen to sense finger-action of the user;

wherein the processing unit is configured to generate an electronic signal to cause a change in a content displayed by the image display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus.

2. The apparatus ofclaim 1, wherein the screen has a touch-sensitive region, and wherein the assigned first area is less than a total area of the touch-sensitive region.

3. The apparatus ofclaim 1, wherein the processing unit is also configured to operate a feedback component in response to the finger-action of the user.

4. The apparatus ofclaim 3, wherein the first area has a boundary, and wherein the finger-action of the user comprises a finger of the user crossing, reaching, or moving to a location that is within a prescribed distance from, the boundary.

5. The apparatus ofclaim 3, wherein the processing unit is configured to operate the feedback component to generate different types of feedback based on different respective spatial relationships between one or more finger(s) of the user with respect to the first area.

6. The apparatus ofclaim 5, wherein the different respective spatial relationships comprise different distances between (1) one of the one or more finger(s), or a point that is between two fingers, of the user and (2) a reference location within the assigned first area.

7. The apparatus ofclaim 5, wherein the different respective spatial relationships comprise one of the one or more finger(s), or a point that is between two fingers, reaching different respective distances from a boundary of the assigned first area.

8. The apparatus ofclaim 5, wherein the assigned first area has a first boundary and a second boundary;

wherein the different types of the feedback comprise at least a first type of feedback and a second type of feedback;

wherein the processing unit is configured to operate the feedback component to generate the first type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the first boundary; and

wherein the processing unit is configured to operate the feedback component to generate the second type of feedback when one or more finger(s) of the user crosses, reaches, or moves to a location that is within a prescribed distance from, the second boundary.

9. The apparatus ofclaim 1, wherein the processing unit is configured to obtain an input signal associated with a pinching or un-pinching action performed on the assigned first area of the screen; and

wherein the processing unit is configured to generate the electronic signal to cause a size of the content displayed by the image display device to change based on the input signal that is associated with the pinching or un-pinching action.

10. The apparatus ofclaim 9, further comprising an orientation sensor for sensing an orientation of the apparatus, wherein the processing unit is also configured to generate the electronic signal to cause the content displayed by the image display device to change based on the input signal associated with the pinching or un-pinching action and the sensed orientation of the apparatus.

11. The apparatus ofclaim 10, wherein the processing unit is configured to generate the electronic signal to cause the content to contract or expand in a first plane if the pinching or un-pinching action is sensed by the apparatus while the apparatus is at a first orientation; and

wherein the processing unit is configured to generate the electronic signal to cause the content to contract or expand in a second plane if the pinching or un-pinching action is sensed by the apparatus while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane.

12. The apparatus ofclaim 11, wherein the apparatus has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°; and

wherein the apparatus has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.

13. The apparatus ofclaim 1, wherein the processing unit is configured to obtain an input signal associated with a swiping action performed on the assigned first area of the screen.

14. The apparatus ofclaim 13, wherein the processing unit is configured to generate the electronic signal to cause the content to change by moving the content in response to the input signal associated with the swiping action.

15. The apparatus ofclaim 13, further comprising an orientation sensor for sensing an orientation of the apparatus, wherein the processing unit is configured to generate the electronic signal to cause the content displayed by the image display device to change based on the input signal associated with the swiping action and the sensed orientation of the apparatus.

16. The apparatus ofclaim 15, wherein the processing unit is configured to generate the electronic signal to cause the content to move in a first plane if the swiping action is sensed by the apparatus while the apparatus is at a first orientation; and

wherein the processing unit is configured to generate the electronic signal to cause the content to move in a second plane if the swiping action is sensed by the apparatus while the apparatus is at a second orientation different from the first orientation, the second plane being different from the first plane.

17. The apparatus ofclaim 16, wherein the apparatus has the first orientation when a major axis of the apparatus forms an angle with a horizontal plane that is less than 45°; and

wherein the apparatus has the second orientation when a major axis of the apparatus forms an angle with a vertical plane that is less than 45°.

18. The apparatus ofclaim 15, wherein the content is in a virtual three-dimensional environment, and wherein the processing unit is configured to generate the electronic signal to cause the content displayed by the image display device to change by moving the content closer to or further from the user when the swiping action is sensed by the apparatus while the orientation of the apparatus is approximately parallel to a horizontal plane.

19. The apparatus ofclaim 15, wherein the content is in a virtual three-dimensional environment, and wherein the processing unit is configured to generate the electronic signal to cause the content displayed by the image display device to move in a vertical plane in the three-dimensional environment when the swiping action is sensed by the apparatus while the orientation of the apparatus is approximately perpendicular to a horizontal plane.

20. The apparatus ofclaim 1, further comprising an orientation sensor for sensing an orientation of the apparatus, wherein the processing unit is configured to generate the electronic signal (1) to cause the content to expand in one or more directions, (2) to cause the content to rotate, or (3) to cause the content to move, based on the sensed orientation of the apparatus.

21. The apparatus ofclaim 1, wherein the apparatus is a handheld apparatus.

22. The apparatus ofclaim 1, wherein the processing unit is also configured to assign a second area of the screen as a first button.

23. The apparatus ofclaim 1, wherein the processing unit is also configured to assign a second area of the screen as a keyboard activation button, and wherein the processing unit is configured to operate the screen to display a keyboard in response to the user touching the assigned second area of the screen.

24. The apparatus ofclaim 1, wherein the change in the content comprises a change in a size of the content, a change in a position of the content, a change in a shape of the content, a change in a color of the content, a replacement of information in the content, an increase or decrease in a quantity of information in the content, or any combination of the foregoing.

25. A method comprising:

assigning a first area of a screen of an apparatus to sense finger-action of a user of an image display device, wherein the image display device is configured for head-worn by the user, and wherein the apparatus is different from the image display device;

generating an electronic signal to cause a change in a content displayed by the image display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus.

26. A product comprising a non-transitory medium storing a set of instructions, an execution of which will cause a method to be performed, the method comprising:

assigning a first area of a screen of an apparatus to sense finger-action of a user of an image display device, wherein the image display device is configured for head-worn by the user, and wherein the apparatus is different from the image display device;

generating an electronic signal to cause a change in a content displayed by the image display device based on the finger-action of the user sensed by the assigned first area of the screen of the apparatus.

Images