US20220221932A1 - Controlling a function via gaze detection - Google Patents

Abstract

Aspects of the present disclosure relate to systems and methods for controlling a function of a computing system using gaze detection. In examples, one or more images of a user are received and gaze information may be determined from the received one or more images. Non-gaze information may be received when the gaze information is determined to satisfy a condition. Accordingly, a function may be enabled based on the received non-gaze information. In examples, the gaze information may be determined by extracting a plurality of features from the received one or more images, providing the plurality of features to a neural network, and determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

Description

BACKGROUND
• Striking the right balance between predicting user actions and conserving computing resources is difficult to do in computing systems. Many computing systems rely on inactivity timers, such as watchdog timers to control one or more functions or applications in an effort allow a user to interact with a computing system without consistently expending computing resources. For example, a computing system may enter a screensaver mode, dim a display, or lock a display in an effort to save power and protect content from unauthorized access. In some instances, however, actions taken by a computing system may frustrate a user. For example, while a user may be viewing content at a computing system display, the computing system may enter a locked state or enact a screensaver due to a lack of user activity or interaction with the computing system. In other examples, computing systems that rely on image-based activity systems, such as a facial recognition system, to authenticate and/or authorize users may inadvertently unlock or grant access to a user or increase a brightness of display device merely because the user appears within a field-of-view of a camera of the computing system. Thus, while the computing system may be readied for user interaction, the user may not desire to engage with the computing system thereby wasting computational resources such as energy. Further, while an image-based activity system may monitor user movement, monitoring user movement in real-time becomes impractical due at least in part to the excessive consumption of resources, such as power and/or computing cycles. It is with respect to these and other general considerations that embodiments have been described. Also, although relatively specific problems have been discussed, it should be understood that the embodiments should not be limited to solving the specific problems identified in the background.
SUMMARY
• Aspects of the present disclosure are directed to determining a user intent based on gaze information acquired for the user. The user intent, or gaze information, may then be used to control one or more functions associated with a computing device or system. As an example, a user may be interacting with a computing system that uses an image of a user together with a facial recognition system and/or a face-based authentication system to authenticate the user to a computing system. Accordingly, the user may be granted access to the computing system and/or one or more capabilities provided by the computing system. However, in accordance with examples of the present disclosure, a user's gaze is acquired and is used as a basis for controlling a subsequent function, such as invoking the facial recognition system and/or the face-based authentication system to authenticate the user. Accordingly, a user is not inadvertently granted access to a computing system merely because an image includes the user. Similarly, the user's gaze may be utilized to control a brightness of a display, provide extra security with respect to displayed content, and/or enable device functions based on one or more detected activities.
• In accordance with some examples of the present disclosure, a method for controlling a function via gaze detection is described. The method may include receiving one or more images of a user; determining gaze information from the received one or more images; receiving non-gaze information when the gaze information is determined to satisfy a condition; and enabling a function based on the non-gaze information.
• In accordance with some examples of the present disclosure, a system for controlling a function via gaze detection is described. The system may include a processor; an image sensor; and memory including instructions, which when executed by the processor, cause the processor to receive one or more images of a user from the image sensor; determine gaze information from the received one or more images; determine if the gaze information satisfies a condition; receive non-gaze information associated with at least one of a function or application of a computing system when the gaze information satisfies the condition; and enable the at least one function or application based on non-gaze information.
• In accordance with some examples of the present disclosure, a computer storage medium including instructions, which when executed by a processor, cause the processor to control at least one of a function or application of a computing system is described. The instructions may cause the processor to receive one or more images of a user from an image sensor; determine gaze information from the received one or more images; determine if the gaze information satisfies a condition; receive device dependent information associated with at least one of a function or application of a computing system when the gaze information is determined to satisfy the condition; and enable at least one function or application based on the device dependent information. In accordance with at least one aspect of the above example, the instructions, which when executed by a processor, cause the processor to: extract a plurality of features from the received one or more images; provide the plurality of features to a neural network; and determine, utilizing the neural network, a location at a display device at which a gaze of the user is directed, wherein, the gaze information satisfies the condition when the location at a display device at which the gaze of the user is directed coincides with a region including a graphical element associated with the at least one function or application.
• This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
BRIEF DESCRIPTION OF THE DRAWINGS
• Non-limiting and non-exhaustive examples are described with reference to the following Figures.
• FIGS. 1A-1B depict an example of controlling one or more functions of a computing system in accordance with examples of the present disclosure.
• FIGS. 2A-2D depict another example of using a user gaze to control one or more functions of a computing system in accordance with examples of the present disclosure.
• FIGS. 3A-3B depict another example of using a user gaze to control one or more functions of a computing system in accordance with examples of the present disclosure.
• FIGS. 4A-4C depict another example of relying on user gaze to control one or more functions of a computing system in accordance with examples of the present disclosure.
• FIGS. 5A-5D depict signal diagrams associated with controlling one or more functions and/or applications of a computing system using eye gaze in accordance with examples of the present disclosure.
• FIG. 6 depicts additional details of a computing system associated with controlling one or more functions and/or applications using eye gaze in accordance with examples of the present disclosure.
• FIG. 7 depicts example data structures in accordance with examples of the present disclosure.
• FIG. 8 depicts details of a method for controlling a function and/or application using a detected gaze of a user in accordance with examples of the present disclosure.
• FIG. 9 depicts details of a method for controlling a function and/or application using a detected gaze of an identified user in accordance with examples of the present disclosure.
• FIG. 10 depicts details of a method for controlling a function and/or application using a detected gaze of a user in accordance with examples of the present disclosure.
• FIG. 11 depicts a block diagram illustrating physical components of a computing system with which aspects of the disclosure may be practiced.
• FIGS. 12A-12B depict details of one or more computing systems in accordance with examples of the present disclosure.
• FIG. 13 depicts an architecture of a system for processing data received at a computing system in accordance with examples of the present disclosure.
DETAILED DESCRIPTION
• In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustrations specific embodiments or examples. These aspects may be combined, other aspects may be utilized, and structural changes may be made without departing from the present disclosure. Embodiments may be practiced as methods, systems, or devices. Accordingly, embodiments may take the form of a hardware implementation, an entirely software implementation, or an implementation combining software and hardware aspects. The following detailed description is therefore not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims and their equivalents.
• Aspects of the present disclosure are directed to determining a user intent based on gaze information acquired for the user and then using the intent, or gaze information, to control one or more functions associated with a computing device or system. For example, a user may be interacting with a computing system, that may utilize a facial recognition system and/or a face-based authentication system to authenticate the user, thereby allowing the user access to the computing system and/or one or more capabilities provided by the computing system. In examples, the computing system may include an image sensor having a field-of-view and capable of acquiring one or more images of the user, such that the one or more images of the user may be processed and used as a basis for authenticating the user with the computing system or another capability of the computing system. That is, the one or more images of the user may allow the user to login to the computing system, serve as an access credential, and/or prevent the mobile computing system from locking a display due, at least in part to, a lack of user interaction with the computing system.
• However, situations arise when one or more functions of the computing system may be triggered upon acquisition of the one or more images even though an intent of the user may be otherwise. For example, a user may be holding a computing system, such as a phone, and may not intend to unlock or otherwise interact with the phone. That is, one or more images may be acquired when the user may be within the field-of-view of the image sensor; the one or more images of the user may then be processed and a function, such as a device unlock function may occur. Such unlock function may occur even when the user is not interacting with or otherwise intending to engage with or otherwise use the computing system but the user happens to be within the field-of-view of the image sensor. In addition to unintentionally unlocking the computing system, an amount of power consumed by the computing system may increase when such function is triggered. For example, the display of the computing system displays content and a brightness of the display may be increased, and/or other computational resources may be expended. Thus, while the computing system may be readied for user interaction, the user may not desire to engage with the computing system thereby wasting computational resources such as energy.
• As another example, a user may be using a computing system, such as the computing system, and the computing system may invoke one or more functions based on a timed event in order to reduce an amount of power consumed by the device. For example, a user may be viewing content on a device, and other than viewing such content, may not interact with the computing system. Based on a lack of detected interaction, the device may dim the display thereby reducing an amount of power consumed by the computing system without regard to whether or not the user is viewing content or otherwise interacting with the device. In addition to potentially annoying the user, the dimmed display may serve as a distraction to the user, diverting the user's attention from an immediate task at hand and instead to focus on increasing the brightness of the display.
• In accordance with examples of the present disclosure, one or more functions of the computing system may be controlled by or otherwise depend on a user gaze. The user gaze generally occurs when a user looks steadily and intently at an object or in a particular direction. The user gaze may be detected by or otherwise determined by the computing system using the image sensor. That is, the image sensor may acquire one or more images and determine the user gaze based on the one or more images. The gaze may then be used to control one or more functions, such as causing a display to increase brightness, unlocking a computing system, etc.
• In accordance with examples of the present disclosure,FIG. 1A-1B depict an example of acquiring a user intent from a user's gaze and using such intent to control one or more functions and/or applications of the computing device. A user may be interacting with a computing system, such as but not limited to a mobile device, phone, or other computing system. In some examples, thecomputing system104 may utilize a facial recognition system and/or a face-based authentication system to authenticate a user, such as theuser102, thereby allowing the user access to thecomputing system104 and/or one or more capabilities provided by the computing system. In examples, thecomputing system104 may include an image sensor having a field-of-view and capable of acquiring one or more images of theuser102, where the user may be included in the field-of-view of theimage sensor112. Theimage sensor112 may be any image sensor capable of acquiring one or more images, where the one or more images may be an image derived from light of the infrared spectrum, the visible light spectrum, and/or or another light spectrum. The one or more images of theuser102 may be processed such that one or more features of the user, obtained from the one or more images, may be used as a basis for authenticating the user with thecomputing system104 or another capability of thecomputing system104. For example, the one or more images of theuser102 may be processed to identify the user and allow the user access to one or more applications or files stored on or otherwise accessible via thecomputing system104. As another example, the one or more images of theuser102 may allow the user to login to thecomputing system104, serve as an access credential, and/or prevent the mobile computing system from locking a display due, at least in part to, a lack of user interaction with thecomputing system104. In some examples, the one or more images of the user may be processed at thecomputing system104 and/or at a computing system or system accessible by thecomputing system104. As another example, the one or more images of the user may be processed at thecomputing system104 and one or more features of the user obtained from the one or more images may be provided to an access control system such that the access control system provides an indication to thecomputing system104 that theuser102 is allowed to use or otherwise interact with thecomputing system104.
• In accordance with examples of the present disclosure, theimage sensor112 may obtain one or more images of theuser102, where theuser102 is within a field-of-view of theimage sensor112. The one or more images acquired by theimage sensor112 may be provided to a neural network model executing at a neural processing unit. The neural network model may determine and provide gaze information of theuser102 to one or more applications executing at thecomputing system104 such than another function, for example a facial recognition system or other face-based authentication system, may be activated or otherwise enabled. Because the neural processing unit is specifically designed and/or programmed to process neural network tasks, the consumption of resources, such as power and/or computing cycles, is less than the consumption would be if a central processing unit were used. The gaze information determined and provided by the neural network model may include a gaze location of the user; the gaze location of the user may correspond to a location, such as an X, Y, Z coordinate, on and/or around thecomputing system104. As depicted inFIG. 1A, the gaze information may indicate that the user is staring intently at an object, in a direction, and/or at a location that does not involve thecomputing system104. For example, theuser gaze108 may be above thecomputing system104. Accordingly, a facial recognition system or other face-based authentication system may not be activated. Accordingly, as depicted inFIG. 1A, no user engagement is detected according to a detected gaze of the user and therefore no identification occurs.
• As depicted inFIG. 1B, thecomputing system104 may determine that theuser gaze114 of theuser102 is directed to a portion of thecomputing system104; based on this determination, the computing system may activate a facial recognition system or other face-based authentication system to unlock thecomputing system104. Accordingly, as depicted inFIG. 1B, a user engagement is detected according to the user gaze and therefore an identification and/or authentication process may occur. In examples, the image sensor, such as theimage sensor112, may acquire one or more images of theuser102 and determine gaze information for the user. The one or more images acquired by theimage sensor112 may be provided to a neural network model and executing at a neural processing unit.
• The neural network model may determine and provide the gaze information to one or more applications executing at thecomputing system104 such than another function, for example a facial recognition system or other face-based authentication system may be activated or otherwise enabled. The gaze information determined and provided by the neural network model may include a gaze location of the user; the gaze location of the user may correspond to a location, such as an x, y, z coordinate, on and/or around thecomputing system104. As an example, the gaze location of the user may correspond to a location on the display of thecomputing system104 associated with a login prompt. Accordingly, until the gaze location of the user matches the location of the login prompt or other displayed control for example, the facial recognition system or other face-based authentication system may not be invoked, enabled, and/or activated. As depicted inFIG. 1B, theuser gaze114 is directed to a display of the computing system; in examples, when the gaze location of the user matches the location of the login prompt or other displayed control, the facial recognition system or other face-based authentication system may be invoked, enabled, and/or activated.
• FIGS. 2A-2D depict another example of using a user gaze to control one or more functions of a computing system in accordance with examples of the present disclosure. In examples, the user gaze may control a brightness functionality of the computing system. For example, a maps application may be executed at the computing system. In some maps applications, when the maps application is providing directions, a computing system display may be maintained at a constant brightness such that the directions provided by the maps application are always immediately viewable by the user. However, maintaining the computing system display at a constant brightness may lead to excessive power utilization by the computing system and therefore cause a battery associated with the computing system to discharge. Accordingly, some maps application may dim the computing system display after a certain amount of time has passed. In examples, the dimming of the computing system display may be initiated by the maps application or may be initiated by an operating system of the computing system when a threshold amount of time has passed without user interaction. Accordingly, a user wishing to view directions provided by the maps application may need to tap the computing system or otherwise interact with the computing system in order to return the brightness of the computing system display to a level at which the user can view the directions. Requiring the user to interact with the computing system in order to view the directions for example, may cause the attention of the user to be directed off of or otherwise away from an immediate task of driving.
• In accordance with examples of the present disclosure, the user gaze may be utilized to control the brightness of thecomputing system display216A. As depicted inFIG. 2A, theimage sensor212 of thecomputing system204 may obtain one or more images of theuser202, where theuser202 is within a field-of-view of theimage sensor212. Thecomputing system204 may be the same as or similar to thecomputing system104 previously described; theimage sensor212 may be the same as or similar to theimage sensor112 previously described. The one or more images acquired by theimage sensor212 may be provided to a neural network model executing at a neural processing unit. The neural network model may determine and provide gaze information of theuser202 to one or more applications, such as the maps application, executing at thecomputing system204. In some examples, the neural network model may determine and provide gaze information of theuser202 to an operating system executing at thecomputing system204. The gaze information determined and provided by the neural network model may include a gaze location of the user; the gaze location of the user may correspond to a location, such as an x, y, z coordinate, on and/or around thecomputing system204. As depicted inFIG. 2A, the gaze information may indicate that the user is staring intently at an object, in a direction, and/or at a location that does not involve thecomputing system204. For example, theuser gaze208 may be directed to an area above thecomputing system204. Accordingly, the maps application and/or the operating system may dim thecomputing system display216A or otherwise allow the computing system display216A to dim in accordance with a timed event. Accordingly, as depicted inFIG. 2A, thecomputing system display216A is dim or may be turned off when theuser gaze208 is not directed to thecomputing system204 and/or thecomputing system display216A.
• As depicted inFIG. 2B, thecomputing system204 may determine that thegaze214 of theuser202 is directed to a portion of thecomputing system204; based on this determination, the computing system may cause thecomputing system display216B to turn on and/or brighten. That is, thecomputing system display216B may be brighter than thecomputing system display216A. Accordingly, as depicted inFIG. 2B, theimage sensor212 may acquire one or more images of theuser202 and determine gaze information for the user. The one or more images acquired by theimage sensor212 may be provided to a neural network model and executing at a neural processing unit.
• The neural network model may determine and provide gaze information of theuser202 to an operating system executing at thecomputing system204 and/or to a maps application. The gaze information determined and provided by the neural network model may include a gaze location of the user; the gaze location of the user may correspond to a location, such as an x, y, z coordinate, on thecomputing system204. As depicted inFIG. 2B, the gaze information may indicate that the user is staring intently at a portion of a display of thecomputing system204. For example, theuser gaze208 may be directed to the map and/or directions of the maps application. Accordingly, the maps application and/or the operating system may cause thecomputing system display216B to brighten based on the gaze information.
• In some examples, a predicted user gaze may be utilized to control one or more functions and/or capabilities of a computing system. For example, one or more images acquired by theimage sensor212 may be provided to a neural network model. In examples, the one or more images may include a field-of-view that captures anarea218. Accordingly, the neural network model may determine that the gaze information for a user, such asuser202, includes agaze location220A corresponding to a first point in time t₁; agaze location220B corresponding to a second point in time t₂; and agaze location220C corresponding to a third point in time t₃. The neural network model may generate a predictedgaze location220D based on the one or more of theprevious gaze locations220A,220B, and/or220C. Accordingly, thecomputing system204 may cause the computing system display216A to brighten based on the predictedgaze location220D such that thecomputing system display216B is in an “on” or otherwise bright state so that a user can view the content displayed to thecomputing system display216B without having to wait for the neural network model to determine a gaze location that is on thecomputing system display216A. As depicted inFIG. 2C, thegaze locations220A,220B, and/or220C may be directed to locations on and/or around thecomputing system204. As depicted inFIG. 2D, the predictedgaze location220D may be directed to a location on thecomputing system display216B causing the display to brighten. As previously mentioned, the application, such as the maps application, and/or an operating system of thecomputing system204 may cause thecomputing system display216B to turn on and/or brighten.
• FIGS. 3A-3B depict another example of relying on user gaze to control one or more functions of a computing system in accordance with examples of the present disclosure. In examples, auser302 may be working at acomputing system304 having one ormore image sensors306. Thecomputing system304 may be the same as or similar to thecomputing system104 previously described; theimage sensor306 may be the same as or similar to theimage sensor112 previously described. In examples, theuser gaze307 may be directed to one ormore locations308A-D or areas on thedisplay305. A subsequent function, such as performing a facial recognition process or other face-based authentication process, may be dependent upon determining that theuser302 intends for the subsequent function, such as the facial recognition process or other face-based authentication process, to be enabled or otherwise activated. For example, a determined gaze location corresponding tolocations308A,308B, and/or308C may indicate that the user does not intend for a subsequent function to be executed. Accordingly, when the gaze location correspond tolocations308A,308B, and/or308C, a subsequent function, such as performing a facial recognition process or other face-based authentication process, may not be enabled or otherwise activated. However, when the gaze location correspond to gaze locations within an area orregion312 for example, the user may intend for the subsequent function, such as performing a facial recognition process or other face-based authentication process to be activated or otherwise enabled. As depicted inFIG. 3B, thegaze locations308D and308E are within theregion312; thus a subsequent function, such as performing a facial recognition process or other face-based authentication process, may be enabled or otherwise activated.
• In examples, each of thegaze locations308 may be determined by a neural network model as previously described. Theregion312 may be application and/or operation system specific and/or may be defined by the user. While the example ofFIGS. 3A-3B are directed to performing a facial recognition process or other face-based authentication process, it should be understood that that functions other than a facial recognition process or other face-based authentication process, are contemplated. For example, when a gaze location corresponding to any location on thedisplay305 is detected, an inactivity timeout of the operating system or other application may be reset. As one example, detecting a user gaze atlocation308B may reset a timer associated with a screensaver and/or an auto-lock feature of the operating system executing on thecomputing system304. Thus, for example, if thecomputing system304 is not moving and/or a user is not physically engaging with thecomputing system304, but a user is looking at thecomputing system304, thecomputing system304 will not lock the display. As another example, a display refresh rate, resolution, font size, and/or font smoothing settings may be controlled based on a gaze location and/or a gaze detection.
• FIGS. 4A-4C depict another example of relying on user gaze to control one or more functions of a computing system in accordance with examples of the present disclosure. In examples, auser402 may be working at acomputing system405 associated with adisplay404, where thecomputing system405 may be in communication with one ormore image sensors406. Thecomputing system405 may be the same as or similar to thecomputing system104 previously described; theimage sensor406 may be the same as or similar to theimage sensor112 previously described. In examples, theuser gaze407 may be directed to one ormore locations408A-408C or areas on thedisplay404. Thecomputing system405 may track theuser gaze407 of the user and associate theuser gaze407 with an identity or unique identifier specific to theuser402. In examples, thecomputing system405 may determine that theuser gaze407 strays from thedisplay404 for a period of time, for example theuser gaze407 of theuser402 is determined to be atlocation408G for a period of time and then vanishes. Thecomputing system405 may cause the display to lock and/or dim when the period of time exceeding a threshold. Accordingly, although asecond user410 may be viewing thedisplay404, because theuser402 is determined to be absent, thecomputing system405 may cause the display to lock such that the potentially unauthorized information is not obtained and/or unauthorized use of one or more applications does not occur.
• As another example, theuser402 may be working at thecomputing system405 associated with thedisplay404. A document or other content may be presented in aregion418 of thedisplay404. Auser gaze407 associated with theuser402 may be determined by a neural network model and associated with the user and/or a unique identifier as previously discussed. In examples, asecond user410 may attempt to view content, such as the content in theregion418, while theuser402 is viewing content. Accordingly, one or more images acquired by theimage sensor406 may be provided to a neural network model where the one or more images include portions ofuser402 andsecond user410. The neural network model may generate two types of gaze locations. A first type of gaze location may be agaze location408A-408D associated with theuser402; a second type of gaze location may be agaze location412 associated with thesecond user410. In some examples, based on the determination that two or more gazes have been detected, thecomputing system405 may cause thedisplay404 to lock or otherwise not display content.
• In some examples, thecomputing system405 may attempt to identify thesecond user410 and perform an operation based on the identity of thesecond user410. For example, thecomputing system405 may initiate a facial recognition process or other face-based recognition system based on an indication from the neural network model that two or more gazes have been detected. In some examples, rather than lock thedisplay404 or otherwise hide content, thecomputing system405 may determine whether thesecond user410, or an account and/or identity associated with thesecond user410, possesses rights to view content currently displayed, such as the content in theregion418. Upon determining that both theuser402 and thesecond user410 are authorized or otherwise possess the appropriate privileges to view the content at thedisplay404 such as the content in theregion418, thecomputing system405 may take no action regarding blocking, locking, or otherwise will not obstruct the content from being viewed by thesecond user410. In some examples, the computing system may determine that both theuser402 and thesecond user410 are authorized or otherwise possess the appropriate privileges to view the content at thedisplay404 when thegaze location412 is directed to the displayed content or otherwise in aregion418. Accordingly, thecomputing system405 may not perform the facial recognition and/or face-based authentication processes if a gaze location is atgaze location412A and/or412B. However, should the gaze location be on top of or otherwise at a specified region, such as theregion418, thecomputing system405 may perform the facial recognition and/or face-based authentication processes to verify thesecond user410 is authorized to view the displayed content.
• In some examples, rather than lock or block all content displayed on adisplay404 if thesecond user410 is determined to not have the necessary access rights and/or privileges to view the displayed content in theregion418, only those files, documents, or content which thesecond user410 is not authorized to view may be blocked or otherwise obstructed from view. As depicted inFIG. 4C, a obstructingblock422 may be overlaid or otherwise cover content in theregion418 thereby preventing thesecond user410 from viewing the content in theregion418. In accordance with some examples, a subset of micro-pixels that correspond to an authenticated user's gaze may be illuminated; similarly, display backlighting, LEDs, and panels may be controlled based on an authenticated user's gaze.
• FIGS. 5A-5D depict signal diagrams associated with controlling one or more functions and/or applications of a computing system using eye gaze in accordance with examples of the present disclosure. As depicted inFIG. 5A, an intent of a user at a computing system may be based on a corresponding eye gaze location of a user. In examples, where an eye gaze location associated with a user is not at a display device or otherwise predicted to be at a display device, a user gaze may not be detected and therefore the intent of the user to interact with the display device may be expressed as a logic low as depicted in the signal diagram504. Accordingly, a subsequent function, process, or application may not be enabled or otherwise invoked. Where an eye gaze location associated with a user is detected or otherwise determined to be at a display device at a first point in time t₁or otherwise predicted to be at a display device at the first point in time t₁, the intent of the user to interact with the display device may be a logic high as depicted in the signal diagram508. Accordingly, a subsequent function, process, or application may be enabled or otherwise invoked at a subsequent time t₂. In examples, the application control in the signal diagram508 may correspond to increasing a brightness of the display, initiating a facial recognition process or other face-based authentication process, unlocking a computing system, granting access to a file, document, or otherwise, taking a picture, initiating or launching an application, etc. Where an eye gaze location associated with a user is at a display device at a first point in time t₁or otherwise predicted to be at a display device at the first point in time t₁, and then is absent at a second point in time t₃, the intent of the user to interact with the display device may be a logic low as depicted in the signal diagram508. Accordingly, a subsequent function, process, or application may be disabled or otherwise prevented from executing at a subsequent time t₄.
• As depicted inFIG. 5C, a subsequent function, process, or application may be enabled or otherwise invoked at a subsequent time t₄. For example, an eye gaze location associated with a user may be detected or otherwise determined to be at a display device at a first point in time t₁or otherwise predicted to be at a display device at the first point in time t₁; the intent of the user to interact with the display device may be a logic high as depicted in the signal diagram512. Accordingly, a subsequent function, process, application, or state may be enabled or otherwise invoked at a subsequent time t₂. In examples, the application control in the signal diagram512 may correspond to maintaining a login state of a user; maintaining a display enable state of a user, displaying content of a window, initiating a facial recognition process or other face-based authentication process, unlocking a computing system, granting access to a file, document, or otherwise, taking a picture, initiating or launching an application, etc. At a time t₃in response to a gaze being detected of both first user and a second user, the enabled function, process, application, or state at t₃may be disabled at t₄. For example, at a time t₄, corresponding to a dual gaze detection signal at a logic high and an action/state signal at a logic low, a display of computing device may be locked and/or content displayed at a computing device display may be obstructed with a graphically generated block for example.
• In instances where the dual gaze detection is no longer detected, such as at t₅, the subsequent function, process, application, or state may be disabled or otherwise invoked at a subsequent time t₆. In examples, the application control in the signal diagram512 again may correspond to maintaining a login state of a user; maintaining a display enable state of a user, displaying content of a window, initiating a facial recognition process or other face-based authentication process, unlocking a computing system, granting access to a file, document, or otherwise, taking a picture, initiating or launching an application, etc. Once the intent of the user is detected to be absent, for example at instances where the user leaves the field-of-view of an imaging device associated with the computing system, such as at time t₆, the enabled function, process, application, or state at t₆may be disabled at t₅.
• As depicted inFIG. 5D, a subsequent function, process, or application may be enabled or otherwise invoked based on an eye gaze location of a user indicating intent and another signal, such as Signal₁. For example, an eye gaze location associated with a user may be detected or otherwise determined to be at a display device at a first point in time t₁or otherwise predicted to be at a display device at the first point in time t₁; the intent of the user to interact with the display device may be a logic high as depicted in the signal diagram516. Accordingly, a signal, such as Signal₁corresponding to another function, process, application, or state may be received at time t₂. In examples, receiving eye gaze information, such as intent, may cause a computing system to determine whether a signal, such as Signal₁from another function, process, application, or state is received. For example, based on the intent signal of516, a computing system may poll another state or signal, such as Signal₁. If a signal associated with the other function is received and is at a logic high, for example, at time t₂, a subsequent function, process, application, or state may be may be enabled or otherwise invoked at a subsequent time t₃. In examples, the application control in the signal diagram516 may correspond to maintaining a login state of a user; maintaining a display enable state of a user, displaying content of a window, initiating a facial recognition process or other face-based authentication process, unlocking a computing system, granting access to a file, document, taking a picture, initiating or launching an application, etc, where the application control may be based on both an intent of a user based on eye gaze information and another signal, such as Signal₁. In some examples, the Signal₁may correspond to non-gaze information, such as an image of a user used for identification and/or authentication purposes.
• In examples, the application control signal may be associated with one or more applications or functions. For example, based on the gaze information (e.g., intent) and another signal, such as Signal₁, a function and/or application may be executed, enabled, disabled, or otherwise invoked. As one non-limiting example, a brightness of a computing system display may be increased in response to the gaze information (or predicted gaze information) (e.g., intent), and where Signal₁corresponds to movement of a computing system, such as a smartphone. Accordingly, a brightness of an app, such as a maps application, may increase. In some examples, a face-based authentication or other facial recognition system may be invoked to log a user into a computing system. Based on a predicted gaze location based on the gaze information, a maps application may be enabled and/or a brightness associated with a display providing the maps application may be increased when the Signal₁indicates a computing system is moving (e.g., in an automobile, moved by a user, etc.). Alternatively, or in addition, an interactivity timer, such as a countdown or watchdog timer associated with a computing system display may be reset such that a computing system is not locked or the computing system display is not turned off—where the resetting of the interactivity timer is based on the intent of the user and Signal₁, such as a computing system not moving.
• As another example, a motion signal Signal₁indicating that a computing system has not been moved or otherwise is being held in a “still” state together with gaze information of the user may indicate to the computing system that a photograph is to be acquired and added to a “quick shot” gallery to solve the “missed moment” situation where a user does not have enough time to frame the subject of a photograph due to a quickly changing environment. As another example, the computing system may classify a user's activity as a type of activity, such as but not limited to “running”, “driving”, etc. and output a detected activity as Signal₁. Thus, together with the detected signal, a user's gaze may be used to invoke an emergency call button rendered to the computing device display. Alternatively, or in addition, atypical device motions, such as motions passing between people may be detected; for example, the computing system may take photos of users preemptively and may provide assistance in use cases such as pick pocketing, muggings, accidents or other atypical situations. Thus, when a signal indicates the computing system is in motion, gaze information corresponding to a person to whom the computing system does not belong may be obtained.
• In some examples, when an eye gaze location associated with a user is no longer received or otherwise determined to be at a location other than a computing system display, such as at a time t₄, the application control may enter a logic low state at time t₆, even though the Signal₁may be at a logic high (e.g., at t₅). Accordingly, the application associated with the application control signal may be disabled for example, or enter another state.
• FIG. 6 depicts additional details of acomputing system602. Thecomputing system602 may be the same as or similar to thecomputing system104 previously described. Thecomputing system602 may include a central processing unit (CPU)604, a neural processing unit (NPU)608, agaze detector612 and a function/action/state controller636. In examples, thecomputing system602 may include anapplication interface624 and/or optionally anidentity management system628. In addition, thecomputing system602 may include or otherwise be communication with animage sensor620 and optionally animage sensor632. Thegaze detector612 may include agaze estimator616.
• In examples, thecomputing system602 may receive one or more images from theimage sensor620. In some examples, thegaze detector612 may execute processing at theCPU604 and/or theNPU608. For example, processing of thegaze estimator616 may occur at theNPU608. TheNPU608, being configured to efficiently execute processing associated with neural network models, such as thegaze estimator616, may allow thegaze detector612 to operate in or near real-time such that a gaze of a user may be detected in or near real-time without consuming resources traditionally expended by theCPU604. Accordingly, thegaze detector612 may receive the one ormore images640 from theimage sensor620. Thegaze estimator616 may take the received one ormore images640, and extract one or more features from theimage640 using thefeature extractor644. For example thefeature extractor644 may determine and/or detect a user's face and extract feature information such as, but not limited to, a location of a user's, eyes, pupils, nose, chin, ears etc. In examples, the extracted information may be provided to aneural network model648, where the neural network model may provide gaze information as an output. In examples, the neural network model may include but is not limited to a transformer model, a convolutional neural network model, and/or a support vector machine model. The gaze information output from the neural network model may be provided to anapplication interface624 where an application associated with theapplication interface624 may utilize the gaze information to determine a function to execute, and/or enabled, disable, or otherwise invoke a function. For example, the gaze information may include coordinates, (e.g., x,y,z coordinates) of a user's gaze in relation to an origin point on a display associated with a computing device. The application associated with theapplication interface624 may then determine a function or application to invoke, enable, or execute, and then invoke, enable, or execute the function.
• As another example, based on the gaze information, a maps application may be enabled and/or a brightness associated with a display providing the maps application may be increased. In some examples, the gaze information may include a signal indicating a user's intent. In examples the gaze information may be provided to a function/action/state controller636 which may determine a function or application to invoke, enable, or execute, and then invoke, enable, or execute the function. For example, based on the gaze information, a maps application may be enabled and/or a brightness associated with a display providing the maps application may be increased.
• In examples, the determination of gaze information may occur over one or more images. For example, a user's gaze may be detected and then confirmed when a gaze of the user determined for a satisfactory number of images occurs. For example, to account for natural user eye movement, a gaze of a user may be determined over multiple images, where a confirmation of a user's gaze occurs when the determined gaze for each image of the multiple images, or a subset of images of the multiple images, is directed to approximately the same location. That is, a user gaze may be based on a gaze in each image, or a subset of images, acquired during the span of 1.8 seconds for example, that is directed to the same location. Of course other time spans and/or number of images are contemplated. In addition, the number images and/or the time span necessary to determine a gaze of a user may be dependent on a function and/or application that is invoked, enabled, or otherwise executed. For example, launching an application based on a user's gaze may require a user to gaze at an icon for 1.8 seconds while preventing a display from going dim may require a user to gaze at the display for 0.9 seconds.
• In some examples, anidentity management system628 may be invoked based on the gaze information. For example, animage652 may be provided by theimage sensor632; the image may be the same as or different from theimage640. In examples, theimage sensor620 and theimage sensor632 may be the same image sensors, or may be different. Theimage652 may be provided to afacial recognition process656 which may perform one or more facial recognition routines to determine whether a user associated with theimage652 should be granted access to a specific application or function. As an example, theidentity management system628 may be utilized to authenticate a user or otherwise log a user into a computing system utilizing facial recognition. Accordingly, theidentity management system628 may communicate with anaccess control store664; the access control store may include access credentials for a user. In some examples, theaccess control store664 may include information indicating what document a user has access credentials to view or the like.
• FIG. 7 depictsexample data structures704 and716 in accordance with examples of the present disclosure. Thedata structure704 may associate agaze706 of a user with agaze location708, identity of theuser710, a time at which point the gaze was detected712, and a corresponding function/action/state control714. For example, a first gaze at time T₁may correspond to a gaze location X₁,Y₁and initiate a function, action, or control having an identifier of FB0723 which may correspond to increasing the brightness of an application, such as a maps application, displayed at a computing device display. As another example, at time T2, another gaze associated with a different user may be detected, which may initiate a function, action, or control having an identifier of 00EF23 corresponding to locking a display of a device.
• Thedata structure716 is an example data structure associating content or anapplication718 to anidentity720 of a user and whether a user has access to view such content. For example, a content identifier CDA₁may refer to a first application or view of content displayed to a computing system display. In examples, a user associated with an identity EFA034 may possess access credentials to view such content such that the content is displayed at a computing system display. In examples, a user associated with an ID EGA034 may not have access credentials to view such content; accordingly, if a gaze associated with a user identity of EGA034 is detected, the content CDA₁will not be displayed to a computing system display. For example, a computing system may enter a locked state such that a login screen is depicted at the computing system display. In another example, content CDA₁is obscured or otherwise blocked from being viewable—for example, content is blurred, covered with other content or a graphic, and/or an application window displaying such content is minimized or closed.
• FIG. 8 depicts details of amethod800 for controlling a function and/or application using a detected gaze of a user in accordance with examples of the present disclosure. A general order for the steps of themethod800 is shown inFIG. 8. Generally, themethod800 starts at802 and ends at818. Themethod800 may include more or fewer steps or may arrange the order of the steps differently than those shown inFIG. 8. Themethod800 can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. In examples, aspects of themethod800 are performed by one or more processing devices, such as a computer or server. Further, themethod800 can be performed by gates or circuits associated with a processor, Application Specific Integrated Circuit (ASIC), a field programmable gate array (FPGA), a system on chip (SOC), a neural processing unit, or other hardware device. Hereinafter, themethod800 shall be explained with reference to the systems, components, modules, software, data structures, user interfaces, etc. described in conjunction withFIGS. 1-7.
• The method starts at802, where flow may proceed to804. At804, an image is received from an image sensor. The image sensor may be the same as or similar to theimage sensor112 previously described. The method may proceed to806 and gaze information may be determined. In examples, the gaze information may include an enable/disable signal or may include a location of a user's gaze. For example, the gaze information may include a coordination location (e.g., X,Y,Z) in relation to an origin point on a display associated with a computing device indicating a location of a user's gaze or a predicted location of a user's gaze. In examples, a gaze estimator, such as agaze estimator616, may receive the image and extract one or more features from the image using a feature extractor, such as thefeature extractor644. The feature extractor may extract feature information such as, but not limited to, a location of a user's, eyes, pupils, nose, chin, ears etc. In examples, the extracted information may be provided to a neural network model, such as theneural network model648, where the neural network model may generate the gaze information.
• Themethod800 may proceed to808 where the gaze information may be provided to a gaze dependent function and/or application. For example, the gaze information may be provided to an application interface, such as theapplication interface624. An application associated with the application interface may utilize the gaze information to determine a function and/or application to execute, enable, disable, or otherwise invoke. In some examples, the gaze information may be provided directly to a function or application. Alternatively, or in addition, the gaze information may be provided to the application interface such that the application interface may determine the function and/or application to execute, enable, disable, or otherwise invoke.
• Themethod800 may proceed to810, where a determination may be made based on whether the gaze information satisfies a condition. In examples, the gaze information may satisfy a condition if the gaze information corresponds to a signal going from an logic low to a logic high and/or vice versa. In some examples, the gaze information may correspond to a location associated with or otherwise directed to a location on a computing system display; if the gaze location coincides with another location, such as a login prompt, a location on a displayed app, a location on the computing system display, then themethod800 may proceed to812. In some examples, the gaze information may be generated for a predicted gaze location associated with or otherwise directed to a location on a computing system display; if the predicted gaze location coincides with another location, such as a login prompt, a location on a displayed app, a location on the computing system display, then themethod800 may proceed to812. In some examples, where the gaze information does not satisfy a condition, the flow of the method may return to804.
• At812, the method may enable and/or control a function or application. For example, based on the gaze information satisfying a condition, the determined function and/or application to execute, enable, disable, or otherwise invoke may be executed, enabled, disabled, or otherwise invoked. As one non-limiting example, a brightness of a computing system display may be increased in response to the gaze information (or predicted gaze information). Accordingly, a brightness of an app, such as a maps application, may increase. In some examples, a face-based authentication or other facial recognition system may be invoked to log a user into a computing system. At810, based on the gaze information and/or the predicted gaze information, a maps application may be enabled and/or a brightness associated with a display providing the maps application may be increased. In some examples, the gaze information may include a signal indicating a user's intent. In examples the gaze information may be provided to a function/action/state controller636 which may determine a function or application to invoke, enable, or execute, and then invoke, enable, or execute the function based on the signal and/or intent. For example, an interactivity timer, such as a countdown or watchdog timer associated with a computing system display may be reset such that a computing system is not locked or the computing system display is not turned off. As another example, based on the gaze information, a computing system may invoke a photograph capture process to take a photograph as intended by the a user, rather than requiring the user to navigate or more graphical user interfaces to take a photograph.
• The method may optionally proceed to814, where one or more additional signals may be received such that a second function or application may be executed at816. For example, a motion signal indicating that a computing system has not been moved or otherwise is being held in a “still” state together with gaze information of the user may indicate to the computing system that a photograph is to be acquired and added to a “quick shot” gallery to solve the “missed moment” situation where a user does not have enough time to frame the subject of a photograph due to a quickly changing environment. As another example, the computing system may classify a user's activity as a type of activity, such as but not limited to “running”, “driving”, etc. In some examples, a user's gaze may be used to invoke an emergency call button rendered to the computing device display based on the detected activity. Alternatively, or in addition, atypical device motions, such as motions of the computing device passing between people, may be used to trigger the taking of photos of users preemptively to provide assistance in use cases such as pick pocketing, muggings, accidents or other atypical situations. For example, when a signal indicates the computing system is in motion, the gaze detector may obtain and store gaze information corresponding to a person to whom the computing system does not belong. Themethod800 may end at818.
• FIG. 9 depicts details of amethod900 for controlling a function and/or application using a detected gaze of an identified user in accordance with examples of the present disclosure. A general order for the steps of themethod900 is shown inFIG. 9. Generally, themethod900 starts at902 and ends at914 Themethod900 may include more or fewer steps or may arrange the order of the steps differently than those shown inFIG. 9. Themethod900 can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. In examples, aspects of themethod900 are performed by one or more processing devices, such as a computer or server. Further, themethod900 can be performed by gates or circuits associated with a processor, Application Specific Integrated Circuit (ASIC), a field programmable gate array (FPGA), a system on chip (SOC), a neural processing unit, or other hardware device. Hereinafter, themethod900 shall be explained with reference to the systems, components, modules, software, data structures, user interfaces, etc. described in conjunction withFIGS. 1-8.
• The method starts at902, where flow may proceed to904. At904, an image is received from an image sensor. The image sensor may be the same as or similar to theimage sensor112 previously described. The method may proceed to906 and gaze information may be determined. In examples, the gaze information may include an enable/disable signal or may include a location of a user's gaze. For example, the gaze information may include a coordination location (e.g., X,Y,Z) in relation to an origin point on a display associated with a computing device indicating a location of a user's gaze or a predicted location of a user's gaze. In examples, a gaze estimator, such as agaze estimator616, may receive the image and extract one or more features from the image using a feature extractor, such as thefeature extractor644. The feature extractor may extract feature information such as, but not limited to, a location of a user's, eyes, pupils, nose, chin, ears etc. In examples, the extracted information may be provided to a neural network model, such as theneural network model648, where the neural network model may generate the gaze information. In addition, an identity may be assigned to the generated gaze information.
• For example, an identity management system may be invoked based on the gaze information such that a facial recognition process may perform one or more facial recognition routines to determine whether a user associated with the gaze information should be granted access to a specific application or function. That is, an identity of a user may be assigned to gaze information. The method may proceed to908 where a determination may be made as to whether the gaze information and/or the identity associated with the gaze satisfies a condition. Thus, in instances where the gaze information and/or the identity satisfy a condition, the method may proceed to910; alternatively, the method may proceed to912. As an example, gaze location of a user may be directed to content that is access controlled but is displayed at a computing system display; accordingly, if the identity of the user is associated with the proper access credentials, the method may proceed to910 for example, where the access controlled content may be displayed or may continue to be displayed. However, if the identity of the user is not associated with the proper access credentials, the method may proceed to912 where a second function, such as the display of the access controlled content may be disabled (e.g., the computing system may enter a locked state such that information is not displayed).
• As another example, themethod900 may operate according to a timed event, such that the gaze information associated with a user is checked to ensure temporal consistency. That is, instances where a first user is using a computing system and leaves the computing system unlocked and unattended, themethod900 may be executed and determine that new gaze information is now associate with a different user. Accordingly, at910 and/or912, the display of content may be blocked, restricted, and/or in some instances enabled based on the identity of the user.
• FIG. 10 depicts details of amethod1000 for controlling a function and/or application using a detected gaze of a user in accordance with examples of the present disclosure. A general order for the steps of themethod1000 is shown inFIG. 10. Generally, themethod1000 starts at1002 and ends at1014. Themethod1000 may include more or fewer steps or may arrange the order of the steps differently than those shown inFIG. 10. Themethod1000 can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. In examples, aspects of themethod1000 are performed by one or more processing devices, such as a computer or server. Further, themethod1000 can be performed by gates or circuits associated with a processor, Application Specific Integrated Circuit (ASIC), a field programmable gate array (FPGA), a system on chip (SOC), a neural processing unit, or other hardware device. Hereinafter, themethod1000 shall be explained with reference to the systems, components, modules, software, data structures, user interfaces, etc. described in conjunction withFIGS. 1-9.
• The method starts at1002, where flow may proceed to1004. At1004, an image is received from an image sensor. The image sensor may be the same as or similar to theimage sensor112 previously described. The method may proceed to1006 and gaze information may be determined. In examples, the gaze information may include an enable/disable signal or may include a location of a user's gaze. For example, the gaze information may include a coordination location (e.g., X,Y,Z) in relation to an origin point on a display associated with a computing device indicating a location of a user's gaze or a predicted location of a user's gaze. In examples, a gaze estimator, such as agaze estimator616, may receive the image and extract one or more features from the image using a feature extractor, such as thefeature extractor644. The feature extractor may extract feature information such as, but not limited to, a location of a user's, eyes, pupils, nose, chin, ears etc. In examples, the extracted information may be provided to a neural network model, such as theneural network model648, where the neural network model may generate the gaze information.
• Themethod1000 may proceed to1008 where the gaze information may be provided to a gaze dependent function and/or application. For example, the gaze information may be provided to an application interface, such as theapplication interface624. An application associated with the application interface may utilize the gaze information to determine a function and/or application to execute, enable, disable, or otherwise invoke. In some examples, the gaze information may be provided directly to a function or application. Alternatively, or in addition, the gaze information may be provided to the application interface such that the application interface may determine the function and/or application to execute, enable, disable, or otherwise invoke.
• Themethod1000 may proceed to1010, where a determination may be made based on whether the gaze information satisfies a condition. In examples, the gaze information may satisfy a condition if the gaze information corresponds to a signal going from an logic high to a logic low and/or vice versa. In some examples, the gaze information may correspond to a location associated with or otherwise directed to a location on a computing system display or at a location other than the computing system display. If the gaze location coincides with another location that is not the computing system display, then themethod1000 may proceed to1012. In some examples, the gaze information may be generated for a predicted gaze location associated with or otherwise directed to a location on a computing system display; if the predicted gaze location coincides with another location that is not directed to the computing system display, then themethod1000 may proceed to1012. In some examples, where the gaze information does satisfy a condition, the flow of the method may return to1004.
• At1012, the method may disable and/or prevent the execution of a function or application. For example, based on the gaze information not satisfying a condition (e.g., a user is looking away from the computing system display), a function and/or application may be disabled, or otherwise prevented from executing. As one non-limiting example, a brightness of a computing system display may be decreased in response to the gaze information (or predicted gaze information) not coinciding with the computing system display. Accordingly, a brightness of an app, such as a maps application, may be decreased. In some examples, a face-based authentication or other facial recognition system may be invoked to record or log a user to whom the gaze belongs. Themethod1000 may end at1018.
• FIGS. 11-13 and the associated descriptions provide a discussion of a variety of operating environments in which aspects of the disclosure may be practiced. However, the devices and systems illustrated and discussed with respect toFIGS. 11-13 are for purposes of example and illustration and are not limiting of a vast number of computing device configurations that may be utilized for practicing aspects of the disclosure, described herein.
• FIG. 11 is a block diagram illustrating physical components (e.g., hardware) of a computing device1100 with which aspects of the disclosure may be practiced. The computing device components described below may be suitable for the computing and/or processing devices described above. In a basic configuration, the computing device1100 may include at least oneprocessing unit1102 and asystem memory1104. Depending on the configuration and type of computing device, thesystem memory1104 may comprise, but is not limited to, volatile storage (e.g., random-access memory (RAM)), non-volatile storage (e.g., read-only memory (ROM)), flash memory, or any combination of such memories.
• Thesystem memory1104 may include anoperating system1105 and one ormore program modules1106 suitable for runningsoftware application1120, such as one or more components supported by the systems described herein. As examples,system memory1104 may include thegaze detector1121, theapplication interface1122, theidentity management system1123, and/or the function/action/state controller1124. Thegaze detector1121 may be the same as or similar to thegaze detector612 as previously described; theapplication interface1122 may be the same as or similar to theapplication interface624 as previously described; theidentity management system1123 may be the same as or similar to theidentity management system628 as previously described; and the function/action/state controller1124 may be the same as or similar to the function/action/state controller636 as previously described. Theoperating system1105, for example, may be suitable for controlling the operation of the computing device1100.
• Furthermore, embodiments of the disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program and is not limited to any particular application or system. This basic configuration is illustrated inFIG. 11 by those components within a dashedline1108. The computing device1100 may have additional features or functionality. For example, the computing device1100 may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated inFIG. 11 by aremovable storage device1109 and anon-removable storage device1110.
• As stated above, a number of program modules and data files may be stored in thesystem memory1104. While executing on theprocessing unit1102, the program modules1106 (e.g., software applications1120) may perform processes including, but not limited to, the aspects, as described herein. Other program modules that may be used in accordance with aspects of the present disclosure may include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided programs, etc.
• Furthermore, embodiments of the disclosure may be practiced in an electrical circuit discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic elements or microprocessors. For example, embodiments of the disclosure may be practiced via a system-on-a-chip (SOC) where each or many of the components illustrated inFIG. 11 may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communications units, system virtualization units and various application functionality, all of which are integrated (or “burned”) onto the chip substrate as a single integrated circuit. When operating via an SOC, the functionality, described herein, with respect to the capability of client to switch protocols may be operated via application-specific logic integrated with other components of the computing device1100 on the single integrated circuit (chip). Embodiments of the disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluidic, and quantum technologies. In addition, embodiments of the disclosure may be practiced within a general-purpose computer or in any other circuits or systems.
• The computing device1100 may also have one or more input device(s)1112 such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or swipe input device, etc. The one ormore input device1112 may include an image sensor, such as theimage sensor620 and/or theimage sensor632. The output device(s)1114 such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are examples and others may be used. The computing device1100 may include one ormore communication connections1116 allowing communications withother computing devices1150. Examples ofsuitable communication connections1116 include, but are not limited to, radio frequency (RF) transmitter, receiver, and/or transceiver circuitry; universal serial bus (USB), parallel, and/or serial ports.
• The term computer readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable, and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, or program modules. Thesystem memory1104, theremovable storage device1109, and thenon-removable storage device1110 are all computer storage media examples (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture which can be used to store information and which can be accessed by the computing device1100. Any such computer storage media may be part of the computing device1100. Computer storage media does not include a carrier wave or other propagated or modulated data signal.
• Communication media may be embodied by computer readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.
• FIGS. 12A-12B illustrate amobile computing device1200, for example, a mobile telephone, a smart phone, wearable computer (such as a smart watch), a tablet computer, a laptop computer, and the like, with which embodiments of the disclosure may be practiced. In some respects, the client may be a mobile computing device. With reference toFIG. 12A, one aspect of amobile computing device1200 for implementing the aspects is illustrated. In a basic configuration, themobile computing device1200 is a handheld computer having both input elements and output elements. Themobile computing device1200 typically includes adisplay1205 and one ormore input buttons1210 that allow the user to enter information into themobile computing device1200. Thedisplay1205 of themobile computing device1200 may also function as an input device (e.g., a touch screen display).
• If included, an optionalside input element1215 allows further user input. Theside input element1215 may be a rotary switch, a button, or any other type of manual input element. In alternative aspects,mobile computing device1200 may incorporate greater or fewer input elements. For example, thedisplay1205 may not be a touch screen in some embodiments.
• In yet another alternative embodiment, themobile computing device1200 is a portable phone system, such as a cellular phone. Themobile computing device1200 may also include anoptional keypad1235.Optional keypad1235 may be a physical keypad or a “soft” keypad generated on the touch screen display.
• In various embodiments, the output elements include thedisplay1205 for showing a graphical user interface (GUI), a visual indicator1220 (e.g., a light emitting diode), and/or an audio transducer1225 (e.g., a speaker). In some aspects, themobile computing device1200 incorporates a vibration transducer for providing the user with tactile feedback. In yet another aspect, themobile computing device1200 incorporates input and/or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., a HDMI port) for sending signals to or receiving signals from an external device.
• FIG. 12B is a block diagram illustrating the architecture of one aspect of a mobile computing device. That is, themobile computing device1200 can incorporate a system (e.g., an architecture)1202 to implement some aspects. In one embodiment, thesystem1202 is implemented as a “smart phone” capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, and media clients/players). In some aspects, thesystem1202 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.
• One ormore application programs1266 may be loaded into thememory1262 and run on or in association with theoperating system1264. Examples of the application programs include phone dialer programs, e-mail programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, maps programs, and so forth. Thesystem1202 also includes anon-volatile storage area1268 within thememory1262. Thenon-volatile storage area1268 may be used to store persistent information that should not be lost if thesystem1202 is powered down. Theapplication programs1266 may use and store information in thenon-volatile storage area1268, such as e-mail or other messages used by an e-mail application, and the like. A synchronization application (not shown) also resides on thesystem1202 and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in thenon-volatile storage area1268 synchronized with corresponding information stored at the host computer. As should be appreciated, other applications may be loaded into thememory1262 and run on themobile computing device1200 described herein (e.g., search engine, extractor module, relevancy ranking module, answer scoring module, etc.).
• Thesystem1202 has apower supply1270, which may be implemented as one or more batteries. Thepower supply1270 might further include an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the batteries.
• Thesystem1202 may also include aradio interface layer1272 that performs the function of transmitting and receiving radio frequency communications. Theradio interface layer1272 facilitates wireless connectivity between thesystem1202 and the “outside world,” via a communications carrier or service provider. Transmissions to and from theradio interface layer1272 are conducted under control of theoperating system1264. In other words, communications received by theradio interface layer1272 may be disseminated to theapplication programs1266 via theoperating system1264, and vice versa.
• Thevisual indicator1220 may be used to provide visual notifications, and/or anaudio interface1274 may be used for producing audible notifications via theaudio transducer1225. In the illustrated embodiment, thevisual indicator1220 is a light emitting diode (LED) and theaudio transducer1225 is a speaker. These devices may be directly coupled to thepower supply1270 so that when activated, they remain on for a duration dictated by the notification mechanism even though theprocessor1260 and other components might shut down for conserving battery power. The LED may be programmed to remain on indefinitely until the user takes action to indicate the powered-on status of the device. Theaudio interface1274 is used to provide audible signals to and receive audible signals from the user. For example, in addition to being coupled to theaudio transducer1225, theaudio interface1274 may also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation. In accordance with embodiments of the present disclosure, the microphone may also serve as an audio sensor to facilitate control of notifications, as will be described below. Thesystem1202 may further include avideo interface1276 that enables an operation of an on-board camera1230 to record still images, video stream, and the like. The onboard camera may be the same as or similar to the previously describedimage sensor620 and/or632.
• Amobile computing device1200 implementing thesystem1202 may have additional features or functionality. For example, themobile computing device1200 may also include additional data storage devices (removable and/or non-removable) such as, magnetic disks, optical disks, or tape. Such additional storage is illustrated inFIG. 12B by the non-volatile storage area3068.
• Data/information generated or captured by themobile computing device1200 and stored via thesystem1202 may be stored locally on themobile computing device1200, as described above, or the data may be stored on any number of storage media that may be accessed by the device via theradio interface layer1272 or via a wired connection between themobile computing device1200 and a separate computing device associated with themobile computing device1200, for example, a server computer in a distributed computing network, such as the Internet. As should be appreciated such data/information may be accessed via themobile computing device1200 via theradio interface layer1272 or via a distributed computing network. Similarly, such data/information may be readily transferred between computing devices for storage and use according to well-known data/information transfer and storage means, including electronic mail and collaborative data/information sharing systems.
• FIG. 13 illustrates one aspect of the architecture of a system for processing data received at a computing system from a remote source, such as apersonal computer1304,tablet computing device1306, ormobile computing device1308, as described above. Thepersonal computer1304,tablet computing device1306, ormobile computing device1308 may include thegaze detector1320 as previously described. Content at aserver device1302 may be stored in different communication channels or other storage types. For example, various documents may be stored using adirectory service1322, aweb portal1324, amailbox service1331, aninstant messaging store1328, orsocial networking services1330.
• One or more of the previously describedprogram modules1106 orsoftware applications1120 may be employed by server device1310 and/or thepersonal computer1304,tablet computing device1306, ormobile computing device1308, as described above. For example, theserver device1302 may include thegaze detector1329, theapplication interface1325, theidentity management system1326, and/or the function/action/state controller1327. Thegaze detector1329 may be the same as or similar to thegaze detector612 as previously described; theapplication interface1325 may be the same as or similar to theapplication interface624 as previously described; theidentity management system1326 may be the same as or similar to theidentity management system628 as previously described; and the function/action/state controller1327 may be the same as or similar to the function/action/state controller636 as previously described
• Theserver device1302 may provide data to and from a client computing device such as apersonal computer1304, atablet computing device1306 and/or a mobile computing device1308 (e.g., a smart phone) through anetwork1315. By way of example, the computer system described above may be embodied in apersonal computer1304, atablet computing device1306 and/or a mobile computing device1308 (e.g., a smart phone). Any of these embodiments of the computing devices may obtain content from thestore1316, in addition to receiving graphical data useable to be either pre-processed at a graphic-originating system, or post-processed at a receiving computing system.
• In addition, the aspects and functionalities described herein may operate over distributed systems (e.g., cloud-based computing systems), where application functionality, memory, data storage and retrieval and various processing functions may be operated remotely from each other over a distributed computing network, such as the Internet or an intranet. User interfaces and information of various types may be displayed via on-board computing device displays or via remote display units associated with one or more computing devices. For example, user interfaces and information of various types may be displayed and interacted with on a wall surface onto which user interfaces and information of various types are projected. Interaction with the multitude of computing systems with which embodiments of the invention may be practiced include, keystroke entry, touch screen entry, voice or other audio entry, gesture entry where an associated computing device is equipped with detection (e.g., camera) functionality for capturing and interpreting user gestures for controlling the functionality of the computing device, and the like.
• Aspects of the present disclosure, for example, are described above with reference to block diagrams and/or operational illustrations of methods, systems, and computer program products according to aspects of the disclosure. The functions/acts noted in the blocks may occur out of the order as shown in any flowchart. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved.
• Examples of the present disclosure are directed to a method for controlling a function via gaze detection. The method may include receiving one or more images of a user; determining gaze information from the received one or more images; receiving non-gaze information when the gaze information is determined to satisfy a condition; and enabling a function based on the non-gaze information.
• In accordance with at least one aspect of the above example, the non-gaze information includes a stored image of the user; and enabling the function includes authenticating the user for access to an access controlled system. In accordance with at least one aspect of the above example, the method may include receiving one or more second images including the user and a second user; determining second gaze information for the second user from the received one or more second images; identifying the second user associated with the second gaze information; and enabling the function includes causing a display of content displayed at a display device to be modified based on the identity of the second user. In accordance with at least one aspect of the above example, the method may include determining content displayed at the display device; and obscuring less than all of the content displayed at the display device based on the identity of the second user. In accordance with at least one aspect of the above example, the function includes one or more of controlling a brightness of a display device, resetting an inactivity timer, and obtaining a second image. In accordance with at least one aspect of the above example, determining gaze information from the received one or more images includes: extracting a plurality of features from the received one or more images; providing the plurality of features to a neural network; and determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed. In accordance with at least one aspect of the above example, the gaze information is determined to satisfy the condition when the location at the display device at which the gaze of the user is directed coincides with a region including a graphical element. In accordance with at least one aspect of the above example, determining gaze information from the received one or more images includes: extracting a plurality of features from the received one or more images; providing the plurality of features to a neural network; and determining, utilizing the neural network, a location at a location other than a display device at which a gaze of the user is directed. In accordance with at least one aspect of the above example, the method may include determining, utilizing the neural network, a future predicted location at the display device associated with a future predicted gaze of the user.
• Examples of the present disclosure are directed to a system for controlling a function via gaze detection. The system may include a processor; an image sensor; and memory including instructions, which when executed by the processor, cause the processor to receive one or more images of a user from the image sensor; determine gaze information from the received one or more images; determine if the gaze information satisfies a condition; receive non-gaze information associated with at least one of a function or application of a computing system when the gaze information satisfies the condition; and enable the at least one function or application based on non-gaze information.
• In accordance with at least one aspect of the above example, at least one processor is a neural processing unit. In accordance with at least one aspect of the above example, the instructions, when executed by the processor, cause the processor to receive second one or more images including the user and a second user; determine second gaze information for the second user from the received second one or more images; identify the second user associated with the second gaze information; and cause a display of content displayed at a display device to be modified based on the identity of the second user. In accordance with at least one aspect of the above example, the instructions, when executed by the processor, cause the processor to: obscure less than all of the content displayed at the display device based on the identity of the second user. In accordance with at least one aspect of the above example, the at least one function or application at least one of controls a brightness of a display device, resets an inactivity timer, obtains a second image, changes a font size, changes resolution, and changes a display refresh rate. In accordance with at least one aspect of the above example, the instructions, when executed by the processor, cause the processor to: extract a plurality of features from the received one or more images; provide the plurality of features to a neural network; and determine, utilizing the neural network, a location at a display device at which a gaze of the user is directed. In accordance with at least one aspect of the above example, the gaze information satisfies the condition when the location at the display device at which the gaze of the user is directed coincides with a region including a graphical element. In accordance with at least one aspect of the above example, the instructions, when executed by the processor, cause the processor to: extract a plurality of features from the received one or more images; provide the plurality of features to a neural network; and determine, utilizing the neural network, a location at a location other than a display device at which a gaze of the user is directed. In accordance with at least one aspect of the above example, the instructions, when executed by the processor, cause the processor to determine a predicted location at the display device associated with a predicted gaze of the user.
• Examples of the present disclosure are directed to a computer storage medium including instructions, which when executed by a processor, cause the processor to: receive one or more images of a user from an image sensor; determine gaze information from the received one or more images; determine if the gaze information satisfies a condition; receive device dependent information associated with at least one of a function or application of a computing system when the gaze information is determined to satisfy the condition; and enable at least one function or application based on the device dependent information. In accordance with at least one aspect of the above example, the instructions, which when executed by a processor, cause the processor to: extract a plurality of features from the received one or more images; provide the plurality of features to a neural network; and determine, utilizing the neural network, a location at a display device at which a gaze of the user is directed, wherein, the gaze information satisfies the condition when the location at a display device at which the gaze of the user is directed coincides with a region including a graphical element associated with the at least one function or application.
• The description and illustration of one or more aspects provided in this application are not intended to limit or restrict the scope of the disclosure as claimed in any way. The aspects, examples, and details provided in this application are considered sufficient to convey possession and enable others to make and use the best mode of claimed disclosure. The claimed disclosure should not be construed as being limited to any aspect, example, or detail provided in this application. Regardless of whether shown and described in combination or separately, the various features (both structural and methodological) are intended to be selectively included or omitted to produce an embodiment with a particular set of features. Having been provided with the description and illustration of the present application, one skilled in the art may envision variations, modifications, and alternate aspects falling within the spirit of the broader aspects of the general inventive concept embodied in this application that do not depart from the broader scope of the claimed disclosure.

Claims (20)

What is claimed is:

1. A method for controlling a function via gaze detection, the method comprising:

receiving one or more images of a user;

determining gaze information from the received one or more images;

receiving non-gaze information when the gaze information is determined to satisfy a condition; and

enabling a function based on the non-gaze information.

2. The method ofclaim 1, wherein the non-gaze information includes a stored image of the user; and

enabling the function includes authenticating the user for access to an access controlled system.

3. The method ofclaim 1, further comprising:

receiving one or more second images including the user and a second user;

determining second gaze information for the second user from the received one or more second images;

identifying the second user associated with the second gaze information; and

enabling the function includes causing a display of content displayed at a display device to be modified based on the identity of the second user.

4. The method ofclaim 3, further comprising:

determining content displayed at the display device; and

obscuring less than all of the content displayed at the display device based on the identity of the second user.

5. The method ofclaim 1, wherein the function includes one or more of controlling a brightness of a display device, resetting an inactivity timer, and obtaining a second image.

6. The method ofclaim 1, wherein determining gaze information from the received one or more images includes:

extracting a plurality of features from the received one or more images;

providing the plurality of features to a neural network; and

determining, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

7. The method ofclaim 6, wherein the gaze information is determined to satisfy the condition when the location at the display device at which the gaze of the user is directed coincides with a region including a graphical element.

8. The method ofclaim 1, wherein determining gaze information from the received one or more images includes:

extracting a plurality of features from the received one or more images;

providing the plurality of features to a neural network; and

determining, utilizing the neural network, a location at a location other than a display device at which a gaze of the user is directed.

9. The method ofclaim 8, further comprising determining, utilizing the neural network, a future predicted location at the display device associated with a future predicted gaze of the user.

10. A system for controlling a function via gaze detection, the system comprising:

a processor;

an image sensor; and

memory including instructions, which when executed by the processor, cause the processor to:

receive one or more images of a user from the image sensor;

determine gaze information from the received one or more images;

determine if the gaze information satisfies a condition;

receive non-gaze information associated with at least one of a function or application of a computing system when the gaze information satisfies the condition; and

enable the at least one function or application based on non-gaze information.

11. The system ofclaim 10, wherein at least one processor is a neural processing unit.

12. The system ofclaim 10, wherein the instructions, when executed by the processor, cause the processor to:

receive second one or more images including the user and a second user;

determine second gaze information for the second user from the received second one or more images;

identify the second user associated with the second gaze information; and

cause a display of content displayed at a display device to be modified based on the identity of the second user.

13. The system ofclaim 12, wherein the instructions, when executed by the processor, cause the processor to:

obscure less than all of the content displayed at the display device based on the identity of the second user.

14. The system ofclaim 10, wherein the at least one function or application at least one of controls a brightness of a display device, resets an inactivity timer, obtains a second image, changes a font size, changes resolution, and changes a display refresh rate.

15. The system ofclaim 10, wherein the instructions, when executed by the processor, cause the processor to:

extract a plurality of features from the received one or more images;

provide the plurality of features to a neural network; and

determine, utilizing the neural network, a location at a display device at which a gaze of the user is directed.

16. The system ofclaim 15, wherein the gaze information satisfies the condition when the location at the display device at which the gaze of the user is directed coincides with a region including a graphical element.

17. The system ofclaim 15, wherein the instructions, when executed by the processor, cause the processor to:

extract a plurality of features from the received one or more images;

provide the plurality of features to a neural network; and

determine, utilizing the neural network, a location at a location other than a display device at which a gaze of the user is directed.

18. The system ofclaim 17, wherein the instructions, when executed by the processor, cause the processor to determine a predicted location at the display device associated with a predicted gaze of the user.

19. A computer storage medium including instructions, which when executed by a processor, cause the processor to:

receive one or more images of a user from an image sensor;

determine gaze information from the received one or more images;

determine if the gaze information satisfies a condition;

receive device dependent information associated with at least one of a function or application of a computing system when the gaze information is determined to satisfy the condition; and

enable at least one function or application based on the device dependent information.

20. The computer storage medium ofclaim 19, wherein the instructions, which when executed by a processor, cause the processor to:

extract a plurality of features from the received one or more images;

provide the plurality of features to a neural network; and

determine, utilizing the neural network, a location at a display device at which a gaze of the user is directed,

wherein, the gaze information satisfies the condition when the location at a display device at which the gaze of the user is directed coincides with a region including a graphical element associated with the at least one function or application.

Images