US20120216152A1 - Touch gestures for remote control operations - Google Patents

Abstract

In general, this disclosure describes techniques for providing a user of a first computing device (e.g., a mobile device) with the ability to utilize the first computing device to control a second computing device (e.g., a television). Specifically, the techniques of this disclosure may, in some examples, allow the user to use drawing gestures on a mobile computing device to remotely control and operate the second computing device. Using a presence-sensitive user interface device (e.g., a touch screen), the user may use drawing gestures to indicate characters associated with operations and commands to control the second computing device.

Description

TECHNICAL FIELD
• This disclosure relates to a gesture-based user interface for mobile devices.
BACKGROUND
• Computing devices are continuously improving and becoming more commonly used. Additionally, touch-based interaction with touch screens of computing devices is also becoming a more common and major interaction modality for mobile device user interfaces. Touch-based interaction may be, for example, finger-based touch input.
• Furthermore, computing devices are being used more to interact with other devices and perform operations other than simple tasks traditionally associated with computing devices. In some instances, computing devices may be used to remotely control operations of other devices.
SUMMARY
• In general, this disclosure describes techniques for providing a user of a computing device with the ability to utilize a first computing device to control a second computing device. Specifically, the techniques of this disclosure may, in some examples, allow the user to use drawing gestures on a mobile computing device to remotely control and operate another device. Using a presence-sensitive user interface device (e.g., a touch screen), the user may use drawing gestures to indicate characters associated with operations and commands to control the second computing device.
• In one example, the disclosure is directed to a method receiving, via a presence-sensitive user interface device of a first device, a first user input comprising a first drawing gesture, transmitting, by the first device, first data representing the first drawing gesture to a second device wirelessly coupled to the first device, wherein the second device includes a display displaying one or more selectable elements, and wherein at least one of the selectable elements is graphically-highlighted on the display based upon the first data, receiving, via the presence-sensitive user interface device of the first device, a second user input comprising a selection gesture, and transmitting, by the first device, second data representing the selection gesture, wherein the at least one graphically-highlighted element is selected in response to the second data.
• In another example, the disclosure is direction to a computer-readable storage medium encoded with instructions that, when executed, cause one or more processors at a first device to perform operations comprising receiving, via a presence-sensitive user interface device of the first device, a first user input comprising a first drawing gesture, transmitting, by the first device, first data representing the first drawing gesture to a second device wirelessly coupled to the first device, wherein the second device includes a display displaying one or more selectable elements, and wherein at least one of the selectable elements is graphically-highlighted on the display based upon the first data, receiving, via the presence-sensitive user interface device of the first device, a second user input comprising a selection gesture, and transmitting, by the first device, data representing the selection gesture, wherein the at least one graphically-highlighted elements is selected in response to the second data.
• In another example, the disclosure is directed to a first device comprising one or more processors, a presence-sensitive user interface device, a gesture module operable by the one or more processors to receive, via the presence-sensitive user interface device, a first user input comprising a first drawing gesture, means for transmitting first data representing the first drawing gesture to a second device wirelessly coupled to the first device, wherein the second device includes a display displaying one or more selectable elements, and wherein at least one of the selectable elements is graphically-highlighted on the display based upon the first data, and a user interface controller operable by the one or more processors to receive, via the presence-sensitive user interface device, a second user input comprising a selection gesture, wherein the means for transmitting transmits data representing the selection gesture, and wherein the at least one graphically-highlighted elements is selected in response to the second data.
• In another example, the disclosure is directed to a system comprising a first device comprising a presence-sensitive user interface device that receives a first user input comprising a first drawing gesture and means for transmitting data, and a second device coupled to the first device, the second device comprising a display that displays one or more selectable elements, wherein the means for transmitting data transmits first data representing the first drawing gesture to the second device, wherein at least one of the selectable elements is graphically-highlighted on the display based upon the first data, wherein the presence-sensitive user interface device receives a second user input comprising a selection gesture, wherein the means for transmitting transmits data representing the selection gesture, and wherein the at least one graphically-highlighted elements is selected in response to the second data.
• Certain techniques of the present disclosure may, as one non-limiting example, allow a user of a computing device to perform certain remote operations using drawing gestures on a touch screen of the computing device. The user may enter different patterns using the gestures on the touch screen, to indicate a desired operation, resulting in remotely controlling another device.
• The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the disclosure will be apparent from the description and drawings, and from the claims.
BRIEF DESCRIPTION OF DRAWINGS
• FIG. 1 is a block diagram illustrating an example system in accordance with techniques of the present disclosure.
• FIG. 2 is a block diagram illustrating further details of one example of the first computing device shown inFIG. 1.
• FIGS. 3A and 3B are block diagrams illustrating example screen displays of a first computing device as a user interacts with the first computing device to remotely control a second computing device, in accordance with one or more aspects of the present disclosure.
• FIGS. 4A and 4B are block diagrams illustrating other example screen displays of a first computing device as a user interacts with the first computing device to remotely control a second computing device, in accordance with one or more aspects of the present disclosure.
• FIG. 5 is a flow diagram illustrating a method that may be performed by a computing device in accordance with one or more aspects of the present disclosure.
• FIG. 6 is a state diagram illustrating example state transition functionality associated with one or more aspects of the present disclosure.
DETAILED DESCRIPTION
• In general, this disclosure describes techniques for providing a user the ability to remotely control operation of a device (e.g., television, projector display, computer, or the like) using gestures (e.g., using a presence-sensitive user interface device, such as a touch screen user interface) on a computing device. These techniques may allow the user to use gestures on a first computing device to select items on the second computing device via interactions with a touch screen of the computing device. These techniques may be integrated with existing systems that allow the user to draw gestures on the touch screen, where the gestures may be associated with characters corresponding to items displayed on the second computing device. Using the touch screen, the user may draw gestures anywhere on the screen to interact with the second computing device, regardless of what the second computing device is or what is displayed on the second computing device. In this manner, the user is not limited to a predefined keypad corresponding to a specific second computing device, and can interact with a variety of computing devices using the same user interface. Additionally, the user may operate the second computing device without having to look at the computing device, as long as the user draws the gesture somewhere within the user interface, which is in some examples the entire screen of the computing device.
• In one example, at least a portion of the touch screen of the computing device may be allocated for drawing gestures. Using the allocated portion, the user may utilize gestures to draw characters corresponding to items displayed on the second computing device. In one example, the items may be menu items, and the characters that the user draws may correspond to a first letter of one or more of the items or a shape associated with the item. The one or more items corresponding to the drawn character may be highlighted, and the user may utilize the same technique to define more characters, until the desired item is selected. The user may then utilize the computing device to select the item (e.g., selectable menu item) or operate the item (e.g., a control item). As the user draws gestures on the computing device, a representation of the drawn gesture may be displayed on the computing device and/or the display of the second computing device. The user may also utilize gestures on the computing device to delete the last character or the sequence of characters.
• Some computing devices provide user interfaces that allow the user to utilize the first computing device as a remote control for a second computing device. Often, the user interface displays a keypad that resembles a remote control. Additionally, the user interface may display a remote control keypad specific for the second computing device. For example, the remote control keypad may be an application that the user needs to acquire for a specific second computing device, and therefore, if the user wishes to use the first computing device to remotely control a second computing device, a different remote control keypad would need to be acquired, as different computing devices are associated with different keypads. Furthermore, some computing devices may require keypads with a lot of keys to enable different functionalities. However, with the size limitations of computing devices, a remote control keypad may have keys that are too small to easily navigate and select. This can become frustrating and cumbersome for a user.
• The techniques of this disclosure allow a user to use a first computing device (e.g., mobile phone) to control a second computing device (e.g., a Google television device, television, projector, or the like), by drawing gestures on a presence-sensitive user interface device (e.g., a touch screen) of the first computing device. Rather than utilizing a keypad resembling a remote control for controlling the second computing device, the techniques allow the user to draw gestures on at least a portion of the touch screen of the computing device to indicate a desired operation. The gestures may be translated into operations by associating a character represented by the drawn gesture with one or more items that appear on the second computing device.
• FIG. 1 is a block diagram illustrating anexample system100 in accordance with techniques of the present disclosure. In the example ofFIG. 1,system100 includesfirst computing device105 andsecond computing device110. Using techniques of this disclosure,first computing device105 may be used as a remote control device ofsecond computing device110. In other examples,system100 may include one ormore computing devices105 and/or one ormore computing devices110. Devices insystem100 may interact with one another over a communication link such as, for example,connection112, when properly configured. In one example,first computing device105 andsecond computing device110 may include Wi-Fi or Bluetooth capabilities and may be configured to communicate wirelessly over aconnection112.
• In some examples,first computing device105 may comprise a mobile device. For example,first computing device105 may comprise or be part of a wireless communication device (e.g., wireless mobile handset or device), a video telephone, a digital multimedia player, a personal digital assistant (PDA), a video game console, a laptop computer, a tablet computer, or other devices. In some examples,first computing device105 may communicate with external, distinct devices via one or more networks (not shown), such as one or more wired or wireless networks, which may, in some cases, provide access to the Internet.First computing device105 may communicate with one or moresecond computing devices110 such as, for example, a standalone smart television, or a set-top box connected to a television set.
• As shown in the example ofFIG. 1,first computing device105 may includeuser interface102. At least a portion ofuser interface102 may be a presence-sensitive user interface device. The presence-sensitive user interface device may be, for example, a touch screen offirst computing device105, responsive to tactile input via a user's finger or a stylus pen, for example.First computing device105 may execute applications such as, for example, remote control application104 that enables a user offirst computing device105 to interact with and control operations ofsecond computing device110. During execution, user interaction with remote control application104 may result in controlling operations associated withsecond computing device110.
• In one example, the user may initiate connection betweenfirst computing device105 andsecond computing device110 by establishing communication overconnection112. When remote control application104 is initially launched, it may prompt the user with a list ofsecond computing devices110 with whichfirst computing device105 may communicate and control using remote control application104. The user may select the second computing device he/she wishes to connect to and control usingcomputing device105. When the connection is established betweenfirst computing device105 andsecond computing device110,second computing device110 is added to the list of devices that a user can control using remote control application104.
• Once a connection is established betweenfirst computing device105 andsecond computing device110, the user may configure remote control application104 onfirst computing device105 such that input provided by the user into remote control application104 controls operation associated withsecond computing device110. In one example, remote control application104 onfirst computing device105 may be configured to control one or moresecond computing devices110. In one example, when remote control application104 is running onfirst computing device105, a list of associatedsecond computing devices110 may be presented to the user to select the desired second computing device to control. In another example, when remote control application104 is running onfirst computing device105, thesecond computing device110 that is closest tofirst computing device105 may be automatically launched. In one example, the user may utilize gestures to switch amongsecond computing devices110, while remote control application104 is running
• In one example, remote control application104 may be operable onfirst computing device105 to perform, during execution, functions in accordance with the techniques of this disclosure. For example, remote control application104 may interact and/or exchange data with a device that is external tofirst computing device105, such assecond computing device110.First computing device105 may, in various examples, download or otherwise obtain remote control application104 from an external server via one or more networks (not shown). For example, a web browser hosted byfirst computing device105 may download one or more applications, such as remote control application104, upon access of one or more web sites hosted by such as external server (e.g., web server).
• During execution, remote control application104 may implement, invoke, execute, or otherwise utilizeuser interface102 as a mechanism to obtain user input. For example, during an initial interaction, remote control application104 may prompt the user viauser interface102 to provide set-up information associated with the user and anysecond computing devices110 that the user wishes to interact with using remote control application104. In another example, during execution, remote control application104 may present to the user, viauser interface102, a list of one or moresecond computing devices110 from which the user may select a second computing device that he/she wishes to interact with. Remote application104 may then invokeuser interface102 in gesture-based mode such that a user may draw gestures inuser interface102 to indicate remote control operations that he/she wishes to utilize to control operation ofsecond computing device110.
• User interface102 may, during execution of remote control application104, provide the user with at least one portion that is presence-sensitive and with which the user may interact via touch (e.g., by finger or a stylus pen) to draw gestures corresponding to items onsecond computing device110. A representation of the drawn gestures may be displayed onuser interface102 as the user draws the gestures. Remote control application104 may include gesture-recognition capabilities, which may be capable of recognizing a drawn gesture and translating it into a matching character, e.g., a letter. As the user draws gestures on the portion ofuser interface102 dedicated for gesture input, remote control application104 may establish communication, viaconnection112, withsecond computing device110 and transfer information regarding the drawn gesture tosecond computing device110. In one example,first computing device105 may execute an algorithm that performs the gesture recognition to determine the corresponding character, and sends the drawn gesture and corresponding character tosecond computing device110. In another example,first computing device105 may send the drawn gesture tosecond computing device110, andsecond computing device110 may execute an algorithm that performs the gesture recognition to determine the corresponding character. In this example,second computing device110 may comprises one or more processors operable to execute an algorithm to perform gesture recognition, which may be stored on a memory and/or storage ofsecond computing device110.
• The character corresponding to the drawn gesture may be associated with an item displayed on a display ofsecond computing device110. An indication may be displayed onsecond computing device110 to indicate the associated item. In one example, the associated item may be highlighted by displaying a box around the item, or displaying the item in a different color or using a different pattern, or the like. The user may then utilize a gesture (e.g., touching user interface102) to indicate desire to select the highlighted item. In one example, selecting the highlighted item may result in displaying further items asFIG. 1 illustrates, if the selected item (e.g.,item2 inFIG. 1) hassub-items114. The user may then utilize the same techniques described above to select one of the sub-items fromsub-items114. In another example, selecting the highlighted item may result in executing an action associated with the item (e.g., changing a channel, recording a program, or launching a website).
• In one example, more than one item displayed onsecond computing device110 may be associated with the character. For example, the drawn gesture may correspond to the letter A, and two or more items displayed onsecond computing device110 may begin with the letter A. In this example, the one of the items listed first (e.g., closer to the top of the list) may be highlighted. The user may decide that the highlighted item is not the one he/she wished to interact with, and may utilizefirst computing device105 to draw a gesture associated with the second letter of the desired item. In the same manner as described above, a character corresponding to the drawn gesture may be transferred tosecond computing device110, and one of the items associated with the first character may be selected based on association with the second character, and highlighted for user selection, as noted above.
• In one example, the user may utilize other gestures for editing purposes. For example, if the user draws a gesture that is erroneous or results in highlighting the wrong item, the user may make a swiping motion acrossuser interface102 to delete the last character corresponding to the last drawn gesture (e.g., by swiping horizontally leftward on the screen) or to delete the sequence of characters corresponding to the drawn gestures (e.g., by swiping horizontally rightward on the screen). When the user deletes a character or a sequence of characters, the highlighted item on the screen changes accordingly. For example, if the user deletes a character in a sequence of characters, the item corresponding to the remaining characters may be highlighted. In another example, if the user deletes the entire sequence of characters, resulting in none of the items being highlighted.
• In one example, as noted above, the user may define certain operations and delete characters corresponding to drawn gestures using gestures on the screen. For example, to delete the character corresponding to the last drawn gesture, the user may swipe horizontally leftward acrossuser interface102. In another example, to delete the sequence of characters corresponding to a sequence of drawn gestures, the user may swipe horizontally rightward acrossuser interface102. As noted above, the user may select a highlighted item by tapping onuser interface102. Other gestures onuser interface102 may be also utilized to perform other operations such as, for example, moving the highlight box from one item to another by swiping vertically upward or downward onuser interface102.
• In another example, the user may define certain operations and delete characters corresponding to drawn gestures using motion gestures, i.e., by movingcomputing device105 in certain directions. For example, to delete the character corresponding to the last drawn gesture, the user may movecomputing device105 horizontally leftward, and to delete a sequence of characters, the user may movecomputing device105 horizontally rightward.
• In this example, the user may movecomputing device105 vertically upward or downward to move the highlight box from one item to another.
• FIG. 2 is a block diagram illustrating further details ofcomputing device105 shown inFIG. 1.FIG. 2 illustrates only one particular example ofcomputing device105, and many other example embodiments ofcomputing device105 may be used in other instances. As shown in the example ofFIG. 2,computing device105 includes one ormore processors122,memory124, a network interface126, one ormore storage devices128, user interface130, and anoptional battery132. For example, if computingdevice105 comprises a mobile device,computing device105 may includebattery132. Each ofcomponents122,124,126,128,130, and132 may be interconnected via one or more busses for inter-component communications.Processors122 may be configured to implement functionality and/or process instructions for execution withincomputing device105.Processors122 may be capable of processing instructions stored inmemory124 or instructions stored onstorage devices128.
• User interface130 may include, for example, a monitor or other display device for presentation of visual information to a user ofcomputing device105. User interface130 may further include one or more input devices to enable a user to input data, such as a manual keyboard, mouse, touchpad, trackpad, etc. In some examples, user interface130 may comprise a presence-sensitive user interface device such as, for example, a touch screen, which may be used both to receive and process user input and also to display output information. User interface130 may further include printers or other devices to output information. In various instances in the description contained herein, references made to user interface130 may refer to portions of user interface130 (e.g., touch screen) that provide user input functionality. In one example, user interface130 may be a touch screen that is responsive to tactile input by the user (e.g., by user's finger or stylus pen).
• Memory124 may be configured to store information withincomputing device105 during operation.Memory124 may, in some examples, be described as a computer-readable storage medium. In some examples,memory124 is a temporary memory, meaning that a primary purpose ofmemory124 is not long-term storage.Memory124 may also be described as a volatile memory, meaning thatmemory124 does not maintain stored contents when the computer is turned off. Examples of volatile memories include random access memories (RAM), dynamic random access memories (DRAM), static random access memories (SRAM), and other forms of volatile memories known in the art. In some examples,memory124 may be used to store program instructions for execution byprocessors122.Memory124 may be used by software or applications running on computing device105 (e.g., remote application104 shown inFIG. 1) to temporarily store information during program execution.
• Storage devices128 may also include one or more computer-readable storage media.Storage devices128 may be configured to store larger amounts of information thanmemory124.Storage devices128 may further be configured for long-term storage of information. In some examples,storage devices128 may comprise non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard discs, optical discs, floppy discs, flash memories, or forms of electrically programmable memories (EPROM) or electrically erasable and programmable (EEPROM) memories.
• Computing device105 also includes network interface126.Computing device105 may utilize network interface126 to communicate with external devices (e.g., one or more servers, web servers, second computing device110) via one or more networks, such as one or more wireless/wired networks.Computing device105 may utilize network interface126 in response to execution of one or more applications that require transferring data to and/or from other devices (e.g., other computing devices, servers, or the like).Computing device105 may include Wi-Fi or Bluetooth capabilities, for example, which may be configurable to establish communication with second computing device110 (FIG. 1) through network interface126.
• Any applications implemented within or executed by computing device105 (e.g., remote control application104 shown inFIG. 1) may be implemented or contained within, operable by, executed by, and/or be operatively coupled toprocessors122,memory124, network interface126,storage devices128, and/or user interface130.
• In one example,computing device105 may include remote control application104, which allows a user to input gestures oncomputing device105 to control operations on asecond computing device110. Remote control application104 may include adisplay module142, a user interface controller144, and agesture module146. Remote control application104 may provide ordisplay user interface102 on which a user may provide tactile input to operatesecond computing device110. Remote control application104 may be stored inmemory124 and/or storage devices130, and may be operable byprocessors122 to perform various tasks during execution.
• In one example, during implementation or execution of remote control application104,display module142 may be operable byprocessors122 to define at least a portion of user interface130 to receive gestures via tactile user input. User interface controller144 may be operable byprocessors122 to receive, via user interface130, user input specifying gestures drawn intended to define operations to controlsecond computing device110. The user input may comprise contact with user interface130 (e.g., contact with a touch screen) to draw gestures, where each of the gestures corresponds to an operation associated with asecond computing device110.
• Gesture module146 may be operable byprocessor122 to determine, based on gestures the user draws on user interface130, the appropriate matching characters and operations, which may be subsequently associated with the appropriate operations or items onsecond computing device110. In one example,display module142 may define at least a portion of user interface130 for gesture input. In one example,gesture module146 may display the drawn gestures on user interface130 and determine the corresponding character. In another example,gesture module146 may send the drawn gestures for display onsecond computing device110.
• In some examples,gesture module146 may also determine, based on the direction a user movescomputing device105, operations corresponding to the motions. The operations may result in deleting characters previously defined by user's drawn gestures. In one example, the deleting operations may be defined using drawn gestures, instead of motions, on user interface130. Whethergesture module146 interprets motion gestures in addition to drawn gestures may be based on user selection indicating user's wish to utilize motion gestures, or may be a default setting of remote control application104. In examples where motion gestures are utilized,computing device105 may also include components capable of detecting motions and changes in position ofcomputing device105, e.g., accelerometer, compass, or the like.
• Remote control application104 may define at least a portion on user interface130 where a user may use gestures to draw characters associated with operations onsecond computing device110. The characters may correspond to letters or shapes associated with items and operations ofsecond computing device110. Additionally, using certain drawn gestures or motions, the user may be able to apply operations to the already-entered characters, e.g., delete one or more characters.
• In one example, as the user utilizes gestures to draw characters and operations,gesture module146 may determine the matching desired characters and operations.Display module142 may be operable to receive the data regarding the drawn gestures to display on user interface130. In this manner,display module142 may be operable to display the gestures as the user draws them on user interface130. In some examples,gesture module146 may transfer data regarding the drawn gestures tosecond computing device110, which may display the drawn gestures.Second computing device110 may then determine based on the data regarding the drawn gestures, the corresponding characters or operations. In one example,gesture module146 may also transfer data regarding the characters or operations corresponding to the drawn gestures tosecond computing device110.
• Processor122 may be operable to execute one or more algorithms including, for example, a gesture-interpretation algorithm. In one example, the gesture-interpretation algorithm may determine the characters and operations corresponding to the drawn gestures. In some examples, the algorithm may associate the characters and operations corresponding to drawn gestures with items and operations ofsecond computing device110. In other examples, the algorithm may cause the characters and operations corresponding to the drawn gestures to be transferred tosecond computing device110, where the characters and operations may be associated with items and operations onsecond computing device110.
• FIGS. 3A and 3B are block diagrams illustrating example screen displays of a first computing device as a user interacts with the computing device to remotely control a second computing device, in accordance with one or more aspects of the present disclosure.First computing device305 may operate in the same manner asfirst computing device105 ofFIGS. 1 and 2, andsecond computing device310 may operate in the same manner assecond computing device110 ofFIG. 1.First computing device305 may include a presence-sensitive user interface device such as, for example, user interface302 (e.g., a touch screen).
• AsFIG. 3A illustrates,first computing device305 may communicate wirelessly withsecond computing device310 viaconnection312 utilizing one of the many available wireless technologies, e.g., Wi-Fi, Bluetooth, or the like.Second computing device310 may list target elements, which may be associated with different operations or functionalities ondisplay device316 associated withsecond computing device310. In one example, target elements displayed ondisplay device316 ofsecond computing device310 may be textual objects, asFIG. 3A illustrates, and may be indexed according to their name, e.g., bookmarks, spotlights, applications, etc. Target elements may also be, for example, graphical objects, e.g., slider, progress bar, volume knob, and the like, and may be indexed using keywords corresponding to the graphical objects, e.g., slider, bar, volume, knob, and so forth. In the example ofFIG. 3A, the user may utilizefirst computing device305 to draw a gesture associated with a target elements the user wishes to select. In this example, the user may draw a gesture indicating the letter “B.” As the user draws the gesture onuser interface302 offirst computing device305, a representation of the drawn gesture may be displayed on the user interface asFIG. 3A shows.
• AsFIG. 3B illustrates, when the user draws the gesture for the letter “B,” data associated with the drawn gesture may be sent tosecond computing device310. In one example,first computing device305 may interpret the drawn gesture to determine the corresponding character. In this example, the data associated with the drawn gesture may include data regarding the drawn gesture and the corresponding character. In another example,first computing device305 may send data regarding the drawn gesture tosecond computing device310, which may in turn determine the corresponding character.second computing device310 may display a representation of the drawn gesture ondisplay device316. Additionally,second computing device310 may associate the corresponding character with the target elements displayed ondisplay device316.
• In another example,second computing device310 may communicate data associated with the target elements displayed ondisplay device316. In this example,first computing device305 may interpret the drawn gesture to determine the corresponding character, and associate the corresponding character with the target element based on the communicated data fromsecond computing device310.First computing device305 may display an indication of the associated target element to the user and request confirmation. Upon confirmation,first computing device310 may communicate data indicating selection of the associated target element tosecond computing device305.Second computing device305 may then proceed with the appropriate action associated with selection of the indicated target element.
• In one example,second computing device310 may associate a character with the first indexed element that starts with the same character as the character corresponding to the drawn gesture. In this example, the first indexed target element is “Bookmarks” and may therefore be associated with the character “B” corresponding to the gesture the user drew onuser interface302 offirst computing device305.Second computing device310 may highlight the associated target element by displaying a box around the item (as illustrated inFIG. 3B), or displaying the item in a different color or using a different pattern, or the like.
• If the user wishes to select the highlighted target element, the user may provide a gesture onfirst computing device305 to indicate the desire to activate the highlighted target element. For example, the user may tap onuser interface302 offirst computing device305 to activate “Bookmarks.” In one example, activating a highlighted target element may result in different actions depending on the highlighted element and an operation associated with it. For example, activating the “Bookmarks” element may result in displaying another screen ondisplay device316 listing bookmarks associated withsecond computing device310. The bookmarks may be listed and indexed and the user may select one of the bookmarks as described above with respect to selecting one of the target elements. In another example, activating a highlighted target element may result in launching a website or a screen associated with the highlighted target element. In another example, activating a highlighted target element may result in performing an operation associated with the highlighted target element, e.g., increase/decrease volume, change display settings, fast forward, and the like.
• FIGS. 4A and 4B are block diagrams illustrating other example screen displays of a computing device as a user interacts with the computing device to remotely control a second computing device, in accordance with one or more aspects of the present disclosure. In some examples, asFIG. 4A illustrates, more than one target element may correspond to a drawn gesture. For example, the user may draw the gesture as shown inFIG. 4A, which may correspond to the letter “A.” In this example,second computing device310 may cause the first target element starting with the letter “A” to be highlighted, which is “Applications.” If the highlighted target element is the one the user desired to select, the user may activate the highlighted target element as described above.
• In one example, the user may have intended to select the target element “AM-Radio.” In this example, the user may draw another gesture to indicate the desired target element. AsFIG. 4B illustrates, the user may draw the gesture corresponding to the letter “M” to add it to the sequence of characters. In this example,second computing device310 may determine the corresponding target element by associating the characters corresponding to the drawn gestures in the order they are drawn, and therefore, associating “AM-Radio” with the drawn gestures. As the user draws more gestures, the highlighted target element is updated accordingly, asFIG. 4B illustrates, where the word “AM-Radio” is highlighted. The user may then activate the highlighted target element as described above.
• In one example, if the user makes a mistake and wishes to delete one or more of the gestures already drawn, the user may utilize gestures to delete drawn gestures. For example, the user may swipe horizontally leftward onuser interface302 to delete the last drawn gesture (e.g., “M”), and may swipe horizontally rightward onuser interface302 to delete the entire gesture sequence (e.g., “AM”). As the user deletes one or more drawn gestures, the highlighted target element may be updated accordingly. For example, if the user in the example ofFIG. 4B deleted the gesture corresponding to the letter “M,”second computing device310 may updated the highlighted target element back to “Applications.” If the user deletes the entire sequence of gestures drawn, no target element will be highlighted.
• In one example, the user may wish to move the highlighted target element without having to draw a gesture onuser interface302. The user may utilize motion gestures to move the highlighting from one target element to another, where the motions may be up, down, left, or right, for example. In the example ofFIG. 4B, the user may wish to select the target element “Queue.” The user may movecomputing device305 in an upward motion, resulting in moving the highlighting box up from “AM-Radio” until the desired target element, e.g., “Queue,” is highlighted.
• FIG. 5 is a flow diagram illustrating a method that may be performed by a computing device in accordance with one or more aspects of the present disclosure. For example, the illustrated example method may be performed, at least in part, by first computing device105 (FIGS. 1 and 2). In some examples, a computer-readable storage medium (e.g., a medium included instorage device128 ofFIG. 2) may be encoded with instructions that, when executed, cause one or more processors (e.g., processor122) to perform one or more of the acts illustrated in the method ofFIGS. 1 and 2.
• The method ofFIG. 5 includes receiving, using a presence-sensitive user interface device (e.g., touch screen102) coupled to the first computing device, first user input comprising a first drawing gesture (502). As noted above, the first drawing gesture may define a first characteristic associated with one or more target elements or items displayed on a display device associated with a second computing device (e.g., second computing device110). The first characteristic may be a first letter of the one or more target element or a shape associated with the one or more target element. The first drawing gesture may be associated with at least one of the target elements based on the first characteristic, where the association may be performed by the first computing device and/or the second computing device (504). When an element or item displayed on the second computing device is associated with the drawing gesture, the element or item may be graphically highlighted to indicate the associated element or item (506).
• The method also includes receiving, using the presence-sensitive user interface device, a second user input comprising an activation gesture (508). In response to receiving the activation gesture, the highlighted target element or item may be activated. In one example, activation may result in an operation associated with the highlighted target element or item, such as, changing the channel or manipulating a volume or display setting, for example. In another example, activation may result in displaying a subset of elements or items associated with the highlighted target element or item, such as, a submenu, for example. When activation results in displaying a subset of elements, the same method as described above may be utilized to select an element or menu of the subset of elements.
• In one example, one or more elements or items may correspond to the first drawing gesture defining a characteristic associated with a target element or item. According to techniques of this disclosure, the highlighted target element or item may be the first listed element or item, or the one closest to the top of the menu or list of elements or items. In one example, the user may provide one or more additional drawing gestures corresponding to one or more additional characteristics associated with the target element or item. As the user inputs more drawing gestures, the associated target element or item may be updated and highlighted accordingly. In another example, if the highlighted element or item is not the desired one, the user may utilize directional gesture (e.g., by drawing gesturing in certain directions or by moving the first computing device) to move the highlighting from one element or item to another. In another example, if the highlighted element or item is not the desired one, the user may utilize drawing gestures to delete previously-drawn gestures, as described above.
• FIG. 6 is a state diagram illustrating example state transition functionality associated with one or more aspects of the present disclosure. AsFIG. 6 illustrates there can be two states associated with the techniques of this disclosure: initial602 and anchoring604. Duringinitial state602, the user has not yet drawn any gestures on a user interface of a first computing device (e.g.,first computing device105 ofFIGS. 1 and 2, orfirst computing device305 ofFIGS. 3 and 4). When a user adds a gesture by drawing it on the user interface of the first computing device, the drawn gesture is interpreted to determine the corresponding character. A second computing device with which the first computing device is communicating may associate the character with a target element displayed on a display device associated with the second computing device. The associated target element may be highlighted using, for example, a focus box, as described above. When the target element is highlighted, the associated state is anchoring604, which indicates that a gesture has been drawn and the corresponding character has been associated with a target element.
• While at the anchoringstate604, several actions can occur. For example, the user may draw another gesture corresponding to a second character, and the highlighted target element is updated accordingly such that the sequence of characters associated with the target element corresponds to the sequence of drawn gestures. In another example, from anchoringstate604, the user may utilize a motion gesture to change the highlighted target element by moving the highlighting from one target element to another. In this example, the user may move the highlighting from one element to another without having to draw additional or new gestures. In another example, from the anchoringstate604, the user may utilize a gesture to delete the last drawn gesture, when there is more than one drawn gesture. Deleting the last drawn gesture may change the corresponding sequence of characters and resulting in the highlighted target element to change to the target element corresponding to the remaining drawn gestures.
• From the anchoringstate604, the user may perform some actions that may result in returning to theinitial state602. For example, the user may activate the highlighted target element by using a gesture indicating activation, e.g., tapping on the user interface of the first computing device. Activating the highlighted target element may result in displaying a new list of target elements associated with the highlighted target element or performing an operation associated with the highlighted target element, for example. The user interface of the first computing device may then be cleared and the user may start drawing gestures again to perform operations or select a target element associated with the updated display.
• In one example, from anchoringstate604, the user may swipe rightward to delete the entire sequence of gestures drawn, thus clearing all the gestures, which may result in removing the highlighting from a highlighted target element, and the user may start over from a clear user interface. In another example, if the user had only drawn one gesture, a leftward swipe on the user interface of the computing device may result in clearing the only drawn gesture and therefore, removing highlighting from a highlighted target element and returning to theinitial state602 with a clear user interface.
• The techniques described in this disclosure may be implemented, at least in part, in hardware, software, firmware, or any combination thereof. For example, various aspects of the described techniques may be implemented within one or more processors, including one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or any other equivalent integrated or discrete logic circuitry, as well as any combinations of such components. The term “processor” or “processing circuitry” may generally refer to any of the foregoing logic circuitry, alone or in combination with other logic circuitry, or any other equivalent circuitry. A control unit including hardware may also perform one or more of the techniques of this disclosure.
• Such hardware, software, and firmware may be implemented within the same device or within separate devices to support the various techniques described in this disclosure. In addition, any of the described units, modules or components may be implemented together or separately as discrete but interoperable logic devices. Depiction of different features as modules or units is intended to highlight different functional aspects and does not necessarily imply that such modules or units must be realized by separate hardware, firmware, or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware, firmware, or software components, or integrated within common or separate hardware, firmware, or software components.
• The techniques described in this disclosure may also be embodied or encoded in a computer-readable medium, such as a computer-readable storage medium, containing instructions. Instructions embedded or encoded in a computer-readable medium, including a computer-readable storage medium, may cause one or more programmable processors, or other processors, to implement one or more of the techniques described herein, such as when instructions included or encoded in the computer-readable medium are executed by the one or more processors. Computer readable storage media may include random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electronically erasable programmable read only memory (EEPROM), flash memory, a hard disk, a compact disc ROM (CD-ROM), a floppy disk, a cassette, magnetic media, optical media, or other computer readable media. In some examples, an article of manufacture may comprise one or more computer-readable storage media.
• In some examples, a computer-readable storage medium may comprise a non-transitory medium. The term “non-transitory” may indicate that the storage medium is not embodied in a carrier wave or a propagated signal. In certain examples, a non-transitory storage medium may store data that can, over time, change (e.g., in RAM or cache).
• Various embodiments of the disclosure have been described. These and other embodiments are within the scope of the following claims.

Claims (20)

1. A method comprising:

receiving, via a presence-sensitive user interface device of a first device, a first user input comprising a first drawing gesture;

transmitting, by the first device, first data representing the first drawing gesture to a second device wirelessly coupled to the first device, wherein the second device includes a display displaying one or more selectable elements, and wherein at least one of the selectable elements is graphically-highlighted on the display based upon the first data;

receiving, via the presence-sensitive user interface device of the first device, a second user input comprising a selection gesture; and

transmitting, by the first device, second data representing the selection gesture, wherein the at least one graphically-highlighted element is selected in response to the second data.

2. The method ofclaim 1, wherein more than one selectable element is associated with the first data, the method further comprising:

receiving, via the presence-sensitive user interface device of the first device, a third user input comprising a second drawing gesture; and

transmitting, by the first device, further data representing the second drawing gesture to the second device, wherein the at least one of the selectable elements is graphically-highlighted on the display based on the first data and the further data.

3. The method ofclaim 1, wherein the display of the second device displays a representation of the first drawing gesture based on the first data.

4. The method ofclaim 1, further comprising:

receiving, via the first device, a motion gesture that defines a direction of movement;

transmitting, by the first device, motion data representing the motion gesture to the second device, wherein the at least one graphically-highlighted element is updated by graphically highlighting one of the selectable elements based on the motion data, according to the defined direction relative to the graphically-highlighted target element.

5. The method ofclaim 1, wherein the display displays a new list of one or more selectable elements in response to the second data.

6. The method ofclaim 1, wherein an operation associated with the at least one graphically-highlighted element is executed by the second device responsive in response to the second data.

7. The method ofclaim 1, wherein the one or more selectable elements comprises at least one of: one or more words, one or more graphical user interface elements, or one or more operations of the second device.

8. The method ofclaim 7, wherein the first drawing gesture comprises a drawing corresponding to a first letter of the one or more words, a word associated with the one or more graphical user interface elements, or a word associated with the one or more operations of the second device.

9. The method ofclaim 1, wherein the first data comprises data indicative of the first drawing gesture, and wherein the second device determines a character based on the first data.

10. The method ofclaim 1, further comprising determining, by the first device, a character based on the first drawing gesture, wherein the first data comprises the character.

11. The method ofclaim 1, further comprising:

receiving, from the second device, data representing the one or more selectable elements;

determining, by the first device, a character based on the first drawing gesture; and

selecting at least one of the selectable elements based on the first character, wherein the first data represents the selected at least one of the selectable elements and the selection gesture comprises a confirmation of the selected at least one of the selectable elements.

12. A computer-readable storage medium encoded with instructions that, when executed, cause one or more processors of a first device to perform operations comprising:

receiving, via a presence-sensitive user interface device of the first device, a first user input comprising a first drawing gesture;

transmitting, by the first device, first data representing the first drawing gesture to a second device wirelessly coupled to the first device, wherein the second device includes a display displaying one or more selectable elements, and wherein at least one of the selectable elements is graphically-highlighted on the display based upon the first data;

receiving, via the presence-sensitive user interface device of the first device, a second user input comprising a selection gesture; and

transmitting, by the first device, data representing the selection gesture, wherein the at least one graphically-highlighted elements is selected in response to the second data.

13. The computer-readable storage medium ofclaim 12, wherein more than one selectable element is associated with the first data, further comprising instructions that cause the one or more processors to perform operations comprising:

receiving, via the presence-sensitive user interface device of the first device, a third user input comprising a second drawing gesture; and

transmitting, by the first device, further data representing the second drawing gesture to the second device, wherein the at least one of the selectable elements is graphically-highlighted on the display based on the first data and the further data.

14. The computer-readable storage medium ofclaim 12, wherein the display of the second device displays a representation of the first drawing gesture based on the first data.

15. A first device, comprising:

one or more processors;

a presence-sensitive user interface device;

a gesture module operable by the one or more processors to receive, via the presence-sensitive user interface device, a first user input comprising a first drawing gesture;

means for transmitting first data representing the first drawing gesture to a second device wirelessly coupled to the first device, wherein the second device includes a display displaying one or more selectable elements, and wherein at least one of the selectable elements is graphically-highlighted on the display based upon the first data; and

a user interface controller operable by the one or more processors to receive, via the presence-sensitive user interface device, a second user input comprising a selection gesture, wherein the means for transmitting transmits data representing the selection gesture, and wherein the at least one graphically-highlighted elements is selected in response to the second data.

16. A system comprising:

a first device comprising:

a presence-sensitive user interface device that receives a first user input comprising a first drawing gesture;

means for transmitting data; and

a second device coupled to the first device, the second device comprising a display that displays one or more selectable elements,

wherein the means for transmitting data transmits first data representing the first drawing gesture to the second device,

wherein at least one of the selectable elements is graphically-highlighted on the display based upon the first data,

wherein the presence-sensitive user interface device receives a second user input comprising a selection gesture,

wherein the means for transmitting transmits data representing the selection gesture, and

wherein the at least one graphically-highlighted elements is selected in response to the second data.

17. The system ofclaim 16, wherein more than one selectable element is associated with the first data, wherein the presence-sensitive user interface device receives a third user input comprising a second drawing gesture, wherein the means for transmitting transmits to the second device further data representing the second drawing gesture, and wherein the at least one of the selectable elements is graphically-highlighted on the display based on the first data and the further data.

18. The system ofclaim 16, wherein the first data comprises data indicative of the first drawing gesture, and wherein the second device determines a character based on the first data.

19. The system ofclaim 16, wherein the first device determines a character based on the first drawing gesture, and wherein the first data comprises the character.

20. The system ofclaim 16, wherein the first device:

receives, from the second device, data representing the one or more selectable elements;

determines a character based on the first drawing gesture; and

selects at least one of the selectable elements based on the first character, wherein the first data represents the selected at least one of the selectable elements and the selection gesture comprises a confirmation of the selected at least one of the selectable elements.

Images