US20170285931A1 - Operating visual user interface controls with ink commands - Google Patents

Abstract

A facility enabling a user to operate visual user interface controls with ink commands is described. The facility causes to be displayed a control operable with mouse input, and receives an ink command directed to the displayed control. In response, the facility operates the displayed control in accordance with the received ink command.

Description

BACKGROUND
• Computer visual interfaces—such as those of operating systems, application programs, and web pages—have conventionally been designed to operate on user input received via a mouse and/or a keyboard. For example, it is common for word processing documents to be scrollable in the vertical dimension. In order to move down one page in such a document, the user can press a page down button on the keyboard, or with the mouse click beneath an elevator portion of a displayed scrollbar. To move down a fraction of a page, the user can press a down arrow button on the keyboard, or use the mouse to click on a down arrow portion of the scrollbar. The scrollbar in this example is one kind of a variety of different “visual user interface controls,” sometimes referred to herein simply as “controls.”
SUMMARY
• This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This summary is not intended to identify key factors or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.
• A facility enabling a user to operate visual user interface controls with ink commands is described. The facility causes to be displayed a control operable with mouse input, and receives an ink command directed to the displayed control. In response, the facility operates the displayed control in accordance with the received ink command.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 is a block diagram showing some of the components that may be incorporated in at least some of the computer systems and other devices on which the facility operates.
• FIG. 2 is a flow diagram showing a process performed by the facility in some examples to manage controls in a way that enables them to be operated with ink commands.
• FIGS. 3 and 4 are display diagrams showing the input of a sample command symbol ink command and the facility's response thereto.
• FIGS. 5 and 6 are display diagrams showing the input of a sample slider hashing ink command and the facility's response thereto.
• FIGS. 7 and 8 are display diagrams showing the input of a sample numerical textual ink command and the facility's response thereto.
• FIGS. 9 and 10 are display diagrams showing the input of a sample enclosing shape item selection command and the facility's response thereto.
• FIGS. 11 and 12 are display diagrams showing the input of a sample item transfer ink command and the facility's response thereto.
• FIGS. 13 and 14 are display diagrams showing the input of a sample text filtering ink command and the facility's response thereto.
• FIGS. 15 and 16 are display diagrams showing the input of a sample text filtering reversal ink command and the facility's response thereto.
• FIGS. 17 and 18 are display diagrams showing the input of a sample check selection command and the facility's response thereto.
• FIGS. 19 and 20 are display diagrams showing the input of a sample lineout deletion command and the facility's response thereto.
• FIGS. 21 and 22 are display diagrams showing the input of a sample item picker enclosing shape ink command (similar to the enclosing shape item selection command discussed above) and the facility's response thereto.
• FIGS. 23 and 24 are display diagrams showing the input of a sample item picker textual date command (similar to the numerical textual ink command discussed above) and the facility's response thereto.
DETAILED DESCRIPTION
• Computer operating systems have recently added support for electronic ink: a user input technique where the user handwrites, draws, or performs other arbitrary spatial movements. As a result of such user input, visual marks are added to the display that correspond to the path of the movement. For example, in the case of handwriting, handwritten words appear on the display in the same handwriting style that they were executed by the user.
• In some cases, the user performs this input by moving a stylus or his or her finger across the surface of the display device. In such cases, the electronic ink generally appears on the display in the positions where the interactions occurred. In some cases, the user performs the inking input by moving a displayed position pointer on the screen, using an input device such as a mouse or touchpad.
• In the case of electronic ink corresponding to handwriting, a variety of handwriting shape recognition techniques can be used to identify text represented by the handwriting, which can be displayed in the document along with or in place of the electronic ink handwriting. Similarly, in the case of electronic ink corresponding to shapes, a variety of shape recognition techniques can be used to identify the shapes drawn by the ink.
• The inventors have identified new ways for users to interact with conventional visual user interface controls, such as those created with Extensible Application Markup Language (“XAML”), using ink. In particular, the inventors have conceived and reduced to practice a software and/or hardware facility for operating visual user interface controls with ink commands (“the facility”). In various examples, the user generates ink input by writing or drawing on or near a particular control in particular ways. The facility interprets this ink input as an ink command directed to the particular control, and proceeds to operate the control in accordance with the ink command.
• For example, in contrast to the conventional manner of scrolling a word processing document described in the Background section, a user can draw a line segment perpendicular to the word processing application document window's vertical scrollbar at a particular point in the scrollbar's height in order to jump to a corresponding location in the document. Alternatively, the user can handwrite numbers on or near the scrollbar corresponding to, for example, page number to jump to, or a percentage of the way through the document to jump to. Similar techniques can be used for other one-dimensional value setting controls, such as sliders, word processing formatting rulers, etc.
• In various additional examples, the user can: circle one or more items in a list, grid, or other container control to select the circled items; draw shapes such as a check mark or star next to one or more items in a container to select these items; draw an arrow from items selected in a source container to a destination container in order to transfer the items to the destination container, or to a visual indication of an action that can be performed on the items, such as a visual indication of a printer for printing the items; draw shapes such as a horizontal line segment, a horizontal line segment with a loop at one end, or a scribble over an item in a container to delete the item from the container; draw an approximation of a symbol on a button in order to activate the button, such as “>” to activate a media player play button, or “<<” to activate a media player fast forward button; write a sequence of characters in a container control to subset the items displayed in the containers to those that have text matching the sequence of characters, and/or to select the first or all items having matching text.
• In some examples, the use of ink commands in accordance with the facility enables the user to operate in a mode in which some or all visual user interface controls are hidden, conserving display space to be able to display additional substantive content, display substantive content at a higher level of magnification/detail, display other visual information, operate with a smaller display device, etc. This further saves user interactions that would otherwise be needed to toggle the display of visual user interface controls on and off, saving effort and time on the user's part.
• In some examples, the facility receives ink commands in isolation, in the absence of both visible user interface controls and hidden user interface controls. For example, in various examples, the facility receives ink commands in a modeless display mode without visible user interface controls, such as a virtual reality or augmented reality display, a gaming display, a video-playing display, etc.; in various examples, such ink commands are received via movement of a finger, a stylus or other held object, a laser pointer, etc.
• Compared to spoken commands, the use of ink commands in accordance with the facility can be more accurate, and can benefit from spatial context provided by the display location at which the ink command is written, contrasted with spoken commands that do not relate to any particular display location.
• By performing in some or all of the ways described above, the facility enables users to operate many conventional visual interface controls more quickly and/or more accurately than using the forms of input for which they were originally designed, and do so without the disruption of having to set down a stylus, or resume using a mouse or keyboard. In some examples, the facility further provides a reduction in the level of computing resources consumed, such as by removing the need to expend processing resources to process mouse or keyboard commands for which ink commands are substituted, removing the need to equip certain devices with a mouse and/or a keyboard, allowing some devices to employ a smaller display, etc.
• FIG. 1 is a block diagram showing some of the components that may be incorporated in at least some of the computer systems and other devices on which the facility operates. In various examples, these computer systems andother devices100 can include server computer systems, desktop computer systems, laptop computer systems, tablet computer systems, netbooks, mobile phones, personal digital assistants, televisions, cameras, automobile computers, electronic media players, electronic kiosk devices, electronic table devices, electronic whiteboard devices, etc. In various examples, the computer systems and devices may include any number of the following: a central processing unit (“CPU”)101 for executing computer programs; acomputer memory102 for storing programs and data while they are being used, including the facility and associated data, an operating system including a kernel and device drivers, and one or more applications; apersistent storage device103, such as a hard drive or flash drive for persistently storing programs and data; a computer-readable media drive104, such as a floppy, CD-ROM, or DVD drive, for reading programs and data stored on a computer-readable medium; and/or acommunications subsystem105 for connecting the computer system to other computer systems and/or other devices to send and/or receive data, such as via the Internet or another wired or wireless network and its networking hardware, such as switches, routers, repeaters, electrical cables and optical fibers, light emitters and receivers, radio transmitters and receivers, and the like.
• In various examples, these computer systems andother devices100 may further include any number of the following: adisplay106 for presenting visual information, such as text, images, icons, documents, menus, etc.; and atouchscreen digitizer107 for sensing interactions with the display, such as touching the display with one or more fingers, styluses, or other objects. In various examples, the touchscreen digitizer uses one or more available techniques for sensing interactions with the display, such as resistive sensing, surface acoustic wave sensing, surface capacitance sensing, projected capacitance sensing, infrared grid sensing, infrared acrylic projection sensing, optical imaging sensing, dispersive signal sensing, and acoustic pulse recognition sensing. In various examples, the computer systems andother devices100 include input devices of various other types, such as keyboards, mice, styluses, etc. (not shown).
• While computer systems or other devices configured as described above may be used to support the operation of the facility, those skilled in the art will appreciate that the facility may be implemented using devices of various types and configurations, and having various components.
• FIG. 2 is a flow diagram showing a process performed by the facility in some examples to manage controls in a way that enables them to be operated with ink commands. In some examples, this process is performed by an operating system executing on the computer system or other device on which the facility operates. Examples depicting the performance of this process are shown inFIGS. 3-24 and discussed below.
• At201, the facility displays one or more visual user interface controls, such as in the context of an operating system shell, an application program, or web page, for example. In some examples, the control or controls displayed at201 are defined in manners that are completely conventional. In some examples, the control or controls displayed at201 are defined in a manner that specifies a set of one or more ink commands that can be used to operate each, in some cases including particular ranges of control operation that are permissible with ink commands. At202, the facility receives user input, and branches on the type of input received: if the input received is keyboard or mouse input, then the facility continues at203; if the input received is ink input, then the facility continues at204.
• At203, the facility operates the control displayed at201 in accordance with the received keyboard or mouse input, such as in the manner that is conventional for the control. Afteract203, the facility continues at202 to receive further input. At204, where the received input is ink input, if an ink command is recognizable in the received ink input, then the facility continues at205, else the facility continues at202. At205, the facility operates the control in accordance with the ink command recognized in the received ink input. Afteract205, the facility continues at202.
• Those skilled in the art will appreciate that the acts shown inFIG. 2 may be altered in a variety of ways. For example, the order of the acts may be rearranged; some acts may be performed in parallel; shown acts may be omitted, or other acts may be included; a shown act may be divided into subacts, or multiple shown acts may be combined into a single act, etc.
• FIGS. 3 and 4 are display diagrams showing the input of a sample command symbol ink command and the facility's response thereto.FIG. 3 shows an initial state of amedia player window300 that is paused in the process of playing an audio sequence. In some examples, such a media player can be used to play a variety of time-indexed media types, including audio sequences, video sequences, animations, slide shows, etc. The media player window contains controls including the following: aplay control301 that the user may activate by, for example, using the mouse to click within it to begin playing of the audio file; afast forward control302 that the user may activate by, for example, using the mouse to click within it to advance playback of the audio file quickly to a later time index; arewind control303 that the user may activate, such as by using the mouse to click within it to return playback of the audio file to an earlier time index; astop control304 that the user may activate by, for example, using the mouse to click within it to end playback of the audio file; and a playbackindex time control305 that the user may operate by, for example, dragging itsknob306 to a new horizontal position to change the current playback time index to a corresponding playback time index. The current display of theplay control301 indicates that playback of the audio file is currently paused.
• The display further shows the user's generation ofink input320 forming a “>” play symbol. The facility recognizesink input320 as a “play” symbol ink command.
• FIG. 4 shows the facility's response to recognizingink input320 as a “play” ink command. In many cases, the ink input recognized as a command symbol ink command visually resembles to some extent a symbol associated with the command, such as a symbol displayed within a control that the ink command activates. It can be seen that the effect of this ink command has been to activateplay control301 shown inFIG. 3, resuming playback of the audio file. This playing state is reflected by substitution of apause control401 for theplay control301 shown inFIG. 3.
• Relative to clicking the play button, using the ink command can be performed more quickly, and with less spatial acuity: where clicking the play button requires a touch or click within the small area of the play button, the ink command can be drawn in a much larger area of the screen. Also, the ink command can be performed immediately at a time when the play button is not displayed (not shown), whereas, in this situation, in order to press the play button, the user must first perform an interaction to display the play button, then touch or click within it.
• In various examples, the facility recognizes a variety of similar control symbol ink commands for operating media player controls, in some cases including: a “∥” pause ink command for operating thepause control401; a “>>” fast forward ink command for activating the fast forward control402; a “<<” ink command for activating the rewind control403; a “□” stop ink command for activating the stop control404; etc.
• In some examples, the facility recognizes similar command symbol ink commands in connection with an electronic reader application that displays the contents of a book or other publication for reading. Some examples include a “>” flip single page command to flip one page forward from the current page; a “>>>” triple page flip ink command to flip three pages forward from the current page; a “6>” ink command for flipping forward six pages; etc.
• Those of ordinary skill in the art will appreciate that, in various examples, the facility recognizes various similar command symbol ink commands to operate controls of a variety of types in a variety of contexts.
• FIGS. 5 and 6 are display diagrams showing the input of a sample slider hashing ink command and the facility's response thereto.FIG. 3 shows an initial state of amedia player window500 that is in the process of playing an audio sequence. The media player window contains controls including avolume slider501 having aknob502 that can move to different positions on atrack503 to set the playback volume to points within a permissible range represented by the length of the track. The user can operate the volume slider by, for example, clicking on its knob with the mouse, and dragging the knob to a new horizontal position on the track before releasing the mouse button.
• The display further shows the user's generation ofink input520 forming a roughly vertical hash mark at a particular horizontal position on the volume slider. The facility recognizesink input520 as a slider hash mark ink command.
• FIG. 6 shows the facility's response to recognizingink input520 as a slider hash ink command. It can be seen that the effect of this ink command has been to operate the volume slider, moving itsknob602 to the point where the hash ink input intersected the volume slider.
• In various examples, the facility recognizes a variety of similar slider hash ink commands in connection with media player controls, such as the playbacktime index slider603.
• In various examples, the facility recognizes similar slider hash ink commands to operate a variety of slider controls in a variety of contexts, as well as recognizing similar slider hashing ink commands to operate single-dimensional position controls other than sliders, such as scroll bars, rulers, round analog clock and pie chart controls, etc.
• FIGS. 7 and 8 are display diagrams showing the input of a sample numerical textual ink command and the facility's response thereto.FIG. 7 shows an initial state of a media player window700 that is in the process of playing an audio sequence. The media player window contains controls including a playbacktime index slider701, having aknob702 whose position on the slider indicates a relative position of the current playback time index within the duration of the audio sequence. Themedia player window300 also contains a current playbacktime index display703, showing the current playback time index to be ‘00:10’. The user can operate the playback time index slider by, for example, clicking on its knob with the mouse, and dragging the knob to a new horizontal position on the playback time index slider before releasing the mouse button. Doing so (a) relocates the knob on the slider; (b) updates the current playback time index display; and (c) causes playback to resume at the new playback time index.
• The display further shows the user's generation ofink input720 forming the text ‘2:30’. The facility recognizesink input720 as an ink command for jumping to the absolute time index ‘2:30’.
• FIG. 8 shows the facility's response to recognizingink input720 as an ink command to jump to absolute playback time index ‘2:30’. It can be seen that the effect of this ink command has been to change the current playbacktime index display803 to ‘2:30’, and move theslider knob802 to a corresponding position on the playbacktime index slider801.
• In various examples, the facility recognizes a variety of similar numerical textual ink commands for operating media player controls, such as volume slider804: the user can write a value such as ‘25%’, or ‘85 dBA’, for example.
• In various examples, the facility recognizes similar numerical textual ink commands to operate a variety of controls, such as writing a page number on the page of an electronic reader application to jump to the corresponding page of the book, writing a page number on the vertical scroll bar in a text editing application to jump to the corresponding page of the document being edited, writing a dimension size on a shape in a document in order to scale the shape to the corresponding dimension size, etc.
• FIGS. 9 and 10 are display diagrams showing the input of a sample enclosing shape item selection command and the facility's response thereto.FIG. 9 shows an initial state of a file systemcontainer control window900 that contains text document items901-912. The user can select items in the container by, example, holding down the control key on the keyboard and clicking each item to be selected in turn. Once items are selected in the display, the user can subject them all to the same command, such as deleting them, opening them, moving or copying them to a different container, etc.
• The display further shows the user's generation ofink input920 forming an enclosing shape arounditems903,904,908, and912. The facility recognizesink input920 as an enclosing shape item selection command.
• FIG. 10 shows the facility's response to recognizingink input920 as an enclosing shape item selection ink command. While in some cases the ink input recognized by the facility as an enclosing shape item selection command is a closed shape, in some examples, the facility recognizes as enclosing shape item selection commands ink input whose shape, while not closed, demonstrates an intent to enclose one or more items. It can be seen that the effect of this ink command has been to selectitems1003,1004,1008, and1012, such as to perform a particular file system operation with respect to all of these four items simultaneously.
• In various examples, the facility recognizes various similar enclosing shape item selection commands for selecting items of a variety of types within container controls of a variety of types, or in other collections of items, such as lists, trees, diagrams, etc.
• FIGS. 11 and 12 are display diagrams showing the input of a sample item transfer ink command and the facility's response thereto.FIG. 11 shows an initial state of a filesystem container control1100. Within this file system container control, it can be seen thatitems1103,1104,1108, and1112 are selected. In order to move or copy the four selected items to a different container, the user can, for example, click on one of the selected items with the mouse, and drag this item to the container before releasing the mouse button.
• The display further shows the user's generation ofink input1120 forming an arrow from one of the selected items to arepresentation1119 of a different file system container, ‘folder H’. The facility recognizesink input1120 as an item transfer ink command.
• FIG. 12 shows the facility's response to recognizinginput1120 as an item transfer ink command. In some cases, ink input recognized by the facility as an item transfer command specifies transferring the effected items to a variety of destinations, including file system containers and containers of other types; visual representations of actions in order to perform an action for the transferred items, such as a visual representation of a printer in order to print the transferred items; etc.FIG. 12 shows a new filesystem container control1200 corresponding to the destination ‘folder H’ container. It can be seen thatitems1203,1204,1208, and1212 have been transferred to the ‘folder H’ container, such as by moving—such that these items cease to be contained by their original ‘text snippets’ container, or by copying—such that they continue to be contained by their original ‘text snippets’ container.
• FIGS. 13 and 14 are display diagrams showing the input of a sample text filtering ink command and the facility's response thereto.FIG. 13 shows an initial state of a filesystem container control1300 containing items1301-1312, each of which has a different name that is displayed in connection with the item.
• The display further shows the user's generation ofink input1320 forming the text string ‘new’. The facility recognizesink input1320 as a text filtering ink command specifying to filter the items in the container that are displayed to those whose name includes the string ‘new’.
• FIG. 14 shows the facility's response to recognizingink input1320 as a text filtering command. In some examples, text filtering ink commands filter the displayed items based upon attributes of the items other than name. In some examples, text filtering ink commands filter the items in a container that are displayed based upon attributes of the items that are not displayed with the items in the container. It can be seen that the effect of this ink command has been to display within thecontainer1400 onlyitems1401,1404, and1412 that have the string ‘new’ in their names, omitting to displayitems1302,1303, and1305-1311 shown inFIG. 13 that do not have the string ‘new’ contained in their names.
• In various examples, the facility recognizes various similar text filtering ink commands to filter items displayed in a variety of containers and other collections of items, such as lists, trees, diagrams, etc.
• FIGS. 15 and 16 are display diagrams showing the input of a sample text filtering reversal ink command and the facility's response thereto.FIG. 15 shows an initial state of afile system container1500 whose display has been filtered as shown inFIGS. 13 and 14 and discussed above to include only items contained by the container that have the string ‘new’ in their names—items1501,1504, and1512.
• The display further shows the user's generation ofink input1520 forming a ‘

  

  ’ revert symbol. The facility recognizesink input1520 as a text filtering reversal command.
• FIG. 16 shows the facility's response to recognizingink input320 as a text filtering reversal ink command. In some examples, the facility recognizes similar ink input as ink commands to reverse a variety of other positive ink commands, such as item transfer commands discussed above, deletion commands discussed below, etc. It can be seen that the effect of this ink command has been to reverse the filtering of items infile system container1600, reverting to the display of all of the items contained by the container, items1601-1612.
• FIGS. 17 and 18 are display diagrams showing the input of a sample check selection command and the facility's response thereto.FIG. 17 shows an initial state of an emailinbox container control1700. The container contains at least email items1701-1709. The user can select email items by, for example, using the mouse to click to the left of each of one or more of the displayed email items.
• The display further shows the user's generation ofink input1721,1722, and1723 forming checkmarks near each ofemail message items1704,1706, and1707. The facility recognizes ink input1721-1723 as one or more check selection ink commands.
• FIG. 18 shows the facility's response to recognizing ink input1721-1723 as one or more check selection commands. It can be seen that the effect of this ink command has been to selectemail items1804,1806, and1807, such as to perform a single action with respect to all three of these email items simultaneously.
• In various examples, the facility recognizes a variety of similar symbols or other inking actions as selection ink commands. In various examples, the facility recognizes various similar selection commands for selecting items of a variety of types within container controls of a variety of types, or among other collections of items such as lists, trees, diagrams, etc.
• In some examples (not shown), the facility recognizes symbols such as a star or asterisk as ink commands for marking an item in a container control as a favorite. Additionally, in order to mark an item as a 3-star favorite, the user can draw a series of 3 star symbols, or write thenumeral 3 followed by a single star symbol.
• FIGS. 19 and 20 are display diagrams showing the input of a sample lineout deletion command and the facility's response thereto.FIG. 19 shows an initial state of an inboxemail container control1900, which contains at least email items1901-1909. To delete one of the displayed email items, the user can, for example, click to its left to select it and press a delete key on the keyboard.
• The display further shows the user's generation ofink input1920 forming a horizontal line through at least a portion ofemail item1902. The facility recognizesink input1920 as a lineout deletion ink command. In some examples, the facility recognizes other ink input shapes as lineout deletion ink commands, such as horizontal lines that, unlike the one shown inFIG. 19, omit a loop at the right end; ink input shapes constituting a back-and-forth scribble; etc.
• FIG. 20 shows the facility's response to recognizingink input1920 as a lineout deletion command. It can be seen that the effect of this ink command has been to deleteemail item1902 shown inFIG. 19, such that it is no longer contained byinbox email container2000.
• In various examples, the facility recognizes lineout deletion commands to delete items of a variety of types within container controls of a variety of types, or other collections of items, such as lists, trees, diagrams, etc.
• FIGS. 21 and 22 are display diagrams showing the input of a sample item picker enclosing shape ink command (similar to the enclosing shape item selection command discussed above) and the facility's response thereto.FIG. 21 shows ahotel search webpage2100 on a travel website. The webpage contains adate picker control2101 for selecting a date range for which to reserve a hotel room. Using a mouse, the user could do so by, for example, first clicking on the date at the beginning of the range, then clicking on the date at the end of the range.
• The display further shows the user's generation ofink input2120, forming a loop around dates2102 (Mar. 9, 2016) and2103 (Mar. 10, 2016). The facility recognizesink input2120 as an item picker enclosing shape ink command.
• FIG. 22 shows the facility's response to recognizingink input2120 as an item picker enclosing shape ink command. It can be seen that the effect of this ink command has been to select a date range from date2202 (Mar. 9, 2016) to date2203 (Mar. 10, 2016).
• In a variety of examples, the facility recognizes item picker enclosing shape and commands to make selections in item picker controls of a variety of kind.
• FIGS. 23 and 24 are display diagrams showing the input of a sample item picker textual date command (similar to the numerical textual ink command discussed above) and the facility's response thereto.FIG. 23 shows an initial state of ahotel reservation webpage2300 of a travel website. The webpage includes individual date picker controls2301 and2302 for specifying check-in and check-out dates for which to reserve a hotel room. The user can operate these date-picker controls by, for example, clicking on each with the mouse and typing a date with the keyboard; clicking on the calendar icon within todisplay control2101 shown inFIG. 21; etc.
• The display further shows the user's generation ofink input2320 forming the text ‘March 9-10’ near the date-picker controls. The facility recognizesink input2320 as an item picker textual ink command.
• FIG. 24 shows the facility's response to recognizingink input2320 as an item picker textual ink command specifying the date range ‘March 9-10’. It can be seen that the effect of this ink command has been to select indate picker control2401 the check-in date of Mar. 9, 2016, and select indate picker control2402 the check-out date of Mar. 10, 2016.
• In some examples, the facility omits to recognize ink commands in connection with text input field controls, and/or omits to transcribe handwritten text directly into a field in any type of control, and/or transcribes handwritten text into a field in a control only after performing transformation on the recognized text.
• In various examples, the facility recognizes item picker textual ink commands to make selections in a variety of types of item pickers.
• In some examples, the facility provides a processor-based device, comprising: a processor; and a memory having contents that cause the processor to, in response to execution of an operating system by the processor: cause to be displayed a control operable with mouse and/or keyboard input; receive ink input proximate to the displayed control; interpret the received ink input as an ink command; and operate the displayed control in accordance with the interpreted ink command.
• In some examples, the facility provides a computer-readable medium having contents configured to cause a computing system to: cause to be displayed a control operable with mouse input; receive an ink command directed to the displayed control; and operate the displayed control in accordance with the received ink command.
• In some examples, the facility provides a method in a computing system, the method comprising: causing to be displayed a control operable with mouse input; receiving ink input proximate to the displayed control; interpreting the received ink input as an ink command; and operating the displayed control in accordance with the interpreted ink command.
• It will be appreciated by those skilled in the art that the above-described facility may be straightforwardly adapted or extended in various ways. While the foregoing description makes reference to particular examples, the scope of the invention is defined solely by the claims that follow and the elements recited therein.

Claims (20)

1. A processor-based device, comprising:

a processor; and

a memory having contents that cause the processor to, in response to execution of an operating system by the processor:

cause to be displayed a control operable with mouse and/or keyboard input;

receive ink input proximate to the displayed control;

interpret the received ink input as an ink command; and

operate the displayed control in accordance with the interpreted ink command.

2. The device ofclaim 1 wherein the displayed control is a one-dimensional value-setting control for setting an index value, and wherein the received ink input comprises a handwritten number, and wherein the interpreted ink command specifies an index value that is based on the handwritten number.

3. The device ofclaim 2 wherein the displayed control is a one-dimensional value-setting control for setting a media playback index time, a page number, or an audio volume level.

4. The device ofclaim 1 wherein the displayed control is for setting a quantity, and has

a handle and a track, the track having a first end and a second end, the handle being repositionable to different points on the track corresponding to different values of the quantity,

wherein the received ink input comprises ink that crosses the control's track a particular percentage of the distance from the first end of the track to the second end of the track,

and wherein the interpreted ink command specifies a value for the quantity that is based upon the percentage.

5. The device ofclaim 4 wherein the displayed control is a slider, scrollbar, or ruler.

6. The device ofclaim 1 wherein the displayed control is a button bearing a visual symbol,

and wherein the received ink input comprises an approximation of the visual symbol borne by the button,

and wherein operating the displayed control in accordance with the interpreted ink command comprises activating the button.

7. The device ofclaim 1 one wherein the displayed control is one or more electronic document reader pages,

and wherein the received ink input comprises a selected number of hand-drawn carets pointing in a selected direction,

and wherein operating the displayed control in accordance with the interpreted ink command comprises performing the selected number of page flips in the selected direction.

8. The device ofclaim 1 wherein the displayed control is a date picker control for setting a date, the control presenting a plurality of dates arranged in rows each corresponding to a week,

and wherein the received ink input is a containing shape that contains a selected date presented by the control,

and wherein operating the displayed control in accordance with the interpreted ink command comprises setting the selected date.

9. The device ofclaim 1 wherein the displayed control is a date picker control for setting a date, the control that present either (1) a plurality of dates arranged in rows each corresponding to a week, and/or (2) two or more controls for independently setting different components of a date,

and wherein the received ink input is text indicating a date,

and wherein operating the displayed control in accordance with the interpreted ink command comprises setting the indicated date.

10. The device ofclaim 1 wherein the displayed control is an enumerated textual string picker control for setting one of a plurality of the enumerated textual strings each comprised of a sequence of characters,

and wherein the received ink input is a handwritten sequence of characters matching the beginning of the character sequence of exactly one of the enumerated textual strings,

and wherein operating the displayed control in accordance with the interpreted ink input comprises setting the enumerated textual string the beginning of whose sequence matches the handwritten sequence of characters.

11. The device ofclaim 1 wherein the displayed control is a container control for selecting items in a container, the container control containing a visual representation of each of at least a portion of the items in the container,

and wherein the received ink input is an enclosing shape surrounding one or more of the item visual representations,

and wherein operating the displayed control in accordance with the interpreted ink input comprises selecting the items whose visual representations are surrounded by the enclosing shape.

12. The device ofclaim 11 wherein the displayed container control shows items in the container in a grid, list, or tree.

13. The device ofclaim 1 wherein the displayed control is a container control for selecting items in a container, the container control containing a visual representation of each of at least a portion of the items in the container, each of the items in the container having a textual string associated with it,

and wherein the received ink input is a handwritten textual string comprising a sequence of characters,

and wherein operating the displayed control in accordance with the interpreted ink input comprises subsetting the items for which the container control contains visual representations to only those items whose associated textual strings match the handwritten textual string.

14. The device ofclaim 1 wherein the displayed control is a container control containing a visual representation of each of at least a portion of the items in a source container, at least one of the item visual representations indicating that the represented item is selected,

and wherein the received ink input is an arrow beginning near one of the item visual representations that indicates that the represented item is selected, and ending near a visual indication of a target container,

and wherein operating the displayed control in accordance with the interpreted ink input comprises transferring the selected items from the source container to the target container.

15. The device ofclaim 1 wherein the displayed control is a container control containing a visual representation of each of at least a portion of the items in a source container, at least one of the item visual representations indicating that the represented item is selected,

and wherein the received ink input is an arrow beginning near one of the item visual representations that indicates that the represented item is selected, and ending near a visual indication of an operation that can be performed against items,

and wherein operating the displayed control in accordance with the interpreted ink input comprises subjecting the selected items to the indicated operation.

16. The device ofclaim 1 wherein the displayed control is a container control containing a visual representation of each of at least a portion of the items in the container,

and wherein the received ink input comprises a shape drawn over at least a portion of the representation of the selected item,

and wherein operating the displayed control in accordance with the interpreted ink input comprises deleting the selected item from the container.

17. The device ofclaim 16 wherein the shape comprised by the received ink input is a horizontal segment, a horizontal segment with a loop at one end, or a scribble.

18. The device ofclaim 1 wherein the displayed control is a container control for selecting items in a container, the container control containing a visual representation of each of at least a portion of the items in the container,

and wherein the received ink input comprises a check mark, a star shape, or an asterisk drawn near the representation of each of one or more distinguished items,

and wherein operating the displayed control in accordance with the interpreted ink input comprises selecting the distinguished items.

19. A computer-readable medium having contents configured to cause a computing system to:

cause to be displayed a control operable with mouse input;

receive an ink command directed to the displayed control; and

operate the displayed control in accordance with the received ink command.

20. A method in a computing system, the method comprising:

receiving ink input at a particular location relative to a display region;

based on the received ink input's shape and location, interpreting the received ink input as an ink command; and

performing the interpreted ink command.

Images