US20130106754A1 - Determining when a touch is processed as a mouse event - Google Patents

Abstract

A multi-touch driver handling user inputs or touches on a multi-touch input device. The method includes receiving touch data associated with a touch detected by the multi touch input device. The multi-touch driver determines a target user interface (UI) component for the touch by processing the touch data and then determines whether the target handles touch events. If so, the driver sends the touch data to the target for handling as a touch event, thereby saving a touch-to-mouse conversion for later touches. The touch may be a first touch received in a particular input region of a GUI displayed on a touch surface. The driver receives touch data for a second touch input later than the first touch but while the first touch is still active. The driver determines a target UI component for the second touch and passes the second touch data to the second target as a mouse event.

Description

RELATED MATTERS
• This application is a continuation of and claims the benefit of priority under 35 U.S.C. §120 to U.S. application No. 12/463,837, filed May 11, 2009, the entire contents of which are incorporated herein by reference.
BACKGROUND
• 1. Field
• The present disclosure relates, in general, to computers and other electronic devices with multi-touch sensitive display surfaces or screens, and, more particularly, to an improved method for determining whether a touch or contact with the multi-touch surface or screen is processed (e.g., treated or handled) as a touch event or as a mouse event.
• 2. Relevant Background
• Multi-touch surface computing is growing rapidly and is spreading quickly from large table computers to handheld electronic devices such as phones and personal music players. Multi-touch refers to a set of interaction techniques that allow computer users to control graphical applications with two or more fingers or pointers. Generally, multi-touch systems include a touch screen or surface and software that recognizes and processes multiple touch points on the surface rather than just a single touch point as may have been the case with typical touchpads.
• Multi-touch systems may use a wide variety of technologies to implement to touch screen or touch input device to determine where a touch occurs and to track a continued touch along a surface. For example, the touch surface may use heat, pressure, high capture rate cameras, infrared light, optic capture, tuned electromagnetic induction, laser rangefinders, shadow capture, and other techniques to provide touch or contact input information to the software or drivers (e.g., multi-touch drivers or the like) that determine points of contact and provide the user input to applications.
• Multi-touch displays surfaces are used by computer and electronic device designers to creatively present interfaces to one or more users such as graphical objects that may be selected, sized, moved, scrolled, and otherwise manipulated through the use of fingers or other physical contact devices (e.g., a pen, a stylus, or the like). Typically a computer renders a graphical user interface (GUI) on the multi-touch display surface or screen, and users may manipulate GUI objects or displayed elements directly with their hands using multi-touch technology rather than traditional input devices such as a mouse or a keyboard. In other words, multi-touch devices and systems merge the user input and data output devices into a single surface, which provides an intuitive and efficient mechanism for users to interact with a computer and applications running on the computer (e.g., to manipulate drawings, photographs, data files, and the like).
• A number of problems arise from the use of a multi-touch display surface. Often, the multi-touch technology is added onto an existing computing system and its applications may have been adapted for use with prior user input technologies. For example, a computer system or electronic device may have software applications in which all or portions of its GUI are configured to receive and process mouse events and mouse gestures such as to point at or select a displayed object in the GUI. In other cases, an application running on a multi-touch system or device may have one portion or area that receives and processes touch events (i.e., a non-mouse event application or multi-touch application portion) and other portions or areas, such as a menu or control interface, that is responsive to mouse events but not touch events.
• By default, current multi-touch systems have software, such as a multi-touch driver, that treats a first touch or contact by the user on the multi-touch surface as a mouse event and second and later touches are treated as touch events until the first touch is release or removed from the surface (e.g., the mouse adapter becomes available). As a result, portions of the GUI and/or portions of applications may not be able to receive user input or to interface with the users. In an exemplary multi-touch system, a first touch within an interactive graphics portion of a GUI may be processed or handled as a mouse event and block a concurrent second touch from being able to interact or use a portion of the GUI corresponding to a non-multi-touch application, which requires a mouse event. This is particularly problematic when multi-touch display technology is implemented in a system with legacy or existing software or applications as these will often include applications that have one or more windows or portions in the GUI that not configured to handle touches or touch events.
SUMMARY
• Briefly, a touch handling method is described that may be implemented in a multi-touch driver to selectively handle or treat touch events as mouse events. In many multi-touch systems, a graphical user interface ((UI) is presented upon the touch input surface or device. The GUI may include one or more regions or areas that are associated with multi-touch (MT) aware UI component, and the GUI may also include one or more regions or areas that are associated with non-MT aware UI components (i.e., applications or UI components of applications that expect and handle mouse-based inputs or events, keyboard inputs, and so on). In prior multi-touch systems, a first touch at any location within the GUI, even within the MT aware region, was translated into a mouse event and a mouse adapter (or touch-to-mouse conversion) was associated with the first touch and unavailable to second or later touches. This often resulted in regions of the GUI that were associated with non-MT aware UI components being disabled or unable to accept user input until the mouse adapter was released or available (e.g., the first touch was ended).
• In contrast, the touch handling method described herein includes receiving a touch and determining a target application (or UI component) for the touch based on location/geography, on application mode, or the like. The method also includes determining whether the target application handles or accepts touches or touch events, and, if so, the received or sensed touch is handled as a touch event with the mouse conversion or adapter being saved for use by other portions of the GUI or UI components associated with regions or areas of the MT surface or screen. When the target does not accept/handle touches, the method may include determining whether the target accepts mouse events and, if so, whether the mouse adapter or touch-to-mouse conversion is available. When both are found to be true or satisfied, the method may further include marking the adapter or conversion as unavailable, translate the touch to a mouse event, and pass the mouse event to the target for further processing as user input. While the touch is maintained, updates are also sent as mouse events to the target(s). When the touch is ended (e.g., a finger is lifted away from the surface or the like), the mouse adapter or touch-to-mouse conversion may again be made available for use in handling later touches received by the multi-touch surface or input device. In some cases, the method or its associated functionality may be provided on a per-application basis in addition to a per-system basis, e.g., each application may have its own touch-to-mouse adapter.
• More particularly, a method is provided for handling or processing user input touches for use with a multi-touch input device such as a computing or electronic device with a multi-touch surface or screen. The method includes running a multi-touch driver with a processor, and the multi-touch driver may include a touch event processing module configured to provide many of the functions of the method. The method further may include receiving touch data associated with a user touch or input event detected by the multi-touch input device. The multi-touch driver may be used to determine a target user interface (UI) component for the touch or received touch data by processing this data (e.g., by comparing location or geometry information with regions or areas of the touch surface associated with UI components, by examining application mode data, and so on). The multi-touch driver may further determine whether the target UI component handles touch events, and, if so, sending the touch data to the target user interface for handling as a touch event, which may include translating the touch into a touch event. In some cases, parts or portions of the multi-touch driver may reside on the touch device itself While in other implementations parts or portions may be provided on the host system and/or be part of applications as well.
• In the method, the touch may be a first touch received in a particular input region or area of a GUI displayed on the touch surface of the multi-touch input device, whereby the first touch is not by default processed or handled (with these terms used interchangeably in this description) as a mouse event. In such a case, the method may include receiving touch data for a second touch that is received later in time than the first touch but while the first touch is maintained (or not yet ended/released). The method may include determining a target UI component for the second touch and passing the second touch data to the target UI component for handling as a mouse event. In this manner, a second touch may be processed or handled as a mouse event.
• In some embodiments of the method, the target UI component may be determined to not handle touch events. The method may then involve determining whether the target UI component handles mouse events, and, if so, sending the touch data to the target UI component for handling as a mouse event (e.g., translating the touch into a mouse event and passing to the appropriate application or UI component of the application). The method may include determining whether a touch-to-mouse conversion module or mouse adapter is available prior to sending the touch data to the target UI component as a mouse event. If available, the method may include marking the conversion module is unavailable to later received touches or to other UI components (until the current touch is ended/released). The touch data may be received after the multi-touch driver has processed or begun processing of an earlier touch associated with another target UI component, which is adapted for handling or accepting touch events. In this manner, a second or later-in-time touch may be processed as a mouse event when one or more prior touches have been received (and maintained or are concurrent/active touches) that are being processed as touch events (e.g., by MT aware UI components). In some cases, the method is used to retrofit system to provide a technique for getting a new behavior even though the underlying system is providing the old style or convention of first-only event conversion.
• According to another aspect or embodiment, a computer readable medium is provided for handling touch events on a multi-touch surface. For example, the computer readable medium may include computer useable and/or readable code stored in a computer memory or in storage medium readable by a processor of a computer (e.g., a multi-touch driver program run by a CPU, a routine used or called by a multi-touch driver when run by a computing or electronic device receiving contact or touch data from a MT surface or device, or the like). The computer readable medium may include computer readable program code devices configured to cause a computer to receive first touch data associated with a first touch provided at a first time in a first input region of a graphical user interface displayed on a multi-touch surface. The computer readable medium may also include computer readable program code devices configured to cause the computer to receive second touch data associated with a second touch provided at a second time later than the first time in a second input region of the graphical user interface. Further, the computer readable medium may include computer readable program code devices configured to cause the computer to handle the first touch data as a touch event and the second touch data as a mouse event, whereby the first touch may be maintained or active at the second time (e.g., the second touch is received while the first touch is still being updated or processed by the first UI component).
• In some embodiments, the first input region corresponds to a first user input component adapted for accepting touch events and the second input region corresponds to a second user input component adapted for accepting mouse events and not accepting touch events. The computer readable medium may also include computer readable program code devices configured to cause the computer to determine the first user input component is a target of the first touch and to determine the second user input component is a target of the second touch. In some cases, the computer readable medium includes computer readable program code devices configured to cause the computer to determine whether the first user input component accepts touch events prior to handling the first touch data as a touch event. Yet further, the computer readable medium may include computer readable program code devices configured to cause the computer to determine whether the second user input component accepts touch events and, when touch event are not accepted, performing the handling of the second touch data as a mouse event. Optionally, the computer readable medium may include computer readable program code devices configured to cause the computer, prior to handling the second touch as a mouse event, to determine that a touch-to-mouse conversion module is available for use by the second user input component. Note, also, that the methods described herein and the medium/software providing the functions described herein may be provided as middleware, as part of an interpretation environment, and/or as part of a virtual machine.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 illustrates in functional block form a computer or computer system including a multi touch (MT) surface, screen, or input device and showing use of a multi-touch driver that is configured to handle user inputs (e.g., contacts or touches) as mouse or touch events in a selective manner;
• FIG. 2 illustrates in functional block form another multi-touch system or device illustrating state and/or data that may be processed to process detected/received touch events (also termed point or contact events and touches (which may also include proximity inputs depending upon the multi touch technology used to detect or sense “touches” or inputs from a user) as either mouse or touch events that are provided as inputs to UI components or portions of one or more running applications;
• FIGS. 3 is a flow chart of a touch (or contact, point, or the like) handling process for use with a multi-touch surface, screen, or display device and as may be implemented, for example, by operation of the system ofFIG. 1 or the system ofFIG. 2;
• FIG. 4 illustrates a multi-touch input display surface during use to display an application GUI with two interface or input regions or areas (e.g., one that corresponds to a multi-touch (MT) aware component or portion of the application and one that corresponds to a non-MT aware component or portion of the application, such as a legacy interface or the like);
• FIG. 5 illustrates a conventional or prior art multi-touch device in which touches to a non-MT aware area or input region are only allowed or processed when they occur prior to touches to a MT aware area or input region (or when no other touches are occurring); and
• FIG. 6 illustrates a multi-touch device using a user input or touch processing module described herein such that a touch or user input to a non-MT aware area or region is allowed or processed even when occurring after a prior, concurrent (or simultaneous) touch or user input in a MT aware area or input region (e.g., the user input or “touch” in a MT aware region of the multi-touch surface is sent to the application as a touch event and a touch-to-mouse converter or adapter is kept as available for a later user input or “touch” in a non-MT aware region of the multi-touch surface).
DETAILED DESCRIPTION
• The following description describes use of software (and/or hardware) implementations of user input handler or processor for use with a multi-touch system or device to more effectively handle a first user input on a multi-touch surface or screen. For example, prior multi-touch system's typically used a multi-touch driver that would treat the first user touch as a mouse event in all cases, which caused a touch-to-mouse event converter or adapter to be unavailable for later touches until the first touch was ended. The user input handler or mouse touch processing module described herein may, for example, be implemented in a multi-touch driver of a computer or electronic device that is running an application that has first and second user interface regions or windows on a multi-touch surface or user input device. The multi-touch driver functions to receive user input (e.g., a user's first touch, point (e.g., proximal but not always requiring contact), or contact (e.g., contact with an input device with a stylus, pen, or the like) of a multi-touch screen—all of which may be considered “touches” in this description), and the multi-touch driver maps the coordinates or location of the user input or touch to surface areas or regions corresponding to the first and second applications.
• The first application may accept only mouse touch events while the second application may accept multi-touch events or touch events (or simply a touch). If the received user input is mapped to the first application interface region, the multi-touch driver sends the input on to the first application as a mouse event (and marks touch-to-mouse conversion as unavailable). If the received user input is mapped to the second application interface region, the multi-touch driver sends the input on to the second application as a touch event (not as a mouse event), which reserves the touch-to-mouse conversion or adapter for use by a next user input or touch. In this mariner, the first application may receive user input processed as mouse events even if it is not the first to receive a user input or touch, whereas in prior multi-touch systems the first application would have had wait idle until a first touch (or the touch-to-mouse conversion) again became available. In this manner, concurrent or simultaneous interfacing with more applications or components of an application is supported by the touch processing methods and techniques (and multi-touch and computer systems/devices incorporating such methods) described in herein with reference toFIGS. 1-6.
• FIG. 1 illustrates a computer or computer system (or electronic device system)100 with a multi-touch surface orinput device110 that is coupled to a computer/electronic device140 via power and/orcommunication links130. Themulti-touch surface110 is configured with a multi-touch (MT)input region114 in Which an application(s) graphical user interface (GUI)112 may be displayed via data transferred to and from thecomputer device140. For example,computer device140 may include a processor orCPU142 that runs one ormore applications144 that generate aGUI112 that displays data, graphics, images, and the like on theMT input region110 of the multi-touch surface ordevice110. Theapplication144 may include one ormore components146 that are legacy or other components that are not adapted for receiving touch events but instead expect and process mouse events (e.g., non-MT aware components), and theapplication144 may also include one or more MTaware components148 that are able to and adapted for receiving and processing touches or touch events provided via themulti-touch surface110.
• As shown, during use of thesystem100, theapplication GUI112 includes a multi-touch (MT)input region114 corresponding to the MT aware component(s)148 ofapplication144 and anon-MT input region120 corresponding to the non-MT aware component(s)146. A user may use theirhand116 or an input implement118 to provide user input (e.g., a touch) at location orpoint119 in theMT input region114, Themulti-touch surface110 may act to sense this input and transfer touch or contact information vialink130 to thecomputer device140. Theprocessor142 may manage operation ofmemory160 and act to store the received contact/input data162. Amulti-touch driver150 may be provided on the computer/electronic device140 and be adapted to separate theinput data162 intotouch event data164 and mouse event data168 (both of which may at least temporarily be stored inmemory160 of the computer140). Thetouch event data164 andmouse event data168 may include location information and/or identification information163 (e.g., a touch location and ID to uniquely identify the touch or input from the user on the MI' surface110).
• A user point/contact handler152 (e.g., a software routine or module run or executed by processor142) may be provided in themulti-touch driver150 to process the user input (e.g., a touch or pointing at location119) to determine whether the input or touch is associated with the MTaware component148. If so determined, such as by mapping of the location or coordinates of thelocation119 to theMT input region114 ofGUI112, the input or touch is passed to the MTaware component148 as a touch event (e.g., as a touch event only) and the touch-to-mouse adapter orconversion module156 is maintained as available for other applications/components to use. The same or a different user may use ahand122 to provide additional user input or another touch, such as with theirfinger124, at a second contact point orlocation125. This location orpoint125 is in aninput region120 of the GUI112 (or different area of the multi-touch surface110) that is associated with the non-MTaware components146 of theapplication144.
• In this case, contact input data1.62 is received by the computer140 (over thelink130 from the MT surface or device110) and processed by the user point/contact handler152 of theMT driver150. The location orpoint125 has coordinates that map to thenon-MT input region120 and non-MTaware component146, and, hence, thehandler152 causes thedriver150 to classify theinput data162 as mouse event data168 (e.g., a touch location and ID). The touch is passed to the non-MTaware component146 as a mouse event by theprocessor142 and the touch-to-mouse adapter orconversion module156 is associated with thecomponent146 and marked as unavailable for use by other applications. Additional inputs or touches may be received while the input atlocation125 or inregion120 is maintained if the inputs are provided in the MT input region114 (e.g., additional touch events may be processed concurrently with the contact at point125). If an additional input is provided inregion120, however, the touch would not be forwarded to the non-MT aware components until after the contact or touch initiated atlocation125 is removed or released. In thesystem100, the location information for the contact points119,125 and other data such as time received may be determined by the multi-touch surface orinput device110 and its hardware/software and/or it may be provided as raw data overlink130 toprocessor142 for processing byMT driver150 or other components (not shown) to produce the information used by the user point/contact handler152 in determining whether to treat the input as a touch event or as a mouse event (as described herein such as with reference toFIGS. 3-6).
• The system100 (andsystem200 ofFIG. 2) may utilize a wide variety of technologies for themulti-touch surface110. Multi-touch (MT) is intended to denote nearly any technology and implementing device that allows interaction between a user and running software/applications using two or more input devices to “touch” a graphical user interface (GUI). For example, the MT device may include a touch screen or touchpad as well as software that recognizes multiple simultaneous touch points. The ability to receive or sense these inputs may be achieved by use of sensors or detection devices based on heat, pressure, cameras, infrared light, optic capture, tuned electromagnetic induction, ultrasonic receivers, transducer microphones, laser rangefinders, shadow capture, or another technology for detecting receipt of a touch and its location on a MT surface or screen. A “touch” in this description refers, therefore, to a user input event that is detected or sensed by a MT technology and transmitted with location information to a computer (such as computer140) for use as input to an application via its interface (displayed on the MT surface). For example, a touch may also be sensed or detected by mechanical actuators, strain gauges, and the like.
• Likewise, the computer or electronic device140 (and230 inFIG. 2) may take numerous forms to practice the invention. Generally, thedevices140,230 may be any electronic device with a processor running one or more applications presenting user interfaces via a MT surface or screen and software for processing the touches such as one or more multi-touch drivers adapted to perform as described herein. For example, thedevices140,230 may be a surface computer, a desktop or workstation computer, a laptop, notebook, handheld, mini, or other portable computer, a wireless communication device such as a cellular phone, personal digital assistant (PDA), or the like, a video game system (portable or home unit), and so on.
• Embodiments described herein ofsystems100,200 and the like (e.g., computer architectures running multi-touch drivers functioning according to one or more embodiments of the invention) may include or utilize a special purpose or general-purpose computer including computer hardware. Embodiments may include physical and other computer-readable media for carrying or storing computer-executable instructions and/or data structures (such as the applications, drivers, and data stored in memory shown insystems100,200 and the like). The computer-readable media may be any available media that can be accessed by a general purpose or special purpose computer system. For example, computer-readable media may include media that stores computer-executable instructions or physical storage media. Physical storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store desired program code in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.
• The computer or electronic devices typically are linked or coupled closely to the MT surface (and its hardware/software), but, in some embodiments, the MT surface may be linked to the computer by a digital communication network or the applications, memory, drivers, or the like may be accessed by the computer or electronic device over a network. A “network” may include one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices, The network and/or data links may also be used to carry program code in the form of computer-executable instructions or data structures, which may be accessed by a general purpose or special purpose computer. Combinations of the above should also be included within the scope of computer-readable media. Computer-executable instructions may include, for example, instructions and data which cause a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. The computer executable instructions may be, for example, binaries, intermediate format instructions such as assembly language, or even source code.
• FIG. 2 illustrates a computer system orelectronic device200 illustrating in a functional block manner a multi-touch technique for more effectively handling touches or user inputs to utilize non-MT aware (or legacy) interfaces. Specifically,system200 is adapted with amultitouch driver250 that functions with the use, in this example, of a MT event processing module252 to process touches or user inputs on a MT surface or input device to classify the touches as touch events or mouse events and then to pass the classified user input events to the corresponding application or UI components of an application(s). In this manner, the first user input or touch is not by default assumed to be a mouse event but each touch is instead processed to determine if the touch can be handled or processed by a MT aware component prior to attempting to handle or process the input as a mouse event. This allows mouse events to be generated on second, third, or later of a set of concurrent touches on a MT surface.
• As shown, thecomputer system200 includes atouch input device210, Which may be part of computing device230 (e.g., mounted on or as integral part of a computer housing such as in a MT computer surface/table or the like) or external to the computing device230 (e.g., spaced apart and communicatively coupled to the device230). Thetouch input device210 includes one ormore sensors212 that operated to sense touches to theinput device210, and aprocessor214 may be included to interpret the data from the sensor(s)212 such to determine whether a touch has occurred, whether a touch has been maintained or released, a location or point of contact for a touch relative to a surface/screen of theinput device210, whether a touch location has been moved on the screen, and the like. Theprocessor214 sends this contact or touch input data to abus interface216, which writes the data tocommunication bus220 that links or couples thetouch input device210 to thecomputing device230. The communicated data often will include a touch ID as well as location data (such as X-Y coordinates on the screen or surface) for each detected or sensed touch or user input.
• Thecomputing device230 uses abus interface232 to read data from thecommunications bus220 including the contact or touch input data from theprocessor214. Thecomputing device230 includes one or more processors orCPUs234 that run one ormore applications240 as shown with a runtime242 running code246 (Which, in turn uses an application framework248). Theapplication240 is shown to includelegacy UT components255, which are not configured for touch events but instead expect and process mouse events254 (or keyboard events/input (not shown)), and theapplication240 also include MTaware UT components257 that expect and handletouch events256. Theevents254,256 are provided as user input via GUI generated by theapplication240 duringruntime242 and displayed on thetouch input device210 to a user of thesystem200.
• Theprocessing unit234 also runs an operating system (OS)238 and a device driver acts to read user input or touch data from thebus interface232, which it passes to theOS238. TheOS238 processes the touch data and provides the data or user input/contact event data to a multitouch driver250 (shown here to be run by theprocessing unit234 as part of theapplication240 runtime242). Themultitouch driver250 includes an MT event processing module252 that is configured to process the touch data and/or user input/contact event data to determine whether to handle these inputs/touches asmouse events254 or as touch events256 (and to determine whichUI component255,257 to provide the events to). In practice, this may be accomplished by employing themethod300 shown inFIG. 3. Briefly, though, the MT event processing module252 may determine whether the touch or input corresponds to a MTaware UT component257 or to a non-MT orlegacy UI component255. If to a MTaware UI component257, then the touch or input is characterized or classified as a touch event and passed as atouch event256 to the appropriate one of the MTaware UI components257. If to a non-MT aware orlegacy UI component255, then thedriver250 characterized or classifies the touch or input as amouse event254 and passes it as such to a corresponding one of the legacy UI components255 (e.g., the component with an input region on the GUI displayed on theinput device210 that includes the coordinates or location of the touch) for further processing. Often, asystem200 may only support one mouse event at a time, and, in such cases, a touch-to-mouse conversion may be marked as not available upon or prior to generation of themouse event254 to the non-MTaware UI component255, and the touch-to-mouse conversion module or adapter may be released or marked as available upon release of user input (e.g., upon receipt of touch data indicating a user has ended a touch associated with the mouse event),
• FIG. 3 illustrates atouch handling process300 as may be implemented by theMT drivers150,250 ofFIGS. 1 and 2. Theprocess300 starts at305 such as by a multi-touch surface or touch input device displaying a GUI with regions or windows associated with two or more applications or components of one or more applications, and these applications or components may have user interfaces that are MT aware or that are not adapted for receiving/processing touches (i.e., a non-MT aware interfaces or application components). At310, the multi-touch surface or input device may receive a touch or user input (with a finger, hand, or other input device), and touch or contact data such as a touch ID and location may be passed to a computing device for processing by its multi-touch driver or other mechanism.
• When it is determined at310 that a touch or input has been received, themethod300 continues at320 with determining with the multi-touch driver a target application, and if a touch is not received themethod300 continues at310. As part ofstep320, thedriver150 may access the contact or touch data to obtain a touch location (e.g., X-Y coordinates, by other geometry processes, or the like) and compare this with a set of application component display areas, which may define areas or regions of the MT surface or screen associated with the user interface of each application or application component presently accessible via a displayed GUI. In some cases, though, the touch or contact data may include a target application due to prior processing by the MT device and/or computing device components. Once a target application for a touch is determined or identified at320, themethod300 may continue at324 with the MT driver determining whether the target accepts touches or is a MT aware UI component.
• If the touch corresponds to a MT aware target, the MT driver acts at328 to send the touch (or touch data) to the target application or target UI component as a touch event. When updates are detected for this particular touch, the updates are also sent as touches or touch events to a target or corresponding UI component. For example, a user may touch a MT screen at a first location and drag their finger to a second location, and the location update information would be sent as a touch event (associated with the same touch event such as by touch ID) to the MT aware UI component for further processing. At334, a detected end or release of a touch is also sent as a touch or touch event to the MT aware UI component, and themethod300 continues at310 with waiting for a next or an additional touch or contact from a user on the MT surface or screen.
• At324, when the MT driver determines that the target application or target UI component does not accept touches (or is non-MT aware as may be the case for a legacy UI component), the driver acts at340 to determine whether the target accepts mouse events and whether the touch-to-mouse adapter or conversion module is available. If not, which may be the case if a touch has previously been converted to a mouse event, themethod300 continues at310 with waiting for a next or additional user input or touch on the MT screen or surface. If mouse aware and conversion available, themethod300 continues at346 with MT driver marking touch-to-mouse conversion as unavailable. At350, the touch or user input is processed such that its record in memory (e.g., theevent data162,164,168 shown inFIG. 1) is updated to note that touch associated with the current ID has been converted into a mouse event (e.g., the ID of the converted touch may be changed to a mouse-touch ID or the like). At360, the touch is sent as a mouse event (with location information) to the target application or target UI component for further processing/use.Method300 may also include at358 converting the touch into a mouse-down event that is sent to the target application.
• At370, themethod300 may include determining whether the mouse-event touch has been maintained, and when it has, themethod300 may include converting updates to mouse events that are then sent to the target application or UI component. For example, a mouse-down event may act to select a graphical object in a legacy portion or region of GUI, and a user may act to manipulate the selected object with mouse updates, e.g., to drag the selected object to a new location, to rotate/position the object, to size the object, and other mouse-based gestures or interactions. When the touch is not maintained on the MT surface, the end of touch is converted to a mouse-up event at380 by the MT driver and sent to the target application or UI component as a mouse event. At384, themethod300 continues with marking the conversion or touch-to-mouse adapter as available, and themethod300 may again wait for additional touches at310.
• Note, themethod300 is intended to handle more than one simultaneous or at least concurrently applied touches, and the steps shown for themethod300 may be performed for two or more touches at the same time. For example, two touches may be received simultaneously and both be processed as shown inFIG. 3 to determine whether to handle one or neither of the touches as a mouse event. In another case, a first touch may be received at a first time at a first location in a region associated with a first UI component andprocess300 may be initiated for this touch, and a second touch may be received at a second time (later than the first time) at a second location in the region associated with the first UI component or a different region associated with a second UI component. This second touch would be processed concurrently with the first touch (but the steps ofmethod300 may be offset in time, for example). Additionally, third, fourth, fifth, and more touches may be received concurrently and sequentially on a MT surface and processed by themethod300 implemented by one or more MT drivers as described herein. In this manner, each touch is processed to determine whether it was received in a non-MT aware region that handles mouse events and, if so, whether the touch-to-mouse conversion is available for use by this touch. A touch occurring in a MT aware region of a GUI or MT surface will not cause the conversion to be marked as unavailable, which allows a GUI with non-MT aware components to be more effectively used in a MT-based system or device.
• FIG. 4 illustrates an exemplary multi-touchinput display surface400 such as may be provided in a MT device or system as shown inFIGS. 1 and 2 and that may be used during implementation ofmethod300 ofFIG. 3. As shown, anapplication GUI410 is displayed within or via thedisplay surface400. Theapplication GUI410 may be generated when a computing device associated with thedisplay surface400 runs one or more applications by a processor. The application(s) associated with theGUI410 may be an application that includes components that are adapted specifically for use with touch devices or display surfaces (such as surface400), and these components may generate user interface regions orwindows430 that may be considered multitouch-aware portions, regions, orcanvas430. In contrast, though, the application associated with theapplication GUI410 may also include one or more UI components that are not adapted for touch inputs but instead expect more conventional inputs such as keyboard inputs or mouse events. Such UI components may generate alegacy interface420 that accepts mouse events (and keyboard events), which may also be called a non-MT aware interface portion or region of theGUI410.
• In a typical implementation, the MTaware portion430 may be associated with interface graphics portion of the application such as may be used to display photographic images, models, animated video images, and so on that may a user may select, move, size, or otherwise manipulate with one, two, or more fingers (or other input devices). The non-MTaware interface portion420 may, in contrast, be a controller or control portion (such as a menu) that may include buttons, labels, menus, popup menus, scrollbars, sliders, and other input objects more typically operated based on mouse gestures. Such alegacy interface420 may be built from parts owned by parties other than those that developed the MT aware portion430 (e.g., UI widgets and the like), may be written or created prior to multitouch technologies were available (but this is, of course, not necessary to practice the invention), and may be expensive or even nearly impossible/impractical to upgrade to be multitouch aware.
• In use with a multi-touch driver (and/or a user touch/contact handler) as described herein, theapplication GUI410 is displayed on the MTinput display surface400. A first contact on a first point orlocation432 in the MTaware portion430 will be processed and handled as a touch event because the driver determines that the UI component or portion of the application associated with theGUI410 can handle touches or touch events. In such a case, the touch-to-mouse conversion is still available such that a second contact, point, or touch by a user at a second point orlocation422 in thelegacy interface420 results in the touch being handled or processed as a mouse event. This also would occur if the touch at location orpoint422 were the first-in-time touch received in theGUI410. Likewise, if two, three, or more touches occur inportion430 prior to a touch ininterface portion420, the touch at location orpoint422 will still be processed as a mouse event as the touch-to-mouse conversion remains available as touches to the MTaware portion430 do not cause the conversion to become unavailable (as is the case of a touch in interface or non-MT aware portion or region420). Hence,interface420 may be interacted with by a user even when it is not touched or contacted first by a user, with a first touch in this region (or another non-MT aware region) of theGUI410 causing the touch-to-mouse conversion module or adapter to be marked as unavailable.
• FIG. 5 illustrates a conventional multi-touch system ordevice500 with amulti-touch surface510 for receiving user inputs as one, two, or more touches that are sensed by an MT technology to provide contact data (e.g., location of a touch on the surface510). As shown, thesurface510 may include or provide a legacy UI region orportion520 that displays selectable objects and the like522, and thisportion520 is provided by a non-MT aware UI component of one or more applications run by thesystem500. Thesurface510 also displays or provides a touchaware portion530 of the GUI that displays interactive graphics, objects, and so on532, and thisportion530 corresponds to (or is generated by) a MT aware UI component of one or more applications run by thesystem500.
• In this conventionalmulti-touch system500, a first touch is by default processed as a mouse event regardless of where on thesurface510 it occurs. Hence, when auser550 uses a finger552 (or other input device) to touch the MTaware portion530 of the GUI at point orlocation554 at a first time, t_MT, thesystem500 determines that this is a first-in-time touch and translates the touch event into a mouse event. The touch-to-mouse conversion is, hence, unavailable while the contact withfinger552 is maintained, e.g., if theuser550 continues to manipulate theobject532 by dragging it or the like, additional mouse events cannot occur in thesystem500. If auser540 uses their finger542 (or other input device) to contact or touch anobject522 or other point orlocation544 in non-MTaware region520, the touch can only be processed and translated by thesystem500 into a mouse event when it occurs at a time, t, prior to time, t_MT, of the touch in the MT-aware region530 (or after the touch in theregion530 has been ended/released). The handling of touch events such that a first (and only first) touch event is translated into a mouse event allows compatibility of touch UI components with legacy or non-MT aware UI components, but it produces the undesirable result that the legacy or non-MT aware UI components often may be unavailable or non-interactive (even though they are being displayed in the surface510) as the MT-aware UI portion orregion530 has received the first touch and is blocking other touches as being processed as mouse events.
• FIG. 6 illustrates a multi-touch system or device (such as a surface computing table or the like)600 that may be adapted as shown insystems100 and200 to handle touches as described in themethod300 ofFIG. 3 and elsewhere herein. Themulti-touch system600 includes an MT input device orsurface610 that is adapted to sense touches by users of thesystem600 to allow the user to provide input using their hands (as shown) or other “touch” input devices. The input device orsurface610 may provide a touch location and ID or other contact information to a multi-touch driver (or UI component of an application) ofsystem600. As with thesystem500, one or more applications run bysystem600 act to create a GUI onsurface610 with a MT aware region orportion630, which corresponds to a touch aware UI component, and with a non-MTaware region620, which corresponds to a non-MT aware UI component (e.g., a UI component that only recognizes keyboard, mouse, and similar inputs or events).
• However, in contrast, a touch insystem600 may be handled as or translated into a mouse event even when it is not a first touch. In other words, thesystem600 is adapted such that a first touch in the non-MTaware region620 of the GUI ofsurface610 is translated into a mouse event while touches in the MTaware region630 of GUI ofsurface610 are handled or processed as touch events (e.g., touch to mouse conversion remains available no matter how many touches occur in region630). In operation ofsystem600, for example, auser640 may use their finger (or other device)642 to provide a touch at location orpoint644 at a first time, t₁(such as to select an object to be added to displayed object632), The user may maintain this touch from the first time, t₁, to a second time, t₂, and on to a third time, t₃, as shown at641 withfinger643 providing a touch at location orpoint645. This may occur when a user selects an object inregion630 and drags646 it to a new location/position as is common in many GUI applications. Updates inregion630 will be passed as touches during thedrag646. Similarly, additional touches may be made inregion630 by the same or other users, and each of these additional touches would be translated into touch events.
• Without the described touch handling processes, thesystem600 would not be able to handle touches in non-MTaware region620 from the first time, t₁, to the third time, t₃. However, in thesystem600, a user650 may use their finger orother input device652 to provide a user input or touch at a location orpoint654 in the non-MT aware region at a time that is simultaneous with or after the first time, t₁, and the touch will still be properly handled or processed as a mouse event. In the illustrated example, a user650 uses afinger652 to provide a touch at location orpoint654 in non-MTaware region620, which coincides with anobject622. The touch atlocation654 occurs at the second time, t₂(Le., while thedrag646 is occurring in the MT-aware region630 of the GUI on surface610).
• The system600 (e.g., a MT driver configured as described herein) determines that the UI component associated with the region orarea620 of the GUI does not accept touches but does accept mouse events and also determines that the touch-to-mouse conversion is available, as it has not been used or marked unavailable by the touches in the MT-aware region orarea630. As a result, the touch or click atlocation654 on thebutton622 at the second time, t₂, which is after a first touch in the MT-aware region orarea630, succeeds, and the user650 is able to interact with the legacy UI component associated with the non-MT aware region orarea620 concurrently with interact with the MT-aware region orarea630.
• Although the invention has been described and illustrated with a certain degree of particularity, it is understood that the present disclosure has been made only by way of example and that numerous changes in the combination and arrangement of parts can be resorted to by those skilled in the art without departing from the spirit and scope of the invention, as hereinafter claimed. As will be understood from the above description, the described processes and systems enable multi-handed interaction for a system with a control panel that includes legacy components and a canvas that may consume touch events. For example, this enables a system where the user may work on the canvas and activate action modifiers by pressing virtual buttons on the input surface (such asarea620 ofFIG. 6), The user is not limited to simple “buttons” but may use any legacy control, such as but not limited to sliders, color pickers, and the like. The user may use such legacy controls without having to remove their hand from the canvas area of the system or its multi-touch display.

Claims (20)

What is claimed is:

1. A method comprising:

identifying a user interface that corresponds to a touch event detected on an input surface that displays the user interface;

determining, using at least one processor, that the user interface does not accept touch events and that a touch-to-mouse adapter is available; and

providing the user interface with a mouse event based on the determining that the user interface does not accept touch events and that the touch-to mouse-adapter is available,

the mouse event being generated from the touch event by the touch-to-mouse adapter.

2. The method ofclaim 1 wherein:

the identifying of the user interface is based on a location of the touch event being within a display area that displays the user interface on the input surface.

3. The method ofclaim 1, wherein:

the identifying of the user interface includes comparing a location that corresponds to the touch event with a display area in which the user interface is displayed on the input surface.

4. The method ofclaim 1, wherein:

the determining that the user interface does not accept touch events and that the touch-to-mouse adapter is available includes determining that the user interface accepts mouse events.

5. The method ofclaim 1 further comprising:

generating the mouse event by processing the touch event with the touch-to-mouse adapter determined to be available.

6. The method ofclaim 5, wherein:

the processing of the touch event includes converting the touch into a mouse-down event.

7. The method ofclaim 1 further comprising:

marking the touch-to-mouse adapter as being unavailable.

8. The method ofclaim 7, wherein:

the marking of the touch-to-mouse adapter as being unavailable is in response to the determining that the user interface does not accept touch events.

9. The method ofclaim 1, wherein:

the touch event is a first touch event that indicates a start of a touch on the input surface;

the mouse event is a mouse-down event; and

the method further comprises:

detecting a second touch event that indicates an end of the touch; and

providing the user interface with a mouse-up event generated from the second touch event by the touch-to-mouse adapter.

10. The method ofclaim 9 further comprising:

detecting a third touch event that indicates the touch is maintained on the input surface;

providing the user interface with a mouse update generated from the third touch event by the touch-to-mouse adapter.

11. The method ofclaim 9 further comprising:

marking the touch-to-mouse adapter as being available in response to the providing of the mouse-up event.

12. The method ofclaim 1, wherein:

the touch event is a second touch event that indicates a second touch on the input surface while a first touch indicated by a first touch event is maintained on the input surface.

13. The method ofclaim 12 further comprising:

detecting the first touch event that indicates maintenance of the first touch on the input surface; and

identifying a further user interface that corresponds to the first touch event;

determining that the further user interface accepts touch events; and

providing the further user interface with the first touch event based on the determining that the further user interface accepts touch events.

14. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

identifying a user interface that corresponds to a touch event detected on an input surface that displays the user interface;

determining that the user interface does not accept touch events and that a touch-to-mouse adapter is available; and

providing the user interface with a mouse event based on the determining that the user interface does not accept touch events and that the touch-to mouse-adapter is available,

the mouse event being generated from the touch event by e touch-to-mouse adapter.

15. The non-transitory machine-readable storage medium ofclaim 14, wherein the operations further comprise:

generating the mouse event by processing the touch event with the touch-to-mouse adapter determined to be available.

16. The non-transitory machine readable storage medium ofclaim 14, wherein the operations further comprise:

marking the touch-to-mouse adapter as being unavailable in response to the determining that the user interface does not accept touch events.

17. A system comprising:

a device with an input surface configurable to display a user interface and detect a touch event that corresponds to the user interface; and

a processor configured to:

identify the user interface that corresponds to the touch event detected on the input surface;

determine that the user interface does not accept touch events and that a touch-to-mouse adapter is available; and

provide the user interface with a mouse event based on the determining that the user interface does not accept touch events and that the touch-to mouse-adapter is available,

the mouse event being generated from the touch event by the touch-to-mouse adapter.

18. The system ofclaim 17, wherein:

the processor, in determining that the user interface does not accept touch events and that the touch-to-mouse adapter is available, is configured to determine that the user interface accepts mouse events.

19. The system ofclaim 17, wherein:

the processor is configured to generate the mouse event by processing the touch event with the touch-to-mouse adapter determined to be available.

20. The system ofclaim 17, wherein:

the processor is configured to mark the touch-to-mouse adapter as being unavailable in response to a determination that the user interface does not accept touch events.

Images