US20100020221A1 - Camera Interface in a Portable Handheld Electronic Device - Google Patents

Abstract

In accordance with some embodiments, a method is performed at a handheld electronic device having a built-in digital camera and a touch sensitive screen. The method includes detecting a multi-finger gesture on the touch sensitive screen, wherein the touch sensitive screen is serving as part of an electronic viewfinder of the camera; storing coordinates of a location corresponding to the detected multi-finger gesture; translating the stored coordinates to a selected area of an image that is captured by the camera and that is being displayed on the touch sensitive screen; contracting or expanding the selected area in response to the user's fingers undergoing a pinching movement or a spreading movement, respectively, while the detected multi-finger gesture remains in contact with the touch sensitive screen; and applying an automatic image capture parameter adjustment process that gives priority to the selected area.

Description

RELATED APPLICATION
• This application claims priority to U.S. Provisional Patent App. No. 61/083,455, “Camera Interface in a Portable Handheld Electronic Device,” filed Jul. 24, 2008, which is incorporated by reference herein in its entirety.
TECHNICAL FIELD
• The disclosed embodiments relate generally to portable handheld electronic devices, such as cellular telephone handsets and digital cameras, and more particularly to a user interface having a touch sensitive screen for controlling camera functions.
BACKGROUND
• Portable handheld electronic devices, such as the IPHONE multifunction device by Apple Inc., have a built-in digital camera, in addition to other functions such as cellular telephony and digital audio and video file playback. The IPHONE device, in particular, has a touch sensitive screen as part of its user interface. The touch screen lets the user select a particular application program to be run, by performing a single finger gesture on the touch sensitive screen. For example, the user can point to (touch) the icon of a particular application, which results in the application being automatically launched in the device. The camera application, in particular, allows the user to navigate amongst previously stored pictures taken using the camera directly on the touch screen. In addition, there is a shutter button icon that can be touched by the user to release the shutter and thereby take a picture of the scene that is before the camera. Other uses of the touch sensitive screen include navigating around a Web page that is being displayed by single finger gestures, and zooming into a displayed Web page by performing a so called multi-finger spread gesture on the touch sensitive screen. The user can also zoom out of the Web page, by performing a multi-finger pinch gesture.
SUMMARY
• Several methods for operating a built-in digital camera of a portable, handheld electronic device are described. In some embodiments, a method is performed at a handheld electronic device having a built-in digital camera and a touch sensitive screen. The method includes detecting a multi-finger gesture on the touch sensitive screen, wherein the touch sensitive screen is serving as part of an electronic viewfinder of the camera; storing coordinates of a location corresponding to the detected multi-finger gesture; translating the stored coordinates to a selected area of an image that is captured by the camera and that is being displayed on the touch sensitive screen; contracting or expanding the selected area in response to the user's fingers undergoing a pinching movement or a spreading movement, respectively, while the detected multi-finger gesture remains in contact with the touch sensitive screen; and applying an automatic image capture parameter adjustment process that gives priority to the selected area. This gives the user finer control of auto focus, auto exposure, and auto while balance (“3A”) adjustments in the camera.
• In some embodiments, a handheld electronic device is provided which comprises a touch sensitive screen, a detector configured to detect a multi-finger gesture on the touch sensitive screen and store coordinates of a location of the detected gesture; an a digital camera. The digital camera includes an image sensor, a lens to form an optical image on the image sensor, a viewfinder module configured to display on the touch sensitive screen a scene at which the lens is aimed, and a priority module coupled to the detector. The priority module is configured to translate the stored coordinates to a selected area of a digital image of the scene that is being displayed on the touch sensitive screen by the viewfinder module, contract or expand the selected area in response to the user's fingers undergoing a pinching movement or a spreading movement, respectively, and apply an automatic image capture parameter adjustment process that gives priority to the selected area for taking a picture of the scene. Thus, a multi-touch pinch or spread gesture may define the hint or priority area, for calculating exposure parameters.
• In some embodiments, a method is performed at a handheld electronic device having a built-in digital camera and a touch sensitive screen. The method includes detecting an initial finger gesture by a user on the touch sensitive screen, wherein the touch sensitive screen serves as part of an electronic viewfinder of the camera; storing coordinates of the initial finger gesture; detecting a closed path on the touch sensitive screen that includes the location of the detected initial finger gesture, wherein the user's finger moves while remaining in contact with the touch sensitive screen to define the closed path, and storing coordinates of the closed path. The method also includes translating the stored coordinates of the closed path to a selected portion of an image that is captured by the camera and that is being displayed on the touch sensitive screen; and applying an automatic image capture parameter adjustment process that gives priority to the selected portion.
• In some embodiments, an apparatus is provided, which comprises a handheld electronic device configured to operate at least in a digital camera mode and a mobile telephone mode. The digital camera mode is configured to permit a user of the apparatus to take a digital picture of a scene, while the mobile telephone mode is configured to permit the user of the apparatus to participate in a wireless telephone call and hear the call through a built-in receiver of the apparatus. Further, the apparatus has a button exposed to the user that alternatively controls loudness of the built-in receiver when the apparatus is operating in the mobile telephone mode, and the button acts as shutter button when the apparatus is operating in the digital camera mode.
• Other embodiments are also described.
• The above summary does not include an exhaustive list of all aspects of the present invention. It is contemplated that the invention includes all systems and methods that can be practiced from all suitable combinations of the various aspects summarized above, as well as those disclosed in the Detailed Description below and particularly pointed out in the claims filed with the application. Such combinations may have particular advantages not specifically recited in the above summary.
BRIEF DESCRIPTION OF THE DRAWINGS
• The embodiments of the invention are illustrated by way of example and not by way of limitation in the figures of the accompanying drawings in which like references indicate similar elements. It should be noted that references to “an” or “one” embodiment of the invention in this disclosure are not necessarily to the same embodiment, and they mean at least one.
• FIG. 1 shows a portable handheld device having a built-in digital camera and a touch sensitive screen, in the hands of its user undergoing a single finger gesture during a still image capture process.
• FIG. 2 is a flow diagram of operations in the electronic device during a still image capture process, in accordance withFIG. 1.
• FIG. 3 shows the portable handheld electronic device undergoing a multi-finger gesture during a still image capture process.
• FIG. 4 is a flow diagram of operations in the electronic device during a still image capture process, in accordance withFIG. 3.
• FIG. 5 illustrates another embodiment of the invention, where the user draws a polygon through a single finger touch gesture, to define the priority area for image capture.
• FIG. 6 is a flow diagram of still image capture in the electronic device, in accordance with the embodiment ofFIG. 5.
• FIG. 7 shows a block diagram of an example, portable handheld multifunction device in which an embodiment of the invention may be implemented.
DETAILED DESCRIPTION
• In this section several preferred embodiments of this invention are explained with reference to the appended drawings. Whenever the shapes, relative positions and other aspects of the parts described in the embodiments are not clearly defined, the scope of the invention is not limited only to the parts shown, which are meant merely for the purpose of illustration.
• FIG. 1 shows a portable handheldelectronic device100 having a built-in digital camera and a touchsensitive screen104 in the hand of its user, undergoing a finger gesture during a still image capture process. In this example, theportable device100 is shown while it is held in the user'sleft hand107, and the user'sright hand109 is making the finger gesture on the touch screen. Thedevice100 may be an IPHONE device by Apple Inc., of Cupertino, Calif. Alternatively, it could be any other portable handheld electronic device that has a built-in digital camera and a touch sensitive screen. The built-in digital camera includes alens103 located in this example on the back face of thedevice100. The lens may be a fixed optical lens system or it may have focus and optical zoom capability. Although not depicted inFIG. 1, inside thedevice100 are an electronic image sensor and associated hardware circuitry and running software that can capture a digital image of ascene102 that is before thelens103.
• The digital camera functionality of thedevice100 includes an electronic or digital viewfinder (also referred to as a preview function). The viewfinder displays live, captured video of thescene102 that is before the camera, on a portion of the touchsensitive screen104 as shown. In this case, the digital camera also includes a soft or virtual shutter button whoseicon105 is displayed by thescreen104, directly below the viewfinder image area. As an alternative or in addition, a physical shutter button may be implemented in thedevice100. Thedevice100 includes all of the needed circuitry and/or software for implementing the digital camera functions of the electronic viewfinder (726,FIG. 7), shutter release, and automatic image capture parameter adjustment as described below.
• InFIG. 1, the user performs a single-finger gesture on the touchsensitive screen104 as shown. In this example, the finger gesture is formed by the user's right hand109 (although it could just as well be the user's left hand107). The user positions the single-finger touch gesture on a preview portion of the touch screen. Thedevice100 has detected this touch down and has automatically drawn a marker96 (in this case, the closed contour that has a box shape), centered around the location of the touch down. The user then moves herright hand109 around the preview portion of the touch screen, to a location of the image of thescene102 that corresponds to an object in the scene (or some portion of the scene) to which priority should be given when the digital camera adjusts the image capture parameters in preparation for taking a picture of the scene. For example, the user may move themarker96 from up above the mountains and the trees down towards a location near the ground or where a man is walking. After the marker has been dragged to the desired portion of the scene where the user wants the camera to give priority, the user may lift off her finger gesture, which in turn signals the camera to accept the final location of the marker and the underlying portion of the image as the priority area of the scene. Once the user has finalized the selection of this priority area, he can command the digital camera to take a picture, after adjusting the image capture parameters to give priority to the selected area. This may be done by, for example, lifting her finger off the touch sensitive display screen which not only finalizes the location of hint area but also automatically signals the device to take the picture after adjusting the parameters. A flow diagram of operations for taking the digital picture, in accordance with the above, is shown inFIG. 2.
• Referring now toFIG. 2, after having powered on thedevice100 and placed it in digital camera mode, a view finder function begins execution which displays video of thescene102 that is before the camera lens103 (block22). The user aims the camera lens so that the desired portion of the scene appears on the preview portion of thescreen104. While monitoring the screen, a camera application (or a touch screen application) running in thedevice100 detects a single-finger touch gesture and stores screen coordinates of its location (block24). Amarker96 is then automatically displayed around the screen location of the touch gesture (block26). Themarker96 is moved around the preview portion of the touch screen, in lock step with the user moving her finger touch gesture along the surface of the touch screen (block28). An area of the image of the scene being shown in the preview portion and that underlies the final location of the marker is defined to be an area selected by the user for priority (block29). This priority area may be finalized, for example, in response to the user lifting off her finger. The priority area of the image may be a fixed chunk of pixels that are about coextensive with the boundary of themarker96. Alternatively, the priority area may be an object in the scene located at or near themarker96, as detected by the camera application using digital image processing techniques.
• Once the selected area has been determined, an automatic image capture parameter adjustment process is applied by thedevice100 to give priority to the selected area (block30). Additional details of this process will be explained below. Once the parameters have been adjusted inblock30, the picture can be taken, for example, when the user gives the shutter release command (block32). Several ways of defining the shutter release command are also described below. Thus, the process described above gives the user finer control of picture taking adjustments.
• In some embodiments, during the automatic image capture parameter adjustment process, themarker96 is displayed in a variable state to indicate that one or more parameters are being adjusted.
• For example, in some embodiments, themarker96 is displayed in an alternating sequence of colors, such as white, blue, white, blue, while the automatic image capture parameter adjustment process sets priority to the selected area (e.g., themarker96 changes color while the camera is focusing). In some embodiments, the display ofmarker96 includes an animation of the boundary of the marker oscillating or “wiggling” on screen while the automatic image capture parameter adjustment process gives priority to the selected area under the location of the marker.
• In some embodiments, after the automatic image capture parameter adjustment process is completed, display ofmarker96 is terminated.
• InFIG. 3, another embodiment of the invention is shown where the user defines the priority area this time by a multi-finger gesture. In this example, the multi-finger gesture is also formed by the user's right hand109 (although it could just as well be the user'sleft hand107, while the device is held by the user's right hand109). In particular, the thumb and index finger are brought close to each other or touch each other, simultaneously with their tips being in contact with the surface of thescreen104 to create two contact points thereon. The user positions this multi-touch gesture, namely the two contact points, at a location of the image of thescene102 that corresponds to an object in the scene (or portion of the scene) to which priority should be given when the digital camera adjusts the image capture parameters in preparation for taking a picture of the scene. In this example, the user has selected the location where a person appears between a mountainside in the background and a tree in the foreground.
• In response to detecting the multi-touch finger gesture, thedevice100 may cause acontour106, in this example, the outline of a box, to be displayed on thescreen104, around the location of the detected multi-finger gesture. Thecontour106 is associated, e.g. by software running in thedevice100, with a taken or selected priority area of the image (to which priority will be given in the image capture parameter adjustment process). The user can then contract or expand the size of the priority area, by making a pinching movement or a spreading movement, respectively, with her thumb and index fingers of herright hand109 while the fingertips remain in contact with the touchsensitive screen104. Thedevice100 has the needed hardware and software to distinguish between a pinching movement and a spreading movement, and appropriately contracts or expands the size of the priority area.
• Once the user has finalized the selected area to which priority is to be given, he can command the digital camera to take a picture after adjusting the image capture parameters to give priority to the selected area. This may be done by, for example, lifting her fingers off of the touchsensitive display screen104 and then actuating the shutter release button (e.g., touching and then lifting off the soft shutter button icon105). If instead the user would like default image capture parameter values to be used, then he would simply actuate the genericshutter button icon105 without first touching the preview of the scene that is being displayed. Several alternatives to this process will now be described.
• Turning now toFIG. 4, a flow diagram of operations for taking a digital picture using thedevice100, in accordance with an embodiment of the invention is shown. After having powered on thedevice100 and placed it in digital camera mode, an electronic viewfinder function begins execution which displays video of thescene102 that is before the camera lens103 (block402). The user can now aim the camera lens so that the desired portion of the scene appears on the touchsensitive screen104. While monitoring the screen, a camera application (or a separate touch screen application) running indevice100 detects a multi-finger touch gesture, and stores screen coordinates of its location (block404). These screen coordinates are then translated to refer to a corresponding area of an image of the scene (block406). Thus, for example, referring back toFIG. 3, the screen location of thecontour106 is compared to the pixel content of the displayed image, within and near that contour (or underlying the contour), to determine that an object, in this case an image of a man walking, is present in that location. The pixels that make up the man thus become the selected or taken area of the image of the scene.
• Next, thedevice100 may contract or expand the selected area, in response to the user's fingers undergoing a pinching movement or a spreading movement, respectively while remaining in contact with the touch sensitive screen (block408). Thus, in the example ofFIG. 3, the user can expand the selected area, to include more pixels of the image, by spreading the index finger and thumb of theright hand109, while they are in contact with the screen. This may be reflected by thedevice100 enlarging thecontour106 that is being displayed. Thedevice100 detects the spreading movement and in response allocates more pixels to the selected area, for example, equally in all directions. In a similar manner, thedevice100 will contract the selected area (i.e., allocate fewer pixels of the image to define the selected area) in response to detecting that the user's fingers are undergoing a pinching movement, that is, the thumb and index finger are moved closer towards each other.
• With the selected area having been determined in this manner (block408), thedevice100 next applies an automatic image capture parameter adjustment process that gives priority to the selected area (block410). This process may include making automatic adjustments to focus, exposure, and color correction (e.g., white balance) parameters. These are sometimes referred to as3A adjustments. The adjusted parameters will be applied by the camera when “taking the next picture” of a scene that is before it. Focus adjustment may include making movement adjustments to an optical lens system of thedevice100, so as to focus on the selected area of the scene. Exposure adjustments include changing the exposure time or integration time of the entire image sensor or portions of the image sensor, based upon characteristics of the selected area including its brightness (rather than that of the entire image). Similarly, adjustments to the color correction parameters may include changing the parameters used to apply a white balance algorithm to a raw image obtained from the image sensor of the camera (that will ultimately become the “picture” of the scene). As described above, in some embodiments, during the automatic image capture parameter adjustment process, themarker96 is displayed in a variable state to indicate that one or more parameters are being adjusted.
• Once the parameters have been adjusted inblock410, the picture is taken when the user gives the shutter release command. There are several ways of actually completing the process of taking the picture, in response to thedevice100 detecting that a shutter release command has been invoked (block412). For example, output from the image sensor may not be accepted until after having detected that the multi-finger gesture has been lifted off the touch sensitive screen. In other words, the camera takes the shot only after the user lifts her fingers off the touch screen. The picture is, of course, taken using the image capture parameters that have been adjusted to give priority to the selected area.
• In another embodiment, the picture is taken only after expiration of a timer that was set upon the parameters having been adjusted. For example, a shutter button may be depressed half-way by the user, to signify that the image capture parameters be adjusted to give priority to the selected area, and then after the parameters have been adjusted, thedevice100 waits a predetermined time interval before accepting the user's command to take the shot (e.g., upon the user pressing the shutter button the rest of the way).
• In another embodiment, the camera function of thedevice100 tracks movement of an object in the scene, that has been captured in the selected area of the image, as the object and thedevice100 move relative to each other and while the multi-touch gesture is still present on the touch screen. The camera could, for example, maintain focus on the moving object only so long as the multi-touch gesture is on the screen, and then at the moment the multi-touch gesture has lifted off the screen, a still picture of the moving object is taken. Alternatively, focus would be maintained even after the multi-touch gesture has lifted off, and then the picture of the moving object is taken when a separate virtual or physical shutter button is actuated by the user.
• The above-described process, for the use of a multi-touch pinch or spread gesture to define the hint or priority area for calculating exposure parameters, may be extended to complex scenes in which there may be two or more distinct priority areas that the user would like to define. For example, the user may wish to select for priority both a dark shadow portion and a medium tone portion, but not a bright portion, of the scene.
• Referring now toFIGS. 5 and 6, another embodiment of the invention is now described for the portable handheld electronic device having a built-in digital camera and touch sensitive screen. In this embodiment, the user can draw an arbitrarily sized andshaped contour306 on thetouch screen104, around the priority or hint area of the scene that is being displayed by the viewfinder. Referring now toFIG. 6, thedevice100 captures and displays live video of thescene102 to which thecamera lens103 is pointed, using a digital viewfinder function on the touch sensitive screen104 (block602). While showing this live video, thedevice100 monitors thescreen104 for a single finger touch gesture. The initial single finger touch gesture is then detected and screen coordinates of its location are stored, while monitoring the screen (block604). Next, as the user's finger moves, while remaining in contact with the screen, to define a closed path, the closed path is detected on the touch sensitive screen and coordinates of the closed path are stored (block606). Essentially simultaneously, thecontour306 is being drawn on the preview area of the touch screen that underlies the user's finger touch. The detected closed path may include the location of the detected initial finger gesture. The stored coordinates of the closed path are translated to a portion of the image that is being displayed on the screen104 (block608). Thus, as seen inFIG. 5, the user's finger is tracing a closed path, which is depicted by acontour306, that is surrounding an image of a man walking in the scene. The stored screen coordinates of the closed path are translated to the selected area of the image of the scene, i.e. a graphical object representing the walking man (block608). The rest of the process may be as described above, namely, the application of an automatic image capture parameter adjustment process that gives priority to the selected area defined within the contour or path306 (block610), and taking a picture of the scene using the parameters as they have been adjusted to give priority to the selected area of the scene, in response to detecting a shutter release command (block612).
• As above, the shutter release command may include simply the act of the user lifting off her finger, following the initial single finger touch gesture and the tracing of thecontour306. As an alternative, the user may simply wish to accept the default image capture parameter values available in thedevice100, and so may take the picture by pressing the generic shutter button (e.g.,icon105 orphysical menu button108 of the device100), without first touching the preview area of thetouch screen104.
• In another embodiment of the invention, the user can zoom in and zoom out of the preview of the scene, using multi-touch pinch and spread gestures, respectively. In other words, with the electronic view finder function of the camera running, so that thetouch screen104 has a preview portion that is displaying live video of the scene to which the camera lens is pointed, a multi-finger gesture is detected on the touch screen. The preview portion then displays either a zoomed-in or zoomed-out version of the scene, in response to the user's fingers undergoing a spreading movement or a pinching movement on the touch screen. Thereafter, the user can lift off the multi-touch gesture and reinitiate a second multi-touch gesture or a single touch gesture, that selects an area of the zoomed in (or zoomed out) preview for purposes of 3A adjustment. In other words, zooming in or zooming out is performed immediately prior to selecting the priority area of the previewed scene, and just prior to taking a picture of the scene (according to the zoom setting and priority area selected). Note that zooming into or out of the scene may be implemented using an optical zoom capability of thedevice100, and/or a digital zoom capability.
• In yet another embodiment of the invention, a volume control button of thedevice100 can also be used as a shutter release button. Referring back toFIG. 1, the portable handheldelectronic device100 has at least two modes of operation, namely a digital camera mode and a mobile telephone mode. In the camera mode, a user of the device is to point thelens103 of the device to a scene and then take a digital picture of the scene using the built-in camera functionality. In the telephone mode, the user is to participate in a wireless telephone call and will hear the call through a built-in receiver112 (ear speaker phone) of the device. Thedevice100 also has abutton110, in this case, located on a side of the external enclosure of thedevice110 as shown (as opposed to its front face, its back face, and its top and bottom sides). Thebutton110 is exposed to the user and in the telephone mode controls the loudness of thereceiver112. In the camera mode, however, thebutton110 acts as a generic shutter button. Thebutton110 may be any one of a variety of different types, and generally is actuated by the user in different directions to increase and decrease, respectively, the loudness of thereceiver112 in the telephone mode. Shutter release occurs in this case when the button is actuated by the user in either direction (while in the camera mode). Thus, in camera mode, there may be two generic shutter buttons available for the user where either one can be used to take a picture, namely the shutter button110 (which also acts as the loudness or volume button in telephone mode), and the virtualshutter button icon105 which is positioned immediately below the preview image area of thetouch screen104.
• Turning now toFIG. 7, a block diagram of an example portable, handheldelectronic device100 is shown, in accordance with an embodiment of the invention. Thedevice100 may be a personal computer, such as a laptop, tablet, or handheld computer. Alternatively, thedevice100 may be a cellular phone handset, personal digital assistant (PDA), or a multi-function consumer electronic device, such as the IPHONE device.
• Thedevice100 has aprocessor704 that executes instructions to carry out operations associated with thedevice100. The instructions may be retrieved frommemory720 and, when executed, control the reception and manipulation of input and output data between various components ofdevice100. Although not shown, thememory720 may store an operating system program that is executed by theprocessor704, and one or more application programs are said to run on top of the operating system to perform different functions described below. The touchsensitive screen104 displays a graphical user interface (GUI) to allow a user of thedevice100 to interact with various application programs running in thedevice100. The GUI displays icons or graphical images that represent application programs, files, and their associated commands on thescreen104. These may include windows, fields, dialog boxes, menus, buttons, cursors, scrollbars, etc. During operation, the user can select and activate various graphical images to initiate functions associated therewith.
• Thetouch screen104 also acts as an input device, to transfer data from the outside world into thedevice100. This input is received via, for example, the user's finger touching the surface of thescreen104. Thescreen104 and its associated circuitry recognizes touches, as well as the position and perhaps the magnitude of touches and their duration on the surface of thescreen104. These may be done by agesture detector program722 that may be executed by theprocessor704. Note that a dedicated processor may be provided to process touch inputs, in order to reduce demand for a main processor of the system. The touch sensing capability of thescreen104 may be based on technology such as capacitive sensing, resistive sensing, or other suitable solid state technologies. The touch sensing may be based on single point sensing or multi-point or multi-touch sensing. Single point touch sensing is capable of only distinguishing a single touch, while multi-point sensing is capable of distinguishing multiple touches that occur at the same time.
• The input device aspect of thetouch screen104 may be integrated with its display device. The input device may be positioned “on top of” the display device, so that the user can manipulate the GUI directly by, for example, placing her finger on top of an object that is being displayed, in order to control that object. Note that this is different than how a touchpad works, because in a touchpad there is no one-to-one relationship such as this. With touchpads, the input device is not aligned with the display device, and the two are sometimes in different planes altogether. Additional details concerning the touchsensitive screen104 and operation of thegesture detector722 to detect user gestures (in this case, single and multi-touch finger gestures) are described in U.S. Patent Application Publication No. 2006/0026535, entitled “Mode-Based Graphical User Interfaces for Touch Sensitive Input Devices”. Thegesture detector722 recognizes the occurrence of gestures and informs one or more software agents running in thedevice100 of these gestures and/or what actions to take in response to such gestures. A gesture may be identified as a command for performing certain action in an application program, and in particular, a camera application as described below.
• A wide range of different gestures may be defined and used. For example, a static gesture does not involve motion, while a dynamic gesture is one that includes motion, e.g. movement of a single or multi-touch point on thescreen104. A continuous gesture is one that is performed in a single stroke in contact with thescreen104, whereas a segmented gesture is one that is performed in a sequence of distinct steps or strokes, including at least one lift off from thetouch screen104. In addition, thedevice100 may recognize a gesture and take an associated action at essentially the same time as the gesture, that is, the gesture and the action simultaneously occur side-by-side rather than being a two-step process. For example, during a scrolling gesture, the graphical image of the screen moves in lock step with the finger motion. In another example, an object presented on the display device continuously follows the gesture that is occurring on the input device, that is, there is a one-to-one relationship between the gesture being performed and the object shown on the display portion. For example, during a zooming gesture, fingers may spread apart or close together (pinch) in order to cause the object shown on the display to zoom in during the spread and zoom out during the close or pinch. These are controlled by theprocessor704 executing instructions that may be part of thegesture detector program722, or another application program such as a priority (camera)application program728.
• Still referring toFIG. 7, camera functionality of thedevice100 may be enabled by the following components. A solidstate image sensor706 is built into thedevice100 and may be located at a focal plane of an optical system that includes thelens103. An optical image of a scene before the camera is formed on theimage sensor706, and thesensor706 responds by capturing the scene in the form of a digital image or picture consisting of pixels that will then be stored inmemory720. Theimage sensor706 may include a solid state image sensor chip with several options available for controlling how an image is captured. These options are set by image capture parameters that can be adjusted automatically, by the priority (camera)application728. Thepriority application728 can make automatic adjustments, that is without specific user input, to focus, exposure and color correction parameters (sometimes referred to as3A adjustments) based on a hint or priority portion of the scene that is to be imaged. This selected or target area may be computed by thepriority application728, by translating the stored coordinates of the detected gesture to certain pixel coordinates of a digital image of the scene that is being displayed at the moment of the touch gesture occurring. Thepriority application728 may contract or expand this selected area in response to receiving an indication from thegesture detector722 that the user's fingers are undergoing a pinch or spread movement, respectively. Once the selected area (hint) has been finalized, thepriority application728 will apply an automatic image capture parameter adjustment process that adjusts one or more image capture parameters, to give priority to the selected area for taking a picture of the scene.
• Still referring toFIG. 7, thedevice100 may operate not just in a digital camera mode, but also in a mobile telephone mode. This is enabled by the following components of thedevice100. Anintegrated antenna708 that is driven and sensed byRF circuitry710 is used to transmit and receive cellular network communication signals from a nearby base station (not shown). Amobile phone application724 executed by theprocessor704 presents mobile telephony options on the touchsensitive screen104 for the user, such as a virtual telephone keypad with call and end buttons. Themobile phone application724 also controls at a high level the two-way conversation in a typical mobile telephone call, by allowing the user to speak into the built-inmicrophone714 while at the same time being able to hear the other side of the conversation through the receive orear speaker112. Themobile phone application724 also responds to the user's selection of the receiver volume, by detecting actuation of thephysical volume button110. Although not shown, theprocessor704 may include a cellular base band processor that is responsible for much of the digital audio signal processing functions associated with a cellular phone call, including encoding and decoding the voice signals of the participants to the conversation.
• Thedevice100 may be placed in either the digital camera mode or the mobile telephone mode, in response to, for example, the user actuating aphysical menu button108 and then selecting an appropriate icon on the display device of the touchsensitive screen104. In the telephone mode, themobile phone application724 controls loudness of thereceiver112, based on a detected actuation or position of thephysical volume button110. In the camera mode, the priority (camera)application728 responds to actuation of thevolume button110 as if the latter were a physical shutter button (for taking pictures). This use of thevolume button110 as a physical shutter button may be an alternative to a soft or virtual shutter button whose icon is simultaneously displayed on the display device of thescreen104 during camera mode (see, e.g.FIG. 3, whereicon105 may be a generic virtual shutter button (default exposure parameters) and is displayed below the preview portion of the display device of the touch sensitive screen104).
• An embodiment of the invention may be a machine-readable medium having stored thereon instructions which program a processor to perform some of the operations described above. In other embodiments, some of these operations might be performed by specific hardware components that contain hardwired logic. Those operations might alternatively be performed by any combination of programmed computer components and custom hardware components.
• A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), not limited to Compact Disc Read-Only Memory (CD-ROMs), Read-Only Memory (ROMs), Random Access Memory (RAM), Erasable Programmable Read-Only Memory (EPROM), and a transmission over the Internet.
• The invention is not limited to the specific embodiments described above. For example, the multi-finger touch down may be defined as a set of one or more predetermined patterns detected in the input device of the touchsensitive screen104. For example, a particular pattern may be defined for the joint tips of the index finger and thumb of the same hand, being pressed against the touch screen, for a certain interval of time. Alternatively, a pattern may be defined by the tips of the index finger and thumb being spaced apart from each other and held substantially in that position for a predetermined period of time. There are numerous other variations to different aspects of the invention described above, which in the interest of conciseness have not been provided in detail. Accordingly, other embodiments are within the scope of the claims.

Claims (21)

1. A method, comprising:

at a handheld electronic device having a built-in digital camera and a touch sensitive screen:

detecting a multi-finger gesture on the touch sensitive screen, wherein the touch sensitive screen is serving as part of an electronic viewfinder of the camera;

storing coordinates of a location corresponding to the detected multi-finger gesture;

translating the stored coordinates to a selected area of an image that is captured by the camera and that is being displayed on the touch sensitive screen;

contracting or expanding the selected area in response to the user's fingers undergoing a pinching movement or a spreading movement, respectively, while the detected multi-finger gesture remains in contact with the touch sensitive screen; and

applying an automatic image capture parameter adjustment process that gives priority to the selected area.

2. The method ofclaim 1, wherein applying the automatic image capture parameter adjustment process includes making automatic adjustments to one or more parameters selected from the group consisting of focus, exposure, and color correction, and wherein the camera is configured to apply the automatic adjustments when taking a picture.

3. The method ofclaim 1, further comprising:

displaying on the touch sensitive screen a contour around the location of the detected multi-finger gesture; and

associating the contour with the selected area of the image.

4. The method ofclaim 1, further comprising:

tracking movement of an object captured in the selected area of the image, as the object and the device move relative to each other.

5. The method ofclaim 1, further comprising:

taking a picture, using image capture parameters that have been adjusted to give priority to the selected area in response to detecting that the multi-finger gesture has lifted off the touch sensitive screen.

6. The method ofclaim 1, further comprising:

taking a picture, using image capture parameters that have been adjusted to give priority to the selected area, in response to an expiration of a timer that was set upon the parameters having been adjusted.

7. The method ofclaim 1, further comprising:

detecting a further multi-finger gesture on the touch sensitive screen;

storing coordinates of a location corresponding to the further multi-finger gesture; and

translating the stored coordinates to a further selected area of the image that is captured by the camera and that is being displayed on the touch sensitive screen,

wherein the automatic image capture parameter adjustment process is applied to give priority to both the selected area and the further selected area.

8. The method ofclaim 7, wherein the selected area and the further selected area are two distinct user-defined priority areas.

9. The method ofclaim 8, wherein the selected area corresponds to a dark shadow area of the image to be captured by the camera, and the further selected area corresponds to a medium tone area of the image to be captured by the camera.

10. The method ofclaim 1, further comprising:

while the touch sensitive screen is displaying a scene at which the camera is pointed:

detecting a multi-finger gesture made by a user on the touch sensitive screen; and

zooming into or out of the scene, in response to detecting the user's fingers undergoing a spreading movement or a pinching movement on the touch sensitive screen.

11. The method ofclaim 10, wherein the zooming into or out of the scene comprises performing a digital zoom.

12. The method ofclaim 10, wherein the zooming into or out of the scene comprises performing an optical zoom.

13. The method ofclaim 1, further comprising:

while the automatic image capture parameter adjustment process sets priority to the selected area, displaying the marker in a variable state to indicate that one or more parameters are being adjusted.

14. A handheld electronic device, comprising:

a touch sensitive screen;

a detector configured to detect a multi-finger gesture on the touch sensitive screen and store coordinates of a location of the detected gesture; and

a digital camera, including:

an image sensor,

a lens to form an optical image on the image sensor,

a viewfinder module configured to display on the touch sensitive screen a scene at which the lens is aimed, and

a priority module coupled to the detector, wherein the priority module is configured to:

translate the stored coordinates to a selected area of a digital image of the scene that is being displayed on the touch sensitive screen by the viewfinder module,

contract or expand the selected area in response to the user's fingers undergoing a pinching movement or a spreading movement, respectively, and

apply an automatic image capture parameter adjustment process that gives priority to the selected area for taking a picture of the scene.

15. The handheld electronic device ofclaim 14, wherein the application of the automatic image capture parameter adjustment process includes making at least one automatic adjustment to one or more parameters selected from the group consisting of focus, exposure, and color correction, and wherein the camera is configured to apply the automatic adjustments when taking a picture.

16. The handheld electronic device ofclaim 14, wherein the digital camera is configured to:

display a graphical object on the touch sensitive screen that is associated with a virtual shutter button of the digital camera; and

take the picture of the scene in accordance with image capture parameters, which are set to a default setting when the virtual shutter button is actuated, and adjusted to give priority to an area of the scene selected by the multi-finger touch gesture.

17. The handheld electronic device ofclaim 16, wherein the graphical object representing the virtual shutter button is displayed below the preview.

18. A method, comprising:

at a handheld electronic device having a built-in digital camera and a touch sensitive screen:

detecting an initial finger gesture by a user on the touch sensitive screen, wherein the touch sensitive screen serves as part of an electronic viewfinder of the camera;

storing coordinates of the initial finger gesture;

detecting a closed path on the touch sensitive screen that includes the location of the detected initial finger gesture, wherein the user's finger moves while remaining in contact with the touch sensitive screen to define the closed path;

storing coordinates of the closed path;

translating the stored coordinates of the closed path to a selected portion of an image that is captured by the camera and that is being displayed on the touch sensitive screen; and

applying an automatic image capture parameter adjustment process that gives priority to the selected portion.

19. An apparatus, comprising:

a handheld electronic device configured to operate at least in a digital camera mode and a mobile telephone mode, wherein:

the digital camera mode is configured to permit a user of the apparatus to take a digital picture of a scene, and

the mobile telephone mode is configured to permit the user of the apparatus to participate in a wireless telephone call and hear the call through a built-in receiver of the apparatus,

wherein the apparatus has a button exposed to the user that alternatively:

controls loudness of the built-in receiver when the apparatus is operating in the mobile telephone mode, and

acts as shutter button when the apparatus is operating in the digital camera mode.

20. The apparatus ofclaim 19, wherein:

the button is configured to be actuated by the user in two different directions to increase and decrease, respectively, loudness in the mobile telephone mode; and

shutter release occurs when the button is actuated by the user in either one of said directions in the digital camera mode.

21. The apparatus ofclaim 19, further comprising:

a built-in touch sensitive screen that serves as part of an electronic viewfinder in the digital camera mode, and

wherein the digital camera mode is configured to display a shutter release button on the touch sensitive screen.

Images