US20150185882A1 - Method and apparatus for recognition of writing, for remote communication, and for user defined input templates - Google Patents

Abstract

A device including a housing, a writing tip connected to the housing, a writing surface position indicator, a processor in the housing, a memory device in the housing connected to the processor, and a sensor in the housing and cooperative with the writing surface position indicator. The device may be used to record writings and drawings applied to a surface by a user, to transmit that data to a remote device, to download data from remote devices, and to otherwise communicate with remote devices.

Description

CLAIM OF PRIORITY
• This application is a continuation of pending U.S. application Ser. No. 11/474,833 filed Jun. 26, 2006. U.S. application Ser. No. 11/474,833 is a continuation of pending U.S. application Ser. No. 10/430,182 filed May 5, 2003, which is a continuation of U.S. application Ser. No. 10/146,243 filed May 5, 2002 and issued as U.S. Pat. No. 6,628,847 on Sep. 30, 2003. U.S. application Ser. No. 10/146,243 was a divisional of Ser. No. 09/258,560, which was filed on Feb. 26, 1999 and which issued as U.S. Pat. No. 6,456,749 on Sep. 24, 2002. U.S. application Ser. No. 09/258,560 claimed priority to U.S. Provisional Application Ser. No. 60/076,338 filed Sep. 27, 1998.
STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT
• Not Applicable
BACKGROUND OF THE INVENTION
• 1. Field of the Invention
• The present invention is directed generally to a method and device for recognition of writing and communication with remote devices.
• 2. Description of the Background
• The use of computers for data entry and communication has become limiting and inconvenient in some aspects. For example, the use of a computer keyboard requires repetitive motion that is known to cause physical discomfort and, in some cases, physical injury. In addition, computers are relatively large devices. Even the advent of laptop computers has not made data entry and communication easy. One reason is that a laptop cannot be smaller than a keyboard and, to be efficient, a keyboard must have keys at least as large as a typical user's fingers. As a result, the size of a laptop computer is not foreseen to reduce significantly in the future. In any event, it is not foreseen to become small enough to be carried by a person with little or no inconvenience.
• To address these problems, some relevant art discloses hand-held devices for data entry, typically in the form of handwriting recognition. Many of those devices, however, require a written or sensory grid in order to determine the motion of the device and read the lines drawn by a user. Some devices require special purpose input pads to monitor the writing and drawing of a user. Other devices require that a user only print using special characters in order for handwriting recognition to work. All of those devices also suffer the inability to monitor multiple writings by a user on an ordinary piece of paper. Other devices allow for multiple writings on an ordinary piece of paper, but do not compensate for movement of the paper relative to the writing instrument. Thus, the prior art devices inaccurately recognize the writings because the user compensates for the movement of the paper, but the device is not able to determine that user has so compensated.
• As a result, the need exists for a device that provides for easy and convenient data entry and communication, as well as the ability to use any surface and to be able to accurately monitor a user's actions.
BRIEF SUMMARY OF THE INVENTION
• The present invention is directed to a device including a housing, a writing tip connected to the housing, a writing surface position indicator, a processor in the housing, a memory device in the housing connected to the processor, and a sensor in the housing and cooperative with the writing surface position indicator. The device may be used to record writings and drawings applied to the surface by the user, to transmit that data to a remote device, to download data from remote devices, and to otherwise communicate with remote devices.
• The present invention is also directed to a device including a housing, writing tip connected to the housing, a processor in the housing, and a memory device in the housing connected to the processor having stored thereon instructions which when executed by the processor cause the processor to recognize an interface defined by writing on a writing surface.
• The present invention provides the advantage in that it permits data entry with a hand-held device which does not require a special writing surface. The present invention also provides the advantage that data can be entered accurately because the device compensates for movement of the writing surface during the writing process. The present invention also provides the advantage that the data can be stored into the writing instrument itself, rather than having to transmit the data to a remote device. The present invention also provides the advantage that the device can communicate with remote devices. The present invention also provides the advantage that a user can define a personal interface, which can be stored by the device, to enter data or instruct the device to perform certain commands. These and other advantages and benefits of the present invention will become apparent from the detailed description of the invention hereinbelow.
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING
• For the present invention to be clearly understood and readily practiced, the present invention will be described in conjunction with the following figures, wherein:
• FIG. 1 is a side plan view of a device constructed in accordance with the present invention;
• FIG. 2 is a block diagram illustrating components of the device;
• FIG. 3 is a top plan view of the device being used to write on a piece of paper;
• FIG. 4 is a top plan view of the device being used to define a personal interface on a piece of paper;
• FIG. 5 is a top plan view of the device being used to define an alternative personal interface on a piece of paper;
• FIG. 6 is a side plan view of the device in a base; and
• FIG. 7 is a side plan view of an alternative embodiment of the device formed as a ring on a user's finger.
DETAILED DESCRIPTION OF THE INVENTION
• It is to be understood that the figures and descriptions of the present invention have been simplified to illustrate elements that are relevant for a clear understanding of the present invention, while eliminating, for purposes of clarity, other elements. Those of ordinary skill in the art will recognize that other elements may be desirable in order to implement a particular embodiment of the present invention. However, because such elements are well known in the art, and because they do not facilitate a better understanding of the present invention, a discussion of such elements is not provided herein.
• FIG. 1 is a side plan view of adevice10 constructed in accordance with the present invention. Thedevice10 includes awriting instrument12 and a writingsurface position indicator14. Thewriting instrument12 may be a writing device such as, for example, a pen, a mechanical pencil, a marker, and chalk. Thewriting instrument12 includes ahousing15 and awriting tip16 connected to thehousing15 to enable a user of thedevice10 to place marks on a surface by applying thewriting tip16 to any surface.
• The writingsurface position indicator14 provides information indicative of the relative position of thewriting tip16 to the surface on which the user is using thewriting instrument12. The writingsurface position indicator14 communicates with the writinginstrument12 to better determine the position of thewriting tip16 relative to the writing surface. The communication between the writingsurface position indicator14 and the writinginstrument12 helps thedevice10 track the marks being applied to the writing surface, even if the surface is translated or rotated during use. Thedevice10 may track movement of the writinginstrument12 with a number ofaccelerometers36, and periodically correct the perceived position of the writinginstrument12 with the writingsurface position indicator14, as discussed hereinbelow.
• Alternatively, the writingsurface position indicator14 may be used alone to determine position and movement of the writinginstrument12. To communicate the relative position of thewriting tip16 to the surface, the writingsurface position indicator14 is affixed to or engaged with the writing surface and transmits a signal to one ormore sensors17 in thehousing15. For example, the writingsurface position indicator14 may be a clip fastenable to an edge or corner of a writing surface. When the writingsurface position indicator14 is not being used to provide information indicative of the relative position of thewriting tip16 to the writing surface, it may be fastened to thewriting instrument12. For example, the writingsurface position indicator14 may be fastened to an end of the writinginstrument12 opposite thewriting tip16, such as is illustrated inFIG. 1. Further, the writingsurface position indicator14 may be fastened to the end of the writinginstrument12 to protect thewriting tip16 when thedevice10 is not in use.
• In one embodiment, the writingsurface position indicator14 may be a clip, such as on a clipboard, for holding the writing surface. In another embodiment, the writingsurface position indicator14 may be, for example, a sticker fastenable to the writing surface. In another embodiment, the writingsurface position indicator14 may be placed on the writing surface and have sufficient mass, such as a paper weight, to maintain a constant position relative to the writing surface even if the writing surface is translated or rotated during writing.
• In addition, the writingsurface position indicator14 may be active or passive. For example, the writingsurface position indicator14 may include a number oftransmitters18 to transmit a signal to thesensor17 in thewriting instrument12. Thedevice10 determines the relative position of thewriting tip16 based on the received signal, as further described hereinbelow. Alternatively, the writingsurface position indicator14 may be passive, such as by utilizing magnetic components. For example, the writingsurface position indicator14 may be a magnetic substance, such as magnetic ink applied to the surface or a magnetic sticker fixably placed on the writing surface. In an embodiment in which the writingsurface position indicator14 is passive, thesensor17 detects the position and orientation of the writingsurface position indicator14, such as by sensing a magnetic field generated by the writingsurface position indicator14, to determine the relative position of thewriting tip16.
• Thedevice10 may also include one or more other features to facilitate its use. For example, thedevice10 may include aninput19, such as a push button, to allow the user to select from various modes of operation and functions to be performed by thedevice10, as further described hereinbelow. Thedevice10 may also include adisplay20, such as a liquid crystal display, for providing a user with visual information, including information regarding user input (feedback). Thedisplay20 may also, for example, serially display received and sent information, such as e-mail messages. In addition, a facsimile message may be received by thedevice10, for example, by converting the facsimile message into electronic form and displaying it on thedisplay20, much like an e-mail message.
• Thedevice10 may further include anaudio interface system21, including aspeaker22 and amicrophone24. Theaudio interface21 may be detachably connectable to thehousing15. When theaudio interface21 is connected to thehousing15, it may allow thedevice10 to provide conventional telephony communications using, for example, cellular technology contained within thehousing15, as discussed hereinbelow. In addition, thedevice10 may, for example, be programmed to record audio dictation, to translate audio input into text to send via e-mail, or to convert the text of a received e-mail message into audio output. Also, theaudio interface system21 may provide an audio signal when an e-mail message is received by thedevice10.
• FIG. 2 is a block diagram illustrating one embodiment of thedevice10 including aprocessor30. Theprocessor30 may be any one of many processors, such as one of the Pentium® family of processors manufactured by Intel Corp., Santa Clara, Calif. Amemory device32 is coupled to theprocessor30, and may be used to store instructions for the operation of thedevice10, as well as to store data collected by thedevice10. Thememory device32 may be, for example, 10 Mbytes of solid state memory. Thememory device32 may be volatile or non-volatile. Thememory device32 may include ROM for storing instructions to be executed by theprocessor30. Thememory device32 may include RAM, such as dynamic RAM or static RAM, for storing instructions and data. Thememory device32 may be or include an EPROM for storing instructions and data.
• Thedevice10 may track the movement of thewriting tip16 relative to the writing surface with the writingsurface position indicator14 in conjunction with thepressure sensor34 andaccelerometers36. For example, onepressure sensor34 may be connected to and located near the writingtip16 of the writinginstrument12. Alternatively, thepressure sensor34 may be located along or at the end of ashaft38 connected to thewriting tip16. Thepressure sensor34 allows thedevice10 to determine when thewriting tip16 is against a writing surface and the force with which thewriting tip16 is being applied to the writing surface, thereby allowing thedevice10 to determine when the user is writing as well as the fineness or thickness of a line being drawn. Thepressure sensor34 may be, for example, a strain gauge based force sensor.
• Movement of the writinginstrument12 may be determined from theaccelerometers36, which may be connected to theshaft38. Eachaccelerometers36 may be, for example, a dual axis, micro-electromechanical accelerometer. Threeaccelerometers36 may be used, as illustrated inFIG. 1, so that collectively they provide data indicative of motion, including speed and direction, of the writinginstrument12 in all three planes of three dimensional space. In an alternative embodiment, threeadditional accelerometers36 may be employed to provide data indicative of roll, pitch, and yaw. Information relating to roll, pitch, and yaw of the writinginstrument12 during the writing operation may provide for better handwriting recognition because information relating to rotation of the writinginstrument12 about a particular point is obtained. Alternatively, the present invention may only utilize twoaccelerometers36 to determine movement on the writing surface, and thepressure sensor34 alone is used to determine when the writinginstrument12 is engaging the writing surface.
• Used in conjunction with thepressure sensor34 andaccelerometers36, the writingsurface position indicator14 facilitates determining the position of thewriting tip16 relative to the writing surface, even if the writing surface moves during use, such as is often the case during writing. Furthermore, the writingsurface position indicator14 may be used by thedevice10 to periodically calibrate the position of thewriting tip16 relative to surface as determined by theaccelerometers36 to compensate for the error in the position determined thereby, which typically accumulates over time.
• In an alternative embodiment, the writingsurface position indicator14 may alone be used to track the movement of the writinginstrument12. In that embodiment, the writingsurface position indicator14 continually determines the position of thewriting tip16 relative to the writing surface in three dimensions, thus enabling thedevice10 to determine when thewriting tip16 is in contact with the surface, and the movement of thewriting tip16 as it is applied to the surface. The writingsurface position indicator14 communicates the information to thewriting instrument12 as described hereinbelow.
• Thememory device32 may be programmed with instructions which when executed by theprocessor30 cause theprocessor30 to recognize handwriting using conventional handwriting recognition algorithms. Thememory device32 may also be programmed to cause theprocessor30 to record in thememory device32 non-recognized writings, such as drawings and doodles written by the user.
• Acommunication system40 may also be provided so that thedevice10 may engage in external communication. For example, cellular telecommunication technology, such as PCS digital cellular technology, may be contained in thewriting instrument12, and used to allow thedevice10 to send and receive information. Thecommunication system40 may be used, for example, to allow a user to use thedevice10 as a telephone, with the aid of theaudio interface system21. Thecommunication system40 may also allow a user to connect to the Internet, via an internal modem and Internet connection (not shown), so that thedevice10 may, for example, receive and send e-mail.
• FIG. 3 is a top plan view of thedevice10 being used to write on a piece ofpaper42. The writinginstrument12 may be used to write on thepaper42 in a conventional manner by applying thewriting tip16 to thepaper42. In the embodiment of the present invention illustrated inFIG. 3, the writingsurface position indicator14 is a clip fastenable to an edge of thepaper42. According to that embodiment, the writingsurface position indicator14 may be removed from the writinginstrument12 and clipped, or otherwise fastened, to thepaper42. An embodiment of the writingsurface position indicator14 includes its own power supply, such as a rechargeable Lithium-ion solid polymer or liquid gel battery. The writingsurface position indicator14 may communicate one or more signals that are used by thedevice10 to determine the relative position of thewriting tip16. The signals may be, for example, infrared signals, ultrasonic signals, or electromagnetic signals. The signals from the writingsurface position indicator14 are received by thesensor17, which may be, for example, a receiver as through part of the communication system40 (illustrated inFIG. 2).
• Thedevice10 may use the signals communicated by the writingsurface position indicator14 in any of several ways. For example, the signals from the writingsurface position indicator14 may be used to provide a reference for thedevice10 of the relative position between the writingtip16 and the writing surface because the writingsurface position indicator14 maintains a constant position relative to thepaper42. As a result, even if thepaper42 is moved during use, as almost always happens during writing, thedevice10 will be able to maintain its position relative to thepaper42, allowing thedevice10 to maintain an accurate record of what is being written. The writingsurface position indicator14 also allows for periodic calibration of thedevice10. For example, by maintaining a known reference point, established at the beginning of a writing session, the writingsurface position indicator14 helps thedevice10 compensate for small errors in itsaccelerometers36 that accumulate over time. Without this feature, the movements recorded and used by thedevice10 may become unusably noisy within a few minutes of use.
• In one embodiment of the present invention, the writingsurface position indicator14 has twosignal transmitters18 separated by a small distance. The writingsurface position indicator14 periodically transmits signals from the twotransmitters18 which are received by thesensor17. The signals may be transmitted at different frequencies so that they may be distinguished. Thedevice10 determines the exact position of the writinginstrument12 relative to the writingsurface position indicator14 by, for example, triangulation. Thedevice10 then resets the internal value of where the writinginstrument12 is relative to the surface (having over the past small fraction of a second been approximated by the accelerometers36) to the value calculated based on the information provided by the writingsurface position indicator14. In this way, “creep” can be avoided. The frequency of signals transmitted by the writingsurface position indicator14 may vary depending on the accuracy required for a specific application and the accuracy of theaccelerometers36. Furthermore, thedevice10 can maintain the position of thewriting tip16 relative to thepaper42 even if thepaper42 is rotated, or otherwise moved, during the writing. In an alternative embodiment, the writingsurface position indicator14 may produce a single signal from onetransmitter18, and thedevice10 may sense that signal with two spaced apartsensors17 on the writing instrument12 (as illustrated inFIG. 1), so that the position of thewriting tip16 relative to the writingsurface position indicator14 may be determined using, for example, triangulation.
• Thedevice10 may operate in several modes. The different modes may be selected with theinput19. For example, thedevice10 may include an operation mode, a command mode, and an off mode. To operate in the different modes, instructions are stored in thememory device32 which when executed by theprocessor30 cause theprocessor30 to operate thedevice10 in the various modes. For example, thedevice10 in the operation mode monitors and records what is being written or drawn by the user on the writing surface. Thedevice10 can be programmed to recognize the writing using handwriting recognition algorithms stored in thememory device32.
• In the command mode, thedevice10 monitors the writing of the user and interprets that writing as commands. For example, the user may send previous writings and drawings, such as those monitored and stored in operation mode, by writing “send” followed by parameters for sending. For example, by writing “send 555-1234”, thedevice10 will send the data via facsimile. In contrast, if “send” is followed by an e-mail address, the data will be sent via e-mail. Of course, separate commands, such as “fax” and “e-mail,” may be used to avoid confusion. An “end” command, or any other command defined by the user, may be used to initiate action by the device. For example, the symbol “.fwdarw.” may be used to end and execute the command. The device may also use other commands, such as “save”, “read”, “contact”, “download”, and “phone”.
• In “off” mode, thedevice10 operates as a conventional writing device and does not perform other functions, such as handwriting recognition.
• Thedevice10 may also be used to define a personal interface. The interface may be used, for example, to enter data into thedevice10 or instruct thedevice10 to execute specified commands. The interface may be defined by the user as a template including, for example, icons, words, and symbols on a surface, which are drawn with thedevice10. For example, the user may enter command mode, write the word “interface”, and draw a template of numbers and functions that define a calculator, as illustrated inFIG. 4. The handwriting recognition feature of thedevice10 allows it to recognize the numbers and letters, and thedevice10 can determine the writing tip's16 position relative to those drawn numbers and functions. To better define the personal interface, each element, such as eachnumber44 and eachfunction46 may be surrounded by a circle orbox48 to more clearly define acommand area50 so that, for example, touching thewriting tip16 inside thecommand area50 invokes that element or function, and touching thewriting tip16 outside of thecommand area50 does not invoke it. The user may touch thewriting tip16 of the writinginstrument12 to thecommand area50, and thedevice10 will perform the command indicated by the user. For example, the user may draw a calculator then interact with the drawn interface as though it were a physical calculator. Answers may be provided on thedisplay20. As a result, a user may balance their checkbook on a piece of scrap paper while waiting at a bus stop. When finished, the user may throw away or recycle the paper.
• Other, potentially limitless, interfaces may also be created. For example, the template may be a keyboard, with the keys arranged in any order desired by the user, and the user may interact with the keyboard with thewriting tip16 of the writinginstrument12. The entries made by the user may be shown on thedisplay20, and the user may review the results by drawing on thepaper42 up and down arrows, and then touching the arrows.
• Also, the interface may be a user-defined macroinstruction. For example, as illustrated inFIG. 5, the user could write the name of a person on the paper and circle it to define acommand area50, and instruct thedevice10 that whenever the user touches thewriting tip16 to thecommand area50 thedevice10 is to transmit stored data to the person's e-mail address. Moreover, the macroinstruction may be stored in thememory device32 of thedevice10, such that it is saved, and will be recognized by thedevice10 without the user having to re-define the interface during each subsequent use of thedevice10.
• Another example of an interface is that the user could write on thepaper42 “FONT=COURIER,” circle it, and instruct thedevice10 that whenever the user touches thewriting tip16 to thepaper42 within the circle, thedevice10 is to convert information stored in the memory device into ASCII text in courier font.
• The various modes may also be used in conjunction. For example, a user could select the operate mode using theinput19, and then write on a piece of paper. In the operate mode, thedevice10 recognizes the writing, as described hereinbefore. Also, the user could instruct thedevice10 to store the writing in thememory device32 under a particular index or file.
• Alternatively, in the command mode, the user could instruct thedevice10 to perform a macroinstruction saved as a user-defined interface, such as to e-mail the writing to a particular address.
• FIG. 6 is a side plan view of thedevice10 in abase54. The base54 may be used to download data from thedevice10 and transmit it to a remote device, such as a personal computer. The base54 may also charge the power supplies of thedevice10. The power supplies for thedevice10 may be, for example, an internal rechargeable Lithium-ion solid polymer or liquid gel battery. In addition, thebase54 may be utilized to upload instructions into thememory device32 of thedevice10. Acord56 on thebase54 may be used to connect thedevice10 to a remote device, such as to a computer and/or to a power supply. Alternatively, thebase54 could be wireless.
• The present invention also includes a method of operating the present invention. The method includes defining and using a personal interface. The interface may be defined with a pen, as described hereinbefore, or it may be defined in other ways. For example, as illustrated inFIG. 7, the invention may be embodied as a ring or sleeve that fits on a user's finger. The writinginstrument12, for example, may be a ring with anextension60 that engages thepaper42 and houses apressure sensor34. A writingsurface position indicator14 may be removably attached to thewriting instrument12 and attached topaper42 when writing. In that way, the user may write and draw with the finger as if it were thedevice10 described hereinbefore. Feedback may be provided to the user by adisplay20 mounted on the ring.
• Those of ordinary skill in the art will recognize that many modifications and variations of the present invention may be implemented. For example, a user need not define an interface on a writing surface, but could instead define an interface in space, such as by moving the writinginstrument12 similar to waving a baton. A user could distinguish to thedevice10 when he is defining the interface and interacting with it from merely moving the writinginstrument12 from point to point, such as by indicating so to thewriting instrument12 by depressing theinput19. The foregoing description and the following claims are intended to cover all such modifications and variations.

Claims (1)

What is claimed is:

1. A device comprising:

a. a housing;

b. a writing tip connected to the housing;

c. a processor in the housing;

d. at least one of a speaker and a display supported by the housing; and

e. a memory device in the housing connected to the processor and having stored thereon instructions which when executed by the processor cause the processor to (i) recognize an interface defined by writing on a writing surface, and (ii) to cause the processor to recognize user interaction with the interface, and (iii) to cause the at least one of a speaker and display to provide output based on said user interaction.

Images