US20030123733A1 - Recognizing commands written onto a medium - Google Patents

Abstract

A method includes obtaining a scanned command mark written with a conventional writing implement onto a conventional medium, and recognizing the scanned command mark as a command that may be executed by a processor. The method may further include recognizing a command mark that comprises a notational, transformational and operational mark.

Description

TECHNICAL FIELD
• This invention relates to a machine-executable process for recognizing commands written onto a medium, such as paper.[0001]
BACKGROUND
• In the modern workplace a variety of computer resources are generally available, such as desk-top or lap-top computers. Entering information or commands to a computer is typically accomplished by using a keyboard or a mouse connected to a computer.[0002]
• Despite the availability of computer resources, many people still use more traditional methods and mediums to record information, such as writing notes and action items on paper using a pen or pencil.[0003]
DESCRIPTION OF THE DRAWINGS
• FIG. 1 is a flowchart showing a command recognition process.[0004]
• FIG. 2A-[0005]2B shows exemplary command marks written on pieces of paper.
• FIG. 3 is a block diagram of computer hardware on which the command recognition process may be implemented.[0006]
DESCRIPTION
• Referring to FIG. 1, a[0007]process10 is shown for recognizing commands written on a piece of paper.Process10 is basically a three part process. First, a command mark is written on a piece ofpaper12 using a conventional writing implement, such as a pen or pencil. Second, the command mark is converted into digital information, either as adigital image file18 ordigital stroke data20, using either ascanner14 or a digitalink capture device16, respectively. Third, the digital information is processed by arecognizer program22 that performs pattern recognition to recognize the command mark and convert the recognized command mark into an executable command, i.e., a command that may be executed by acomputer24. The executable command is either executed immediately bycomputer24, or, the digital information and the converted executable command are stored for later review and execution, respectively, on astorage device26 connected tocomputer24.
• Performing[0008]process10 enables a writer to write specific marks (or strokes) on a piece of paper, which are interpreted as executable commands that are either executed immediately by a computer processor, or executed at a later time.Process10 simplifies the input, recognition and subsequent management of written commands and digital images, using a very common and widespread method of storing information, e.g., by writing commands and information on a conventional medium, such as a piece of paper or cardboard, with a conventional writing implement, such as a pen or pencil. Pre-printed material may also be processed usingprocess10 where the pre-printed materials are supplemented with written command marks and then processed by therecognizer program22.
• Referring to FIG. 2, exemplary pieces of paper[0009]12aand12binclude several written command marks that may be recognized and interpreted byprocess10. In an embodiment according to the invention, command marks are categorized as notational, operational and transformational command marks. Notational marks typically indicate important phrases, dates and keywords. For example, on paper12a, astar symbol52 may indicate an important item, an underlinedphrase54,56 and58, may indicate a date, a heading or a keyword, respectively. Operational marks typically indicate specific operations to be performed by acomputer24. For example, on paper12a, a box symbol60 may indicate an item of information to be included in a task list (a “to-do” item), anenvelope symbol62 may indicate an item is to be mailed to another computer user, and afiling command symbol64 may indicate that the digital image file represented by paper12ais to be stored as a file onstorage device26. Transformational marks are typically used to indicate the transformation (a change) to a specific section of text included on a piece of paper, for example, on paper12b, a bold command mark66 indicates the circled section of text66ashould be re-formatted using bold text.
• Referring to FIG. 2B, an exemplary piece of paper[0010]12c, includes preprintedtext region90, aspecial command mark80 and secondary command marks82-84. In an embodiment,special command mark80 must be recognized byrecognizer program22 before secondary command marks,82-84 are recognized byrecognizer program22. In an embodiment ofprocess10,special command mark80 is written with a conventional writing implement, such as a pen or pencil, onto a conventional pre-printed medium, such as piece of paper12c(or, onto a pre-printed piece of cardboard or plastic), and the conventional medium12cis scanned as a digital image and processed byrecognizer program22. In this embodiment, recognizerprogram22 recognizesspecial command mark80 before recognizing secondary command marks82-84 in the digital image of paper12c. Ifspecial command mark80 is written on paper12c, then secondary command marks82-84 are recognized byrecognizer program22 and processed as executable commands which may be executed immediately or a later time. In an embodiment ofprocess10,special command mark80 is only recognized by recognizer program if written on a specific region of paper12c, for example, near a corner of paper12c.
• Recognizer[0011]program22 uses a pattern recognition process to identify the command marks. The pattern recognition process may be based on a statistical model, a neural network model, and/or a Hidden Markov model.
• [0012]Recognizer program22 may apply a machine learning process to better identify command marks from a particular writer. The machine learning process may use heuristic techniques to enhance the pattern recognition accuracy by determining common traits and tendencies included in previously recognized command marks made by a particular writer.
• The recognition of command marks may be intentionally constrained by recognizing only those command marks made in specific regions of[0013]paper30. For example, on paper12a,process10 may only recognize command marks written to the left of aside margin line68 or above a top margin line70.
• The recognized commands may be executed immediately by[0014]computer24, or may be stored as executable commands in a file onstorage device26. Similarly, the digital image file may be stored onstorage device26. By storing the command file and the digital image file, the writer has the opportunity to review both files for accuracy, and make corrections as required, before the commands are executed or the text is changed by executing a recognized command.
• As discussed previously,[0015]process10 may be performed by writing a command mark that is detected by a digitalink capturing device16. Detecting digital ink refers to a process of detecting the stroke of a writing implement on the surface of a digital ink capture pad. The detected strokes are then converted todigital stroke data20 which may be analyzed byrecognizer program22 in a manner similar to the scanneddigital image18 described previously. In an embodiment ofprocess10, a piece ofpaper12 is placed on top of a digitalink capture pad16 and a writer writes a command mark onpaper12.Capture pad16 detects the strokes made through the paper onto the capture pad and converts the stroke information intodigital stroke data20. Recognizerprogram22 recognizes the command mark represented by thedigital stroke data20 as an executable command and executes the command oncomputer24, or stores the executable command in a file onstorage device26.
• In an embodiment of[0016]process10, a conventional writing implement, such as a pen or pencil, includes a digital ink capturing device attached to the writing implement. Command marks are then written onto a conventional writing medium, such as a piece of paper or cardboard, and the digital ink capturing device outputs stroke information which is used by recognizerprogram22 to recognize command marks. An example of a digital ink capturing device that may be attached to a writing implement is an optical scanner attached near the tip of the writing implement, the optical scanner using a paper's surface patterns to detect movements and stroke information from the movement of the pen or pencil. Other types of digital ink capture devices could be used withprocess10 to capture digital stroke information based on command marks made with a conventional writing implement, such as a pen or pencil, onto a conventional writing medium.
• FIG. 3 shows[0017]computer24 on whichprocess10 may be implemented.Computer24 includes aprocessor30, amemory32, and a storage medium26 (see view36).Storage medium26stores data38 for digital image files and machine-executable instructions40 that are executed byprocessor30 out ofmemory32 to performcommand recognition process10.
• [0018]Process10 is not limited to use with the hardware and software of FIG. 3. It may find applicability in any computing or processing environment.Process10 may be implemented in hardware, software, or a combination of the two.Process10 may be implemented in computer programs executing on programmable computers or other machines that each include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage components), at least one input device, and one or more output devices. The input/output device may include wireless communications port to send and received signals to a digital ink capture device. The input/output device may include a device that communicates using a wireless protocol, such as a Bluetooth protocol.
• Program code may be applied to data entered using an input device (e.g., a mouse or keyboard) to perform[0019]process10 and to generate output information.
• Each such program may be implemented in a high level procedural or object-oriented programming language to communicate with a computer system. However, the programs can be implemented in assembly or machine language. The language may be a compiled or an interpreted language.[0020]
• Each computer program may be stored on a storage medium (e.g., a magnetic disk, RAM or ROM) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform[0021]process10.Process10 may also be implemented as a machine-readable storage medium, configured with a computer program, where, upon execution, instructions in the computer program cause a machine to operate in accordance withprocess10.
• In an embodiment,[0022]process10 is performed by writing a command mark on a conventional medium, such as a piece of paper, cardboard or plastic, with the command mark being written with a conventional writing implement such as a pen or a pencil and the written command mark on the conventional medium is then scanned as a digital image which is then processed by a recognizer program.
• In an embodiment,[0023]process10 is performed by writing a command mark onto a pre-printed medium, such as a pre-printed piece of paper or cardboard, with the command mark being written with a conventional writing implement such as a pen or a pencil, the pre-printed document with the written command mark is then scanned as a digital image which is then processed by the recognizer program.
• As used herein, the term “command mark” may refers to written alphabetic characters and also may refer to glyphs that represent non-verbal information (such as a circled letter, a star symbol, etc.)[0024]
• In an embodiment, the recognizer program uses a combination of pattern matching and image processing schemes with applied heuristics. The recognizer program may operate by performing pattern recognition using either a template-based recognition scheme or by employing a Hidden Markov Model (HMM) scheme combined with an appropriate feature extraction method. The following pseudo-code describes a template-based recognition scheme, which may be used to recognize command marks by broadly classifying them into “X-based” commands and “O-based” commands, and then analyzing the recognized command marks for further classification. The following pseudo-code may also be used to distinguish between a check mark and an X-mark, for example:[0025]
• 1. Segment the scanned image into connected areas.[0026]
• 2. For each such segment (corresponding to a letter, a word portion, a drawing element, or a command):[0027]
• [0028]2a. Is the image of the appropriate size? (Discard segments that are too small or too large, to avoid conflicts with letters and text);
• [0029]2b. Is the image in the right portion of the page? (To enforce a convention that command markes must be made in the left half of the image, for example);
• 3. Match the segment to a template:[0030]
• [0031]3a. Does this segment match an O-shaped command mark? If it does: Draw axes through a centroid of a bounding box surrounding the command mark. If most of the image data is at approximately the same distance from the center of the bounding box, this command mark is recognized as an O-shaped command pattern.
• [0032]3b. Or does this segment match an X-shaped command? If it does: Draw strips through a centroid of a bounding box surrounding the command mark. If most of the image data is within these strips, then this command mark corresponds to an X-shaped command.
• 4. Perform more analysis to distinguish the recognized command mark from other command marks. For example is the command mark an X-shape or an asterisk? Or, is it an O-shape or a cipher?[0033]
• In an embodiment, the recognizer program may be used to identify text printed on a conventional medium, as described previously. The recognizer program may include a scheme to recognize the printed text by recognizing at least one common characteristic associated with a section of a scanned image. For example, recognizing the uniformity of spacings (inter-word, inter-line, inter-paragraph) and/or recognizing that several letters share the same size or the same font attributes (ascenders, descenders, curves), and/or recognizing “neatness”, i.e., recognizing at least one margin is substantially aligned with another margin line (for example, where margin lines are horizontal, and/or parallel).[0034]
• The recognizer program may apply an image processing scheme that enables the recognition of image elements that are blurred and/or smeared through resolution changing mechanisms. By applying such a scheme in conjunction with Hough transforms, printed text of a certain size may be further recognized. Similarly, templates based on font attributes (curves of ‘d’, ‘b’, ‘p’, or ascenders for ‘k’, descenders for ‘g’, ‘j’, etc.) can be applied on each recognized text line to ascertain uniformity and classify elements as printed.[0035]
• The invention is not limited to the specific embodiments described above. For example, we mentioned a piece of paper as the medium on which command marks are made. However, any medium that can be written on and subsequently scanned could be used, such as cardboard, metal, plastic, cloth, etc. Also, we mentioned using a pen or pencil as the writing implement. However, any implement that can be used to make a command mark on a suitable medium could be used.[0036]
• Other embodiments not described herein are also within the scope of the following claims.[0037]

Claims (30)

What is claimed is:

1. A method comprising:

obtaining a scanned command mark written with a conventional writing implement onto a conventional medium; and

recognizing the scanned command mark as a command that may be executed by a processor.

2. The method ofclaim 1, wherein the command mark comprises one of a notational, transformational and operational mark.

3. The method ofclaim 2, wherein recognizing comprises:

recognizing a pattern associated with the command mark based on one of a statistical model, a neural network model, and a Hidden Markov model.

4. The method ofclaim 2, wherein recognizing further comprises:

applying heuristic techniques to enhance accuracy of the pattern recognition, the heuristic techniques being based on previous interpretations of a command mark.

5. The method ofclaim 2, wherein the command mark is recognized only if the command mark is placed in a specified area of the medium.

6. The method ofclaim 5, wherein a first command mark must be recognized before any other command marks are recognized as executable commands.

7. The method ofclaim 2, wherein the medium includes printed text, and wherein the recognized command mark is executed to affect the printed text.

8. The method ofclaim 2, further comprising:

executing the recognized command in the processor.

9. The method ofclaim 2, further comprising:

storing the recognized command in memory.

10. A method comprising:

detecting stroke information associated with making a command mark with a conventional writing implement on a conventional medium; and

recognizing the command mark as a command that may be executed by a computer processor.

11. The method ofclaim 10, wherein the command mark comprises one of a notational, transformational and operational mark.

12. The method ofclaim 11, wherein recognizing comprises:

recognizing a pattern associated with the stroke information based on one of a statistical model, a neural network model, and a Hidden Markov model.

13. The method ofclaim 11, wherein recognizing further comprises:

applying heuristic techniques to enhance accuracy of the pattern recognition, the heuristic techniques being based on previous interpretations of a command mark.

14. The method ofclaim 11, wherein the command mark is recognized only if the command mark is placed on a specific region of the medium.

15. The method ofclaim 14, wherein a first written command mark is recognized before any other command marks are recognized as executable commands.

16. The method ofclaim 11, further comprising:

executing the recognized command in the processor.

17. The method ofclaim 11, further comprising:

storing the recognized command in memory.

18. An article comprising a machine-readable medium that stores machine-executable instructions for recognizing a command mark written with a conventional writing implement onto a conventional medium, the instructions causing a machine to:

recognize the written command mark as a command that may be executed in a processor.

19. The article ofclaim 18, wherein the command mark comprises one of a notational, transformational and operational mark.

20. The article ofclaim 19, wherein recognizing comprises recognizing a pattern associated with the command mark based on one of a statistical model, a neural network model, and a Hidden Markov model.

21. The article ofclaim 19, wherein the command mark is recognized only if the command mark is written in a specific area of the medium.

22. The article ofclaim 19, wherein the medium includes printed text, and wherein the recognized command mark is executed to affect the printed text.

23 The article ofclaim 19, wherein the instructions cause the machine to execute the recognized command.

24 The article ofclaim 19, wherein the instructions cause the machine to store the recognized command in memory.

25. An apparatus for recognizing a command mark written with a conventional writing instrument onto a conventional medium, comprising:

a memory that stores executable instructions; and

a processor that executes the instructions to:

recognize a scanned image of the written command mark as a command that may be executed by a computer processor.

26. The apparatus ofclaim 21, wherein the command mark comprises one of a notational, transformational and operational mark.

27. The apparatus ofclaim 26, wherein the recognized command is executed by the processor.

28. The apparatus ofclaim 26, wherein the processor executes instructions to store the recognized command.

29. An apparatus for recognizing a command mark written with a conventional writing instrument onto a conventional medium, comprising:

a memory that stores executable instructions;

a processor that executes the instructions to:

obtain stroke data that corresponds to the written command mark, and

recognize the stroke data as an executable command.

30. The apparatus ofclaim 29, further comprising:

a digital ink detecting device which detects the command mark as stroke data.

Images