US20030117378A1 - Device and system for retrieving and displaying handwritten annotations - Google Patents

Abstract

An apparatus for capturing and displaying handwritten annotations, for use with a computer system, comprises a transparent digitizing tablet adapted to cooperate with a stylus in order to capture handwritten annotations. It also comprises a transparent electro-luminescent display adapted to display captured handwritten annotations, the transparent digitizing tablet being superposed on the transparent electro-luminescent display.

Description

FIELD OF THE INVENTION
• The present invention generally relates to computer interfaces used to input information relating to a physical document. More particularly, the invention relates to a device for capturing and displaying handwritten annotations and a computer system using such a device.[0001]
BACKGROUND
• For most people, handwriting on a physical medium is the most natural manner of recording information. Thus, digitizing tablets are commonly used as input devices for computer systems allowing the entry of handwritten information into a computer. Moreover, in the prior art, several techniques have been described using a transparent digitizing tablet superposed on a display device, such as an LCD screen. Such a device allows an electronic form to be displayed by the LCD screen, through the digitizing tablet, and the strokes or annotations produced by a pencil-like stylus on the digitizing tablet to be displayed on the underlying display device. An example of these techniques is described in U.S. Pat. No. 6,212,297, by Sklarew, entitled “Handwritten keyboardless entry computer system”. Such a device is not adapted to be used with a physical document but only with an electronic document.[0002]
• Other types of interfaces make it possible to write with real ink over a physical document, in paper form for example, superposed on a digitizing tablet. This latter is enabled to digitize and capture information handwritten on physical documents. However, such a method degrades the integrity of original physical documents since the document is marked with real ink annotations.[0003]
SUMMARY
• A main object of the present invention is to overcome the above-mentioned drawbacks and to propose a new apparatus and system adapted to associate additional information with physical documents while keeping intact the integrity of the physical documents.[0004]
• In a first aspect, the present invention relates to an apparatus for capturing and displaying handwritten annotations, for use with a computer system, comprising a transparent digitizing tablet adapted to cooperate with a stylus in order to capture handwritten annotations. The apparatus comprises a transparent electro-luminescent display adapted to display captured handwritten annotations, said transparent digitizing tablet being superposed on said transparent electro-luminescent display.[0005]
• Thanks to the transparent electro-luminescent display and the transparent digitizing tablet superposed thereon, handwritten annotations can be marked and displayed on a physical document without physically marking the document. Thus, a user can make annotations over a physical document using “electronic ink” so that what is being “written” by the user on the physical document (on the digitizing tablet) can appear to the user electronically illuminated by the transparent electro-luminiscent display, on the same physical document.[0006]
• Such an apparatus enables the user to draw, sketch, annotate, construct, outline, enhance, erase, associate and display information directly over physical documents while keeping intact the integrity of said physical documents.[0007]
• In a second aspect of the invention, a system for capturing and displaying electronic handwritten annotations comprises:[0008]
• an apparatus as described above;[0009]
• a computer system;[0010]
• a stylus adapted to cooperate with a transparent digitizing tablet of said apparatus;[0011]
• said transparent digitizing tablet comprising means for digitizing annotations made by said stylus and means for sending digitized annotations to the computer system;[0012]
• said computer system comprising means for transmitting digitized annotations to the transparent electro-luminescent display of said apparatus; and[0013]
• said transparent electro-luminescent display comprising means for highlighting said digitized annotations.[0014]
• As mentioned above, such a system makes it possible to “write” over physical documents without physically marking said documents.[0015]
• According to a preferred embodiment of the invention, the computer system comprises means for storing digitized annotations sent by the transparent digitizing tablet. The handwritten annotations are stored in order to be retrieved and displayed in the future by the same or another user. Moreover, a user can handwrite several pages over a single form without physically marking it, and store said handwritten annotations in a computer system for later use.[0016]
• This information can be selectively displayed over the same form. According to a third aspect of the invention, a method for capturing and displaying handwritten annotations, using a system as described above comprises the following steps:[0017]
• digitizing annotations made by the stylus on the transparent digitizing tablet;[0018]
• sending digitized annotations to the computer system;[0019]
• transmitting digitized annotations to the transparent electro-luminescent display; and[0020]
• highlighting said digitized annotations.[0021]
• According to a fourth aspect of the invention, a method for retrieving and displaying electronic handwritten annotations on a physical document, using a system as described above comprises the following steps:[0022]
• identifying a physical document;[0023]
• identifying digitized annotations stored in an annotation table associated with the identified document; and[0024]
• sending for display the digitized annotations to the transparent electro-luminescent display.[0025]
• The invention also includes a computer program comprising computer readable instructions for carrying out the method for capturing and displaying handwritten annotations and/or the method for retrieving and displaying handwritten information according to the invention.[0026]
• The method for capturing and displaying handwritten information, the method for retrieving and displaying handwritten information and the computer program present the same advantages and the same characteristics as the apparatus and system which perform them.[0027]
• Other features and advantages of the invention will also emerge from the following description.[0028]
BRIEF DESCRIPTION OF THE DRAWINGS
• In the accompanying drawings, given by way of non-limiting example:[0029]
• FIG. 1 is a schematic view of an apparatus for capturing and displaying handwritten annotations according to a first aspect of the invention;[0030]
• FIG. 2 is a block diagram illustrating a system for capturing and displaying handwritten information according to a second aspect of the invention;[0031]
• FIG. 3 illustrates an algorithm for a method of capturing and displaying handwritten information according to a third aspect of the invention;[0032]
• FIGS.[0033]4 to6 illustrate the method for capturing and displaying handwritten annotations on a physical document;
• FIG. 7 illustrates an algorithm for a method of retrieving and displaying handwritten information according to a fourth aspect of the invention; and[0034]
• FIGS. 8 and 9 illustrate a method for retrieving and displaying handwritten information on a physical document;[0035]
DETAILED DESCRIPTION
• Referring first to FIG. 1, a description will now be given of an apparatus for capturing and displaying handwritten annotations in accordance with the invention.[0036]
• This[0037]apparatus10 comprises a transparentdigitizing tablet11 adapted to cooperate with astylus12 in order to capture handwritten annotations. Such atransparent digitizing tablet11 is commonly used to generate signals that are proportional to the coordinates of each point pressed by thestylus12. Digitizingtablet11 andstylus12 can be considered as electronic equivalents of pencil and paper used for writing, drawing, tracing or sketching.
• In a preferred embodiment of the invention, the digitizing tablet may be built from a substrate having a matrix formed by a plurality of first and second lines used to determine the position of the[0038]stylus12 when the latter is applied to the tablet. The first lines correspond to a plurality of scanning lines and the second lines correspond to a plurality of data lines. These first and second lines are made from a transparent conductive material, for example from indium tin oxide (ITO). The application of the stylus to the tablet creates an electric signal, the succession of electric signals corresponding to a drawing on the tablet. An example of transparent digitizing tablets it would be possible to use for implementing the present invention is produced by Wacom Technology Co. and sold under the name Wacom, PL Series, LCD pen tablet system. Such a tablet is used with a pen-like stylus. An example of a stylus it would be possible to use, in combination with the digitizing tablet, is Wacom's wireless, pressure sensitive UltraPen sold by Wacom Technology Co.
• According to the invention, the apparatus further comprises a transparent electro-[0039]luminescent display13 adapted to display captured handwritten annotations. Thisdisplay13 is of the type of a transparent organic bright light-emitting device. In a preferred embodiment of the present invention, thedisplay13 comprises a substrate having an array or a mesh formed by a plurality of first and second lines. The first lines correspond to transparent scanning lines and the second lines correspond to transparent data lines. These first and second lines cross one other and an electro-luminescent element is located at each intersection of the scanning and data lines. Such a light-emitting display, with a passive matrix, may be made of an array of Transparent Organic Light Emitting Devices (TOLED's) which are commonly used to create vision area displays on windshields, cockpits, and so forth. TOLED are today manufactured with standard silicon semiconductors. An example of light emitting foil technology it would be possible to use here is that used for the TOLED's manufactured by UNIVERSAL DISPLAY CO.
• In its basic form, a TOLED is a monolithic, solid-state device consisting in a series of molecules of small size forming organic thin films sandwiched between two transparent, conductive layers. When a voltage is applied across the device, it emits light. This light emission is based upon a luminescence phenomenon wherein electrons and holes are injected and migrate from the contacts toward the organic junction under the applied electric field. When these carriers meet, they form excitons (electron-hole pairs) that recombine radiatively to emit light. As a result, TOLED's are bright and can be directed to emit from either or both surfaces of the sheet. This is possible because, in addition to the transparent contacts, the organic materials are also transparent over their own emission spectrum and throughout most of the visible spectrum. Moreover, TOLED displays can be nearly as clear as glass, since they can be up to 85% transparent when not energized.[0040]
• As shown on FIG. 1, the[0041]transparent digitizing tablet11 is superposed on the transparent electro-luminescent display13. Thus, the apparatus comprises two superposed, functionally independent transparent foils. This apparatus consists in a portable transparent electro-luminescent digitizing tablet10 which can be superposed on any kind ofphysical document5.
• A shown in FIG. 1, the electro-[0042]luminescent display13 is adapted to emit light at thepoint1001 that is pressed by the stylus (see the arrow1002). Since the digitizing tablet superposed thereon is transparent, the light emitted by the display is transmitted through the tablet in order to appear on the top of theapparatus10. Moreover, since this display is transparent, it is also adapted to transmit light (see the arrow1003) from thephysical document5 through both thedisplay13 and thetablet11, so that the surface of thephysical document5 is visible on unmarked areas. Consequently, such an apparatus is perfectly adapted to write annotations in “electronic ink” and to display these annotations over thedocument5, without physically marking the document.
• As shown in FIG. 2, this[0043]apparatus10 is used with acomputer system20. Thecomputer system20 can be a personal computer, an Internet enabled cell phone, a Personal Digital Assistant, an onboard computer, a network computer, an Internet appliance or a wireless IP enabled device. Theapparatus10 can communicate with thecomputer system20 over an infrared link, a serial wired connection or any other communication means. Theapparatus10, thestylus12 and thecomputer system20 thus constitute a system for capturing and displaying electronic handwritten annotations.
• The[0044]apparatus10 comprises means14 for digitizing annotations made by thestylus12 and means15 for sending digitized annotations to thecomputer system20. In practice, apulse driving circuit16 alternately transmits driving pulses to the X- and Y-axis directions of the digitizingtablet11 for sensing the current position of thestylus12. Thepulse driving circuit16 is controlled by themicrocontroller15. The position of thestylus12 is detected by capacitive coupling sensed in the digitizingtablet11, that is thestylus12 senses a position signal in a potential distribution on the digitizingtablet11 using capacitive coupling, and provides the position to the digitizing means14 which is referred to as aposition sensing circuit14.
• The[0045]position sensing circuit14 receives the current X- and Y-axis coordinate data of thestylus12 and converts the coordinate data into digitized position data. The micro-controller15 transfers data of the position detected from theposition sensing circuit14 to thecomputer system20. Thus, themicro-controller15 is used for sending digitized annotations to thecomputer system20.
• Upon reception of position data from the[0046]position sensing circuit14, themicro-controller15 analyses the position data to calculate the current position of the stylus and updates thecomputer system20. This latter comprises means21 for transmitting digitized annotations to the transparent electro-luminescent display13 of theapparatus10. Thus the data of the position of thestylus12 is directly transmitted by thecomputer system20 to an electro-luminescentdisplay driving circuit17 controlled by thecomputer system20. The electro-luminescent display driving circuit provides X-, and Y-axis coordinate driving signals to the electro-luminescent display13 so that this latter can display the pixel on which is positioned the stylus through thetransparent tablet11.
• The[0047]computer system20 also comprisesmeans22,23,24 for storing digitized annotations sent by thetransparent digitizing tablet11. As described in detail in the following description, in reference to the method performed by the system, the storing means22,23,24 are adapted to store digitized annotations associated with a reference to a physical document which is used in relation with theapparatus20.
• As illustrated on FIG. 4, the[0048]apparatus10, connected to thecomputer system20 is used to create annotations relating to aphysical document5. Here, this document to be annotated is a book of several pages, entitled “Early British Kingdoms”. Any kind of document can be used, and for example, geographic maps, technical plans, commercial catalogs or even any other type of hard-copy, engraved, written or printed surfaces. Thephysical document5 could also be a blank paper sheet or a printed form to be filled in. The material of the physical documents can be paper, plastic, wood or any other material.
• As shown on the embodiment of FIG. 3, the method comprises first a[0049]step30 of selecting a physical document and assigning an identification number to it. Here, the identification number is a reference, such as Doc: 170301, which can be written on the document, for example on the first page of the book. A document's annotation table is then created in a creatingstep31. As shown on FIG. 5, the annotation table is created on thecomputer system20. It comprises the reference number of the document and possibly, other relevant information related to the document such as the title, author, ISBN (International Standard Book Number), date, and so forth.
• In a selecting[0050]step32, a page of the document, here thepage115, is selected and entered in the annotation table. This step corresponds to open a new page entry on the annotation table. An aligningstep33 is then performed wherein theapparatus10 comprising digitizingtablet11 and electro-luminescent display13 in a stack superposed over the selected page of the document. This location of the apparatus is clearly shown in FIG. 5. Since the apparatus comprises superposed sheets that are both transparent, the page of the document underneath the apparatus can be read through theapparatus10. Preferably, theapparatus10 is aligned with the borders of the page by some conventional means, for example by adjusting the upper left corner of the page with the upper left corner of theapparatus10.
• A writing[0051]step34 is then performed by using thestylus12 as shown in FIG. 6, in order to create annotations relating to the document. Any type of annotations (marks, texts, drawing, underlines, and so forth) can be handwritten by thestylus12 on the digitizingtablet11 of theapparatus10. These annotations made by a user can be directly related to the content of the page seen through theapparatus10 by virtue of its transparency.
• A digitizing[0052]step35 is adapted to digitize annotations made by thestylus12 on thetransparent digitizing tablet11. In practice, as illustrated in FIG. 6, the digitizingtablet11 takes samples, digitizes the sequence Xi, Yi of coordinates of each point pressed by thestylus12 on thetablet11 and sends this sequence to thecomputer system20. The succession of sequences of coordinates thus constitute a digitizing annotation sent to thecomputer system20 during a sendingstep36. Under control of thesame computer system20, the sequence of coordinates Xi, Yi are transmitted, i.e. sent back, to theapparatus10, and more precisely, to the transparent electro-luminescent display10. This transmittingstep37 is adapted to transmit digitized annotations in order to further highlight the digitized annotations during a displayingstep38.
• In practice, each of the point coordinates Xi, Yi received by the elctro-luminescent display is decoded and the corresponding row and column drivers of the matrix of passive electro-luminescent elements become active. When active, the element (or pixel) located at the intersection of the selected row and column emits light. Obviously, only the electro-luminescent elements corresponding to the stylus strokes sensed by the digitizing[0053]tablet11 must be continuously scanned and refreshed by thecomputer system20. Unmarked areas of theapparatus10 remain transparent, while only the areas marked by thestylus12 are highlighted in order to display annotations made by the user directly over the selected page of the document, without physically marking this page. Thus, the method described above permits the entering and displaying of annotations simultaneously.
• According to the preferred embodiment described here, the method further comprises a[0054]step39 of storing, in the annotation table stored in the computer system, digitized annotations sent by theapparatus10.
• FIG. 6 shows how the sequence Xi, Yi of point coordinates, corresponding to stylus strokes is stored on the[0055]computer system20, as a “Page Annotation File” (.PAF). that encodes the graphic information of the annotations made by the user. Once created, this .PAF file is referenced in the document's annotation table, associated with the selected document's page.
• In this embodiment, the name of the file to be referenced is composed of the document's reference (here 170301) followed by the page number (here 115).[0056]
• As an example, the annotation table associated with a document after annotating different pages (for[0057]example pages 3, 12, 57, 115, . . . , 213) would be as follows:

  DOG: 170301	Title: “Early British Kingdoms”	Author: David Gnash Ford of Infield, Berkshire, UK
  	Date: Jan. 28, 2000	ISBN: 84-344-0856-2
  Pg.: 3	File: 170301-003.PAF
  Pg: 12	File: 170301-012.PAF
  Pg: 57	File: 170301-057.PAF
  Pg: 115	File: 170301-115.PAF
  Pg: . . .	File: 170301- . . .
  Pg: 213	File: 170301 -213.PAF

• A method for retrieving and displaying electronic handwritten annotations on a[0058]physical document5 is now described with reference to FIGS.7 to9. This retrieving method can be performed by the same user as the one who has made the annotations over the document or by another user from a computer system memorizing the annotation table associated with the document.
• Firstly, the method of retrieving and displaying annotations comprises a step for identifying a physical document. In practice, a first selecting step[0059]70 is adapted to select a physical document by entering a reference number of this document. The user can enter, by means of any interface (keyboard, mouse, etc.) the reference number, here Doc: 170301. As shown in FIG. 8, theapparatus10 comprises here anarray18 of buttons, preferably touch sensitive, printed on a dedicated touch pad area of the apparatus, directly on the digitizingtablet11. The user enters the reference of the physical document via a button marked “Doc” and number buttons. When this reference entered in the apparatus is transmitted via themicrocontroller15 to thecomputer system20, this latter retrieves the annotation table associated with the selected document.
• Next, a second selecting step[0060]71 is performed to identify a page of thephysical document5, for example thepage115 of the book under consideration. The entering of the number of the page can be made as described above, via thebutton area18. The number of the page is transferred from theapparatus10 to thecomputer system20 via themicro-controller15 and in a retrieving step72 the computer system identifies the page annotation file (e.g. 170301-115.PAF) of the selected page within the annotation table associated with the selected document. During the selecting step70 and71 and the retrieving step72, theapparatus10 can be disposed in any way vis-á-vis the physical document.
• An aligning step[0061]73 is then performed for aligning theapparatus10 over the selected document page. This step is shown in FIG. 9. The upper left corner of theapparatus10 can be adjusted in a conventional manner with the upper left corner of the page. Thanks to the transparency of theapparatus10, the user can see the content of the selected document page through theapparatus10.
• Next, a step[0062]74 of identifying digitized annotations stored in the page annotation file is performed. In fact, in the page annotation file, all the annotations made previously over the page of the document are encoded as graphic information. Next, during a sending step75, thecomputer system10 sends the retrieved annotations to theapparatus10 for displaying the annotations via the transparent electro-luminescent display13.
• Consequently, the annotations that have been made by the user or by another user, are displayed by the[0063]apparatus10 directly over the physical document, without marking it. Thus, if two remotely located users share copies of the same document, annotations made by any one user over his copy can be transmitted via a communication network, by sending the annotation table relating to the document, to the other user. This latter can then see the same annotations over his document, through an analogous system shown in FIG. 2. If the physical document is a form to be filled in, this form can be filled in for different persons without physically marking the form. Moreover, it may be noted that, when the physical document is simply a blank sheet of paper, the methods and the system according to the invention can be applied to create a handwritten document or drawing which is memorized in a page annotation file to be displayed again by the user or another user.
• The above-described methods can be performed by microprocessors incorporated respectively in the[0064]apparatus10 and thecomputer system20.
• The[0065]computer system20 comprises also a Read Only Memory storing a computer program comprising computer readable instructions for carrying out the method for capturing and displaying handwritten annotations and for retrieving and displaying handwritten annotations.
• The[0066]computer system20 comprises a Random Access Memory adapted to store the modified parameters when the program is run, such as the Page Annotation Files associated with each selected physical document.
• The computer program could also be stored on a hard disk of the[0067]computer system20 or in a movable storage means such as a diskette or a CD-ROM which can be read by a reader incorporated in thecomputer system20.
• Persons skilled in the art will recognize that many variations of the teachings of this invention can be practiced that still fall within the claims of the invention which follow.[0068]

Claims (5)

I claim:

1. Apparatus for capturing and displaying handwritten annotations, for use with a computer system, the apparatus comprising:

a substantially transparent digitizing tablet responsive to a stylus for capturing handwritten annotations; and

a substantially transparent electro-luminescent display for displaying handwritten annotations captured by the digitizing tablet;

wherein the digitizing tablet is superposed on the electro-luminescent display.

2. The apparatus ofclaim 1, wherein the electro-luminescent display is a transparent organic light-emitting device.

3. A system for capturing and displaying electronic handwritten annotations, comprising:

a substantially transparent digitizing tablet responsive to a stylus for capturing handwritten annotations and a substantially transparent electro-luminescent display for displaying and highlighting handwritten annotations captured by the digitizing tablet, wherein the digitizing tablet is superposed on the electro-luminescent display;

a computer system;

means for digitizing annotations made by the stylus on the digitizing tablet and sending the digitized annotations to the computer system; and

means for transmitting digitized annotations from the computer system to the electro-luminescent display.

4. The system ofclaim 3, wherein the computer system includes means for storing digitized annotations received from the digitizing table.

5. The system ofclaim 4, wherein a digitized annotation is stored in the computer system in association with a reference to a physical document.

Images