SPECIFICATIONDisplay apparatusThe present invention relates to a display apparatus for displaying hand written symbols, such as alpha numeric characters.
In general, personal computers include three identifiable discrete components, namely a visual output device such as a display screen, a manual input device which is generally a keyboard, and a housing for the processing and storage components of the computer.
Many personal computers are used for handling text, generally by means of word processing software. However, the use of such personal computers for informal handling of text causes problems, especially to untrained personnel, since the ability to enter and manipulate text with speed and precision requires a high degree of typing skill.
For handling text, conventional computers have the disadvantage that input to the computer is in serial form. For the majority of applications, text input is provided by means of the keyboard so that the computer sees a serial string of numbers representing the desired alpha numeric characters. The practical consequence of this is demonstrated by the need for a cursor in order to indicate to the computer where the next character is to be placed. Also, an additional input device is required for graphics applications.
Another problem is that the input and output devices are physically separated. Thus, when entering text via a keyboard, there is a constant change of attention point between the keyboard and the display, which is fatiguing and limits efficiency. Because of the separation, such computers also occupy a comparatively large working space.
Much work is presently going into speech recognition/synthesis techniques for computers and, when developed, such systems would avoid the problems of large working space requirement and change of attention point mentioned above. However, input by speech recognition, would still represent a serial input to the computer, with the disadvantages mentioned above. Also, there will be situations in which speech recognition is not practical, for instance in an office full of people all trying to talk to their computers.
Light pens have been used to permit some degree of interaction between a user and a computer. A light pen is a device which cooperates with a raster-scan display to enable inputting of graphical information which is displayed on a visual display unit as the light pen moves over its surface. Light pens are also used to perform command and control functions, for instance selecting commands displayed on a screen, moving graphics and text from one position on a screen to another, etc.
However, light pens have certain disadvantages because of the mode of operation.
Speed of movement is limited if data is to be entered reliably because of the finite scanning speed on conventional displays. Also, resolution is limited because of the line-scanning and is generally less than one picture element (pixel). Thus, light pens and other position determining systems dependent on the display scanning process are unsuitable for inputting of conventional hand-written symbols.
According to the invention, there is provided an apparatus for displaying a symbol, comprising a display, first means independent of the display for determining the path of a writing implement on the surface of the display, second means for causing the display to display the path of the writing implement on the surface of the display in registration with the path of the implement, and third means for detecting when the path corresponds to a predetermined symbol and for causing the display to display the predetermined symbol in place of the path.
The display is preferably a flat display ie with a thickness which is much smaller than the height and width of the surface providing the display. Thus, an outline similar to a book or writing pad may be achieved. The display itself may comprise a liquid crystal display, plasma display, or any other suitable display.
A liquid crystal display has the advantage that no potentially or allegedly harmful radiation is emitted. Again, much work is currently being done on producing flat-screen displays and any display capable of displaying the path of the implement may be used.
The position determining means may determine physically the position of the implement on the display surface or screen. The position determining means may also be capable of determining the position of the implement on a surface outside the screen surface, for instance to permit the digitization of existing drawings. Techniques which rely on eiectromagnetic or acoustic signals, pressure, and the like may be used.
The writing implement need not be a part of the apparatus and may be a conventional pen, preferably a pen which does not dispense ink since the display itself shows the path followed by the implement. With suitable position determining means, the implement could even be a human finger.
The third means preferably includes character recognition means for recognising alpha numeric characters. Other characters, such as shorthand or Japanese characters or mathematical symbols could also be made recognisable by the use of the appropriate character recognition means. Such means may operate in any suitable way, for instance on the basis of the shape of the path or on the basis of the strokes used to form the path.
The apparatus may form part of a computer including a processing unit and storage. Such a computer may be used for word processing or other handling of text, or even simply for storage of text. In a preferred embodiment, the writing implement is used in the way of a conventional pen or pencil to "write" on the display surface or screen of the display. The display produces a visual output of the path followed by the implement, just as though the implement were a conventional pen leaving a trail of ink on paper. At a suitable opportunity, for instance after recognition of each character or of each word (each blank space) or at the end of each line, recognised characters are substituted on the display in place of the actual path of the implement.Such characters may be in the form of a conventional dot matrix representation of the Roman alphabet and Arabic-based numbers in common usage throughout the world. However, for particular applications, different character recognition means and different "fonts" of characters or graphic symbols may be used.
The apparatus may be arranged to scale the recognised symbols automatically. Thus, when data characters are recognised, the replacement symbol may be the same size as the hand-written character or may be taken from a set of predetermined sizes.
Any suitable recongition means may be used. Although present character recognition means cannot adequately handle connected writing but require the input of text in separate characters, future developments should improve the recognition abilities with connected writing.
Another advantage of such an embodiment is that text and graphics may be mixed. The conventional form of input corresponding directly to the use of pen and paper avoids the problem of serial input of symbols and allows graphic symbols, mathematical formulae, and even conventional drawings to be mixed with text in the same way as is possible with conventional pen and paper. For entering technical drawings, such as circuit diagrams, the recognition means could be arranged to recognise conventional symbols, for instance for transistors, resistors, and other components and to substitute standard symbolic representations of these so as to produce a neat circuit diagram.The recognition means is preferably arranged to allow control and command functions to be recognised and performed, such as erasure of symbols, moving of symbols ("picking" and "dragging"), and selection from a command menu.
An apparatus of this type removes the need for conventional typing skills and makes computing more accessible to untrained personnel.
problems with change of attention point are avoided because thd apparatus can simulate very closely conventional writing by means of a pen or pencil with the input and output occurring at essentially the same point in space.
Also, by combining input and output devices, the space requirement can be substantially reduced.
Such an apparatus has many applications and some examples of such applications are: automatic translation of shorthand into text; access to personal computers for non-european language users, for instance Japanese users, in their own language; programming and representation of information using graphical and/or mathematical symbols; and teaching of handwriting to children.
A dim graticule may be displayed on the display in order to guide writing by means of the implement. In order to provide multi-colour input, an electronically switched graticule may be used to select the input colour. For instance, each graticule may be coloured in accordance with the colour which can be input when that graticule is selected. This technique again is very similar to conventional use of different coloured pens or pencils, where the pen or pencil has to be exchanged for another when it is desired to change colour. This represents a considerable advantage and is no less convenient than other forms of "pallet selection" used for multi-colour graphics with conventional computers.
The character recognition means may be of a programmable or "trainable" type, in which, for instance, a symbol is drawn once or several times and the character recognition means is informed of which corresponding "formalised" symbol corresponds thereto. Once this basic training or programming has been finished, drawing of the symbol on the display by means of the implement will thereafter result in the path of the implement on the display being replaced by the formalised symbol.
The invention will be further described, by way of example, with reference to the accompanying drawings, in which:Figure 1 shows a display apparatus constituting a preferred embodiment of the invention;Figure 2 is a cross-sectional view of the apparatus of Figure 1;Figure 3 is a functional block diagram of the apparatus of Figure 1;Figure 4 is a more detailed block diagram of the apparatus of Figure 1;Figure 5 shows a writing implement of the apparatus of Figure 1;Figure 6 is a cross-sectional view of part of the implement of Figure 5;Figure 7 shows a detail of Figure 6 to an enlarged scale;Figure 8 is a block diagram of part of the apparatus of Figure 1;Figures 9A and 9B are a flow diagram illustrating operation of the apparatus of Figure 1;Figure 10 illustrates a possible screen layout;Figure 11 illustrates some control character operations; andFigure 12 is a flow diagram illustrating oper ation of a character recognition system learning phase.
The display apparatus shown in Figures 1 and 2 comprises a wedge-shaped housing 1 with rear feet 2 which may be adjusted for height. Most of the sloping major surface of the housing is taken up by a large flat display 3, for instance of LCD type, covered by a thin sheet of protective glass 4. The surface of the sheet 4 has a thin transparent conductive film which is electrically connected to earth, and a faint switchable graticule 5. A pen 6 and sensors 8 and 9 are provided for a position determining system for determining the position of the pen on the surface of the sheet 4. A slot 7 is provided for housing the pen when not in use.
The sloping face also carries function switches 10, 11, a removable indicator strip 12 for the function switches, cartridge slots 13 for insertion of external cartridges holding preprepared programs, a loudspeaker 15 for a speech synthesiser, and light emitting diode indicators 16. The side of the housing has a disc drive 14 for mass storage and a power switch 17. The front edge of the housing carries a connector 18 for an external keyboard, which may more conveniently be used by touch typists who can type faster than they can write.
Figure 2 illustrates the internal construction of the apparatus. The electronic circuitry is carried by one or more boards 19 fixed to the bottom of the housing by supports 20. A recess 21 is provided at the rear of the housing and contains various external connectors 22.
As shown in Figure 3, the apparatus comprises a processing system 31 which contains a character recognition system 32. The processing system 31 has various interfaces including a serial input/output port 33, a cartridge controller 34 for magnetic tape cartridges 24 or the like, a mass storage interface 35 for the disc drive 14, a speech synthesiser 36 for the loudspeaker 15, a display driver 37 for the display 3 and for an external monitor line 38, a coordinate digitizing interface 39 for the pen 6, and a switch and keyboard interface 40 for an external keyboard 23 if required. With the exception of the coordinate digitizing interface 39, the other parts are generally conventional and frequently provided in personal computers. Figure 4 illustrates in more detail the hardware which constitutes the apparatus shown in Figure 3.
The pen 6 has a body which contains a switch 130 and a tip 131 which acts as a point source of ultrasonic radiation. The ultrasonic wave front emitted by the tip is received by the sensors 8 and 9 and, by measuring the time delay between transmission and reception of an ultrasonic burst, the pen position may be determined as indicated hereinafter. The tip 131 is made of metal and is earthed by the coating on the glass sheet 4 so as to determine when the pen touches the display surface. An ultrasonic transmitter 132 surrounded by absorptive rubber 133 communicates via a channel 134 with the tip 131.
As shown in Figure 7, the tip 131 has a small convexly tapered reflector 135 for reflecting the ultrasonic signals from the transducer 132 generally transversely of the pen and in all directions. The tip 135 is fixed to the body by means of thin supports 136 and a connecting tag 137. The diameter of the tip is approximately equal to a wavelength at the frequency of the ultrasonic signal.
Figure 8 illustrates the coordinate digitizing system in more detail. This system comprises a timing generator 150 which supplies timing signals for clocking, clearing, and starting counting of two counters 151 and 152. The timing generator receives clock signals on an input line 153 from the microprocessor timing generator shown in Figure 4. The timing generator 150 also supplies an ultrasonic signal via a bandpass filter 154 to the pen 6 via a control circuit comprising an operational amplifier 155 resistors RA, RB, and RF, and an electronic switch 156 controlled by a start burst signal from the generator 150. Thus, the ultrasonic transducer transmits the ultrasonic signal continuously but, at the start of each digitizing step, the switch 156 is closed temporarily to provide an increased intensity or transient level.
The sensors 8 and 9 are connected to amplifiers 157 and 158, bandpass filters 159 and 160, detectors 161 and 162, and comparators 163 and 164, respectively. The sig nals received by the sensors are thus processed and the comparators compare the envelopes of the signals with a reference voltage Vref. When the reference voltage is exceeded, the comparators 163 and 164 stop the counters 151 and 152. Latching of the count is controlled by a latch signal from the timing generator 150. The tip 131 of the pen 6 is earthed when the tip is in contact with the transparent conductive coating on the glass sheet 4 and this is used to indicate pen up/pen down.The counters 151 and 152 are each twelve bit counters and are cleared and instructed to start counting the clock pulses coincident with the start burst emitted by the transducer 132 after a delay to allow for the delay introduced by the operation of the bandpass filters 159, 160 and detectors 161, 162.
The counters are stopped upon detection of the start burst signals by the sensors 8 and 9so that the counts held represent the distance from the sensors 8 and 9 to the pen tip 131.
In order to convert these positions to cartesian coordinates, the microprocessor is preferably arranged to determine the x and y coordinates from the following equations: x = (D12 - D22)/4a + a (D12 - x2)1/2 where D1 and D2 are the counts in the counters 151 and 152 and a is half the spacing between the sensors 8 and 9.
Figures 9A and 9B illustrate operation of the display apparatus. Upon switching on the apparatus, following certain initialising and housekeeping functions, the apparatus proceeds from the start block 200. Block 201 determines whether a character has been started and, if not, loops back until a character has been started. Block 202 determines the position of the pen tip as described hereinbefore and stores this in display memory so that the display displays a path following the movements of the pen tip. The apparatus then enters a character recognition routine.
In the character recognition routine, movements of the pen tip are separated into individual strokes at block 203. Until the end of a character is detected in block 204, operation cycles back to block 202. Upon detection of the end of a character, the salient features defining the character are extracted in block 205 and a dictionary of characters is searched in block 206. If the search is successful and the character is recognised, the operation moves on. If not, the pen trace on the display is replaced by an indication that an unrecognised character has been written in block 208, the unrecognised character is placed into a buffer in block 209 and control returns to block 201 for the next character.
When a character is recognised, block 210 determines whether the character is a data character or some other character, such as a control character. If the character is a control character, block 211 determines whether it is a valid control character and if not informs the user in block 212 and returns to the start block 200. If a valid character is detected, the desired control operation is performed in block 213 and the pen trace is removed from the screen in block 214.
If a data character is recognised, block 215 determines whether the character is in a command window as will be described hereinafter.
If not, the character written by the pen is replaced, in block 216, with a formalised display of the character, the character is placed in a data buffer 217, and the contents of the buffer are passed in block 218 for further processing, after which the operation returns to the start block 200.
If the data character occurs in a command window, the pen trace is replaced by a formalised display of the character in block 219 and the character is placed in a command buffer in block 220. Block 221 determines whether the character is a correct valid command or part thereof, and, if not, the user is informed at block 222 and control returns to the start block 200. If valid, block 223 determines whether the command is complete. If not, control is passed to the start block 200.
If the command is complete, the desired operation is performed at block 224 and control returns to the start block 200.
Figure 10 illustrates a typical screen display partioning. The screen has a command area 325 in which characters may be written by means of the pen for causing certain commands to be performed. The four corners 326 of the display are allocated to scroll functions in the directions indicated by the arrows and placing the pen tip in any of these corners will cause corresponding scrolling of the display.
A status area 327 is set aside for allowing the apparatus to display status and other information to the user, such as errors. placing the pen in the status area normally has no effect. A data region 328 is provided for writing characters for data or control by. means of the pen. One or more menu areas 329 may be provided in the command area 325 to allow such commands to be executed instantly by placing the pen tip at the relevant menu area.
In order to allow free-hand graphics to be entered, the apparatus may be arranged, upon receipt of a command written in the command window or upon actuation of a programmable function switch, to disable character recognition. In this mode, the apparatus functions in much the same way as a conventional graphic system, for instance based on "picking" and "dragging".
Figure 11 illustrates various control characters which control the display of data. Thus, rapid horizontal movements erase a character and permit insertion of a fresh character. Encircling of one or more previous characters allows a block operation to be executed, such as move, delete, and the like. Rapid vertical movements followed by a path crossing previous characters causes a text line to be deleted. Touching a character and performing a curved path causes the character to be rotated.
Prior to normal operation of the apparatus, the character recognition system may be trained to recognise characters written by a particular user. Figure 12 illustrates a method of performing such training. Following a start block 400, a user enters a character representation code sequence for selecting a specific character in block 401. For instance, the AS Cll code may be entered to select a symbol which is to be programmed. The user may then produce a desired replacement graphic display from an inbuilt graphic editor in block 402. The user then provides a number of hand drawn examples of the character by means of the pen on the display at block 403. The features of the hand drawn examples are extracted and, in block 404, a test is performed to assess whether these features clash or conflict with those already existing. If so, control returns to the block 403 for another attempt. If not, the features are stored in a dictionary in block 405 and the graphic and character representations are stored in block 406. This routine may be repeated for all characters which are required to be recognised.
Various modifications may be made within the scope of the invention. For instance, the ultrasonic means for digitizing the position of the pen on the display may be replaced by alternative systems. One example is a spark generator provided at the tip of the pen and controlled by the timing generator.