US20060088215A1 - Handwriting recognition system - Google Patents

Abstract

In order to improve the accuracy of recognition of hand-written input using a stylus, output signals from a plurality of accelerometers representing x and y axis acceleration and deceleration are sampled at a predetermined rate and passed through a digital bandpass filter to remove high frequency components and dc components arising from gravity. X and y vectors derived from the original x and y input signals are passed to a classifier using a hidden Markov model. Bandpass filtering improves the robustness of the interpretation of the vectors against stored templates which may be templates of individual characterisations or of whole words.

Description

RELATED APPLICATION
• This is a division of our copending commonly assigned application Ser. No. 09/914,262 filed Aug. 24, 2001 which was, in turn, a 35 U.S.C. §371 US national phase of PCT/GB00/01052 filed Mar. 21, 2000 and claiming priority from European application 99302270.6 filed Mar. 24, 1999.
BACKGROUND OF THE INVENTION
• 1. Field of the Invention
• The present invention relates to a handwriting recognition system and more particularly to a hardware and an algorithm for implementing such a system.
• 2. Related Art
• In PCT application number GB98/0316 (Publication No. WO99/22338) there is disclosed a portable computer in the form of a pen-type casing. Incorporated within the casing is at least one accelerometer which is used to detect movement of the pen with respect to its environment. By using the instrument for handwriting it is possible to effect data entry or transmission of signals reflecting movement, the user using either a pen tip mounted switch or a finger operated switch to indicate that movement is effecting a written input.
• There are many other pen-type input devices on the market in addition to stylus scroll pallets typically used in so-called palm top computers where handwriting recognition has been used. Such devices often require very precise movement which may not reflect natural handwriting movements for the user. One of the problems which makes characteristic handwriting recognition difficult is that while the underlying movement made by an individual to represent a particular letter may be consistent an element reflecting user movements due to stress and other factors will be present.
BRIEF SUMMARY OF THE INVENTION
• According to an exemplary embodiment the present invention there is provided a handwriting recognition system comprising means responsive to input analogue signals representative of movement of a handheld writing device, sampling means to provide signals representative of the acceleration of the writing device in at least x axis and y axis channels at a predetermined capture sampling rate, and filtering means to remove dc level components and to provide smoothing of the output whereby signals representative of movement of the pen over a period are supplied to a classifier for comparison with a template representative of characters formed.
• The classifier may use hidden Markov modelling (HMM) techniques using a large number of states to determine the character defined by movement. The system may include an input indicative of a user's intention that the movement is representative of character writing.
• According to a feature of the present there is provided a method of analysing signals from a moving handheld device, the method comprising sampling signals at a predetermined rate, passing signals through a bandpass filter to remove dc level and excess acceleration components, sampling the filtered output to provide a series of vectors representing the position of the handheld device at periodic intervals and using a classifier to compare the sample sets with predetermined templates to determine the character for output.
BRIEF DESCRIPTION OF THE DRAWINGS
• A handwriting recognition system in according with the invention will now be described by way of example only with reference to the accompanying drawing of which:
• FIG. 1 is a block schematic diagram of the system;
• FIG. 2 shows relative positioning of the x and y axis of the handwriting device ofFIG. 1;
• FIG. 3 is a schematic diagram of the handheld writing device ofFIG. 1 in a particular position;
• FIG. 4 shows relative input and output vector streams from the system ofFIG. 1; and
• FIGS.5 to9 show comparative templates for a number of different letters.
DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS
• Referring first toFIG. 1, aninput device1 such as a stylus produces x andy vector streams2 and3 which are fed into asampling unit4. The outputs x and y are generally from accelerometers or other position sensing devices within thestylus1. Also feeding thesampling unit4 is anoutput5 from a switch indicated here as being in the nib section of thestylus1 such that contact between theswitch6 and a surface is indicative of the stylus being used in a writing mode. It will be appreciated that thenib switch6 when incorporated in a non-surface contacting stylus, such as that disclosed in the previously referred PCT application, may be replaced by a user operable switch.
• The output of thesampling unit4, which samples the incoming streams at 60 Hz for example, is passed to abandpass filter arrangement7 and thence to a downsampling unit8 which produces digitised vectors x and y over a period of time. The x and y vectors are passed to aclassifier9, which uses a hidden Markov model to carry out a comparison between the vectors and templates representing written characters. Theclassifier9 may be arranged to output to avisual display10.
• More specifically, thestylus1 for example comprises a simple plastic casing containing the electronics for transferring information to a PC. Two accelerometers mounted in the stylus, for example, are used to produced the x and y outputs. The nib switch is a simple on/off switch connected to determine when pressure is being applied to the pen nib and can therefore detect when a pen forstylus1 is writing.
• Turning briefly toFIG. 2, the two accelerometers mounted in the top of the pen measure acceleration across their plane such that effectively they measure acceleration along thex axis11 and they axis12 of thewriting surface14. The acceleration measured by the sensors is made up of two components, acceleration due to gravity and the acceleration as a result of stylus movement. It will be appreciated that the acceleration due to gravity is always present, such that when the pen is exactly horizontal both sensors would measure acceleration of 1 g. As the angle of the pen to the horizontal changes (as shown for example inFIG. 3), the accelerometers are subject to Sin θ×1 g where θ is the inclination angle of thestylus1.
• The other component, acceleration as a result of the pen moving is produced by the acceleration and deceleration effect as the user writes.
• The acceleration of the two sensors x_totaland y_totalcan be expressed as
  y_total=y_g+y_movementandx_total=x_g+x_movement
  The remaining items ofFIG. 1 are incorporated in a computer unit, for example a PC, and three signals, as previously indicated,2,3 and5 being the two acceleration signals and binary signal from the pen switch are provided to the PC.
• In one embodiment the two acceleration signals are read into a normal PC using an RS232 port and the binary switch signal by means of the games port of a sound card.
• The sampling section must sample sufficiently regularly to capture the movement of the pen but preferably should not over-sample, which would result in a waste of processing and storage within the PC. It has been found satisfactory for the purposes of the current embodiment to sample at a rate of 60 Hz. The acceleration signals for each channel are read in as two byte words giving a dynamic range for each acceleration signal from 0 to 65535. The pen nib switch is similarly sampled at the same rate.
• As previously mentioned, the accelerometer signals are partly dependent on a component of the earth's gravitation field passing through the accelerometer of thestylus1. This results in an almost constant dc level present on the output corresponding to the average pen angle θ while writing. Thebandpass filter7 is thus arranged to filter the signals from the two accelerometers to remove the offsets. Additionally, the bandpass filter smoothes the output from the sensors thus correcting for instability introduced by the user so that the smoothed output from the sensors increases robustness and facilitates matching between the x and y vectors and stored templates.
• Turning now toFIG. 4, for each sample received on the PC from the accelerometers, the pen nib switch (or manually operable switch) indicates whether thestylus1 was being used in a writing mode or not. The downsampling process8 uses the information to down sample acceleration samples and to retain only those when it was known that thestylus1 was writing. Thus, considerFIG. 4, assuming that the x axis and y axis vectors are as shown at15 and16, then the down sampling vectors need only be taken into account when the nibswitch signal indicator17 is high. This will reduce the number of samples significantly so that the output from the down sampling process is a time series of two dimensional vectors x and y as indicated at18 and19.
• Having completed processing of the acceleration measurement from thestylus1 the vector stream is passed to theclassifier stage9 which takes in a series of vectors representing the acceleration measurements made within a given word. These are then compared to a set of templates which cover the range of words within the system vocabulary and the word which matches most closely with the unknown input word is deemed the recognised word. In this system the classifier is a hidden Markov model. Such models have been widely used in speech recognition and using a large number of states in the hidden Markov model will give the best performance for corresponding handwriting recognition.
• Thedisplay10, which displays the output from the PC allows display of a word, for example, on a screen.
• While the above handwriting recognition system is intended for use with a series of known words which, depending on the system vocabulary entered into the PC, may be a large number, it will be possible to use the same kind of system to validate single character entry. Using single character recognition and using cursive entry it is still possible to build individual words which may not be present in the vocabulary. There may be a lower level of confidence in words created rather than template determined. However, over time, the vocabulary may be expanded where multiple entries of the same word have occurred such that higher confidence levels may be achieved.
• A typical single accelerometer output can be seen respectively for the letters c, b, f and h in FIGS.5 to8. In each case the template developed here shows three entries on a single accelerometer for each of the letters.
• InFIG. 9, f, b, h and c are shown in comparison so that a suitable template for comparison may be derived. It will be appreciated that the combination of an x accelerometer trace and a y accelerometer trace will serve further to emphasise the difference between each input letter.
• Bandpass filtering in digital form to remove dc components and high frequency components increases the reliability of the recognition process and therefore the reliability of the interpretation of thestylus input1. It will be appreciated that where thestylus1 carries other components, for example an internal processing arrangement, some of the functions may be transferred from the PC to thestylus1. All of the components of sampling, bandpass filtering, down sampling and classifying can be implemented in a suitable computer program.

Claims (16)

1. (canceled)

2. A handwriting recognition system as inclaim 5, in which the classifier uses a hidden Markov model for comparison purposes.

3. A handwriting recognition system as inclaim 5, in which the sampling means, filtering means and classifier are implemented in a digital computer environment.

4. (canceled)

5. A handwriting recognition system comprising:

a stylus including at least one accelerometer to detect acceleration of the stylus with respect to gravitational pull and to provide analog signals representative of movement of the stylus;

a signal sampler responsive to the analog signals to derive a plurality of signal streams therefrom at a predetermined sampling rate, respective ones of said plurality of signal streams respectively representing acceleration of the stylus in at least an x-axis channel and a y-axis channel;

a bandpass filter acting upon the signal streams to remove low frequency and constant components caused by fixed angular gravitational pull on the stylus and to remove high frequency components caused by operator instability to provide a corresponding plurality of signal streams representing only wanted acceleration components; and

a classifier acting on the corresponding plurality of signal streams to produce data defining alpha-numeric characters.

6. A handwriting recognition system as inclaim 5 in which the stylus includes a switch which controls the signal sampler so that sampling occurs only when the switch indicates that the operator intends writing.

7. A handwriting recognition system comprising:

a stylus including two accelerometers respectively detecting acceleration of the stylus with respect to gravity in an x-axis and a y-axis direction and providing respective analogue signals representative of acceleration of the stylus in the x-axis and the y-axis direction;

a signal sampler acting upon the respective analog signal streams to derive a respective plurality of signal streams therefrom at a predetermined sampling rate:

a bandpass filter acting upon the plurality of signal streams to remove low frequency and constant components caused by fixed angular gravitational pull on the stylus and to remove high frequency components caused by operator instability to provide a corresponding plurality of signal streams representing only wanted acceleration components; and

a classifier acting on the corresponding plurality of signal streams to produce data defining alpha numeric characters.

8. A handwriting recognition system as inclaim 7 in which the classifier uses a hidden Markov model for comparison purposes.

9. A handwriting recognition system as inclaim 7 in which the signal sampler, bandpass filter and classifier are implemented in a digital computer.

10. A handwriting recognition system as inclaim 7 in which the stylus includes a switch which controls the signal sampler so that sampling occurs only when the switch indicates that the operator intends writing.

11. A handwriting recognition system as inclaim 2, in which the sampling means, filtering means and classifier are implemented in a digital computer environment.

12. A handwriting recognition system as inclaim 2 in which the stylus includes a switch which controls the signal sampler so that sampling occurs only when the switch indicates that the operator intends writing.

13. A handwriting recognition system as inclaim 8 in which the signal sampler, bandpass filter and classifier care implemented in a digital computer.

14. A handwriting recognition system as inclaim 8 which the stylus includes a switch which controls the signal sampler so that sampling occurs only when the switch indicates that the operator intends writing.

15. A method for processing accelerometer signals from a handheld movable electronic writing device, said method comprising:

bandpass filtering said accelerometer signals to reduce (a) dc components caused by gravity and (b) high frequency components caused by operator instability; and

thereafter analyzing the resulting bandpass-filtered signals to determine a recognized electronic writing output.

16. Apparatus for processing accelerometer signals from a handheld movable electronic writing device, said method comprising:

a bandpass filter arranged to process said accelerometer signals by reducing (a) dc components caused by gravity and (b) high frequency components caused by operator instability; and

signal analyzing and recognition structure arranged to analyze the resulting bandpass filtered signals to determine a recognized electronic writing output.

Images