US20080119271A1 - Hand-Held Sensor Devices for Use with Printed Material - Google Patents

Abstract

Interactive printed material and sensor apparatus for use therewith is described. The sensor responds to characteristics of printing on the printed material, for example the degree of infrared reflectance or absorption. The printed material has intelligible components and non-intelligible components. By configuring the printed material and/or the internal programming in the sensor device appropriately, the sensor device can be calibrated to the printed material automatically as the sensor device is applied to the printed material as part of a structured intellectually mediated interaction between the two. In one example, the sensor device is run across a sequence of pre-printed areas specifically designed to enable the values of the property to be presented in a specific order enabling sophisticated calibration and programming functions to be implemented.

Description

• A number of proposals have been made, and some commercial products have been successfully produced and marketed, in the educational and amusement field providing a book or the like associated with a pen, wand, or like device, which is used in conjunction with the book. By arranging for the pen or wand to detect a property of the page, a suitable interactive relationship can be established between them which, in the context, may have an amusement or educational value.
• WO-A-88/05951 and WO-A-83/02842 describe systems of this type. Although disclosing that other interactive mechanisms may be used, these specifications disclose in particular hand-held sensor devices which are configured to detect the infrared reflectance or absorption of the portion of the printed material located immediately adjacent the head of the sensor device.
• In order for such combinations of sensor devices for printed materials to work effectively with one another, substantial care must be taken both in the printing of the printed material and in the calibration of the sensor device. It is wholly commercially unsatisfactory for a sensor device to work properly only with a single piece of printed material. Rather, the need is for the sensor device to work with numerous different pieces of printed material which may indeed have been printed at different times and in different places, even by different printing methods. This is difficult to achieve in practice and, in particular, leads to substantial restrictions in the nature and scope of interactive activity which can be contemplated between the sensor device and the printed material, because the number of different values of a property of the printed material, e.g. reflection of infrared radiation, is limited to a very small number, usually 3 or 4.
• WO 2005/013237-A relates to improvements in interactive printed material and sensor apparatus for use therewith and discloses sensor devices which are configured so that they may operate in a calibration mode, or even a recalibration mode, to take into account variations between different printed materials. While some details are disclosed in this earlier application as to how such calibration or recalibration may be carried out, they rely on the user of the interactive sensor device and printed material consciously applying the sensor device sequentially to a number of differently identified areas. This is a disadvantage since the preliminary task of consciously calibrating the sensor apparatus is not in itself stimulating to or enjoyable for, the user, and accordingly it can simply be ignored, which may not immediately have any adverse consequences, but which can lead, e.g. as a game is being played or a quiz sheet worked through, to instabilities, errors and eventually failure of the system to work properly as designed.
• In accordance with the present invention, the printed material, and/or the internal programming in the sensor device, is arranged to ensure that calibration of the sensor device to the printed material is carried out automatically as the sensor device is applied to the printed material as part of a structured, intellectually mediated, interaction between the two.
• As indicated in the published disclosures referred to above, interactive sensor devices and printed materials may be configured in a myriad of ways providing educational or amusement materials. In all of these, the printed material contains intelligible printing, i.e. text and/or pictorial matter which can be read and viewed by the user and which has meaning, and non-intelligible material, the presence or absence of which, or the degree of presence of which, can be picked up by the sensor. Put very simply, the printed page contains information which is visible and intelligible to the human observer and other printed information which is “invisible”.
• One way in which the present invention can be put into effect is to design an interactive game which requires the user to place the sensor on a given number of differently reactive areas (i.e. differently reactive as far as the sensor is concerned) of the printed material. If, for example, the sensor is programmed to discriminate between six different levels of infrared absorption, then the game structure may be such as to impel the user to apply the sensor successively to six differently infrared reflective areas.
• One form of attractive game is that of a track or maze which must be navigated by the user. If this contains an identified “start” and the initial instruction is, e.g., “walk up the steps one at a time to start the game”, the steps may be printed with different infrared reflectivities. An alternative is to provide some form of track which means that the user is graphically driven to pass the sensor head successively over differently reflective areas, for example by having the sensor pass over a picture of a bridge or a pedestrian crossing to enter a graphically illustrated “field of play”.
• Yet a further approach is to provide that the sensor unit will not work in an interactive fashion with the product, for example by not reacting to a printed speech bubble (which may or may not have any words printed in it) until the step of applying the sensor to differentially reflective areas has been undertaken. This may involve, for example, simply moving the sensor around randomly on the page.
• An alternative approach which, as will appear below, brings with it substantial advantages particularly in terms of operation of such devices for sophisticated game play, or for use in quiz book or revision test exercise applications, is to programme the sensor device so that it records different values of a sensed property of the printed material and then processes the record of the different values to extract from it information as to the different values themselves.
• Thus, according to a specific feature of the present invention, there is provided interactive information apparatus consisting of printed material and a sensor device adapted to respond to characteristics of printing on the printed material, and wherein the printing on the printed material has intelligible components and non-intelligible components, the latter being sensed by the sensor device, and wherein the sensor device senses differing values of a property of the printed non-intelligible image, and the device includes a processing unit adapted to recognise maxima and minima of the values of the sensed property following the sequential continuous application of the sensor device to a sufficiently large number of differentially (unintelligibly) printed areas of the printed material.
• Operating in this way, the specific levels which are maxima or minima correspond to actual values which are all different and which can be set by the sensor device itself as the specific levels it is thereafter programmed to recognise, i.e. the sensor will recognise printing as possessing a property at a specific level if the property sensed is at or within a preset tolerance band of the level determined by the analysis of maxima and minima as described above.
• The detection of the levels and the self-calibration or recalibration thereby achievable in the sensor device is valuable as enabling the device to adapt to the printed material and accordingly to adapt to printed materials produced by different producers and at different times. This recalibration can occur irrespective of the sequence in which the various levels are detected by the sensor device. However, as will appear from the discussion below, it is particularly preferred to present those levels in a definite fixed sequence because that can enable the sensor device not only to calibrate itself, but, more particularly, to change operational mode from a default mode or some earlier operating mode to a new operating mode. Having read the code, it can also say a word or phrase which corresponds with the graphics.
• This is of particular value in the sophisticated area of use of interactive printed material and sensor devices for quiz and early learning books where the rules for interaction between the sensor pen and the printed material may change e.g. from page to page or from one printed book or worksheet to the next. It is straightforward to provide very substantial quantities of stored programme material in a sensor pen of conventional construction including a variety of different programmed modes of operation, each of which may chosen to drive the way in which the pen operates.
• Applying the pen to the printed material areas in a desired sequence may be secured by the techniques noted above, for example by some form of pictorially or graphically driven track which has the individual printed areas in the appropriate sequence, by providing all the sequences embedded in the sequential areas which the user applies the pen to in accordance with the instructions on the page or the questions and answers.
• As generally described above, the sequential input of different values enables the sensor device to self-calibrate. There is, however, a set of games where calibration is not required at all. These work by comparison alone. For use in such games, the sensor module is preset with one threshold which can thus differentiate between “right” and “wrong”. This is factory set in software at a level where readthrough and other tolerances are taken into account—between 5% and 10% carbon black. Any patch with a carbon black content greater than this threshold is deemed to be ‘correct’ and is remembered. 4 or 5 different levels can then be used for correct answers, enabling various games to be played including sets, sequences, matching, tracking and mazes where simple sequences are analysed and given appropriate responses. It is possible to use a similar approach (threshold between “right” and “wrong”) at the start of a revision test, but this time calibrating the correct answers once a whole set has been collected.
• Specific examples of how the present invention can be put into practice will now be described with reference to the accompanying drawings. In these drawings:
• FIG. 1 is a representation of a play board and frog used to play a game;
• FIG. 2 is a diagrammatic illustration of part of a track printed on a game board;
• FIG. 3 is a graphical illustration of a sensed value response of a sensor passing along the track ofFIG. 2;
• FIG. 4 is a block circuit diagram showing a circuit useful in sensor devices according to the invention; and
• FIG. 5 is a circuit diagram showing a preferred form of circuit for use in a sensor device according to the invention.
• Referring first toFIG. 1, this shows a frog located to one side of a printed image of five lilypads in a pond. The sensor unit is encased in a plastic frog which, when applied to any part of the printed board, gives out an audible signal. Applying the frog to different areas of the board which have different infra-red absorption levels, generates a different signal. Internally, the frog may be programmed to operate the game, as explained below.
• As shown inFIG. 1, there are five lilypads which have been printed using five different tints of infrared absorbent ink, and which are also differentiated by means of colours or numbers so that they look different to the player. The player starts the game by placing the frog on the board, actuating a switch which turns the internal electronics on and which causes the frog to say “Jump to any pad”. The frog is then lifted off the board and placed back on it by the player, preferably on a lilypad (the background is the same level as one of the lilypads). The sensor module in the frog records the level read and confirms the “landing” with positive audio/visual feedback. The frog then says “Jump to another pad”. If the frog is now placed on a new pad, the new landing is confirmed as before and the new level recorded too. However, if the frog jumps to the same pad again, then the level read will be the same, a negative feedback is given followed by “Jump to another pad” again. Eventually, the infrared absorption levels for all five lilypads will have been recorded. The sensor module can now assign the levels with predetermined attributes—i.e. the highest level with, say, “yellow”, the next highest with “green”, etc. The game continues with the frog randomly calling out a colour. The frog must now be landed on the specific pad. The sensor module compares the latest reading with the remembered level, and feeds back accordingly. The game is made challenging by speeding up and by recording the number of hits within a time limit.
• Turning now toFIGS. 2 and 3,FIG. 2 is a typical printed code which can be present on a printed sheet and which consists of a sequence of areas of infrared absorptive ink printed at different densities. The infrared absorption detectable by a sensor will accordingly vary depending upon which of the areas the sensor is looking at. The technique can, of course, be applied to other properties than infrared absorption.
• As evident fromFIG. 2, the successive areas will give different levels when a sensor is passed across them and, as can be seen, this will actually produce an output as diagrammatically illustrated inFIG. 3.
• The specific actual measured values of absorption are not important. What can be seen very clearly fromFIG. 3 is that, as a sensor is passed from left to right across the area shown inFIG. 2, the infrared absorption starts (when the sensor is on the white paper) at zero, goes up to a level arbitrarily denoted as 5 at A, then drops to a lower level arbitrarily assigned thevalue 1 at B, then up again tolevel 4 at C, down tolevel 2 at D and up tolevel 3 at E.
• This produces a sensor sequence 51423 in terms of the output. The absolute levels of absorption can be set in an entirely arbitrary fashion, but the relative levels need to be set to allow sufficient differentiation, which can obviously be achieved without too much difficulty.
• The reading of these levels in that sequence can be by way of sliding a sensor across the series of areas as shown inFIG. 2, or it may be, for example, applying the sensor successively to such areas.
• As explained above, if the sequence is maintained, then, as well as enabling the sensor device to self-calibrate to the individual levels (which it can detect via the maxima and minima which are obvious from looking atFIG. 3), the sequence in which those maxima and minima occur can in fact represent a code which can be decoded and applied to change the operational mode of the sensor device.
• As will appear from the discussion below, the sensor device may thus act both in self-calibrating and in decoding fashion. The coding is there, but it is there in a way which also allows the sensor device to learn the absolute print levels without the sensor device having any prior knowledge of those absolute levels. The coding mechanism, however, not only allows unique identification of each printed level, but produces a unique code sequence which can be used when decoded to make the sensor respond by operating, e.g., in a particular programmed mode.
• Turning to the specific example shown inFIG. 2 and which produces the coding sequence shown inFIG. 3 of 5-1-4-2-3, it can be seen that this will act to identify a particular operational mode for proper interaction between the sensor device and the printed material bearing the marking as shown inFIG. 2.
• Applied to the easily practically achievable approach of printing with infrared absorptive black ink on to white paper, it is straightforward to provide seven discrete levels of response from an infrared sensor. These seven levels include white (where there is no absorbent ink present), black (where the absorbent black ink is present at an arbitrarily defined 100%) and a number of intermediate levels designated1 to5 and where the amount of infrared absorptive black ink is, relative to the amount used for the 100% black areas which are printed, 8% forlevel 1, 17% forlevel 2, 27% forlevel 3, 38% forlevel 4 and 54% forlevel 5. Printing at these levels with infrared absorptive black ink can be rendered essentially undetectable if the printing is over coloured areas which are printed with conventional cyan, magenta and yellow printing inks (all of which are non-infrared absorptive).
• As can be seen very simply, measuring each successive level as a maximum or minimum as illustrated inFIG. 3, a variety of different sequences can be envisaged. Operating on the simplest approach that one wishes to generate a sinuous response as shown inFIG. 3, each sequence of responses must be “high-low-high-low-high-low”. If the sensor is moved across a sequence of areas as shown inFIG. 2 and as designated onFIG. 3, the following code sequences which meet this sinuous requirement are:

• ABCDE

  	0513240
  	0514230
  	0523140
  	0524130
  	0534120
  	0413250
  	0415230
  	0423150
  	0425130
  	0435120
  	0315240
  	0324150
  	0325140
  	0314250
  	0214350
  	0215340

• The software within the sensor device may be programmed to react, once the various levels of reflectants have been read, i.e. once a sufficient number of maxima and minima have been detected, to store those levels for use in subsequently classifying an unknown response when the sensor device is placed against paper and, separately, because the sequence can be identified, the “code” can be identified.
• Using this technique of identifying the maxima and minima, in order to detect the first level in any particular sequence, all that is required is that the previous level is lower than that first level. Likewise, the final level detected only requires that the subsequent level is lower than that level to identify it as a maximum. Accordingly, on printed material, the level before and after the sequence of differentially printed areas does not have to start with zero absorption (i.e. white paper). The code levels can, of course, be read in reverse, and it is possible to operate if desired in a fashion where it does not matter in which order, reversed or not, the successive maxima and minima are detected, but that does reduce the possible number of uniquely identifiable codes which can be detected in this way.
• If desired, in order to economise on processing power in the sensor device itself, the coded sequences can be manipulated to represent them by single low resolution values. This can be achieved simply by processing each sequence by taking each maximum or minimum as it is read and summing the number of previous levels that are less than that level. It should be noted, in this connection, that the first two measurements, i.e. the first maximum and minimum measured, are always one of each so the first two terms of a sequence are, in this sense, insignificant. If the measured sequence is 0514230, then starting at the third maximum/minimum figure, in this case theFIG. 4, then calculating the number of previous elements which are lower in value to that element, gives the compressed data: 223, and then using weighting factors of 1, 2 and 8 respectively, one can calculate the sum of the occurrences factored by the weighting factors as follows:
• 
  (2×1)+(2×2)+(3×8)=30
• An alternative approach is to use the formula:
• $\sum _{x=3}^n{f\left(x\right)}_0^{x-1}$
• where n=the number of non-zero elemental levels in the sequence, x is the index of element in the code sequence (running from zero to n) and the formula
• ${f\left(x\right)}_a^b=\left(\sum _{m=a}^{b-1}\left(\left(x_a\right)<\left(x_b\right)\right)\right)*{\mathrm{<figure-callout\; label="f"\; filenames="US20080119271A1-20080522-D00002.png,US20080119271A1-20080522-D00003.png"\; state="\{\{state\}\}">f</figure-callout>}}_2\left[x\right]$
• where f₂(x)=[1,2,8]^x-1
  a unique code for each sequence can be generated as shown below:

• 	Sequence	Code No
  	0534120	21
  	0435120	21
  	0524130	28
  	0425130	29
  	0514230	30
  	0415230	31
  	0523140	36
  	0325140	37
  	0513240	38
  	0315240	39
  	0215340	41
  	0423150	44
  	0324150	45
  	0413250	46
  	0314250	47
  	0214350	49

• As can be seen, this generates 16 different codes, each represented by two digits, thus enabling the device to decide which of 16 possible programme modes in which to operate.
• If it is desired to select from an even wider variety of operating modes, then the code sequence may be simply extended so long as the final level measurement (which tells the device that the reading mode for the code should terminate) is not repeated. Thus, a sequence 0514230 could be extended to 051514230 or 051424230. Doing this doubles the number of code sequences available and, clearly, for each pair of added levels, which, of course, would be the same as other levels in terms of printing, the number of possible sequences doubles so that the number of uniquely identifiable codes rapidly becomes very large. When using a longer sequence in this way, the code compression algorithm discussed above needs to have more weighting factors, for example 1, 2, 8, 16 and 32. In each case, doing the processing results in the production of a uniquely identifiable short number code.
• FIG. 2 shows a sequence of printed areas which abut one another. If desired, the areas may be spaced from one another so that, as the sensor is passed over them, it detects the reflectance of the background paper between each area. This makes it easier to interpret the signals recorded and, in practice, leads to greater operational stability, especially if the number of different levels of reflectance is increased. Such an approach is more resistant to variations in print tolerances of the printed material, and indeed in variations in the sensor head itself.
• By using such a sequence of areas printed on each page of, for example, a revision exercise book, automatic calibration and recalibration may be carried out simply and reliably. In terms of the internal programming of the device, considerable savings are achieved by avoiding the need to have an internally stored “look-up table”. The levels in question are essentially stored when use of the sensor device commences, and lost at the end of a user session, though the software may retain them if it goes into a power-saving ‘sleep’ mode, e.g. after 60 seconds have occurred with no input changes, prior to going into a shut-down mode if no change occurs e.g. in the first 10 minutes of the sleep mode.
• Referring now toFIG. 4, this shows in block diagram form how a differential comparator detection technique may be used to overcome the variances and tolerances associated with detection of absolute levels of IR ink absorbency. The technique measures only the differences in absorbency levels as a relative value. In order to achieve this at minimal cost, the circuit configuration shown inFIG. 4 may be used.
• Amicrocontroller1 processes a constant current analogue output (Aout), usually utilised to drive an external audio device (speaker2) either directly, or via a buffer circuit (buffer3). The technique utilises this feature to compare the output current of Aout with the current drawn by an infrared sensor circuit in which asensor4 acts primarily as a variable current source. Both the analogue output from themicrocontroller1 and the output from thesensor4 are conditioned and scaled in passive signal conditioning circuits to allow direct comparison of the two current levels by acomparator6. The digital output ofcomparator6 then indicates to the microcontroller's digital input (Din) whether the analogue output is greater than or less than that of thesensor4.
• By varying its analogue output, Aout,microcontroller1 can determine in software the relative analogue output level of thesensor4. This then provides a direct digital representation of the sensor's analogue output in much the same way as a traditional successive approximation Analogue-to-Digital Converter (ADC).
• Under normal operation, this mechanism would require that the audio output ofmicrocontroller1 be temporarily suspended whilst the analogue output is used to compare with that of the sensor's output. However, a novel feature of this circuit is that the analogue comparison is made continuously, even when an audio output is present. This works because the range of the audio output level is tailored to cover the equivalent full range of the sensor output by the signal conditioning circuit. Since the audio signal comprises primarily alternating current (AC) waveforms over any audio output sequence (i.e. a sound or a phrase), the analogue output level inherently covers the full range of the sensor's possible output. Therefore, by continuously monitoring the digital output of thecomparator6, the sensor's absolute level can be continuously monitored. This does requiremicrocontroller1 to be able to determine the digital equivalent level of the analogue output in order for the microcontroller's software to determine the sensor's relative level.
• Thecomparator device6 may be internal to the microcontroller if desired.
• The analogue output, Aout, may be generated externally (by use of a Digital-to-Analogue converter or DAC) or internally by the microcontroller. In either case, the microcontroller must be able to read or determine the equivalent digital level of the instantaneous analogue output level.
• Theaudio buffer3 may be controlled by the digital output of the microcontroller (Dout). This allows the microcontroller to drive its analogue output for comparison purposes, even when no audio output is required, without generating any output from the speaker.
• Referring now toFIG. 5, this shows a preferred circuit for a sensor device in accordance with the present invention, though the power supply (which is conventional) has been omitted for the sake of clarity. Shown in the drawing is amicroprocessor unit10 which is programmed to operate as desired and which may be rendered active when aswitch11 is closed.Switch11 may be, for example, a “tip switch” mounted in the end of an elongate “sensor pen” so that when the tip is pressed against the printed material, switch11 closes. Also connected tomicroprocessor10 is anaudio transducer12.Microprocessor unit10 also controls whether or not current flows through the emitter diode of an optical infrared emitter/receiver package13. The receiving sensor portion is connected to one input of acomparator14. Connected to the other input ofcomparator14 is a resistance ladder network generally indicated at15, theladder network15 being also connected to outputs of themicroprocessor unit10.
• In operation, when theswitch11 is closed, the circuitry within themicroprocessor10 applies signals to the “rungs” of theresistance ladder15 which varies the applied voltage to the plus input ofcomparator14. The voltage applied at that input may be decreased incrementally at very rapid intervals (for example every 100 microseconds) while at the same time the voltage applied to the minus terminal ofcomparator14 will depend on the amount of infrared radiation reflected from the surface against which the optical emitter/receiver package13 is placed. The voltage on the minus input ofcomparator14 is always less than the supply voltage, and the output ofcomparator14 accordingly stays as a logical 1 until the voltage applied to the positive input ofcomparator14 drops below the level applied to the negative input at which point the output drops tologic 0 and accordingly, since the output ofcomparator14 is connected to an input ofmicroprocessor10, the microprocessor then knows precisely what voltage level is applied to the minus input of the comparator, i.e. it has a measure of the infrared reflectance of the surface next to package13. Aspackage13 is moved across a sequence of differently reflective areas of printing, the individual levels of absorption can be detected and decoded for use as indicated above.
• The circuit shown inFIG. 5 is very simple and inexpensive to produce and accordingly ideal for use in hand-held sensor devices used in conjunction with printed materials for early learning activity books and quiz books. By appropriate programming, the output fromtransducer12 may provide an attractive and stimulating level of interactivity between the sensor unit, the printed material and the user.

Claims (9)

1-8. (canceled)

9. Interactive information apparatus comprising printed material and a sensor device adapted to respond to characteristics of printing on the printed material, and wherein the printing on the printed material has intelligible components and non-intelligible components, the non-intelligible components being sensed by the sensor device, and wherein the sensor device senses differing values of a property of the non-intelligible components, and wherein the printed material and/or internal programming in the sensor device is arranged to ensure that calibration of the sensor device to the printed material is carried out automatically as the sensor device is applied to the printed material as part of a structured, intellectually mediated, interaction between the sensor device and the printed material.

10. The apparatus according toclaim 9, wherein the sensor device and the printed material are configured as an interactive game which requires a user to place the sensor device on a given number of differently reactive areas of the printed material.

11. The apparatus according toclaim 10, wherein the printed material includes a printed track or maze which must be navigated by the user passing the sensor device successively over different printed areas having differing values of the property sensed.

12. The apparatus according toclaim 9, wherein the printed material has a form of a quiz book or revision test exercise and the sensor device is programmed so that the sensor device records different values of the property sensed of the printed material and then processes a record of the different values to extract from the different values information as to the different values themselves.

13. The apparatus according toclaim 9, wherein the sensor device includes a processing unit adapted to recognize maxima and minima of the different values of the property sensed following sequential continuous application of the sensor device to a sufficiently large number of the unintelligible components of the different printed areas of the printed material.

14. The apparatus according toclaim 13, wherein the sensor device is programmed to set values of the property which are maxima or minima as specific values the sensor device is thereafter programmed to recognize, and wherein the sensor device will recognize printing as possessing a property at one of the specific values if the property sensed is at or within a preset tolerance band of the values of the maxima and minima.

15. The apparatus according toclaim 14, wherein the printed material presents different maxima and minima in a definite fixed sequence, whereby the sensor device may both calibrate itself to specific printed material and change operational mode of the sensor device from a default mode or an earlier operating mode to a new operating mode.

16. The apparatus according toclaim 15, wherein the printed material and the sensor device constitute a quiz or early learning book and wherein rules for interaction between the sensor device and the printed material change from page to page or from one printed book or worksheet to a next printed book or worksheet.

Images