TITLE: Method and device for automatically repeating computer-actions
FIELD OF THE INVENTION
The present invention relates in general to the field of automation.
5 BACKGROUND OF THE INVENTION
Persons working with computers or managing servers, especially those used in offices, for instance PCs, laptops, etc, will recognize that some labourous tasks need to be repeated again and again. Examples may be found on the level 10 of working in individual projects (for instance: word processing documents), but also on a higher level.
First, it should be noted that a computer as such is just a machine designed to read and execute instructions. A coherent set of computer instructions will be indicated as 15 computer program. Without computer program, a computer will do nothing, and it is just an expensive paper weight. For a computer to perform a task, a computer program must be loaded into an operative memory, and the program must be run, meaning that the subsequent instructions will be executed. Depending 20 on the nature of the program, the apparatus will now behave as a text processor, a data base, a video processor, a TV tuner/receiver, a building designer, an environment management system, etc. While humans tend to indicate the computer on the basis of the task that it is currently performing (for 25 instance: "the video processor is working"), one has to keep in mind that it is actually a computer program that is running, i.e. being executed. Such computer program is also indicated as "application".
In some cases, when running, an application is running 30 autonomously. For instance, a control system that monitors temperature and light level in a building and control shutters, lamps, air conditioners, heating appliances, etc, 1039199 2 does not need any human intervention. On the other hand, other applications require data input and/or instructions input from a user: an example is a word processor, which requires user input via a keyboard to enter characters to make words, 5 sentences, pages, and which further requires user input instructions for defining layout, and for instructing the apparatus to store, print, retrieve, etc.
Most applications have a "repeat" facility, which is an instruction causing the application to repeat the last user 10 instruction. An example of such instruction is a formatting instruction. As a simple example, consider a user using a text-processing application to draft a patent application, who wishes to have the wording "characterized in that" be formatted in bold print. The user actions are to select the 15 intended text fragment and give the instruction 'bold'. In a next claim, the user wishes to do the same. Again, he selects the intended text fragment, and now he can give the instruction 'repeat last instruction', which in this case was the instruction to put the text in bold.
20 The above example is just a simple example, but even in a text-processing application there may be found repetitive tasks that are slightly more complicated. For instance, consider that the patent application has reference numerals between brackets in the claims. While this is allowed in some 25 coutries, it is not allowed on others. When the applicant wishes to use the same text in other countries, he has to remove these reference numerals including the brackets. While the task as such is simple, it may nevertheless be rather time-consuming, so it is desirable to have this task performed 30 by the computer itself. Again, the user may select one reference numeral including its brackets and give the instruction 'delete', and repeat these actions for all reference numerals. However, it is clearly desirable to delegate this repetition to the computer, i.e. to the text-35 processing application.
Modern text-processing applications, for instance Microsoft Word, have a facility for this, capable of doing so. Stated briefly, the application is capable of memorizing a series of consecutive instructions, and to store these 3 instructions as a small program indicated as MACRO. In this example, the instructions may be: 'proceed to next bracket-open', 'select', 'extend selection to the right until bracket close', 'delete selection'. The stored MACRO will be available 5 to the user, and by pressing a key the user can have this MACRO executed. For generating the MACRO, the application will have a MACRO RECORDER facility, which basically consists of a START-command and a STOP-command, defining that the instructions received between the START-command and the STOP-10 command constitute the MACRO instructions.
As mentioned, the above example concerns a facility offered by the application itself, and the use is limited to the application itself. The present invention aims to provide a task repeating facility on a higher level.
15
Figure 1 is a block diagram schematically illustrating the operation of a computer, whether a mainframe computer, a PC, a laptop, etc. A user is indicated by reference numeral 1, the computer is generally indicated by reference numeral 100. 20 Reference numeral 10 indicates a computer program, for instance a text-processing application, running in an operational memory of the computer 100. Apart from such application, the computer 100 comprises a control system 20, also indicated as operating system. The operating system 20 25 can be considered as a main computer program controlling the basic functioning of the computer apparatus and also controlling the operation of the application. It can be considered as a shell between the user and the application.
For communication with the user, the computer 100 further 30 comprises a user interface 30. The user interface 30 typically comprises hardware components such as a keyboard 31, a display screen 32, a mouse 33. These hardware components have associated software programs indicated as drivers, but these are not illustrated separately. The user interface 30 35 communicates with the operating system 20. If the user presses a key on his keyboard 31, the keyboard 31 will generate a signal that is interpreted by the operating system 20 as, for instance, input of a character "K", and the operating system 20 will communicate this character to the application 10. The 4 application 10 may tell the operating system 20 that this character "K" is to be displayed, and the operating system 20 controls the display screen 32 in such a way that the screen pixels present to the user the image of the character "K", in 5 a letter type (font) determined by the application 10. Apart from keyboard input, the operating system 20 will typically also allow for graphical user input, indicated as "clicking", "double-clicking", etc. It is assumed that these concepts are clear to a person skilled in the art: the user takes the mouse 10 and clicks on a certain item displayed on the display screen. While every experienced computer-user will know exactly what this means, it is useful to elaborate in some more technical detail: the operating system 20 drives the display screen 32 to display an image of a pointer, for instance an arrow, and 15 the position of this pointer on the display screen 32 is changed in accordance with instructions received from the user via either keyboard commands or mouse commands. Then, the user issues an 'enter' command, either via the keyboard (pressing the 'enter' key) or via the mouse (pressing a key of the mouse 20 device). This command has a meaning depending on the actual position on screen of the pointer at that precise moment, which position is of course known to the operating system 20. So, the operating system 20 interprets this command, taking into account the position on screen of the pointer, and 25 translates this command to an instruction for the application.
How does the operating system 20 "know" where the user is clicking, i.e. where the pointer is located? For driving the display device such as to render an image on its display screen 32, the computer 100 comprises a video memory 132. The 30 display screen 32 comprises a plurality of pixels, organized in rows and columns, and the video memory 132 comprises a plurality of memory locations, each memory location being associated with a respective screen pixel. The operating system 20 regularly (at an image repetition rate) sends the 35 actual contents of the video memory 132 to the display screen 32 such as to generate an image. The data contents of a memory location determines the colour and brightness of the corresponding pixel. Whenever the operating system 20 is generating a pointer image, it writes the data corresponding 5 to the pointer image into a collection of memory locations. User input intended to result in a movement of the pointer will cause the operating system 20 to amend said collection of memory locations. Depending on the size of the pointer image, 5 said collection of memory locations will always consist of a certain number of memory locations in horizontal direction and a certain number of memory locations in vertical direction, typically having the shape of a rectangle, and the position of the pointer may be defined by the identity of one of said 10 memory locations, for instance the one corresponding to the top-left corner of said rectangle. The operating system 20 obviously has data defining the memory locations where it has written the pointer pixel data: this is simply indicated as "the operating system 20 knows where the pointer is located on 15 the screen". Whenever a user is actuating a click-button, whether a button on the mouse, a key of the keyboard, a touchpad, etc, this action is interpreted in conjunction with the actual position of the pointer on the screen, and this is briefly indicated as "clicking at that position".
20 In the above-described example of a MACRO in a text- processing application, the task-repetition facility is inherently available within the application. In the case of an application without such facility, it is not possible to alter the application. Therefore, the present invention aims to 25 provide an intelligent repetition tool that operates on the level of the operating system 20. A simple approach for such tool is illustrated in figure 2. This figure is identical to figure 1, with the exception of a repetition tool 50 being added. The repetition tool 50 has an input 53 coupled to the 30 user interface 30 and an output 52 coupled to the operating system 20. The repetition tool 50 further has an input 54 for receiving the contents of the video memory 132. It may be that the repetition tool 50 has direct access to the video memory 132: in that case, the input 54 is directly coupled to the 35 video memory 132, as shown. Alternatively, it may be that the video memory 132 is only accessible through the operating system 20, in which case the input 54 would be coupled to the operating system 20, as indicated in dotted lines, but this will be indicated as an indirect coupling to the video memory 6 132. The repetition tool 50 further is provided with a repetition memory 133.
Thus, the repetition tool 50 not only receives the user input from the keyboard 31 and from the mouse 33, but it also, 5 at any moment, receives the contents of the video memory 132.
This allows the repetition tool 50 to repeat a sequence of user actions. Not only can the repetition tool 50 repeat keyboard strokes, it can also repeat an action as "clicking at a certain screen position" by sending data to the operating 10 system 20 defining the screen position and defining the click action. More specifically, the repetition tool 50 detects the pointer position and detects the user actions, from monitoring the user interface 30 and the video memory 132, while further the repetition tool 50, via its output 52, sends data to the 15 operating system 20 putting the pointer in a certain position and simulating the user actions. The operating system 20 does not distinguish between input received from the repetition tool 50 on the one hand, and input received from the user interface 30 on the other hand: as far as the operating system 20 20 is concerned, it is as if the user himself is performing the actions represented by the data sent by the repetition tool 50, and the operating system 20 is responding in exactly the same way.
Depending on the circumstances, the intended recipient of 25 these actions may be the application 10 or the operating system 20 itself. For instance, when an application is running and an interaction screen is displayed, clicking at a certain position of such interaction screen will be interpreted by the operating system 20 as being an instruction intended for the 30 application 10, and it will pass on this instruction to the application 10. On the other hand, it may be that no application is running, or that the user is clicking at a position outside such interaction screen, for instance on an icon linking to a certain program or a certain folder or a 35 certain document: in such case, clicking at that position will be interpreted by the operating system 20 as being an instruction intended for itself, and it will execute the corresponding instruction, i.e. open said certain folder or said certain document or start said program.
7
SUMMARY OF THE INVENTION
A repetition tool as described so far can already be 5 useful for repeating exactly the same actions in exactly the same circumstances. However, the applicability of such tool would be limited. It is to be noted that, in the introduction, a repetition tool offered by an application has been described, which tool would repeat the last user action 10 immediately preceding the 'repeat' instruction. Also a MACRO has been described, as a series of consecutive instructions; such MACRO can be stored and recalled for executing at a later moment. The repetition tool envisaged by the present invention is also a software program that can be stored and run at a 15 later moment, as desired by the user, but it can also be copied and run at another computer. Whether run at a later moment on the same computer, or on a different computer, there is no guarantee that the graphical user interface will have exactly the same format.
20 Figure 3A is a diagram schematically illustrating a front view of a computer display screen 32 displaying just one icon 35 and a pointer 34. It is noted that the screen will typically display more than one icon, but this is not shown for sake of simplicity. The icon position is defined by 25 coordinates XI and Yl, being the horizontal distance and vertical distance, respectively, of the top-left corner of the icon 35 from the top-left corner of the display screen 32. For sake of clarity, the pointer 34 is shown just below the icon 35, but for clicking on the icon, the pointer should be at 30 least partially overlapping the icon, as should be clear to a skilled person. Assume, in a simple example, that the repetition tool 50 is to repeat the clicking action on this icon on another computer (or on the same computer at a later moment).
35 Figure 3B is a diagram comparable to figuren 3A, of the display screen 32 of the other computer (or the same computer at a later moment), also just displaying the same one icon. However, the icon 35 has been moved to a different location. When the repetition tool 50 would repeat the same action, i.e.
8 click at the position [XI, Yl], nothing would happen. Or something unintentional might happen. In any case, in the interaction between user and computer, it is not important that the user has clicked at a certain position [XI, Yl], what 5 is important is that the user has clicked on the icon 35. The user action as such is, indeed, clicking at (or close to) the position [XI, Yl]. However, this action is received by the operating system 20, which also 'knows' that the display screen 32 is showing the icon 35 at that position, and the 10 operating system 20 will "translate" the user click action as meaning an instruction to perform an action defined by the icon, which typically represents a link to a program to be started, a folder to be opened, etc. In the situation of figure 3B, the operating system 20 will receive the user click 15 at the position [XI, Yl], but now there is no icon located at that position (or a different icon might be located there). In any case, the operating system 20 will not perform the action defined by icon 35.
The present invention aims to solve this problem.
20
BRIEF DESCRIPTION OF THE DRAWINGS
These and other aspects, features and advantages of the present invention will be further explained by the following description with reference to the drawings, in which same 25 reference numerals indicate same or similar parts, and in which: figure 1 is a block diagram schematically illustrating the operation of a computer; figure 2 is a block diagram schematically illustrating a 30 computer provided with a repetition tool; figures 3A and 3B are diagrams schematically illustrating a display screen showing an icon and a pointer; figure 4 is a block diagram schematically illustrating a computer provided with a repetition tool and image recognition 35 software.
DETAILED DESCRIPTION OF THE INVENTION
9
In accordance with the present invention, the repetition tool 50 is equipped with image recognition software, generally indicated by reference numeral 150 in figure 4. Such image recognition software, also indicated as "computer vision", is 5 known per se and available both commercially and non- commercially, so that a detailed description and explanation may be omitted here. Using the image recognition software, the repetition tool 50 always analyses the content of the video memory 132. In a learning mode or recording mode, for instance 10 illustrated in figure 3A, when the user is performing actions and the repetition tool 50 is recording the user actions in its repetition memory 133 such as to be able to repeat them later, the repetition tool 50 analyses the video memory 132 and recognizes the icon 35. When the user now clicks at (or 15 close to) the position [XI, Yl], the repetition tool 50 will recognize that the user action to be stored is a click on that specific icon 35. The repetition tool 50 will now not only store the action concerned (i.e. 'click'), but will also store data defining the icon picture (typically a copy of the 20 portion of the video memory associated with the icon picture). In a repeat mode or playback mode, for instance illustrated in figure 3B, when the user has instructed the repetition tool 50 to repeat the recorded actions, the repetition tool 50, when arriving at the moment when it is to repeat this specific user 25 action, again analyses the video memory 132 and, using the image recognition software 150, looks for the icon 35, which it will find at position [X2, Y2]. The repetition tool 50 will now send to the operating system 20 data defining a click action at position [X2, Y2] (or at any other position 30 coinciding with the icon 35).
In this respect it is noted that the icon 35 occupies a certain screen area, and that a click at any position within this area is interpreted by the operating system 20 as being a click on this icon. The precise size of this area may depend 35 on the computer system, on the screen definition, and on user settings defining the image resolution and the icon size. In the context of the present invention, the phrase "icon area [X, Y]" will be used to indicate the screen area occupied by the icon having its topleft corner located at position [X, Y].
10
Thus, the repetition tool 50 will send to the operating system 20 data defining a click action in icon area [X2, Y2].
In the above example, it is assumed that the user action is a 'click', but the same applies, mutatis mutandis, if the 5 user action is a 'right click', 'double click', etc. Further, the same applies, mutatis mutandis, if the user action is for instance the typing of text in a text box. Further, instead of an icon, the relevant image area may also represent for instance a folder or a written list of items.
10 It is further to be noted that the operation of the repetition tool 50 is aimed at repeating a certain effect, for instance opening a folder or running an application represented by an icon. It is not necessarily so that the same effect is achieved by the same action. For instance, depending 15 on user-defined settings of the operating system, opening a folder or running an application may be effected by a single click action or by a double click action. So it may for instance be that the repetition tool 50, when running on one computer, should perform a single click action but, when 20 running on another computer, should perform a double click action to achieve the same effect. Therefore, in a preferred embodiment, the repetition tool 50 is adaptable in that the user is allowed to replace a certain action (single click) by a certain other action (double click).
25
In a further elaboration, it may be that, on repetition, the icon 35 is represented by a different icon picture as compared to the moment of recording. In that case, the repetition tool 50 will not be able to find the icon and it 30 can not repeat the action.
According to the present invention, the repetition tool 50 may now display a message for the user, asking the user to indicate the icon picture that now represents the icon concerned. The user response will be interpreted by the 35 repetition tool 50, and it will replace the picture in its icon memory by the picture now indicated by the user.
Alternatively, the user may have predefined an alternative action: if the repetition tool 50 is not able to 11 find the icon and can not repeat the action, the repetition tool 50 will continue with performing the alternative action.
Alternatively, if the repetition tool 50 is not able to find the icon and can not repeat the action, the repetition 5 tool 50 may be programmed to end its repetition task, and to send an error message together with a print of the current screen contents to a predefined location, for instance an e-mail address, to allow the user to analyse the problem.
10 In the above explanation, the advantages and strength of the repetition tool 50 proposed by the present invention have been explained in the context of clicking on an icon. From this explanation, it shoud be clear that the inventive tool is very user friendly. However, it should also be clear that the 15 tool is not restricted to clicking icons. When interacting with an application, such application may display graphical objects that need to be clicked, or text boxes where text needs to be inputted, and in the same manner as described above, the repetition tool 50 in a recording mode or learning 20 mode, by always consulting the video memory, at each user step can see what the user is doing where. The user task may be more elaborate than just clicking one icon; in fact, the user task may be very complicated. One example of a complicated user task is the installation of a computer program on a 25 user's PC, for instance installing an email client. Another example, in line with the example given above of an applicant drafting a patent application, is the online filing of such application: the repetition tool of the present invention is quite conveniently able to perform all the required actions, 30 while at certain junctions asking for necessary information, such as 'name of inventor', if this information is not available in a database accessible to the repetition tool 50.
In any case, the repetition tool according to the present invention is not blindly repeating user action, but is 35 intelligently monitoring the video memory to determine where on the display screen the user action needs to be repeated.
12
It should be clear to a person skilled in the art that the present invention is not limited to the exemplary embodiments discussed above, but that several variations and modifications are possible within the protective scope of the 5 invention as defined in the appending claims. For instance, instead of a 'click'-event, the invention also applies to the case of 'double click' or 'multiple click', or any other event resulting in a control of the mouse pointer or an input of a character. Further, instead of a separate mouse and display 10 screen, a touch screen may be used.
In the above, the present invention has been explained with reference to block diagrams, which illustrate functional blocks of the device according to the present invention. It is to be understood that one or more of these functional blocks 15 may be implemented in hardware, where the function of such functional block is performed by individual hardware components, but it is also possible that one or more of these functional blocks are implemented in software, so that the function of such functional block is performed by one or more 20 program lines of a computer program or a programmable device such as a microprocessor, microcontroller, digital signal processor, etc.
It is further noted that the repetition tool 50 proposed by the present invention is typically implemented as a 25 software program. The program may be resident on a computer, i.e. stored in the computer's filing system, to be executed when so instructed by a user; even while not actually being executed, the computer having the repetition tool 50 stored in its memory nevertheless falls under the protective scope of 30 the present invention. It is also possible that the program is stored on a computer-readable medium, such as for instance a floppy disk, an optical disc, a USB stick, an SD card, etc, either for being copied to a computer memory or for being directly operated from such medium: such medium having the 35 repetition tool 50 stored also falls under the protective scope of the present invention. Finally, even the computer program as such falls under the protective scope of the present invention.
1039199