US20020101510A1

Movatterモバイル変換

Info

Publication number: US20020101510A1
Application number: US09/774,488
Authority: US
Inventors: Sara Basson; Dimitri Kanevsky
Original assignee: International Business Machines Corp
Current assignee: Hulu LLC
Priority date: 2001-01-31
Filing date: 2001-01-31
Publication date: 2002-08-01
Also published as: US7474335B2

Abstract

Apparatus, and associated methods, for stabilizing an image. The apparatus includes an aiming device to be aimed at a moving image, a transforming device which transforms the image into a non-moving image; and a displaying device which displays the non-moving image. The apparatus and methods are of particular benefit, for example, to individuals with Parkinson's disease and/or individuals attempting to work on a laptop computer or read a book while travelling via airplane, boat, train or road vehicle.

Description

FIELD OF THE INVENTION

The present invention relates generally to items that normally provide optical assistance, such as eyewear, as well as to screens and the like that display images. The present invention relates more particularly to the use of the aforementioned items in less than ideal circumstances, such as when the user's field of vision, or the image(s) being viewed, are unstable.[0001]

BACKGROUND OF THE INVENTION

Eye strain and fatigue have long been recognized as undesirable consequences presented to individuals attempting to read or do close work under given conditions or in given medical contexts. For instance, an individual suffering from Parkinson's disease may experience eye strain and fatigue when reading a book since the person's hands may be shaking too much.[0002]

Other problems may be encountered by any individual when attempting to read or do close work in or on a moving vehicle. For example, when an individual attempts to read a book while riding in a car, his or her eyes will almost inevitably grow tired since the book will likely be shaking continually. On the other hand, an individual riding in an airplane, along with reading material or another subject of close work, may be shaking at a constant rate due to turbulence experienced by the plane. Similar problems may be encountered on a boat. Thus, all of these vehicular movement issues will typically affect the manner in which an individual visually perceives his or her environment, leading almost inevitably to eye strain and fatigue and possibly leading to nausea. Normally, a recommended solution for “sea-sickness”, “air-sickness” or “car-sickness” in instances such as these is to look at a perceptually stationary entity, such as the horizon.[0003]

However, a need has been recognized in connection with proceeding beyond simplistic solutions for relieving eye strain and fatigue, and feelings of nausea, in the context of vehicular movement, and to provide somewhat of a user-friendly mechanism to attend to such matters. A need has also been recognized in connection with attending to such matters, in a similar manner, in connection with medical issues that might affect an individual regardless of whether he or she is in vehicular transit.[0004]

SUMMARY OF THE INVENTION

In accordance with at least one presently preferred embodiment of the present invention, it is broadly contemplated that an individual may wear a special pair of eye glasses or head mounted display that is connected to a camera. The image recorded by the camera may then be displayed on the eye-glasses in a way that it is not shaking in front of the eyes of the wearer. For example, a book that is being read by an individual with Parkinson's and, in the process, may be shaking, can preferably be recorded by a camera and is sent through an object recognition device that recognizes and tracks the book. If the book appears to have moved from a stationary position, the device may thence adjust for such movement and display the book on the eye glasses by special projectors so that it appears not to be shaking, thus enabling the reader to enjoy the book without getting sick or frustrated. A similar solution may be applied to the problem of somebody reading in a car or other vehicle, where the image of the book, newspaper, etc., being read may be projected onto eye glasses of the reader in such a way as to appear virtually immobile.[0005]

In another embodiment of the present invention, if an individual is looking at the lap-top or other computer, or even an e-book, and it is shaking (e.g. when riding in a car or train) then the image on the lap-top may preferably be adjusted such that its vibrations relative to the motion of the user's eye are dampened or even eliminated in order to keep the user from feeling sick. A camera on the lap-top may preferably note the position of the user's eye and its motion. An intelligent device within the laptop will preferably display the image on the screen of the laptop so that its motion is always relative to that of the user's eyes. A similar mechanism may be used for somebody reading on an airplane or other vehicle. In such instances, the person's glasses may register the object as being read as if on the horizon (i.e., virtually stationary), thereby preventing the person from becoming sea-sick, air-sick or otherwise feeling unwell.[0006]

In one aspect, the present invention provides an apparatus for stabilizing an image, the apparatus comprising: an aiming device to be aimed at a moving image; a transforming device which transforms the image into a non-moving image; and a displaying device which displays the non-moving image.[0007]

In another aspect, the present invention provides a method of stabilizing an image, the method comprising the steps of: providing an aiming device; aiming the aiming device at a moving image; transforming the image into a non-moving image; and displaying the non-moving image.[0008]

Furthermore, the present invention provides, in another aspect, a program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for stabilizing an image, the method comprising the steps of: providing an aiming device; aiming the aiming device at a moving image; transforming the image into a non-moving image; and displaying the non-moving image.[0009]

For a better understanding of the present invention, together with other and further features and advantages thereof, reference is made to the following description, taken in conjunction with the accompanying drawings, and the scope of the invention will be pointed out in the appended claims.[0010]

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 schematically illustrates an embodiment of the present invention involving eyewear.[0011]

FIG. 2 is a schematic representation of a CPU from FIG. 1.[0012]

FIG. 3 schematically illustrates an image transformator.[0013]

FIG. 4 is a flow diagram depicting the function of a transformator such as that shown in FIG. 3.[0014]

FIG. 5 schematically illustrates an arrangement for stabilizing an image on a laptop computer.[0015]

FIG. 6 is a flow diagram depicting several basic steps applicable to different embodiments of the present invention.[0016]

DESCRIPTION OF THE PREFERRED EMBODIMENTS

FIG. 1 schematically illustrates an embodiment of the present invention involving eyewear. Indicated at[0017]100 is a hypothetical object that is shaking (e.g., moving up and down). Thisobject100 may be a book, laptop, magazine, as well as many other things. Theobject100 includes animage110, which is also shaking.

Indicated at[0018]102 are the eye glasses (or other eyewear, such as goggles) of the users where102brepresents the right side of the glasses and102arepresents the left side. Theglasses102 preferably have acamera101 attached (though the camera need not necessarily be attached and in fact can be located remotely). Camera101 is preferably connected to acomputer105, and any image information sent through thecamera101 thus preferably progresses through aCPU103.CPU103 is preferably adapted to determine how thecamera101 is moving, shaking or vibrating relative toeyeglasses102.

Information relating to how the[0019]

camera

101 is moving, as well as the image110 (whose presence in the camera is indicated at130) enters animage recorder106 and acoordinate recorder108. Thecoordinate recorder108 records and stores information about how the camera is moving. Theimage recorder106 contains the image (indicated there at140) that was observed by thecamera101.

Information from the[0020]

image recorder

106 andcoordinate recorder108 is sent to animage transformator107 which transforms the image so that it appears as though it is not moving in theglasses102 of the wearer. The image so transformed is sent through theprojector104 and displayed on the left and right lenses109a/bof the user's eye glasses. An image is thus displayed at120aon the left lens109aand at120bon the right lens109b. It is assumed here that theglasses102 are stationary relative to the person's eyes; therefore, if the image is displayed relative to the movement of theeye glasses102 then it will appear stationary to the user. Preferably, there will be some shift in the left and right images120a/bwith respect to each other, in known manner, so as to invoke the perception of a three-dimensional image. Thus, it is possible to create an illusion of depth of field using two images due to stereo imaging phenomena. However, most displays have a fixed depth of focus which would restrict the placement of an image to be transformed at different focal lengths. Presumably, techonologies still to be developed might remove these restrictions. For present purposes, it should be understood that it is not essential that a transformed image be displayed in both eyes.

FIG. 2 is a schematic representation of the[0021]

CPU

103 from FIG. 1, that is, the computer device attached to thecamera102. In one embodiment,CPU103 may preferably include agyroscope200 that identifies how thecamera102 is angled with respect to a vertical axis. Indicated at201 is a height or altitude measurement device which contains a sensor that identifies the height, with respect to some stationary entity, such as the ground, of the camera/glasses device101/102. Indicated at202 is a clock that measures and records the time at which any motions of the camera/glasses device101/102 occurred. This information is sent to acommunication module203. The information from thegyroscope200 and theheight measurer201 thus provides an essentially complete sense of the movements of thecamera101 connected to theglasses102. The information from thecommunication module203 enters the image recorder and coordinate recorder,106 and108. Theimage recorder106 receives the actual image data whereas thecoordinate recorder108 receives information recorded in200 and201.

FIG. 3 is an illustration of the[0022]

image transformator

107. Continued reference is also made to FIG. 1. The input for theimage transformator107 is shown as original image frames301 and302, time-stamped at303 with times t1 and t2. The original image frames301/302, indeed, correspond to theimage130 that was recorded by thecamera101 and that entered theimage recorder106. The original image frames301/302 may preferably be taken at specific time intervals. These time intervals could conceivably depend upon the situation, e.g., how quickly the images are moving/shaking. If, for example, a person is holding a book in a car and the vibrations of the book are relatively slow, then the intervals of time at which a picture of the book is taken can be relatively longer. If a person reading a book has Parkinson's disease and the hand holding the book is shaking relatively fast, then the intervals at which a picture of the book is taken can preferably be much smaller. Indicated at306 are camera coordinate change data entering theimage transformator106.

The image frames[0023]301 and302 are preferably input into a comparator304 which compares the frames and, particularly, how the images have changed. For example, if the system sees that the user is reading a book (page remaining the same) then the system can display the book with incremental movements just to keep the display natural. In order to “see” that the user is reading a book, a known manner of object recognition may be utilized, such as that described in copending and commonly assigned U.S. patent application Ser. No. 09/238,245, filed on Jan. 28, 1999 (“A Virtual Map System and Method for Tracking Objects”). If, on the other hand, the person is turning a page, then the system must recognize that if the motions it records do not result in identical display of object being observed then the motions must be deliberate and not motions of the car. Contour image recognition occurs at305; that is, the contours of an object are recognized.

A rationale for including contour image recognition is that, as a general rule, people tend to look at what is written in a book and not at the contours of the book itself For example, if a person is watching television, the content itself is changing but not the contours of the television. Therefore, in such cases it can be necessary to compare the contours of the television at different times.[0024]

Comparator[0025]304 and contour recognizer305 preferably communicate between themselves. It is possible that contours of the same image will be compared at different times.

After it has been determined how the contour of an image has moved, this information preferably enters an image shifter[0026]307 along withinformation306 relating to the change in camera coordinates. The information about how the object has moved is thence compared to how the camera has moved.

Preferably, the position at which the image “should” be displayed relative to the last display is computed by just enough to account for the motion of the camera. For example, if the contour (e.g. of a book) has shifted 3 cm up and the camera has shifted 1 cm up then the image shifter[0027]307 will preferably serve to move theobject display 2 cm down. Similar computations are preferably made with respect to horizontal and rotational coordinates. If, for example, the contour of the objects shifts 30 degrees in direction “A” whereas the camera shifts 15 degrees in a direction opposite to “A”, then the image shifter307 will preferably account for this by displaying the image at −15 degrees with respect to direction “A”. The shifter307 thus serves to make the object appear as though it has not moved. The image is then preferably sent to theprojector104.

For as long as the user has not replaced the object being viewed or read, the eyeglasses will preferably contain a constant picture of the original image taken at time t[0028]1, as will be better appreciated from FIG. 4. Only when a person turns the page or puts the object down will the image in the glasses change. This method is applicable when the person only needs to see what he/she is reading. In other words, if a person reads a book, he/she will need to see the picture of a page that s/he currently reads. He/she can read an image of the page, as captured by the camera. When he/she finishes reading the page, he/she can turn the page and the camera captures a new picture that will be displayed for the reader, and so on.

FIG. 4 provides a flow chart of which elaborates on how the transformator from FIG. 3 may work. It depicts a situation in which an image frame is taken at a time interval and this image is displayed on the user's eye glasses until a deliberate (e.g., page turning) motion is made. This is possible only if the person is, for example, reading a book and then the initial image is displayed until the system registers that the user has turned the page and it again takes and initial image frame at that time of the new page.[0029]

Image frame[0030]1 (i.e. the first image frame) is indicated at400. Subsequent image frames2,3, . . . K are indicated at401. Atstep402, it is determined as to whether the last image content has changed by comparing the latest image taken with the one before it. If the content did not change, then the system opts to keep theinitial frame404 displayed on theglasses403. If the answer to the question at402 question, i.e., “Did the kth image content change?”, is “yes”, then the Kth frame is displayed on theglasses403 and step405 takes place, in which there is a continual comparison of frames k+1, k+2, . . .

FIG. 5 depicts a different method, in which an image is stabilized an on a lap-top without the user making use of the inventive eyeglass arrangement discussed above. Indicated at[0031]500 is the eye of a person looking at the display of a lap-top500a. A camera501 is fixedly mounted on the lap-top. The laptop display preferably contains an image or othervisual content502, which may be a book, a text document, a movie, a graphical image, etc . . . Indicated at503 is a running application on the lap-top. Indicated at504 is an eye shift detector which will preferably note how the user's eye is moving relative to theimage502 displayed on the lap-top. The detector504 preferably functions analogously to theimage transformator107 described heretofore. Depending on the data received from the detector of eye shift504, the image (content) shifter505 will preferably move the image being displayed on the laptop in a manner that will make the image appear immobile to the user. Widely available technology currently exists for detecting eye shift; examples of such technology may be found at http://www.lctinc.com/doc/linksem.htm.

FIG. 6 is a flowchart of a process in accordance with at least one embodiment of the present invention, and in fact may relate to a wide range of embodiments, including “eyeglasses” and “laptop” embodiments. Step[0032]600 involves getting the image frame i+1. Step602 involves comparing the image i+1 with the previously stored image i. Step603 involves analyzing whether the image contour has changed. If the contour did not change, then the system gets thenext image frame606, which returns the process back to step600. If the image contour did change, then it is queried atstep604 as to whether the content has changed. If the content has not changed, then the camera/glasses coordinates are obtained at601 and the image is shifted to make it balanced, atstep607. If the content did change, then frame i+1 is stored and, atstep606, the next frame is obtained.

The disclosure now turns to a discussion of various components that may be utilized in accordance with the embodiments of the present invention.[0033]

An eyeglasses-based microdisplay system, as generally depicted in the figures, is available from MicroOptical Corporation (Westwood, Mass.). A wide variety of other head-mounted display systems (e.g. glasses and/or goggles) may be used, e.g. the ergonomic wearable personal display from Invisio Inc. (Sunnydale, Calif.); and the compact optical display system associated with the wearable personal computer from IBM Corporation (Armonk, N.Y.) as described for example, in U.S. patent applications Ser. Nos. 09/437,972 (filed Nov. 10,1999), 09/437,793 (filed Nov. 10, 1999), and 29/110,706 (filed Sep. 10, 1999).[0034]

The miniature projectors contemplated herein are preferably located inside the head-mounted glasses/goggles device. The projectors are preferably configured for projecting computer screens on a suitable area, such as a lens area visible to the user. Preferably, the effect will be that the user has impression that he or she is looking at a computer screen. Such projector arrangements can be small enough to fit in the frames of glasses or goggles. In one possible embodiment, the projected image can be projected on the lenses with the intermediary of a mirror and prism.[0035]

Conceivably, the eyeglasses/goggles, particularly the lenses or lens areas thereof, could be transparent or opaque. For instance, if it is desired that no projected image be confused with an actual image, the lens or lens areas of the eyeglasses/goggles could be opaque. The user would then need only “aim” the eyeglasses/goggles at the object for which the image transformation described heretofore is desired. Transparent lenses or lens areas, however, could be used in instances where viewing the actual (originating) image or object may be desired. Here, an overlayed image could be displayed on the lens with a distinct contrast as compared to the general background (e.g., the displayed matter could be much brigther than what is otherwise viewd through the lens).[0036]

In another conceivable scenario, it is possible for the image of a book to be “printed” (e.g. downloaded) into the lens or lens area of the glasses/goggles while a user is looking at a blank wall. In other words, the image of a book or other item to be read that is shaking or moving can be stabilized by downloading or “printing” text and/or images from the book or other item onto the lens or lens area of the glasses/goggles from CPU storage, rather than necessitating the user looking at the book (or other item to be read) in the first place. Thus, in such an instance, the glasses/goggles will not necessarily have to be “aimed” at the book or other item to be read when the user wishes to read it.[0037]

Among components that could be utilized for displaying a stabilized image in accordance with the embodiments of the present invention are: a head-mounted display, glasses/goggles, a laptop display, a TV, a telephone screen, a pager screen, or any of many other possible analogous items.[0038]

A projecting arrangement, for getting an image on a displaying arrangement, could include a projector, a TV tube, a computer, or any of many other possible analogous items.[0039]

A sensing arrangement, for sensing the position and/or state of a moving image that is to be transformed into a non-moving image, could include a gyroscope, camera, motion sensors, or any of many other possible analogous items.[0040]

It is to be understood that the present invention, in accordance with at least one presently preferred embodiment, includes an aiming device to be aimed at a moving image, a transforming device which transforms the image into a non-moving image, and a displaying device which displays the non-moving image. Together, the aiming device, transforming device and displaying device may be implemented on at least one general-purpose computer running suitable software programs. These may also be implemented on at least one Integrated Circuit or part of at least one Integrated Circuit. Thus, it is to be understood that the invention may be implemented in hardware, software, or a combination of both.[0041]

If not otherwise stated herein, it is to be assumed that all patents, patent applications, patent publications and other publications (including web-based publications) mentioned and cited herein are hereby fully incorporated by reference herein as if set forth in their entirety herein.[0042]

Although illustrative embodiments of the present invention have been described herein with reference to the accompanying drawings, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be affected therein by one skilled in the art without departing from the scope or spirit of the invention.[0043]

Claims

What is claimed is:

1. An apparatus for stabilizing an image, said apparatus comprising:

an aiming device to be aimed at a moving image;

a transforming device which transforms the image into a non-moving image; and

a displaying device which displays the non-moving image.

2. The apparatus according toclaim 1, wherein said aiming device is adapted to be mounted on a human head.

3. The apparatus according toclaim 2, wherein said aiming device comprises eyeglasses.

4. The apparatus according toclaim 2, wherein said aiming device comprises goggles.

5. The apparatus according toclaim 1, wherein said transforming device comprises a camera which records the moving image.

6. The apparatus according toclaim 5, wherein said camera is adapted to record successive discrete frames of the moving image.

7. The apparatus according toclaim 5, wherein said camera is adapted to record the moving image continuously.

8. The apparatus according toclaim 1, wherein said transforming device comprises a CPU which accepts data relating to the moving image and transforms the same into data relating to a non-moving image.

9. The apparatus according toclaim 1, wherein said transforming device is adapted to stabilize the moving image relative to an eye of an individual viewing the moving image.

10. The apparatus according toclaim 1, wherein said transforming device is adapted to stabilize the moving image relative to an object external to an individual viewing the moving image.

11. The apparatus according toclaim 1, wherein:

said aiming device comprises goggles;

said goggles comprise at least one lens area for being viewed;

said displaying device comprising a projector which projects the non-moving image on said at least one lens area.

12. The apparatus according toclaim 11, wherein said at least one lens area is transparent.

13. The apparatus according toclaim 11, wherein said at least one lens area is opaque.

14. The apparatus according toclaim 1, wherein said displaying device comprises a computer screen.

15. The apparatus according toclaim 14, wherein said displaying device comprises a laptop computer screen.

16. The apparatus according toclaim 1, further comprising a contour recognizer for recognizing an outer contour of the moving image.

17. A method of stabilizing an image, said method comprising the steps of:

providing an aiming device;

aiming the aiming device at a moving image;

transforming the image into a non-moving image; and

displaying the non-moving image.

18. The method according toclaim 17, wherein said step of providing an aiming device comprises mounting the aiming device on a human head.

19. The method according toclaim 18, wherein said step of providing an aiming device comprises providing eyeglasses.

20. The method according toclaim 18, wherein said step of providing an aiming device comprises providing goggles.

21. The method according toclaim 17, wherein said transforming step comprises recording the moving image with a camera.

22. The method according toclaim 21, wherein said recording step comprises recording successive discrete frames of the moving image.

23. The method according toclaim 21, wherein said recording step comprises recording the moving image continuously.

24. The method according toclaim 17, wherein said transforming step comprises accepting, at a CPU, data relating to the moving image and transforming the same into data relating to a non-moving image.

25. The method according toclaim 17, wherein said transforming step comprises stabilizing the moving image relative to an eye of an individual viewing the moving image.

26. The method according toclaim 17, wherein said transforming step comprises stabilizing the moving image relative to an object external to an individual viewing the moving image,

27. The method according toclaim 17, wherein:

said step of providing said aiming device comprises providing goggles, said goggles comprising at least one lens area for being viewed; and

said displaying step comprises projecting the non-moving image on said at least one lens area.

28. The method according toclaim 27, wherein said at least one lens area is transparent.

29. The method according toclaim 27, wherein said at least one lens area is opaque.

30. The method according toclaim 17, wherein said displaying step comprises displaying the non-moving image on a computer screen.

31. The method according toclaim 30, wherein said displaying step comprises displaying the non-moving image on a laptop computer screen.

32. The method according toclaim 17, further comprising the step of recognizing an outer contour of the moving image.

33. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for stabilizing an image, said method comprising the steps of:

providing an aiming device;

aiming the aiming device at a moving image;

transforming the image into a non-moving image; and

displaying the non-moving image.