US10223777B2

Movatterモバイル変換

Info

Publication number: US10223777B2
Application number: US15/420,945
Authority: US
Inventors: Al Oleg Baymar
Original assignee: Freedom Scientific Inc
Current assignee: Freedom Scientific Inc
Priority date: 2017-01-26
Filing date: 2017-01-31
Publication date: 2019-03-05
Anticipated expiration: 2037-01-31
Also published as: EP3355274A1; US20190019276A1

Abstract

A method of selectively modifying output of a computer screen. An application hosts an electronic document having a background, alphanumerical characters, and one or more images. A modification tool performs a first modification step in which the modification tool modifies color and/or brightness of all content displayed on the computer screen. The invention identifies and analyzes images within that content to determine whether each image is a real-world image, such as a photograph. A second modification step, which is opposite to the first modification step, is applied to each real-world image, thereby reversing initial modification and returning the image to original, unmodified form. The computer screen outputs the modified electronic document having a modified background and modified text, but outputs the real-world image in its original, unmodified form.

Description

BACKGROUND OF THE INVENTIONField of the Invention

This invention relates to modification of visual output displayed on a computer screen. More specifically, it relates to a method of displaying modified content on a computer screen whereas real-world images are displayed in their original, unmodified form.

Background of the Invention

Computer applications and electronic documents often include dark text on a light background and images having various color characteristics. For some visually-impaired user, it may be difficult to discern conventional text and illustrations. For this reason, there is a number of software products that provide color inversion. These products are designed to modify the colors of the output of the computer screen, which helps the visually-impaired users to have improved clarity and to reduce eye strain.

One of the major disadvantages of the current software products having color/brightness modification capabilities is that the modification affects the entire area of the screen, including real-world images. Such indiscriminate application of color modification results in a serious shortcoming: when the colors of a photograph are inverted, people, animals, and real-world objects depicted in the photograph may become difficult to recognize. This flaw of the existing color inversion products is a source of frustration for many visually-impaired users and, thus, substantially undermines the benefits offered by these products.

Accordingly, what is needed is a screen magnification software that selectively applies color/modification to background, alphanumerical content, and abstract images, while displaying the real world images in their original, non-altered form.

SUMMARY OF THE INVENTION

The invention is a computer-executable method of selectively modifying colors or brightness of a computer screen output. The invention involves a step of displaying a graphic user interface (GUI) desktop and an application window displaying an electronic document. The electronic document includes a background, an alphanumerical content, and an image. The invention modifies colors and/or brightness of the GUI desktop and the electronic document thereby creating a modified graphic user interface desktop and a modified electronic document having a modified background, a modified alphanumerical content, and a modified image. The invention analyzes the electronic document to identify any images contained therein. Then, the invention analyzes the images to determine whether each image is a photograph. If the image is a photograph, the invention applies a second modification step to the modified image. The second modification step is opposite to the first modification step and, therefore, reverses the first modification step creating a final image. The final image is visually equivalent to the image in its original, unmodified form prior to the first modification step. The invention displays the modified GUI desktop and the modified electronic document having the modified background and the modified alphanumerical content. However, the final image is displayed instead of the modified image. This allows the user to see all content in modified colors that the user prefers, while seeing the photographs in their original colors.

In an embodiment, the first modification step is performed by a module external to the application hosting the electronic document. The external module obtains a copy of the modified image and applies the second modification step to the copy of the modified image to produce the final image. The final image replaces the modified image. Consequently, the modified electronic document is displayed on the computer screen with the final image instead of the modified image.

In another embodiment, the second modification step is applied to the modified image directly by the application hosting the electronic document. The application performs this modification as directed by the document processing module of the modification tool. The document processing module runs in the context of the application. In this embodiment, the application hosting the document creates a final image that is visually equivalent to the original, unmodified image.

The first and second modification steps can be an inversion of color or an inversion of brightness. The inversion of color includes a step of assigning to red, green, and blue color channels of each pixel corresponding opposite values. The inversion of brightness includes a step of assigning an opposite value of to each pixel's brightness index. The key is that the second modification step produces the exact opposite effect of the first modification step.

The determination of whether the image is a photograph may be based on an image size, an aspect ratio of the image, a file name of the image, an origin of the image, a number of different colors within the image, a pattern analysis of the image, or a combination thereof.

BRIEF DESCRIPTION OF THE DRAWINGS

For a fuller understanding of the invention, reference should be made to the following detailed description, taken in connection with the accompanying drawings, in which:

FIG. 1A is a conceptual view depicting a computer screen displaying an unmodified output;

FIG. 1B is a conceptual view depicting the computer screen displaying a modified output;

FIG. 1C is a conceptual view depicting the computer screen displaying a selectively modified output, wherein the real-world image is displayed in its unmodified form;

FIG. 2 is a schematic view depicting the components of the invention and their interaction with one another;

FIG. 3 is a flowchart depicting the steps of the method of selectively displaying modified content while displaying real-world images in their original, unmodified form.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The invention pertains to selective modification of color or brightness of the content displayed on a computer screen.FIG. 1A depicts acomputer screen10.Computer screen10 displays a graphic user interface (GUI)desktop12.Computer screen10 also displays anapplication window14 displayingelectronic document16.Electronic document16 includes abackground18,text20, and an image22.

The conventional color scheme and brightness of the content displayed oncomputer screen10 may be uncomfortable for visually-impaired users. Mostelectronic documents16 havedark text20 on a bright,light background18. Some visually-impaired users find such scheme to inflict discomfort and strain on their eyes. These users often prefer the inverted color scheme: lightcolored text20 displayed on adark background18.

Standard screen inversion is known in the art and is illustrated inFIG. 1B. InFIG. 1B, a screen modification tool is used to invert the output ofcomputer screen10.FIG. 1B depicts thatGUI desktop12 andbackground18,text20, and image22 ofelectronic document16 are all displayed using the inverted colors of the original output.FIG. 1A depicts a modifiedGUI desktop24 and a modified electronic document26 having a modifiedbackground28, modifiedtext30, and a modifiedimage32. Although such inversion may help the visually-impaired users to better discern text and icons, the inversion of real-world image22 may alter the appearance of people, animals, and objects depicted therein to an extent where they become difficult to recognize. The present invention addresses this major flaw of the prior art by outputting modifiedGUI desktop24 and modified electronic document26 with an unmodified image22 as depicted inFIG. 1B

FIG. 2 schematically depicts the components of the invention and illustrates their interaction with one another. The invention includescomputer application34 hostingelectronic document16. Adocument processing module36 runs in the context ofapplication34. Animage processing module38 is external toapplication34 and is in communication withdocument processing module36. Amodification tool40 is used to apply color and/or brightness modifications to the entire output ofcomputer screen10.

Modification tool

40 is deployed on the computing device coupled tocomputer screen10.Modification tool40 performs a first modification step to indiscriminately modify color and/or brightness of all visible content output oncomputer screen10. If visible content includes image22, the first modification step creates color-modifiedimage32.Modification tool40 analyzes the visible output to determine whether the output includes image22. After image22 is identified, the modification tool performs analysis to determine whether image22 depicts text, an abstract picture, or a real-world image. In one embodiment of the invention, whenmodification tool40 determines that image22 is a real-world image—i.e., image22 is a photograph or another type of a rendering depicting people or real-world objects—themodification tool40 applies a second modification step to modifiedimage32. The second modification step is opposite of the first modification step, thereby reversing the first modification step. The final image produced by this second modification step has the same colors and brightness as original image22.FIG. 1B depicts the selectively modified output produced by the modification tool:computer screen10 is displaying modifiedGUI desktop24, modifiedbackground28, modifiedtext30, and an unmodified image22.

The invention works only for those modifications which are reversible—i.e. they are applied to original image22 with no information loss and hence can be reverted with an opposite operation. Two examples of such color enhancements are the following: (1) the color of image22 is inverted (by assigning opposite values to red, green and blue color channels of each pixel of image22); and (2) the brightness of image22 is inverted (by calculating the brightness index of each pixel and assigning to it the opposite value while retaining the pixel color). Because the inversion operation is applied to color channels, calculated brightness index can be presented as X=256−Y, it is possible to restore the original Y value by knowing X.

When one of these two modes is applied,modification tool40 analyzes the content ofelectronic document16 displayed inapplication window14. For each identified image22, the modification tool applies certain heuristics to decide whether the image represents text, an abstract picture, or an image of the real world (most often a photograph). The analysis takes into account the image size, aspect ratio, file name, origin, number of different colors, or a combination of these properties.

For those images22 recognized as depicting real objects, a second color modification operation that is opposite to the first color modification operation is applied. There are at least two embodiments of the invention that can be used to accomplish this task. In the first embodiment,modification tool40 interacts withapplication34 hostingelectronic document16.Modification tool40 instructsapplication34 to perform the second modification operation on modifiedimage32. In the second embodiment, instead of performing the second modification step directly withinapplication34, an externalimage processing module38 creates acopy42 of modifiedimage32 and applies the second color modification to copy42. When modified electronic document26 is output ontocomputer display10, modifiedimage32 is replaced withfinal copy44, which is visually equivalent to original, unmodified image22.

Modification tool

40 can use either one of these two methods depending on a runtime decision based on factors such as the image nature, its origin, current color modification mode, or when recovering from a failure while executing one method or the other.

As the result of such manipulations, images22 of the real world object will go through color modification process twice, with one of the phases being performed by themodification tool40 and the other phase performed bydocument processing module36 internal toapplication34 or by externalimage processing module38. The two color modifications will cancel each other so the net effect producesfinal image44 not affected by the color modification at all, whereas the rest ofelectronic document16—text, document background etc.—will still have the desired colored modification.

FIG. 3 is a flowchart illustrating the steps of the selective visual output modification according to the present invention. Instep50

application

34 sends instructions to outputelectronic document16 ontocomputer screen10. Instep52,modification tool40 intercepts those instructions and applies color/brightness modification toelectronic document16 and also toGUI background14. Instep54, the invention analyzes whether modification applied instep52 is reversible. If the modification is not reversible, the process ends. If the modification applied instep52 is reversible, the invention identifies and enumerates images22 withinelectronic document16 instep56. The invention then proceeds to a loop which begins atstep58 and is repeated for each image22 enumerated instep56.

Instep60, the invention determines whether image22 is a real world image (such as a photograph). If image22 depicts texts or an abstract drawing, the invention moves on to the next enumerated image instep62. If, however, image22 is a real world image, the invention proceeds to step64, in which the invention checks whether the image is an original image22 or modifiedimage32. If the image is modifiedimage32, the invention proceeds to step66, in which it determines whether the second modification step can be executed withinapplication34. If so, instep68,document processing module36 applies the second modification step to modifiedimage32 directly within theapplication34 hostingelectronic document16. Otherwise, instep70, a request is sent to externalimage processing module38 to apply the second modification step to modifiedimage32.

Instep72, externalimage processing module38 obtainscopy42 of modifiedimage32. Instep74, externalimage processing module38 applies the second modification step to copy42 to createfinal copy44, which is visually equivalent to original, unmodified image22. Instep76, modifiedimage32 in modified electronic document26 is replaced withfinal copy44.

These steps are asynchronous, whereby images are sent to the queue and the results are provided as they are ready. The steps explained above result in an output on thecomputer screen10 that includes modifiedGUI desktop24, modifieddocument background28, modifiedtext30, and afinal image44 that is visually equivalent to original, unmodified image22.

Hardware and Software Infrastructure Examples

The present invention may be embodied on various computing platforms that perform actions responsive to software-based instructions and most particularly for low-vision user software. These include both traditional desktop and notebooks devices and also smartphones and tablets. The following provides an antecedent basis for the information technology that may be utilized to enable the invention.

The computer readable medium described in the claims below may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any non-transitory, tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. However, as indicated above, due to certain statutory subject matter restrictions, claims to this invention as a software product are those embodied in a non-transitory software medium such as a computer hard drive, flash-RAM, optical disk or the like.

Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire-line, optical fiber cable, radio frequency, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C#, C++, Visual Basic, Objective C, Python or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages.

Aspects of the present invention are described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

It should be noted that when referenced, an “end-user” is an operator of the software as opposed to a developer or author who modifies the underlying source code of the software. For security purposes, authentication means identifying the particular user while authorization defines what procedures and functions that user is permitted to execute.

Glossary of the Claim Terms

Background (of an electronic document)—one or more colors filling the space that is not occupied by alphanumerical characters or images.

Computing device—an electronic device for storing and processing data according to instructions issued by an application or an operating system.

Computer screen—an electronic visual display that is capable of presenting electronic information in a visual form.

Electronic document—information stored in a digital form that can be accessed by an application and output onto a computer screen. Examples of electronic documents include webpages, PDF files, word processing files, electronic spreadsheet, presentation slides, etc.

Final image—the image after it has undergone two modification steps. The second modification step reverses the first modification step and, therefore, the final image is visually equivalent to the original, unmodified image.

Graphic user interface (GUI) desktop—a primary display area of a computer screen that allows users to interact with the computing device through icons and visual indicators.

Image—a set of data that, when processed by a computing device, produces a visual output on a computer screen.

Modified background—the background whose color and/or brightness have been altered from its original, unmodified form.

Modified electronic document—the electronic document having at least some content whose color and/or brightness have been altered from its original, unmodified form.

Modified GUI desktop—the GUI desktop whose color and/or brightness have been altered from its original, unmodified form.

Modified image—the image whose color and/or brightness have been altered from its original, unmodified form.

Photograph—a subcategory of an image that depicts a real-world object and was created using a camera.

The advantages set forth above, and those made apparent from the foregoing description, are efficiently attained. Since certain changes may be made in the above construction without departing from the scope of the invention, it is intended that all matters contained in the foregoing description or shown in the accompanying drawings shall be interpreted as illustrative and not in a limiting sense.

Claims

What is claimed is:

1. A method of selectively modifying colors or brightness of a computer screen output, the method comprising:

providing a computing device coupled to a computer screen;

receiving from the computing device a first data set for outputting onto the computer screen a graphic user interface desktop and an application window displaying an electronic document, the electronic document comprising a background, an alphanumerical content, and an image, the image having original colors;

applying a first modification step to the graphic user interface desktop and the electronic document, the first modification step modifying colors or brightness of the graphic user interface desktop and the electronic document, thereby generating a second data set corresponding to a modified graphic user interface desktop and a modified electronic document, the modified electronic document having a modified background, a modified alphanumerical content, and a modified image, wherein the modified image is visually different from the image in its original form;

analyzing the electronic document to identify the image;

analyzing the image to determine whether the image is a photograph without alphanumeric content;

responsive to identifying the image as a photograph without alphanumeric content, applying a second modification step to the modified image thereby generating a third data set corresponding to a final image, the third data set being separate from the first and second data sets, wherein the second modification step cancels out visual alterations to the image imparted by the first modification step, wherein the final image is visually equivalent to the image in its original form prior to the first modification step;

displaying the modified graphic user interface desktop and the modified electronic document having the modified background and the modified alphanumerical content; and

displaying the final image instead of the modified image.

2. The method ofclaim 1, wherein the first modification step is performed by a module external to the application hosting the electronic document.

3. The method ofclaim 1, further comprising:

obtaining a copy of the modified image;

applying the second modification step to the copy of the modified image to produce the final image;

replacing the modified image in the modified electronic document with the final image, whereby the modified electronic document is displayed on the computer screen with the final image instead of the modified image.

4. The method ofclaim 3, wherein the steps of obtaining the copy of the modified image and applying the second modification step thereto are executed by a module external to the application hosting the electronic document.

5. The method ofclaim 1, wherein the second modification step is applied to the modified image by the application hosting the electronic document.

6. The method ofclaim 1, wherein the first and second modification steps are selected from the group consisting of an inversion of color and an inversion of brightness.

7. The method ofclaim 6, wherein the inversion of color comprises a step of assigning to red, green, and blue color channels of each pixel corresponding opposite values.

8. The method ofclaim 6, wherein the inversion of brightness comprises a step of assigning an opposite value of to each pixel's brightness index.

9. The method ofclaim 1, wherein determination of whether the image is a photograph is based on an image size, an aspect ratio of the image, a file name of the image, an origin of the image, a number of different colors within the image, a pattern analysis of the image, or a combination thereof.

10. A non-transitory computer-readable medium having stored thereon computer-executable instructions that, in response to being executed by a computing device, cause the computing device to perform operations comprising:

using a first set of data to display on a computer screen a graphic user interface desktop and an application window displaying an electronic document, the electronic document comprising a background, an alphanumerical content, and an image;

analyzing the electronic document to identify the image;

responsive to identifying the image as a photograph without alphanumeric content, applying a second modification step to the modified image thereby generating a third data set corresponding to a final image, the third data set being separate from the first and second data sets, wherein the second modification step cancels out visual alterations to the image imparted by the first modification step, wherein the final image is visually equivalent to the image in its original form prior to the first modification step.

11. The media ofclaim 10, wherein the first modification step is performed by a module image processing module external to the application hosting the electronic document.

12. The media ofclaim 10, further comprising:

obtaining a copy of the modified image;

13. The media ofclaim 12, wherein the steps of obtaining the copy of the modified image and applying the second modification step thereto are executed by an image processing module external to the application hosting the electronic document.

14. The media ofclaim 10, wherein the second modification step is applied to the modified image by the application hosting the electronic document.

15. The media ofclaim 10, wherein the first and second modification steps are selected from the group consisting of an inversion of color and an inversion of brightness.

16. The media ofclaim 15, wherein the inversion of color comprises a step of assigning to red, green, and blue color channels of each pixel corresponding opposite values.

17. The media ofclaim 15, wherein the inversion of brightness comprises a step of assigning an opposite value of to each pixel's brightness index.

18. The media ofclaim 10, wherein determination of whether the image is a photograph is based on an image size, an aspect ratio of the image, a file name of the image, an origin of the image, a number of different colors within the image, a pattern analysis of the image, or a combination thereof.