US20090027558A1 - Apparatus and Method for Rendering High Dynamic Range Images for Standard Dynamic Range Display - Google Patents

Abstract

The method, system and apparatus embodiments of the present invention provide for rendering original images having high dynamic range into display images having lower dynamic range while retaining a visual quality of rendered images comparable to the original or reference images. Tone-mapping parameters are applied to the display images and are iteratively adjusted in order to reduce the differences between the conditioned original image and the conditioned display image.

Description

FIELD OF THE INVENTION
• The embodiments of the present invention relate to apparatuses and methods of rendering high dynamic range images for standard dynamic range display, particularly renderings using visual models, display models, and the tuning of parameters to achieve perceptually acceptable renderings at dynamic ranges lower than the dynamic ranges of the source images.
BACKGROUND
• A principal goal of a tone-mapping problem is to reproduce a scene on a display so that the displayed image is visually perceived to be an acceptably close or is a completely faithful reproduction of the original image. An identical, or perfect, match between the original image and its rendering on a display or in a hard-copy format is almost never practicable. The impracticability may be attributed in part to an output medium that is typically insufficiently bright in the face of ambient light and may be unable to display the image with a dynamic range, i.e. contrast, and color gamut, that faithfully represents the source image. Often the rendering on a perceptually acceptable output device becomes a trade-off between preserving certain image features at the cost of a degradation of others image features. For example, high contrast and brightness of an image can be often preserved only at the cost of incurring clipping, i.e., the saturating of certain amount of pixels in bright or in dark regions. The determination of those features deemed more important than others for faithful rendering may be driven by particular applications, and for a particular application, a metric may be applied, based on some aspects of the visual perception, to drive the resulting display image perceptually close to the original image.
SUMMARY
• The invention, in its several embodiments includes methods, systems and apparatuses for rendering original images having high dynamic range into display images having lower dynamic range while retaining a visual quality of rendered images comparable to the original or reference images. Tone-mapping parameters are applied to the display images and are iteratively adjusted in order to reduce the differences between the conditioned original image and the conditioned display image. An exemplary method of image rendering of the present invention includes steps of: (a) emodifying a reference image luminance based on a luminance perception model as part of a human visual system model; (b) shaping or filtering the spatial frequency transform of the modified display image with a contrast sensitivity function and generating a reference response image; (c) modifying a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter; (d) optionally modifying a display image chrominance based on a white light bias parameter; (e) if, on a pixel-by-pixel basis, a display image color radiance component exceeds a display-limited range, then optionally limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value; (f) modifying a display image luminance based on the luminance perception model of the human visual system model; (g) optionally shaping or filtering a spatial frequency transform of the modified display image with the contrast sensitivity function and thereafter generating a display response image; (h) comparing the display response image with the reference response image to generate an error value; (i) if the error value exceeds a threshold value; then modifying at least one of: the brightness parameter, the sharpness parameter, the contrast parameter, and optionally the white light bias; and (j) repeating steps (c)-(i) while the error value exceeds the threshold value. In some process or method embodiments, the contrast sensitivity function may be a function of viewing distance. In some method embodiments, the human visual response model comprises, as a human luminance perception model, a just-noticeable-difference (JND) encoding function. In some method embodiments, the error value may be generated via a mean square error (MSE) process and in some embodiments, a process iteration test threshold may be based on either a minimal MSE value and/or a determination that the maximal pixel difference between the display response image and the reference response image is below a preset threshold.
• An exemplary system embodiment of the present invention includes: a first display image luminance processing module having a processing unit, and addressable memory, the processing module adapted to modify a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter; a second display image luminance processing module adapted to modify a display image luminance based on a human luminance perception model as part of the human visual system model; a shaping or filtering module adapted to shape the spatial frequency response of the modified display image with a contrast sensitivity function and adapted to generate a display response image; a comparator module adapted to generate the error value based on the generated display visual response image and a reference visual response image; and a parameter processing module having a processing unit, and addressable memory adapted to modify at least one of: the brightness parameter, the sharpness parameter, the contrast parameter, and, optionally, a white light bias; wherein the parameter modification is based on the generated error value. One of ordinary skill in the art will appreciate that functionality may be distributed across or within different modules of various system embodiments and still fall within the contemplated scope of the invention. The exemplary system embodiment may further include a display processing module having a processing unit, and addressable memory wherein the display processing module is adapted to modify a display image chrominance based on a white light bias parameter. The display processing module of the exemplary system embodiment may be further adapted to limit, on a pixel-by-pixel basis, when a display image color radiance component exceeds a display-limited range, then limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value. The exemplary system embodiment may further include a reference visual image processing module having a processing unit, and addressable memory wherein the reference image processing module is adapted to modify a reference image luminance based on the human luminance perception model of the human visual system model and shaping or filtering the modified display image spatial frequency transform with a contrast sensitivity function and generate a reference visual response image.
• An exemplary apparatus embodiment of the present invention may include a processing module having a processing unit, and addressable memory wherein the processing module tests on whether a generated error value is less than a threshold value and wherein the processing module is adapted to: modify a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter; modify a display image luminance based on a human luminance perceptual model as part of a human visual system model; shape or filter the spatial frequency transform of the modified display image with the contrast sensitivity function and generate a display visual response image; compare the generated display visual response image with a reference visual response image to generate the error value; and modify at least one of: the brightness parameter, the sharpness parameter, the contrast parameter, and the white light bias; when the generated error value is above a threshold value. The processing module of the exemplary apparatus may be further adapted to modify a display image chrominance based on a white light bias parameter, and limit, on a pixel-by-pixel basis, when a display image color radiance component exceeds a display-limited range, then limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value. The processing module of the exemplary apparatus may be further adapted to modify a reference image luminance based on the human luminance perception model as a portion of the human visual system model and shape or filter the spatial frequency transform of the modified display image with a contrast sensitivity function and generate a reference visual response image.
BRIEF DESCRIPTION OF THE DRAWINGS
• The present invention is illustrated by way of example and not limitation in the figures of the accompanying drawings, and in which:
• FIG. 1 is an exemplary top-level functional block diagram of a process embodiment of the present invention;
• FIG. 2 is an exemplary top-level flow chart of a process embodiment of the present invention;
• FIG. 3 is an exemplary functional block diagram of a portion of a process embodiment of the present invention;
• FIG. 4 is an exemplary functional block diagram of a portion of a process embodiment of the present invention;
• FIG. 5 is an exemplary functional block diagram of a portion of a process embodiment of the present invention;
• FIG. 6 is an exemplary functional block diagram of a portion of a process embodiment of the present invention;
• FIG. 7 is an exemplary functional block diagram of a portion of a process embodiment of the present invention;
• FIG. 8 is an exemplary functional block diagram of a portion of a process embodiment of the present invention;
• FIG. 9A is an exemplary graph of sensitivity models as functions of luminance;
• FIG. 9B is an exemplary graph of a contrast sensitivity model as a two-dimensional function of spatial frequency;
• FIG. 10 is an exemplary graph of normalized contrast sensitivity functions shown along one spatial frequency axis;
• FIG. 11 is an exemplary functional block diagram of a portion of a process embodiment of the present invention;
• FIG. 12 is an exemplary graph of a visual error performance index as a function of brightness and contrast tone-mapping parameters;
• FIG. 13A is an exemplary graph of a resulting brightness, contrast, and sharpness tone-mapping parameters for a particular HDR display effective dynamic range as functions of ambient illumination;
• FIG. 13B is a sample HDR image
• FIG. 13C is a sample HDR image at a higher ambient illumination level;
• FIG. 14A is an exemplary graph of a resulting brightness, contrast, and sharpness tone-mapping parameters for a particular SDR display effective dynamic range as functions of ambient illumination;
• FIG. 14B is a sample SDR image;
• FIG. 14C is a sample SDR image at a higher ambient illumination level;
• FIG. 15A is an exemplary graph of a resulting brightness, contrast, and sharpness tone-mapping parameters as functions display brightness;
• FIG. 15B is a sample image at 10 candelas per square meter;
• FIG. 15C is a sample image at 1,000 candelas per square meter;
• FIG. 16A is an exemplary graph of a resulting brightness, contrast, and sharpness tone mapping parameters as functions display distance, normalized by display height;
• FIG. 16B is a sample image at twice the display height;
• FIG. 16C is a sample image at twelve times the display height;
• FIG. 17 is a high level functional block diagram of an exemplary system embodiment of the present invention; and
• FIG. 18 is a high level functional block diagram of an apparatus embodiment of the present invention.
DETAILED DESCRIPTION
• When the dynamic range, i.e., contrast ratio, or color gamut of an original image exceeds the capacity of the target display device intended to be used to reproduce a faithful image, processing techniques may be applied to generate an image as perceptually faithful an image as practicable within the constraints of the display device itself. Some display methods map the image code value to the achievable range of the display by applying a tone curve. Such a tone curve in application is rarely adapted to adjust for image content or viewing conditions. For an example as to viewing conditions, there may be no adjustments in the applied tone curve for the effect of changes in ambient light level. Accordingly, more faithful reproductions in a displayed image may be achieved where the rendering method adapts to the viewing conditions, display capabilities of the target display and image content.
• The present invention, in its several embodiments, adaptively renders an image of an arbitrary dynamic range and a wide color gamut onto a specified display device, that is, a rendering made under the constraints of the capabilities and capacity of the display device, where rendering with limited display capabilities can mean rendering within the available range of white and black level and the color gamut. That is, the present invention, in its several embodiments, renders an image via the target display device that, when compared to existing methods, has a reduced humanly perceptible visual difference between the rendered and the original images.
• Method embodiments of the present invention take, as an input, a high quality—preferably high dynamic range—image and data on viewing conditions (e.g., distance from the display screen, ambient light level, and surround) and produces as a result a display-adapted image. The display-adapted image is, for purposes of the selected objective function, the result of an iteratively optimized rendering of the input image under given display conditions and constraints of the target display, particularly constraints such as maximum brightness and contrast range. Exemplary system embodiments of the present invention, by which a display-adapted image may be iteratively rendered according to a selected portion of the human visual system, may be comprised of two or more processing modules, or network nodes, wherein each node or processing module has computing functionality, e.g., a general or special microprocessor, and an addressable memory. One of ordinary skill in the art will appreciate that functionality may be distributed across or within different modules of various system embodiments and still fall within the contemplated scope of the invention. In addition, exemplary apparatus embodiments of the present invention, having at least one processing module and addressable memory, may execute steps of the disclosed method embodiments and in doing so iteratively render a display-adapted image according to a selected portion of the human visual system.
• Having an original image as input, which can be in high dynamic range (HDR) or any high quality format, an iterative goal may be to generate a display-adapted image that would appear to a human viewer to be a faithful rendering of a reference original image. A response image, R_disp, of the human visual system for an image shown on the display device may not be exactly the same as a response image, R_orig, evoked by the original or reference scene due to differences in viewing conditions between the two images and to such limitations in the display device as a limited dynamic range and color gamut. For purposes of illustration, one may presume that the iterative goal of nearly faithful reproduction is practicably achieved if the response of the human visual system for an image shown on the display, R_disp, is made practicably close to the response of the human visual system evoked by the original or reference scene, R_orig. Tone mapping parameters may be adjusted to modify the rendered image on the display device and the direction of their adjustments may be driven in directions to reduce the difference between R_origand R_dispin an iterative fashion.
• Referring toFIG. 1, what may be termed anoriginal image102, having a high dynamic range, may be operated on by a tone-mapping operator104, having tone-mapping parameters106 and thereby render a display-adaptedimage108. Embodiments of the present invention, via anerror minimizing process110, iteratively adjust the tone-mapping parameters106 from their initial values in directions that also reduce the difference between a response of theoriginal image112, R_ORIG, and a response of the display-adaptedimage114, R_DISP. In order to generate the representation of theoriginal image112, R_ORIG, a humanvisual system model116 may be applied that takes in theoriginal image102 that may be optionally enhanced118 and may incorporate viewing conditions, e.g., viewing distance, of theoriginal image120 to produce the response of theoriginal image112. In order to generate the response of the display-adaptedimage114, R_DISP, a humanvisual system model122 may be applied that takes in the display-adaptedimage108 after it is conditioned by adisplay model124 and may incorporateviewing conditions126, the viewing conditions of the display-adaptedimage108, to produce the response of the display-adaptedimage114, R_DISP.
• An exemplary process flow may be represented inFIG. 2 where the inputs are ahigh quality image202, e.g., an image having high dynamic range, and initial values for tone-mapping parameters204. The high quality image may be optionally enhanced206. A human visual system model (HVSM) is selected210 to map thehigh quality image207 into a representation of the human-perceived model of thehigh quality image211. The human-perceived model of thehigh quality image211 is used to determine212 the response of theoriginal image112, R_ORIG. The high quality image also provides areference207 from which the display-adaptedimage209 is generated208 based on tone-mapping parameters205. A display model may be applied214 to the display-adaptedimage209, or DAI, and after application of the display model the resultingDAI215 may then provide the basis for determining216, via the human visual system model, the response of the display-adaptedimage114, R_DISP. A performance index based on characterizing thedifference219 between the response of theoriginal image112, R_ORIGand the response of the display-adaptedimage114, R_DISP, may be generated (step218). The display-adaptedimage114, R_DISP, may then be iteratively regenerated using adjusted tone-mapping parameters224 to reduce a characterization of the difference between the response of the display-adaptedimage114, R_DISP, and the response of theoriginal image112, R_ORIG. A stopping rule for iterative adjustments to the tone-mapping parameters may be based on comparisons of current and previous performance index values and/or absolute performance index values and/or maximum global pixel difference.Such test values219 may be compared220 with a threshold value and if the test value being applied219 is found to be below a threshold, then tone-mapping parameters used to generate the display-adapted image are treated as satisfactory, requiring nofurther adjustment222. If however, the test value being applied219 is above the threshold, then the tone-mapping parameters are adjusted224, typically based on an error metric generated in the course of comparing the display-adaptedimage114, R_DISP, and theoriginal image112, R_ORIG, and the adjusted tone-mapping parameters are used in the determination of a recalculated response of the display-adaptedimage114, R_DISP. In this way, the cycle may be repeated until the test value being applied219 is found to be below the selected threshold.
Tone-Mapping Operator
• As part of a tone-mapping operator (TMO) embodiment for a particular application, a suitable perceptual model may be applied and error metric may be selected. For example, an operator that may produce a visual response found to be visually pleasing to humans, could be applied with an appearance model, while an operator that may maximize visibility, e.g., for aircraft cockpit displays, could be applied with a detection model. The particular tone-mapping process may be customized to its particular application, where such customization may include ranging for brightness adjustment, e.g., the level of backlight in a mobile display, to local tone-mapping operators used to process digital photographs. Often in computationally practicable applications, an optimization framework may be a simplified framework that, if possible, may be reduced to a close-form solution. Existing tone-mapping operators may be integrated into embodiments of the present invention where steps are taken to automatically adjust the TMO parameters for a given image, display and viewing condition.
• An exemplary tone mapping operator is described having adjustable parameters.FIG. 3 illustrates a process where animage310 may be characterized by itsluminance311 and its chrominance. It may also be represented in XYZ color space havingpixel components X312,Y313, andZ314. The image luminance may be termed as theinput luminance311, L_IN, and may be processed by luminance tone-mapping320 using a subset of the exemplary tone-mapping parameters330 comprised ofbrightness331,contrast332 andsharpness333, in order to generate a tone-mappedoutput luminance321, L_OUT. That is, theoutput luminance321, L_OUT, may be expressed as a linear combination of a mean luminance component, a high pass luminance component, and a low pass luminance component where each component is weighted by a tone-mapping parameter from the set of tone-mapping parameters330. A resulting tone-mappedimage340 may be generated based on theoutput luminance321, L_OUT, and this resulting tone-mappedimage340 may be represented in XYZ color space where the tone-mapped X component may be represented asX_TMO341, the tone-mapped Y component may be represented asY_TMO342 and the tone-mapped Z component may be represented asZ_TMO343. The tone-mappedcomponents X_TMO341,Y_TMO342, andZ_TMO343 may be further processed via white light tone-mapping350 that employs awhite light weighting334 and references theimage310 as represented inXYZ space312, i.e., the original pixel component values,X312,Y313, andZ314. The further processed tone-mappedcomponents X_TMO351,Y_TMO352, andZ_TMO353 may then represent a tone-mapped imaged360 affected by four tone-mapping parameters:brightness331,contrast332,sharpness333, and awhite light weighting334.
• FIG. 4 illustrates and exemplary functional block diagram of the luminance tone-mapping processing320 for forming theoutput luminance321, L_OUT. The mean luminance of all the pixels of an input image may be determined via amean processing module410 as the “direct current,” or DC,luminance412, L_DC. Abilateral filter420 such as a fast or piecewise-linear bilateral filter of U.S. Pat. No. 7,146,059 that issued to Durand et al. Dec. 5, 2006, may be applied to theinput luminance311, L_IN, to determine thelow pass component421, L_LP, after theDC luminance412 is extracted from theoutput422 of thebilateral filter420. The exemplarybilateral filter420, when represented as a function, has as arguments L_IN, δ_Rand δ_S. In this example, δ_Ris the width of theGaussian kernel424 used for value range weighting in the bilateral filter processing and may be set to a value of 1.46, which is the logarithm of the ratio between black and white diffuse surfaces (i.e., log₁₀(0.90/0.31)=1.46). Also in this example, δ_Sis a parameter representing the width of the Gaussian kernel forspatial filtering426 and may be set adaptively based on the viewing distance. Accordingly, the lowpass luminance component421, L_LP, may be expressed as the output of the bilaterally filtered input luminance, L_IN, less theDC component412, L_DC, as follows:
• 
  L_LP=bilateral(L_IN,δ_S,δ_R)−L_DC.  [Eq. 1]
• The highpass luminance component430, L_HP, may be expressed as theinput luminance311, L_IN, less the lowpass luminance component421, L_LP, and less theDC component412, L_DC, as follows:
• 
  L_HP=L_IN−L_LP−L_DC.  [Eq. 2]
• Three of the tone-mapping parameters330 may be applied to the highpass luminance component430, L_HP, the lowpass luminance component421, L_LP, and theDC component412, L_DC. In particular, thebrightness modulation value331, bri, thecontrast modulation value332, con, and thesharpness modulation value333, shr, may be applied respectively to weight themean pixel luminance412, L_DC, the lowpass luminance component421, L_LP, and the highpass luminance component430, L_HP, and to generate theoutput luminance321, L_OUT, as follows:
• 
  L_OUT=bri*L_DC+con*L_LP+shr*L_HP.  [Eq. 3]
• The saturation of colors of an image is perceived by human vision to increase with the compression of luminance contrast. While one may invoke desaturation of all of the pixels of the image to address this perceptual artifact of the luminance contrast compression, a present exemplary process adds white light to an image. In particular, white light may be added to components of the luminance tone-mapped image in the XYZ space of pixel component values. Also, X_W, Y_Wand Z_Wmay represent the normalized luminance components of white color (i.e., 0.95; 1; 1.089 for a D65 white point, International Commission on Illumination), and with may be a white light weighting.
• FIG. 5 illustrates further tone-mapping350 using white light weighting where the normalized luminance ofwhite color510, expressed inpixel components X_W512,Y_W514 andZ_W516 are each weighted by the whitelight weighting parameter334, wlt. A tone-mapped component,X_TMO341, may be expressed in relation to the original pixel component values, X, Y and Z, and as a function of the white light weighting, wlt, as follows:
• 
  X_TMO=((Y_TMO/Y)*X+wlt*X_W)/(R*(X+Y+Z)+wlt*(X_W+Y_W+Z_W));  [Eq. 4]
• and similarly, a tone-mapped component, Y_TMO, may be expressed as follows:
• 
  Y_TMO=((X_TMO/X)*Y+wlt*Y_W)/(R*(X+Y+Z)+wlt*(X_W+Y_W+Z_W)).  [Eq. 5]
Image Enhancement
• Image enhancement, which modifies the original image to improve its appearance, may optionally be integrated into tone-mapping framework embodiments of the present invention. Human viewers tend to prefer images that are sharper, have higher contrast and more saturated colors, as would be expected when viewing the original scenes from which the images are based. Image enhancement may also be described as a restoration process, where distortions produced in the recording of images, such as camera distortions, are removed from the captured image. The typical modulation transfer function (MTF) of a lens system is lower than one at low spatial frequencies, thus accounting for degraded image contrast, and the MTF is even lower for high spatial frequencies, which accounts for reductions in sharpness. Accordingly, sharpening and contrast enhancement together may be regarded as a restoration process for such camera distortions.
Display Model
• A display model processing module or component may be applied to account for some of the limited capabilities of the target, or low dynamic range, display device, where the limited capability may include maximum brightness, dynamic range (contrast ratio) and color gamut (color primaries). These display capabilities are affected by the technical aspects of a display, as well as viewing conditions, such as ambient light that is reflected from a display panel. Illustrated by example inFIG. 6 is aprocess600 whereby the tone-mappedimage360 may be adapted for the limitation of the target display resulting in a display-adaptedimage610. The tone-mappedimage360 may be expressed in code values, e.g., a gamma-corrected color space, such as sRGB. That is, the tone-mappedimage360 may be expressed as a redradiance color component621, R_TMO,RED, a greenradiance color component622, R_TMO,GREEN, and a blueradiance color component623, R_TMO,BLUE. A display model may provide, as a result, the actual calorimetric values of a displayed image. If these values exceed the display capabilities, the values may be limited or clamped to the corresponding extremes or limits. For example, the lowest radiance producible by the target display for a particular color component, i.e., red, green, or blue color components, may be represented by a known value, d_min, and the highest radiance producible by the target display for the same particular color component may be represented by a known value, d_max. So, the redradiance color component621, R_TMO,RED, may be tested and limited by ared threshold limiter630 from above by a d_maxvalue for thered component631 and limited from below by d_min, value for thered component632. Likewise, the greenradiance color component622, R_TMO,GREEN, may be tested and limited by agreen threshold limiter640 from above by a d_maxvalue for thegreen component641 and limited from below by d_min, value for thegreen component642; and the blueradiance color component623, R_TMO,BLUE, may be tested and limited by ablue threshold limiter650 from above by a d_maxvalue for theblue component651 and limited from below by d_min, value for theblue component652. The output radiance value of the display model for a particular color component may be expressed as:
• 
  R_disp=[R_tmo]_{d min}^{d max}+R_refl  [Eq. 6]
• where [.] operator restricts the values to be within the range from d_minto d_maxand where a radiance bias, R_refl, is added to the potentially limited radiance to account for the amount of ambient light reflected from the panel. The red color component ambient light reflection bias661, R_refl,red, may be approximated by:
• 
  R_refl,red=w_R*E_amb/πk,  [Eq. 7]
• where E_ambis ambient luminance given in lux, w_Ris a weighting factor for the red color component (e.g., w_R=0.2126; for the green color component: w_G=0.7152; and for the blue color component: w_B=0.0722, all for a D65 reference), and k is the reflectivity for a display panel, e.g., about 1% for LCD displays and may be expressed as a larger percentage for CRT displays. Accordingly, the green ambientlight reflection bias662, R_refl,greenmay be approximated by:
• 
  R_refl,green=w_R*E_amb/πk,  [Eq. 8]
• And the blue ambientlight reflection bias663, R_refl,blue, may be approximated by:
• 
  R_refl,blue=w_R*E_amb/πk  [Eq. 9]
• Thered radiance component621 is tested630 and may be limited and the testedred radiance component635 may be combined with the red ambient light reflection bias661 to generate a display-adaptedred radiance component671. Thegreen radiance component622 is tested640 and may be limited and the testedgreen radiance component645 may be combined with the green ambientlight reflection bias662 to generate a display-adaptedgreen radiance component672. Theblue radiance component623 is tested650 and may be limited and the testedblue radiance component655 may be combined with the blue ambientlight reflection bias663 to generate a display-adaptedblue radiance component673. The display adapted radiance components671-673 may be combined to generate a display-adaptedimage610.
Human Visual System Model
• In general, a human visual system model may include representations of the human such as color, eye movements, attention, temporal response, masking and stochastically corrupted representations that may be termed noise and for the present description, include luminance and spatial frequency responses. Illustrated inFIG. 7 is an exemplary top-level architecture model of an exemplary humanvisual system710 that processes, as its input image data, e.g., luminance andchrominance data711, in order to produce anestimated response715 of the human visual system. The tone-mapping framework of the several embodiments of the present invention is general enough to accommodate a range of the visual models. A visual model, when applied to an image, can be a key component of the overall tone-mapping framework and accordingly, the visual model applied is typically profiled for a particular application. The estimatedresponse715 of the human visual system typically scales image features relative to their importance, according to the human visual system model selected716 based on the intended application orapplication state712. For example, thevisual detection model714, described below as an example of a luminance perception model, reduces the importance of very low and high frequencies because the actual human visual system is less sensitive to very low and high frequencies. For example, a tone-mapping for processing photographs may be better processed by the application of anappearance model713, while the tone mapping for maximizing visibility of displayed information (e.g. cockpit display) may be better processed by the application of adetection model714.
• FIG. 8 illustrates an example where the processing of the exemplary detection model as exemplary portions of an HVS model is comprised of two stages. The exemplary first stage comprises a humanperception luminance model820 of high dynamic range images via an exemplary extension of gamma curves used for encoding of standard dynamic range (SDR) images to the full visible range of luminance. In this example, the luminance is converted via a luminance sensitivity function to a response value that is approximately scaled in units of luma, which is a pixel gray-value used with a human luminance perception model such at the exemplary just-noticeable-difference (JND) encoding. The exemplary encoding may be derived from theluminance sensitivity function822 and it may be referenced in order to scale luminance in approximately perceptually uniform units.
• The luminance may be modified by a human luminance perception model.FIG. 9A shows the shape of the exemplary luminance encoding or modifyingfunction910, labeled JND encoding, compared with atypical SDR encoding920, e.g., gamma correction for thesRGB color space920, andlogarithmic compression930.FIG. 9B shows an exemplary CSF as a function of two spatial dimensions, where the spatial dimensions are in units of cycles per degree. The exemplary CSF may be applied to shape the spatial frequency response, for example, the response of the image after modification by a human luminance perception model.
• Returning toFIG. 8, as part of the second stage, the image of the response values823 may be filtered830 in the 2D Fourier (spatial frequency)domain824 by a normalized contrast sensitivity function (CSF)840. A CSF may be characterized as a two-dimensional spatial filter. An exemplary CSF may be applied by example by assuming a test patch size of one-visual degree and by neglecting no effects of orientation and eccentricity. The CSF values may be adjusted for low frequencies, so that the minimum normalized sensitivity is 20%. An exemplary normalized CSF may be approximated by:
• $\begin{array}{cc}\mathrm{CSF}\left(\rho ,l_{\mathrm{adapt}}\right)=\{\begin{array}{cc}\exp \left(-\frac{{\rho /{\rho }_{\mathrm{peak}}-1}^{{\mathrm{<figure-callout\; label="\beta "\; filenames="US20090027558A1-20090129-D00009.png,US20090027558A1-20090129-D00010.png"\; state="\{\{state\}\}">\beta </figure-callout>}}_1}}{2{\mathrm{<figure-callout\; label="\sigma "\; filenames="US20090027558A1-20090129-D00009.png,US20090027558A1-20090129-D00010.png"\; state="\{\{state\}\}">\sigma </figure-callout>}}_1^{{\beta }_1}}\right);\mathrm{for}& \rho \le {\rho }_{\mathrm{peak}}\\ \exp \left(-\frac{{\rho /{\rho }_{\mathrm{peak}}-1}^{{\mathrm{<figure-callout\; label="\beta "\; filenames="US20090027558A1-20090129-D00010.png,US20090027558A1-20090129-D00013.png"\; state="\{\{state\}\}">\beta </figure-callout>}}_2}}{2{\mathrm{<figure-callout\; label="\sigma "\; filenames="US20090027558A1-20090129-D00010.png,US20090027558A1-20090129-D00013.png"\; state="\{\{state\}\}">\sigma </figure-callout>}}_2^{{\beta }_2}}\right);\mathrm{for}& \rho >{\rho }_{\mathrm{peak}}\end{array}& \left[\mathrm{Eq}.10\right]\end{array}$
• where ρ is the spatial frequency given in cycles per degree, σ₁=0.61059, β₁=2.362, σ₂=0.95127, β₂=1.1453, and where ρ_peakis the frequency of the peak sensitivity and is given by:
• $\begin{array}{cc}{\rho }_{\mathrm{peak}}=\frac{a}{b+l_{\mathrm{adapt}}^d}+c& \left[\mathrm{Eq}.11\right]\end{array}$
• where the parameter l_adaptis the adaptation luminance in candela per square meter (cd/m²) and the constants are: a=1.1453, b=1.4607, c=5.1159 and d=0.50285.
• Human contrast sensitivity and accordingly the various models of CSF strongly depend on the viewing distance. Since observers are unlikely to watch images from a fixed distance, a CSF may be generated for the minimum and maximum assumed viewing distances and then an envelope of these two resulting functions may be used to approximate CSF for the range of viewing distances. Such refined CSF models may then be applied in the various embodiments of the present invention. Examples of CSF functions for different viewing distances are shown inFIG. 10 plotted as the modulation of the contrast sensitivity as a function of spatial frequency in cycles per pixel. Normalized for display heights, the following CSF curves may be referenced for fixed viewing distances: (a) F1 represents a viewing distance of 0.5 display heights; (b) F2 represents a viewing distance of 1.5 display heights; (c) F3 represents a viewing distance of 2.5 display heights; (d) F4 represents a viewing distance of 3.5 display heights; (e) F4 represents a viewing distance of 11.5 display heights; and (f) F4 represents a viewing distance of 12.5 display heights. Normalized for display heights, the following CSF curves may be referenced for a range of viewing distances: (a) G1 represents a viewing distance range of 0.5 to 1.5 display heights; (b) G2 represents a viewing distance range of from 2.5 to 3.5 display heights; and (c) G3 represents a viewing distance range of from 11.5 to 12.5 display heights. Accordingly, the CSF curves of G1, G2, and G3 may be applied as shaping functions in two spatial dimensions, e.g., in the 2D Fourier conditioning or shaping of the image in the spatial frequency domain.
Error Metric
• An error metric in the framework of the several embodiments of the present invention may be responsible for computing the difference between the response of the selected portions of the human visual system model as applied to the original scene and the response of displayed image, as affected by the tone-mapping parameters via the TMO. Such an error metric may yield a single number, which can be designed to be the value of the objective function that may be minimized via the variation in the tone-mapping parameters. One example of an objective function may be the mean square error, MSE, across a display image comprised of M by N pixels:
• $\begin{array}{cc}\mathrm{MSE}=\frac{1}{M\times N}\sum _{m=0}^{M-1}\sum _{n=0}^{N-1}{\left(R_{\mathrm{orig}}\left[m,n\right]-R_{\mathrm{disp}}\left[m-n\right]\right)}^2.& \left[\mathrm{Eq}.12\right]\end{array}$
• An exemplary stopping rule is where the MSE is below a threshold, MSE_min, i.e.,
• 
  MSE<MSE_min,  [Eq. 13]
• and/or where largest pixel-by-pixel difference, i.e., the maximal pixel difference, is below a threshold, R_error, i.e.,
• 
  max_M,N(R_orig[m,n]−R_disp[m,n])<R_error.  [Eq. 14]
• Illustrated inFIG. 11 is an exemplary process flow diagram for both the determination of an error and a stopping test where avisual display response114, R_disp, and the original orreference scene112, R_orig, are compared via, for example, the objective function, where the resulting error metric made may be driven practicably close by changes in the tone-mapping parameters224 to effect changes in the responsevisual display response114, R_disp. In some embodiments, once the error metric1110, or changes in the error metric1140, are driven to values below athreshold1120, the adjusting of the tone-mapping parameters is stopped1130. Otherwise, the adjustments to the tone-mapping parameters224 may be made to cause the resulting objective function to further decrease, as in the case of a minimization formulation. In other embodiments, the error metric may drive the magnitude and direction of adjustments in one or more of the tone-mapping parameter while a separate stopping test is made, pixel-by-pixel, of the maximum difference between thevisual display response114, R_disp, and the originalvisual display response112, R_orig. Other parameter adaptation mechanisms may be employed as well as other stopping rules without deviating from the scope of the invention disclosed by way of exemplary embodiments herein.
• Accordingly, the reduction in thedifference display response114, R_disp, and the original orreference scene112, R_orig, may be accomplished by characterizing the process as a nonlinear optimization problem, where a performance index, J, is a nonlinear function of the image response error having tone-mapping parameters: brightness (bri), contrast (con), sharpness (shr), and optionally white light weighting (wlt). The pixel-by-pixel comparison of the two two-dimensional matrices may be cast as, for example:
• 
  J=MSE(R_orig−R_disp)  [Eq. 15.1]
• 
  Min J  [Eq. 15.2]
• Where the error term, R_orig-R_disp, is a nonlinear function of: con, bir, shr and wlt for a fixed CSF, as has been described by example herein.
• 
  Constraints: 0.0≦con≦1.5;  [Eq. 15.3]
• 
  −2.0≦bri≦1.0;  [Eq. 15.4]
• 
  −2.0≦shr≦3.0;  [Eq. 15.5]
• 
  and optionally,
• 
  −2.0≦wlt≦3.0.  [Eq. 15.6]
• For example, the minimization algorithm may be selected, by those of ordinary skill in the field, from constrained numerical gradient search processes including Modified Method of Feasible Directions.
• An exemplary objective function, termed also as visual error or error metric, may yield a parabolic surface as shown by example inFIG. 12 when variations in contrast and brightness tone-mapping parameters are made. Such a parabolic function indicates a local stable solution that is independent of the particular bounded, initial, or starting, tone-mapping parameters.
EXAMPLES
• The invention, in its several embodiments accommodates a range of tone-mapping operators and ranges, viewing conditions including ambient light and viewing distance applied with visual models used for modifying images. The following exemplary results include studies in ambient illumination, brightness of a target display, and the viewing distance. For these examples, the visual detection model explained above is applied, as are the tone mapping operator and its tone-mapping parameters as described in the previous sections. Calibrated HDR and SDR images were used as input each exemplary process.
Ambient Illumination Scenario
• In the ambient illumination scenario, a change of an ambient light level is simulated to reflect the environment when the same, presumably mobile, display is viewed both indoors and outdoors. In cases of immobile displays, the change in ambient light may be due to such circumstances as daylight through windows and changes in lamp illumination. The exemplary display has a contrast ratio, i.e., dynamic range, of 1:100 and maximum brightness of 200 candles or candelas per square meter (cd/m²). The change of tone-mapping parameters for the HDR image is shown inFIG. 13A with the images of the extreme cases shown inFIGS. 13B and 13C. The change of tone-mapping parameters for the SDR image is shown inFIG. 14A with the images of the extreme exemplary ambient illumination cases shown inFIGS. 14B and 14C.
• Elevated ambient light level has the largest impact on the black level of the display, i.e., the luminance of a black pixel. As the ambient light increases, more light is reflected from the panel, effectively reducing the dynamic range of a display. Also, as a displayed image is pushed toward higher luminance levels, with increasing black levels, the contrast is reduced and the perceived sharpness of an image is typically reduced as well. To compensate for this and to preserve low contrast details, the exemplary tone-mapping process increases sharpness of an image. In this scenario, there is a significant difference in contrast modulation between HDR and SDR image. Since the dynamic range of HDR image cannot be accommodated on the target display, it must be reduced to at least 50%. However for the SDR image, the contrast is evenly enhanced for high ambient, to compensate for the loss of contrast. Such enhancement is usually done at the cost of clipping the brightest and the darkest pixels.
Brightness of a Display
• In the brightness of a display scenario, a set of simulated displays of different maximum brightness were presumed which vary from 10 to 1000 cd/m². This is also the situation when the backlight of an LCD display is modulated. The dynamic range of the display is the same as in the previous example, i.e., 1:100. The change of the tone mapping parameters is shown inFIG. 15A with the images of the extreme cases shown inFIGS. 15B and 15C. As the display gets brighter, its effective dynamic range, as may be limited by panel reflections, gets higher and better image reproduction can be achieved. For this reason, both contrast and sharpness parameters converge toward the value of one. Note that the value of one would indicate no change of contrast and sharpness and therefore perfect reproduction.
Viewing Distance
• In the viewing distance scenario, an observer is simulated as moving away from the display. This effect is illustrated inFIG. 16A by the viewing distance starting at a normalized distance of two display heights and stopping at twelve display heights. The viewing distance significantly influences the contrast sensitivity for high and low spatial frequencies as was illustrated by example inFIG. 10. At larger distances, relatively small details in the window and in the shades ofFIG. 16B are no longer visible inFIG. 16C and this effect is because preserving them is no longer weighted heavily when the CSF is accommodated. Accordingly, the exemplary tone-mapping process incorporating the CSF increased contrast at large distances, thus offering better contrast reproduction at the cost of clipping the brightest and the darkest regions of the image.
SYSTEM EMBODIMENTS
• An exemplary system embodiment of the present invention is illustrated inFIG. 17 where the system1700 includes a first display imageluminance processing module1710 having a processing unit, and addressable memory, the processing module is adapted to execute computer instructions to modify a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter; a second display imageluminance processing module1720 is adapted to execute computer instructions to modify a display image luminance based on a human luminance perception model as part of the human visual system model, a shaping orfiltering module1730 adapted to execute computer instructions to shape the spatial frequency response of the modified display image with a contrast sensitivity function and adapted to generate a display response image, acomparator module1740 adapted to execute computer instructions to generate the error value based on the generated displayvisual response image1732 and a referencevisual response image1752; and aparameter processing module1760 having a processing unit, and addressable memory adapted to modify at least one of: the brightness parameter, the sharpness parameter, the contrast parameter, and, optionally, a white light bias; wherein the parameter modification is based on the generated error value. One of ordinary skill in the art will appreciate that functionality may be distributed across or within different modules of various system embodiments and still fall within the contemplated scope of the invention. The exemplary system embodiment may also have the firstdisplay processing module1710 having a processing unit, and addressable memory wherein the firstdisplay processing module1710 is adapted to modify a display image chrominance based on a white light bias parameter. The firstdisplay processing module1710 of the exemplary system embodiment may be further adapted to limit, on a pixel-by-pixel basis, when a display image color radiance component exceeds a display-limited range, then limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value. The exemplary system embodiment may further include a reference visualimage processing module1750 having a processing unit, and addressable memory wherein the reference image processing module is adapted to modify a reference image luminance based on the human luminance perception model of the human visual system model and shaping or filtering the modified display image spatial frequency transform with a contrast sensitivity function and generate a reference visual response image.
APPARATUS EMBODIMENTS
• An exemplary apparatus embodiment of the present invention is illustrated inFIG. 18 where theapparatus1800 includes a processing module having aprocessing unit1810, andaddressable memory1820 wherein the processing module tests on whether a generated error value is less than a threshold value and wherein the processing module is adapted to execute computer instructions to: modify a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter; modify a display image luminance based on a human luminance perceptual model as part of a human visual system model; shape or filter the spatial frequency transform of the modified display image with the contrast sensitivity function and generate a display visual response image; compare the generated display visual response image with a reference visual response image to generate the error value; and modify at least one of: the brightness parameter, the sharpness parameter, the contrast parameter, and the white light bias; when the generated error value is above a threshold value. Theprocessing module1810 of the exemplary apparatus may be further adapted to execute computer instructions to modify a display image chrominance based on a white light bias parameter, and limit, on a pixel-by-pixel basis, when a display image color radiance component exceeds a display-limited range, then limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value. Theprocessing module1810 of the exemplary apparatus may be further adapted to execute computer instruction to modify a reference image luminance based on the human luminance perception model as a portion of the human visual system model and shape or filter the spatial frequency transform of the modified display image with a contrast sensitivity function and generate a reference visual response image.
• Although this invention has been disclosed in the context of certain embodiments and examples, it will be understood by those of ordinary skill in the art that the present invention extends beyond the specifically disclosed embodiments to other alternative embodiments and/or uses of the invention and obvious modifications and equivalents thereof. In addition, while a number of variations of the invention have been shown and described in detail, other modifications, which are within the scope of this invention, will be readily apparent to those of ordinary skill in the art based upon this disclosure. It is also contemplated that various combinations or subcombinations of the specific features and aspects of the embodiments may be made and still fall within the scope of the invention. The embodiments of the present invention may be embodied in a set of program instructions, e.g., software, hardware, or both—i.e., firmware. Accordingly, it should be understood that various features and aspects of the disclosed embodiments can be combined with or substituted for one another in order to form varying modes of the disclosed invention. Thus, it is intended that the scope of the present invention herein disclosed should not be limited by the particular disclosed embodiments described above.

Claims (23)

1. A method of image rendering comprising:

if an error value exceeds a threshold value, then iteratively performing the steps of:

modifying a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter;

transforming a display image luminance based on a human luminance perception model and generating a display visual response image;

comparing the display visual response image with a reference visual response image to generate the error value; and

modifying at least one of a set of tone-mapping parameters comprising: the brightness parameter, the sharpness parameter, and the contrast parameter.

2. The method of image rendering ofclaim 1 further comprising, after the step of modifying a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter, the step of modifying a display image chrominance based on a white light bias parameter, and, wherein the set of tone-mapping parameters further comprises the white light bias parameter.

3. The method of image rendering ofclaim 1 further comprising, after the step of modifying a display visual image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter, the step of: if, on a pixel-by-pixel basis, a display image color radiance component exceeds a display-limited range, then limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value.

4. The method of image rendering ofclaim 1 further comprising, after the step of modifying a display image luminance based on the human luminance perception model, the step of shaping a spatial frequency transform of the modified display image with a contrast sensitivity function and thereafter generating the display visual response image.

5. The method of image rendering ofclaim 1 further comprising, preceding the step of testing whether the error value exceeds the threshold value, the steps of modifying the reference image luminance based on a human luminance perception model and shaping a spatial frequency transform of the modified display image with a contrast sensitivity function and thereafter generating the reference visual response image.

6. The method of image rendering ofclaim 5 wherein the contrast sensitivity function is a function of viewing distance.

7. The method ofclaim 1 wherein the human luminance perception model comprises a just-noticeable-difference (JND) encoding function.

8. The method ofclaim 1 wherein the error value is a mean square error value.

9. The method ofclaim 1 wherein a second error value is generated for comparing with the threshold value via a determination of the maximal pixel difference between the display response image and the reference response image.

10. A method of image rendering comprising:

modifying a reference visual image luminance based on a human luminance perception model;

shaping a spatial frequency transform of the modified reference image with a contrast sensitivity function and generating a reference visual response image;

if an error value exceeds a threshold value, then iteratively performing the steps of:

modifying a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter;

if, on a pixel-by-pixel basis, a display image color radiance component exceeds a display-limited range, then limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value;

transforming a display image luminance based on the human luminance perception model;

shaping a spatial frequency transform of the modified display image with the contrast sensitivity function and generating a display visual response image;

comparing the display visual response image with the reference visual response image to generate an error value; and

modifying at least one of a set of tone-mapping parameters comprising: the brightness parameter, the sharpness parameter, and the contrast parameter.

11. The method of image rendering ofclaim 10 further comprising modifying a display image chrominance based on a white light bias parameter prior to limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value and, wherein the set of tone-mapping parameters further comprises the white light bias parameter.

12. The method of image rendering ofclaim 10 wherein the contrast sensitivity function is a function of viewing distance.

13. The method ofclaim 10 wherein the human luminance perception model comprises a just-noticeable-difference (JND) encoding function.

14. The method ofclaim 10 wherein the error value is a mean square error value.

15. The method ofclaim 10 wherein the error value is generated via a determination of the maximal pixel difference between the display visual response image and the reference visual response image.

16. A system for image rendering comprising:

a first display image luminance processing module having a processing unit, and addressable memory, the first display image luminance processing module adapted to modify a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter;

a second display image luminance module, executable via a processing unit having addressable memory, wherein the second display image luminance module is adapted to modify a display image luminance based on a human luminance perception model;

a shaping module, executable via a processing unit having addressable memory, wherein the filtering module is adapted to shape the spatial frequency response of the modified display image with a contrast sensitivity function and adapted to generate a display visual response image;

a comparator module, executable via a processing unit having addressable memory, wherein the comparator module is adapted to generate an error value based on the generated display visual response image and a reference visual response image; and

a parameter processing module, executable via a processing unit having addressable memory, wherein the parameter processing module is adapted to modify at least one of a set of tone-mapping parameters comprising: the brightness parameter, the sharpness parameter, and the contrast parameter; wherein the parameter modification is based on the generated error value.

17. The system ofclaim 16 further comprising a display processing module, executable via a processing unit having addressable memory, wherein the display processing module is adapted to modify a display image chrominance based on a white light bias parameter; and, wherein the set of tone-mapping parameters further comprises the white light bias parameter.

18. The system ofclaim 17 wherein the display processing module is further adapted to limit, on a pixel-by-pixel basis, when a display image color radiance component exceeds a display-limited range, then limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value.

19. The system ofclaim 16 further comprising a reference image processing module, executable via a processing unit having addressable memory, wherein the reference image processing module is adapted to modify a reference image luminance based on a human luminance perception model and shaping the spatial frequency transform of the modified reference image with a contrast sensitivity function and generate a reference response image.

20. The system ofclaim 16 wherein the first display image luminance processing module is further adapted to modify a reference image luminance based on a human luminance perception model and shaping the spatial frequency transform of the modified reference image with a contrast sensitivity function and generate a reference response image.

21. An apparatus for image rendering comprising:

a processing module having a processing unit, and addressable memory wherein the processing module is adapted to execute the following steps while an error value is above a threshold value:

modify a display image luminance based on an input luminance and a brightness parameter, a sharpness parameter, and a contrast parameter;

modify a display image luminance based on a human luminance perception model;

shape the modified display image spatial frequency transform with a contrast sensitivity function and generate a display visual response image;

compare the generated display visual response image with a reference visual response image to generate the error value; and

modify at least one of a set of tone-mapping parameters comprising: the brightness parameter, the sharpness parameter, the contrast parameter.

22. The apparatus for image rendering ofclaim 21 wherein the processing module is further adapted to modify a display image chrominance based on a white light bias parameter, and limit, on a pixel-by-pixel basis, when a display image color radiance component exceeds a display-limited range, then limiting, on a pixel-by-pixel basis, the display image color radiance component to the exceeded display-limited range value and, wherein the set of tone-mapping parameters further comprises the white light bias parameter.

23. The apparatus for image rendering ofclaim 22 wherein the processing module is further adapted to modify a reference image luminance based on a human luminance perception model and shape a spatial frequency transform of the modified reference image with a contrast sensitivity function and thereafter generate the reference visual response image.

Images