US9489043B2 - Display device and controlling method - Google Patents

Abstract

An image display device and controlling method capable of optimizing a state of the image display device for a user at a desired position. The display device includes: an imaging section that takes an image of a predetermined range of a dynamic image with respect to an image display direction; an image analyzing section that analyzes the dynamic image taken by the imaging section and calculates a position of a user; a system optimization processing section that calculates system control information for optimizing a system based on the position of the user calculated by the image analyzing section; and a system controlling section that optimizes the system based on the system control information calculated by the system optimization processing section.

Description

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No. 13/395,035, filed Jun. 21, 2012, which is a National Stage of PCT/JP2010/062311, filed Jul. 22, 2010, which claims the benefit of priority to Japanese Patent Application No. 2009-213377, filed Sep. 15, 2009, the contents of which is incorporated in its entirety.

TECHNICAL FIELD

The present invention relates to a display device and a controlling method.

BACKGROUND ART

In the recent years, accompanying an expansion of a flat-screen television market, demands for an image display device such as a large screen television for installing in for example a living room is increasing. In such a situation, an image display device having various functions is being proposed.

SUMMARY OF INVENTIONTechnical Problem

Incidentally, since a user can watch an image that an image display device is displaying at any desired position, depending on a viewing position of the user, a state of the image display device, namely an audio property such as a volume balance of sound output from an audio output section, an image property of an image display section of the image display device and display contents, a displaying direction of the image display device and the like may not be optimal for the user at the desired position.

Thus, the present invention has been made in view of the above problem, and an aim of the present invention is to provide a novel and improved image display device and controlling method capable of optimizing the state of the image display device for the user at the desired position.

Solution to Problem

To solve the above problem, according to an aspect of the present invention, a display device including an imaging section that takes an image of a predetermined range of a dynamic image with respect to an image display direction; an image analyzing section that analyzes the dynamic image taken by the imaging section and calculates a position of a user; a system optimization processing section that calculates system control information for optimizing a system based on the position of the user calculated by the image analyzing section; and a system controlling section that optimizes the system based on the system control information calculated by the system optimization processing section is provided.

The system optimization processing section may calculate system control information for optimizing a volume balance of sound output from an audio output section based on the position of the user calculated by the image analyzing section.

The system optimization processing section may calculate system control information for optimizing an image property of an image display section based on the position of the user calculated by the image analyzing section.

The system optimization processing section may calculate system control information for optimizing display contents of an image display section based on the position of the user calculated by the image analyzing section.

The system optimization processing section may calculate system control information for optimizing a device direction of the display device itself based on the position of the user calculated by the image analyzing section.

The image analyzing section may analyze the dynamic image taken by the imaging section and calculate a three-dimensional position of the user.

The image analyzing section may analyze the dynamic image taken by the imaging section and calculate respective positions of a plurality of users, and the system optimization processing section may calculate positions of center of balance of the plurality of users based on the positions of the plurality of users calculated by the image analyzing section, and calculate system control information for optimizing a system based on the calculated positions of center of balance of the plurality of users.

Further, to solve the above problem, according to another aspect of the present invention, a controlling method including an imaging step of taking an image of a predetermined range of a dynamic image with respect to an image display direction; an image analyzing step of analyzing the taken dynamic image and calculating a position of a user; a system optimization processing step of calculating system control information for optimizing a system based on the calculated position of the user; and a system controlling step of optimizing the system based on the calculated system control information is provided.

Advantageous Effects of Invention

As explained above, according to the present invention, a novel and improved image display device and controlling method capable of optimizing the state of the image display device for the user at the desired position can be provided.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram that explains an outer appearance of animage display device100 in an embodiment of the present invention.

FIG. 2 is an explanatory diagram that explains a configuration of theimage display device100 of the embodiment of the present invention.

FIG. 3 is an explanatory diagram that explains a configuration of acontrol section110.

FIG. 4 is an explanatory diagram that explains the configuration of thecontrol section110.

FIG. 5(A) is an explanatory diagram for explaining a case where auser 1 and a user 2 are present in an imaging area of animaging section104, andFIG. 5(B) is an explanatory diagram for explaining a face detecting position [a1, b1] and a face size [w1, h1] of theuser 1, and a face detecting position [a2, b2] and a face size [w2, h2] of the user 2 that are included in an image taken by theimaging section104.

FIG. 6(A) is an explanatory diagram for explaining a case where users are present at a reference distance d0 and a distance d1 in the imaging area of theimaging section104,FIG. 6(B) is an explanatory diagram for explaining the face size [w1, h1] of the user at the distance d1 in the image taken by theimaging section104, andFIG. 6(C) is an explanatory diagram for explaining a reference face size [w0, h0] of the user at the reference distance d0 in the image taken by theimaging section104.

FIG. 7(A) is an explanatory diagram for explaining a device center [0,0,0] of theimage display device100 and a camera position [Δx, Δy, Δz] of theimaging section104, andFIG. 7(B) is an explanatory diagram for explaining the device center [0,0,0] of theimage display device100, a frontward axis [0,0], the camera position [Δx, Δy, Δz] of theimaging section104, and an installation angle [Δφ, Δθ].

FIG. 8 is a flow diagram showing an example of an optimization process in accordance with a position of the user by theimage display device100 of the embodiment of the present invention.

FIG. 9 is a flow diagram showing an example of an optimization process in accordance with positions of one or more users by theimage display device100 of the embodiment of the present invention.

FIG. 10 is a flow diagram showing an example of an optimization process in accordance with ages of one or more users by theimage display device100 of the embodiment of the present invention.

FIG. 11 is an explanatory diagram for explaining an optimization process of a volume balance.

FIG. 12 is an explanatory diagram for explaining a calculation method of positions of center of balance of a user group.

FIG. 13(A) is an explanatory diagram for explaining a position [D0, Ah0] of a user A and a position [D1, Ah1] of a user B in a horizontal direction, andFIG. 13(B) is an explanatory diagram for explaining a position [D0, AV0] of the user A and a position [D, Av1] of the user B in a vertical direction.

FIG. 14(A) andFIG. 14(B) are explanatory diagrams for explaining an optimization process of a letter size of a GUI.

FIG. 15(A) toFIG. 15(C) are explanatory diagrams for explaining a correction method of the reference face size [w0, h0] at the reference distance d0 in a calculation of the user distance.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the appended drawings. Note that, in this specification and the drawings, elements that have substantially the same function and structure are denoted with the same reference signs, and repeated explanation is omitted.

Note that, explanation will be given in the following order.

<1. One Embodiment of the Present Invention>

[1-1. Configuration of Image Display Device]

[1-2. Configuration of Control Section]

[1-3. Optimization Process according to User Position]

[1-4. Optimization Process according to User Positions of One or More Users]

[1-5. Optimization Process according to Ages of One or More Users]

1. One Embodiment of the Present Invention

[1-1. Configuration of Image Display Device]

Firstly, a configuration of an image display device of an embodiment of the present invention will be explained.FIG. 1 is an explanatory diagram that explains an outer appearance of animage display device100 in the embodiment of the present invention.FIG. 1 is a front view diagram in viewing theimage display device100 from a front side. Hereinbelow, the outer appearance of theimage display device100 of the embodiment of the present invention will be explained with reference toFIG. 1.

As shown inFIG. 1, theimage display device100 of the embodiment of the present invention includesimaging sections104 that take a dynamic image arranged at an upper center position and left and right center positions of adisplay panel section102 that displays a still image or a dynamic image. Theimaging sections104 take the dynamic image with respect to a direction by which theimage display device100 displays the still image or the dynamic image in thedisplay panel section102. Theimage display device100 of the embodiment analyzes an image taken by theimaging sections104, and detects a face of a user taken in the image. Theimage display device100 analyzes the detected face of the user, and detects a face detecting position and a face size. Theimage display device100 calculates a relative position of the user with respect to an optical axis of a camera of therespective imaging sections104 based on the detected face detecting position and face size. Then, theimage display device100 calculates a position of the user with respect to a device center and a frontward axis of theimage display device100 based on a calculation result of the relative position of the user with respect to the optical axis of the camera of therespective imaging sections104 and appended information such as a position, an angle of the camera of therespective imaging sections104. Theimage display device100 of the embodiment characteristically optimizes a state of theimage display device100 in accordance with the position of the user; that is, an audio property such as sound volume, an image property, display contents, a device direction with respect to the position of the user.

Further, theimage display device100 of the embodiment includes asensor section106 arranged at a lower center position of thedisplay panel section102. Thesensor section106 detects presence/absence of a person in front of theimage display device100.

Note that, inFIG. 1, although theimage display device100 had theimaging sections104 that take the dynamic image at three portions around thedisplay panel section102, it is needless to say that positions of theimaging sections104 that take the dynamic image are not limited to the above example in the present invention; for example, a device independent from theimage display device100 may be provided, the device may be connected to theimage display device100, and the dynamic image may be taken by this device. Further, it is needless to say that a number of theimaging section104 is not limited to three; images may be taken by providing two or less, or four ormore imaging sections104. Moreover, it is needless to say that a number of thesensor section106 is not limited to one; and two or more sensor sections may be provided.

Further, although not shown inFIG. 1, theimage display device100 may further include a signal receiving section that can receive a control signal from a remote controller (not shown) through an infrared scheme, wireless scheme.

As above, the outer appearance of theimage display device100 of the embodiment of the present invention has been explained with reference toFIG. 1. Next, a configuration of theimage display device100 of the embodiment of the present invention will be explained.

FIG. 2 is an explanatory diagram that explains the configuration of theimage display device100 of the embodiment of the present invention. Hereinbelow, the configuration of theimage display device100 of the embodiment of the present invention will be explained with reference toFIG. 2.

As shown inFIG. 2, theimage display device100 of the embodiment of the present invention is configured by including thedisplay panel section102, theimaging sections104, thesensor section106, aspeaker section108, amechanism section109, and acontrol section110.

Further, thecontrol section110 is configured by including animage input section112, animage processing section114, a viewingstate analyzing section116, a user positioninformation storing section117, a viewingstate recording section118, a systemoptimization processing section120, and asystem control section122.

Thedisplay panel section102 displays the still image or the dynamic image based on a panel drive signal. In the embodiment, thedisplay panel section102 displays the still image or the dynamic image using liquid crystals. Of course in the invention, it is needless to say that thedisplay panel section102 is not limited to the above example. Thedisplay panel section102 may display the still image or the dynamic image using a self-emitting display device such as an organic EL (electroluminescence).

As described above, theimaging sections104 are arranged at the upper center position and the left and right center positions of thedisplay panel section102 that displays the still image or the dynamic image. Theimaging sections104 take the dynamic image with respect to the direction by which theimage display device100 displays the dynamic image in thedisplay panel section102 when the panel drive signal is supplied to thedisplay panel section102 and thedisplay panel section102 is displaying the dynamic image. Theimaging sections104 may take the dynamic image by CCD (Charge Coupled Device) image sensors, or may take the dynamic image by CMOS (Complementary Metal Oxide Semiconductor) image sensors. The dynamic image taken by theimaging sections104 is sent to thecontrol section110.

Thesensor section106 is provided at the lower center position of thedisplay panel section102 that displays the still image or the dynamic image, and is for example for detecting the presence/absence of a person in front of theimage display device100. Further, if a person is present in front of theimage display device100, thesensor section106 can detect a distance between theimage display device100 and that person. A detection result and distance information by thesensor section106 are sent to thecontrol section110. Thespeaker section108 outputs sounds based on a sound outputting signal. Themechanism section109 controls a displaying direction of thedisplay panel102 of theimage display device100 for example based on a drive signal.

Thecontrol section110 controls operations of theimage display device100. Hereinbelow, respective sections of thecontrol section110 will be explained.

Theimage input section112 receives the dynamic image taken in theimaging sections104. The dynamic image received by theimage input section112 is sent to theimage processing section114, and is used in image processes in theimage processing section114.

Theimage processing section114 is an example of an image analyzing section of the present invention, and performs the respective image processes on the dynamic image that is taken by theimaging sections104 and sent from theimage input section112. The image processes performed by theimage processing section114 includes a detection process for an object included in the dynamic image taken by theimaging sections104, a detection process for a number of persons included in the dynamic image, and a detection process for a face and a facial expression included in the dynamic image. Results of the respective image processes by theimage processing section114 are sent to the viewingstate analyzing section116, and are used for analyses of presence/absence of a person watching theimage display device100, and a viewing state and a viewing position of the person.

As for the face detection process to detect the face included in the image by theimage processing section114, techniques described for example in Japanese Patent Application Publication No. 2007-65766 and Japanese Patent Application Publication No. 2005-44330 can be used. Hereinbelow, the face detection process will briefly be explained.

In order to detect the face of the user from the image, firstly, a face position, a face size, and a face direction in the supplied image are respectively detected. By detecting the position and size of the face, a face image portion can be cut out from the image. Then, based on the cut-out face image and information of the face direction, feature portions of the face (facial feature positions), for example, eyebrows, eyes, nose, mouth, and the like, are detected. In the detection of the facial feature positions, for example, a method called an AAM (Active Appearance Models) may be adapted so as to be capable of detecting the feature positions.

When the facial feature positions are detected, a local feature value is calculated for each of the detected facial feature positions. By calculating the local feature values and storing the calculated local feature values together with the face image, face recognition becomes possible from the image taken by theimaging section104. As for the method of face recognition, since techniques described for example in Japanese Patent Application Publication No. 2007-65766 and Japanese Patent Application Publication No. 2005-44330 can be used, thus a detailed description thereof will herein be omitted. Further, it is also possible to determine whether the face taken in the supplied image is a male or a female, and how old the person is, by the face image and the face feature position. Further, by predeterminedly recording information of faces, the person taken in the supplied image can be searched among the faces, and an individual can be specified.

The viewingstate analyzing section116 is an example of an image analyzing section of the present invention. It receives a result of the respective image processes by theimage processing section114 and a detection result and distance information by the detection of thesensor section106, and performs an analysis of the viewing state and the viewing position of the person watching the image displayed by theimage display device100 by using the result of the respective image processes by theimage processing section114 and the detection result and the distance information by the detection of thesensor section106. By the viewingstate analyzing section116 analyzing the viewing state and the viewing position of the person watching, theimage display device100 can control the audio property such as a sound balance of an output from thespeaker section108, control the image property of thedisplay panel section102, control the display contents of thedisplay panel section102, and control the display direction of thedisplay panel section102 by using themechanism section109 in accordance with the viewing position of the user watching theimage display device100. An analysis result of an analysis process by the viewingstate analyzing section116 is sent to the viewingstate recording section118, the user positioninformation storing section117 and the systemoptimization process section120.

Note that, in a case where although a dynamic body is detected from the detection result and the distance information by the detection of thesensor section106 but a distance between thesensor section106 and that dynamic body is at or exceeding a predetermined distance, the viewingstate analyzing section116 can exclude that object from being a detection target.

The viewingstate recording section118 records the analysis result obtained by the analysis process by the viewingstate analyzing section116. The analysis result in the viewingstate analyzing section116 recorded by the viewingstate recording section118 is used in the system optimization process by the systemoptimization processing section120. Further, the analysis result in the viewingstate analyzing section116 recorded by the viewingstate recording section118 may be sent to an externaldata collection server200.

The user positioninformation storing section117 stores the analysis result of an analysis process by the viewingstate analyzing section116.

The systemoptimization processing section120 calculates the system control information for performing the system optimization process on the respective sections of theimage display device100 by using the analysis result obtained by the analysis process of the viewingstate analyzing section116. The system optimization process on the respective sections of theimage display device100 includes the control of the audio property such as the sound balance of the output from thespeaker section108, the control of the image property of thedisplay panel section102, the control of the display contents of thedisplay panel section102, and the control of the display direction of thedisplay panel section102 of theimage display device100 by themechanism section109.

Theimage display device100 can perform the optimization process in accordance with the position of the user based on the system control information calculated by the systemoptimization processing section120. The system control information calculated by the systemoptimization processing section120 is sent to thesystem control section122.

Thesystem control section122 performs the system optimization process on the respective sections of theimage display device100 based on the system control information calculated by the systemoptimization processing section120. For example, thesystem control section122 performs the control of the volume balance of the output from thespeaker section108, the control of the image property of thedisplay panel section102, the control of the display contents of thedisplay panel section102, the control of the display direction of thedisplay panel section102 of theimage display device100 by themechanism section109 and the like based on the system control information calculated by the systemoptimization processing section120.

As above, the configuration of theimage display device100 of the embodiment of the present invention has been explained with reference toFIG. 2. Next, a configuration of thecontrol section110 included in theimage display device100 of the embodiment of the present invention will be explained in more detail.

[1-2. Configuration of Control Section]

FIG. 3 is an explanatory diagram that explains the configuration of thecontrol section110 included in theimage display device100 of the embodiment of the present invention. Within thecontrol section110,FIG. 3 explains a configuration of the viewingstate analyzing section116 included in thecontrol section110. Hereinbelow, the configuration of the viewingstate analyzing section116 will be explained with reference toFIG. 3.

As shown inFIG. 3, the viewingstate analyzing section116 is configured by including a user direction/distance calculating section132 and a user positioninformation calculating section134.

The user direction/distance calculating section132 receives the results of the respective image processes by theimage processing section114 and optical information such as information of an angle of view and a resolution of each camera of theimaging sections104, and calculates a relative position of the user (direction [φ1, ƒ1], distance d1) with respect to the optical axis of each camera of theimaging sections104 by using the results of the respective image processes by theimage processing section114 and the optical information from theimaging sections104. Here, the taken image, as well as face detection information (for example, information such as the face detecting position [a1, b1], the face size [w1, h1], and other attribute information such as the age and sex) for each user using theimage display device100 are sent from theimage processing section114 to the user direction/distance calculating section132 of the viewingstate analyzing section116.FIG. 5(A) is an explanatory diagram for explaining a case where auser 1 and a user 2 are present in the imaging area of theimaging section104, andFIG. 5(B) is an explanatory diagram for explaining the face detecting position [a1, b1] and the face size [w1, h1] of theuser 1, and the face detecting position [a2, b2] and the face size [w2, h2] of the user 2 that are included in the image taken by theimaging section104. Further,FIG. 6(A) is an explanatory diagram for explaining a case where users are present at a reference distance d0 and a distance d1 in the imaging area of one of theimaging sections104,FIG. 6(B) is an explanatory diagram for explaining the face size [w1, h1] of the user at the distance d1 in the image taken by theimaging section104, andFIG. 6(C) is an explanatory diagram for explaining a reference face size [w0, h0] of the user at the reference distance d0 in the image taken by theimaging section104.

The direction [φ1, θ1] is calculated as follows from the face detecting position [a1, b1] normalized by a taken image size [xmax, ymax] and the angle of view [φ0, θ0] of the camera of theimaging section104.
Horizontal direction: φ1=φ0*a1
Vertical direction: θ1=θ0*b1

Further, the distance d1 is calculated as follows from the reference face size [w0, h0] at the reference distance d0.
Distance:d1=d0*(w0/w1)

The user positioninformation calculating section134 receives a calculation result of the relative position of the user with respect to the optical axis of the camera of therespective imaging sections104 by the user direction/distance calculating section132 and the appended information such as the position, the angle of the camera of therespective imaging sections104, and calculates a three-dimensional position of the user with respect to the device center of theimage display device100 and the frontward axis by using the calculation result of the relative position of the user by the user direction/distance calculating section132 and the appended information of therespective imaging sections104. The user position information calculated by the user positioninformation calculating section134 is sent to the user positioninformation storing section117.FIG. 7(A) is an explanatory diagram for explaining the device center [0,0,0] of theimage display device100 and a camera position [Δx, Δy, Δz] of theimaging section104, andFIG. 7(B) is an explanatory diagram for explaining the device center [0,0,0] of theimage display device100, the frontward axis [0,0], the camera position [Δx, Δy, Δz] of theimaging section104, and an installation angle [Δφ, Δθ].

When the relative position of the user with respect to the optical axis of the camera of theimaging section104 is of the direction [φ1, θ1] and distance d1, the device center of theimage display device100 is [0,0,0], and, with respect to the frontward axis [0,0], a positional difference [Δx, Δy, Δz] is the camera position of theimaging section104, an angular difference [Δφ, Δθ] is the installation angle, a position [x1, y1, z1] of the user with respect to the device center [0,0,0] of theimage display device100 is calculated as follows.
x1=d1*cos(θ1−Δθ)*tan(φ1−Δφ)−Δx
y1=d1*tan(θ1−Δθ)−Δy
z1=d1*cos(θ1−Δθ)*cos(φ1−Δθ)−Δz

FIG. 4 is an explanatory diagram that explains the configuration of thecontrol section110 included in theimage display device100 of the embodiment of the present invention. Within thecontrol section110,FIG. 4 explains configurations of the user positioninformation storing section117, the systemoptimization processing section120 and thesystem control section122 included in thecontrol section110. Hereinbelow, the configurations of the user positioninformation storing section117, the systemoptimization processing section120 and thesystem control section122 will be explained with reference toFIG. 3.

As shown inFIG. 4, the systemoptimization processing section120 is configured by including an audio propertyoptimization processing section142, an image propertyoptimization processing section144, and a device directionoptimization processing section146. Further, thesystem control section122 is configured by including an audioproperty control section152, an imageproperty control section154, and a devicedirection control section156.

The user positioninformation storing section117 stores user position information that is the calculation result of the position of the user with respect to the device center and the frontward axis of theimage display device100 by the user positioninformation calculating section134 of the viewingstate analyzing section116. The user position information stored in the user positioninformation storing section117 is sent to the systemoptimization processing section120.

The audio propertyoptimization processing section142 of the systemoptimization processing section120 calculates audio property control information for performing an audio property optimization process on thespeaker section108 of theimage display device100 based on the user position information sent from the user positioninformation storing section117, so as to optimize the audio property of theimage display device100 for the user at the desired position. The audio property control information calculated by the audio propertyoptimization processing section142 is sent to the audioproperty control section152 of thesystem control section122.

As the audio property optimization process, there are a process for optimizing the volume balance of stereo sound output from thespeaker section108 on the left and right sides, and an optimization process related to a surround effect of the stereo sound output from thespeaker section108. Since a difference is generated in the volume balance of the left and right sides of the stereo sound output from thespeaker section108 depending on the position of the user, the optimization process optimizes gains on the left and right sides. For example, as shown inFIG. 11, a difference from a reference point is calculated as follows according to a principle that the volume of the stereo sound output from thespeaker section108 attenuates in inverse proportion to a square of distance; the difference (gain_dif) in the volume in thespeaker section108 at the left and right sides is calculated, and the volume balance on the left and right sides can be optimized.
gain_Lch=20*log(d_Lch/d_org_LRch)
gain_Rch=20*log(d_Rch/d_org_LRch)
gain_dif=gain_Rch−gain_Lch=20*log(d_Rch)−20*log(d_Lch)=20*log(d_Lch/d_Rch)
Here, note that: gain_Lch
: the gain difference of Lch
gain_Rch
: the gain difference of Rch
d_org_LRch
: the distance from the left and right speaker sections to the reference point
d_Lch
: the distance from the Lch speaker section to the user
d_Rch
: the distance from the Rch speaker section to the user
gain_dif
: voltage gain difference of L/R

Further, if a plurality of users is present, the left and right volume balance of the sound output from thespeaker section108 may be optimized with respect to a center of balance of that user group, or alternatively may optimize the left and right volume balance of the sound output from thespeaker section108 with priority for a particular user. As a method of calculating the center of balance of the user group, when a plurality of users, namely users A, B, C, and D as shown inFIG. 12 is present, the calculation can be performed as follows.

	d_cent_dif
	= 0;
	temp_CAM_DIS
	= 0;
	if(CAM_AUDIENCE
	!= 0){
	for(int i=0;i <
	CAM_AUDIENCE;i++){
	d_cent_dif +=
	CAM_DIS[i]* tan(CAM_HOR_ANGLE[i]* PI/180);
	temp_CAM_DIS +=
	CAM_DIS[i];
	}
	}
	d_cent_dif
	= d_cent_dif/CAM_AUDIENCE;
	// return (use) value center of balance processed angle
	CAM_DIS
	= temp_CAM_DIS/CAM_AUDIENCE;
	// return (use) value center of balance processed distance

Here, note that CAM AUDIENCE
: User within the imaging area of theimaging sections104
CAM_HOR_ANGLE[0]
: Angle of user A
CAM_HOR_ANGLE[1]
: Angle of user B
CAM_HOR_ANGLE[2]
: Angle of user C
CAM_HOR_ANGLE[3]
: Angle of user D
CAM_DIS[0]
: distance of user A
CAM_DIS[1]
: distance of user B
CAM_DIS[2]
: distance of user C
CAM_DIS[3]
: distance of user D

The image propertyoptimization processing section144 of the systemoptimization processing section120 calculates image property control information with respect to the user at the desired position for performing the image property optimization process on thedisplay panel102 of theimage display device100 based on the user position information sent from the user positioninformation storing section117 that is for optimizing the image property of theimage display device100. The image property control information calculated by the image propertyoptimization processing section144 is sent to the imageproperty control section154 of thesystem control section122.

As the image property optimization process, there are processes such as a gamma correction for optimizing an appearance of black, and a correction of color gain for complying with a color change.

For example, the gamma correction is performed as follows.
γ=2.2+image quality correction−0.1×user direction

Further, for example, the correction of the color gain is performed as follows.
ColorGain=User Color+image quality correction±α×user direction
R(G,B) Gain=R(G,B)×image quality correction±α×user direction

Further, if the plurality of users is present, the gamma correction and the correction of the color gain can be performed on the center of balance of the user group, or alternately the gamma correction and the correction of the color gain can be performed with priority for a particular user. In assuming that a position of the user A is [D0, Ah0] and a position of the user B is [D1, Ah1] in the horizontal direction as shown inFIG. 13(A), and a position of the user A is [D0, AV0] and a position of the user B is [D, Av1] in the vertical direction as shown inFIG. 13(B), an average viewing angle correction coefficient and setting data for the system optimization process are calculated by the following formula.
Average viewing angle correction coefficient={(1/D0*(Ah0+Av0)+1/D1*(Ah1+Av1)+1/Dn*(Ahn+Avn))/n}*correction value
Setting data=(basic data)*{1+(correction value at the maximum viewing angle)*(average viewing angle correction coefficient)}

An average viewing angle gives weight to close-by users, and calculates an average of a horizontal angle and a vertical angle for a number of the users. A correction coefficient is calculated by multiplying the correction value to the average viewing angle, and an overall correction amount is calculated by multiplying the correction coefficient to the maximum correction value. The setting data is calculated by adding or subtracting the correction amount to or from the basic data (data with no correction: γ=2.2+image quality correction).

Further, in the optimization of the image property, since the optimal value varies depending on the age and sex of the user, the optimization process can be performed by using the attribute information such as the age and sex obtained from the image processing section in addition to the user position information. For example, a luminance correction of thedisplay panel102 is performed as follows.
BackLight=basic setting value*correction value
Correction value=10^(A*log screen illumination+B*log viewing angle+C*log image average level+D*age)/screen illumination

Further, in an experiment, an optimal luminance is known to have a relationship with the screen illumination, viewing angle, image average luminance level, and age as follows.
Log optimal luminance=A*log screen illumination+B*log viewing angle+C*log image average level+D*age

The device directionoptimization processing section146 of the systemoptimization processing section120 calculates device direction control information for performing a device direction optimization process to themechanism section109 of theimage display device100 with respect to the user at the desired position based on the user position information sent from the user positioninformation storing section117, for optimizing the device direction of theimage display device100. The device direction control information calculated by the device directionoptimization processing section146 is sent to the devicedirection control section156 of thesystem control section122.

As the device direction controlling process, if themechanism section109 is equipped with a swivel mechanism in the vertical and horizontal direction, theimage display device100 is rotated so that the frontward axis [0,0] of theimage display device100 comes to be at the direction [φ1, θ1] of the user. Due to this, thedisplay panel102 of theimage display device100 can be optimized to a direction that is head-on as seen from the user.

The audioproperty control section152 of thesystem control section122 performs the audio property optimization process based on the audio property control information sent from the audio propertyoptimization processing section142. For example, the audioproperty control section152 performs the control of the volume balance of the sound output from thespeaker section108 based on the audio property control information sent from the audio propertyoptimization processing section142.

The imageproperty control section154 of thesystem control section122 performs the image property optimization process based on the image property control information sent from the image propertyoptimization processing section144. For example, the imageproperty control section154 performs the control of the imaging property of thedisplay panel section102 based on the image property control information sent from the image propertyoptimization processing section144.

The devicedirection control section156 of thesystem control section122 performs the device direction optimization process based on the device direction control information sent from the device directionoptimization processing section146. For example, the devicedirection control section156 performs the control of themechanism section109 of theimage display device100 based on the device direction control information sent from the device directionoptimization processing section146.

As above, the configuration of thecontrol section110 included in theimage display device100 of the embodiment of the present invention has been explained with reference toFIG. 3 andFIG. 4. Next, the optimization process in accordance with the position of the user by theimage display device100 of the embodiment of the present invention will be explained.

[1-3. Optimization Process according to User Position]

FIG. 8 is a flow diagram showing an example of the optimization process in accordance with the position of the user by theimage display device100 of the embodiment of the present invention. Hereinbelow, the optimization process in accordance with the position of the user by theimage display device100 of the embodiment of the present invention will be explained with reference toFIG. 8.

InFIG. 8, firstly, when theimaging sections104 of theimage display device100 starts taking an image, theimage input section112 of thecontrol unit110 receives the image taken by the imaging sections104 (step S802).

Then, theimage processing section114 of thecontrol section110 performs the process of detecting the face included in the image received by the image input section112 (step S804).

Then, the viewingstate analyzing section116 of thecontrol section110 calculates the relative position of the user with respect to the optical axis of the camera of therespective imaging sections104 in the user direction/distance calculating section and calculates the user position with respect to the device center and the frontward axis of theimage display device100 in the user position information calculating section134 (step S806).

Then, the systemoptimization processing section120 of thecontrol section110 calculates system control information for performing the system optimization process for optimizing a state of theimage display device100 with respect to the user at the desired position based on the user position information calculated in step S806 (step S808). For example, in step S808, the system control information for performing the optimization of the volume balance of the sound output from thespeaker section108 on the left and right sides is calculated. Further, in step S808, the system control information for performing the process such as the gamma correction for optimizing the appearance of black and the correction of the color gain for complying with the color change is calculated. Further, in step S808, the system control information for performing the process for optimizing the device direction of theimage display device100 is calculated.

Then, thesystem control section122 of thecontrol section110 performs the system optimization process based on the system control information calculated in step S808 (step S810), and ends this process.

By the optimization process according to the position of the user inFIG. 8, the state of theimage display device100 with respect to the user at the desired position can be optimized. For example, the left and right volume balance of the sound output from thespeaker section108 is optimized, so the user can watch theimage display device100 without feeling uncomfortable. Further, the appearance of black and the color change are optimized, so the user can satisfactorily watch the image displayed on theimage display device100. Further, theimage display device100 is optimized to the direction that is head-on to the user, so the user can satisfactorily watch the image displayed on theimage display device100.

[1-4. Optimization Process according to User Positions of One or More Users]

Next, an optimization process in accordance with positions of one or more users by theimage display device100 of the embodiment of the present invention will be explained.FIG. 9 is a flow diagram showing an example of an optimization process in accordance with the positions of the one or more users by theimage display device100 of the embodiment of the present invention.

InFIG. 9, firstly, when theimaging sections104 of theimage display device100 starts taking an image, theimage input section112 of thecontrol unit110 receives the image taken by theimaging section104, and theimage processing section114 of thecontrol section110 performs the detection process of the face included in the image received by theimage input section112 and the like (step S902).

Then, the viewingstate analyzing section116 of thecontrol section110 receives a result of the face detection process by theimage processing section114, and determines whether a detected number of the user is one or more by using the result of the face detection by the image processing section114 (step S904).

As a result of the determination of step S904, if the detected number of the user is one, the viewingstate analyzing section116 of thecontrol section110 calculates a horizontal angle and a vertical angle of the user (step S906).

Then, the systemoptimization processing section120 of thecontrol section110 calculates a correction coefficient for the system control information for performing the system optimization process based on the horizontal angle and the vertical angle of the user calculated in step S906 (step S908).

As a result of the determination of step S904, if the detected number of the user is a plurality, the viewingstate analyzing section116 of thecontrol section110 determines whether the plurality of users is at a center of the image or not (step S910).

As a result of the determination of step S910, if the plurality of users is not at the center of the image (NO at step S910), the viewingstate analyzing section116 of thecontrol section110 calculates the horizontal angle, the vertical angle and a distance for respective ones of the plurality of users, and averages them (step S912). Further, in step S912, the horizontal angle, the vertical angle, and a distance may be calculated for respective ones of the plurality of users, and positions of center of balance for the plurality of users may be calculated.

Then, the systemoptimization processing section120 of thecontrol section110 calculates the correction coefficient for the system control information for performing the system optimization process (step S914).

As a result of the determination of step S910, if the plurality of users is at the center of the image (YES at step S910), the systemoptimization processing section120 of thecontrol section110 calculates the correction coefficient without the correction coefficient for the system control information for the system optimization process, or by changing a weight thereon (step S916).

After having calculated the correction coefficients in steps S908, S914, and S916, the systemoptimization processing section120 of thecontrol section110 calculates the system control information by adding the correction coefficients to basic data of the system control information for the system optimization process (step S918), and ends the process.

By the optimization process according to the positions of the one or more users inFIG. 9, the state of theimage display device100 with respect to the plurality of users at the desired positions can be optimized.

[1-5. Optimization Process according to Ages of One or More Users]

Next, an optimization process in accordance with ages of the one or more users by theimage display device100 of the embodiment of the present invention will be explained.FIG. 10 is a flow diagram showing an example of the optimization process in accordance with the ages of the one or more users by theimage display device100 of the embodiment of the present invention.

InFIG. 10, firstly, when theimaging sections104 of theimage display device100 starts taking an image, theimage input section112 of thecontrol unit110 receives the image taken by theimaging section104, and theimage processing section114 of thecontrol section110 performs the detection process of the faces included in the image received by theimage input section112 and the like (step S1002).

Then, the viewingstate analyzing section116 of thecontrol section110 receives a result of the face detection process by theimage processing section114, and determines whether a detected number of the user is one or more by using the result of the face detection by the image processing section114 (step S1004).

As a result of the determination of step S1004, if the detected number of the user is one, the viewingstate analyzing section116 of thecontrol section110 analyzes the age of the user (step S1006).

Then, the systemoptimization processing section120 of thecontrol section110 calculates a correction coefficient for the system control information for the system optimization process based on an analysis result of the age in step S1006 (step S1008).

As a result of the determination of step S1004, if the detected number of the user is a plurality, the viewingstate analyzing section116 of thecontrol section110 analyzes the ages of the respective users (step S1010).

Then, the viewingstate analyzing section116 of thecontrol section110 calculates a correction coefficient without the correction of the system control information for the system optimization process, or calculates correction coefficients independently, and averages the same (step S1012).

After steps S1008, S1012, the systemoptimization processing section120 of thecontrol section110 calculates the system control information by adding the correction coefficient to the basic data of the system control information for the system optimization process (step S1014), and ends the process.

By the optimization process according to the ages of the one or more users inFIG. 10, the state of theimage display device100 with respect to the plurality of users of various ages can be optimized.

Further, in the embodiment, the systemoptimization processing section120 may calculate system control information for optimizing a letter size of a GUI displayed on thedisplay panel section102 based on the user position information. The system control information calculated by the systemoptimization processing section120 is sent to thesystem control section122, and the optimization process for optimizing the letter size of the GUI displayed on thedisplay panel section102 is performed by thesystem control section122.

As the optimization process of the letter size of the GUI, as shown inFIG. 14(A), there is a process to enlarge the letter size of the GUI displayed on thedisplay panel section102 when the user approaches theimage display device100. In this case, the letter size of the GUI is enlarged when the user approaches theimage display device100, and the letter size of the GUI is reduced when the user backs away from theimage display device100. For example, when the letter size of the GUI is small and causing difficulty in reading, the letter size of the GUI increases as the user approaches, whereby the user can easily recognize the GUI.

Further, as the optimization process of the letter size of the GUI, as shown inFIG. 14(B), there is a process to refine information to be displayed, that is, to increase information amount, by reducing the letter size of the GUI displayed on thedisplay panel section102 when the user approaches theimage display device100. In this case, the letter size of the GUI is reduced and the information to be displayed is increased when the user approaches theimage display device100, and the letter size of the GUI is increased when the user backs away from theimage display device100. For example, if the user is far away when a program list is displayed on thedisplay panel section102, the amount of information is reduced by enlarging the letter size, and when the user approaches, the amount of information is increased by reducing the letter size.

Further, in the present embodiment, as shown inFIGS. 15(A) to 15(C), in the calculation of the viewing positions of the users, the variation in the face size may be corrected by using the reference face size [w0, h0] at the reference distance d0 in a following correction table. For example, from the attribute information such as the user's age, a data table of an average face size of that age is predeterminedly stored; for example, if the user is a child, the reference face size [w0, h0] may be made to be a face size [w0C, h0C] that is smaller than the reference face size shown inFIG. 15(C), and if the user is an adult, the reference face size [w0, h0] may be made to be a face size [w0A, h0A] that is larger than the reference face size shown inFIG. 15(B).

Further, in the present embodiment, in the calculation of the viewing position of the user, in predeterminedly registering the user who uses theimage display device100 in theimage display device100, for example a family of an installation site of theimage display device100, the face size of each user may be registered as a data table. Due to this, the reference face size can be changed for each user. A method to register the face size for each user can be realized by taking images together with distance information in cooperation with another distance sensor (not shown), taking images by guiding the user to a predetermined distance, or taking images at the same distance as a scale to be the reference.

Further, in the present embodiment, even if the user is out of the imaging area of theimaging section104 by estimating the position of the user outside the imaging area from a chronological transition information, the aforementioned system optimization process can be continued.

Further, in the present embodiment, in the system optimization process, an appropriate time constant may be set according to a viewing environment of the user. Due to this, even if the user performs a steep positional transition, the system optimization process can be continued.

Note that, the aforementioned series of processes may be performed by hardware, or may be performed by software. In having the software to perform the series of processes, a program configuring that software is installed from a program storing medium to a computer installed in a dedicated hardware, or for example to a general purpose personal computer that is capable of performing various functions by installing various programs.

The preferred embodiment of the present invention has been described above with reference to the appended drawings, but the present invention is not limited thereto. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

REFERENCE SIGNS LIST

• 100 Image display device
• 102 Display panel section
• 104 Imaging section
• 106 Sensor section
• 108 Speaker section
• 109 Mechanism section
• 110 Control section
• 112 Image input section
• 114 Image processing section
• 116 Viewing state analyzing section
• 117 User position information storing section
• 118 Viewing state recording section
• 120 System optimization processing section
• 122 System control section
• 132 User direction/distance calculating section
• 134 User position information calculating section
• 142 Audio property optimization processing section
• 144 Image property optimization processing section
• 146 Device direction optimization processing section
• 152 Audio property control section
• 154 Image property control section
• 156 Device direction control section
• 200 Data collection server

Claims (18)

The invention claimed is:

1. An information processing device, comprising:

a sensor that detects a user within a detectable range from the sensor; and

processing circuitry that

detects a face of the user based on image data captured by the sensor;

calculates system control information for optimizing an output of a display system based on a respective position of the user detected by the sensor;

analyzes the face of the user to determine an age of the user;

calculates a correction coefficient to correct the system control information according to the age of the user;

corrects the system control information based on the correction coefficient; and

optimizes the output of the display system based on the corrected system control information.

2. The information processing device according toclaim 1, wherein the processing circuitry calculates the system control information to optimize a sound output of the display system based on the respective position of the user detected by the sensor.

3. The information processing device according toclaim 2, wherein the processing circuitry calculates the system control information to optimize a volume balance of the sound output from an audio output of the display system based on the respective position of the user detected by the sensor.

4. The information processing device according toclaim 2, wherein the processing circuitry calculates the system control information to adjust an output direction of the sound output of the display system based on the respective position of the user detected by the sensor.

5. The information processing device according toclaim 1, wherein

the sensor detects a plurality of users within the detectable range from the sensor, and

the processing circuitry calculates the system control information to optimize the output of the display system based on respective positions of each of the plurality of users detected by the sensor.

6. The information processing device according toclaim 1, wherein the processing circuitry calculates the system control information to optimize an image property of the output of the display system based on the respective position of the user detected by the sensor.

7. The information processing device according toclaim 1, wherein the processing circuitry calculates the system control information to optimize a display content that is output by the display system based on the respective position of the user detected by the sensor.

8. The information processing device according toclaim 1, wherein

the processing circuitry calculates the system control information to modify a display direction of the display system based on the respective position of the user detected by the sensor, and

the display system modifies the display direction of the display system according to the system control information.

9. The information processing device according toclaim 1, wherein the processing circuitry further

calculates a position of the user according the face of the user, and

calculates the system control information for optimizing the output of the display system based on calculated position of the user.

10. The information processing device according toclaim 1, wherein the processing circuitry, to correct the system control information based on the correction coefficient, adjusts a size of the output of the display system.

11. The information processing device according toclaim 1, wherein the processing circuitry, to correct the system control information based on the correction coefficient, adjusts a brightness of the output of the display system.

12. The information processing device according toclaim 1, wherein the processing circuitry further

calculates the system control information to optimize a sound output of the display system based on the respective position of the user detected by the sensor, and

adjusts a volume of the sound output of the display system to correct the system control information based on the correction coefficient.

13. The information processing device according toclaim 1, wherein

the output of the display system includes a graphical user interface, and

the processing circuitry, to correct the system control information based on the correction coefficient, adjusts a position of the graphical user interface output by the display system.

14. The information processing device according toclaim 1, wherein

the output of the display system includes a graphical user interface, and

the processing circuitry, to correct the system control information based on the correction coefficient, adjusts a brightness of the graphical user interface output by the display system.

15. The information processing device according toclaim 1, wherein the processing circuitry sets a time constant based on the respective position of the user.

16. A controlling method, comprising:

detecting, by a sensor, a user within a detectable range from the sensor;

detecting a face of the user based on image data captured by the sensor;

calculating, by processing circuitry, system control information to optimize an output of a display system based on a respective position of the user detected by the sensor;

analyzing the face of the user to determine an age of the user;

calculating a correction coefficient to correct the system control information according to the age of the user;

correcting, by the processing circuitry, the system control information based on the correction coefficient; and

optimizing, by the processing circuitry, the output of the display system based on the corrected system control information.

17. The controlling method according toclaim 16, further comprising:

setting, by the processing circuitry, a time constant based on the respective position of the user.

18. A non-transitory computer readable medium storing computer readable instructions that, when executed by a processor, cause the processor to:

control a sensor to detect a user within a detectable range from the sensor;

detect a face of the user based on image data captured by the sensor;

calculate system control information to optimize an output of a display system based on a respective position of the user detected by the sensor;

analyze the face of the user to determine an age of the user;

calculate a correction coefficient to correct the system control information according to the age of the user;

correct the system control information based on the correction coefficient; and

optimize the output of the display system based on the corrected system control information.

Images