CN101394500A - Electronic appliance and control method thereof - Google Patents

Abstract

Provided is an electronic appliance and manufacturing method thereof, when the electronic appliance is controlled by image recognization, it is able to detect a single motion for recognization more accurately without the effect of noise and the like. A control method of the electronic appliance shooting an operator, and part of the operator bode is displaced in a plurality of detection regions where image is divided horizontally by N and vertically by M, wherein the N, M are an integer above 2 separately; according to the image from a video camera, first detection signals are detected in the plurality of detection regions separately based on the parts of the operator body; second detection signals are generated based on the first detection signals, and the second detection signals represent the regions of the plurality of detection regions where part of the operator body is displaced.

Description

Electronic equipment and control method thereof

The application be the denomination of invention submitted on December 14th, 2006 for " electronic equipment ", application number is dividing an application of 200610169367.1 application for a patent for invention.

Technical field

The present invention relates to carry the electronic equipments such as television receiver of video camera, related to the motion images that is used to discern staff etc., carried out the remote operated electronic equipment and the control method thereof of electronic equipment.

Background technology

The eighties in 20th century popularized widely be attached to infrared remote controller (generally being called remote controller) with the home appliance headed by the television receiver on, the user interface that can control has on hand changed the mode of utilizing of home appliance significantly.Present this mode of operation or main flow, but the basic comprising of remote controller is to realize a function by one-touch, for example in television receiver, keys such as " power supply ", " channel ", " volume ", " input is switched " belong to such function, are remote operation methods very easily for television receiver before.

But, at remote controller not on hand or when can not find remote controller, can make us feeling inconvenience extremely.For this, studied the action or the shape of recognition image, carry out the mode of the handover operations such as switch of power supply.For example, Japanese kokai publication hei 11-338614 communique (patent documentation 1) discloses the action or the shape of identification hand, is applied to the technology of operation of equipment.In this technology, with the infrared ray sensor of special use or imageing sensor as being used to detect the action of hand or the checkout gear of shape.

On the other hand, in the data broadcasting that begins recently, for the menu screen of selecting to wish, need push repeatedly remote controller " on ", D score, " left side ", " right side " reach " determining " key, the complex operation on the remote controller uses inconvenient.In addition, for EPG (electronic program list), be after the position of selecting the picture to wish, to press key by introducing of arranged, therefore, there be the problem identical with data playback.And,, also wish similarly to apply flexibly the action of image or shape and mode that can corresponding various operation for so very fine selection operation.

In the TOHKEMY 2003-283866 communique (patent documentation 2), in order to solve such problem following control device has been proposed, promptly, with the position appointed information of using mouse or similar position assigned operation device and obtaining, being encoded to key, to press the timing graph of signal be that key is pressed the sequential code, and this key is pressed the control device that the sequential code sends to television receiver.

Patent documentation 1: the spy opens flat 11-338614 communique

Patent documentation 2: the spy opens the 2003-283866 communique

In normal domestic AV equipment such as television receiver (stereo set or video equipment), apply flexibly former remote controller and realized remote operation.Therefore, not under on hand the situation, when for example wanting power-on, remote controller is obtained at the place that needs to confirm remote controller at remote controller, the key of selection operation correspondence, and the user can feel inconvenience like this.In addition, when not knowing the place of remote controller, just must open the switch of the main power source of television receiver body.The problem points that the frequent in the past experience of these things is crossed about the remote controller operation.

On the other hand, for the operation of powered-down, on hand the time, can utilize remote controller easily, the power supply of closing television receiver at remote controller.But, not under on hand the situation, have same problem when powered-down at the remote controller that leaves seat etc. a little.

The easy action that the action that utilizes in the control mode ofpatent documentation 1 record is circular motion, move up and down, side-to-side movement is such if can realize operation by image recognition, then becomes method of operation very easy to use.But, the action carry out easily in, also have problem to the permission of wrong identification, and with the appropriate scale implement device, and other image recognizing and processing equipments between shared aspect, also have insoluble problem.

Control device shown in thepatent documentation 2 is the indication ball that exactly likes by the operation and the operation of personal computer, and television receiver is carried out remote operated device.Therefore, for the people who does not use personal computer, use inconvenience, from the viewpoint of information usability, it is inappropriate that the handled easily on the personal computer is directly introduced electronic equipment.Therefore, requiring remote operated current utilizing of television receiver to need new operating mechanism in the mode.

In ON/OFF from power supply, require in the image recognition of various selection operation to the image recognition of 2 stage selection operations and menu screen selection etc., using same mechanism and realize new operating mechanism with the proper device scale, is very important problem providing aspect the cheap civil equipment.In addition, the simple image recognition action identification that makes a mistake easily, for example comprise carry out the action similar in the way of seeing TV to identification maneuver and powered-down such cause the possibility that fatal error is moved.

Summary of the invention

The present invention makes in view of such problem, and its purpose is to provide a kind of electronic equipment and control method thereof, when utilizing image recognition control electronic equipment, can not be subjected to the influence of noise etc. and more correctly detects the simple action that is used to discern.

In order to solve above-mentioned problem, the invention provides the electronic equipment and the control method thereof of following (a)～(f).

(a) a kind of control method of electronic equipment, the operator is photographed, the part of this operator's health be put into picture is carried out in the horizontal direction N cut apart, on vertical direction is carried out a plurality of surveyed areas after M is cut apart, wherein, above-mentioned N, M are the integers more than 2; According to the image of being photographed, detect first detection signal that uses the action that the part of health carries out based on the aforesaid operations person respectively from above-mentioned a plurality of surveyed areas; Generate second detection signal according to above-mentioned first detection signal, this second detection signal represents to be put into the zone of a part of the health of the aforesaid operations person on above-mentioned a plurality of surveyed area.

(b) as the control method of (a) described electronic equipment, it is characterized in that, obtain the additive value of each above-mentioned second detection signal of having added up in specified time limit; Use above-mentioned additive value surpass predetermined threshold surveyed area, and comprise at least and near second detection signal of the surveyed area of the surveyed area adjacent detect aforesaid operations person's action with above-mentioned surveyed area.

(c) as the control method of (b) described electronic equipment, it is characterized in that, obtain from surpassing the summation of detected second detection signal of a plurality of surveyed areas that the surveyed area of predetermined threshold intersects with above-mentioned additive value; To first signal waveform of the action that is illustrated in the aforesaid operations person on above-mentioned a plurality of surveyed area of calculating, compare with the pairing secondary signal waveform of predefined action on above-mentioned surveyed area according to above-mentioned summation meter, when the consistent degree of above-mentioned first signal waveform and above-mentioned secondary signal waveform is high, judges and on above-mentioned surveyed area, carried out above-mentioned predefined action.

(d) as the control method of (a) described electronic equipment, it is characterized in that, obtain from the summation of above-mentioned second detection signal of first surveyed area group output and from the summation of above-mentioned second detection signal of second surveyed area group output, the above-mentioned first surveyed area group comprises the full surveyed area of vertical direction of each coordinate of horizontal direction, and the above-mentioned second surveyed area group comprises the full surveyed area of horizontal direction of each coordinate of vertical direction; Obtain the additive value of each above-mentioned summation of having added up in specified time limit; Use additive value surpass predetermined threshold surveyed area, and comprise near second detection signal of the surveyed area of the surveyed area adjacent at least, the action of the part of detecting operation person's health with surveyed area.

(e) as the control method of (d) described electronic equipment, it is characterized in that, to above-mentioned additive value surpass the action of the aforesaid operations person on the surveyed area group of predetermined threshold first signal waveform, compare with the pairing secondary signal waveform of predefined action on above-mentioned surveyed area, when the consistent degree of above-mentioned first signal waveform and above-mentioned secondary signal waveform is high, judges and on above-mentioned surveyed area, carried out above-mentioned predefined action.

(f) a kind of electronic equipment is characterized in that, comprising: video camera, the operator is photographed; Test section, have with the picture of the image that above-mentioned video camera is exported and carry out a plurality of detectors that N is cut apart, carried out the corresponding setting of a plurality of surveyed areas difference after M is cut apart in vertical direction in the horizontal direction, use above-mentioned a plurality of detector to produce first detection signal that uses the action that the part of health carries out based on aforesaid operations person by above-mentioned video camera photography, wherein, above-mentioned N, M are the integers more than 2; Maker generates second detection signal according to above-mentioned first detection signal, and this second detection signal represents to be put into the aforesaid operations person's of above-mentioned surveyed area the zone of a part of health; And motion detector, to first reference signal waveform of the expression aforesaid operations person's that generates according to above-mentioned second detection signal action, compare with the representative detection signal waveform of the predetermined action of decision in advance, whether the predetermined action that detects above-mentioned prior decision is consistent with aforesaid operations person's action.

Description of drawings

Fig. 1 is used to illustrate that electronics of the present invention sets the figure of the summary of method of operation.

Fig. 2 is the block diagram of the major part structure of the television receiver that relates to of one embodiment of the present invention.

Fig. 3 is the figure that the example of television receiver is controlled in action that explanation identifies the operator.

Fig. 4 is the figure of the user's that photographed by video camera of expression appearance.

Fig. 5 is the figure that is used to illustrate the surveyed area in y axis detector, the picture and controls the relation between its commutator pulse.

Fig. 6 is the figure that is used to illustrate the surveyed area in x axis detector, the picture and controls the relation between its commutator pulse.

Fig. 7 is the block diagram of the structure of expression detector shown in Figure 2.

Fig. 8 is the block diagram of the structure of expression object extractor displacer shown in Figure 7.

Fig. 9 is used to illustrate the form and aspect of the object of being extracted out by special color filter shown in Figure 8 and the figure of saturation.

Figure 10 is the flow chart that calculates the processing of form and aspect from color difference signal.

Figure 11 is the figure of expression by the luminance level of the object of gray scale delimiter extraction shown in Figure 8.

Figure 12 is the block diagram of expression motion detection Filter Structures shown in Figure 8.

Figure 13 is the figure of the characteristic of expression motion detection filter.

Figure 14 is the figure that the output of having described the object extractor displacer is presented at the appearance on the picture.

Figure 15 is the block diagram of the structure of expression control information determining device (CPU).

Figure 16 carries out the figure that modelling shows with the output signal of histogram detector in the characteristics of objects Data Detection portion and average brightness detector.

Figure 17 is the figure that is used to illustrate hand that is presented at the vertical action on the picture and the relation between the coordinate of representing surveyed area.

Figure 18 is the detection data of expression x axis detector and y axis detector and according to the table (when the action of hand is longitudinal oscillation) of the value of the center of gravity that detects data computation.

Figure 19 is the sequential chart (when the action of hand is longitudinal oscillation) of change of the barycentric coodinates of expression hand position.

Figure 20 is the block diagram of the structure of expression high pass filter.

Figure 21 is the picture when having described to have limited surveyed area by activation marker (Flg_x) and the figure of commutator pulse.

Figure 22 is the figure of generation method that is used to illustrate the x axle commutator pulse of y axis detector.

Figure 23 is the figure that is used to illustrate the control content that two commutator pulses of x axle by the y axis detector and y axle carry out.

To be expression removed the detection data of x axis detector after the data of unwanted detector and y axis detector and according to the figure (when the action of hand is longitudinal oscillation) of the value of the center of gravity of detection data computation by activation marker (Flg_x) to Figure 24.

Figure 25 is the figure (when the action of hand is longitudinal oscillation) that is used to illustrate the content of cross-correlation digital filter.

Figure 26 is the sequential chart (when the action of hand is longitudinal oscillation) of the change of expression cross-correlation digital filter output.

Figure 27 is the figure that is used to illustrate hand that is presented at the crosswise movement on the picture and the relation between the coordinate of representing surveyed area.

Figure 28 is the detection data of expression x axis detector and y axis detector and according to the table (when the action of hand is teeter) of the value of the center of gravity that detects data computation.

Figure 29 is the sequential chart (when the action of hand is teeter) of change of the barycentric coodinates of expression hand position.

Figure 30 is the picture when having described to have limited surveyed area by activation marker (Flg_y) and the figure of commutator pulse.

Figure 31 is the figure of generation method that is used to illustrate the y axle commutator pulse of x axis detector.

Figure 32 is the figure that is used to illustrate the control content that two commutator pulses of x axle by the x axis detector and y axle carry out.

To be expression removed the detection data of x axis detector after the data of unwanted detector and y axis detector and according to the table (when the action of hand is teeter) of the value of the center of gravity of detection data computation by activation marker (Flg_y) to Figure 33.

Figure 34 is the figure (when the action of hand is teeter) that is used to illustrate the content of cross-correlation digital filter.

Figure 35 is the sequential chart (when the action of hand is teeter) of the change of expression cross-correlation digital filter output.

Figure 36 is the flow chart of the treatment step of expression motion detection method.

Figure 37 is the surveyed area of expression second execution mode and the figure of the detector of correspondence.

Figure 38 is the figure that is illustrated in the hand that has carried out the longitudinal oscillation action on the surveyed area of second execution mode.

Figure 39 is the block diagram of detector with second execution mode of characteristics of objectsData Detection portion 530.

Figure 40 is illustrated in the figure that the surveyed area of difference has taken place in second execution mode.

Figure 41 is the figure of the expression secondobject extractor displacer 510.

Figure 42 is the table of expression from the detection data of the x axis detector of second execution mode and y axis detector.

Figure 43 is illustrated in the figure that has carried out the surveyed area of shielding processing in second execution mode.

Figure 44 is that the embodiment to the menu screen of image that has mixed the operator in the one embodiment of the present invention and menu image has carried out the figure that describes.

Figure 45 is the figure that has described menu screen is carried out the operator's who moves appearance.

Embodiment

With reference to the description of drawings one embodiment of the present invention.

Fig. 1 is used to illustrate the mode of operation of former remote control and the figure of the difference between the mode of operation of the present invention.During user (operator) 3operation television receivers 1, be thatuser 3 presses the key that function is wished in startup by hand-heldremote control device 4 towardstelevision receiver 1 in the past, operated.Therefore, if there is notremote control 4 just can not operate, experience bigger inconvenience often.

In the present embodiment, ontelevision receiver 1, be provided withvideo camera 2 as illustrated in fig. 1, takeusers 3, go outuser 3 action, carry out the operation oftelevision receiver 1 and relative equipment according to the image detection ofvideo camera 2 byvideo camera 2.

Detected user's 3 action specifically refers to the demonstration/non-demonstration control of the control of the electric power on/off oftelevision receiver 1, menu screen, from the menu screen specific action ofuser 3 health (hand, pin, face etc.) of having selected to wish use that the control of button is corresponding.By detecting this specific action, carry out the operation of electronic equipment.In the present embodiment, illustrate that most probable is realized, with the method that moves forward into line operate of hand.

Fig. 2 is the block diagram of the structure of expression television receiver 1.Television receiver 1 hasbenchmark synchronizing generator 11,timing pulse generator 12,graphic generator 16,video camera 2,mirror transformation device 17, scaler (scale) 15,first blender 17,pixel count converter 21,second blender 22,display unit 23, test section 19 and control information determining device (CPU) 20.

Horizontal cycle pulse and vertical cycle pulse thatbenchmark synchronizing generator 11 produces the benchmark that becomes television receiver 1.During when television broadcasting receives with from external equipment input signal of video signal, generate the synchronous pulse of synchronizing signal with this input signal.Timing pulse generator 12 generates and described laterly shown in Figure 4 detects respectively that piece (surveyed area) is necessary, phase place arbitrarily with horizontal direction and vertical direction and the pulse of width.

Video camera 2 is positioned at the front of television receiver shown in Figure 11, takes the image of user (operator) 3 ortelevision receiver 1 front.The output signal ofvideo camera 2 is brightness (Y) signal and aberration (R-Y, B-Y) signal, with horizontal cycle pulse and the vertical cycle impulsive synchronization from 1 output of benchmark synchronizing generator.In addition, in the present embodiment, the pixel count of the image of being taken byvideo camera 2 is consistent with the pixel count of display unit 23.Under the inconsistent situation of pixel count, insert the pixel converter, make that pixel count is consistent to get final product.

Mirror transformation device 14 is used for will being shown reversing about as mirror on thedisplay unit 23 by photography body image (user 3) byvideo camera 2 photography.Therefore, when character display, with mirror in the same manner about the counter-rotating.In the present embodiment, utilize memory, and pass through method, carry out mirror transformation the image inversion of horizontal direction.

Use under the situation of CRT (cathode ray tube) asdisplay unit 23,, can obtain identical effect by operant level deflection on the contrary.In this case, need prior image to reverse in the horizontal direction with figure and other mixing side.

Scaler 15 is to adjust the device of the size of the body image of being taken byvideo camera 2 of being photographed, and adjusts magnification ratio and minification by two-dimentional the going up of being controlled at of control information determining device (CPU) 20.In addition, can also not amplify and dwindle, just carry out level and vertical phase place adjustment.

Graphic generator 16 is the devices that are used for launching the menu screen that sends from control information determining device (CPU) 20, even the signal on the memory is unfolded with the primary signal of R (red) signal, G (green) signal, B (orchid) signal, in back level output signal compound with signal of video signal or that mix is brightness (Y) signal and aberration (R-Y, B-Y) signal.In addition, do not limit the number of planes of the figure of generation, but what be used to illustrate is a plane.

In the present embodiment, make pixel count consistent with the pixel count of display unit 23.Under inconsistent situation, need to insert the pixel count converter and make its unanimity.

First blender 17 utilizes controllingvalue α 1 control mixed proportion, comes the output signal Gs ofmixed graph maker 16 and the output signal S1 of calibration device 15.Specifically, represent output signal M1o with following formula.

M1o＝α1·S1+(1-α1)·Gs

Controllingvalue α 1 is set to the value between 0 to 1, if controllingvalue α 1 is bigger, then the ratio of the output signal S1 ofscaler 14 becomes big, and the ratio of the output signal Gs ofgraphic generator 16 diminishes.Example as blender is not limited to this, but in the present embodiment, if comprise the signal message of the two system of input, then can obtain identical effect.

Test section 19 comprise the 1st detector the 301, the 2nd detector the 302, the 3rd detector 303 ... n detector (300+n).Do not limit the detector number that comprises in the test section 19, but have 25 detectors in first execution mode of the present invention, they are included in 16 of the1st detector 301～the 16th detector 316 of timing working of horizontal direction described later and 9 of the17th detector 317～the25th detector 325 of the timing working of vertical direction.

Detector number is not limited thereto, and in order to improve accuracy of detection, detector number is The more the better, but wishes according to adjusting quantity with the relation of treatment scale.Use 25 detectors at first execution mode, use 144 detectors at second execution mode.

Control information determining device (CPU) 20 carries out exporting various control signals from the parsing of the data of test section 19 outputs.The contents processing of controlinformation determining device 20 is realized by software, is described in detail later about algorithm.In the present embodiment, mixed processing by hardware (each functional block) and the processing by software (on CPU20, launching), but be not limited to boundary shown here.

Pixel count converter 21 makes from the pixel count of the external input signal of outside input and the consistent pixel count conversion of pixel of display unit 23.External input signal is assumed to the signal from the outside input of television receiver such as the TV signal (comprising data broadcasting etc.) of broadcasting, (VTR) input of recording a video.In this explanation, omitted the synchro system of external input signal, but its formation is, obtains synchronizing signal (level and vertical), make synchronous unanimity bybenchmark synchronizing generator 11.

Second blender 22 has and first blender, 17 identical functions.That is,, mix the output signal Mio offirst blender 17 and the output signal S2 ofpixel count converter 21 with controllingvalue α 2 control mixed proportions.Specifically, represent output signal M2o with following formula.

M2o＝α2·M1o+(1-α2)·S2

Controllingvalue α 2 is set to the value between 0 to 1, if controllingvalue α 2 is bigger, then the ratio of the output signal M1o offirst blender 17 becomes big, and the ratio of the output signal S2 ofpixel count converter 21 diminishes.Example as blender is not limited to this, but in the present embodiment, if comprise the signal message of the two system of input, then can obtain identical effect.

Display unit 23 is assumed to CRT, LCD (LCD), plasma scope (PDP) or projection display equipment etc., but is not the display mode that limits display.The input signal ofdisplay unit 23 is luminance signal and color difference signal, is transformed to the RGB primary signal in the internal matrix ofdisplay unit 23 and shows.

Comprisinguser 3 action, the action of thetelevision receiver 1 that constitutes as described above is being described.Fig. 3 is the action of the hand that is used to illustrate thatuser 3 carries out and to the figure of the control content of thetelevision receiver 1 that should move.The diagram of the manual work that theuser 3 that stands intelevision receiver 1 front carries out is shown with arrow among Fig. 3 (A).In the present embodiment, establish the action that the user carries out be " hand is (up and down) swing vertically " and " hand laterally (about) swing " 2 actions.

Fig. 3 (B) (1)～(B) (3) (below (B) (1) is expressed as B-1, other is also identical) represents to have carried out the manual transition of carrying out withuser 3television receiver 1, that be presented at the content on thedisplay unit 23 of doing corresponding control respectively.In the present embodiment, the control content fortelevision receiver 1 is three controls of " power supply is become unlatching from closing ", " display menu picture ", " elimination menu screen ", " from unlatching become close power supply ".

Corresponding relation between the manual work thatuser 3 carries out and the control content oftelevision receiver 1 is, the action ofuser 3 " hand longitudinal oscillation " is corresponding to the control of " at the power supply oftelevision receiver 1 is when closing; power-on " and " at the power supply oftelevision receiver 1 is when opening, the display menu picture ".The action of " hand teeter " is corresponding to the control of " nomatter television receiver 1 is in what kind of picture state, all powered-down ".

Fig. 3 (B-1) shows the power-off oftelevision receiver 1, the state that does not show ondisplay unit 23 whatever.Under this state, ifuser 3 swings hand vertical (up and down), thenvideo camera 2 photographs this action, opens the power supply oftelevision receiver 1, shows the television image (program) shown in Fig. 3 (B-2) ondisplay unit 23.

At first, shown in Fig. 3 (B-1), ondisplay unit 23, not show whatever, so the user can't confirm self image byvideo camera 2 photographies.Therefore,user 3 must stand in the position thatvideo camera 2 one photographs surely, evenuser 3 had been photographed in a certain position in the video image ofvideo camera 2 photographies,television receiver 1 also needs to discernuser 3 action.Under this situation,, also no problem even withoutdisplay unit 23 andgraphic generator 16.

Further, under the state that shows the television image shown in Fig. 3 (B-2) on the display unit 23 (audiovisual state), when the user with hand vertically during (up and down) swing, the displaying contents ofdisplay unit 23 becomes the menu screen shown in Fig. 3 (B-3), transfers to the selection action of channel alteration etc. thus.Under this situation,display unit 23 has also just shown television image at first, and the user can not be presented at self the image that is photographed byvideo camera 2 on thedisplay unit 23 and confirm.Therefore, same with foregoing, no matteruser 3 is arranged in any position by the image ofvideo camera 2 photographies,television receiver 1 also must be discerneduser 3 action.

Show indisplay unit 23 under the state (Fig. 3 (B-2)) of television images, whenuser 3 with hand laterally (about) during swing, the power-off oftelevision receiver 1 becomes the state shown in Fig. 3 (B-1).Under the state (Fig. 3 (B-3)) that has shown all pictures such as menu, data broadcasting, EPG,, become the state shown in Fig. 3 (B-2) or Fig. 3 (B-1) as user during with the hand teeter.

Action in the present embodiment employing with the vertical or horizontal swing of hand, be that people are at the daily movement content that carries out, the implication that the action of hand longitudinal oscillation is generally had " coming; come " such greeting, from having given the implication that enters (transferring to) NextState as previously mentioned, also can be described as suitable action.In addition, be action when saying the difference of " good-by " with the action of hand teeter, from given the implication that breaks away from from particular state for it, also can be described as suitable action.The implication of this action is different and different with ethnic group because of country, also can consider to adopt the situation of other action, but consider from angle easy to use, wishes to carry out according to the implication of action as far as possible.

Exemplified the control example oftelevision receiver 1 simple and that understand easily at this, but, come the appropriate change control content to get final product according to commodity design according to the function thattelevision receiver 1 has.

In addition, when the power supply of consideringtelevision receiver 1 was opened from being closed to, the situation thatuser 3 is leaving from best audiovisual point wished the camera coverage ofvideo camera 2 is made wide-angle, increases the scope of identification maneuver as far as possible.And, with display frame from the state of audiovisual TV programme when the changes such as menu screen, can imagine thatuser 3 is positioned at from the near position of best audiovisual point, so, the camera coverage ofreduction camera 2 to a certain extent.

Fig. 4 is the figure that is used to illustrate the surveyed area of the manual work that detects user 3.Fig. 4 represents by the user's 3 ofvideo camera 2 photographies image and the coordinate of horizontal direction x and vertical direction y.In the present embodiment, will carry out in the horizontal direction from the picture of the image ofvideo camera 2output 16 cutting apart, carry out 9 in vertical direction and cut apart and a plurality of surveyed areas of being provided with identification user's 3 manual work.As shown in Figure 4, in the level that image is presented on the display unit 23: vertically in thetelevision receiver 1 for the length-width ratio of 16:9, if as described above picture segmentation is becomelevel 16, vertical 9, then 1 interval that forms of cutting apart by vertical (y axle) direction and level (x axle) direction is square.Respectively cut apart number and be the integer more than 2, suitably set and get final product.

When detecting manually the doing ofuser 3, can consider following 2 situations: use picture is cut apart 16 surveyed areas of setting and cut apart the situation of whole 25 surveyed areas (one dimension) of 9 surveyed areas of setting at the y direction of principal axis at the x direction of principal axis; With, picture is carried out 16 cutting apart at the x direction of principal axis, carries out 9 at the y direction of principal axis and cut apart, as 1 surveyed area, use the situation of whole 144 surveyed areas (two dimension) by these 1 intervals cutting apart formation.If surveyed area is 25, then can cut down hardware size, therefore wish.In addition, when surveyed area is handled as 144,, can adapt to processing same when being 25 with surveyed area as long as will be the information separately of x axle and y axle from the information conversion of each surveyed area.

At first, as first execution mode of the present invention, the situation that whole 25 surveyed areas are set is described on picture.Fig. 5 is used to illustrate to be divided into the figure of 9 surveyed area from the picture of the image ofvideo camera 2 outputs at the y direction of principal axis.Fig. 5 represents by the image of the user's 3 ofvideo camera 2 photography hand, cuts apart and the quadrangle of with dashed lines is represented 9 surveyed areas and commutator pulse, alsoexpression 17th detector 317～25th detector 325 (y axis detector) corresponding with each surveyed area in the y direction.

At each surveyed area, the axial picture of y center is made as 0, given the coordinate of the position relation of expression from-4～+ 4.The y axial coordinate is-4 corresponding the17th detector 317 of surveyed area, the y axial coordinate is-3 corresponding the18th detector 318 of surveyed area, the y axial coordinate is-2 corresponding the19th detector 319 of surveyed area, and the coordinate of y axle is respectively corresponding the 20th detector 320 of each surveyed area～the25th detector 325 of-1～+ 4.Eachy axis detector 317～325th produces the detector of the detection signal of the manual work that carries out based onuser 3.

Eachy axis detector 317～325 is moved based on the commutator pulse of supplying with from timing pulse generator 12.Among Fig. 5, at y direction of principal axis (vertically) and x direction of principal axis (level), illustrate respectively and be used to make the19th detector 319 corresponding y axial coordinates for the commutator pulse of-2 surveyed area action with to be used to make the25th detector 325 corresponding y axial coordinates be the commutator pulse of 4 surveyed area action.

The commutator pulse that the x direction of principal axis is represented be have with effective image during the pulse of the suitable width of the width of horizontal direction, be to be divided into 9 the suitable pulse of width with width at the commutator pulse that the y direction of principal axis is represented with the vertical direction during effective image.Each y axis detector to other is also imported same commutator pulse respectively.

Fig. 6 is used to illustrate to be divided into the figure of 16 surveyed area from the picture of the image ofvideo camera 2 outputs at the x direction of principal axis.Fig. 6 represents by the image of the user's 3 ofvideo camera 2 photography hand, cuts apart and the quadrangle of with dashed lines is represented 16 surveyed areas and commutator pulse, alsoexpression 1st detector 301～16th detector 316 (x axis detector) corresponding with each surveyed area in the x direction.

At each surveyed area, the approximate centre of the axial picture of x is made as 0, given the coordinate of the position relation of expression from-8～+ 7.The coordinate of x axle is-8 corresponding the1st detector 301 of surveyed area, the x axial coordinate is-7 corresponding the 2nd detector 302 of surveyed area, the x axial coordinate is-6 corresponding the 3rd detector 303 of surveyed area, and the coordinate of x axle is respectively corresponding the 4th detector 304 of each surveyed area～the 16th detector 316 of-5～+ 7.Eachx axis detector 301～316th produces the detector of the detection signal of the manual work that carries out based onuser 3.

Eachx axis detector 301～316 is moved based on the commutator pulse of supplying with from timing pulse generator 12.Among Fig. 6, at x direction of principal axis (level) and y direction of principal axis (vertical), illustrate respectively and be used to make the 2nd detector 302 corresponding x axial coordinates for the commutator pulse of-7 surveyed area action with to be used to make the 16th detector 316 corresponding x axial coordinates be the commutator pulse of 7 surveyed area action.The commutator pulse that the y axle is represented be have with effective image during the pulse of the suitable width of the width of vertical direction, be to be divided into 16 the suitable pulse of width with width at the commutator pulse that the x axle is represented with the horizontal direction during effective image.X axis detector to other is also imported same commutator pulse respectively.

As shown in Figure 7, the1st detector 301～the25th detector 325 has the firstobject extractor displacer 51,time grate device 52, characteristics of objectsData Detection portion 53 respectively.Time grate device 52 is according to Fig. 5 and commutator pulse shown in Figure 6, and control is passed through from the picture signal ofvideo camera 2.

The zone that picture signal is passed through is located in each surveyed area that the quadrangle of Fig. 5 and Fig. 6 with dashed lines represents.To the signal that in this surveyed area, limits, carry out various Filtering Processing described later, extract the user's 3 who captures byvideo camera 2 hand out.

The firstobject extractor displacer 51 has the filter that has added characteristics of image, in the present embodiment in order to detectuser 3 hand, is conceived to the Filtering Processing of the colour of skin especially and detects the motion detection Filtering Processing of moving.As shown in Figure 8, the firstobject extractor displacer 51 specifically comprisesspecial color filter 71,gray scale delimiter 72,motion detection filter 75,recombiner 73,object gate 74.

With reference to Fig. 9special color filter 71 is described.Fig. 9 is the aberration plane graph, and the longitudinal axis is made as R-Y, and transverse axis is made as B-Y.The all colours signal of TV signal can be estimated with polar coordinates with the vector representation on this coordinate.Special color filter 71 is to limit the form and aspect of the color signal of importing with color difference signal and the device of colour saturation (saturation).For specific it, be that the angle that benchmark (0 degree) turns left shows form and aspect in order to the B-Y axle of first quartile.In addition, saturation is the scalar of vector, and the initial point on aberration plane is that saturation is 0 not have the state of color, and along with away from initial point, it is big that saturation becomes, and the expression color thickens.

In Fig. 9, scope by 71 extractions of special color filter, be set to than little corresponding to theangle θ 1 of isochrome phase line L1 and with etc. the scope of thecorresponding angle θ 2 of aberration line L2, in addition, color depth is set to the big and scope littler than L4 than equisaturation line L3.This scope of second quadrant is equivalent in the color of the staff of present embodiment extraction, is the zone of the colour of skin that still, the zone of the color of extraction is not particularly limited in this.

Whetherspecial color filter 71 calculates angle and saturation according to the color difference signal (R-Y, B-Y) fromvideo camera 2 inputs, detect color difference signal and enter by waiting aberration line and equisaturation line area surrounded.

As an example of angle calculation, be to handle by the angle calculation shown in the flow chart of Figure 10, each input pixel is calculated angulation on aberration plane shown in Figure 9.In the present embodiment, utilize hardware to realize the angle calculation processing, but also can realize with the some of software and hardware.

At first, at step S401 shown in Figure 10, according to the color difference signal R-Y of each input pixel, the symbol of B-Y composition, the form and aspect of detection input pixel are positioned at which quadrant on the aberration plane.

Then, at step S402 comparison colours difference signal R-Y, B-Y composition absolute value separately | R-Y|, | B-Y|, will be bigger as A, with less calculating as B.

Then, at step S403, according to B/A detection angles T1.From the processing of step S402 as can be known, this angle T1 becomes 0 °～45 °.Can calculate angle T1 with piecewise linear approximation, ROM table etc.

At step S404, judge that whether A is | whether R-Y| promptly is | R-Y|〉| B-Y|.If judge it is "No", promptly be not | R-Y|〉| B-Y| then directly enters step S406.If be judged as "Yes", promptly be | R-Y|〉| B-Y|, then enter step S405, angle T1 is replaced with angle T as (90-T1).Thus, obtain tan^-1((R-Y)/(B-Y)).

The reason that will be made as 0 °～45 ° at the angle T1 that step S403 detects is, if tan^-1The bending of ((R-Y)/(B-Y)) surpasses 45 °, and it is big that its gradient sharply becomes, and is not suitable for the calculating of angle.

Moreover, at step S406, use the quadrant data that detect at step S401 to judge whether it is second quadrant, if second quadrant enters step S407, calculate T=180-T1.If not second quadrant, enter step S408, judge whether it is third quadrant, if third quadrant enters step S409, calculate T=180+T1.

If not third quadrant, enter step S410, judge whether it is four-quadrant, if four-quadrant enters step S411, calculate T=360-T1.If neither four-quadrant, when promptly being first quartile, angle T be made as T1 at step S412.Then, finally export angulation T on each input pixel aberration plane at Fig. 9 at step S413.

By above processing, can in 0 °～360 ° scope, obtain color difference signal R-Y, the B-Y angle on the aberration plane of input.Step S404～S411 will be modified to the processing of angle T at the angle T1 of step S403 detection.In addition, step S404～S411 according to first quartile～four-quadrant correction angle T1.

Then, utilize following formula to carry out calculating as the saturation of color depth.

Vc＝sqrt(Cr×Cr+Cb×Cb)

Vc is the scalar of vector, in this expression saturation.Cr is color signal shown in Figure 9 (R-Y) axle composition, and Cb is (B-Y) axle composition.In addition, sqrt () is the operator of carrying out square root calculation.

The processing here is not specific to software or hardware, but multiplication and square root are not easy realization with hardware, and calculation step is also a lot of in software in addition, is undesirable, therefore can also use following approximate expression.

Vc＝max(|Cr|，|Cb|)+0.4×min(|Cr|，|Cb|)

Wherein, max (| Cr|, | be to select Cb|) | Cr| and | bigger one calculation process among the Cb|, min (| Cr|, | be to select Cb|) | Cr| and | less one calculation process among the Cb|.

Whether the angle that evaluation is obtained by above step (form and aspect) T and saturation Vc the scope of theangle θ 1 toθ 2 of aberration line such as enter into, and whether the concentration of color enters into the scope littler and bigger than L3 than equisaturation line L4.The effect ofspecial color filter 71 shown in Figure 8 is that the signal that enters this scope is passed through.

As shown in figure 11, thegray scale delimiter 72 of Fig. 8 limits the specific tonal range of luminance signal.Under the situation of 8 position digital signals, in 0～255 256 grades, at random set the maximum level Lmax and the minimum levels Lmin of gray scale, output is comprised in the luminance signal of the grey level between the Lmax to Lmin.

Use themotion detection filter 75 of Figure 12 and Figure 13 key diagram 8.As shown in figure 12,motion detection filter 75 has a frame delay device 75-1, subtracter 75-2, absolute value device 75-3, nonlinear processor 75-4, quantizer 75-5, according to moving of the luminance signal detected image of importing.

At a frame delay device 75-1, be delayed 1 frame from the picture signal ofvideo camera 2, be input to subtracter 75-2.Subtracter 75-2 calculates from the picture signal ofvideo camera 2 with from the difference of the picture signal of a frame delay device 75-1, exports to absolute value device 75-3.The symbol direction of not special provision subtraction.Because according to the level of signal, differential signal can be exported the value of two positive and negative directions,, export to nonlinear processor 75-4 so absolute value device 75-3 carries out absolute value to the difference value from subtracter 75-2 input.

Nonlinear processor 75-4 implements Nonlinear Processing based on input-output characteristic shown in Figure 13 to differential signal input, that carried out absolute value.In Figure 13 (A), transverse axis has been represented from the carrying out of absolute value device 75-3 input the differential signal of absolute value, and the longitudinal axis is represented from the signal of nonlinear processor 75-4 output.A value and b value are variable in the scope of scope R1 and R2 respectively.

The output signal of nonlinear processor 75-4 is input to quantization unit 75-5, carries out binaryzation based on the defined threshold shown in Figure 13 (B).

The recombiner 73 compound signals fromspecial color filter 71,gray scale delimiter 72 and 75 inputs of motion detection filter of Fig. 8 are transformed into regional pulse.In the present embodiment, when the signal that has passed throughspecial color filter 71, the signal that passed through the signal ofgray scale delimiter 72 and passed throughmotion detection filter 75 all existed, output became the regional pulse of high level.

Supply withobject gate 74 by the regional pulse that recombiner 73 generates.When regional pulse was high level,object gate 74 passed through luminance signal and color difference signal.When regional pulse is low level (scope beyond the regional pulse), input signal (luminance signal and color difference signal) is passed through, the signal of output setting.In the present embodiment, the luminance signal of output black level and the color difference signal ofsaturation 0.

Special color filter 71 limits form and aspect (angle) and the saturation with the color signal of color difference signal input, andgray scale delimiter 72 limits the specific tonal range of luminance signal, andmotion detection filter 75 is according to the mobile qualification luminance signal of image.

By limiting form and aspect and saturation byspecial color filter 71, can focus on people's the colour of skin, but people's the colour of skin changes because of the Exposure to Sunlight situation, because of ethnic group is different, the colour of skin is various in addition.Therefore, if according to the control signal from 20 inputs of control information determining device, adjust form and aspect, saturation withspecial color filter 71, the gray scale limited range withgray scale delimiter 72 adjustment luminance signals then roughly can detect staff.Moreover, by the action ofmotion detection filter 75, can extract and discern staff out according to image.

Figure 14 (A) is that the output signal of the firstobject extractor displacer 51 is presented at the figure on the display unit 23.In the image byvideo camera 2 photographies, the image of hand shows that based on the signal of being extracted out by the firstobject extractor displacer 51 part beyond the image of hand is made as black level with luminance signal, therefore shows whatever.According to the signal of this extraction, the feature that analysis image had and position on the picture and movement content,identification user 3 has made goal-directed action.

At Figure 14 (B) expression commutator pulse, by the image of the signal of video signal oftime grate device 52 gatings of Fig. 7 based on each detector action that corresponding each surveyed area is provided with.At this, as typical example, expression and y axial coordinate be the 21st corresponding detector 321 of the surveyed area of 0 (zero) and with the y axial coordinate be output signal the 20th corresponding detector 320 of-1 surveyed area, that export fromtime grate device 52.

Characteristics of objectsData Detection portion 53 carries out also detecting the Filtering Processing of its feature from the picture signal shown in Figure 14 (A).As shown in figure 15, characteristics of objectsData Detection portion 53 has the functional block that detects various feature from image, i.e.histogram detector 61, mean flow rate (APL)detector 62, high frequency generatingcapacity detector 63,minimum detector 64 and maximum value detector 65.Though also have the important document of other definite characteristics of image, but in the present embodiment, according to the detection signal that produces from thesedetectors 61～65, generate expression by the detection signal of the 1st motion detector 20-1～the 5th motion detector 20-5 in the zone of the detected hand of surveyed area, differentiating signal of video signal is the signal of having taken hand, and discerns the action of this hand.

Histogram detector 61, mean flow rate (APL)detector 62, high frequency generatingcapacity detector 63,minimum detector 64 andmaximum value detector 65, constitute by hardware in present embodiment, with picture unit (and frame unit: generate the data (detection signal) of each interior feature of surveyed area in the representation space vertical cycle unit), send to controlinformation determining device 20 through cpu bus.

The data that controlinformation determining device 20 will send from eachdetector 61～65 as variable storage, are carried out data processing on software.

Histogram detector 61 will be divided into for example eight rank from the gray scale of the luminance signal oftime grate device 52 output, the pixel count that is present in each rank is counted, will be represented that by each picture (or a frame picture) histogrammic data export to the 1st motion detector 20-1.Meanflow rate detector 62 similarly carries out add operation to the intensity level in the picture and the average brightness value of a picture removing with whole pixel counts to the 2nd motion detector 20-2 output.

High frequencygenerating capacity detector 63 is extracted radio-frequency component out by spatial filter (two dimensional filter), the generating capacity of the radio-frequency component in picture of the 3rd motion detector 20-3output.Minimum detector 64 is to the minimum gradation value of the luminance signal of a picture of the 4th motion detector 20-4 output, and in addition,maximum value detector 65 is exported the maximum gradation value of the luminance signal in the picture to the 5th motion detector 20-5.

The 1st motion detector 20-1～the 5th motion detector 20-5 with the data that receive as variable storage, by software data processing.The processing that detection described later is manually done is the processing of being undertaken by software in the present embodiment.Controlinformation determining device 20 has based on the control information generator 20-10 that produces control signal from the detection signal of the 1st motion detector 20-1～the 5th motion detector 20-5.

Figure 16 shows the data of histogram detector from characteristics of objectsData Detection portion 53 61 andaverage brightness detector 62 outputs has been carried out modeled result.Figure 16 is a transverse axis as the gray scale that is divided into 0～7 8 rank, the longitudinal axis histogram as frequency.Mean flow rate (APL) represents with arrow, so that understand its size sensuously.

Figure 16 (A) expression constitutes thehistogram detector 61 and the on average output ofbrightness detector 62 of the 20th detector 320 of Figure 14 (B).Shown in Figure 14 (B), hand is not placed on surveyed area in the 20th detector 320, does not sell so detect infirst object detector 51, shields with black level from the signal of the first object extractor displacer, 51 outputs.Therefore, the histogram shown in Figure 16 (A) becomes the partial data of having only minimum gray scale (0).In addition, signal is deceived basically, and APL is 0 (zero), but lower in order to express signal level, represents with short arrow.

Figure 16 (B) expression constitutes thehistogram detector 61 and the on average output ofbrightness detector 62 of the 21st detector 321 of Figure 14 (B).Shown in Figure 14 (A), in the21st detector 21, detect the hand that is placed on surveyed area with the firstobject extractor displacer 51, therefore, the output of thehistogram detector 61 shown in Figure 16 (B) is, except thegray scale 0 of the black level of conductively-closed, frequency is distributed on the gray scale of brightness of hand.In addition, about APL, the signal component by hand rises mean flow rate, so represent with long arrow.

In the present embodiment, ask, be placed on the data in zone of the hand of surveyed area as expression from minimum gray scale (0) summation in addition of the data ofhistogram detector 61 output.Promptly, extracted output signal out according to theobject extractor displacer 51 of the detector of corresponding surveyed area setting at the hand of action,histogram detector 61 produces first and detects data, the 1st motion detector 20-1 detects data based on first, generates the second detection data of expression from the zone of the hand of surveyed area extraction.

And, calculate frequency byhistogram detector 61 according to being divided into the gray scale black and second order of composition beyond it, also can extract out by carrying out compulsory exercise and be placed on the hand of surveyed area.Therefore, can also obtain the second detection data in the zone of expression hand according to detecting data from first of thehistogram detector 61 that is reduced to 0 gray scale and other two gray scales.

Moreover, in the present embodiment, based on detecting data from first ofhistogram detector 61 outputs, generated second at the 1st motion detector 20-1 and detected data, but be not limited to this, as long as detect data, detect data in 20 generations second of control information determining device and get final product according to first of characteristics ofobjects data detector 53 outputs that had from eachdetector 301～325.

Figure 17 isuser 3 when vertically (up and down) is mobile with hand in the zone ofvideo camera 2 photography, an example of the image of the hand of being photographed by video camera 2.Show the arrow and the xy coordinate that is configured in the surveyed area in the picture of the direction of representing that hand moves simultaneously.Represent in four positions of the hand that moves in Figure 17 (A), (B), (C), (D) extraction.Figure 17 (A) expression hand is present in the situation of the top, and slightly to the situation of lower swing, further to the situation of lower swing, Figure 17 (D) expression hand is positioned at the situation of below with hand in Figure 17 (C) expression with hand in Figure 17 (B) expression.

In the present embodiment, will move four times down on hand.That is, (A), (B) of Figure 17, (C), (D), (D), (C), (B), (A) as 1 cycle, four cycles have been moved with hand.Under the situation about moving up and down like this, move hardly, be positioned on the identical coordinate at x direction of principal axis hand.On the other hand, at the y direction of principal axis, the coordinate of hand changes up and down.Therefore, detected data have become repetition up and down four cycles of peak value, show as the change value of the dateout of each detector that is provided with from the surveyed area of corresponding each coordinate.

Data value that Figure 18 is exported eachhistogram detector 61 in the testing result that moves up and down of hand shown in Figure 17, eachdetector 301～325 and the content of having handled these data are as representing.The left column of this table is an entry name, shows the data value of the projects that change along with the process of time on the right side that project ranks.

The Cycle of project represents the cycle that moves up and down of above-mentioned hand, and 2 initial cycles of having write out in this table in whole four cycles are represented.The frame number of project n presentation image under the situation of general vision signal, is made as the 60Hz cycle, under interleaved situation, constitutes a frame by 2 fields, and a vertical cycle is made as the 60Hz cycle.

Project ph represents which position the hand that moves up and down is positioned at, and A, B, C, D be (A), (B), (C), (D) of corresponding Figure 17 respectively.The x of project (i) (i=-8～+ 7) represent respectively the histogram detector from the1st detector 301～the 16th detector 316 61 that obtain as previously mentioned, based on first the surveyed area of correspondence detect data represent hand the zone second detect data.Similarly, the y of project (j) (j=-4～+ 4)expression 61 that obtain from the histogram detector of the17th detector 317～the25th detector 325, based on first of the hand of extracting out at the surveyed area of corresponding eachdetector 301～325 detect data represent hand the zone second detect data.In addition, project XVS, XVSG, XG, YVS, YVSG, YG are the contents of having handled the data that obtain from eachdetector 301～325, are described in detail later.

In an example shown in Figure 17 (A)～(D), hand moves at above-below direction, and at the x direction of principal axis, the position of mobile hand does not change, so the data of project x (i) do not change.Shown in Figure 17 (A)～(D), hand for being centered close on the x coordinate 4～6, shows the value that has detected hand with x coordinate 5 on project x (4), the x (5) of the table of Figure 18 and x (6).Other project x (i) shields with the firstobject extractor displacer 51, and therefore value is 0 (except project x (1), y (2), the y (3) of frame number 11).

This only is an ideal situation, if the part of the expression colour of skin beyonduser 3 the hand is moving, then also produces detected value in the coordinate beyond the coordinate of the surveyed area of having placed hand, and the detection of moving for hand is a noise.Its emphasis is, how to suppress such noise, and the action of hand is discerned as operation information.

At the y direction of principal axis, will move down on hand, so the data movement of project y (j).Therefore at Figure 17 (A), hand is positioned on y coordinate 2 and 3, in the frame number of the project y (2) of Figure 18 and y (3) is 0 hurdle detected value has been shown.Similarly, for Figure 17 (B), (C), (D), in the project (y) corresponding, show detected value separately with separately the y coordinate that is placed with hand.

The value of the data of project x shown in Figure 180 (i), y (j) (second detects data) is based onhistogram detector 61 detected signals.In the present embodiment, picture is divided into 16, is divided into 9 and a interval that 25 surveyed areas being provided with intersect is made as the value in 100 (zones) at the y direction of principal axis at the x direction of principal axis, carried out the first ratio adjustment that detects data.Second detects data is based on according to what the output signal of having extracted the hand that is placed on surveyed area out generated and first detects data, represents to be placed on the data of size in zone of the hand of surveyed area.

In the present embodiment, with follow comparing of effluxion by the change of the value shown in projects, the change that promptly detects data based on second of the1st detector 301～the25th detector 325 output separately, what is more important is according to the change based on second summation that detects data that is detected data by first of a plurality of detector outputs data that obtain, that the expression center of gravity moves.Therefore, by the hand placed and based on output signal, obtain all centers of gravity of a plurality of surveyed areas of being placed with hand (below, simply be called " center of gravity of hand "), and it is estimated from each extraction of a plurality of surveyed areas.

Can obtain the center of gravity XG of the hand on the x coordinate that frame number is n with following mathematical expression 1.XVS is the summation that second of each x axis detector (the1st detector 301～the 16th detector 316) detects data, is based on the value by the output signal of being extracted out by the firstobject extractor displacer 51 at a plurality of surveyed areas at the hand of action.XVSG detects the x coordinate that multiply by corresponding surveyed area on the data in second of each x axis detector, has carried out the summation of the detection data of weighting.

[mathematical expression 1]

$\mathrm{XG}\left(n\right)=\frac{\mathrm{XVSG}\left(n\right)}{\mathrm{XVS}\left(n\right)}=\frac{\underset{i=-8}{\overset{7}{\mathrm{\Σ}}}i\×x(n,i)}{\underset{i=-8}{\overset{7}{\mathrm{\Σ}}}x(n,i)}---\left(1\right)$

In the present embodiment, the project XG of Figure 18 becomes 5 (except frame numbers 11) in nearly all frame, and therefore, the x coordinate of the center of gravity of hand is 5, is that centre data increases with x coordinate 5.

Obtain the center of gravity YG of the hand on the y coordinate that frame number is n with following mathematical expression 2.YVS is the summation that second of each y axis detector (the17th detector 317～the 25th detector 325) detects data, and YVSG detects the y coordinate that multiply by corresponding surveyed area on the data in second of each y axis detector, has carried out the summation of the detection data of weighting.

[mathematical expression 2]

$\mathrm{YG}\left(n\right)=\frac{\mathrm{YVSG}\left(n\right)}{\mathrm{YVS}\left(n\right)}=\frac{\underset{i=-4}{\overset{4}{\mathrm{\Σ}}}j\×y(n,j)}{\underset{i=-4}{\overset{4}{\mathrm{\Σ}}}y(n,j)}---\left(2\right)$

In the present embodiment, the project YG of Figure 18 becomes 2.5 when frame number 0.The y coordinate of the center of gravity of this expression hand is 2.5.For other frame, the value of project YG is also represented the y coordinate of the center of gravity of the hand in this frame.In the present embodiment, the value of project YG is the value (except frame number 11) of 0～2.5 scope, the moving up and down of the hand on the change denotation coordination of the value of project YG.

Analyze the change of center of gravity YG in the present embodiment, the action of hand is discerned as operation information.Figure 19 is a sequential chart of having represented the coordinate change of the heavy-handed heart along the process of time.The change of the y coordinate of the heavy-handed heart of Figure 19 (A) expression, i.e. the change of the value of the project YG of Figure 18 is illustrated in 0～2.5 scope interior span and four cycles and have fluctuation.The change of the x coordinate of the heavy-handed heart of Figure 19 (B) expression, i.e. the change of the value of the project XG of Figure 18.As illustrated in fig. 17 with x coordinate 5 as center of gravity with hand in longitudinal oscillation, do not change in the horizontal, so from the principle, shown in Figure 19 (B), become the straight line of certain level.

Though should analyze the waveform of this x and two axles of y, explanation is to the protection of wrong identification earlier.Ideal data when theperiod 1 of table shown in Figure 180 becomes the hand longitudinal oscillation.The data of project x (i) of having extracted the x coordinate beyond the x coordinate 4,5 and 6 of hand out are 0.For the y coordinate also is same, and the surveyed area data in addition of having extracted hand out are 0.But, in fact, promptly use the firstobject extractor displacer 51 to carry out various Filtering Processing, also have omission, therefore can imagine the situation of the data (noise) of the action beyong contemplation in addition that produces hand.

In theframe number 11 of second round, except the data relevant with hand, also have second to detect data, these second detection data are illustrated in each surveyed area, detect the zone of the hand (object) that is equivalent tozone 100 by the detector of corresponding x (1), detect the zone of the hand (object) that is equivalent tozone 50 by the detector of corresponding y (2), detect the zone of the hand (object) that is equivalent tozone 50 by the detector of corresponding y (3).These data may upset the barycentric coodinates of detected hand.Shown in Figure 17 (A)～(D), the x coordinate of the heavy-handed heart is 5 and constant, but the value of the project XG offrame number 11 illustrates 3.361.In addition, in the y coordinate of the center of gravity of the hand offrame number 11, withframe number 3 similarly the value of project YG should be 0, but become-1.02, x axle and y axle all become value affected by noise.

If noise is occurrent, can utilize the isolated point that often uses in Digital Signal Processing to remove filter (median filter) suppresses, but, become the main cause that discrimination reduces if can pass under the bigger situation of this filter or noise number and amount.

In the present embodiment, in order to suppress this noise effectively, close the processing of thetime grate device 52 of unnecessary detector.In table shown in Figure 180, controlinformation determining device 20 is confirmed in specified time limit the additive value that the second detection data add up have been surpassed certain value (if the x axis detector is threshold value th1x; If the y axis detector is threshold value th1y) detector, promptly represent peaked detector, wherein, this second detects data based on the output from each detector ofx axis detector 301～316,y axis detector 317～325.

Shown in the table of Figure 18, detect data (output signal) change based on detecting second of data from first ofy axis detector 317～325 outputs, there is not detector above threshold value th1y.On the other hand, detect the data based on detecting second of data from first ofx axis detector 301～316 outputs, based on detecting data x (5) expression maximum from the 2nd of the output of the 14th detector 314 of corresponding x coordinate 5, the end value that at a time adds up surpasses threshold value th1x, so be judged as being the relevant detection device.Thus, the action of distinguishing hand is the action that swings up and down.In addition,, will detect data, and only be expressed as second of regulation detector and detect data based on detecting second of data from first of regulation detector output for easy later on.

Figure 19 (C) expression second of 14th detector 314 corresponding with x coordinate 5 that add up detects the process of data x (5).Surpassed moment (frame 9) of threshold value th1x at the additive value that adds up, activation marker Flg_x becomes 1 of specified time limit from 0.When additive value surpassed threshold value th1x, controlinformation determining device 20 as a token of maker generated activation marker Flg_x.Activation marker Flg_x become 1 during, do not detect unnecessary interval or the hand in the surveyed area (object) as described later.At this, the additive value that adds up has surpassed threshold value th1x atframe 9, still, as long as surpass threshold value th1x in specified time limit.

The specified time limit that activation marker Flg_x is raise is as between active period, its length be set to identification manually do four required cycles during.Describe in the back about Figure 19 (D).

Figure 21 is used to illustrate that the surveyed area that will be positioned at which position on the picture is as effectively scheming.When in the present embodiment, surveyed area (interval) being used to detect the action of hand of placement as effectively.The image, the noise contribution of representing with black surround of the hand of longitudinal oscillation on x coordinate 5 of having described thatvideo camera 2 photographs at Figure 21 and the commutator pulse of supplying with for control the 21st detector 321.

The x axle commutator pulse that the x direction of principal axis is described with the single-point setting-out be have with effective image during the pulse of the suitable width of the width of horizontal direction, in the moment thatuser 3 begins to swing hand, supply with whole y axis detector (the17th detector 317～the 25th detector 325).

When generating activation marker Flg_x (becoming 1) based on additive value, the 2nd x axle commutator pulse that generation is described with solid line, this additive value are to add up and the value that obtains in the output signal of detector that specified time limit will be corresponding with the surveyed area of having extracted the hand of swinging out.The 2nd x axle commutator pulse be have with effective image during the pulse of the suitable width of the Rack of horizontal direction, supply with wholey axis detector 317～325.Eachy axis detector 317～325 is only exported the detection signal of the essential surveyed area in order to detect the hand that is placed on surveyed area Min. ground based on the 2nd x axle commutator pulse.

Use Figure 22 that the generation method of the 2nd x axle commutator pulse is described.The initial x axle commutator pulse of supplying with eachy axis detector 317～325 is an x axle commutator pulse.The one x axle commutator pulse is that the axial whole width of x with each surveyed area corresponding with eachy axis detector 317～325 are made as effective commutator pulse.

When moving when the surveyed area of x coordinate 5 has been extracted hand out by hand shown in Figure 21, as previously mentioned, in the present embodiment, second of 14th detector 314 corresponding with the coordinate 5 of x axle detects data, detect data with second of other detector and compare, get maximum (with reference to Figure 18) continuously.When the value that detects data when second of the 14th detector 314 that added up surpasses threshold value th1x, generate activation marker Flg_x (becoming 1).Controlinformation determining device 20 confirms to have generated activation marker Flg_x, and the x axle control data of the surveyed area of x coordinate 5 is made as 1.

In the present embodiment, consider according to the distance between television receiver 1 (video camera 2) and theuser 3, the situation that the size of the hand on the picture also changes a little will comprise the surveyed area corresponding with the detector that has generated activation marker Flg_x at least and be made as 1 with near the x axle control data of the surveyed area of the surveyed area of this surveyed area adjacency.For example, the x axle control data of x coordinate 4 and 6 surveyed area is made as 1.In addition, the x axle control data of surveyed area in addition is made as 0.

Control information controller 20 is supplied with aforesaid x axle control data totiming pulse generator 12; X axle commutator pulse active controller 12x in thetiming pulse generator 12 generates the 2nd x axle commutator pulse based on the x axle control data of importing, and supplies with wholey axis detector 317～325.Therefore, if state shown in Figure 21, generate that to have with the x coordinate be the 2nd x axle commutator pulse of the suitable width of the width of 4 to 6 surveyed area.That is,timing pulse generator 12 generations are with the 2nd x axle commutator pulse of the narrowed width of an x axle commutator pulse.Being supplied to eachy axis detector 317～325 of the 2nd x axle commutator pulse, only is 4 to 6 interval output detection signal from the x coordinate of each surveyed area of correspondence.Its result, do not detect coordinate shown in Figure 21 (x, y)=(1 ,-2), (1 ,-3) noise contribution of taking place.

When having generated the 2nd x axle commutator pulse, controlinformation determining device 20 carries out control after this based on the output from eachy axis detector 317～325.Reference is not by each xaxis detector 301～316 detected detection signal.In addition, can not supply with commutator pulse yet, just stop the output of detection signal to thetime grate device 52 of eachx axis detector 301～316.

Figure 23 represents the17th detector 317～the 2nd x axle commutator pulse that the25th detector 325 is supplied with to the y axis detector and the commutator pulse (y direction of principal axis) that is used for eachy axis detector 317～325 pairing surveyed area.317～325 outputs of each y axis detector from the surveyed area of correspondence and with three detected signals in interval that overlap based on the corresponding surveyed area of the x coordinate 4～6 of the 2nd x axle commutator pulse.By doing like this, in surveyed area, can not detect do not extract out hand, do not need the interval detected.

In addition, adopted in the present embodiment, still, also can adopt the starting point of specific pulse and the method for pulse duration etc., flexibly the circuit methods of control impuls width as Figure 22 and method shown in Figure 23 with surveyed area unit's control impuls width.

Table shown in Figure 24 is the content roughly the same with table shown in Figure 180, become after 1 the 2nd x axle commutator pulse that generates but show utilization at the activation marker Flg_x of the 14th detector 314 shown in Figure 19 (C), and restriction never need to detect interval or from the detection that detects the zone obtain, based on detecting data from second of the output signal of eachdetector 301～325.In Figure 19 (C), surpassed threshold value th1x,frame number 10 is later second detects data and be equivalent to this, x (1), y (3) and the y (2) of theframe number 11 that exists as noise contribution in the table of Figure 18 become 0.This is because (x y)=(1 ,-2), the interval of (1 ,-3), supplies with the 2nd x axle commutator pulse by the18th detector 318 that is provided with to correspondence and eachtime grate device 52 of the19th detector 319, and is not detected at coordinate.

By the removal of noise contribution, eliminated the disorder of the value of center of gravity XG and YG, improved the discrimination of the 1st motion detector 20-1～the 5th motion detector 20-5 of back grade of each y axisdetector 317～325.In addition, in the present embodiment, being subjected to the influence of noise contribution up toframe 9, still, up to this constantly, is to be purpose so that activation marker Flg_x is put 1, and the noise contribution that makes the maximum of the additive value that adds up not change this degree does not exert an influence to detecting.

The 1st motion detector 20-1 in the controlinformation determining device 20～the 5th motion detector 20-5 accepts and handles data shown in Figure 24.Return Figure 19, illustrate to be used to detect the processing how hand moves.

The change of the y coordinate YG of Figure 19 (A) expression center of gravity, the change of the x coordinate XG of Figure 19 (B) expression center of gravity, expression does not have the waveform of noise respectively.Become moment more than the threshold value th1x in the value that the output signal of the x axis detector shown in Figure 19 (C) (the 14th detector 314) has been carried out adding up, activation marker Flg_x becomes 1.Near axial surveyed area of x and the axial surveyed area of each y intersect and the axial interval of each y beyond a plurality of intervals of forming, be disabled by the 2nd x axle commutator pulse of supplying with eachy axis detector 317～325, near the axial surveyed area of x this comprises at least with the surveyed area adjacent surveyed area corresponding with the detector that has generated activation marker Flg_x.That is, be not used in the detection of hand.Therefore, be not subjected to The noise.

If the waveform of Figure 19 (C) becomes more than the threshold value th1x constantly, then the 2nd x axle commutator pulse is supplied with eachy axis detector 317～325 constantly, therefore, continue unwanted interval and be invalid during, continue not to be subjected to the effect of The noise.If the waveform of Figure 19 (C) becomes below the threshold value th1x, then accumulated value is reset.Wherein, the value that becomes the benchmark that resets is not limited to threshold value th1x.

Then, suppress the processing of DC biasing of the waveform of Figure 19 (A), make the mean value of waveform roughly become 0.During handling, this uses high pass filter shown in Figure 20.

Thedelayer 81 of Figure 20 carries out the delay of 4 frames (time T m) in the present embodiment.Subtracter 82 is asked the signal of delay and the difference between the undelayed signal.At this, symbol is unimportant, does not influence final result.At last, carry out the adjustment of ratio with 1/2 multiplier 83.The waveform of Figure 19 (A) is by the high pass filter of Figure 20, and its result is shown in Figure 19 (D), and the mean value of waveform roughly becomes 0.Thus, get rid of the positional information of hand on the y axle of swing, can obtain being suitable for manually doing the waveform of the analysis of content.In addition, the center of gravity YGH that represents at the longitudinal axis of Figure 19 (D) has carried out the value that high-pass filtering is handled to the center of gravity YG shown in the longitudinal axis of Figure 19 (A).

Then return Figure 15, the 1st motion detector 20-1～the 5th motion detector 20-5 is described.The 1st motion detector 20-1～the 5th motion detector 20-5 has not shown cross-correlation digital filter.In the present embodiment, in order to discern operation information by manual work, will be on hand down or about the highest swing four times just passable.Promptly, determined to discern what kind of action in advance, therefore, the cross-correlation digital filter is obtained the representational detection signal waveform of a certain compulsory exercise (longitudinal oscillation action) that is predetermined and based on the cross-correlation of the detection signal waveform that generates at the 1st motion detector 20-1～the 5th motion detector 20-5 from the detection signal that is produced by actual act of eachdetector 301～325 outputs, by estimating its consistency, the operation information that identification is manually done.

In the present embodiment, with the reference signal waveform (the representational detection signal waveform of compulsory exercise) of the waveform shown in Figure 25 (G) as the pitch action, in the tap coefficients value of the k0～k40 of the cross-correlation digital filter shown in Figure 25 (F), use the value corresponding with this reference signal waveform.In addition, illustrated among Figure 25 (D) and be input to detection signal waveform cross-correlation digital filter kn, that produce by actual act, but this is the identical waveform of signal waveform with Figure 19 (D).The cross-correlation digital filter multiplies each other with tap coefficient with based on second detection signal of the signal that detects actual act, the 1st motion detector 20-1～the 5th motion detector 20-5 is based on the signal waveform from the output of cross-correlation digital filter, and whether the action thatdetection user 3 carries out is the longitudinal oscillation action.Utilize followingmathematical expression 3, ask the output signal wv (n) of cross-correlation digital filter kn.

[mathematical expression 3]

$\mathrm{wv}\left(n\right)=\underset{i=0}{\overset{N-1}{\mathrm{\Σ}}}y(n+i)k\left(i\right)---\left(3\right)$

N is the tap number of digital filter, is 41 taps (0～40) at this.Y (n+i) is the center of gravity YGH through Filtering Processing that represents on the longitudinal axis of Figure 25 (D).Cross-correlation digital filter kn realizes its function by only becoming action in 1 o'clock at activation marker Flg_x.

Cross-correlation digital filter output signal wv (n) becomes the waveform shown in Figure 25 (E), and when the consistency of cross-correlation increased, it is big that amplitude also becomes.In addition, Figure 26 (D) is identical with Figure 19 (D) and Figure 25 (D), as representing with the comparative control of Figure 26 (E).The absolute value of getting output signal wv (n) carries out cumulative integral, reaches threshold value th2v when above in this value, and the cross-correlation that is judged as with reference signal waveform is fully, is identified as and has made specified action (is the longitudinal oscillation action at this).The 1st motion detector 20-1～the 5th motion detector 20-5 is based on from the detection signal of test section 19 outputs, and whether the action that detectsuser 3 is the motion detector of compulsory exercise.

In this action of identification, confirm that expression is that longitudinal oscillation action and the activation marker Flg_x that undertakes the effect of protection window are " 1 ", determined the longitudinal oscillation operation of hand, carry out the incident corresponding (control) with the state of television receiver 1.This incident is according to control information generator 20-10 any of a plurality of motion detector 20-1～20-5 shown in Figure 15 to be become the signal that definite situation carries out exporting after the logic determines to carry out.

Then, action with hand teeter (good-by) is described.In the present embodiment, automatically distinguish the action of vertical and horizontal, can carry out simultaneously.Figure 27 be in the zone ofvideo camera 2photography user 3 with hand laterally (about) example of the image of hand when mobile, that photograph by video camera 2.The xy coordinate of the surveyed area that shows the arrow of expression hand moving direction simultaneously and in picture, dispose.Extract four positions of the hand that is moving out represents at Figure 27 (A), (B), (C), (D).Figure 27 (A) expression hand is positioned at the most left situation, the situation that Figure 27 (B) expression moves right hand a little, and the situation that Figure 27 (C) expression further moves right hand, Figure 27 (D) expression hand is positioned at the rightest situation.

In the present embodiment, with the hand move left and right four times.That is, (A), (B) of Figure 27, (C), (D), (D), (C), (B), (A) as one-period, four cycles have been moved with hand.Carry out under the situation of such side-to-side movement, at the y direction of principal axis, hand moves hardly, is positioned on the identical coordinate.On the other hand, at the x direction of principal axis, change about the coordinate of hand.Therefore, detected data have become repetition four cycles of left and right sides peak value, show as the change value of the dateout of each detector output that is provided with from the surveyed area of corresponding each coordinate.

Figure 28 is with the data value and the content of having handled this data value of eachhistogram detector 61 output of eachdetector 301～325 in the testing result of the side-to-side movement of hand shown in Figure 27, tabulates and represents.This table is to make the side-to-side movement of the corresponding hand of data value with the epiphase form together of Figure 18.

In the example shown in Figure 27 (A)～(D), hand moves at left and right directions, and at the y direction of principal axis, the position of mobile hand does not change, so the data of project y (j) (j=-4～+ 4) do not change.Shown in Figure 27 (A)～(D), hand for being centered close on the y coordinate 1～3, shows the value that detected hand at project y (1), y (2) and the y (3) of Figure 28 with y coordinate 2.Other project y (j) is by the shielding of the firstobject extractor displacer 51, and its value becomes 0 (except project x (7), the x (4) offrame number 11, y (1)).

At the x direction of principal axis, so the hand move left and right is the data movement of project x (i).In Figure 27 (A), hand is positioned on the x coordinate-6 ,-5 ,-4, in the hurdle of theframe number 0 of project x (6), the x (5) of Figure 28 and x (4) value that detects has been shown.Similarly, reach (D), in the project x (i) corresponding, each value that detects has been shown with the x coordinate of each hand placement for Figure 27 (B), (C).

With above-mentionedmathematical expression 1, can obtain frame number and be the center of gravity XG of the hand on the x coordinate of n.In the present embodiment, the project XG of Figure 28 is-5.3 when frame number 0.The x coordinate of the center of gravity of this expression hand is-5.3.For other frame, the x coordinate of the center of gravity of the hand in this frame of the value representation of project XG.In the present embodiment, the value of project XG is the value (except frame number 11) of-5.3～-2.3 scope, and the side-to-side movement of the hand on the coordinate has been represented in the change of the value of project XG.

With above-mentionedmathematical expression 2, can obtain frame number and be the center of gravity YG of the hand on the y coordinate of n.In the present embodiment, the project YG of Figure 28 becomes 2.19 (except frame numbers 11) in nearly all frame, and therefore, the y coordinate of the center of gravity of hand is 2.19, increases with 2.19 centre datas of y coordinate.

Figure 29 is the sequential chart of change of having represented the barycentric coodinates of hand along the process of time.The change of the y coordinate of the center of gravity of Figure 29 (A) expression hand, be the change of the project YG value of Figure 28, since as shown in figure 27 with y coordinate 2.19 as the teeter of center of gravity handle, therefore in vertically not change, shown in Figure 29 (A), become the straight line of certain level from the principle.The change of the x coordinate of the center of gravity of Figure 29 (B) expression hand, be the change of the project XG value of Figure 28, on four cycles, fluctuation arranged between being illustrated in-5.3～-2.3.

Though should analyze the waveform of this x and two axles of y, the ideal data the whenperiod 1 of table shown in Figure 28 becomes the hand teeter.The data of project y (j) of having extracted the y coordinate beyond the y axial coordinate 1,2 and 3 of hand out are 0.For the x coordinate too, the data of having extracted out beyond the surveyed area of hand are 0.

But, in theframe number 11 of second round, except the data relevant with hand, also have second to detect data, these second detection data are illustrated in each surveyed area, detect the zone of the hand (object) that is equivalent tozone 120 by the detector of corresponding y (1), detect the zone of the hand (object) that is equivalent tozone 50, detect the zone of the hand (object) that is equivalent tozone 70 by the detector of corresponding x (7) by the detector of corresponding x (4).These data can upset the barycentric coodinates of detected hand.Shown in Figure 28 (A)～(D), though the y coordinate of the center of gravity of hand is 2.19 and fix, the value representation of the project YG offrame number 11 is 1.351.In addition, for the x coordinate of the center of gravity of the hand offrame number 11, should withframe number 3 in the same manner, the value of project XG should be-2.3, but becomes-0.45, becomes the value that x axle and y axle all are subjected to The noise.

During with the hand teeter,, close the processing of thetime grate device 52 of unnecessary detector also with the situation of hand longitudinal oscillation is identical.In table shown in Figure 28, confirm specified time limit second additive value that detects data that added up surpass certain value first (if the x axis detector then is threshold value th1x; If the y axis detector then is th1y) detector, promptly represent peaked detector, this second detects data based on detecting data from first of each detector output ofx axis detector 301～316,y axis detector 317～325.

Shown in the table of Figure 28, inx axis detector 301～316, do not exist second to detect the detector that data (output signal) change and exceed threshold value th1x.On the other hand, iny axis detector 317～325, second of 23rd detector corresponding with y coordinate 2 detects data (y (2)) expression maximum, and the additive value that at a time adds up surpasses threshold value th1y, is judged as the relevant detection device.Thus, the action of having distinguished hand is the action that swings.

Figure 29 (C) second of 23rd detector 323 corresponding with y coordinate 2 of having represented to add up detects the process of data y (2).Surpassed moment (frame 9) of threshold value th1y at the additive value that adds up, activation marker Flg_y becomes 1 of specified time limit from 0.When additive value surpassed threshold value th1y, controlinformation determining device 20 as a token of maker generated activation marker Flg_y.Activation marker Flg_y become 1 during, do not detect unnecessary interval or the hand in the surveyed area as described later.In addition, in the figure, the additive value that adds up has surpassed threshold value th1x atframe 9, still, as long as surpass threshold value th1x in specified time limit.

The specified time limit that activation marker Flg_y is raise is as between active period, its length be set to about four required cycles of the action of identification hand during.Describe in the back about Figure 29 (D).

Figure 30 is used to illustrate that the surveyed area that will be positioned at which position on the picture is as effectively scheming.In Figure 30, described the images ofvideo camera 2 hands that take, that on y coordinate 2.19, laterally move, 2 noise components representing with black surround, and in order to control the commutator pulse that the 6th detector 306 is supplied with.A y axle commutator pulse of describing with chain-dotted line at the y direction of principal axis, be have with effective image during the pulse of the suitable width of the width of vertical direction, in the moment thatuser 3 begins to swing hand, supply with to all x axis detector (the 1st detector～the 16th detector).

When having generated activation marker Flg_y (becoming 1) based on additive value, generate the 2nd y axle commutator pulse of describing with solid line, this additive value is the value that will add up with the output signal of having extracted out at the corresponding detector of the surveyed area of the hand of swing in specified time limit.The 2nd y axle commutator pulse be have with effective image during the pulse of the suitable width of the Rack of vertical direction, supply with to all x axis detector.Eachx axis detector 301～316 is based on the 2nd y axle commutator pulse, only exports in order to detect the hand that is placed on surveyed area and the detection signal of essential minimal surveyed area.

Utilize Figure 31 that the generation method of the 2nd y axle commutator pulse is described.The initial y axle commutator pulse of supplying with eachy axis detector 301～316 is a y axle commutator pulse.The one y axle commutator pulse is that axial whole width of y with each surveyed area corresponding with eachx axis detector 301～316 are as effectively pulse.

When moving when the surveyed area of y coordinate 2 has been extracted hand out by hand shown in Figure 30, as previously mentioned, in the present embodiment, detect data with second of other detector and compare, second of 23rd detector 323 corresponding with y axial coordinate 2 detects data and gets maximum (with reference to Figure 28) continuously.When the value that detects data when second of the 23rd detector 323 that added up surpasses threshold value th1y, generate activation marker Flg_y (becoming 1).Controlinformation determining device 20 confirms to have generated activation marker Flg_y, and the y axle control data of the surveyed area of y coordinate 2 is made as 1.

In the present embodiment, the size of considering the hand on the picture changes a little with the distance between television receiver 1 (video camera 2) and theuser 3, will comprise the surveyed area corresponding with the detector that has generated activation marker Flg_y at least and be made as 1 with near the y axle control data of the surveyed area of the surveyed area of this surveyed area adjacency.For example, the y axle control data with the surveyed area of y coordinate 1 and 3 is made as 1.In addition, the y axle control data with other surveyed area is made as 0.

Controlinformation determining device 20 is supplied with aforesaid y axle control data totiming pulse generator 12; Y axle commutator pulse active controller 12y in thetiming pulse generator 12 generates the 2nd y axle commutator pulse based on the y axle control data of input, supplies with to allx axis detector 301～316.Therefore, if state shown in Figure 30, generate then that to have with the y coordinate be the 2nd y axle commutator pulse of the suitable width of the width of 1 to 3 surveyed area.That is,timing pulse generator 12 generations are with the 2nd y axle commutator pulse of the narrowed width of a y axle commutator pulse.Being supplied to eachx axis detector 301～316 of the 2nd y axle commutator pulse, only is 1 to 3 interval output detection signal from the y coordinate of each surveyed area of correspondence.Its result, can not detect coordinate shown in Figure 30 (x, y)=(4 ,-1), (7 ,-1) noise component of taking place.

When generating the 2nd y axle commutator pulse, controlinformation determining device 20 carries out its later control based on the output from eachx axis detector 301～316.Reference is not from eachy axis detector 317～325 detected detection signal.In addition, can not supply with commutator pulse yet, and stop the output of detection signal to thetime grate device 52 of each y axisdetector 317～325.

Figure 32 represent to supply with the x axis detector the1st detector 301～the 16th detector 316 the 2nd y axle commutator pulse and be used for the commutator pulse (x direction of principal axis) of eachx axis detector 301～316 pairing surveyed area.Eachx axis detector 301～316 is only exported from the surveyed area of correspondence with three detected signals in interval that coincide based on the corresponding surveyed area of the y coordinate 1～3 of the 2nd y axle commutator pulse and is got final product.By doing like this, can not detect and to extract hand out and the interval that do not need to detect in the surveyed area.

In addition, in the present embodiment, adopted, still, also can adopt the starting point of specific pulse and the method for pulse duration etc., flexibly the circuit methods of control impuls width as Figure 31 and the method with surveyed area unit's control impuls width shown in Figure 32.

The content of table shown in Figure 33 and table shown in Figure 28 is roughly the same, become after 1 the 2nd y axle commutator pulse that generates but show utilization at the activation marker Flg_y of the 23rd detector 323 shown in Figure 29 (C), to the detection of the interval that never needs to detect or surveyed area limit and obtain detect data from second of eachdetector 301～325.

Theframe number 10 second later detection data that surpassed threshold value th1y in Figure 29 (C) belong to this, and x (4), x (7), the y (1) of theframe number 11 that exists as noise contribution in the table of Figure 28 become 0.This be because, owing to supply with the 2nd y axle commutator pulse to eachtime grate device 52 of the13rd detector 13 that is provided with accordingly and the 16th detector 316, thus coordinate (x, y)=(4 ,-1), the interval of (7 ,-1) is not detected.By the removal of noise component, eliminated the confusion of the value of center of gravity XG and YG, improve the discrimination of the 1st motion detector 20-1～the 5th motion detector 20-5 of back grade of eachx axis detector 301～316.

The 1st motion detector 20-1 in the controlinformation determining device 20～the 5th motion detector 20-5 receives and handles data shown in Figure 33.Return Figure 29, illustrate to be used to detect the processing how hand moves.

The change of the y coordinate YG of Figure 29 (A) expression center of gravity, the change of the x coordinate XG of Figure 29 (B) expression center of gravity shows the waveform that does not have noise respectively.Become moment more than the threshold value th1y in the value that the output signal of the y axis detector shown in Figure 29 (C) (the 23rd detector 323) has been carried out adding up, activation marker Flg_y becomes 1.Near axial surveyed area of y and the axial surveyed area of each x intersect and the axial interval of each x beyond a plurality of intervals of forming, be disabled by the 2nd y axle commutator pulse of supplying with eachx axis detector 301～316, near the axial surveyed area of y this comprises the surveyed area with the surveyed area adjacency corresponding with the detector that has generated activation marker Flg_y at least.That is, be not used in the detection of hand.Therefore, be not subjected to The noise.

If the waveform of Figure 29 (C) is more than the threshold value th1y constantly, then the 2nd y axle commutator pulse is supplied with eachx axis detector 301～316 constantly, therefore also continuing unnecessary interval and be invalid during, continuing not to be subjected to the effect of The noise.If the waveform of Figure 29 (C) becomes below the threshold value th1y, then accumulated value is reset.But, become the value of the benchmark that resets, be not limited to threshold value th1y.

Then, suppress the processing of DC biasing of the waveform of Figure 29 (B), make the mean value of waveform roughly become 0.During handling, this uses high pass filter shown in Figure 20.

The waveform of Figure 29 (B) is by the high pass filter of Figure 20, its result, and the mean value of waveform roughly becomes 0 shown in Figure 29 (D).Thus, the positional information on the x axle of hand swing is excluded, and obtains being suitable for analyzing the waveform of manually making content.In addition, the center of gravity XGH that represents on the longitudinal axis of Figure 29 (D) is that the center of gravity XG that represents on the longitudinal axis to Figure 29 (B) has carried out the value that high-pass filtering is handled.

Movement content for the hand of analyzing teeter, same with the action of hand longitudinal oscillation, cross-correlation between the detection signal waveform that the actual act that obtains the representational detection signal waveform of the compulsory exercise (teeter action) of decision in advance and export from eachdetector 301～325 produces is estimated consistency.

In the present embodiment, with the reference signal waveform (the representational detection signal waveform of compulsory exercise) of the waveform shown in Figure 34 (G) as the teeter action, in the tap coefficients value of the cross-correlation digital filter k0～k40 shown in Figure 34 (F), use the value corresponding with this reference signal waveform.In addition, the actual detection signal waveform that is input to cross-correlation digital filter kn has been shown in Figure 34 (D), but this is the identical waveform of signal waveform with Figure 29 (D).The cross-correlation digital filter multiplies each other with tap coefficient with based on second detection signal of the signal that detects actual act, the 1st motion detector 20-1～the 5th motion detector 20-5 is based on the signal waveform from cross-correlation digital filter output, and detecting the action thatuser 3 carries out is the teeter action.Obtain the output signal wh (n) of cross-correlation digital filter kn with followingmathematical expression 4.

[mathematical expression 4]

$\mathrm{wh}\left(n\right)=\underset{i=0}{\overset{N-1}{\mathrm{\Σ}}}x(n+i)k\left(i\right)---\left(4\right)$

N is the tap number of digital filter, is 41 taps (0～40) at this.X (n+i) is the center of gravity XGH that has carried out the Filtering Processing shown in the longitudinal axis of Figure 34 (D).Only become action in 1 o'clock by cross-correlation digital filter kn, realize its function at activation marker Flg_y.

In addition, used cross-correlation digital filter in the present embodiment and had and moved the cross-correlation digital filter of corresponding tap coefficient with teeter with tap coefficient corresponding with longitudinal oscillation action, but, also can switch to a cross-correlation digital filter according to action and supply with in storage such as controlinformation determining device 20 grades corresponding to the tap coefficient of longitudinal oscillation action and the tap coefficient that moves corresponding to teeter.Wherein, be under the situation of same action in longitudinal oscillation action and teeter action, be made as identical tap coefficient and get final product.

The responsiveness and the frame number of hand then, are described.For the responsiveness of hand and the relation of frame number, and though the swing mode of hand be vertically (up and down) still horizontal (about) all identical.

In the present embodiment, with 1 second as 60 frames, with explanation and accompanying drawing simply turn to purpose, with hand up and down or 4 times the action of swinging be made as 32 frames.The tap coefficient of correlation computations also diminishes.

But, if 32 frames are scaled the time, then be about 0.5 second, be too fast as the action of the people in the reality.Can expect that actual manual work is slower, for example establishing 4 hands of swing needs 2 seconds, then needs 120 frames.In order to detect this action, in correlation computations, increase tap number and get final product, suitably adjust tap number according to the required time of action and get final product.

The cross-correlation digital filter output signal wh (n) relevant with the teeter of hand becomes the waveform shown in Figure 35 (E), increases the conforming while of cross-correlation, and it is big that amplitude becomes.In addition, Figure 35 (D) is and Figure 29 (D) and the identical waveform of Figure 34 (D), illustrates as the comparative control with Figure 35 (E).The absolute value of obtaining output signal wh (n) carries out cumulative integral, reaches threshold value th2h when above in this value, and the cross-correlation that is judged as with reference signal waveform is fully, is identified as and has carried out compulsory exercise.The 1st motion detector 20-1～the 5th motion detector 20-5 is based on from the detection signal of test section 19 outputs, and whether the action that detectsuser 3 is the motion detector of compulsory exercise.

In this action of identification, confirmed that expression is that teeter action and the activation marker Flg_y that bears the effect of protection window are " 1 ", be defined as the teeter operation of hand, carry out the incident corresponding with the state of television receiver 1.This incident is, a plurality of motion detector 20-1～20-5 shown in Figure 15 some become that signal that definite situation carries out exporting after the logic determines carries out according to control information generator 20-10.

Figure 36 is the flow chart of treatment step of the method for the expression action that detects the longitudinal oscillation of above-mentioned hand and teeter.At length recorded and narrated the processing in each step shown in the flow chart of Figure 36, so, illustrate at this what kind of function is each step realize in integral body, and illustrate that the longitudinal oscillation of hand and teeter action discerned bytelevision receiver 1 as operation information, and carried out the process of control (incident) content.

The action of the swing hand that flow chart shown in Figure 36 carries out according touser 3 is divided into that system is handled in longitudinal oscillation and teeter is handled 2 that are.When the X-axis of vertically handling system started, input detected data x (8)～x (7) based on detect data from first of eachx axis detector 301～316output 16 second.At first, in steps A 501, each frame added up second detect data x (8)～x (7) based on each of the output of eachx axis detector 301～316.

Then, enter steps A 502, judge whether the msx (i) (i=-8～+ 7) that respectively is worth that is added up is more than the threshold value th1x.When the answer of steps A 502 is "No", promptly during all less than threshold value th1x, return steps A 501 at all additive value msx (i), add up.In steps A 502 when answer is "yes", promptly some additive value msx (i) become threshold value th1x when above, enter next step A503.

More than threshold value th1x, promptly meaning is set about the action of longitudinal oscillation from the additive value msx (i) of some x axis detector, therefore, in steps A 503 activation marker Flg_x is made as 1 from 0, supplies with the 2nd x axle commutator pulse to eachx axis detector 301～316.Thus, control the output of eachx axis detector 301～316, the processing (shielding) of not extracting object (hand) out in unnecessary surveyed area or interval can improve noise immunity.

Handle system too for teeter, when Y-axis starts, input is carried out handling with longitudinal oscillation the identical processing of steps A 501～A503 of system based on 9 second detection data y (4)～y (4) of the output ofy axis detector 317～325 at step B501～B503.

Then, at step B502, when the value msy (j) (j=-4～+ 4) that adds up in some y axis detector reaches threshold value th1y when above, activation marker Flg_y becomes 1 from 0, and being identified as manual work is teeter.

In the present embodiment, become for 1 the moment at activation marker Flg_x or Flg_y some, another activation marker is suppressed.At steps A 504 or B504, carry out the judgement of activation marker.For example handle in the system, become moment of 1 to enter steps A 504 at activation marker Flg_x, and judge that teeter handles whether the activation marker Flg_y that is is 0 in longitudinal oscillation.

When the answer of steps A 504 is "Yes",, determine that then the longitudinal oscillation processing that is treated as after this is to enter steps A 505 if promptly activation marker Flg_y is 0.On the other hand, if the answer of steps A 504 is "No", i.e. teeter is handled system and is activated, activation marker Flg_y is 1, then enter steps A 509, will be used to carry out longitudinal oscillation and handle add up additive value msx (i) and the activation marker Flg_x of system and be reset to 0, return steps A 501.

In addition, handle in the system in teeter, Flg_y becomes for 1 the moment at step B503 activation marker, enters step B504, judges simultaneously whether the activation marker Flg_x of longitudinal oscillation processing system is 0.

When the answer of step B504 was "Yes", if promptly activation marker Flg_x is 0, the processing that then is defined as after this was that system is handled in teeter, enters step B505.On the other hand, if the answer of step B504 is a "No", i.e. longitudinal oscillation is handled system and is activated, activation marker Flg_x is 1, then enter step B509, will be used to carry out teeter and handle add up additive value msy (j) and the activation marker Flg_y of system and be reset to 0 respectively, return step B501.

In the answer of steps A 504 is the answer of "Yes" or step B504 when being "Yes", enters steps A 505 or B505 respectively, carries out y axle center of gravity calculation or x axle center of gravity calculation.Y axle center of gravity calculation or x axle center of gravity calculation are used above-mentionedmathematical expression 1 ormathematical expression 2, obtain the project YG of table shown in Figure 24 or the project XG of table shown in Figure 33.(XG, value YG) is carried out the cross-correlation digital filtering and is handled, and calculates cross-correlation digital filter output signal wv (n) or wh (n) to the center of gravity obtained in the cross-correlation calculation of steps A 506 or B506.

At steps A 507 or B507, cross-correlation digital filter output signal wv (n) or wh (n) are carried out adding up after the absolute value, calculate the swh that adds up of add up swv or the wh (n) of wv (n).

Then, at steps A 508 or B508, the value of judging the swv that adds up whether greater than the value of threshold value th2v or swh whether greater than threshold value th2h.When the answer of steps A 508 or B508 is "Yes", start longitudinal oscillation incident or teeter incident.In addition, steps A 504～A508 and B504～B508 being described at the same time, still, as mentioned above, is not to handle longitudinal oscillation simultaneously to handle system and teeter processing system, only carries out some processing.

In addition, consider easy understanding explanation, processing before Figure 36 handles the cross-correlation calculation of steps A 506 or B506 is separated into 2 systems, but estimate activation marker Flg_x or Flg_y distinguishes that detected action is longitudinal oscillation or teeter at steps A 504 or B504, therefore processing can be made a system.In addition, steps A 504 or B504, and the judgement of steps A 508 or B508 in answer when being "No", enter steps A 509 or B509, accumulated value msx (i) and Flg_x or accumulated value msy (j) and Flg_y are reset to 0 respectively, and turn back to the moment of beginning.

Like this, in the present embodiment, handle the operation of hand longitudinal oscillation and the operation of hand teeter simultaneously, difference is discerned.As this application examples, if the action of hand longitudinal oscillation shown in Figure 3 " come, come ",, for example be that power supply is opened or started menu screen as the incident corresponding (control) with it.In addition, if the action of " good-by " of hand teeter then can be applicable to powered-down.

In addition, also can at this moment, omit steps A 504 or B504 and get final product only a compulsory exercise in longitudinal oscillation action and the teeter action as the control electronic equipment.

In first execution mode of above explanation,, be located at surveyed area on the picture and be and carry out in thehorizontal direction 16 cutting apart, carry out 9 25 of totals cutting apart setting in vertical direction as Fig. 5, shown in Figure 6.The detector corresponding with each surveyed area is 25 from the1st detector 301 to the25th detectors 325, and first embodiment has the advantage that can realize with less hardware size.

On the other hand, if consider more senior and complicated identifying operation, can consider to use second execution mode of the surveyed area of following explanation.The situation that the algorithm of detailed description by the flowchart text of Figure 36 acts in second execution mode equally.In addition, for second execution mode, the part different with first execution mode only is described.

Figure 37 represents to be provided with second execution mode that adds up to 144 (16 * 9) surveyed areas, these 144 surveyed areas are will carry out in the horizontal direction from the picture of the image ofvideo camera 2output 16 cutting apart, carry out 9 in vertical direction and cut apart, and the zone of cutting apart in the horizontal direction and intersect in the zone that vertical direction is cut apart and form.Therefore, the detector that constitutes test section 19 also needs 144, and the data of importing to controlinformation determining device 20 also are 144.The1st detector 301 correspondence in Figure 37 is positioned at coordinate, and (x, y)=(8,4) surveyed area on is exported from this surveyed area detected first and is detected data.

What second execution mode illustrated be, to obtain from the output signal of each surveyed area extraction at each picture (each vertical cycle) is purpose, distribute each detector to each surveyed area,, handle with software to the data of controlinformation determining device 20 each surveyed area of input.In addition, each detector can be with necessary data bulk on the hardware configuration with the realization of getting off by buffer storage is set.

Figure 38 is illustrated in the picture of the image of the overlapping hand of having described the longitudinal oscillation of being photographed byvideo camera 2 on 144 surveyed areas.Surveyed area additional shadow line at the mobile generation frame difference by hand is represented (zone of hand has also added hacures).In the first embodiment, thehistogram detector 61 by characteristics of objectsData Detection portion 53 shown in Figure 7 etc. will add this hatched surveyed area and be transformed to data, via cpu bus to 20 outputs of control information determining device.

In second execution mode, also can be suitable for same structure, but be 144, become the traffic carrying capacity of big and bus, reduced data so consider hardware size from the data that each detector obtains.In addition, in order to compare explanation, the position of the action of the hand of Figure 38 is the position identical with first execution mode shown in Figure 17.

Figure 39 is the test section 19 of second execution mode and the block diagram of control information determining device 200.Constitute the1st detector 301 to the 144th detectors 444 of test section 19, the data of processing object transmit to the 6th motion detector 20-6 of control information determining device 200.The output of the firstobject extractor displacer 51 as shown in Figure 8, the signal of thespecial color filter 71 that is compound,gray scale delimiter 72,motion detection filter 75 is the output signals of having extracted object from the output image ofvideo camera 2 out.

Complex method can be considered various logical operations, but considers logic product here.The output ofobject gate 74 is, has only the surveyed area of the dashed area of Figure 38 to have gray scale, and other surveyed area is not extracted object out, and gray scale is made as 0 (black level).If the black level of thevideo camera 2 of this moment is more than 0 level.

Characteristics of objectsData Detection portion 530 hasblock counter 66 and piece quantizer 67.Can also havehistogram detector 61 andAPL detector 62 etc. as required.

The information of the surveyed area of the output signal that 67 pairs ofblock counter 66 and piece quantizers be that represent with hacures in the picture of Figure 38, obtained the firstobject extractor displacer 51 carries out one than exporting after the specialization.The device that blockcounter 66 is counted the surveyed area in the whole surveyed area, except the shielding level.By in the surveyed area ofblock counter 66 counting, from the output signal of the first object extractor displacer, 51 outputs, compare with threshold value in the setting ofpiece quantizer 67, bethreshold value output 1 when above atpiece quantizer 67, when following,export 0.

When for example having obtained the output signal from 51 outputs of the first object extractor displacer in 1/2 zone of whole surveyed area, setting threshold, and, if to the output signal ofpiece quantizer 67 inputs, then become and added hatched surveyed area output signal shown in Figure 40 from the surveyed area of the dashed area of Figure 38 according to this threshold value.That is, (x y) is two surveyedarea outputs 1 of (5,3), (5,2), other surveyedarea output 0 to the coordinate of surveyed area.

By such setting threshold, 67 that produce byblock counter 66 and piece quantizer, become 144 from the output of test section 19, can realize with Min..

In 144 data of controlinformation judging part 20, handle according to the recognizer of action as picture of variable storage (each vertical cycle).This content is shown in Figure 41.Project x (8) becomes the summation of the output of the whole detectors corresponding with the whole surveyed area of indulging (y axle) direction of each x axle respectively to x (7).For example, project x (0) is based on from first of each detector output and detects the additive value that second of data detect data, the corresponding surveyed area coordinate of this each detector (x, y) be (0,-4), (0 ,-3), (0 ,-2), (0,-1), (0,0), (0,1), (0,2), (0,3), the surveyed area of (0,4).Surveyed area is divided into 9 at the y direction of principal axis, so maximum gets 9.

Project y (4)～y (4) is the summation of the output of whole detectors corresponding with whole surveyed areas of horizontal (the x axle) of each y axle too, and maximum is 16.Thus, as the result of manual work shown in Figure 38, the change of center of gravity can come identification maneuver with identical algorithm process with shown in Figure 180 identical.

Compare the table of Figure 41 and the table of Figure 42, in the row of initial frame n=0, projects x of Figure 18 (6)=x (4)=12, x (5)=120, y (3)=y (2)=72 are equivalent to projects x (6)=x (4)=0, x (5)=2, the y (3)=y (2)=1 of Figure 41.

Figure 41 represents that ratio is different with Figure 18 through quantizing to be corrected for the value of two-value.But locative center of gravity is identical.Therefore, the 6th motion detector 20-6 of second execution mode can utilize with the 1st motion detector 20-1～algorithm that the 5th motion detector 20-5 is identical of first execution mode and discern the action of hand.The algorithm of the 6th motion detector 20-6 is the calculating and the action oftimeing closing gate 52 etc. not to the detector output commutator pulse corresponding with unwanted surveyed area of correlated digital filter of center of gravity calculation, themathematical expression 3 ofmathematical expression 1 andmathematical expression 2, and the flow chart that shows its algorithm is Figure 36.The 6th motion detector 20-6 is based on from the detection signal of test section 19 outputs, and whether the action that detectsuser 3 is the motion detector of compulsory exercise.

At this, processing each detector that will be corresponding with each surveyed area of second execution mode,timeing closing gate 52 is recorded and narrated in the back as shielding processing.

In second execution mode, the surveyed area of eachdetector 301～444 correspondence is equivalent to the interval that illustrates in first execution mode, so the method fortimeing closing gate 52 is identical with first execution mode, still, make the invalid method difference of unwanted surveyed area.

Figure 42 represents the action of swing repeatedly down on hand in the same manner with Figure 38.For this action recognition is operation, the firstobject extractor displacer 51 is carried out the function of the action that detects hand, but can sneak into unconscious action.For example, at surveyed area shown in Figure 42, (x, y)=(1 ,-2), (1 ,-3) noise represented with black circle has taken place.

In the table of Figure 41, in the frame of n=11, become project x (1)=2, y (2)=11, y (3)=1.This can disturb the center of gravity of x axle and y axle, counteracts when detecting the action of hand.This can influence the value of center of gravity, so detect among the present invention of action of hand in the change that utilizes center of gravity, becomes problem.

By to carrying out shielding processing, suppress and remove this noise at the surveyed area that detects beyond the surveyed area that moves up and down of hand.

Shielding processing is identical with first execution mode, and the value of projects x (8)～x (7) of x axle of adding up in specified time limit respectively when having surpassed threshold value th1x shown in Figure 19 (C), is put 1 with activation marker Flg_x and got final product.Therefore, in second execution mode, when the output summation of the whole detectors corresponding with whole surveyed areas of vertical (y axle) in each x coordinate or with the output summation of the corresponding whole detectors of whole surveyed areas of horizontal (x axle) in each y coordinate, when having surpassed threshold value th1x or th1y respectively, generate activation marker Flg_x or Flg_y and get final product.In addition, if the value that adds up has surpassed setting, also boundary can be set.

In Figure 19 (C), detect the output signal of each detector corresponding with each surveyed area with x axial coordinate 5 has been carried out the additive value that adds up, through having surpassed threshold value th1x specified time limit (frame 10).Therefore, detect the hand of swing at least a portion of each surveyed area with x coordinate 5.

After the signal from detector output surpasses threshold value th1x, during regulation, activation marker Flg_x is put 1, by correlation being estimated in the change of the center of gravity YG of vertical (the y axle) shown in Figure 19 (A) with the cross-correlation digital filter, with the action recognition of hand for operating.

In second execution mode, in horizontal and vertical cutting apart each surveyed area is set from the picture of the image ofvideo camera 2 output, and first of each surveyed area is detected data to be supplied with to controlinformation determining device 20, variable as two-dimensional arrangements is handled, therefore, by this variable is operating as 0, can realize shielding processing.Certainly, can also control the commutator pulse ofincoming timing gates 52 withtiming pulse generator 12.

In the present embodiment, the 10th frame has been implemented shielding processing later on, so can suppress the noise component (frame number 11) shown in the table of Figure 41.As mentioned above, shielding processing has the action beyond the inhibition hand, only finds out the effect of compulsory exercise.

The hatched part of Figure 42 is the surveyed area that has carried out shielding processing as described above.Table based on Figure 41, the detector corresponding with the surveyed area in addition of the surveyed area with x axial coordinate 5 carried out shielding processing to get final product, but consider the fixed situation of having light fingers in the present embodiment, be controlled to being that each corresponding detector of each surveyed area of 5 ± 1 does not carry out shielding processing, with regard to output detection signal with whole surveyed areas of x axial coordinate 5 and x axial coordinate.

Promptly, to becoming each corresponding detector of each surveyed area of 1 x coordinate 5 and each detector corresponding with each surveyed area of x coordinate 4 and 6 with activation marker Flg_x, supply with commutator pulses by timingpulse generator 12, x coordinate 4 and each surveyed area of 6 and each surveyed area adjacency of x coordinate 5.

In addition, according to the table of Figure 41, do not relate to moving up and down of hand, with the x axial coordinate be 4～6 and the y axial coordinate supply with commutator pulse for each corresponding detector of each surveyed area of-4 ,-3 ,-2,4, implement shielding processing.Through the surveyed area of shielding processing at Figure 42 with mark * expression.Thus, can further suppress The noise.

This shielding processing is, activation marker flg_x is put for 1 the moment in Figure 19 (C), estimates the center of gravity YG shown in the Figure 19 (A) before this moment and carries out.Center of gravity YG in the memory in control information determining device 20 (not shown) during the recording prescribed is in the moment that activation marker Flg_x has taken place, with reference to the center of gravity YG that writes down before this.In the present embodiment, with reference to the scope of representing witharrow 1 among Figure 19 (A).At the y axial coordinate is-4 ,-3 ,-2,4 surveyed area, can be judged as and not place hand, and the action of hand is to take place beyond the scope of-4 ,-3 ,-2,4 surveyed area at the y axial coordinate, carries out shielding processing as described above.

That is, when during mobile hand, determining the surveyed area of having extracted the hand of placing out in order to carry out compulsory exercise, pass through of becoming that detection signal passes through is regional.At surveyed area in addition, owing to do not supply with commutator pulse totime grate device 52, so, detected nonpassage of signal mistake.Moreover, the additive value of the output signal of each detector surpasses threshold value th1x if added up, then with reference to detecting data second of the specified time limit more Zao than the moment that surpasses threshold value th1x, the detector corresponding with the surveyed area of placing hand surveyed area in addition carried out shielding processing, make and suppress noise thus by not output detection signal.

Second execution mode is, to being located at each surveyed area from the picture of the image ofvideo camera 2 outputs, corresponding detector is set, and utilizes block surveyed area to detect the action of hand.Owing to can carry out the shielding processing on the two dimensional surface, can focus on the surveyed area of having extracted out at the hand of action by first execution mode, improve noise robustness.In addition, can carry out shielding processing with software, so can carry out and the processing of not implementing the data parallel of shielding processing, also having increases the advantage of handling the degree of freedom.

And, according to shown in Figure 36 second algorithm that detects the action of data identification hand, be irrelevant and similarly carry out its function with the execution mode of surveyed area, can determine to control television receiver after the identifying operation.

Figure 43 is the figure of the expression secondobject extractor displacer 510, and the secondobject extractor displacer 510 is another execution modes of the firstobject extractor displacer 51 shown in Figure 8.The compound signals fromspecial color filter 71 and 72 outputs of gray scale delimiter of the second object extractor displacer, 510 usefulness recombiners 73 in series disposemotion detection filter 75 in level thereafter, apply gating by 74 pairs of signals fromvideo camera 2 of object gate and handle.

In addition, in second execution mode, only count with theblock counter 66 of characteristics of objectsData Detection portion 530 at the surveyed area corresponding with the detector of having supplied with commutator pulse, so, the output of themotion detection filter 75 of Figure 41 directly is added in the block count of characteristics of objectsData Detection portion 530 of Figure 39, equally also can obtains the mobile message of hand of the output surveyed area unit ofpiece quantizer 67.

Figure 44 is the figure that is used to illustrate an embodiment who has used one embodiment of the present invention.Show the menu image of describing with graphic generator 16 (operation image) at Figure 44 (A), this image is divided into five zones of (1-1)～(1～5), anduser 3 carries out specified action to these five zones.Figure 44 (B) expression is by the user's 3 ofvideo camera 2 shootings the image of image behind mirror transformation.

Figure 44 (C) is illustrated in and mirrors the appearance of having mixed Figure 44 (A) and image (B) on thedisplay unit 23, has known menu image and user's 3 position relation.In the mechanism of this second execution mode,display unit 23 shown in Figure 2 andgraphic generator 16 are essential functional blocks.

Figure 45 shows the drawing of the menu screen of having seen the mixing shown in Figure 44 (C) and user's 3 mirror image, anduser 3 is operating the state of television receiver 1.Show at Figure 45 (A) thatuser 3 swing hands are selected the image of the menu content that expression is wished in a plurality of menu images, the state of the action button of promptlywishing.User 3 has for example selected the action button of " film " among Figure 45 (A).

As in the explanation of first execution mode,, just know which detector in the x axis detector is represented maximum, and activation marker Flg_x becomes " 1 " if at the longitudinal oscillation hand.Therefore, if handle has generated thegraphic generator 16 of menu image and the detector corresponding with the surveyed area of each coordinate is mapped, just can start the corresponding control of selecting withuser 3 of action button.

According to thetelevision receiver 1 shown in the above-mentioned the embodiments of the present invention, can realize following effect.When opening the power supply oftelevision receiver 1,, just can carry out the ON/OFF of power supply and the control that EFR STK shows if the action of hand is in the image pickup scope of video camera 2.The action with the vertical or horizontal swing of hand of this moment is the action of being not difficult to the people.In addition, vertically action expression " come, come ", crosswise movement is represented " good-by ", be significant action concerning the people, be used for the control oftelevision receiver 1, can realize being very easy to understand and operation easy to use according to the form of in these actions, giving its implication.

In addition,, can carry out motion detection, can further utilize the control of activation marker to carry out the few correct detection of wrong identification no matteruser 3 is positioned at any position of the image pickup scope of video camera 2.And, can also be applied on the picture of user self image that has mixed the menu image that generates bygraphic generator 16 andvideo camera 2 photographies, to select the method for operation of the menu of wishing, can realize various applying flexibly with identical circuit and software processes.

In above-mentioned the embodiments of the present invention,television receiver 1 is represented as the example of electronic equipment, but, can on other electronic equipment, be carriedvideo camera 2 and use not specific to this.In addition, for from the menu screen of the image thatmixed video camera 2 at the EFR STK image, the user selects and the method for the menu image of the control content that the operation expression is wished, so long as the electronic equipment with display (display unit) just, can use.The present invention provides useful device having constructed on the basis that does not have the structure that remote controller also can operating electronic equipment.

Claims (6)

1. the control method of an electronic equipment is characterized in that,

The operator is photographed, the part of this operator's health be put into picture is carried out in the horizontal direction N cut apart, on vertical direction is carried out a plurality of surveyed areas after M is cut apart, wherein, above-mentioned N, M are the integers more than 2;

According to the image of being photographed, detect first detection signal that uses the action that the part of health carries out based on the aforesaid operations person respectively from above-mentioned a plurality of surveyed areas;

Generate second detection signal according to above-mentioned first detection signal, this second detection signal represents to be put into the zone of a part of the health of the aforesaid operations person on above-mentioned a plurality of surveyed area.

2. the control method of electronic equipment as claimed in claim 1 is characterized in that,

Obtain the additive value of each above-mentioned second detection signal of having added up in specified time limit;

Use above-mentioned additive value surpass predetermined threshold surveyed area, and comprise at least and near second detection signal of the surveyed area of the surveyed area adjacent detect aforesaid operations person's action with above-mentioned surveyed area.

3. the control method of electronic equipment as claimed in claim 2 is characterized in that,

Obtain from surpassing the summation of detected second detection signal of a plurality of surveyed areas that the surveyed area of predetermined threshold intersects with above-mentioned additive value;

To first signal waveform of the action that is illustrated in the aforesaid operations person on above-mentioned a plurality of surveyed area of calculating, compare with the pairing secondary signal waveform of predefined action on above-mentioned surveyed area according to above-mentioned summation meter, when the consistent degree of above-mentioned first signal waveform and above-mentioned secondary signal waveform is high, judges and on above-mentioned surveyed area, carried out above-mentioned predefined action.

4. the control method of electronic equipment as claimed in claim 1 is characterized in that,

Obtain from the summation of above-mentioned second detection signal of first surveyed area group output and from the summation of above-mentioned second detection signal of second surveyed area group output, the above-mentioned first surveyed area group comprises the full surveyed area of vertical direction of each coordinate of horizontal direction, and the above-mentioned second surveyed area group comprises the full surveyed area of horizontal direction of each coordinate of vertical direction;

Obtain the additive value of each above-mentioned summation of having added up in specified time limit;

Use above-mentioned additive value surpass predetermined threshold surveyed area, and comprise near second detection signal of the surveyed area of the surveyed area adjacent at least with above-mentioned surveyed area, detect the action of a part of aforesaid operations person's health.

5. the control method of electronic equipment as claimed in claim 4 is characterized in that,

To representing that above-mentioned additive value surpasses first signal waveform of the action of the aforesaid operations person on the surveyed area group of predetermined threshold, compares with the pairing secondary signal waveform of predefined action on above-mentioned surveyed area, when the consistent degree of above-mentioned first signal waveform and above-mentioned secondary signal waveform is high, judges and on above-mentioned surveyed area, carried out above-mentioned predefined action.

6. an electronic equipment is characterized in that, comprising:

Video camera is photographed to the operator;

Test section, have with the picture of the image that above-mentioned video camera is exported and carry out a plurality of detectors that N is cut apart, carried out the corresponding setting of a plurality of surveyed areas difference after M is cut apart in vertical direction in the horizontal direction, use above-mentioned a plurality of detector to produce first detection signal that uses the action that the part of health carries out based on aforesaid operations person by above-mentioned video camera photography, wherein, above-mentioned N, M are the integers more than 2;

Maker generates second detection signal according to above-mentioned first detection signal, and this second detection signal represents to be put into the aforesaid operations person's of above-mentioned surveyed area the zone of a part of health; And

Motion detector, to first reference signal waveform of the expression aforesaid operations person's that generates according to above-mentioned second detection signal action, compare with the representative detection signal waveform of the predetermined action of decision in advance, whether the predetermined action that detects above-mentioned prior decision is consistent with aforesaid operations person's action.

Images