Summary of the invention
The purpose of the embodiment of the present invention be to provide a kind of food control method of intelligent refrigerator and, it is intended to solve above-mentioned existingSome intelligent refrigerators need nonetheless remain for user and are manually operated warehouse-in and the outbound recording food, and its operating procedure still comparesLoaded down with trivial details problem.
The embodiment of the present invention is achieved in that a kind of food control method of intelligent refrigerator, including:
When the door of described intelligent refrigerator is opened, automatically starts and be arranged on the photographic head in described intelligent refrigerator, make instituteState the image in photographic head continuous acquisition detection region, generate food image sequence；
It is analyzed described food image sequence processing, dynamic with obtain that user after described intelligent refrigerator is opened performsMaking type and the food classification corresponding with described type of action, described type of action includes taking out food and putting into food；
Record type of action that user after described intelligent refrigerator is opened performs and corresponding with described type of actionFood classification.
On the basis of technique scheme, it is analyzed described food image sequence processing, to obtain described intelligenceRefrigerator be opened after user perform type of action and the food classification corresponding with described type of action specifically include:
Use mixed Gauss model background modeling technology to carry out background modeling according to described food image sequence, obtain backgroundModel；
Image in described food image sequence is all compared with described background model, identifies each image respectivelyIn foreground pixel and background pixel, and generate the binaryzation sport foreground image corresponding with each image according to recognition result,Obtain sport foreground image sequence；
Sport foreground image in described sport foreground image sequence is all carried out connected domain differentiation, extracts each respectivelyThe iconic element corresponding to largest connected territory in sport foreground image；
Identify iconic element corresponding to the described largest connected territory direction of motion in described sport foreground image, according to instituteState the direction of motion in described sport foreground image of the iconic element corresponding to largest connected territory and judge that described intelligent refrigerator is beatenOpen the type of action that rear user performs；
Identify in the image in described food image sequence that comprise with described action by the grader of training in advanceType corresponding food classification.
On the basis of technique scheme, described sport foreground image in described sport foreground image sequence is all enteredRow connected domain differentiates, also wraps before extracting the iconic element corresponding to largest connected territory in each sport foreground image respectivelyInclude:
Sport foreground image in described sport foreground image sequence is carried out binary morphology closed operation process, the most rightCarry out the sport foreground image after binary morphology closed operation process and carry out binary morphology opening operation process, to remove described fortuneNoise pixel in dynamic foreground image.
On the basis of technique scheme, described according to the iconic element corresponding to described largest connected territory in described fortuneThe direction of motion in dynamic foreground image judges that described intelligent refrigerator also includes before opening the type of action that rear user performs:
Judge whether the pixel count in the iconic element corresponding to described largest connected territory is more than or equal to predetermined threshold value, ifMore than or equal to predetermined threshold value, then enter described according to the iconic element corresponding to described largest connected territory in described sport foregroundThe direction of motion in image judges that described intelligent refrigerator opens the step of the type of action that rear user performs, and otherwise, terminates flow process.
On the basis of technique scheme, the described grader by training in advance identifies described food image sequenceIn image in the food classification corresponding with described type of action that comprise specifically include:
The image in described food image sequence is identified by the SVM classifier advancing with the training of BOW characteristics of imageIn the food classification corresponding with described type of action that comprise.
The another object of the embodiment of the present invention is to provide the food management system of a kind of intelligent refrigerator, including:
Control module, for when the door of described intelligent refrigerator is opened, automatically starts and is arranged in described intelligent refrigeratorPhotographic head, make described photographic head continuous acquisition detection region in image, generate food image sequence；
Image analysis processing module, for being analyzed process, to obtain described intelligence ice to described food image sequenceCase be opened after user perform type of action and the food classification corresponding with described type of action, described type of action bagInclude taking-up food and put into food；
Logging modle, for record type of action that user after described intelligent refrigerator is opened performs and with described actionThe food classification that type is corresponding.
On the basis of technique scheme, described image analysis processing module specifically includes:
Background modeling unit, is used for using mixed Gauss model background modeling technology to carry out according to described food image sequenceBackground modeling, obtains background model；
Sport foreground image acquiring unit, for all entering the image in described food image sequence with described background modelRow compares, and identifies the foreground pixel in each image and background pixel respectively, and generates and each image according to recognition resultCorresponding binaryzation sport foreground image, obtains sport foreground image sequence；
Connected domain extraction unit, for all carrying out connected domain to the sport foreground image in described sport foreground image sequenceDifferentiate, extract the iconic element corresponding to largest connected territory in each sport foreground image respectively；
Type of action judging unit, for identifying that iconic element corresponding to described largest connected territory is in described motion foreground pictureThe direction of motion in Xiang, according to the motion side in described sport foreground image of the iconic element corresponding to described largest connected territoryTo judging that described intelligent refrigerator opens the type of action that rear user performs；
Food classification acquiring unit, for identifying the figure in described food image sequence by the grader of training in advanceThe food classification corresponding with described type of action comprised in Xiang.
On the basis of technique scheme, described image analysis processing module also includes:
Noise pixel processing unit, for carrying out two-value shape to the sport foreground image in described sport foreground image sequenceState closed operation processes, and then the sport foreground image after carrying out binary morphology closed operation process is carried out binary morphology and opensCalculation process, to remove the noise pixel in described sport foreground image.
On the basis of technique scheme, described image analysis processing module also includes:
Iconic element judging unit, whether the pixel count in the iconic element judging corresponding to described largest connected territoryMore than or equal to predetermined threshold value, if more than or equal to predetermined threshold value, then entering described according to corresponding to described largest connected territoryThe iconic element direction of motion in described sport foreground image judge described intelligent refrigerator open after user perform action classThe step of type, otherwise, terminates flow process.
On the basis of technique scheme, food classification acquiring unit specifically for:
The image in described food image sequence is identified by the SVM classifier advancing with the training of BOW characteristics of imageIn the food classification corresponding with described type of action that comprise.
The food control method and system implementing a kind of intelligent refrigerator that the embodiment of the present invention provides have following useful effectReally:
The embodiment of the present invention is owing to by when the door of described intelligent refrigerator is opened, automatically starting and being arranged on described intelligencePhotographic head in refrigerator, makes the image in described photographic head continuous acquisition detection region, generates food image sequence；To described foodProduct image sequence is analyzed processing, to obtain the type of action of user's execution after described intelligent refrigerator is opened and with describedThe food classification that type of action is corresponding, described type of action includes taking out food and putting into food；Record described intelligent refrigeratorAfter being opened user perform type of action and the food classification corresponding with described type of action such that it is able to intelligence iceAutomatically identify when case is opened and record user from intelligent refrigerator, put into or take out the action of food and corresponding with this actionFood classification, simplify user operation step, improve Consumer's Experience.
Detailed description of the invention
In order to make the purpose of the present invention, technical scheme and advantage clearer, below in conjunction with drawings and Examples, rightThe present invention is further elaborated.Should be appreciated that specific embodiment described herein only in order to explain the present invention, andIt is not used in the restriction present invention.
Fig. 1 is the schematic flow diagram of the food control method of a kind of intelligent refrigerator that the embodiment of the present invention provides.See Fig. 1Shown in, the food control method of a kind of intelligent refrigerator that the present embodiment provides may comprise steps of:
In S101, when the door of described intelligent refrigerator is opened, automatically starts and be arranged on taking the photograph in described intelligent refrigeratorAs head, make the image in described photographic head continuous acquisition detection region, generate food image sequence.
In the present embodiment, described photographic head is arranged on upper wall in the fresh-keeping layer of described intelligent refrigerator, and shoots down, describedDetection region is the field range of described photographic head.When the door of described intelligent refrigerator is opened, described intelligent refrigerator can be automaticallyStart described photographic head, and make described photographic head according to detecting the image in region described in default frequency continuous acquisition, with lifeOne-tenth may include user and put into or take out the image sequence of food motion images.
In S102, it is analyzed described food image sequence processing, uses after described intelligent refrigerator is opened to obtainFamily perform type of action and the food classification corresponding with described type of action, described type of action include take out food andPut into food.
What Fig. 2 showed step S102 in the present embodiment implements flow chart.Shown in Figure 2, in the present embodiment, stepRapid S102 may comprise steps of:
In S201, use mixed Gauss model background modeling technology to carry out background according to described food image sequence and buildMould, obtains background model.
In S202, the image in described food image sequence is all compared with described background model, identifies respectivelyGo out the foreground pixel in each image and background pixel, and generate the binaryzation motion corresponding with each image according to recognition resultForeground image, obtains sport foreground image sequence.
In the present embodiment, before in the image that photographic head obtains, the hand of user and the pixel in food region can be identified asScene element, remainder is background pixel, thus can carry out binary conversion treatment to obtain binaryzation according to recognition resultSport foreground image.See Fig. 3 a and Fig. 3 b, respectively illustrate the image acquired in photographic head shooting and this image through processingAfter sport foreground image.
In S203, the sport foreground image in described sport foreground image sequence is all carried out connected domain differentiation, respectivelyExtract the iconic element corresponding to largest connected territory in each sport foreground image.
In S204, identify iconic element corresponding to described largest connected territory motion side in described sport foreground imageTo, judge described intelligence according to the direction of motion in described sport foreground image of the iconic element corresponding to described largest connected territoryThe type of action that rear user performs can be opened by refrigerator.
In the present embodiment, by the way of setting up coordinate system iconic element described in identification in described sport foreground imageThe direction of motion, concrete:
Making sport foreground image top left corner apex is coordinate origin, and to the right, (x-axis is " square vertically downward for y-axis for x-axis levelTo " level to the right, vertically downward, and y-axis positive direction is the direction away from refrigerator to y-axis " positive direction ", and y-axis negative direction is near iceThe direction of case), if the pixel that the iconic element corresponding to largest connected territory is topmost is at the coordinate of current frame motion foreground imageY-axis coordinate in system is yt, then iconic element corresponding to largest connected territory described in previous frame sport foreground image is topmostThe y-axis coordinate of pixel be yt-1, the like, yt-n, yt-n+1…….yt-1, ytHave recorded in the past n frame in current frame imageThe y-axis coordinate of the pixel of the top of the iconic element corresponding to largest connected territory, so we according to largest connected territory institute rightThe variation tendency of the y-axis coordinate of the pixel of the top of the iconic element answered can obtain described iconic element in motion foreground pictureThe direction of motion in Xiang.
Further, in the present embodiment, if the y-axis of the pixel of the top of the iconic element corresponding to largest connected territoryCoordinate from yt-n, yt-n+1…….yt-1, ytMore and more less, then the type of action that user performs is for putting into food；Otherwise, ifThe y-axis coordinate of the pixel of the top of big iconic element corresponding to connected domain from yt-n, yt-n+1…….yt-1, ytIncreasinglyGreatly, then the type of action that explanation user performs is for taking out food.
In S205, identified by the grader of training in advance the image in described food image sequence comprises withDescribed type of action corresponding food classification.
In the present embodiment, step S205 specifically includes:
The image in described food image sequence is identified by the SVM classifier advancing with the training of BOW characteristics of imageIn the food classification corresponding with described type of action that comprise.
Optionally, shown in Figure 4, in another embodiment, can also include before step S203:
In S202-1, the sport foreground image in described sport foreground image sequence is carried out binary morphology closed operationProcess, then the sport foreground image after carrying out binary morphology closed operation process carried out binary morphology opening operation process,To remove the noise pixel in described sport foreground image.
In the present embodiment, some noise pixels can be comprised due in the sport foreground image that obtains in step S203, because ofThis, in order to remove the impact of these noise pixels, by entering described sport foreground image after getting sport foreground imageRow two-value form closed operation is carrying out binary morphology opening operation to it after processing, such that it is able to eliminate in sport foreground imageNoise pixel, it is simple to the extraction in follow-up largest connected territory.
Optionally, shown in Figure 4, in another embodiment, can also include before step S204:
In S203-1, it is judged that whether the pixel count in iconic element corresponding to described largest connected territory is more than or equal toPredetermined threshold value, if more than or equal to predetermined threshold value, then enters and described exists according to the iconic element corresponding to described largest connected territoryThe direction of motion in described sport foreground image judges that described intelligent refrigerator opens the step of the type of action that rear user performs, noThen, flow process is terminated.
In the present embodiment, if the pixel in the iconic element corresponding to described largest connected territory is more than or equal to presetting thresholdValue, then illustrate to include in this two field picture user and put into food or take out the image of action of food, now enter step S204Carry out the identification of user action type；On the contrary, if the pixel detected in the iconic element corresponding to described largest connected territory is littleIn predetermined threshold value, then illustrate that being not detected by user in this two field picture is taken in and out the action of food, now, it is not necessary to moveMake the judgement of type, terminate the flow process that this two field picture is analyzed to be processed.
In S103, record type of action that user after described intelligent refrigerator is opened performs and with described type of actionCorresponding food classification.
In the present embodiment, intelligent refrigerator record user perform type of action and the corresponding food of described type of actionThe information of record can also be uploaded to high in the clouds after classification, make high in the clouds according to the outbound of the food of intelligent refrigerator record and warehouse-inSituation to user recommend user may need the food bought or when food fast expired time user carried out the convenient clothes such as promptingBusiness.
Above it can be seen that the food control method of a kind of intelligent refrigerator of the present embodiment offer is due to by when described intelligenceWhen the door of energy refrigerator is opened, automatically starts and be arranged on the photographic head in described intelligent refrigerator, make described photographic head continuous acquisitionImage in detection region, generates food image sequence；It is analyzed described food image sequence processing, to obtain described intelligenceCan refrigerator be opened after user perform type of action and the food classification corresponding with described type of action, described action classType includes taking out food and putting into food；Record the type of action of user's execution after described intelligent refrigerator is opened and with describedThe food classification that type of action is corresponding such that it is able to automatically identify when intelligent refrigerator is opened and record user from intelligence iceCase is put into or taken out the action of food and the food classification corresponding with this action, simplifies user operation step, improveConsumer's Experience.
Fig. 5 is the schematic block diagram of the food management system of a kind of intelligent refrigerator that the embodiment of the present invention provides, this systemFor running the method that embodiment illustrated in fig. 1 provides.Illustrate only part related to the present embodiment for convenience of description.
Shown in Figure 5, the food management system of a kind of intelligent refrigerator that the present embodiment provides, including:
Control module 1, for when the door of described intelligent refrigerator is opened, automatically starts and is arranged in described intelligent refrigeratorPhotographic head, make described photographic head continuous acquisition detection region in image, generate food image sequence；
Image analysis processing module 2, for being analyzed process, to obtain described intelligence ice to described food image sequenceCase be opened after user perform type of action and the food classification corresponding with described type of action, described type of action bagInclude taking-up food and put into food；
Logging modle 3, for recording the type of action of user's execution after described intelligent refrigerator is opened and moving with describedMake the food classification that type is corresponding.
Shown in Figure 6, in the present embodiment, described image analysis processing module specifically includes:
Background modeling unit 21, is used for using mixed Gauss model background modeling technology to enter according to described food image sequenceRow background modeling, obtains background model；
Sport foreground image acquiring unit 22, for by the image in described food image sequence all with described background modelCompare, identify the foreground pixel in each image and background pixel respectively, and generate and each figure according to recognition resultAs corresponding binaryzation sport foreground image, obtain sport foreground image sequence；
Connected domain extraction unit 23, for all connecting the sport foreground image in described sport foreground image sequenceTerritory differentiates, extracts the iconic element corresponding to largest connected territory in each sport foreground image respectively；
Type of action judging unit 24, for identifying that iconic element corresponding to described largest connected territory is in described sport foregroundThe direction of motion in image, according to the motion in described sport foreground image of the iconic element corresponding to described largest connected territoryIntelligent refrigerator described in walking direction opens the type of action that rear user performs；
Food classification acquiring unit 25, for identifying in described food image sequence by the grader of training in advanceThe food classification corresponding with described type of action comprised in image.
Wherein, described food classification acquiring unit 25 specifically for:
The image in described food image sequence is identified by the SVM classifier advancing with the training of BOW characteristics of imageIn the food classification corresponding with described type of action that comprise.
Optionally, shown in Figure 7, in another embodiment, described image analysis processing module also includes:
Noise pixel processing unit 26, for carrying out two-value to the sport foreground image in described sport foreground image sequenceClosing operation of mathematical morphology processes, and then the sport foreground image after carrying out binary morphology closed operation process is carried out binary morphologyOpening operation processes, to remove the noise pixel in described sport foreground image.
Optionally, shown in Figure 7, in another embodiment, described image analysis processing module also includes:
Iconic element judging unit 27, for judging that the pixel count in the iconic element corresponding to described largest connected territory isNo more than or equal to predetermined threshold value, if more than or equal to predetermined threshold value, then entering described according to corresponding to described largest connected territoryThe iconic element direction of motion in described sport foreground image judge described intelligent refrigerator open after the action that performs of userThe step of type, otherwise, terminates flow process.
It should be noted that modules in the said system of embodiment of the present invention offer, due to real with the inventive methodExecuting example based on same design, its technique effect brought is identical with the inventive method embodiment, and particular content can be found in the present inventionNarration in embodiment of the method, here is omitted.
Above it can be seen that the food management system of a kind of intelligent refrigerator of the present embodiment offer equally can be at intelligence iceAutomatically identify when case is opened and record user from intelligent refrigerator, put into or take out the action of food and corresponding with this actionFood classification, simplify user operation step, improve Consumer's Experience.
The foregoing is only presently preferred embodiments of the present invention, not in order to limit the present invention, all essences in the present inventionAny amendment, equivalent and the improvement etc. made within god and principle, should be included within the scope of the present invention.