US20160138859A1

Movatterモバイル変換

Info

Publication number: US20160138859A1
Application number: US14/898,187
Authority: US
Inventors: Peter Stimpfig; Gerhard Wetzl; Wolfgang Schramm; Jens Wichert
Original assignee: BSH Hausgeraete GmbH
Current assignee: BSH Hausgeraete GmbH
Priority date: 2013-06-14
Filing date: 2014-06-04
Publication date: 2016-05-19
Also published as: CN105358925A; DE102013211096A1; WO2014198610A1; EP3008413A1

Abstract

A refrigeration appliance system includes a refrigeration appliance having a camera module for acquiring or detecting image data of an item to be refrigerated in a cold or refrigeration compartment of the refrigeration appliance and a wireless camera-module operating device for operating the camera module.

Description

The present invention relates to a refrigeration appliance system which comprises a refrigeration appliance having a camera module.
When a refrigeration appliance has a camera module and a fixed display, recorded image data can be displayed only locally on the refrigeration appliance. The refrigeration appliance is controlled by means of separately provided switches or controllers.
The object of the invention is to define a refrigeration appliance system that is easier to operate.
This object is achieved by the subject matter having the features claimed in the independent claim. The subject matter of the figures, description and dependent claims contains advantageous embodiments of the invention.
The object is achieved according to one aspect of the invention by a refrigeration appliance system which comprises a refrigeration appliance having a camera module for acquiring image data about refrigerated goods in a cold compartment of the refrigeration appliance and comprises a wireless camera-module operating device for operating the camera module. This achieves the technical advantage, for example, that the camera module is easier to control.
A refrigeration appliance is understood to include in particular a domestic refrigeration appliance, i.e. a refrigeration appliance which is used for domestic tasks in households or in the catering sector, and in particular is used for storing food and/or beverages at specific temperatures, for instance an appliance such as a refrigerator, a freezer, a combined fridge-freezer, a chest freezer or a wine cooler.
In an advantageous embodiment of the refrigeration appliance system, the camera-module operating device comprises a touch-sensitive display for controlling the camera module and displaying the acquired image data. This achieves the technical advantage, for example, that the camera modules can be controlled easily without buttons.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a plurality of camera modules, and the camera-module operating device comprises means for selecting a camera module. This achieves the technical advantage, for example, that a specific camera module can be selected for control and display.
In another advantageous embodiment of the refrigeration appliance system, the camera modules are arranged for the purpose of generating an all-round view of the refrigerated goods, and the camera-module operating device is designed to control a viewing angle of the all-round view. This achieves the technical advantage, for example, that the refrigerated goods can be displayed on the display device from different sides.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a first camera module for acquiring first image data about the refrigerated goods from a first perspective, and a second camera module for acquiring second image data about the refrigerated goods from a second perspective. This achieves the technical advantage, for example, that the refrigerated goods can be recorded and displayed from different sides.
In another advantageous embodiment of the refrigeration appliance system, the first camera module is arranged in a door of the refrigeration appliance, and the second camera module is arranged in a compartment ceiling. This achieves the technical advantage, for example, that the perspectives obtained thereby complement each other well, and it is possible to obtain extensive information about the refrigerated goods.
In another embodiment of the refrigeration appliance system, the camera-module operating device is designed to switch between displaying the first perspective and the second perspective on the camera-module operating device being tilted. This achieves the technical advantage, for example, that it is easier to make a change in perspective.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a camera array comprising a multiplicity of camera modules for generating a view composed from individual images. This achieves the technical advantage, for example, that the camera modules can be generated on the basis of a panoramic view.
In another advantageous embodiment of the refrigeration appliance system, the camera array is arranged in a door of the refrigeration appliance. This achieves the technical advantage, for example, that the panoramic view is obtained from the same perspective from which a user would also perceive the contents of the refrigeration appliance with the door open.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a position-detection device for detecting a position of the camera-module operating device placed on an outer surface of the refrigeration appliance. This achieves the technical advantage, for example, that the refrigeration appliance can use the position to perform further control actions, for instance for a lighting device.
In another advantageous embodiment of the refrigeration appliance system, the camera-module operating device is designed to display an image region associated with the detected position of the camera-module operating device. This achieves the technical advantage, for example, that according to the detected position, those images that correspond to the position are displayed on the display device. This allows the contents of the refrigeration appliance to be displayed in an X-ray-like manner.
In another advantageous embodiment of the refrigeration appliance system, the refrigeration appliance comprises a lighting device for illuminating the refrigerated goods in the cold compartment. This achieves the technical advantage, for example, that the refrigerated goods can be recorded under defined light conditions.
In another advantageous embodiment of the refrigeration appliance system, the lighting device can be controlled by the camera-module operating device. This achieves the technical advantage, for example, of implementing straightforward and switch-free control of the lighting device.
In another advantageous embodiment of the refrigeration appliance system, the camera-module operating device is designed to compare the acquired image data about the refrigerated goods with stored image data about the refrigerated goods. This achieves the technical advantage, for example, of being able to show differences in the cold compartment.
In another advantageous embodiment of the refrigeration appliance system, the camera-module operating device is designed to identify the refrigerated goods from the acquired image data. This achieves the technical advantage, for example, that a list of contents of the refrigeration appliance can be generated or the type of the refrigerated goods can be determined.

Exemplary embodiments of the invention are described in more detail below and illustrated in the drawings, in which:

FIG. 1 is a schematic view of a refrigeration appliance;

FIG. 2A is a perspective view of a cold compartment having two camera modules;

FIG. 2B is a front view of the cold compartment;

FIG. 3A is a perspective view of a cold compartment having one camera module;

FIG. 3B is a plan view of the cold compartment;

FIG. 4 is a perspective view of a cold compartment having a plurality of camera modules;

FIG. 5 shows two views of different camera modules;

FIG. 6 is a plan view and a side view of the cold compartment in conjunction with an angle of view;

FIG. 7 is a plan view of the cold compartment in which a plurality of camera modules are used;

FIG. 8 is a perspective view of a cold compartment having a sensor mat;

FIG. 9 is a view of a refrigeration appliance having a data bus;

FIG. 10 is another view of a refrigeration appliance having a data bus;

FIG. 10A is a schematic view of an arrangement of the camera modules in a door;

FIG. 11 is a schematic view of a data bus having distribution nodes;

FIG. 12 is a view of a processor unit;

FIG. 13 shows images of refrigerated goods in the cold compartment and shows an image analysis;

FIG. 14 is a view of a camera-module operating device having a touch-sensitive display;

FIG. 15 is a schematic view of a refrigeration appliance having a camera module in a door; and

FIG. 16A-C are schematic views of different connections of camera modules.

FIG. 1 shows a refrigerator as an example of arefrigeration appliance100 comprising a top andbottom refrigerator door111. Therefrigeration appliance100 is a stand-alone appliance.

The refrigerator is used, for example, for chilling food and comprises a refrigerant circuit containing an evaporator, a compressor, a condenser and a flow regulator. The evaporator is a heat exchanger in which the liquid refrigerant, after expansion, is evaporated by absorbing heat from the medium to be cooled, i.e. from the air inside the refrigerator.

The compressor is a mechanically driven component which extracts the refrigerant vapor from the evaporator and discharges said vapor at a raised pressure to the condenser. The condenser is a heat exchanger in which the vaporized refrigerant, after compression, becomes a liquid by releasing heat to an external cooling medium, i.e. to the surrounding air. The flow regulator is a device designed to constantly reduce the pressure by narrowing the cross section.

The refrigerant is a fluid which is used for heat transfer in the refrigeration system and which absorbs heat when at a low temperature and low pressure, and releases heat when at a higher temperature and higher pressure, usually involving a change in state of the fluid.

Therefrigeration appliance100 comprises the top andbottom door111, which closes a cold compartment inside therefrigeration appliance100. Therefrigeration appliance100 also comprises aside wall109. Inside therefrigeration appliance100 is an electronic camera module, which is able to record the refrigerated goods inside therefrigeration appliance100 using a single image, and to generate image data. Thecamera module101 may be able to be directed, pivoted or adjusted by the customer, for example.

FIG. 2A shows a perspective view of thecold compartment105 having twocamera modules101. Thecold compartment105 forms a receptacle for therefrigerated goods103 inside therefrigeration appliance100. A beaker and a food bag are inside thecold compartment105 as examples of therefrigerated goods103. The twocamera modules101 are arranged at the same height and spaced horizontally apart from one another on afront face131 of thecold compartment105, for example in thedoor111 of therefrigeration appliance100. Therefrigerated goods103 are recorded by the twocamera modules101 from the front face of therefrigeration appliance100. Thecamera modules101 on thefront face131 of thecold compartment105 make it possible to emulate the view of a user of the contents of thecold compartment105.

Alighting device102 is provided for illuminating thecold compartment105 during a recording. For example thelighting device102 is an infrared light or illumination by visible light. Thelighting device102 is arranged in a compartment ceiling of thecold compartment105 for instance. One and the same image can be recorded using different illumination for additional image information or higher resolution.

The design of the interior of thecold compartment105 can be optimized with respect to thecamera module105, for example in terms of adjustability, a compartment floor arrangement, the position of the door racks relative to the compartment floors or adjustability of the door racks. Thecamera modules101 are, for example, inclined towards the center of the refrigeration appliance. Thecamera modules101 can be movable. Acamera module101 in a side wall has the advantage of gaining a view towards thedoor111. In addition, reflective coatings, glass beads, mini retro-reflectors can be used.

A keep-fresh compartment does not have a dedicated camera, and has a transparent lid, transparent compartment floor and no opaque strip at the compartment floor edge.

Thecamera modules101 comprise a light sensor array containing color sensors, for instance a CCD chip. The light sensor array has, for example, a minimum resolution of 640×480 pixels (VGA) and three color channels each with a color depth of at least 8 bits, for example RGB color channels. Thecamera module101 can be designed, for example, such that it records not only image data in the visible region but additionally also image data in the infrared region.

CCD image sensors suitable for use in the light sensor array consist of an array of light-sensitive photodiodes. These may be rectangular, square or polygonal with edge lengths of 1.4 μm to more than 20 μm. The larger the surface area of the photodiodes, the higher the light sensitivity and the dynamic range of the CCD image sensor. For the same sensor size, however, the image resolution is lower in this case. Thecamera modules101 have an interface for communicating with a processor unit. The sensor data obtained by the light sensor array can be processed by the processor unit. The power of the system is of the order of 3 W when active, which is only on demand, and 0.1 W in standby.

FIG. 2B shows a front view of thecold compartment105 from the perspective of one of thecamera modules101. In the front view, the outlines of therefrigerated goods103 can be seen in front of the background of arear wall129 of thecold compartment105.

By using twocamera modules101 spaced horizontally apart, it is possible, for example, to record two images of therefrigerated goods103, which in a stereoscopic view convey to the user a three-dimensional image of therefrigerated goods103. In addition, using twocamera modules101 results in the advantage that even if one of thecamera modules101 should be obstructed byrefrigerated goods103, the interior of thecold compartment105 can be recorded by the other of thecamera modules101.

FIG. 3A shows another perspective view of thecold compartment105 having onecamera module101. Thecamera module101 is arranged in thecompartment ceiling113 of thecold compartment105. Arranging thecamera module101 in thecompartment ceiling113 allows a fundamental change in the direction of view and an overhead view onto the contents of thecold compartment105.

FIG. 3B shows an overhead view onto thecold compartment105 from the perspective of thecamera module103 arranged in thecompartment ceiling113. This overhead view can be used for visualizing the contents of thecold compartment105.

As a general rule, the camera module can also be arranged on therear wall129 or on aside wall109 of thecold compartment105. It is hence possible to enlarge the field of vision for visualizing the contents and to record different perspectives inside thecold compartment105 or an all-round view.

FIG. 4 shows a perspective view of acold compartment105 having a plurality ofcamera modules101. Thecamera modules101 are arranged such that therefrigerated goods103 can be recorded from every side and under different viewing angles. By combining the images recorded by theindividual camera modules101, it is possible to generate an all-round view of therefrigerated goods103 in thecold compartment105.

In order for thecamera module101 to be able to record therefrigerated goods103 with sufficient brightness, alighting device102 is implemented inside thecold compartment105. Thecamera module101 and thelighting device102 can be arranged in a shared installation space, for example.

A flash light unit for emitting a flash of light during a recording is arranged inside therefrigeration appliance100 as thelighting device102 for instance, so that a recording can be obtained using flash light. The flash light unit is arranged on thecamera module101 for example. The flash light unit can be actuated by an additional controller, which outputs a control signal for eachcamera module101. The light intensity of the flash can be measured using a special flash-light meter or a dual-function exposure meter.

In addition, therefrigeration appliance100 can comprise an LED array as thelighting device102, which is arranged such that therefrigerated goods103 in thecold compartment105 are illuminated as uniformly as possible from all sides. The LED array can be controlled centrally. In addition, the LED array provides a simple and energy-saving way of implementing uniform or adaptive illumination of thecold compartment105 and of reducing cast shadows.

As a general rule, however, any other suitable lighting devices can also be used, such as LED strips, incandescent lamps or neon tubes for instance. When the refrigeration appliance door is open, ambient light can be used as the interior lighting.

FIG. 5 shows two different camera modules101-1 and101-2. Two types of camera modules101-1 and101-2, which are suitable for fitting in therefrigeration appliance100, are available for image acquisition. The assembled camera modules101-1 and101-2 for instance comprise a light sensor array, a lens and amicrocontroller104.

Camera modules101-1 and101-2 having built-inmicrocontroller104 can be integrated easily into therefrigerated appliance100. Integration is also made simpler if there is a driver for the particular camera module101-1 and101-2. If the camera modules101-1 and101-2 comprise a USB interface, the camera modules101-1 and101-2 can be connected directly to a USB data bus or a PC. The camera modules101-1 and101-2 comprise an imager, which provides the raw image data for analysis.

The camera modules101-1 and101-2 are protected against fruit acid, dust and moisture. The camera modules101-1 and101-2 can be arranged in an encapsulated housing and/or provided with lens heating by an IR-LED and/or with a lens coating in order to prevent fogging.

The first camera module101-1 comprises a pin-hole lens135 and acircuit board133, on which are arranged a chip comprising a light sensor array and amicrocontroller104. The camera module101-1 is formed by a pin-hole camera107, which can have a glass cover to protect against soiling. Although the light intensity of the pin-hole camera107 is less than that of focusing cameras, the pin-hole camera107 has a greater depth of field. The lower light intensity, however, can be compensated by a longer exposure time, because no movements occur inside thecold compartment105.

The second camera module101-2 comprises alens109 and acircuit board133, on which are arranged a chip comprising a light sensor array and a microcontroller. Thelens109 may have a focal length of less than 10 mm (miniature lens), for example, so that thelens109 also produces a large depth of field and a near-field region can be imaged. Thelens109 is a wide-angle lens, for instance, with an angle of view of greater than 60°. The angle of view defines the angle range that can be captured by the camera module101-2. In addition, the camera modules101-1 and101-2 can comprise additional deflection mechanisms, for instance devices such as hinged or rotating prisms, mirrors, lenses or semiconductor mirrors.

The dimensions of thecircuit board133 are 32 mm×32 mm for example. On thecircuit board133 are formed, for example, electrical circuits for themicrocontroller104 and an interface for connecting thecamera module101 to a data bus so that image data can be provided as a file. Thecircuit board133 is formed in the foam in thedoor111, for example.

The height of the camera module101-1 and101-2 is defined by thelens109 or the pin-hole lens135 and lies between 10 mm and 30 mm for instance. Pin-hole lenses135 allow a lower depth of thecamera module101 of less than 20 mm. Furthermore, a camera module101-1 having a pin-hole lens135 requires a smaller chamber or recess for fitting in a wall of therefrigeration appliance100. For a camera module101-1 having a pin-hole lens135, the depth of the recess equals 2 mm to 3 mm, for example, whereas for a camera module101-2 having alens109, the depth of the recess equals about 10 mm. The camera module101-1 having the pin-hole lens135 can also be implemented withoutcircuit board133 and can have dimensions of 10 mm×10 mm×7 mm, for example. A pin-hole camera107 having a pin-hole lens135 can permit an angle of view of 120°, for example.

The camera modules101-1 and101-2 can have different interfaces for transmitting image data, for instance interfaces such as an RS232 interface, an SPI interface, an IIC interface or a USB interface.

FIG. 6 shows a plan view and a side view of thecold compartment105 in conjunction with different angles of view for acold compartment105 of size 20 cm×60 cm×50 cm. The angle of view defines the angle range that can be imaged by thecamera module101. Different viewable regions for thecamera module101 are obtained with different angles of view. There areindividual areas137 to the sides of thecamera module101 in thecold compartment105 that are not visible and that become smaller as the angle of view gets bigger.

FIG. 7 shows a plan view of thecold compartment105 in which a plurality ofcamera modules105 are used. An arrangement having fourcamera modules101 with an angle of view 40° is shown on the left. An arrangement having threecamera modules101 with an angle of view 60° is shown on the right by way of comparison. With a larger angle of view,fewer camera modules101 can be used to cover an identicalviewable region139.

The angle of view determines the number ofcamera modules101 to be able to generate a continuous image. A compromise can be found between the angle of view and the number ofcamera modules101 depending on the size of thecold compartment105. A larger angle of view, however, produces greater distortion. For instance a cushion-shapeddistortion141 or a barrel-shapeddistortion143 is produced. The nature and size of the

distortion

141 and143 can be identified and corrected for further image processing. A pattern can be used for this purpose.

FIG. 8 shows acold compartment105 having asensor mat115 for detecting a change in a load on acompartment floor117. The membrane-like sensor mat115 can be used for detecting weight changes. Thesensor mat115 is a few millimeters thick and is placed on thecompartment floor117. Thesensor mat115 measures the change in loads on the horizontal, flat and fixedcompartment floor117 and is suitable for monitoring free-standing and stackedrefrigerated goods103 on thecompartment floor117. This can be used, for example, for determining how full containers are by comparing the weight with an initial value. In addition, it is also possible to track changes in position of therefrigerated goods103 and therefore to detect the position.

Moreover, the position of newly-inserted refrigerated goods103-2 can be detected. If, for example, the refrigerated goods103-2 are added to the refrigerated goods103-1 on thecompartment floor117, thesensor mat117 detects the surface region on which the refrigerated goods103-2 have been added. Changes in thecold compartment105 can thereby be detected in addition to a visual change in the image data acquired by thecamera module101.

In addition, therefrigeration appliance100 can comprise a reader for radio tags (RFID tags), so that therefrigerated goods103 in thecold compartment105 can be registered by reading the tags. The reader for radio tags is an electronic circuit which by means of radio waves is able to read the information stored in electronic tags.

Both the radio-tag reader and thesensor mat115 can be provided as additional sensors to complement optical image acquisition, for instance as an alternative or addition to the optical recognition of an EAN code. This increases reliability and opens up further applications.

In addition, thecamera module101 can comprise a controllable pivoting device for turning a recording direction of thecamera module101 and/or a controllable zooming device for magnifying a recording by thecamera module101. Furthermore, thecamera module101 can comprise a displacement device for controllable horizontal and/or vertical movement of thecamera module101 along a wall of therefrigeration appliance100. The pivoting device, the displacement device and the zooming device can be implemented using a suitable mechanism driven by an electric motor and can be controlled by a processor unit via a data bus.

FIG. 9 shows arefrigeration appliance100 having adata bus121. Thecold compartment105 comprises fourcamera modules101, which are arranged in various corners of thecold compartment105. In addition, therefrigeration appliance100 comprises aprocessor unit119, to which the images are transmitted from theindividual camera modules101. Theprocessor unit119 comprises a processor and a memory and is used for processing the transmitted image data.

Adata bus121 is used for transmitting the image data from thecamera module101 to theprocessor unit119. A data bus is a system for data transmission between a plurality of bus users over a shared transmission path, with the bus users not being involved in the data transmission between other bus users.

Each of the fourcamera modules101 is connected to aseparate data bus121. Further devices can be connected to thesedata buses121 and controlled. Part of thedata bus121 is integrated in theside wall109 of therefrigeration appliance100. Thedata bus121 is formed by electrical conductors that transmit signals from thecamera modules101 to theprocessor unit119.

Thedata bus121 is a serial bus, for instance, which is used to transmit the individual bits or data packets in a serial data transmission sequentially (serially) over the conductor. Thedata bus121 implements, for example, a USB standard, an I2C standard, an SPI standard or a CAN standard, or, for instance, a DBus2 standard or DBus3 standard. Thedata bus121 is a high-speed bus (USB) for example. Aserial data bus121 has the advantage that it can be implemented using a small number of conductors.

Thecamera modules101 and theprocessor unit119 likewise have a serial interface for connecting thedata bus121. It is advantageous to usecamera modules101 and systems for which drivers already exist. For this purpose, thedata bus121 can be pre-fitted with connecting devices such as plug-fit connectors, for example, for connecting thecamera modules101. In this case, thecamera module101 can be changed and replaced easily if it has a fault. In additional thedata bus121 and connected components can be designed such that the components, forinstance camera modules101 orlighting devices102, are supplied with electrical power or current via thedata bus121. This reduces the cabling costs inside therefrigeration appliance100.

FIG. 10 shows anotherrefrigeration appliance100 having thedata bus121. Thecold compartment105 comprises fourcamera modules101, which are arranged in the respective corners of thecold compartment105. The fourcamera modules101 are connected in pairs to thesame data bus121. Two of thecamera modules101 are arranged in adoor111 of therefrigeration appliance100 and generate a front view of thecold compartment105. Thedata bus121 to thesecamera modules101 is taken through ahinge123 of thedoor111 via a cable grommet.

A logical or physical address is assigned to therespective camera modules101 and can be used to address thecamera modules101 on thedata bus121 individually. In addition, if required additional local distribution nodes can be provided. An image retrieval by theprocessor unit119 via thedata bus121 should not take longer than a defined time period.

FIG. 10A shows a schematic view and a cross-sectional view of an arrangement of sixcamera modules101 on the inside of adoor111. Thedoor111 has adoor handle169 on the outside. Thecamera modules101 acquire the image data from an identical region inside acold compartment105 from different viewing angles. Two of thecamera modules101 at a time are grouped into pairs comprising a first and asecond camera module101 for stereoscopic image recording.

Two of thecamera modules101 at a time are arranged as a pair at the same height in thedoor111. Afirst camera module101 of the pair is arranged on a right-hand door pillar167, and asecond camera module101 is arranged on a left-hand door pillar167. Thecamera modules101 are arranged in a corner of thedoor pillars167 on the inside of thedoor111. For example, thecamera modules101 are integrated in a frame of thedoor111 so that they do not protrude out of the inside of thedoor111.

Eachcamera module101 is assigned alighting device102, for example in the form of a white LED in thedoor111. Eachlighting device102 is arranged diagonally beside the associatedcamera module101.

In the center of thedoor111 is formed adoor rack165 into whichrefrigerated goods103 such as bottles, for example, can be placed. Thedoor rack165 is not height-adjustable. Two of thecamera modules101 are located directly beneath thedoor rack165, so that they cannot be obstructed by therefrigerated goods103 inside therefrigeration appliance100. Twofurther camera modules101 are arranged in an upper region of thedoor111 so that they too cannot be obstructed by refrigerated goods, and image recording can be performed from diagonally above.

The pairs ofcamera modules101 are arranged at different heights on the inside of thedoor111 so that the recording can be obtained at different viewing angles. This is done, for example, by each pair ofcamera modules101 capturing an identical region inside acold compartment105 from different viewing angles.

Thedoor111 comprises anintegral data bus121 for connecting thecamera modules101 to aprocessor unit119 so that the image data obtained by thecamera modules101 can be transmitted for further processing. All sixcamera modules101 are connected to thesame data bus121, for example. Thisdata bus121 can be designed as a serial data bus so that the cabling costs for transmitting the image data can be low. The first andsecond camera module101 of each pair can each comprise a zooming device for synchronous magnification of the stereoscopic image recording. In general, thecamera modules101 can also be arranged in different positions inside the refrigerated appliance, for instance on a rear wall of therefrigerated appliance100.

FIG. 11 shows a schematic view of adata bus121 in arefrigeration appliance100 having a plurality ofdistribution nodes125. Thedistribution nodes125 each form an auxiliary module for connecting thecamera modules101 and comprise suitable interfaces. Thedistribution modules125 are implemented, for example, by an embedded platform such as a BeagleBoard for instance.

Thedistribution nodes125 are used for distributing image data and control signals between theprocessor unit119 and thecamera modules101, and can form image conversion units. In addition, data compression can be performed in thedistribution node125. Furthermore, protocol conversion can also take place in thedistribution nodes125.

Thecamera modules101 connected to thedata bus121 can comprise a microcontroller, which performs compression of the image data obtained. A compression facility for compressing the image data can be implemented in this way. This compression involves converting the image data obtained in bitmap format for example into image data based on the JPEG format and transmitting said image data in JPEG format via thedata bus121. This results in the advantage that the transmission is performed with less data and takes a shorter time period.

FIG. 12 shows a perspective view of theprocessor unit119. Theprocessor unit119 comprises aninterface145 for transmitting the image data to an external network. For example, theinterface145 is a wireless WLAN interface, a Bluetooth interface, an Ethernet interface or a mobile communications interface such as GSM, UMTS or LTE. The image data can be supplied to an external network via theinterface145. Where required, a direct connection to a personal computer is possible via theinterface145. The personal computer can be used to process the sensor data, for example to remove distortions and noise, perform artificial sharpening or to increase contrast.

Theprocessor unit119 is used for processing the sensor data and image data from thecamera modules101. A program library, such as OpenVC for instance, containing algorithms for image processing and computer vision can be used for image analysis. Implementable image analysis methods are, for example, detecting a change, identifying a barcode or recognizing defined logos (pattern recognition). Linux can be used as the operating system for theprocessor unit119.

Theprocessor unit119 can be used to retrieve the image data from thecamera modules101 and process the image data transmitted in this way. Image processing, for example suppressing noise or assembling the image data into larger units, can be used in this context. The image data can also be analyzed, for example difference images or difference objects can be detected. This analysis can be used to identifyrefrigerated goods103 recorded by thecamera modules101 and to obtain additional information relating to these goods. The image data can be provided via a wireless network.

In addition, theprocessor unit119 can control alighting device102 for illuminating thecold compartment105 during a recording. For this purpose, therefrigeration appliance100 comprises alighting device102, which is connected to thedata bus121 and can be controlled via this data bus.

FIG. 13 shows a first image recorded by thecamera module101 composed offirst image data147, and a second image recorded by thecamera module101 composed ofsecond image data149. Thesecond image data149 is recorded at a later point in time than thefirst image data147. Theprocessor unit119 or an external camera-module operating device on which an application (app) is running can be used to implement a freshness-determination facility, which serves to determine a condition of freshness of the refrigerated goods on the basis of thefirst image data147 and thesecond image data149. The time interval between the first and the

second image data

147 and149 equals one day, for example.

In both cases, thelighting device102 provides substantially identical lighting conditions in thecold compartment105 in order to ensure comparability of the time-offset recordings. To prevent the

image data

147 and149 being falsified by ambient light entering thecold compartment105 on opening thedoor111 of therefrigeration appliance100, therefrigeration appliance100 comprises a sensor for detecting opening of thedoor111. If the sensor detects that thedoor111 of therefrigeration appliance100 is open, no image data is acquired. The image data is only acquired once the door is closed111. The image data is recorded after thedoor111 is closed once uniform illumination of the interior of thecold compartment105 has taken place.

It is also possible by comparing the first image data and the second image data to establish thatrefrigerated goods103 have been added or removed. Theimage data151 shows the subtraction of thesecond image data149 from thefirst image data147. Thefirst image data147 and thesecond image data149 are, for example, three-color image data (RGB) with a color resolution of 24 bits per pixel.

The difference and the change between both sets of

image data

147 and149 can be calculated by the subtraction. In the example shown, theimage region153 displays therefrigerated goods103 that have been removed from thecold compartment105. Showing or hiding the contents of thecold compartment105 and the calculated change can be performed on a display of therefrigeration appliance100, for example on a tablet PC or a smartphone. In addition, an outline of therefrigerated goods103 and the change brought about can be highlighted.

It is likewise possible to determine a change over time in therefrigerated goods103 that are perishable. A freshness-determination facility formed by theprocessor unit119 or the camera-module operating device by means of an application determines the condition of freshness of therefrigerated goods103 on the basis of a color difference between thefirst image data147 and thesecond image data149. Then a suitable algorithm can be used to determine from a change in color that has occurred whether therefrigerated goods103 are still fresh or usable. It is also possible to use the weight-change data acquired by thesensor mat115 to determine freshness.

In addition, the freshness-determination facility may be designed to detect the type of therefrigerated goods103 on the basis of the first image data or the second image data. The freshness-determination facility may, for example, automatically identify what sort of perishable goods therefrigerated goods103 constitute. For instance the freshness-determination facility is able to identify by image analysis whether therefrigerated goods103 in thecold compartment105 consist of bananas or tomatoes. An image analysis algorithm compares the recording from thecamera module101 with pre-saved image data and thereby identifies therefrigerated goods103.

Should automatic identification of the type of therefrigerated goods103 fail, therefrigeration appliance100 can comprise an input apparatus for manual input of the type of therefrigerated goods103. This is formed, for example, by a touch-sensitive display or a display device on which different types ofrefrigerated goods103 are displayed in menu form. A user selects a particular item of refrigerated goods, which the user has placed in thecold component105, by touching the corresponding menu option. From the entered or detected type of therefrigerated goods103, it is possible to determine more precisely the condition of freshness of therefrigerated goods103 by means of a color difference.

Aspecial camera module101, which determines the image data not only in the visible spectrum region but simultaneously also in the infrared spectral region, can be used for detecting a condition of freshness. Thecamera module101 can comprise a dichroic element, which separates a beam path of the infrared light from the beam path of the visible light.

The dichroic element is an optical prism, for example, which splits a light beam into two beams of different spectra or colors. The dichroic element is made of a glass, for instance, certain surfaces of which are provided with dichroic mirrors, which reflect or transmit the light depending on its wavelength.

The visible light is detected by a first light sensor array, and the separated infrared light by a second light sensor array. The required design generally determines the number, arrangement and type of thecamera modules101 and of the lighting.

In order to illuminate therefrigerated goods103 during the image recording, alighting device102 is provided in thecold compartment105, which lighting device provides identical illumination of thecold compartment105 for every recording. In order to be able to perform a spectral analysis of therefrigerated goods103 at a defined frequency, thelighting device102 can emit monochromatic and narrowband light of a defined frequency, for example. For instance, thelighting device102 comprises a tunable LED for emitting a monochromatic light of variable wavelength. Thelighting device102, however, can also be designed such that it emits polychromatic white light or infrared light.

Therefrigeration appliance100 can comprise a control and display device, which displays the determined condition of freshness of therefrigerated goods103 to a user. If the detected color difference or a color change of, for instance, fruit or vegetables, exceeds a defined threshold value, a user can be informed via the display device. A tablet PC or smartphone can be used as the control and display device.

If the

image data

147 and149 is available in theprocessor unit119 or the display device, further information can be extracted therefrom using image analysis algorithms, for instance using an application on the camera-module operating device. This information can then be displayed on the camera-module operating device. For instance, the recorded

image data

147 and149 can be used not only to detect changes but also to detect bar codes or QR codes of therefrigerated goods103 or to perform character recognition. Further analysis of the codes can be performed via a product database on the Internet, in which database the relevant codes are stored. In addition, the ascertained product data can be associated with detected changes in the image data.

Image analysis employing pattern recognition can also be used to identify product logos. For this purpose, the product logos should be uniquely identifiable and not concealed by otherrefrigerated goods103. Reliable optical recognition of the product can be guaranteed if the recording contains sufficient details of the product logo. Using suitable algorithms, theprocessor unit119 can be trained for a multiplicity of products.

A list of contents of therefrigeration appliance100 can be created in combination with product recognition. Product recognition is performed in this case using optical recognition of an EAN code on the product. Products such as fruit or vegetables, for instance, that have no EAN code can be selected manually from a selection list. Product registration can be performed in this context by identifying the EAN code. Once the product has been placed in therefrigeration appliance100, the resultant change is then associated and highlighted in the image data (tracing).

The product database of therefrigeration appliance100 can be extended to the entire kitchen area. This is done by initially selecting a storage location and registering the product by identifying the EAN code. Then the user puts the product away to store. The stock list can be updated when goods are removed. It is also possible to identify a product from the EAN code and to make an association with the image data.

Details providing additional information about a registered product, for instance information such as a best before date or a minimum stock level, can be used for monitoring purposes. In this case, the user can get the additional information (best before date) displayed when selecting the product on a display or in the database. If the product stock runs low or the product runs out, the user can be informed via the camera-module operating device.

In this context, for instance, the image data is merged with the product data by storing the outline information from an image analysis after the registration of a product. The product database can be used to implement other applications in addition to product monitoring. For instance these applications can include creating a recipe suggestion from the stock list or creating a shopping list for frequently purchased products. In this connection, an adaptive system can learn the habits of the user. An online interface to the stock list or to the image data allows convenient access on the move. The product registration and product database together with the display of additional information for the user provides extra added value in addition to pure image analysis and image display.

FIG. 14 shows a view of a portable camera-module operating device300 having a touch-sensitive display303 (touchscreen). The camera-module operating device300 for example is a Tablet PC connected to therefrigeration appliance100 via a WLAN, or a smartphone connected to therefrigeration appliance100 via the mobile communications network. The camera-module operating device300 can also be connected to therefrigeration appliance100 via a USB interface, however. The USB interface, however, can also be used for charging the camera-module operating device300.

An application such as a computer program, for instance, can be executed on the camera-module operating device300. An installation kit can be provided for fitting the camera-module operating device300. The camera-module operating device300 is connected to a router via a wireless network. The camera-module operating device300 can act as an image server or as a graphics processor and can back up the image data via a WLAN. The portable camera-module operating device300 can be replaced or updated by a user.

Data can be transferred bi-directionally between the portable camera-module operating device300 and therefrigeration appliance100. The wireless camera-module operating device300 is a component of a refrigeration appliance system and is used for displaying the image data acquired by thecamera modules101. The user can hence select the displayedcold compartment105 in the application on the camera-module operating device300. In addition, a view can be changed by swiping upwards or downwards on the touch-sensitive display303 or by tilting the camera-module operating device300 to right or left. An angle at which the camera-module operating device300 is held can define, for instance, whichcold compartment105 is displayed.

Thecamera modules101 inside thecold compartment105 have a large angle of view. A plurality ofcamera modules101 are used for a continuous representation of the front of view. Therefrigeration appliance100 can comprise a plurality ofcamera modules101, which are arranged in thecold compartment105 such that therefrigerated goods103 are recorded from all sides, and an all-round view is generated by combining the individual images. The viewing angle in the all-round view onto therefrigerated goods103 is selected by swiping left and right on the touch-sensitive display303.

Acamera module101 having a large angle of view (fish eye) and arranged in acompartment ceiling113 allows an overhead view onto therefrigerated goods103 in thecold compartment105. Anadditional camera module101 is arranged in adoor111 of therefrigeration appliance100. Thecold compartment105 additionally comprises alighting device102 for uniform illumination of therefrigerated goods103. In this case and in all the other cases, the particular view can be selected by tilting the camera-module operating device300.

In general, therefrigeration appliance100 as a component of therefrigeration appliance system200 can comprise afirst camera module101 for acquiring first image data about therefrigerated goods103 from a first perspective, and asecond camera module101 for acquiring second image data about therefrigerated goods103 from a second perspective. The perspective can generally be changed by touching the touch-sensitive display303. Therefrigeration appliance100 can comprise anylighting device102 that is suitable for illuminating therefrigerated goods103 in thecold compartment105 during image-data acquisition. Thelighting device102 can be controllable by the camera-module operating device300.

On one side of the camera-module operating device300 is displayed amenu305 of options, which can be used to select the particularcold compartment105 containing therelevant camera module101. The contents of thecold compartment105 are displayed when thecold compartment105 is selected by touching the touch-sensitive display303. The particular view can also be selected by swiping across the touch-sensitive display103. It is possible to zoom-in or zoom-out the view, for instance, by a spreading movement with the fingers on the touch-sensitive display103. It is also possible to display additional information such as, for example, a condition of freshness of therefrigerated goods103 or changes in the image data compared with stored earlier image data.

The camera-module operating device300 can compare the acquired image data about therefrigerated goods103 with stored image data about therefrigerated goods103. In addition, the camera-module operating device300 can use an analysis algorithm to identify therefrigerated goods103 from the acquired image data. The user can select and view on the camera-module operating device300 the image data from thecamera module101 and the processed images, and get additional information displayed on said console, for instance information such as the allocation of a bar code to a specific product. For this purpose, the image data and the associated additional information are retrieved, processed and provided by theprocessor unit119 of therefrigeration appliance100. For this purpose, the camera data and additional information can be retrieved, processed and provided by an external central computer system. An association between therefrigeration appliance100 and the camera-module operating device300 can be made via a central processing unit, for example by means of Home Connect.

FIG. 15 shows a schematic view of arefrigeration appliance100 having acamera array127 in thedoor111. Therefrigeration appliance100 comprises acamera array127 having a multiplicity ofcamera modules101 arranged in the form of an array on the inside of thedoor111. Each of thesecamera modules101 generates an individual front-view image from the inside of therefrigeration appliance100. Thecamera modules101 of thecamera array127 are arranged such that a combined view of the entire front view, such as when thedoor111 is open, can be generated from the individual images. A panoramic image representing a continuous recording of the front side is composed from thecamera array127 in therefrigerator door111. Thecamera modules101 can be calibrated in order to improve an assembly of the individual images.

In addition, therefrigeration appliance100 comprises a position-detection device for detecting a position of the camera-module operating device300 placed on the outside of therefrigeration appliance100. This position-detection device can be used, for example, to determine a coordinate pair specifying the position on the front face of therefrigeration appliance100 in which the camera-module operating device300 is placed.

Then an image region of the previously assembled view, which region is associated with the coordinate pair, can be transmitted to the camera-module operating device300. This creates the impression that a user can use the wireless camera-module operating device300 to look through the wall of therefrigeration appliance100 in an X-ray-like manner. The user can also zoom in on regions of the assembled view. When the camera-module operating device300 is moved in front of the front face of therefrigeration appliance100, the corresponding image region moves on the camera-module operating device300, and thecold compartment105 that corresponds to the position of the camera-module operating device300 is displayed. While the camera-module operating device300 is moving over therefrigeration appliance100 on the outside, the displayed image segment follows the position of the camera-module operating device300. An image segment can be moved even when only one camera module is being used. All the regions are uniformly illuminated inside therefrigeration appliance100 in this context.

The complexity of software development generally depends on the additional functions required such as, for instance, image acquisition, image analysis, product recognition (EAN code), detecting changes in an image, a product database and using recognition to allocate the products in the image data.

In therefrigeration appliance100, it is possible to provide just thecamera modules101 and an interface to the camera-module operating device300. The camera-module operating device300 is used to control thecamera modules101 and to analyze the image data. Thus a minimum amount of control electronics can be implemented in therefrigeration appliance100. A user is not tied to outdated hardware but can replace the camera-module operating device300 as required. A door-opening and door-closing signal can optionally be transmitted for use by the camera-module operating device300 to control thecamera modules101. The camera-module operating device300 can transmit the image data by WLAN and display the contents of therefrigeration appliance100.

For example, the image data is obtained every time the door is closed, then provided or transmitted to an external storage location and then transmitted immediately on request. Illumination remains available for a period of 300 to 500 ms immediately after a door closure. This achieves the advantage of recording the latest contents after any changes, adding a small amount of extra heat after a large amount of heat input, and allowing illumination solely by standard interior lighting.

FIG. 16A to 16C show different connections ofcamera modules101. Thecamera modules101 and the controller and be integrated in different ways.

InFIG. 16A, thecamera module101 comprises acontrol element155 and acamera157 as an integral unit and is connected to anetwork161. InFIG. 16B, a plurality ofcamera modules101 having thecontrol elements155 are connected to thenetwork161 via an integral image processing unit or adistribution node125. InFIG. 16C, a multi-camera module is connected to thenetwork161. The multi-camera module can be connected to thenetwork161 using an integral or separateimage processing unit125. Themulti-control element155 is used to control a plurality ofcameras157 of the multi-camera module. An interface to aGSM network163 can also be provided. Thecamera157, thecontrol elements155, themulti-control element155 and theimage processing unit125 can be physically identical units that perform different functions.

Thecamera modules101 can be controlled by a processor in the refrigeration appliance or can be controlled separately from the refrigeration-appliance controller, for instance on the camera-module operating device300 such as a smartphone and mobile Tablet PC. The camera-module operating device300 having the screen can be attached to thedoor111.

The camera-module operating device300 can execute a program or an application (app), which is used to analyze the image data and to control thecamera modules101. For instance the application is used for determining freshness on the basis of the image data obtained. The freshness of therefrigerated goods103 can be distinguished from a ripeness of therefrigerated goods103.

In products which “feel the cold”, such as bananas, tomatoes, cucumbers, zucchini, egg plants and tropical fruits, the cell structure is affected by the cold. In general, weight, smell, taste, color, texture, for instance grooves, size, diameter or volume can be detected. The texture can be used to monitor whether a product is still crisp. Turgidity indicates the amount of water in the cells and is a measure of the freshness.

An application for determining the freshness acquires the information about products, analyzes this information and provides this information to other processing devices. For many products, the best before date is a reference date which is associated with the product and provided externally.

Unpackaged food does not have a best before date associated with the product. In addition, an alternative best before date can be used, which depends on the product, the time at which it was stored away, a refrigerator setting, a temperature graduation in thespecific refrigeration appliance100 and an air humidity. A traffic-light system is possible in this context:

- Red: best-before date cannot be identified
- Yellow: alternative best before date
- Green: best before date

Refrigerated goods such as meat may be packaged in opaque packaging, and therefore the condition of the meat cannot be monitored. The packaging can be labeled in a defined way, however, and may have a bar code and a best before date. Some products pose an increased health risk after expiry of the best before date, for instance products such as minced meat, poultry or fish products. The traffic-lights can be adapted for these products. To assess whether a product is still usable, the product specification contains a value that allows an assessment specifically according to the nature of the loss.


	Color

relative

absolute

Texture

	Food	e.g. red/brown	Weight	Size	Shape

Apple	+	+	+	+	+
Lettuce	+	−	+	(−)	+
Melon	+	+	−	+	−

The relative color can be assessed using a difference from the first image and a shade threshold value. Ripeness is identified from the shade of color.

The application can display guidance on storage in therefrigeration appliance100, for instance

- “This food should not be kept in the refrigerator”
- “Minced meat is on top shelf. Note: store on a lower shelf/in a colder area”
- or
- “Note: adjust fridge to cooler setting or adjust automatic function; less food waste outweighs electricity costs.”

Some products have a Time Temperature Integrator (TTI), which indicates a product-specific actuation. The defined colors of the TTI can be read by sensors. Dietary advice can be generated by connecting to a storage guide. When stocking therefrigeration appliance100, a stock list of all the items placed in the appliance can be created by passing these items across a camera module.

In addition, the application can use time-lapse to display the long-term change. In a virtual tour, an image of the contents of the refrigeration appliance can be displayed on the camera-module operating device300 by the application. The functions may be a combination of language and bar code (EAN/QR), image analysis and image display. Products can be added. Synchronization with a photo can be carried out, and shopping lists can be generated.

In addition, therefrigeration appliance100 can be designed such that a user can use a cell phone to request image data, making it possible to monitor freshness potentially even from remote locations. A GSM camera having MMS transmission can be used for this purpose. The camera and cell phone are connected via the GSM infrastructure. An SMS can be used to request an image, which is then transmitted as an MMS image.

Photos and videos can also be sent by email or MMS with a time stamp and date stamp. A photo or video can be sent on call at regular intervals. The photo resolution of the MMS for instance equals 160×120, 320×240, 640×480 pixels. If there is no GSM signal available, a recording is made in the internal memory. Power is supplied by battery, power pack or USB connection. Control can be performed directly at therefrigeration appliance100, at a personal computer or by SMS from the mobile phone.

The mobile phone can be used to allow convenient remote control by Android app. The GSM camera can be actuated via GSM by a call or SMS and supports SMS commands. In addition, a power failure message can be issued. The viewing angle of the camera equals 60° or 90° with magnetic attachable wide-angle lens. A flash function can also be implemented.

In addition, a UMTS/HSDPA camera (WCDMA/TD-SCDMA) with a photo resolution of 1.3 Megapixels can be used. The UMTS/HSDPA camera has a tilt function and pivot function and a night-vision function. A UMTS video server is arranged separately from the camera.

In addition, an RJ45-10/100M Ethernet interface with or without WLAN can be provided. Network-compatible camera-module operating devices300 are a laptop or television set for instance. It is thereby possible to separate multimedia components from refrigeration equipment. A multimedia interface may be a standard multimedia interface. In addition, optical fibers having a lens at one end can be used. The same color sensor can be used to read a plurality of optical fibers.

All the features illustrated and explained in association with the individual embodiments can be provided in various combinations in the subject matter according to the invention to achieve the advantageous effects of these embodiments simultaneously.

The scope of protection of the present invention is defined by the claims and is not restricted by the features explained in the description or illustrated in the figures.

LIST OF REFERENCES

100 refrigeration appliance
101 camera module
101-1 camera module
101-2 camera module
102 lighting device
103 refrigerated goods
104 microcontroller
105 cold compartment
107 pin-hole camera
109 side wall
111 door
113 compartment ceiling
115 sensor mat
117 compartment floor
119 processor unit
121 data bus
123 hinge
125 distribution node
127 camera array
129 rear wall
131 front face
133 circuit board
135 pin-hole lens
137 non-viewable region
139 viewable region
141 distortion
143 distortion
145 interface
147 image data
149 image data
151 subtraction
153 image region
155 control element
157 camera
161 network
163 GSM network
165 rack
167 door pillar
169 door handle
200 refrigeration appliance system
300 display device/camera-module operating device
303 touch-sensitive display
305 menu of options