US20110043631A1 - Use of video camera analytics for content aware detection and redundant storage of occurrences of events of interest - Google Patents

Abstract

Video analytics and a mass storage unit are contained in a camera housing of a video camera. Video data representing a field of view of a scene observed by the camera are stored in the mass storage unit. The video analytics analyzes video data produced by the camera and detects whether there is an occurrence of a defined event of interest. A video clip of the scene representing the event of interest is sent to a remote storage unit. Since only about 1% of security video data is reviewed, storing only video data representing events of interest remotely, while storing more complete video data of the scene observed by the camera local to the camera, reduces the remote storage capacity and bandwidth demand for the video system. Remote redundant storage of events of interest also provides higher reliability and fault tolerant storage for the video data that are most important.

Description

RELATED APPLICATIONS
• This application is a continuation of U.S. patent application Ser. No. 12/105,871, filed Apr. 18, 2008, and claims the benefit under 35 U.S.C. §119(e) of U.S. Provisional Application No. 61/033,290, filed Mar. 3, 2008.
TECHNICAL FIELD
• This disclosure describes a video imaging system that intelligently recognizes the content of video data, reduces system storage and bandwidth capacity demands, and prolongs the operational lifespan of video data mass storage units.
BACKGROUND INFORMATION
• Network camera systems, for example network surveillance camera systems or IP camera systems, have existed for a number of years but have undergone relatively slow industry adoption. Compared to traditional analog camera systems, network camera systems offer advantages such as accessibility, integration, low installation costs, scalability, and an ability to move to higher resolution video. Data produced by network cameras, however, demand large amounts of bandwidth and storage capacity.
• Bandwidth problems associated with network camera systems have lead to more complex camera networks that include an increased number of switches and, in some cases, complete alternative data paths. Storage problems associated with network camera systems become magnified as video resolution and the number of cameras in a system increase. For example, a single standard D1 resolution camera using MPEG-4 compression and operating at 30 frames-per-second (fps) can require 360 gigabytes (GB) of storage for video data representing one month of video data. A camera system with 1000 cameras, therefore, would require 360 terabytes (TB) of storage for data spanning one month. This example demonstrates a huge cost and facility management challenge presented with network camera systems, especially where mega-pixel resolution is desired and where applications require six months or a year of video data storage. Due to the problems identified, most network video data are not recorded at full quality, but are recorded at lower resolutions and frame rates. Because typical high resolution cameras generate video data requiring a large amount of storage resources within a short period of time, it is impractical for a typical camera to include a self-contained storage unit, such as a hard drive, that is able to store a significant amount of video data.
• Typical storage architecture of network camera systems is configured with central storage similarly to traditional analog systems. The architecture includes centrally located digital video recorders (DVRs) or network video recorders (NVRs) connected through a network to IP cameras. The typical architecture for IP cameras is inadequate for a number of reasons. If, for example, the network fails or is made nonoperational for maintenance or any other reason, all video is lost and can never be retrieved. Numerous (e.g., many dozens of) cameras streaming across the network to a central storage device place severe bandwidth demands on the network. Moreover, 99% of the bandwidth used is wasted because typically less than 1% of the video is ever accessed for review. Additionally, typical network camera systems often lack storage scalability such that, as network camera systems expand, central storage systems require “forklift” upgrades.
• Another problem with typical video data storage configurations is that many applications require storage devices to continuously run. Such continuous operation causes the storage devices to fail after three to five years of operation. Unless archived or stored redundantly, data on failed storage devices become lost. The need to replace storage devices, therefore, becomes a significant concern and maintenance issue.
• Recently, some network camera systems have implemented video analytics processing to identify when important events (such as object movement) are being captured by a video camera. Video analytics has been primarily used to alert security of potential unwanted events. Most video analytics is performed by a central processor that is common to multiple cameras, but some video cameras have built-in video analytics capabilities. These video cameras with built-in analytics, however, have not included large capacity storage due to the large storage requirements of the video data generated by the camera and the traditional approach of centralized storage. Also, there are some cameras configured without built-in video analytics but with built-in small storage capacity that is insufficient to serve as a substitute for traditional DVRs and NVRs. Moreover, if the video data are stored only in the camera, the stored video data are vulnerable to attack or being stolen.
• Therefore, a need exists for a network camera system that produces high quality video data, requires less storage capacity and network bandwidth, meets IT standards, is easily scalable, and operates for a longer period of time without storage device replacement.
SUMMARY OF THE DISCLOSURE
• The disclosed preferred embodiments implement methods and systems of content aware storage of video data produced by a video camera, which includes a camera housing and is adapted for connection to a network communication system. The video data produced represent a field of view of a scene observed by the video camera. Video analytics and a mass storage unit are contained in or form part of the camera housing. The video analytics analyzes the video data produced by the video camera and detects whether there is an occurrence of an event of interest. The video data representing the field of view of the scene observed by the video camera are stored in the mass storage unit. The stored video data include video data of a first quality and video data of a second quality. The first quality represents the occurrence in the field of view of the event of interest detected by the video analytics, and the second quality represents nonoccurrence in the field of view of the event of interest detected by the video analytics. By storing video data in the mass storage unit contained in or forming part of the camera housing, the majority of network bandwidth requirements are eliminated because the video data need not be streamed across the network for storage purposes.
• The implementation described above reduces video data storage and network bandwidth requirements of a distributed network video surveillance system that includes network communication paths between network video imaging devices and network video data stores. In such surveillance system, the network video imaging devices produce video data representing fields of view of scenes under observation by the video imaging devices, and the network video data stores store video information corresponding to the video data produced by the network video imaging devices. Each of multiple ones of the network video imaging devices is associated with a content-aware video data storage system that is capable of selective storage of video data produced by its associated network video imaging device. The content-aware video data storage system includes video analytics that analyzes the content of the video data and local video data stores that store portions of the video data in response to the analysis by the video analytics. Video data corresponding to the portions of video data are delivered through the network communication paths to the network video data stores to provide a managed amount of video data representing at a specified quality level the fields of view of the scenes. The managed amount of the video data consumes substantially less network bandwidth and fewer data storage resources than those which would be consumed by delivery to the network video stores the video data produced by the network video imaging devices at the specified quality level and in the absence of analysis by the video analytics. While video surveillance applications are of particular interest, the above approach is applicable across a wide variety of video applications.
• Additional aspects and advantages will be apparent from the following detailed description of preferred embodiments, which proceeds with reference to the accompanying drawings.
BRIEF DESCRIPTION OF THE DRAWINGS
• FIG. 1 depicts an embodiment of a network camera system.
• FIG. 2 is a high level block diagram of a network camera ofFIG. 1.
• FIG. 3 is a block diagram depicting the imaging system, video processing system, and data storage system ofFIG. 2 according to a first embodiment.
• FIG. 4 is a block diagram depicting an access control management unit operating in the video processing system of the first embodiment ofFIG. 3.
• FIG. 5 is a block diagram depicting a second embodiment of the imaging system, video processing system, and data storage system ofFIG. 2.
• FIG. 6 is a block diagram depicting portions of the video processing system of the second embodiment.
• FIG. 7 is a block diagram representing a memory buffer unit and a hard drive storage unit of data storage system.
• FIG. 8 is an image of a bird perched on a birdfeeder, in which image the bird and birdfeeder are displayed as high quality images and a background scene is displayed in low quality.
DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS
• System components with like reference numerals perform the same functions in each of the embodiments of a content aware storage system described below.
• FIG. 1 is a pictorial diagram depicting an embodiment of anetwork camera system100 utilized in an application with local campus buildings and remote sites.Network camera system100 is not limited to video surveillance or to the application depicted, but may be used in any network communication system.Network camera system100 includesnetwork cameras102 connected to acentral monitoring station104 through anetwork106 that includes a wide area network (WAN)108 and a campus local area network (LAN)110.Network106 may also include awireless network112 that includesnetwork cameras102′ with wireless communication capabilities.Network106 establishes multiple network communications paths. The following descriptions ofnetwork camera102 apply also to networkcamera102′.Network106 is not limited to the configuration depicted, but may include various configurations and types of networks. Aremote user114 may also be connected to networkcameras102 throughWAN108.Network cameras102 may be connected to a remote storage unit116 (i.e., a network data store).Network camera system100 may also includevarious switches118 androuters120 to facilitate communication overnetwork106.
• In operation,network cameras102 capture various fields of view and generate data representing the fields of view. Certain applications may require substantially continuous operation ofnetwork camera102. The data are communicated tocentral monitoring station104, in which a user may view images, generated from the data, depicting the fields of view. Also, the data may be communicated toremote user114 to generate images of the fields of view. The data may be stored inremote storage unit116 and later accessed by a user.
• Network camera102 will now be described in more detail with reference toFIG. 2.Network camera102 includes animaging system202, avideo processing system204, a data storage system206 (i.e., a local data store), apower system208, and an input/output interface andcontrol system210.Network camera102 includes a camera housing; and all or portions ofsystems202,204,206,208, and210 may be contained within the housing.Imaging system202 may include a wide variety of units for capturing a field of view and for generating video information including digital data and analog signals. For example,imaging system202 may generate information according to NTSC/PAL formats and mega-pixel formats.Imaging system202 may include programmable imagers, high-definition imagers, no/low light sensors, and specialized imagers that are more sensitive to certain spectrums of light.Imaging system202 may include a scalable video codec, such as MPEG-4 SVC, and other video compression capabilities, such as H.264compression. Power system208 may include any system for receiving and distributing electrical power to various systems ofnetwork camera102. Power may be DC power, including Power over Ethernet (PoE), or AC power. Input/output interface andcontrol system210 includes various hardware and software configurations to facilitate numerous types of communication including Internet; Ethernet; universal serial bus (USB); wireless; asynchronous transfer mode (ATM); Packet over SONET/SDH (POS); pan, zoom, tilt (PZT); and audio information. Input/output interface andcontrol system210 may be implemented in hardware and software to allow a user to configure operation ofnetwork camera102.
• In an alternative embodiment, as depicted inFIG. 1, avideo server122 may be used in place ofnetwork camera102, in whichmultiple imaging systems202 capturing different fields of view are connected tovideo server122.Video server122 includes, within a server housing,video processing system204,data storage system206,power system208, and input/output interface andcontrol system210. For clarity,network camera102 will be referred to in the following descriptions, but the following descriptions are also applicable to situations in whichmultiple imaging systems202 are connected tovideo server122.
Content Aware StorageFirst Embodiment
• A first embodiment ofnetwork camera102 is described in more detail with reference toFIG. 3.Video processing system204 includes a rules basedengine302,video analytics304, and astorage management system306, some or all of which may be implemented in software.Video analytics304 includes video analytics software operating in a video analytics processor. Although video analysis and other video processing described in the following embodiments are performed byvideo processing system204, video data may also be supplied fromnetwork camera102 to a network-connected video processor, such as a video server (not shown), that performs all or part of the video analysis and other video processing described below. In other words, video analysis and processing may be distributed throughoutnetwork camera system100.Video processing system204 may also include video encryption capabilities to prevent unauthorized viewing of video information.Imaging system202 captures a field of view and generates video data representing the field of view.Imaging system202 may be programmable and may be capable of producing multiple quality levels of video data, including higher quality (HiQ) video data and lower quality (LowQ) video data. A quality level refers to multiple video parameters including resolution, frame rate, bit rate, and compression quality. For example, HiQ video data may represent D1 resolution video recorded at 30 frames-per-second (fps) and LowQ video data may represent CIF resolution video recorded at 5 fps. HiQ and LowQ video data are not limited to the parameters above. HiQ video data may represent D1 resolution video recorded at a lower frame rate—for example, 15 fps. In general, HiQ video data are video data that represent higher quality video than LowQ video data. HiQ video data are characterized by large storage requirements, and LowQ video data are characterized by small storage requirements.Imaging system202 may produce more than two quality levels of video data.Imaging system202 may be capable of producing different quality levels for different portions of a field of view within a video frame. For example,imaging system202 may generate HiQ quality video data representing a person in the field of view while simultaneously generating LowQ video data representing background scene images of the field of view. As a further example,FIG. 8 depicts a bird perched on a birdfeeder in high resolution while the background scene is represented in low resolution.
• Imaging system202 communicates video data tovideo analytics304.Video analytics304, via the video analytics engine, analyzes the video data produced byimaging system202 to detect whether a predefined event or object of interest is being captured byimaging system202. The video data analyzed byvideo analytics304 is preferably HiQ video data.Video analytics304 generates metadata that describe the content of video data. The metadata produced byvideo analytics304 may be a textual and semantic description of the content of the video.
• Events and objects of interest may be programmed by a user and specified in an XML definitions file. The definitions file andvideo analytics304 may be updated periodically, and definition files may be shared betweenvideo analytics304 ofdifferent network cameras102 withinnetwork camera system100.Video analytics304 ofdifferent network cameras102 may have different analytic capabilities. Multiple events of interest may be defined, and more than one event of interest may occur at a particular time. Also, the nonoccurrence of one event leaves open the possibility of the occurrence of a second event. The metadata may be supplied todata storage system206 andremote storage unit116 for storage. The metadata representing an arbitrary frame n can be associated with video data representing frame n. Thus, the metadata may be searchable to allow a user to efficiently search and semantically browse large video archives, whether stored locally or remotely.
• An event of interest thatvideo analytics304 detects may be as simple as motion in the field of view.Video analytics304 may also implement blob detection (e.g. detecting a group of moving pixels as a potential moving object, without identifying what type of object it is), lighting change adjustment, and geometric calibration based on object size in the field of view to distinguish objects based on types. For example,video analytics304 may be able to classify an object as a human being, a vehicle, or another type of object and be able to recognize an object when the object appears in any portion within the field of view ofnetwork camera102. Furthermore,video analytics304 may be able to recognize certain identifiable features of an object such as, for example, human faces and vehicle license plates.Video analytics304 may be able to recognize whenimaging system202 is capturing a new object and assign a unique object ID to the new object.Video analytics304 may be able to recognize the speed and trajectory at which an object moves.Video analytics304 may be able to recognize events such as perimeter intrusion, object movement in a particular direction, objects approaching one another, a number of objects located in a specified area, objects left behind, and object removal.Video analytics304 can also recognize specific locations, or coordinates, within the field of view where an event or object of interest is being captured, or a combination of objects and events, as defined by a rule.
• Whenvideo analytics304 detects an event or object of interest within the video data,video analytics304 generates metadata that correspond to the event or object of interest and supplies the metadata to rules basedengine302. Rules basedengine302 includes rules that associate events or objects of interest, specified in the metadata, to specific actions to be taken. The actions associated with the rules may be to perform, for example, one or more of the following: store HiQ or LowQ video data indata storage system206, store HiQ or LowQ video data inremote storage unit116, stream HiQ or LowQ video data tocentral monitoring station104 orremote user114, generate and send a short video clip file of the event of interest tocentral monitoring station104 orremote user114, send an alert (e.g., instructions to generate one or both of a visual display and an audible sound) tocentral monitoring station104 orremote user114, store video data indata storage system206 for X period of time. For example, a user may define the following rule: when a human being enters a defined perimeter, store indata storage system206 HiQ video data representing the intrusion, alertcentral monitoring station104 of the intrusion, generate a short video clip of the intrusion and send the video clip tocentral monitoring station104, and store inremote storage unit116 HiQ video data representing the intrusion. Or, a user may define the following rule: when no event or object of interest is being captured, store indata storage system206 LowQ video data and send no video data tocentral monitoring station104. Becausevideo analytics304 can detect various objects and events, a wide variety of rules may be defined by a user and each rule can have different storage quality settings. Also, because multiple events of interest may occur simultaneously, a rule may correspond to a combination of events.
• Storage management system306 may control storage of video data indata storage system206 and inremote storage unit116.Storage management system306 is intelligently driven by the metadata generated invideo analytics304 and the rules defined in rules basedengine302.Storage management system306 implements the actions defined by the rules. For example,storage management system306 communicates toimaging system202 to generate HiQ and/or LowQ video data to be stored indata storage system206 andremote storage unit116. Becausevideo analytics304 can specify locations, or coordinates, within the field of view where an event or object of interest is being captured,storage management system306 can communicate toimaging system202 which portions of the field of view are to be represented with HiQ video data (portions corresponding to events or objects) and LowQ video data (remaining portions). For example,FIG. 8 depicts a scene of a bird perched on a birdfeeder.Video analytics304 can recognize the bird and portions of the birdfeeder as the most important features of the image or as objects of interest. The bird and birdfeeder, as the objects of interest, are displayed as HiQ images, while the background scene is displayed as a LowQ image. Also,imaging system202 may be controlled such that a “windowed view” of the event or object of interest is generated by creating video data in which only the portion of the field of view corresponding to the event or object is displayed. Because HiQ and LowQ video data may be intelligently generated based upon content, events or objects of interest may be captured and stored as HiQ video data while overall storage requirements are lowered by generating LowQ video data to represent scenes in which no event or object of interest is captured.
• In an alternative embodiment,imaging system202 generates one quality level of video data to be stored indata storage system206.Network camera102 includes a scalable video codec, such as MPEG-4 SVC. After the video data are analyzed byvideo analytics304 and stored indata storage system206, portions of the video data may be processed using the scalable video codec to generate a second quality level (multiple quality levels may be generated using the SVC). For example,network camera102 generates anddata storage system206 stores HiQ video data. Some time later (e.g., minutes, hours, days), the quality level of portions of the HiQ video data that represent the nonoccurrence of an event of interest are reduced to LowQ.
• Storage management system306 can also implement storage management policies that dictate how long portions of video data are stored indata storage system206 based upon content. For example,storage management system306 can controldata storage system206 such that important events are retained for long periods of time while less important video data are replaced with new video data within a short period of time.Storage management system306 also controls communication of video data between sub-storage units ofdata storage system206 as described below. One goal ofstorage management unit306 is to minimize the frequency of writing operations from a first sub-storage unit to a second sub-storage unit.
• Because video data generated bynetwork camera102 are stored indata storage system206 within the camera housing ofnetwork camera102, the video data may be more vulnerable to damage or theft. For example, if an intruder stealsnetwork camera102, the intruder would also have in his possession the video data.
• Becausenetwork camera102 includesvideo analytics304 anddata storage system206, numerous features may be implemented insystem100 to secure video data from loss or unauthorized viewing in the event that networkcamera102 is stolen. For example, when an event of interest (e.g., detection of an intruder) is detected byvideo analytics304, the video data representing the event of interest may be immediately streamed, or sent as video files, toremote storage unit116, or to anothernetwork camera102, for redundant storage. Also, shortly after the event of interest is detected, an alert and a video clip file representing the event of interest may be sent tocentral monitoring station104 orremote user114 beforenetwork camera102 is tampered with. To prevent an intruder from viewing images captured bynetwork camera102, video data stored indata storage system206 is encrypted so that the intruder cannot play back the video data. Also, video data streamed or sent as video files fromnetwork camera102 may be encrypted to prevent unauthorized viewing.
• Imaging system202,video analytics304, rules basedengine302,storage management system306, anddata storage system206 cooperate to establish a content aware storage system. The content aware storage system provides a number of unique benefits not available in traditional camera systems (even those camera systems that include some form of video analytics or small capacity storage). With the content aware storage system, storage capacity needs can be greatly reduced by intelligent recognition and classification of video content. Storage capacity needs can be greatly reduced even for applications that require substantially continuous operation ofnetwork camera102. For example, when an event of interest is captured, the content aware storage system can record the event at a HiQ level. When an event of interest is not being captured, the content aware storage system can record the video data at a LowQ level. The quality level of stored data, therefore, can be matched to the importance of the content.
• Because LowQ video data may be stored when no event or object of interest is being captured,data storage system206 may include practical storage capacity, for example 80 GB, and still be able to store video data spanning long periods of time (for example, one or two months). In comparison, a typical D1 resolution30 fps system without content aware storage can require over 360 GB of storage for one month. Thus, typical video cameras have not been able to include a mass storage unit that can store video data spanning a long period of time. Also, becausenetwork camera102 includesdata storage system206, video data may be stored despite network failure or network downtime due to system upgrades or maintenance. Separate networks no longer need to be set up for network cameras; network cameras can be installed on the same data network used at a particular site, saving installation costs and ongoing maintenance costs. Also, becausenetwork camera102 includesdata storage system206, the capacity ofremote storage unit116 may be greatly reduced andremote storage unit116 may serve primarily as backup or archival storage of important events. Also,data storage system206 eliminates the need to include traditional DVRs and NVRs innetwork camera system100.
• Additionally, becausenetwork camera102 includesdata storage system206, network bandwidth demands can be greatly reduced becausenetwork camera102 does not have to continuously supply video data overnetwork106 toremote storage unit116. Instead,network camera102 can supply a managed amount of video data toremote storage unit116. For example,network camera102 may supply HiQ or LowQ video data overnetwork106 only when an event or object of interest is being captured. For example, events or objects of interest may be captured only ten percent or less of the time in a typical camera system. During the other 90% of the time, a user may choose to send only LowQ video data overnetwork106, or to send no video data at all. Forwireless network112, because network bandwidth demands are lower, morewireless network cameras102′ can be added towireless network112.
• Becausevideo analytics304 can detect when an event or object of interest is being captured, the data and metadata associated with the event or object of interest can be automatically archived inremote storage unit116 to provide added redundancy and fault tolerance. Transmission of alarm information and video data tocentral monitoring station104 orremote user114 may also be prioritized based on importance of the video content.
• Also, because of the content aware storage system, users can categorize different events or objects of interest by assigning priority values. The video data associated with the events or objects of interest can be stored intelligently indata storage system206 for preset time periods that vary based upon the priority values. For example, less important events may be deleted after one month, but more important events may be stored for three months, six months, or a year. Also, when combined with the scalable video codec capabilities ofimaging system202, video data can be retained indata storage system206 but reduced according to different resolutions and frame rates based upon the video content so that the video data take up less space.
• Becausevideo analytics304 generates metadata that can be stored indata storage system206 andremote storage unit116, access to video data stored indata storage system206 andremote storage unit116 may be controlled based on content. Also, access to live video data may be controlled as metadata is created corresponding to the video data. As depicted inFIG. 4,video processing system204 may include an accesscontrol management unit402, which is preferably implemented in software. According to rules in rules basedengine302, different content security levels are assigned to different events or objects of interest so that access to video data may be controlled according to content. Also, different users have one or more security levels assigned to them—the security levels corresponding to one or more of the content security levels. Accesscontrol management unit402 controls access to stored video data such that a user may access only the video data that include a content security level corresponding to the user's security level. Security managers, for example, may access video data flagged for security breaches or threats, but may be prevented from accessing video data that have been captured for business or marketing purposes. Likewise, marketing personnel may access video data identified for their applications but not access video security data. Policies on video encryption may also be controlled based upon content.
• The content aware storage system may also intelligently distribute stored video data to maximize the available storage capacity. For example, to meet its storage needs, afirst network camera102 may need only one-half of the capacity of itsdata storage system206, while asecond network camera102 may require more storage capacity than the capacity of itsdata storage system206. Video data from thesecond network camera102 may be supplied overnetwork106 to thefirst network camera102 to be stored therein. Becausedata storage system206 of one network camera can store data of another network camera, the total storage capacity in thesystem100 can be maximized and redundant storage of important data may be distributed throughoutsystem100. Maximizing total storage capacity and distributing redundant storage of important data make the important data more immune to tampering or failure. Additionally, storage transfer could take place at low bandwidth times of the day.
• The content aware storage system also allowsnetwork camera system100 to be easily scalable. In conventional systems, as the number of cameras increases, storage capacity must increase by adding units to a remote storage facility. Moreover, processing power must increase by adding units to a central processing facility. With a content aware storage system, remote storage and processing facilities do not require upgrades asnetwork cameras102 are added tonetwork camera system100. Instead, eachnetwork camera102 contains its own storage capacity (via data storage system206) and processing power (via video processing system204). Thus, when anetwork camera102 is added tonetwork camera system100, the storage capacity and processing power increase simultaneously.
Second Embodiment
• A second embodiment ofcamera102 is described with reference toFIG. 5 and includesimaging system202,video processing system204′, anddata storage system206.Video processing system204′ of the second embodiment includesvideo analytics304 and animage processing unit502.Image processing unit502 generates video data to be communicated tocentral monitoring station104 and stored indata storage system206 andremote storage unit116.Image processing unit502 may be capable of compressing D1 resolution video data according to the H.264/AVC standard at 30 fps.Image processing unit502 may be, for example, a Freescale Semiconductor® i.MX27 multimedia applications processor.Video analytics304 analyzes data to determine whether the data contain predetermined types of content.Video analytics304 may be capable of performing MPEG4/CIF encoding.Video analytics304 may also deliver video data to a PAL/NTSC monitor (not shown).Video analytics304 may implement, for example, a Texas Instruments DaVinci™ DM6437 Digital Video Development Platform. Additionally,image processing unit502 andvideo analytics304 may communicate with each other through aninter-processor bridge506.
• Video data generated byimaging system202 are supplied toimage processing unit502 andvideo analytics304. Each frame of video represented in the video data receives a time stamp, which is applied by a time-stamp block508 that is positioned betweenimaging system202 andimage processing unit502 andvideo analytics304.
• Image processing unit502 will now be described in more detail with reference toFIG. 6.Image processing unit502 includes afirst encoder602, asecond encoder604, streaming andarchive control unit606, rules basedengine302, accesscontrol management unit402, andstorage management system306. Streaming andarchive control unit606, rules basedengine302,access management unit402, andstorage management unit306 may be implemented in software. First andsecond encoders602 and604 may be implemented in hardware as application specific integrated circuitry or in software. First andsecond encoders602 and604 receive the video data generated byimaging system202. First andsecond encoders602 and604 encode the video data at two different quality levels. As described above, a quality level refers to a number of video processing parameters including resolution, frame rate, bit rate, and video compression quality.First encoder602 encodes the video data at a HiQ level to produce HiQ video data, andsecond encoder604 encodes the video data at a LowQ level to produce LowQ video data.
• Although two encoders are depicted inFIG. 6,image processing unit502 may include more than two encoders to produce several streams video data of multiple quality levels, or the quality levels of a single encoder may change based on the type of event detected byvideo analytics304. Preferably,encoders602 and604 can encode video data simultaneously. For example,encoder604 encodes the video data continuously andencoder602 encodes the video data only when an event of interest is detected such that, when an event of interest is detected,encoder602 encodes the event at a LowQ level whileencoder604 encodes the event at a HiQ level. Continuous recording of LowQ video data and intermittent recording of HiQ video data substantially reduce storage resource requirements. For example, a 24-hour period may include approximately five hours in which an event of interest is being detected. Ifnetwork camera102 recorded five hours of D1 resolution video at 15 fps using H.264 compression, those five hours would require approximately 0.86 GB of storage. Ifnetwork camera102 also recorded 24 hours of CIF resolution video at five fps using H.264 compression, the 24 hours would require approximately 0.4 GB of storage for a total of 1.26 GB/day. Ifdata storage system206 ofnetwork camera102 included 80 GB of storage,data storage system206 could record approximately two months of video. In comparison, a typical camera that continuously records D1 resolution video at 15 fps using MPEG-4 compression requires approximately 5.4 GB of storage per day. Thus, a typical 16 channel video system requires approximately 5 TB of storage for a two-month period.
• Continuous recording at LowQ is desirable incase video analytics304 misses the occurrence of an event of interest. In many cases, it is necessary to have a capability of verifying that no event happened during a period of time, such as, for example, a situation in which a customer claims to have slipped and fallen in a retail store when in fact the customer did not slip and fall. The LowQ video data can be reviewed by a user to view the event missed byvideo analytics304. Thus, LowQ video data can be continuously recorded to ensure that nothing has been missed, while HiQ video data are recorded when important events have been detected byvideo analytics304.
• One or both streams of HiQ and LowQ video data are supplied todata storage system206 and streaming andarchiving control unit606, depending on video content and defined rules of rules basedengine302. Streaming andarchiving control unit606 facilitate communication of live HiQ and LowQ video data, video data stored indata storage unit206, video clip files representing events of interests, and alarms tocentral monitoring station104,remote storage unit116, andremote user114. For example, after an event of interest is captured and stored indata storage system206, HiQ video data representing the event of interest stored indata storage system206 may be communicated toremote storage unit116, via streaming andarchiving control unit606, for redundant storage of the event of interest.
• Rules basedengine302 receives metadata produced byvideo analytics304 and determines from the metadata whether the HiQ or LowQ video data are to be stored indata storage system206 and/or communicated tocentral monitoring station104,remote storage unit116, andremote user114.
• Accesscontrol management unit402 controls access to live and stored video data. Accesscontrol management unit402 allows multiple user accounts to be created with corresponding permissions to access portions of video data based upon the content of the video data.
• Storage management system306 is intelligently driven by the metadata generated invideo analytics304 and the rules defined in rules basedengine302.Storage management system306 also controls communication of video data between sub-storage units ofdata storage system206.
• In operation,imaging system202 captures a field of view and generates video data. Frames of the video data are time-stamped in time-stamp block508 so that metadata generated byvideo analytics304 may be synchronized with video data generated byimage processing unit502.Video analytics304 analyzes the video data generated byimaging system202 and generates metadata based upon the content of the video data.First encoder602 andsecond encoder604 also receive the video data generated byimaging system202 and generate, respectively, HiQ video data and LowQ video data.
• The metadata is communicated to rules basedengine302 overinter-processor bridge506 and rules basedengine302 determines whether a rule has been violated (i.e., whether an event or object of interest detected byvideo analytics304 requires action). Based on the metadata and the rules of rules basedengine302,storage management system306 controlsfirst encoder602 andsecond encoder604 to supply HiQ video data and/or LowQ video data todata storage system206. Portions of the HiQ and LowQ video data may be segmented into video clip files.Storage management system306 also controls whether to send HiQ or LowQ video data, via streaming andarchiving control unit606, to centraldata storage unit116.
• For example, if the metadata communicates that no object or event of interest is being captured within the field of view ofnetwork camera102, a rule may communicate tostorage management system306 to control first andsecond encoders602 and604 such that the LowQ video data are supplied todata storage system206 and no video data are supplied toremote storage unit116. In an alternative example, if the metadata communicates that an object or event of interest is occurring within the field of view ofnetwork camera102, a rule may communicate tostorage management system306 to control first andsecond encoders602 and604 such that HiQ and LowQ video data representing the object or event are supplied todata storage system206. Because both HiQ and LowQ video data are stored indata storage system206, the event of interest may be played back in a higher bandwidth or lower bandwidth mode. Also, when an event of interest is captured, a video clip file of the event of interest stored indata storage system206 may be communicated tocentral monitoring station104 via streaming andarchiving control unit606. A video clip file may be a short HiQ video segment of the event of interest that is stored indata storage system206. A video clip file may represent seven seconds of video in which two seconds are before the event occurs and five seconds are after the event is detected. The duration of the video clip may be programmed to be any length. The video clip file may be played back multiple times, forwarded to other users via e-mail, or stored on a removable disk and sent to, for example, law enforcement. As described in the first embodiment, users may define any combination of actions to be taken based upon the metadata, such as storing HiQ video data indata storage system206 andremote storage unit116 while simultaneously streaming LowQ video data tocentral monitoring station104. Users may also define a rule that an alert be sent tocentral monitoring station104 or toremote user114 via e-mail whenever an event or object of interest is detected. For example,remote user114 may receive an alert on a mobile device such as a cellular phone or personal digital assistant (PDA), together with a video clip that can be played on the mobile device.
• Although the above embodiments have been described,network camera102 is not limited to these two embodiments.Network camera102 may include a camera system having any degree of resolution and having a capability of analyzing the content of video data to detect motion or another event of interest, storing the video data in the camera, and sending video data of the events of interest to remote storage.
Data Storage System
• Data storage system206 is described in detail with reference toFIG. 7.Data storage system206 may be included innetwork camera102, ordata storage system206 may be external tonetwork camera102 and may communicate withnetwork camera102 overnetwork106.Data storage system206 serves as a mass storage unit that includes at least one GB, preferably 80 GB or more, of storage capacity.Data storage system206 includes a solid statememory buffer unit702 and a harddrive storage unit704.
• Memory buffer unit702 may include non-volatile and volatile memory, such as NAND flash memory and RAM. If volatile memory is used asmemory buffer unit702, a secondary power supply may be included innetwork camera102 in case of power failure. In a preferred embodiment, one GB of NAND flash memory is used, butmemory buffer unit702 may include a memory size larger or smaller than one GB. Sectors ofmemory buffer unit702 may be allocated for different types of data, as indicated byportions706,708, and710. For example,portion706 represents 50% of memory allocated to store the most recent video data captured byimaging system202.Portion708 represents 40% of memory allocated to store the most recent events or objects of interest captured byimaging system202.Portion710 represents 10% of memory allocated to store metadata generated byvideo analytics304. Allocation of memory is not limited to the above example and can be adapted to meet the needs of a particular application. Also,portions706,708, and710 may be periodically rotated to correspond to different sectors ofmemory buffer unit702 to prolong the operating lifespan of the sectors ofmemory buffer unit702.
• Harddrive storage unit704 may be any type of mass storage device including a hard-disk drive and a large capacity solid state memory device. For simplicity, harddrive storage unit704 is described herein as a hard-disk drive, but many of the features described are also applicable to a large capacity solid state memory device. The storage capacity of harddrive storage unit704 may be any size, but, preferably, the storage capacity would be 80 GB or more. Harddrive storage unit704 includes a read/write head and a storage memory disk.
• In operation, during a first time interval,memory buffer unit702 receives video data fromimaging system202 orvideo processing system204′ and stores the video data inportions706 or708, as directed bystorage management system306, according to the content of the video data.Memory buffer unit702 also receives metadata generated byvideo analytics304 and stores the metadata inportion710. Because the metadata is synchronized to the video data, the video data can be quickly searched according to content by referring to the metadata. For the majority of time that video data and metadata are being stored inmemory buffer unit702, harddrive storage unit704 is in a powered-down state. A powered-down state includes multiple states such as a total power-off state or one of several idle, standby, or sleep states described in connection with Hitachi's Enhanced Adaptive Battery Life Extender (ABLE)™ technology. For example, in a powered-down state, the read/write head may be in a “loaded” or activated condition without executing read/write commands while the storage memory disk is spinning, the read/write head may be in a “loaded” or activated condition while the storage memory disk is not spinning, the read/write head may be in an “unloaded” or non-activated condition while the storage memory disk is spinning, or the read/write head may be in an “unloaded” or non-activated condition while the storage memory disk is not spinning. Generally, a powered-down state is characterized by a power level less than a power level of a powered-up state. A total memory buffer storage time represents a sum of the first time intervals.
• During a second time interval determined bystorage management system306, harddrive storage unit704 is powered-up (i.e., the read/write head is in an activated condition to execute read/write commands and the storage memory disk is spinning) and video data from one ormore portions706 and708 are communicated frommemory buffer unit702 to harddrive storage unit704 to be stored in harddrive storage unit704. Metadata fromportion710 may also be communicated to harddrive storage unit704 during the second time interval.Storage management system306 determines the amount of data that are to be supplied frommemory buffer unit702 and written to harddrive storage unit704. During the second time interval,memory buffer unit702 may continue to receive and store video data and metadata to prevent interruption in the storing of video data and metadata. At the end of the second time interval (e.g., after a predetermined amount of data ofmemory buffer unit702 are written to hard drive storage unit704)memory buffer unit702 stops supplying data to harddrive storage unit704 and harddrive storage unit704 is powered-down. A total hard drive storage time represents a sum of the second time intervals. The total hard drive storage time spent transferring to and storing a portion of the video data in harddrive storage unit704 is substantially less that the total memory buffer storage time spent storing the video data inmemory buffer unit702.
• Storage management system306 may control a write operation frommemory buffer unit702 to harddrive storage unit704 at any time such as when a predetermined condition is satisfied. For example, a predetermined condition might be to perform the write operation when one ofportions706,708, or710 is near capacity. Or, in mobile applications such as a camera in a vehicle, a write operation may be performed when a motion sensor, accelerometer, or other sensor located within the vehicle indicates that the vehicle is stationary so that damage to the harddrive storage unit704 may be avoided or to eliminate the need to include a hard drive with extensive shock absorption. The motion sensor, accelerometer, or other sensor may be contained within the camera housing. A sensor includes a vehicle on/off switch. For example, when a vehicle is turned off, the vehicle battery may keepdata storage system206 powered-up so that video data may be transferred frommemory buffer unit702 to harddrive storage unit704 during the off state of the vehicle—ideally, the vehicle is stationary during an off state. The write operation may be stopped when all the data from one ormore portions706,708, and710 have been written to harddrive storage unit704 or when another condition is met, such as vehicle movement. Additionally,storage management system306 may control harddrive storage unit704 to be powered-up when a user requests retrieval of video data stored in harddrive storage unit704. Harddrive storage unit704 would be powered-up to supply video data to video processing system204 (204′) so that the video data may be supplied to a user overnetwork106.
• In mobile applications, a singledata storage system206 may servemultiple imaging systems202 of a vehicle (e.g., multiple cameras on a bus), or one harddrive storage unit704 may servemultiple imaging systems202 that include their ownmemory buffer units702. When the vehicle comes to a stop, harddrive storage unit704 is powered-up and data is transferred from memory buffer unit(s)702 quickly by optimizing the communication speed between memory buffer unit(s)702 and the harddrive storage unit704 and by implementing a harddrive storage unit704 with a fast write speed (e.g., approximately 665 Mbits/sec or more). Also, when a vehicle includesmultiple imaging systems202, data transfer can be accomplished quickly by lowering the frame rate of eachimaging system202 without substantially sacrificing video quality. For example, if a vehicle included eight cameras operating at 7.5 fps, the video data generated by the eight cameras would be equivalent to data generated by two cameras operating at 30 fps.
• By storing video data inmemory buffer unit702 and by powering-up harddrive storage unit704 periodically, the operating life of the harddrive storage unit704 may be extended because harddrive storage unit704 is not constantly powered-up. For example, if D1 resolution video data are recorded at 30 fps and compressed using MPEG-4 compression,portion706, with 500 MB of storage capacity, could record approximately one hour of D1 resolution video data. Depending on the data transfer rate ofmemory buffer unit702 and harddrive storage unit704, 500 MB of data may be transferred frommemory buffer unit702 to harddrive storage unit704 in four minutes or less. Thus, out of one hour, harddrive storage unit704 may need to be powered-up for only four minutes. This represents a 1/15 ratio. The powered-up ratio of harddrive storage unit704 is not limited to 1/15, but may be greater or less and may vary over time depending on the content of the video data. For example, a user may choose to store lower than full frame rate video in which harddrive storage unit704 could be powered-up less frequently than once per hour. Preferably, the average powered-up duration will be substantially less than the average powered-down duration. By comparison, a typical hard drive that is powered-up continuously may have a lifespan of approximately five years. By powering-up harddrive storage unit704 for four minutes out of one hour, for example, the lifespan of harddrive storage unit704 may be increased by more than ten times the lifespan of a typical, continuously operated hard drive. Thus, by decreasing the power-up hours of harddrive storage unit704, the lifespan of harddrive storage unit704 is extended.
• Because harddrive storage unit704 is powered-up and down repeatedly, a preferred embodiment of harddrive storage unit704 includes a wear durable hard drive in which the number of on/off cycles does not substantially reduce the lifespan of harddrive storage unit704. A wear durable hard drive includes a read/write head that is physically parked off a storage memory disk (i.e., “unloaded”) during one or more of the powered-down states such that the read/write head does not contact the storage memory disk during a powered-up state or a powered-down state. For example, harddrive storage unit704 may implement Hitachi's ramp load/unload technology described in connection with its Travelstar® hard drive.
• In a conventional contact start-stop (CSS) hard drive, the read/write head rests directly on the storage memory disk during an off state. When the CSS hard drive is powered-up, the read/write head and the storage memory disk remain in contact until air generated by the spinning storage memory disk causes the read/write head to lift off of the storage memory disk. Because the read/write head and the storage memory disk are in contact while the storage memory disk is spinning, the read/write head and the storage memory disk of a CSS hard drive are subject to wear. For example, a CSS hard drive may be able to withstand only 50,000 on/off cycles before failure. If a CSS hard drive was cycled on and off once per hour, the CSS hard drive would last only five or six years.
• In contrast, a wear durable hard drive can withstand, for example, 300,000 on/off cycles or more—preferably 600,000 on/off cycles or more. In other words, if the preferred wear durable hard drive is cycled on and off once per hour, the hard drive would last about 60 years or more. Thus, by implementing the wear durable hard drive of the preferred embodiment, the number of on/off cycles will not substantially reduce the lifespan of harddrive storage unit704. Because the lifespan ofdata storage system206 is relatively long,data storage system206 requires little maintenance or upgrade. Thus, a network edge device, such asnetwork camera102, can practically includedata storage system206. In contrast, without implementing the life extension features described above, a network camera system that provided a large number of mass storage units in network edge devices would require large repair and maintenance costs. This is so because the mass storage units would fail frequently and be distributed in places difficult to serve (such as on tops of tall poles). With the life extension features described above, mass storage units can be provided in network edge devices without the need of frequent replacement.
• It will be obvious to those having skill in the art that many changes may be made to the details of the above-described embodiments without departing from the underlying principles of the invention. The scope of the present invention should, therefore, be determined only by the following claims.

Claims (20)

1. A method of providing redundant storage of video data representing an event of interest while reducing storage and bandwidth requirements of a video camera adapted for connection to a network communication system that includes a remote storage unit, the video camera producing video data representing a field of view of a scene and including a camera housing, comprising:

using a configured event of interest for detection by analysis of the video data produced by the video camera, the video camera provided with video analytics and a mass storage unit contained in or forming part of the camera housing, and the mass storage unit storing the video data representing a field of view of a scene;

causing the video analytics to analyze the video data produced by the video camera and detect whether there is an occurrence of the event of interest; and

sending from the video camera for delivery to the remote storage unit, in response to detection by the video analytics of an occurrence of the event of interest, a clip of the video data representing the field of view of the scene in which the video analytics detected the event of interest.

2. The method ofclaim 1, in which the clip of the video data sent in response to detection of the occurrence of the event of interest includes a video clip file of the event of interest.

3. The method ofclaim 1, in which the clip of the video data sent in response to detection of the occurrence of the event of interest includes a stream of the video data representing the event of interest.

4. The method ofclaim 1, in which the network communication system includes a central monitoring station, and further comprising sending an alert from the video camera for delivery to the central monitoring station in response to detection by the video analytics of an occurrence of the event of interest.

5. The method ofclaim 1, in which the network communication system includes a central monitoring station, and further comprising, in response to detection by the video analytics of an occurrence of the event of interest, supplying from the video camera for delivery to the central monitoring station the clip of video data representing the field of view of the scene in which the video analytics detected the event of interest.

6. The method ofclaim 1, in which the event of interest includes motion detected within the field of view of the video camera.

7. The method ofclaim 1, in which the event of interest includes motion of a blob detected within the field of view of the video camera.

8. The method ofclaim 1, in which the event of interest includes recognizing a predefined object or tracking movement of the predefined object.

9. The method ofclaim 8, in which the video analytics uses geometric calibration to recognize the predefined object as it moves to various portions of the field of view.

10. The method ofclaim 1, in which the storing of video data in the mass storage unit includes storing video data representing a field of view of a scene in which there is no occurrence of the event of interest.

11. The method ofclaim 10, in which the occurrence and nonoccurrence of the event of interest constitute two events of multiple events of interest, and further comprising using multiple configured video data retention policies based on the multiple events of interest such that certain ones or combinations of the multiple events of interest are retained in the mass storage unit for corresponding lengths of time.

12. The method ofclaim 1, in which the video data are stored at a first bit rate when no event of interest is detected and are stored at a second bit rate when the event of interest is being detected, the first and second bit rates being different from each other.

13. The method ofclaim 1, in which the clip of the video data sent for delivery to the remote storage unit includes the video data stored when an event of interest is being detected but not the video data stored when no event of interest is being detected.

14. The method ofclaim 1, in which the clip of the video data sent for delivery to the remote storage unit includes the video data stored when an event of interest is being detected and a portion of the video data representing the field of view of the scene at a time immediately preceding occurrence of the event of interest.

15. The method ofclaim 1, further comprising encrypting at least some of the video data stored in the mass storage unit.

16. The method ofclaim 1, further comprising controlling an amount of time of storage of at least some of the data included in the video clip.

17. The method ofclaim 1, further comprising sending, from the video camera for delivery to the remote storage unit, the video data stored in the mass storage and representing a scene in which no event of interest has been detected.

18. A method of reducing video data storage and network bandwidth requirements of a distributed network video surveillance system that includes network communication paths between network video imaging devices and first video data stores, the network video imaging devices producing video data representing scenes under observation by the network video imaging devices, and the first video data stores storing video information corresponding to the video data produced by the network video imaging devices, comprising:

associating with each of multiple ones of the network video imaging devices second video data stores that are capable of storage of video data produced by an associated network video imaging device, the second video data stores including video analytics that analyzes the content of the video data and local video data stores that store portions of the video data representing an event of interest in response to the analysis by the video analytics; and

delivering from each of the multiple network video imaging devices, through the network communication paths, to the first video data stores, the video data representing scenes under observation to provide redundant storage of the video data representing at least the event of interest.

19. The method ofclaim 18, in which each of the network video imaging devices includes a housing, and in which the second video data stores associated with each network video imaging device are contained within or form part of the housing.

20. The method ofclaim 18, in which the portions of the video data are stored in the local video data stores at a first time, and in which the video data representing scenes under observation include the stored portions of video data and are delivered to the first video data stores at a second time that is later than the first time.

Images