US7106374B1 - Dynamically reconfigurable vision system - Google Patents

Abstract

A closed-loop vision system is disclosed that utilizes a concept known as Dynamically Reconfigurable Vision (DRV), which is adaptive image sensing driven by a computer or human operator's response to changing scenery. The system reduces the amount of irrelevant video information sensed and thus achieves more effective bandwidth and computational resource utilization, as compared to traditional vision systems. One or more reconfigurable photodetector arrays sensitive to either visible, infrared or ultraviolet radiation are present in the DRV system. These photodetector arrays feature on-chip means for spatial and temporal data reduction implemented through multiple independently controllable, time-correlated, frequently overlapping windows on the photodetector array that may be programmed according to their size, location, resolution, integration time, and frame rate. All photodetector array windows are dynamically reconfigurable in real time on a frame-by-frame basis. Furthermore, a DRV system is constructed in a client-server architecture in which a vision processor client passes window request command messages to the reconfigurable photodetector array server, which in turn delivers the requested video back to the client processor. The ability to simultaneously reconfigure, integrate, process, and readout multiple photodetector array video windows is an important characteristic of the DRV system.

Description

This Application claims the benefit of priority to U.S. Provisional Application No. 60/127,698, filed Apr. 5, 1999, which in herein incorporated by reference.

FIELD OF INVENTION

The present invention is related to a vision system (and method), and related particularly to a vision system having reconfigurable CMOS video photodetector arrays providing multiple, time-correlated, overlapping, programmable photodetector array windows. The invention is suitable for single or multiple target viewing, detection, tracking, and identification, especially where such applications are constrained by requirements of limited bandwidth, processing capability, space, weight, or power.

BACKGROUND OF INVENTION

Vision systems for detecting, identifying, and tracking targets must acquire and process large volumes of video data in real time. Most applications requiring these tasks, however, are characterized by imagery where targets and other objects of interest occupy only a small fraction of the total scene. Hence, traditional imaging systems that employ uniform and constant spatial resolution throughout the entire field of view acquire much irrelevant information and thus burden valuable data processing and communication resources in the system. As a result, such systems are slow and do not operate at their full potential due to a flood of unimportant video information. Building vision systems with multiple cameras, several processors, high-bandwidth communication links, and specialized hardware will increase performance, but results in equipment that is complex, expensive, large, heavy, high power consuming, and prone to failures. These constraints make constant resolution vision systems difficult or impossible to use where space, speed of response, and reliability are critical considerations, for instance, in defense applications.

Humans and other vertebrates have foveal vision that allows them to concurrently perform several tasks: survey a wide field of view at a low resolution for situational awareness and detection of features or targets of interest; track moving targets with great accuracy; scan at high resolution these multiple targets of interest; and communicate over channels with limited bandwidth (neurons) the information of interest to the computer (brain). Because high-resolution imaging is limited to the fovea, which is fixed in the center of the retina, the tracking of targets involves movement of the eyes and the head.

The vision system of the present invention operates according to a concept termed Dynamically Reconfigurable Vision, or DRV, which is inspired by the biologically proven concept of foveal vision, but is further extended for practicality in a machine vision implementation. DRV is adaptive image sensing driven by a computer or human operator's response to changing scenery. One goal of this system is to reduce the amount of irrelevant video information sensed and thus achieve more effective bandwidth and computational resource utilization than traditional vision systems. This is achieved by a system controlled by a computer or microprocessor and a photodetector array for imaging a frame of a scene through multiple independently controllable, time-correlated, overlapping photodetector array windows, which are dynamically reconfigurable in real time, and where such windows in a frame are capable of having varying resolution.

Several vision systems or imaging photodetector arrays have been developed which do not enable DRV. U.S. Pat. No. 5,541,654 describes a focal plane array imaging device having sensor windowing, variable integration time, and snapshot mode. U.S. Pat. No. 5,990,469 describes a control system for an image array sensors providing multiple windows. U.S. Pat. No. 5,973,311 provides a pixel array imager with a high and low resolution mode. U.S. Pat. No. 5,949,483 describes a multiresolution image sensor that contains an electronic shutter, can form multiple pixel windows, and possesses a multiresolution imaging circuit for averaging pixels into groups of superpixels. U.S. Pat. No. 5,493,335 provides a color CCD sensor video camera that is adapted for processing images of different resolution to provide a user selectable image size. U.S. Pat. No. 5,926,208 describes a reconfigurable camera with low-bandwidth transmission capability using a conventional image sensor, processor, and memory that can hold several image compression algorithms. These vision systems and sensors do not provide a reduction in the amount of irrelevant information sensed through the use of a computer-controlled system for imaging a scene through multiple independently controllable, time-correlated, overlapping sensor windows.

In a publication by NASA's Jet Propulsion Laboratory, a photodetector array circuitry is described that performs snapshot imaging and pixel averaging, see F B. Pain and X. Zheng, Active Pixel Sensor with Photosites in Substrates, NASA Tech Brief, vol. 23, no. 10, October 1999 (from JPL New Technology Report NPO-20534). This circuitry requires the host computer to send control signals to the sensor for the extraction of the signal from each superpixel, and like the other prior art system, do not provide imaging a scene through a multiple independently controllable, time-correlated, overlapping sensor windows.

In U.S. Pat. No. 5,262,871, a multiresolution image sensor is described that inputs data representing a superpixel to a computer that controls the size of that superpixel. The computer extracts video data from the photodetector array one superpixel at a time. This is time consuming and reduces bandwidth by increasing the amount of interaction between the computer and the camera. Furthermore, the values reporting the level of illumination of the superpixels generated by the system are a function of pixel size in which this value is equal to the sum of the comprising pixels. The superpixel values are normalized so that the image processing algorithms do not erroneously interpret the superpixel to represent a bright scene region. This operation requires extra time and system memory, thus slowing down the image processing. Larger pixel values also require a wider dynamic range in the video communication circuits. In addition, the photodetector array described in U.S. Pat. No. 5,262,871 does not operate in a snapshot mode, so that the exposure time can be different for different pixels after the system has been reconfigured, requiring the computer to carry out additional normalization of pixel values in order to avoid a mistake in interpreting the pixel values in the context of the image. Moreover, because the pixels in this patent are exposed at different times, any motion in the field of view (due to camera and/or target motion) will introduce artifacts, such as target warping, artifacts that reduce the accuracy of the target classification. Overexposed pixels in an image will appear brighter than the properly exposed or underexposed pixels in the same image.

Other developed vision systems that use fixed geometry multiresolution have a two-dimensional photodetector array that has small size pixels in the center and larger size pixels in the periphery, closely simulating the anatomical structure of the retina. Such arrays are not reconfigurable, do not operate in a closed loop fashion with the vision processor, and require a pointing mechanism for gazing. Mechanical pointing suffers from instabilities and is slow. Further, because the topology of the array in a fixed geometry system cannot be changed on demand, some relevant regions of the scene may not be resolved adequately, as a result sacrificing system reliability and usefulness, while the irrelevant regions may be resolved too finely, hence reducing the efficiency of system resource utilization.

In still other systems, referred to as pyramidal machine vision systems, targets are detected using low resolution; these targets are then scanned using localized high-resolution windows. However, the pixels that form the high resolution windows are not combined into superpixels directly in the pixel array, but the information from individual pixels is combined in the computer so as to mimic superpixels. These techniques require the processing of video from a uniform resolution camera to generate the pyramid data structure. As a result, the communications bandwidth and the video processing resources are not used any more efficiently than in a conventional uniform resolution camera.

SUMMARY OF THE INVENTION

It is the principal object of the present invention to provide an improved vision system having a dynamically reconfigurable photodetector array that efficiently uses bandwidth and data processing resources.

A further object of the present invention is to provide an improved vision system that includes a video camera and a vision processor, which is located in a host computer or in the video camera.

Another object of the present invention is to provide an improved vision system having a video camera and vision processor connected in a closed loop system and interacting in a client-server relationship.

Another object of the present invention is to provide an improved vision system having a video camera and vision processor in which the vision processor sends commands specifying the imaging characteristics, such as spatial and temporal parameters, to the reconfigurable photodetector array.

Still further object of the present invention is to provide an improved vision system capable of automatically changing the resolution at which images or parts, such as windows, of images are acquired.

Another object of the present invention is to provide an improved vision system with a reconfigurable photodetector array having multiple windows: a background surveillance window at a low resolution for detecting targets and multiple overlapping higher resolution windows generated any place targets of interest are located in the field of view, such as to resolve additional image details of the targets.

Still another object of the present invention is to provide an improved vision system having a reconfigurable photodetector array for imaging a field of view in windows which can be time-correlated and retrieved by a vision processor simultaneously through multiple output channels of the photodetector array.

A still further object of this invention is to provide a vision system in which the location, size, and shape of the windows is automatically controlled in response to the location and morphology of the target.

Yet another object of this invention is to provide an improved vision system in which the resolution of the windows can progress in multiple steps from the highest (that of a single pixel) to the lowest (that of a single superpixel encompassing the entire field of view).

Briefly described, the present invention embodies a vision system that uses closed-loop photodetector array reconfiguration for efficient utilization of bandwidth and processing resources. The system comprises a video camera and a vision processor arranged in a client-server relationship and is capable of automatically detecting, recognizing, and tracking targets of interest. To accomplish these tasks, the client vision processor determines the spatial and temporal parameters of the incoming video signal, and then passes this information to the video camera server via a window request command. On a reconfigurable photodetector array in the video camera, the system forms a low-resolution background surveillance window capable of detecting targets of interest. Superimposed on this background window in a time-correlated, non-destructive fashion are multiple higher-resolution windows corresponding to the location in the field of view of the targets. The smaller windows supporting target tracking can co-exist in a single video frame with the wide field-of-view window, permitting the detection of new targets. The system can simultaneously retrieve and process the background surveillance window as well as the foreground superimposed target windows.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing objects, features and advantages of the invention will become more apparent from a reading of the following description in connection with the accompanying drawings, in which:

FIG. 1 is a block diagram depicting the major components in accordance with the present invention;

FIG. 2 is a more detailed block diagram of the system ofFIG. 1 showing a host computer, video camera, one photodetector, and an embedded controller external to the photodetector;

FIG. 3 is a block diagram similar toFIG. 2 of another embodiment of the present invention in which the host computer's capability is integrated into a processor within the video camera;

FIG. 4 is a block diagram similar toFIG. 2 of a further embodiment of the present invention in which the controller in the camera and the analog-to-digital converters are located on the same silicon substrate, i.e., chip, as the reconfigurable photodetector array;

FIG. 5 is a block diagram similar toFIG. 2 of still another embodiment of the present invention in which the reconfigurable photodetector of the previous embodiments is replaced with an infrared detector bump-bonded to a read-out integrated circuit;

FIG. 6 illustrates the software architecture of the system in accordance with the present invention; and

FIGS. 7a–cillustrate the progress of target detection and tracking by forming a wide field-of-view, low-resolution background window and higher resolution windows around the instantaneous locations of the target.

DETAILED DESCRIPTION OF THE INVENTION

Referring toFIG. 1, a block diagram of the essential components of theDRV system8 of the present invention is shown having two major modules: a reconfigurable video server11 (typically embodied as a reconfigurable photodetector array within a video camera) and avision processor10. These two components are connected in a closed-loop, client-server relationship. A server represents an object that is waiting to be contacted by a client object so that the server can do something for the client. InDRV system8, theserver11 is thereconfigurable photodetector array23 in thevideo camera29 and the client is thevision processor10. The client vision processor in thecomputer19 is programmed to generate commands, known as window request commands, that specify the spatial and temporal requirements (characteristics or configuration) of the video to be sensed on thereconfigurable photodetector array23 in thevideo server11. These window request commands are transmitted to the reconfigurable photodetector array via a digitalcommand data link12. Once the reconfigurable photodetector array receives a window request command, it reconfigures its resources to satisfy the demands of the window request command. After reconfiguration, the reconfigurable photodetector array acquires the video and transmits it back to thevision processor10 via the digitalvideo data link13.

To begin a client-server cycle, theclient vision processor10 in thecomputer19 identifies a need for one or more video windows, each of a certain field-of-view, spatial resolution, frame rate, and spectral content. It creates a window request command and passes that to thereconfigurable photodetector23 in thevideo server11. The array, which contains on-chip circuitry to enable spatial and temporal data reduction via overlapping windows, then reconfigures itself according to the demands of the window request command. New video frames are sensed according to the current photodetector array configuration and transmitted back to thevision processor client10. Thereconfigurable photodetector array23 may simultaneously transmit multiple windows, such as a background window and overlapping target windows, over multiple on-chip serial ports. The client-server loop can occur on a frame-by-frame basis, if necessary. If one or more video windows do not need to be altered in a given frame, then theDRV system8 andreconfigurable photodetector array23 will remember the window configuration and contents and continue to acquire and transmit those windows.

DRV system8 is useful for applications and platforms requiring single or multiple target detection, tracking, cueing, and identification, especially when such applications are constrained by requirements for limited bandwidth, processing, space, weight, and power. Enabled by a reconfigurable photodetector array,DRV system8 allows simultaneous acquisition and tracking of multiple targets widely separated in the field of view, without the need for smooth-pursuit mechanical tracking. Depending on the number of targets of interest, the system forms one or more high-resolution windows overlapping a low-resolution field-of-view, each high-resolution window containing a target of interest. While targets are being tracked at high resolution, the low-resolution background window remains active and continues to search for new targets over a wide field-of-view. Templates for identifying targets of consequence can be input into the system and stored to be used for automatic selection of such targets for review and tracking at high resolution. In addition, targets can be initially detected based on their movement, contrast, shape, and, with appropriate video photodetector arrays, heat emission or color.DRV system8 is described below in more detail inFIGS. 2–6.

Referring toFIG. 2, one embodiment ofsystem8 is shown having ahost computer19 and adigital video camera29. Thehost computer19 includes thevision processor10 that acts as a client and generates the window request command and processes received digital video.Vision processor10 may be either a single processor or a multi-processor network that is connected to an arbitrary computerperipheral bus18 and the random access memory (RAM)14. Also communicating with theperipheral bus18 is avideo display16 and one or more input/output devices17 that may include, but are not limited to, a mouse, joystick, keyboard, track ball, keypad, and custom user console. A bi-directionaldigital interface15 is responsible for transmitting window request commands from thevision processor10 to thevideo camera29 via the digitalcommand data link12. Thedigital interface15 may be any wirebound or wireless interface capable of transmitting the window request commands and receiving digital video.

Thedigital video camera29 contains a bi-directionaldigital interface20 that is compatible with thehost computer interface15. Thevideo camera interface20 may also have multiple ports for networkingmultiple video cameras29 over additional wirebound or wirelessdigital channels31. Thedigital interface20 connects directly to an embeddedprocessor21. This processor is responsible for handling the digital communication protocol among the host computer, video camera, and any additional networked video cameras. Theprocessor21 shares a bi-directional data link with one or moreprogrammable logic devices22 that act as a controller for the reconfigurableCMOS photodetector array23. Thecontroller22 also interfaces to programmable voltage supplies27, analog-to-digital converters24, as well as FIFOs and registers25 that pipeline multiple video channels into one digitalvideo data bus32. Thevideo bus32 delivers digital video to either theprocessor21 or to a bank ofrandom access memories26.Optics28 are necessary to focus light onto thereconfigurable array23.Additional accessories30 may be embedded in the camera without detracting from the spirit of this invention. These may include, but are not limited to, an embedded web server, microphones, serial ports, or a global positioning system receiver. Theaccessories30 may share a link with the cameradigital interface20 or the embeddedprocessor21.

The closed-loop operation and data flow of the preferred embodiment inFIG. 2 are described in the following. First, thevision processor10 creates an initial window request command to start the video acquisition process. A likely initial command would be for a wide field-of-view, low-resolution window that occupies the entirereconfigurable photodetector array23. The vision processor may use this initial window for detecting target motion or target contrast against a background. Thevision processor10 then dispatches the window request via the host computerperipheral bus18 to the hostdigital interface15. Here the window request is repackaged according to the communication protocol being employed and transmitted via the digitalcommand data link12. Next, the window request command is received by the video cameradigital interface20 and conveyed to the embeddedprocessor21. Here, the window request command is extracted from its digital transmission protocol and transmitted to the embeddedsystem controller22 via a simple serial data link. Within thecontroller22, the window request command is slated into a series of digital vectors for controlling the multiple, programmable windows on thereconfigurable photodetector array23, the photodetector array bias voltages27,multi-channel video digitization24, andvideo multiplexing25. Thephotodetector array23 views a scene throughoptics28. Thecontroller22 sends spatial vectors to the photodetector array directing thephotodetector array23 to output a two-dimensional array of pixels (or pixel signals) imaging the scene in a frame combining the pixels into one or more windows which may be overlapping. Thephotodetector array23 may as described in earlier cited, JPL New Technology Report NPO-20534 and U.S. Pat. No. 5,949,483, and in the NASA Technology Brief from JPL New Technology Report NPO-20555, also JPL New Technology Report NPO 20866 (Real-time Reconfigurable CMOS Imager with Variable High-Fidelity Spatial Sampling with Multiple Windows) A High Speed CMOS Imager with Motion Artifact Suppression and Anti-Blooming, which are herein incorporated by reference.

Thereconfigurable photodetector array23 is capable of changing the resolution of the array by automatically varying the number of individual pixels that constitute a single image resolving element, called a superpixel, having any number of individual pixels from a single pixel to all the pixels in the photodetector array. Said windows are areas of any size, shape, and frame rate within the boundaries of the photodetector array that contain a plurality of said superpixels all having the same resolution. The resolution of the image in each window is controlled by pixel averaging, or other pixel reduction techniques, to achieve the desired low to high resolution. Reconfiguration of imaging characteristics, such as integration time and frame rate, and the size, resolution, location, and overlap of windows in a frame, are controlled in accordance with each window request command. The host computer can direct the resolution of one or more photodetector array windows to be increased or decreased in multiple steps, and each window maintains a consistent resolution regardless of the resolution of overlapping foreground or background windows.

Furthermore, the photodetector array includes electronic circuitry simulating the function of an optical shutter such that when open the electronic shutter allows all the pixels to respond to the light entering the video camera and when closed not to respond to the light from the scene.

In response to signals from thecontroller22, thereconfigurable photodetector array23 outputs one or more channels of analog video to a multi-channel analog-to-digital converter24. Multiple channels of digitized video are then transferred to a FIFO and registers25 where additional control bits are added by thecontroller22 and the digital video is multiplexed onto asingle data bus32. From thedata bus32 the video can go to amemory bank26 where the video pixels will be stored as a contiguous block and can be accessed as appropriate by the embeddedprocessor21. The digital video on thedata bus32 can also be delivered directly to the embeddedprocessor21 where it is packetized into a transmission protocol and delivered directly to thedigital interface20. From theinterface20 the digital video is transmitted outside the camera over a digitalvideo data link13 to thehost computer interface15, where the video is received and delivered to amemory bank14. Frommemory14, thevision processor10 can access the stored digital video and process it. Processing may take the form of image enhancement; compression; target detection, tracking, cueing, and recognition; or other algorithms designed to complete a mission involving surveillance or multi-target detection and tracking. After processing on thevision processor10, the processed video is transferred over theperipheral bus18 to avideo display16 for graphical display and possible user interaction via one or more of various input/output devices17. Finally, thevision processor10 creates a new window request command as a result of video processing results and/or user interaction. Thus begins a new cycle of the closed-loop DRV system.

Referring toFIG. 3, another embodiment of aDRV system8 is shown without a host computer in which a video camera29ais provided having corresponding components tovideo camera29 ofFIG. 2, except forprocessor21 andcontroller22. This embodiment is suitable for embedded applications, such as remote autonomous surveillance and targeting. In such a system, the embeddedprocessor21 doubles as theclient vision processor10. The embeddedcontroller22, in combination with thereconfigurable photodetector array23, acts as the video server. Thus, the basic client-server model ofFIG. 1 is satisfied. Thememory bank26 can be increased in size in this embodiment to hold sequences of video images, intermediate target detection, tracking algorithm data, and executable code. The data links12,13, and31 are used to transmit very low bandwidth control, status, and target information to and from a base station, the reconfigurable video camera29a, and other similar video cameras29a. For example, the base station may be a remote computer system.

Another embodiment ofsystem8 is shown inFIG. 4 in which avideo camera29bis provided having corresponding components tovideo camera29 or29aofFIGS. 2 and 3, respectively, except for the embeddedcontroller22 and the analog-to-digital converter24 which have been integrated onto thereconfigurable photodetector array23. This embodiment forms an application-specific DRV system for platforms that are highly constrained according to size, weight, and/or power.

FIG. 5 shows still another embodiment ofsystem8 having anvideo camera29chaving corresponding components tovideo camera29 or29aofFIGS. 2 and 3, respectively, except forreconfigurable photodetector array23 being replaced by aninfrared detector33 bump-bonded to a CMOS read-outintegrated circuit34 that implements and enables the present invention. The read-out integrated circuit also contains on-chip analog-to-digital converters for digitizing multiple video streams from thedetector33. Although this embodiment depicts an embeddedcontroller22, thiscontroller22 can be integrated onto the read-out integrated circuit (ROIC)34 in the same manner as the embodiment described inFIG. 4.

The software architecture ofsystem8 for providing dynamically reconfigurable vision is shown inFIG. 6. First, a Detection and Tracking Application (DTA)35 generates a request for an initial video surveillance window that covers the entire photodetector field-of-view but possesses a low resolution. TheDTA35 passes this request to the windowrequest command generator36. Here, the request is packaged into awindow request command37 and sent to thevideo server11. First, the window request command must pass through asoftware driver38, which controls digital message transmission via the digitalcommand data link12. At the other end of the data link, anothersoftware driver39 receives the transmitted window request command and passes it onto a windowrequest command interpreter40. Thissoftware module40 takes apart the window request command message and handles communication via aserial command interface41 with reconfigurable photodetector array'scontrol firmware42. Within thefirmware control module42, the window request command is translated into a sequence of digital control vectors and signals43 that are used to manage the circuitry on thereconfigurable photodetector23, the video digitization process, and the movement ofdigital video44 through thevideo camera29. Thedigital video44 is next transferred to avideo packaging node45 where it is bundled into a software data structure and transferred to anoutput driver46 that handles transmission via thevideo data link13. At the opposite end of this data link, anothersoftware driver47 receives the digital video and passes it across abus48 to theDTA35. In the DTA video processing algorithms perform all or some of target detection, tracking, cueing, identification, recognition, and classification. Based upon the digital video content, theDTA35 makes a determination as to what spatial, temporal, and spectral parameters the next video windows should possess. This information is then passed on to the windowrequest command generator36, and the cycle begins again. TheDTA35 may also possess the ability to display the digital video and create a graphical user interface for interaction with a human operator.

FIGS. 7a–7cillustrates schematically the process of target detection, tracking, and identification using first a low resolution window that occupies the entire frame (FIG. 7a), followed by the formation of multiple higher resolution windows around each target (FIG. 7b), and, finally, by multiple small, very high resolution windows each containing a target (FIG. 7c). At this point, the targets can be automatically identified by a human operator or by automatically comparing them to stored templates. This sequence also illustrates the overlapping nature of the multi-resolution windows of the present invention;FIGS. 7band7cdepicts the higher resolution target interrogation windows being formed that non-destructively overlap the lower-resolution background surveillance window. Thus, the resolution in windows are gradually increased as the size of windows is decreased to facilitate the resolution of detail in target or target(s) and the accuracy of tracking of the target(s). All of the windows are time-correlated on thereconfigurable photodetector array23 and can be read from it simultaneously through multiple output ports.

System8 enables Dynamically Reconfigurable Vision (DRV) to provide adaptive image sensing driven by a computer or human operator's response to changing scenery, such that the amount of irrelevant video information sensed is reduced. Thus,system8 has more effective bandwidth and computational resource utilization than traditional vision systems, whereby a scene may be imaged in time-correlated, overlapping windows, with frame-by-frame local (window) resolution selection. In real time, the system adaptively adjusts to changing scene dynamics, thus reducing the amount of irrelevant data read off the reconfigurable photodetector array and making better use of the available communication and data processing resources. Furthermore, the reconfigurable photodetector array can enable multiple windows of varying resolution that can overlap in a non-destructive fashion and can be retrieved from the reconfigurable photodetector array simultaneously. Several technologies make this possible, including the use of one or more reconfigurable photodetector arrays that detect visible, infrared, or ultraviolet radiation. In the case of visible radiation, the reconfigurable photodetector array can use a complementary metal-oxide semiconductor (CMOS) photodetector array and on-chip (monolithic) spatial and temporal data reduction circuitry. For infrared detection, the array may be an infrared photodetector array, where a CMOS readout integrated circuit (ROIC) is bonded to each photodetector array, and there is circuitry on the ROIC that implements spatial and temporal data reduction. Thehost computer19 in the embodiment ofFIG. 2 and theprocessor21 in the embodiments ofFIGS. 3,4, and5,control photodetector array23 orROIC34 viacontroller22 to provide multiple independently controllable, time-correlated, frequently overlapping windows for a frame of an image on thephotodetector array23 orROIC34 that can be reconfigured according to size, location, resolution, integration time, and frame rate. The multiple photodetector array windows can be dynamically reconfigurable in real time on a frame-by-frame basis, on the order of microseconds per reconfiguration. The client-server software architecture ofsystem8 enables the vision processor client to pass window request command messages to the reconfigurable photodetector array server, which in turn delivers the requested video windows back to the client processor. Thesystem8 is capable of simultaneous reconfiguration, integration, processing, and readout of multiple video windows.

One feature ofsystem8 is that the host computer19 (or processor21) issues only one brief high-level configuration command per frame to the photodetector array, as opposed to prior art systems which require a host computer to send multiple control signals to the reconfigurable photodetector array for the extraction of the signal from each superpixel, i.e., averaged pixel set, imaged. Further, by using an electronic shutter on the pixel array, all pixels are exposed simultaneously irrespective of their position, size, and illumination history. The pixel averaging technique in the reconfigurable photodetector array does not require pixel value normalization by the computer, reducing processing and communication requirements to the host computer.

Another feature is that thevideo server11 allows frame-by-frame software control of on-chip photodetector array windowing, pixel resolution, and pixel averaging functions. Low-bandwidth video transmission is provided to the host computer by the reconfigurable photodetector array in the camera having spatial and temporal data reduction means, under control by the host, to reduce the amount of image data communicated to the host in comparison with prior art vision system. Thus, based upon the scene content, a processor or human operator can issue window request commands on a per-frame basis to control the spatial, temporal, and spectral characteristics of windows in a frame. The topology of the windows of the reconfigurable photodetector array can be modified frame-by-frame in an appropriate response to the instantaneous conditions in the scene being imaged, thereby enabling multiple targets in a scene to be extracted such as for tracking, in which the host computer can determine signatures of target objects. This is facilitated by the ability of the system to control the resolution of the photodetector array windows automatically in selected windowed locations of the frame corresponding to the instantaneous location of the image of a target or other targets without changing the resolution of the remainder parts, i.e., windows, of the frame.

It is to be understood that the preceding descriptions are illustrative only and that changes may be made in the dynamically reconfigurable vision system, subject of this invention, in its components, materials and elements, as well as in all other aspects of this invention discussed herein without departing from the scope of the invention as defined in the claims.

Claims (33)

1. A vision system for imaging a scene using a dynamically reconfigurable photodetector array comprising:

a video camera having at least one reconfigurable photodetector array capable of imaging a scene;

means for controlling characteristics of imaging by said photodetector array; and

a computer system for receiving signals from said photodetector array representing the scene imaged by said photodetector array and sending signals to said controlling means to configure the characteristics of imaging by said photodetector, wherein said photodetector array simultaneously provides signals representing a frame with one or more windows having pixels in which said signals are simultaneously readout from the photodetector array, and two or more of said windows are capable of overlapping in said frame.

2. The system according toclaim 1 wherein said photodetector array further comprises means for controlling the resolution of the pixels in each of said windows.

3. The system according toclaim 1 further comprising a client-server interface between said computer system and said controlling means.

4. The system according toclaim 1 wherein said controlling means represents a controller located on said photodetector array.

5. The system according toclaim 1 in which said controlling means represents a logic device separate from said photodetector array.

6. The system according toclaim 1 wherein said computer system is integrated into said video camera.

7. The system according toclaim 1 wherein said computer system is separate from said video camera.

8. The system according toclaim 1 wherein said computer system sends signals to said controlling means representing window request commands that contain information used to dynamically reconfigure one or more of said photodetector array's imaging characteristics.

9. The system according toclaim 1 wherein said characteristics represent spatial and temporal parameters.

10. The system according toclaim 1 wherein said photodetector array contains a plurality of individual pixels that have fixed dimensions.

11. The system according toclaim 1 wherein said photodetector array represents a complementary metal oxide semiconductor photodetector array responsive to radiation in the visible spectral range.

12. The system according toclaim 1 wherein said photodetector array represents a device responsive to infrared wavelengths.

13. The system according toclaim 1 wherein said photodetector array is bonded to a complementary metal oxide semiconductor read-out integrated circuit.

14. The system according toclaim 1 wherein said computer system and said video camera are part of a closed-loop interactive system.

15. The system according toclaim 1 wherein said computer system is capable of analyzing video imagery for target objects and determining the necessary position, size, shape, resolution, and frame rate of multiple, overlapping windows within said vision system's field of view.

16. The system according toclaim 1 wherein said computer system is capable of determining the spectral signatures of said target objects.

17. The system according toclaim 1 wherein said computer system further comprises a user interface enabling a human operator to analyze video imagery for target objects and determine the necessary position, size, shape, resolution, frame rate, and spectral content of multiple windows within said vision system's field of view.

18. The system according toclaim 1 wherein said computer system further comprises means for generating a window request command to said video camera identifying said characteristics.

19. The system according toclaim 1 wherein said computer system further comprises means for generating a window request command to said video camera identifying said characteristics, in which said characteristics represents any of position, size, shape, resolution, frame rate, and spectral content of multiple windows.

20. The system according toclaim 1 wherein when said windows overlap said windows overlap non-destructively in said frame.

21. The system according toclaim 1 wherein said windows which overlap in said frame are of same or different resolutions.

22. The system according toclaim 1 wherein said controlling means controls resolution of each of said windows by averaging pixels.

23. A method of tracking objects in a scene comprising a frame of windows comprising the steps of:

locating at least one object to be tracked in one of said windows in a low resolution;

forming one or more windows having said object in a high resolution which are capable of overlapping each other and overlapping said one or more of said windows at said low resolution in the frame in which said overlapping windows are simultaneously provided in the frame from signals simultaneously outputted from a photodetector array; and

identifying said object in said windows of said high resolution.

24. The method according toclaim 23 wherein a plurality of said windows of high resolution are formed corresponding to the number of targets in said scene.

25. The method according toclaim 23 further comprises the step of:

changing at least one window between different resolutions by averaging pixels of the window.

26. A system for imaging comprising:

at least one photodetector array capable of imaging successive frames in which each frame has windows of pixels at a low resolution; and

means for controlling the resolution of one or more windows of the frames to enable tracking of one or more objects at higher resolution windows in successive ones of said frames in which said windows are dynamically reconfigurable in real time between successive frames to facilitate said tracking, and said windows at said higher resolution non-destructively overlap one or more of said lower-resolution windows in the frame which are in the local of said higher resolution windows and at least one of said higher resolution windows are imaged simultaneously with at least one of said lower resolution windows being overlapped, and the pixels of the overlapping windows that are simultaneously imaged in the frame are simultaneously readout from the photodetector array.

27. The system according toclaim 26 wherein said means is capable of controlling the resolution of windows by averaging pixels of said windows.

28. The system according toclaim 26 wherein said overlapping windows are time-correlated frame-by-frame with location of said one or more objects in each successive frame.

29. The system according toclaim 26 wherein said higher resolution windows are capable of overlapping each other.

30. A video imaging apparatus comprising:

at least one detector array for imaging a scene capable of simultaneous readout of multiple windows; and

a controller for controlling characteristics of imaging by said detector array to simultaneously output one or more windows in which two or more of the windows when simultaneously readout by said detector array are capable of overlapping.

31. The video imaging apparatus according toclaim 30 wherein said controller is operative in response to a host computer system.

32. The video imaging apparatus according toclaim 30 wherein said windows simultaneously outputted are of the same or different resolution.

33. The video imaging apparatus according toclaim 30 wherein said overlapping windows overlap non-destructively.

Images