US20190364263A1 - Computational imaging device - Google Patents

Abstract

Dense field imagers are disclosed which are configured to provide combined, aggregated, fused, and/or stitched light field data for a scene. A dense field imager can include a plurality of imaging elements configured to be joined into image blocks or facets that each provides light field data about a scene. The dense field imager can include a plurality of facets in a fixed or modular fashion such that the dense field imager is configured to combine, aggregate, fuse and/or stitch light field data from the plurality of facets. The facets can be mounted such that one or more facets are non-coplanar with other facets. The facets can be configured to provide a representation of the light field with overlapping fields of view. Accordingly, the dense field imager can provide dense field data over a field of view covered by the plurality of facets.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• This application is a continuation of U.S. patent application Ser. No. 15/283,731, filed Oct. 3, 2016, entitled “DENSE FIELD IMAGING” which is a continuation of U.S. patent application Ser. No. 14/181,547, filed Feb. 14, 2014, entitled “DENSE FIELD IMAGING” which claims the benefit of priority to U.S. Provisional Patent Application No. 61/765,661, entitled “DENSE FIELD IMAGING,” filed Feb. 15, 2013, and to U.S. Provisional Patent Application No. 61/785,494, entitled “IMAGING SYSTEM WITH MULTIPLE IMAGING DEVICES HAVING OVERLAPPING FIELDS OF VIEW,” filed Mar. 14, 2013. Each application referenced in this paragraph is incorporated by reference herein in its entirety so as to form part of this specification.
BACKGROUND
• The present disclosure generally relates to acquiring and processing image data from a plurality of imaging devices.
SUMMARY
• The systems, methods and devices of the disclosure each have innovative aspects, no single one of which is indispensable or solely responsible for the desirable attributes disclosed herein. Without limiting the scope of the claims, some of the advantageous features will now be summarized.
• Systems and methods are disclosed herein for acquiring and processing image data from a plurality of imaging devices, and particularly in some embodiments for acquiring multiple light field representations and generating images from the multiple light field representations. For example, dense field imagers are disclosed which are configured to provide a plurality of light field representations of a scene. Dense field imagers can be configured to combine the plurality of light field representations to generate a dense field data set, which can include the plurality of light field representations of the scene. A dense field imager can include a plurality of imaging elements configured to be joined into image blocks or facets, the imaging blocks providing data that can be used to generate a plurality of light field representations for a scene. In some embodiments, the data from each imaging block is used to generate a light field representation. The dense field imager can include a plurality of image blocks or facets in a fixed or modular fashion such that the dense field imager is configured to aggregate and/or combine light field representations from the plurality of facets. The facets can be mounted such that one or more facets are non-coplanar with other facets. The facets can include processing modules that are configured to generate light field representations from acquired pixel data. The generated light field representations can have overlapping fields of view. Accordingly, the dense field imager can provide dense field data over a field of view covered by the plurality of facets. The dense field data can include the plurality of light field representations that can be used to generate one or more images for the field of view.
• Image processing systems are also disclosed which are configured to receive light field representations from a plurality of sources and to aggregate or combine the light field representations to generate a unitary, dense field data set. The generated dense field data set can be used to dynamically generate viewpoints or images having a variety of desired properties. For example, a depth of focus can be changed dynamically, stereoscopic images can be provided where an inter-ocular distance can be changed dynamically, images from a variety of virtual viewpoints can be generated, and the like. These image processing systems can allow dynamic editing and cinematography such that the importance of the configuration of the cameras acquiring the data is reduced as many of the creative decisions can be made during post-processing. For example, using the image processing systems described herein a director or video editor could re-light a scene virtually, perform depth-based editing or coloring, shift a focus of the image, zoom in or out of a scene, change a viewpoint of the scene, and the like.
• In some embodiments, the image processing system can allow for users to personalize videos or movies. For example, a movie can be made using data acquired with the dense field imager described herein. The dense field image data can then be provided to allow a person not associated with the original creation of the movie to change characteristics of the movie by changing focus, viewpoint, lighting, zoom, etc.
• In some embodiments, the dense field imager and/or image processing systems described herein can allow for a reduction or elimination of the involvement of camera operators in a venue. For example, one or more dense field imagers can be provided in a stadium at an athletic event and a director can dynamically change a focus of output images to focus on an object or region of interest. In some embodiments, a dense field imager can be configured to dynamically track or focus on one or more objects or persons of interest. It can be possible, for example, then to refocus on different objects according to focus criteria.
• Some embodiments described herein provide for a dense field imaging block and array. The imaging block can include a support and at least a first imaging element and a second imaging element carried by the support, each imaging element comprising a sensor and a lens. The imaging block can include a mechanical connector for mechanically connecting the imaging block into an array of imaging blocks. The imaging block can include an electrical connector for electrically connecting the imaging block into an array of imaging blocks.
• In some implementations, the first imaging element of the imaging block comprises a monochromatic filter, wherein the sensor of the first imaging element detects substantially monochromatic light passing through the monochromatic filter. In some implementations, the support comprises a wafer substrate and the wafer substrate can sometimes be made of a semiconductor material. In some implementations, the sensors are formed on the substrate. In some implementations, the lenses are wafer-level lenses.
• Some embodiments provide for an imaging array that includes an array support and at least two imaging blocks carried by the array support. The at least two imaging blocks can include a first imaging block and a second imaging block. At least one of the sensors in the first imaging block can be non-coplanar with at least one of the sensors in the second imaging block. In some implementations, each of the sensors in the first imaging block is coplanar, each of the sensors in the second imaging block is coplanar, and each of the sensors in the first imaging block is non-coplanar with each of the sensors in the second imaging block. In some implementations, each imaging element has a primary optical axis, and the primary optical axis of at least one of the imaging elements in the first imaging block is substantially non-parallel with the primary optical axis of at least one of the imaging elements in the second imaging block. In some implementations, each imaging element has a primary optical axis, the primary optical axes of the imaging elements in the first imaging block are substantially parallel, the primary optical axes of the imaging elements in the second imaging block are substantially parallel, and the primary optical axes of the imaging elements in the first imaging block are substantially non-parallel with the primary optical axes of the imaging elements in the second imaging block. In some implementations, a primary optical axis of at least one imaging element of the first imaging block is angularly adjustable with respect to a primary optical axis of at least one imaging element of the second imaging block. In some implementations, the imaging array comprises a user-actuatable control for achieving the angular adjustment.
• The imaging array can also include, in some implementations, an image processing system configured to use image data captured by the first imaging block to generate a first image data set representative of a first portion of a light field. It can also be configured to use image data captured by the second imaging block to generate a second image data set representative of a second portion of the light field. It can also be configured to derive a third image data set from the first and second image data sets. In some implementations, the image processing system is carried by the array support. In some implementations, the image processing system is physically separate from the array support and receives the first and second image data sets wirelessly. In some implementations, the image processing system derives the third image data set at least partly by creating a spatial relationship tensor that includes spatial relationship information between elements of the first and second image data sets and using the spatial relationship tensor to derive the third image data set. In some implementations, the image processing system derives the third image data set at least partly by using the spatial relationship tensor to combine together the first and second portions of the light field. In some implementations, the image processing system derives the third image data set at least partly by aggregating the first image data set and the second image data set. In some implementations, the third image data set can include information about a spatial relationship between the first image data set and the second image data set, wherein the third image data set includes, for example, relative orientations and positions of the first and second imaging blocks which can be used to associate the first and second image data sets representative of first and second portions of the light field.
• In some implementations, the imaging block can include one or more processors carried by the support and configured to generate a light field representation based on pixel data acquired from the sensors. In some implementations, one or more of the lenses are removably replaceable with lenses having different optical characteristics. In some implementations, each imaging element has a primary optical axis, and most of the primary optical axes are substantially parallel. In some implementations, each imaging element has a primary optical axis, and at least two of the primary optical axes diverge in a direction leading away from the sensor. In some implementations, each imaging element has a primary optical axis, and at least two of the primary optical axes converge in a direction leading away from the sensor. In some implementations, each imaging element has a primary optical axis, and at least a first primary optical axis is angularly adjustable with respect to at least a second primary optical axis. In some implementations, the imaging block includes at least 8 imaging elements. In some implementations, the imaging elements are arranged in two rows of 4. In some implementations, the imaging block includes at least 16 imaging elements. In some implementations, the imaging elements are arranged in a 4×4 grid. In some implementations, at least one sensor is no larger than about 5 mm×5 mm. In some implementations, the imaging block also includes an FPGA chip carried by the support. In some implementations, at least some of the sensors have one or more of different sizes, different resolutions, or different sensitivities.
• Some embodiments provide for an imaging array comprising an array support, and at least two imaging blocks according to any of the above descriptions, the imaging blocks being carried by the array support.
• Some embodiments provide for a method of compiling an image data set. The method can include obtaining a first image data set representative of a first portion of a light field. The method can include obtaining a second image data set representative of a second portion of the light field. The method can include deriving, with one or more processors, a third image data set that is based on at least the first image data set, the second image data set, and information relating to a spatial relationship between the first and second portions of the light field. The method can include storing the third image data set in one or more memory devices.
• In some implementations, the first image data set is derived from pixel data acquired by a first group of at least two imaging elements and the second image data set is derived from pixel data acquired by a second group of at least two imaging elements, each imaging element comprising a sensor and a lens. In some implementations, the method also includes processing the first image data set and the second image data set to determine the spatial relationship between the first portion of the light field and the second portion of the light field. In some implementations, deriving includes accessing a spatial relationship tensor representing the spatial relationship information to perform a geometric transform on the first image data set and second image data set. In some implementations, deriving includes determining the spatial relationship between the first image data set and the second image data set using calibration information. In some implementations, the at least two imaging elements in the first group are coplanar, the at least two imaging elements in the second group are coplanar, and the at least two imaging elements in the first group are non-coplanar with respect to the at least two imaging elements in the second group. In some implementations, the first and second image data sets additionally represent the first portion and the second portion of the light field as a function of time. In some implementations, the third image data set comprises light field information represented as a function of time. In some implementations, the first portion of the light field and the second portion of the light field comprise regions of the light field which at least partially overlap, and wherein the third data set comprises light field information derived from data in both of the first and second data sets that corresponds to the region of the light field lying within the overlap. In some implementations, the first portion of the light field and the second portion of the light field comprise regions of the light field which only partially overlap. The third data set can include light field information derived from data in the first image data set that corresponds to a first portion of a scene, and data in the second image data set that corresponds to a second portion of the scene that does not overlap with the first portion.
• In some implementations, the third image data set comprises at least 4D light field information. In some implementations, deriving the third image data set comprises deriving the third image data set while maintaining the dimensionality of functions that represent the first and second portions of the light field. In some implementations, the first and second image data sets respectively represent the first and second portions of the light field as functions having at least four input parameters. In some implementations, the first and second image data sets respectively represent the first and second portions of the light field as functions having at least five input parameters. In some implementations, the first and second image data sets respectively represent the first and second portions of the light field as functions that represent luminance as a function of a position in space and a pointing direction.
• In some implementations, viewable images are extractable from the third image data set. In some implementations, the viewable images comprise 2D images and/or 3D images. In some implementations, viewable motion video is extractable from the third image data set.
• In some implementations, the method can also include accessing a fourth image data set representative of a third portion of the light field, wherein said deriving comprises deriving the third image data set based on at least the first image data set, the second image data set, the fourth image data set, and spatial relationships between the first, second, and third portions of the light field.
• Some embodiments provide for a stored image data set on a memory device wherein the memory device includes a dense field image data set. The memory device can include a storage medium, a dense field image data set stored in the storage medium and derived by relating a first image data set representative of a first portion of a light field to a second image data set representative of a second portion of the light field, using information relating to a spatial relationship between the first and second light field representations.
• Some embodiments provide for a computer-readable memory device, wherein the image data set comprises light field information represented as a function of time. In some implementations, the computer-readable memory device includes the image data set and the image data set includes at least 4D light field information. In some implementations, the first image data set is derived from pixel data acquired by a first imaging block comprising a support and at least two imaging elements carried by the support, and the second image data set is derived from pixel data acquired by a second imaging block comprising a support and at least two imaging elements carried by the support, each imaging element comprising a sensor and a lens.
• Some embodiments provide for a machine comprising the computer-readable memory device above and one or more processors configured to derive the dense field image data set. In some implementations, the one or more processors are configured to extract viewable images from the dense field image data set.
• Some embodiments provide for a computer-readable memory device having viewable images that are 2D images, 3D images, and/or motion video.
• Some embodiments provide for a method of creating a dense field image set. The method can include acquiring pixel data from a plurality of imaging elements, each imaging element comprising a sensor and a lens. The method can include generating a plurality of light field representations, each of the light field representations generated using pixel data from at least two of the imaging elements. The method can include creating a spatial relationship tensor or matrix representative of spatial relationships among the light field representations. The method can include utilizing the spatial relationship tensor or matrix, combining or aggregating the light field representations and the spatial relationship tensor or matrix to create a dense field image set.
• In some implementations, combining the light field representations comprises stitching or combining light field representations having at least partially non-overlapping fields of view to create a dense field image set having a significantly wider field of view than the individual light field representations. In some implementations, the lateral field of view of the dense field image set is greater than or equal to about 145 degrees. In some implementations, the imaging elements are arranged on a common support. In some implementations, at least some of the imaging elements are not coplanar. In some implementations, each light field representation is generated using pixel data from an imaging block comprising at least two imaging elements. In some implementations, at least some of the imaging blocks are not coplanar. In some implementations, at least one of the imaging elements comprises a monochromatic filter, and wherein light passing through the monochromatic filter is detected by the sensor of the at least one imaging element. In some implementations, at least half of the imaging elements comprise monochromatic filters. In some implementations, substantially all of the imaging elements comprise monochromatic filters.
• Some embodiments provide for an imaging system configured to generate a light field representation for each of a plurality of imaging blocks. The plurality of imaging blocks each can include at least two imaging elements, and each of the imaging elements can include an image sensor and a lens. The imaging system can include a dense field image processor module configured to, for each imaging block of at least two of the plurality of imaging blocks, generate pixel correspondence information for the imaging block, the pixel correspondence information representative of spatial relationships between pixels in each imaging element of the imaging block and corresponding pixels in other imaging elements of the imaging block. The module can be configured to utilize the correspondence information to generate a light field representation using pixel data acquired by the respective imaging block.
• In some implementations, the dense field image processor module is further configured to create a spatial relationship tensor or matrix representative of spatial relationships among the light field representations, and, utilizing the spatial relationship tensor or matrix, combine or aggregate the light field representations to create a dense field image set.
• In some implementations, the imaging blocks are arranged on a common support. In some implementations, the imaging blocks are formed in a wafer. In some implementations, the imaging elements of each imaging block are coplanar with respect to one another. In some implementations, at least some of the imaging blocks are not coplanar with respect to other ones of the imaging blocks. In some implementations, at least some of the imaging blocks are arranged on physically separate supports.
• Some embodiments provide for a dense field imaging system that includes a plurality of imaging blocks, each of the imaging blocks comprising at least two sensor/lens pairs, wherein at least some of the imaging blocks are substantially non-coplanar with each other and have at least partially non-overlapping fields of view. The dense field imaging system can include a dense field image processor module configured to, for each imaging block, generate a light field representation using pixel data acquired by the sensor/lens pairs. The module can be configured to generate a spatial relationship tensor or matrix representative of spatial relationships between the light field representations generated by each imaging block. The module can be configured to use the spatial relationship tensor or matrix to combine or aggregate the light field representations to create a dense field imaging set having a substantially wider field of view than the individual light field representations. In some implementations, each of the imaging blocks comprises a wafer substrate on which the sensors are formed. In some implementations, the dense field imager also includes a control configured to adjust the angular relationship between the primary optical axes of sensor/lens pairs.
• Some embodiments provide for a dense field imaging system that includes a plurality of imaging blocks arranged on a support, each of the imaging blocks comprising at least two imaging elements, each of the imaging elements comprising a sensor and lens, wherein at least some of the imaging elements comprise monochromatic filters. The dense field imaging system can include a dense field image processor module configured to, for each imaging block, generate a light field representation using pixel data acquired by the imaging elements of the imaging block. The module can also be configured to combine or aggregate the light field representations to create a dense field image set.
• In some implementations, most of the sensors are monochromatic. In some implementations, each of the sensors is monochromatic. In some implementations, each of the imaging blocks includes a support comprising a wafer substrate, and wherein the sensors are formed on the wafer substrate. In some implementations, the lenses comprise wafer level lenses.
• Some embodiments provide for any of the dense field imaging systems described herein to be mounted on an aerial vehicle to perform wide-angle persistent surveillance. The dense field imager can include a plurality of imaging blocks, each having a plurality of imaging elements with image sensors, each image sensor having a resolution measured in pixels. The dense field imaging system can be configured to generate a plurality of light field representations covering a field of view. The plurality of light field representations can be combined or aggregated to generate an output image of at least a portion of the field of view wherein the output resolution of the output image is greater than any resolution of the image sensors of the plurality of imaging blocks.
• In some implementations, the resolution of the output image is at least 3× greater than the resolution of any individual image sensor, at least 5× greater than the resolution of any individual image sensor, at least 10× greater than the resolution of any individual image sensor, at least 20× greater than the resolution of any individual image sensor, at least 50× greater than the resolution of any individual image sensor, or some value between any of these values greater than the resolution of any individual image sensor (e.g., between 3× and 50×, between 5× and 20×, etc.). In some implementations, the resolution of the output image is greater than the combined resolution of the imaging sensors. For example, in some implementations, the resolution of the output image is at least 3× greater than the combined resolution the image sensors, at least 5× greater than the combined resolution the image sensors, at least 10× greater than the combined resolution the image sensors, at least 20× greater than the combined resolution the image sensors, at least 50× greater than the combined resolution the image sensors, or some value between any of these values greater than the combined resolution of the image sensors (e.g., between 3× and 50×, between 5× and 20×, etc.). In some implementations, the dense field imaging system can be mounted on a gimbal on the aerial vehicle. In some implementations, the aerial vehicle can be an unmanned aerial vehicle, an airplane, a helicopter, a satellite, or the like. In some implementations, a ground sample distance of each pixel in the image sensors is at least about 0.5 m and/or less than about 4 m, at least about 0.75 m and/or less than about 2 m, at least about 0.9 m and/or less than about 1.25 m wherein the wide-angle persistent surveillance images the ground from the aerial vehicle.
• Some embodiments provide for a dense field imaging system incorporated into a mobile device such as a smartphone, cellular phone, PDA, tablet, laptop, or the like. The dense field imaging system can include a plurality of imaging blocks each configured to acquire image data and generate one or more light field representations of a field of view. The imaging blocks can each include a plurality of imaging elements each comprising a lens and an image sensor. The imaging elements can be implemented using wafer-level optics and image sensors on a wafer-substrate.
• Some embodiments provide for a dense field camera comprising a housing having a first side with a plurality of imaging blocks attached thereto, and a second side, opposite the first side, having a display. The dense field camera includes an image processing system configured to generate a light field representation from the image data acquired by each of the plurality of imaging blocks. The housing can have a length that is at least about 6 in. and/or less than or equal to about 36 in., at least about 12 in. and/or less than or equal to about 24 in., or at least about 15 in. and/or less than or equal to about 20 in. The housing can have a width that is at least about 4 in. and/or less than or equal to about 36 in., at least about 8 in. and/or less than or equal to about 24 in., or at least about 12 in. and/or less than or equal to about 20 in. The housing can have a depth that is at least about 0.5 in. and/or less than or equal to about 8 in., at least about 1 in. and/or less than or equal to about 5 in., or at least about 2 in. and/or less than or equal to about 4 in.
• In some implementations, the dense field camera can include a wireless data transmission system configured to transmit image data to an off-camera processing and/or storage system. In some implementations, the dense field camera can be configured to have a greater effective output image resolution with a smaller depth of the camera housing relative to a standard camera employing a single imaging element.
• Some embodiments provide for a portable dense field camera comprising a housing having a surface with a plurality of imaging blocks attached thereto, the surface having a modifiable geometrical configuration wherein a modification to the surface geometry modifies a field of view of the dense field camera. The portable dense field camera includes an image processing system configured to generate a light field representation from the image data acquired by each of the plurality of imaging blocks.
• In some implementations, the surface geometry can be modified by a user of the dense field camera. In a further implementation, the user can adjust the surface geometry through a single mechanical element, such as a knob, slider, rod, or the like. In some implementations, the surface geometry can be made to be planar, convex, or concave.
• Some embodiments provide for a dense field camera comprising a plurality of imaging blocks, each imaging block having a field of view. The dense field camera is configured such that the field of view of an imaging block substantially overlaps with a field of view of a neighboring imaging block. The dense field camera includes an image processing system configured to enhance a parameter of an output image generated by the image processing system, enhancement of the parameter based at least in part on information provided by pixels within the substantially overlapping fields of view of neighboring image blocks. In some implementations, the image parameter can be a brightness of at least a portion of the image, a dynamic range of at least a portion of the image, or a color of at least a portion of the image.
BRIEF DESCRIPTION OF THE DRAWINGS
• The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the disclosure. Throughout the drawings, reference numbers may be re-used to indicate general correspondence between referenced elements.
• FIG. 1A illustrates a structural block diagram of an embodiment of an imaging system having a plurality of imaging blocks or facets, each facet having a plurality of imaging elements.
• FIG. 1B illustrates a functional block diagram of an embodiment of an imaging system, such as a dense field imager, coupled to a dense field processing system.
• FIG. 2 illustrates an example imaging element that includes a lens, an image sensor, and image acquisition electronics.
• FIG. 3 illustrates an example of a single facet of a dense field imager, the single facet comprising a plurality of imaging elements with overlapping fields of view.
• FIG. 4 illustrates an example of a facet having a plurality of imaging elements coupled together on a common support, the common support including facet processing electronics.
• FIGS. 5A and 5B illustrate schematic drawings showing a dense field imager with an array of imaging elements coupled to a common support and acquisition electronics.
• FIG. 6A illustrates a field of view for an individual imaging element, such as an imaging element in a dense field imager.
• FIGS. 6B and 6C illustrate overlapping fields of view for example arrays of imaging elements that are arranged in rectangular and hexagonal grids, respectively.
• FIG. 7A illustrates imaging elements with partially overlapping fields of view.
• FIG. 7B illustrates imaging elements with substantially overlapping fields of view.
• FIG. 7C illustrates imaging elements with substantially adjacent fields of view.
• FIG. 7D illustrates imaging elements with partially overlapping fields of view, where the angle of view is decreased.
• FIG. 7E illustrates imaging elements with substantially overlapping fields of view, where the angle of view is decreased.
• FIG. 7F illustrates imaging elements with substantially adjacent fields of view, where the angle of view is decreased.
• FIG. 8 illustrates an example dense field imager having an array of imaging elements wherein each imaging element includes a monochromatic sensor.
• FIGS. 9A-C illustrate an example embodiment of a planar dense field imager having integrated cooling elements and integrated acquisition and processing electronics.
• FIGS. 10A-C illustrate an example embodiment of a dense field imager having facets that are mounted in a concave fashion.
• FIGS. 10D-F illustrate fields of view of some of the facets on the dense field imager illustrated inFIGS. 10A-C.
• FIGS. 11A-C illustrate an example embodiment of a dense field imager having facets that are mounted in a convex fashion.
• FIGS. 11D-F illustrate fields of view of some facets on the dense field imager illustrated inFIGS. 11A-C.
• FIGS. 12A and 12B illustrate perspective views of an example dense field imager including a plurality of imaging elements arranged in an array.
• FIG. 12C shows a side perspective view of an embodiment of a dense field imager having a plurality of imaging elements with large, removable lenses.
• FIG. 12D shows a side perspective view of an embodiment of a dense field imager having a plurality of imaging elements with small, removable lenses.
• FIG. 12E shows a side perspective view of an embodiment of a dense field imager having a plurality of imaging elements with non-removable, low-profile lenses.
• FIGS. 12F-H show top perspective views of a dense field imager with a grip attachment, the dense field imager having a plurality of imaging elements with, respectively, large, removable lenses; small, removable lenses; and non-removable, low-profile lenses.
• FIGS. 12I and 12J show top perspective views of a dense field imager, respectively with and without a grip attachment, showing the viewfinder screen.
• FIG. 13 illustrates an embodiment of a dense field imager that is configured to wirelessly transmit acquired data to an image hub.
• FIGS. 14A and 14B respectively illustrate a top plan view and a side elevational view of a multi-imaging device system.
• FIG. 14C illustrates a top plan view of an array of cameras recording an object or other portion of interest in a viewing theater.
• FIG. 15 illustrates a block diagram of an example image processing system for receiving data from a plurality of sources of different types and generating light field data, dense field data, and/or viewpoints from received data.
• FIG. 16 illustrates a flow chart of an example method for acquiring image data with a dense field imager or an array of imaging devices having overlapping fields of view wherein a plurality of light field representations are generated using two or more imaging elements or imaging devices.
• FIG. 17 illustrates a flow chart of an example calibration method for determining corrective factors for imaging elements on a dense field imager or an array of imaging devices.
• FIG. 18A illustrates an example object that can be used to in the calibration method illustrated inFIG. 17.
• FIG. 18B illustrates an example plot showing results of corrected and non-corrected pixel data.
• FIG. 18C illustrates an example plot of relative imaging element orientation.
• FIG. 18D illustrates a plot of a result of calibrating imaging elements on a dense field imager.
• FIG. 19 illustrates a flow chart of an example method for combining light field representations.
• FIGS. 20A and 20B represent a scene to be captured by a plurality of devices configured to generate light field representations and a field of view of one of the plurality of devices.
• FIG. 21 illustrates a timing diagram for capturing images using multiple imaging devices.
• FIG. 22 illustrates a flow chart of an example method for generating a viewpoint using dense field data.
DETAILED DESCRIPTION
• In the following description, reference is made to the accompanying drawings. It is to be understood that other structures and/or embodiments may be utilized. Various aspects of the disclosure will be described with regard to certain examples and embodiments, which are intended to illustrate but not to limit the disclosure. Nothing in this disclosure is intended to imply that any particular feature or characteristic of the disclosed embodiments is essential.
• Generally, image sensors that capture digital images are made up of multiple picture elements, or pixels. Each pixel receives light that is directed to it by an optical system. The light received by the pixel creates an electrical charge, and the cumulative charge caused by light received by the pixel is converted into a digital value. By combining information from multiple image sensors, multiple capabilities can be provided that, for example, improve image quality and/or provide information that generally is not obtained using a single image sensor. For example, dynamic range can be increased, noise can be reduced, directional and/or depth information can be extracted, focus can be adjusted, resolution can be enhanced, and the like.
• A plurality of imaging elements having their own optics and image sensors can be combined to provide advantages outlined above and described in greater detail herein. A plurality of imaging elements can be combined into an image block where each imaging element provides image data from their sensors. The image data from the plurality of imaging elements in the image block can be combined to produce a light field representation over a field of view. Multiple image blocks can be combined or grouped to form an imaging system, such as a dense field imager or dense field camera, which is configured to combine or aggregate light field representations from the multiple blocks.
• Image processing systems can also be configured to receive image data from a plurality of imaging elements to produce light field representations. These or other image processing systems can also be configured to receive light field representations covering multiple fields of view and combine or aggregate the light field representations to enhance or increase information contained in the combined or aggregated light field representation and/or increase a field of view of the combined or aggregated light field representation.
• As will be described in further detail, imaging systems described herein can capture pixel data that can be used to generate one or more light field representations, and in some embodiments provide similar advantages to light-field and plenoptic cameras, in addition to providing other advantages. Such advantages can include the capability to adjust focus and/or depth of field in post-processing. For instance, some of the multi-imaging device cameras described herein can capture light field information about the image scene, which can be used by the by post-processing software (or by the camera system) to provide adjustment of focus, depth of field, and other characteristics. Such adjustments can be made after recording, such as during playback or editing.
• As used herein, the terms light field data, light field representation, and the like can be used to mean, for example and without limitation, a representation of luminance or radiance as a function of space, direction, wavelength, and/or time. A light field can be used to mean the radiance or luminance of light within a scene, independent of any device used to capture light-related information. In some embodiments, a light field can be represented using a function with at least five dimensions as input wherein the dimensions can include three spatial dimensions and two directional dimensions. For example, the spatial dimensions can be coordinates in a Cartesian coordinate system (e.g., x, y, and z) and the directional coordinates can be angles in a spherical coordinate system (e.g., θ and φ, or azimuth and elevation). In some embodiments, a light field can be represented using a function with at least four dimensions as input where the spatial and directional dimensions are represented using a parameterization such as a light slab, light spheres, points on a plane with an angle, or the like, reducing the number of dimensions used to represent position and direction from five to four. A light field representation can include a wavelength as input, thereby representing luminance or radiance as a function of wavelength in addition to position and direction. A light field representation can include time as input, thereby representing luminance or radiance as a function of time in addition to position and direction. A light field representation can include both time and wavelength as input in addition to position and direction. In some embodiments, a separate light field representation can be used for different wavelength bands (e.g., a blue light field representation, a green light field representation, a red light field representation, a broadband light field representation, etc.). The light field representations or light field data, then, can be used to approximate or represent a portion of the light field, e.g., the portion of the light field within a scene or region imaged by one or more imaging devices.
• As used herein, the terms dense field, dense field image data, dense field image set, dense field representations, and the like can be used to mean, for example and without limitation, a unitary data set resulting from combining or aggregating a plurality of input light field representations wherein the unitary data set can be used to generate multiple viewpoints and/or images. The unitary data set can include information about relationships between input light field representations, relationships between pixels in imaging elements used to acquire the input light field data, alignment information, geometric transformations, and the like.
• It is to be understood that data such as light field data and/or dense field image data can include metadata that can be used to provide information related to the light field data and/or dense field image data. Metadata can be used, for example and without limitation, to describe a resolution of one or more imaging elements, color depth, image sensor properties, focal length of imaging elements, zoom of imaging elements, acquisition date and time, aperture settings, dense field imager information, calibration information, and other such data. The metadata can also include information that is associated with a camera operator, a setting in which the data is acquired, a position or orientation of an imaging element or dense field imager, and the like. Metadata can be used to store information that can be used to generate viewpoints, to combine image data, to process acquired data, and the like.
Dense Field Imager
• FIG. 1A illustrates a structural block diagram of an embodiment of animaging system100, such as a dense field imager or dense field camera, having a plurality of imaging blocks orfacets300, eachfacet300 having an array ofimaging elements200. Thedense field imager100 includes anarray support150 configured to position and orient the plurality offacets300. As described herein, thedense field imager100 can have a variety of configurations and can include a plurality offacets300 that are coplanar, non-coplanar, or a combination of coplanar andnon-coplanar facets300. Theimaging elements200 can be grouped intofacets300 where theimaging elements200 are mounted on or attached to a facet support. Theimaging elements200 on a givenfacet300 can be substantially coplanar with theother imaging elements200 on the same facet or theimaging elements200 can be non-coplanar such that the facet support is generally concave, convex, or has some other non-planar shape. In some embodiments, theimaging elements200 on aparticular facet300 have optical axes that are substantially parallel. In some embodiments, the optical axes of the facets300 (e.g., a geometric average of the optical axes of the imaging elements200) can be substantially parallel, can generally diverge, can generally converge, or any combination of these. In some embodiments, thedense field imager100 includesfacets300 that are not coupled to acommon support150, but are loosely physically coupled or physically separate and logically connected through a common wired or wireless connection to one or more acquisition or processing systems. In some embodiments, the plurality offacets300 can be grouped into an imaging system.
• FIG. 1B illustrates a functional block diagram of an embodiment of animaging system100, such as a dense field imager, coupled to a densefield processing system102. Thedense field imager100 includes a plurality ofimaging elements200, wherein theimaging elements200 are grouped into image blocks orfacets300. Thedense field imager100 can comprise a plurality offacets300 which are electrically coupled and which can share at least a portion of image acquisition electronics.
• Each of the plurality ofimaging elements200, as described in greater detail herein, can include optics and an image sensor such that theimaging element200 can provide image data of a scene. For example, theimaging element200 can be a conventional camera configured to provide raw, processed, and/or compressed image data corresponding to light collected from a scene. The plurality ofimaging elements200 can be identical or they can differ from one another in a variety of ways. For example,imaging elements200 can have different focal lengths, dynamic ranges, frame rates, photosensitivity, color gamut, fields of view, and the like. Theimaging elements200 can be configured to provide video in addition to images, and it should be understood that throughout this disclosure where images or image data is discussed video or video data is included. Thus, in some implementations, theimaging system100 is configured to acquire motion and still images at the same time.
• Two ormore imaging elements200 can be combined to form an image block orfacet300. The plurality ofimaging elements200 in afacet300 are configured to provide image data of a scene tofacet acquisition modules304, which can be configured to combine the image data for processing. For example, thefacet acquisition module304 can be configured to combine image data from theimaging elements200 to provide light field data, dense field data, stitched image data, combined image data, or any combination of these. In addition,facets300 can be combined in a modular fashion to dynamically change a size of theimaging system100.
• Thefacet acquisition modules304 can be configured to send the output of the module to a densefield acquisition module118 of the densefield processing system102. The densefield processing system102 can be configured to generate light field data from pixel data received from thefacet acquisition modules304. In some embodiments, the densefield processing system102 is configured to join light field data received from thefacet acquisition modules304 to generate dense field data. In some embodiments, the densefield processing system102 can be configured to perform image processing functions as described herein, such as, for example and without limitation, calibration, registration, filtering, alignment, combining, aggregating, stitching, fusion, and the like. In some embodiments, thefacet acquisition modules304 perform some or all of these functions in addition to or instead of the densefield processing system102 performing them.
• The densefield processing system102 can receive the output of thefacet acquisition modules304 and further process the image data. The densefield processing system102 can includedata storage104 for storing image data, calibration data, metadata, camera data, and the like. The densefield processing system102 can include acontroller106 configured to control operation of the components of thesystem102. Thecontroller106 can include one or more computer processors, FPGAs, ASICs, DSPs, or the like. The densefield processing system102 can include a densefield processing module108 configured to perform analysis and processing functions. Theprocessing module108 can include hardware, firmware, and/or software components. For example, theprocessing module108 can be configured to perform executable instructions stored on a computer readable medium. Theprocessing module108 can include hardware components programmed to perform defined functions, such as a FPGA configured to aggregate, combine, fuse or stitch image data. The image data can be compressed when stored, wherein data compression techniques can include any of a variety of compression and image storage techniques such as those relating to compression and storage of raw image data, including those disclosed in U.S. Patent Pub. No. 2013/0113951, entitled “Video Camera,” and filed Aug. 3, 2012, which is incorporated by reference herein in its entirety.
• The densefield processing system102 can be configured to provide a variety of processing capabilities through the densefield processing module108. For example, the densefield processing system102 can be configured to aggregate light field representations generated by each facet. The aggregated light field representations can be processed to include spatial relationship information (e.g., using a matrix) which includes information about the relative or absolute position and/or orientation of the facet used to generate the light field representation. The aggregated light field representations can also include the positions and/or orientations of each facet relative to one or more facets. As another example, the densefield processing system102 can be configured to fuse image data which can include, without limitation, joining image data at common or coinciding locations to increase information available at those locations. Thesystem102 can also be configured to stitch image data by, for example, identifying corresponding features, pixels, or light rays in regions of overlapping image data and joining image data from two or more imaging elements or facets based at least in part on the image data in the overlapping region. The densefield processing system102 can be configured to enhance a resolution of an output image by, for example, combining a plurality of input images of a scene and performing sub-pixel sharpening. The densefield processing system102 can be configured to provide substantially clear imagery of a scene that is partially occluded by combining image data from a variety of viewpoints to piece together a complete image substantially free of the occlusion. The densefield processing system102 can be configured to provide 3D imagery with a configurable inter-ocular distance by providing imagery from two, adjustable viewpoints. The densefield processing system102 can be configured to provide imagery from a variety of viewpoints and/or viewing angles wherein the viewpoints can correspond to locations with imaging elements or the viewpoints can be virtual viewpoints which do not correspond to locations with at least one imaging element. The densefield processing system102 can be configured to change a depth of field and/or a focus depth of an output image. The densefield processing system102 can be configured to provide output imagery having a higher dynamic range than any individual camera orimaging element200 used to acquire the image data. The densefield processing system102 can be configured to receive image data and/or light field data and output dense field image data.
• Theimaging system100 can be configured to acquire image data in a variety of combinations. Image data from theimaging elements200 can be combined to provide light field data, to enhance resolution of an imaged scene, to provide image data having a relatively high dynamic range, to produce video with a high frame rate, and the like. Image data acquired by theimaging elements200 can be provided to the densefield processing system102 in a raw and/or compressed format, thus allowing the densefield processing system102 to manipulate and process the image data in a variety of manners. For example, theimaging system100 can be configured to record any combination of high resolution images, light field data, dense field data, and/or smaller compressed images or video (e.g., jpeg, mpeg, TIFF, etc.). Accordingly, image data from theimaging system100 can be provided for relatively quick editing or distribution, and/or it can be combined and stored to create a larger data set for post-processing and analysis.
• In some embodiments, theimaging system100 can be an array-based imaging system that includes an array of image blocks orfacets300. Each of thefacets300 can include at least twoimaging elements200 comprising at least a sensor and lens pair. The densefield acquisition module118 can be configured to generate a light field representation for eachfacet300 using pixel data acquired by theimaging elements200 of thefacet300. The densefield acquisition module118 can be configured to utilize alignment information that specifies geometric relationships between theimaging elements200 to join light field representations by additively combining at least overlapping portions of the light field representations to create a dense field image set. The dense field image set, in some embodiments, can be configured to enable generated viewpoints having improved resolution, improved correspondence with the scene, improved perceived quality, and the like as compared to the individual light field representations.
• In some embodiments, the densefield acquisition module118 is configured to create a spatial relationship tensor that includes spatial relationships among the light field representations. The densefield acquisition module118 can use the spatial relationship tensor to stitch together non-overlapping portions of the light field representations. The output of this process can provide a dense field image set that has a substantially wider field of view than the individual light field representations. In some embodiments, the densefield acquisition module118 is configured to obtain the alignment information by processing calibration images acquired by theimaging elements200.
Imaging Elements
• FIG. 2 illustrates anexample imaging element200 that includes alens210,image sensor220, andimage acquisition electronics230. When used in theimaging system100, e.g., a dense field imager, the output of theimage acquisition electronics230 can be sent to afacet acquisition module304 configured to join image data from a plurality ofimaging elements200. Theexample imaging element200 can be a conventional camera configured to produce a color or monochromatic image or video of a scene.
• In some embodiments, a dense field imager includes a plurality ofimaging elements200. Theimaging elements200 can be arranged, for example, in an array, attached to one or more supports. The supports can position theimaging elements200 such that the imaging elements are all substantially coplanar or non-coplanar, or some fraction of theimaging elements200 can be coplanar and another fraction can be non-coplanar, or any combination of coplanar and non-coplanar. The supports can position theimaging elements200 such that the optical axes of theimaging elements200 are all substantially parallel, only a portion are substantially parallel, or none are substantially parallel. Theimaging elements200 can be mounted in such a way that, in a direction moving away from theimaging sensor220, their optical axes converge, diverge, converge along one axis and diverge along another axis, converge along one axis and are substantially parallel along an orthogonal axis, or diverge along one axis and are substantially parallel along an orthogonal axis.
• Eachimaging element200 includes asensor220 and anoptical system210, which can include one or more lens elements. The field of view of theimaging elements200 used in theimaging system100 can be different from one another or substantially identical. By combining images captured from one or more of theimaging elements200, the overall resolution of output images can be increased. Further, the portions of the captured images that correspond to the overlapping regions may be stacked to reduce noise. For instance, the noise from overlapping regions can be averaged together in some cases to provide noise reduction.
• If theimaging elements200 capture images that are aligned along pixel boundaries, the overlapping regions may be stacked directly. However, it is likely that the overlapping portions will not align exactly along pixel boundaries. That is, the portion of the image captured by a particular pixel in animaging element200 will likely not match exactly with the portion of the image captured by a pixel in anadjacent imaging element200. Rather, the portion of the image captured by a particular pixel in animaging device200 will likely partially correspond to two or more pixels inadjacent imaging devices200.
• Imaging elements200 can include an optical low-pass filter to avoid aliasing effects that could occur by sampling an image at the resolution of the sensor. Assuming the optical low-pass filter blocks optical frequencies above the Nyquist frequency for a given sensor resolution, the captured image can be up-sampled to increase the number of sample points, which allows for more accurate alignment of sample points between the images. In some cases, a low-pass filter is not used. For instance, the low pass filter in some cases can be moved out of the optical path or can be removed.
• Stacking of images can be accomplished in substantially real time by theimaging system100 having a plurality ofimaging elements200, particularly if information about the alignment of the plurality ofimaging elements200 is known in advance and/or up-sampling is not needed. Alternatively, stacking of images can be accomplished, for example, as part of a post-processing workflow, where more precise alignment of the images could be achieved using more computationally intensive algorithms.
• Similarly, stitching of images can be accomplished in substantially real time from image data from the plurality ofimaging elements200, particularly if information about the alignment of the plurality ofimaging elements200 is known in advance and/or up-sampling is not needed. Alternatively, stitching of images can be accomplished, for example, as part of a post-processing workflow, where more precise alignment of the images could be achieved using more computationally intensive algorithms.
• Theimaging element200 can include one ormore lens components210 that focuses light passing through an aperture onto an array ofpixels220. Thelens components210 can comprise a variety of lens types, and can be made of a variety of materials including glass and plastic, for example. In some configurations, thelens components210 are fixed-focal length lenses having a relatively wide depth-of-field. The lenses may have a fixed zoom ratio as well. In some embodiments, one or more of thelens components210 includes a liquid lens cell. In some embodiments, thelens components210 can have a dynamic focus and zoom ratio. In some embodiments, thelens components210 provide a relatively narrow depth-of-field. In some embodiments, thelens components210 include dyes or filters such that substantially monochromatic light or light within a defined portion of the spectrum reaches thesensor220. Thelens components210 can be removable, such that different characteristics or properties can be provided to theimaging element200 by changing thelens components210.
• In some embodiments, theimaging elements200 are configured to have a relatively small size. Keeping the size of theimaging elements200 small can have a number of advantages. For instance, maintaining a smaller form factor can increase the number ofimaging elements200 included in the system, providing improved resolution among other advantages.Smaller sensors220 also have reduced noise, improved depth of field, and are less prone to manufacturing defects rates. Also, the number of picture elements, or “pixels,” in a sensor typically determines the resolution. A large sensor may use more pixels than a small sensor. Alternatively, a large sensor may use larger pixels to decrease noise in the image. However, an increase in sensor size means there will also be an increase in the likelihood that the sensor will contain defects. Further, a larger sensor will require a larger lens, with additional expense and weight.
• It can be desirable to maintain or improve the larger sensor advantages, such as greater resolution and decreased noise, while minimizing the larger sensor disadvantages, such as increased overall system physical size and cost.
• While the size and shape of thesensors220 andlens components210 can vary betweenimaging elements200, in some embodiments thesensors220 have a substantially square form factor that is about 5 mm by 5 mm, and the lenses are about 15 mm in diameter. In some embodiments, thesensors220 have a square form factor that is smaller than about 10 mm by 10 mm. In other embodiments, thesensors220 have a rectangular form factor with a surface area that is less than about 400 square millimeters, less than about 225 square millimeters, less than about 100 square millimeters, less than about 50 square millimeters, or less than about 25 square millimeters. In various embodiments, the diameter or width of thelens components210 is less than about 50 mm, less than about 40 mm, less than about 30 mm, less than about 20 mm, less than about 15 mm, less than about 10 mm, or less than about 5 mm.
• Theimaging elements200 can also optionally include a neutral density filter for exposure control, black shading, noise correction and the like. In another embodiment, a common neutral density filter is used for all of theimaging elements200.
• Sensors220 may include, for example, an array of charge-coupled devices (CCD) or Complementary Metal-Oxide-Semiconductor (CMOS) image sensor cells, such as active-pixel sensor cells. Such image sensors are typically built on silicon chips and may contain thousands or millions of image sensor cells.
• Thesensor220 further includesoutput circuitry230 configured to process and output image information for one or more pixels. For example, theoutput circuitry230 is configured to process and digitize the analog pixel values received from the pixel array. Theoutput circuitry230 ofsensor220 in one configuration includes sets of programmable-gain amplifiers (PGAs) and analog-to-digital converters (ADCs), although a variety components may be used in various implementations. Theoutput circuitry230 presents the digitized, processed values of the currently selected set of pixels (e.g., a selected row or subset of rows) for storage and/or further processing. For example, thesensor220 may transmit the values to a memory, image processing module, or other component of theimaging system100 for storage and/or processing. In some instances, thesensor220 buffers the values for one or more rows before transmission. Depending on the embodiment, theoutput circuitry230 can be configured to process and output a single row of pixels or a subset of two or more rows of pixels at a given time. In one embodiment, thesensor220 outputs two rows at a time and can include two instances ofsimilar output circuitry230.
• In some embodiments, one ormore imaging elements200 can include anoptical sensor220 andlens configuration210 with additional lens elements near the sensor, such as a micro-lens array. Configured in this manner, theimaging element200 can be used to capture image data that can be used to produce a light field representation for a portion of the imaged scene, which can be used by theimaging system100, theimage processing system102, and/or a post-processing system to adjust focus, depth of field, or provide other effects.
• Theimaging elements200 can be heterogeneous. In one embodiment, theimaging elements200 use different color filters. For example, one ormore imaging elements200 may capture light in the green range, one or moreother imaging elements200 may capture light in the red range, and yet one or moreother imaging elements200 may capture light in the blue range. In another embodiment, theimaging elements200 use different monochromatic sensors sensitive to a portion of the visible, infrared, or ultraviolet spectrum. The result being similar to the use of different color filters.
• In one embodiment, theimaging system100 includes a first set ofimaging elements200 of fixed optical power, and a second set ofimaging elements200 having a variable optical power (e.g., liquid lens cells). In one embodiment, theimaging system100 includes a first set ofimaging elements200 including liquid lenses, and a second set ofimaging elements200 including non-liquid lenses (e.g., solid glass or plastic).
• In one embodiment, theimaging elements200 use different pixel sizes. For example, one ormore imaging elements200 may capture light in the green range using normal sized pixels, one or moreother imaging elements200 may capture light in the green range using larger pixels, one or moreother imaging elements200 may capture light in the red range using normal sized pixels, and yet one or moreother imaging elements200 may capture light in the blue range using normal sized pixels. In another embodiment, one ormore imaging elements200 may capture light using a Bayer pattern with the sensor having normal sized pixels, while one or moreother imaging elements200 may capture light using a Bayer pattern or light in the green range using larger pixels. The larger pixels tend to have less noise and may provide greater dynamic range, while the smaller pixels provide greater resolution. These descriptions are intended to be exemplary only, and other variations of color patterns and pixel sizes are within the scope of the invention.
• In one embodiment, theimaging elements200 use different sensor sizes. In one embodiment,imaging elements200 use sensors having different aspect ratios.
• Theimaging elements200 can further have differently sized apertures and/or focal lengths. For instance, a first group of one ormore imaging elements200 can have a first aperture size and/or focal length, and a second group of one ormore imaging elements200 can have a second aperture size and/or focal length. In one embodiment,imaging elements200 in the first group have relatively wide apertures and capture images within a relatively short depth of field, while imagingelements200 in the second group have relatively narrow apertures, and capture images within a relatively large depth of field. Depending on the desired photographic effect, image data from the first group, the second group, or a combination thereof can be utilized as appropriate. Similarly, theimaging elements200 having different focal lengths can be used to provide focus control. The use of a depth map to utilizeimaging elements200 having different apertures and/or focal lengths is described in greater detail herein.
• In yet other embodiments, a first group of one ormore imaging elements200 can include a lens element(s)210 made of a first material (e.g., glass), and a second group of one ormore imaging elements200 can include a lens element(s)210 made of a second material (e.g., plastic). Or a first group ofimaging elements200 may include a lens element(s)210 having a first type of coating (e.g., a first type of anti-reflective coating) applied thereon, while a second group ofimaging elements200 may include a lens element(s)210 having a second type of coating (e.g., a second type of anti-reflective coating) applied thereon.
• In addition, theimaging elements200 can be configured to capture image data having different ranges of wavelength and frequency. For instance, particular groups ofimaging elements200 can be configured to image data in one or more of the visible, infrared, ultraviolet, x-ray and/or ultraviolet spectrums.
• In certain cases, theimaging system100 can exploit the multi-imaging element configuration by calibrating theimaging elements200 in theimaging system100 using theother imaging elements200. For instance, one or more test images can be captured, and the results can be normalized or otherwise processed to diagnose theindividual imaging elements200. Based on the processing, theimaging system100 can calibrate theindividual imaging elements200 in the array to adjust color, white balance, camera response curves, and the like. In some cases, instead of, or in addition to, calibrating theindividual imaging elements200, the calibration results are used in post-processing to adjust the captured image data.
• In some embodiments, theimaging system100 includes a control configured to adjust an angular relationship betweenimaging elements200. The control can be configured to adjust a direction of the optical axis, focal length, aperture, zoom ratio, or the like forindividual imaging elements200.
Imaging Blocks or Facets
• FIG. 3 illustrates an example of an imaging block orfacet300 of animaging system100, such as a dense field imager. Thefacet300 comprises a plurality ofimaging elements200 with overlapping fields of view. Thefacet300 includes anelement support302 configured to substantially secure theimaging elements200 in a particular position and/or orientation. Thefacet300 includesfacet acquisition module304 configured to receive data from theimaging elements200. In some embodiments, thefacet acquisition module304 receives raw pixel data from theimaging elements200. In some embodiments, thefacet acquisition module304 receives processed image data from theimaging elements200, such as compressed image data.
• Thefacet acquisition module304 can be a common acquisition board for allimaging elements200 on thefacet300. For example, thefacet acquisition module304 can include a FPGA configured to receive image data from theimaging elements200 and output processed image data for the densefield acquisition module118 of thedense field imager100.
• Theimaging elements200 can be mounted on anelement support302 where theelement support302 can be substantially rigid or it can be flexible. For example, theelement support302 can change shape to affect an overlapping of the fields of view of theimaging elements200, such as becoming convex to decrease the overlap between fields of view of adjacent imaging elements or becoming concave to increase overlap. Theelement support302 can have a variety of shapes such as planar, concave, convex, or some other regular or irregular configuration. Theelement support302 can be configured to emphasize a desirable output, such as increasing an overlap of the fields of view ofimaging elements200 where it is desirable to fuse image data to increase resolution, decrease noise, increase position accuracy, and the like. Theelement support302 can be configured to decrease an overlap of the fields of view ofadjacent imaging elements200 to increase a composite field of view of thefacet300. Theelement support302 can be flat, arched, or some other geometric or non-geometric shape.
• Theimaging elements200 can be arranged in any particular fashion on theelement support302. For example, theimaging elements200 can be in rows, columns, staggered, in a circular pattern, in a random configuration, an irregular configuration, in a circular or hexagonal configuration, or the like.
• In some embodiments, theimaging elements200 can have heterogeneous properties configured to enhance particular aspects of thedense field imager100. For example, theimaging elements200 can be configured to be smaller to increase a density ofimaging elements200 on the facet, to decrease a size of thedense field imager100, to increase sharpness of an image, and the like. As another example, the aspect ratios of theimaging elements200 can be configured to provide a desired composite field of view. The image sensors can be rectangular and can be oriented such that the longer dimension is substantially vertical rather than horizontal, which is a typical orientation for an image sensor when acquiring video, for example. The image sensors can be a mix of square, circular, and/or rectangular shapes. Theimaging elements200 can be oriented pointing in different directions. Theimaging elements200 can be configured to have different focal lengths, resolutions, sensitivities, etc. Differences inimaging elements200 can be accounted for in image processing at thefacet acquisition module304, the densefield acquisition module118, or the densefield processing module108.
• FIG. 4 illustrates an example embodiment of afacet300 comprising a plurality ofimaging elements200 coupled together on afacet support302. Theexample facet300 comprises 16imaging elements200 arranged in a 4×4 grid. Different numbers and configurations ofimaging elements200 are possible, and in some configurations the number ofimaging elements200 perfacet300 is different. Thefacet300 is electrically coupled to afacet acquisition board306, wherein thefacet acquisition board306 is configured to receive image data from theimage sensors200 and/or data from facet acquisition modules as described above with reference toFIG. 3. In some embodiments, thefacet acquisition board306 is configured to receive data fromindividual imaging elements200 and combine the data into a combined output. In some embodiments, thefacet acquisition board306 is configured to receive data from eachimaging element200 and combine the data into a light field representation. Combining image data can include any of the image processing functions described herein, including without limitation, fusing, stitching, enhancing resolution, producing light field data, and the like. As configured in the illustrated example ofFIG. 4,facets300 can be configured to be joined together in a modular fashion, such that a modular imaging system can be built using multiple facets grouped together, e.g., to form a modulardense field imager100.
• Data acquired and processed on thefacet acquisition board306 can be stored on theboard306 and/or it can send the data to another location. Thefacet acquisition board306 can be configured to send the data over a wired connection or over a wireless connection. Wired connections can include any cabled connection including, without limitation, USB cables, FireWire, Ethernet, serial cables, coaxial cables, twisted pairs, custom cabling, or the like. Wireless data transmission can be accomplished using radio frequency techniques including 802.11 standards (e.g., WiFi and/or WiGig), Bluetooth®, CDMA, GSM, NFC, or the like. Theface acquisition board306 can include one or more antennas and one or more transceivers configured to send and receive data. In some embodiments, the wired and/or wireless transfer rates can exceed 20 Gbps, 10 Gbps, 5 Gbps, 1 Gbps, or 500 Mbps. Using parallel wireless transmission techniques with multiple frequency bands and high speed protocols can enable thefacet300 and/or groups of facets to send all acquired data at an acquisition frame rate to processing and/or storage systems that are remote from thefacet acquisition board306. In some embodiments, the acquisition frame rate can be at least about 240 fps, at least about 5 fps and/or less than or equal to about 240 fps, at least about 10 fps and/or less than or equal to about 120 fps, at least about 24 fps and/or less than or equal to about 60 fps, at least about 30 fps and/or less than or equal to about 40 fps, or less than or equal to about 5 fps.
• Thefacet acquisition board306 can include one or more computer processors, FPGAs, ASICs, DSPs, or the like configured to process image data. The components of thefacet acquisition board306 can be configured to receive image data from a plurality ofimaging elements200 and output light field data. The components of thefacet acquisition board306 can be configured to receive pixel data from imagingelements200 and output light field data. In some embodiments, thefacet acquisition board306 provides some preliminary image processing functionality and sends the output of its processing to a processing station for editing, coloring, re-focusing, etc.
• In some embodiments, eachimage facet300 includes acquisition electronics (e.g., computer processors, ADCs, ASICs, FPGAs, DSPs, etc.) configured to collect image data from theimaging elements200 associated with thefacet300. The acquisition electronics can be configured to pass the image data to thefacet acquisition board306, to another image facet acquisition electronics (e.g., chaining together data collection for image facets), to a remote system (e.g., a system that is not physically coupled to the dense field imager but that may be wired or wirelessly connected to the dense field imager for data transfers), or the like. In some embodiments, thefacet acquisition board306 includes electronic elements configured to receive and/or consolidate output from one ormore imaging elements200. For example, thefacet acquisition board306 can be configured to receive a first image data set representative of a first portion of a light field, to receive a second image data set representative of a second portion of a light field, and to combine the first and second image data sets to form a third image data set wherein the third image data set can include light field data, combined image data, dense field data, a dense field image data set, a spatial relationship tensor, a 5 dimensional tensor representative of a light field, a tensor having five or more dimensions representing a combined light field representation and information for generating viewpoints from the combined light field representation, and/or one or more images.
• FIGS. 5A and 5B illustrate schematic drawings showing adense field imager100 with an array ofimaging elements200 coupled to a common support andacquisition board505. As illustrated, thedense field imager100 includes an array of 192imaging elements200, the array having 8 rows and 24 columns. Other configurations are possible including different numbers of imaging elements200 (e.g., less than 10, more than 10, more than 20, more than 50, more than 100, more than 200, more than 500, less than 500, etc.) and different arrangements of imaging elements200 (e.g., a regular array, a staggered array, circular, hexagonal, irregular or random distribution, etc.). The example embodiment illustrated in the schematic ofFIG. 5A includespassive cooling elements510 configured to provide cooling to the processing and acquisition electronics included on the common support andacquisition board505. The example embodiment illustrated in the schematic ofFIG. 5B includes data acquisition andprocessing elements515 at various locations on the common support andacquisition board505. As described herein, the data acquisition andprocessing elements515 can be configured to acquire image data and/or to generate light field data and to output combined image data, light field data, combined light field data, and/or dense field data.
• In some embodiments, the array ofimaging elements200 on the common support andacquisition board505 can be logically and/or physically divided into facets. The facets can include 2 ormore imaging elements200. The multiple facets on the common support andacquisition board505 can be part of a modulardense field imager100, where multiple arrays ofimaging elements200 can be joined to form a largerdense field imager100. For example, adense field imager100 can include two or more arrays ofimaging elements200 mounted to different common support andacquisition boards505. Thecommon supports505 can be physically joined together or they can be physically separate but configured to send their data to a common processing system. In some embodiments, an image processing system can receive data from a plurality of modulardense field imagers100 and combine the data to produce dense field image data.
• Thecommon acquisition board505 can be configured to combine image data in a staged fashion. For example, thecommon acquisition board505 can receive image data from imagingelements200 in a facet and combine the image data to form a light field representation. Adjacent light field representations can then be combined to form dense field image data. This process can proceed until image data from substantially allimaging elements200 has been combined into a dense field representation.
• In some embodiments, a subset of theimaging elements200 on the common support andacquisition board505 are dedicated to monitoring thedense field imager100. In some embodiments, the rest of theimaging elements200 are used to generate light field data or dense field data. In some embodiments, theimaging elements200 that are used to monitor the system can change according to monitor criteria. The monitor criteria can include criteria regarding a quantity of image data to be produced for use in creating light field data, a quality factor for image data, and the like.
Field of View of a Dense Field Imager
• The field of view of adense field imager100 is a result of the fields of view of thecomponent facets300, and the fields of view of thecomponent facets300 are a result of the fields of view of theimaging elements200 on afacet300. Fields of view ofimaging elements200 and fields of view offacets300 can be configured to substantially overlap and/or to overlap primarily along neighboring or adjacent edges. For example,imaging elements200 in afacet300 can be configured to have substantially overlapping fields of view to facilitate the generation of light field data, andfacets300 in adense field imager100 can be configured to have fields of view that primarily overlap along the edges to increase a resulting field of view of a dense field representation. Other configurations are possible as well, such as havingimaging elements200 whose fields of view overlap primarily along the edges andfacets300 whose fields of view substantially overlap. It may be advantageous, in some embodiments, to haveimaging elements200 andfacets300 whose respective fields of view substantially overlap with one another to produce relatively high quality dense field data. It may be advantageous, in some embodiments, to haveimaging elements200 andfacets300 whose respective fields of view overlap with one another primarily along adjacent edges to produce dense field data having a relatively large field of view.
• FIG. 6A illustrates a field of view for anindividual imaging element200, such as animaging element200 in adense field imager100. The lens field ofview212 is larger than the sensor field ofview214 that is actually captured by thesensor220. Only the field of views are depicted inFIG. 6A. Physical components such as aphysical sensor220 orlens210 are not shown inFIG. 6A.
• FIGS. 6B and 6C illustrate overlapping fields of view for example arrays ofimaging elements200 that are arranged in rectangular and hexagonal grids, respectively. Theimaging elements200 can be configured in any number of ways including, for example, in rectangular and hexagonal grids (as shown), along concentric circles, staggered, random distributions, grouped in facets with the facets arranged in any of the preceding configurations, etc. As shown, the fields ofview212 for the lenses and the fields ofview214 for the sensors overlap with one another, providing some commonality in image scene content captured by neighboringimaging elements200. Again, physical components such as aphysical sensor220 orlens210 are not shown inFIG. 6B or 6C. Theimaging elements200 themselves may be physically spaced from one another, although the fields of view overlap. For example, theimaging elements200 can be part of adense field imager100. Theimaging elements200 can be substantially coplanar with substantially parallel optical axes, non-coplanar with substantially parallel optical axes, substantially coplanar with non-parallel optical axes, and/or non-coplanar with non-parallel optical axes. The overlapping fields of view represented inFIGS. 6B and 6C can represent the overlap at a given distance from theimaging elements200. In some embodiments,imaging elements200 that form afacet300 can be co-planar with parallel optical axes andfacets300 that form adense field imager100 can be non-coplanar and/or have non-parallel optical axes. In some embodiments, theimaging elements200 can have fields of view that differ from one another. For example, afirst imaging element200 can have a relatively wide field of view and anadjacent imaging element200 can have a relatively narrow field of view.Imaging elements200 on afacet300 can have similar fields of view or they can cover a range of fields of view to provide different coverage of the image scene.Facets300 can have differing fields of view as well such that adense field imager100 can have facets with non-uniform fields of view. This can increase the modular capabilities of thedense field imager100 as it can utilize a number of different types ofimaging elements200 and/orfacets300, allowing for a number of different configurations that can be tailored for a desired or particular purpose.
• As shown, the overlap can occur primarily in edge regions of the sensor. In other embodiments, the overlap can also occur in the center regions of the sensor. Limiting the overlap to primarily in the edge regions of the sensor provides for a larger effective field of view and/or more resolution given a particular set of sensors, while overlapping towards the center regions of the sensor provides for improved fusion, stitching, and/or noise reduction of images. In some embodiments, the overlapping image content can be used to reduce the effects of vignetting or lens distortions, which can typically be more pronounced in edge regions of the sensor.
• In some embodiments, afacet300 can be configured to haveimaging elements200 whose fields of view substantially overlap. This can be useful when combining the image data from theimaging elements200 to form a light field representation. The resulting field of view of the light field representation can be the total area covered by two ormore imaging elements200, the total area covered by substantially all or all of theimaging elements200 of thefacet300, or an area substantially limited to where two ormore imaging elements200 have overlapping fields of view.
• In some embodiments, image data fromfacets300 of adense field imager100 can be used to generate light field representations whose fields of view substantially overlap. With substantially overlapping fields of view, thedense field imager100 and/or image processing systems can use the overlapping light field representations to produce dense field data whose resulting field of view can be an area covered by all or substantially all of thefacets300 in thedense field imager100. The quality of the resulting dense field data can be increased based at least in part on the quantity of information available over the combined field of view. An increase in the quality of the dense field data can correspond to a reduction in noise, an increase in resolution, a decrease in uncertainties related to luminance or radiance values, improved spatial relationship tensors which can be used to generate higher quality viewpoints or images, and the like.
• In some embodiments, image data fromfacets300 of adense field imager100 can be used to generate light field representations whose fields of view overlap primarily along the edges of their respective fields of view. In these embodiments, the field of view of a resulting dense field representation can be increased or maximized, covering a relatively large area. This can be advantageous where it is desirable to acquire, with a relatively small number ofimaging elements200 orfacets300, dense field data having a wide field of view.
• FIGS. 7A to 7F illustrate a plurality ofimaging elements200a,200b,200cthat are spaced on a variety of surfaces and that have a variety of fields of view. In some embodiments, theimaging elements200a,200b,200care part of afacet300 on adense field imager100. Although the description that follows discusses the effects of these configurations with respect toimaging elements200, the same discussion can be applied to the fields of view offacets300 of adense field imager100. For example, rather than illustratingimaging elements200a,200b,200c,FIGS. 7A to 7F could illustrate a plurality of facets300a,300b,300cthat are spaced on a variety of surfaces and that have a variety of fields of view. The orientation and fields of view of theimaging elements200a,200b,200ccan be configured to enhance or optimize generation of light field data. Similarly, the orientation and fields of view offacets300 in adense field imager100 can be configured to enhance or optimize generation of dense field data.
• FIG. 7A illustrates a plurality ofimaging elements200a,200b,200cthat are spaced on a planar surface. The cross-hatched region is the region of overlap forimaging elements200aand200b. To determine and/or control where the overlap region occurs, the distance between imaging elements, d, and opening angles, θ, can be used. For example, for imagingelements200aand200bwith substantially the same opening angle, θ (which is half of the field of view and is measured from the optical axis OA to the outer edge of the field of view), the distance to the overlap region, l, is equal to d/(2*tan(θ)). The area of overlap at a distance l+x from theimaging elements200a,200bis calculated based on the similarity of the triangles (with the first triangle being defined as the triangle with d as the base and l as the height and the second triangle the overlapping region). At the distance l+x from the imaging elements, the cross-sectional area of overlap is equal to 0.5*d*(x/l).
• FIG. 7B illustrates a plurality ofimaging elements200a,200b,200cthat are spaced on a concave surface. The field of view of theimaging elements200a,200b,200csubstantially overlap. A geometrical calculation that is similar to that described with reference toFIG. 7A is shown in this figure for the overlapping region. Here, because the imaging elements are mounted on a concave surface, an additional parameter is used. The angle α is used to represent an offset angle of the optical axis compared to an axis that bisects the distance between the imaging elements and intersects with the optical axes OA_aand OA_bwhere they intersect. Another way to understand this axis is that it represents a combined or average optical axis forimaging elements200aand200b. The distance, l, to the intersection of the fields of view is then d/(2*tan(θ+α)). The area of overlap at a distance l+x along the combined optical axis is 0.5*d*(x/l), if the respective outer fields of view diverge or are parallel (e.g., θ≥α). If the respective outer fields of view converge (e.g., θ<α), the area is a bounded quadrilateral.
• FIG. 7C illustrates a plurality ofimaging elements200a,200b,200cthat are spaced on a convex surface. The field of view of theimaging elements200a,200b,200care substantially adjacent. The calculation of the overlapping region here proceeds as with that shown inFIG. 7B, with the distance, l, being d/(2*tan(θ−α)). The area of overlap at a distance l+x along the combined optical axis is 0.5*d*(x/l). The condition that is satisfied to provide an overlapping region is θ>α.
• FIG. 7D illustrates a plurality ofimaging elements200a,200b,200cthat are spaced on a planar surface, with a decrease in the angular view.
• FIG. 7E illustrates a plurality ofimaging elements200a,200b,200cthat are spaced on a concave surface, with a decrease in the angular view. The field of view of theimaging elements200a,200b,200csubstantially overlap. Because of the decrease in angular view, the radius of the concave surface increases as compared to the concave surface ofFIG. 7B.
• FIG. 7F illustrates a plurality ofimaging elements200a,200b,200cthat are spaced on a convex surface, with a decrease in the angular view. The field of view of theimaging elements200a,200b,200care substantially adjacent. Because of the decrease in angular view, the radius of the convex surface increases as compared to the convex surface ofFIG. 7C.
• FIGS. 7C and 7F illustrate that varying the alignment of a plurality of imaging elements in a convex fashion can provide increased resolution, even if the field of view changes.
• FIGS. 7A and 7D illustrate that even when the alignment of a plurality of imaging elements does not change, using the plurality of imaging elements provides for increased resolution and reduced noise, even if the field of view changes.
• Theimaging elements200 may be mounted, for example, on a flexible surface or membrane. The surface may adjust with adjustments to the zoom settings of theimaging elements200. For example, the flexible surface may adjust to provide substantial overlap in the field of view betweenimaging elements200. As another example, the flexible surface may adjust to keep the field of view betweenimaging elements200 substantially adjacent, allowing for some overlap to stitch the images together. The configurations illustrated inFIGS. 7A to 7F can each be accomplished in separate devices, or in a single, adjustable device, if the sensors are mounted to a support such as a membrane which is capable of a tympanic displacement. The sensors can be positioned on a membrane radially symmetrically about a displacement axis which is coincident or parallel to the primary viewing axis of the array. Slight displacement of the membrane along the displacement axis, away from the field of view while restraining the periphery of the membrane, will deflect the membrane into a concave configuration as seen inFIG. 7B. Displacement of a point on the membrane along the displacement axis in the direction of the field of view will produce a convex configuration as seen inFIG. 7C. Displacement can be controllably accomplished in any of a variety of ways, such as by magnetic displacement, acoustical displacement or a variety of mechanical or electromechanical systems for providing small mechanical displacement in response to a control signal.
• In some embodiments, the imaging elements200 (or the facets300) can be mounted or attached to a surface having a modifiable geometrical configuration wherein a modification to the surface geometry modifies a field of view of the associated facet300 (e.g., when modifying the surface geometry of the surface supportingimaging elements200a-c) or the dense field imager (e.g., when modifying the surface geometry of the surface supporting facet300). In some embodiments, such a configuration can be implemented on a portable dense field imager or dense field camera. In some implementations, the surface geometry can be configured to be simply modified by a user of the dense field imager. For example, the user can adjust the surface geometry through a single mechanical element, such as a knob, slider, rod, or the like. In other cases, multiple controls can be included, such as where a first control adjusts the surface geometry in a first manner (e.g., along a horizontal aspect of the imager) and a second control adjusts the surface geometry in a second manner (e.g., along a vertical aspect of the imager). In some implementations, a non-mechanical control element can be provided to allow for adjustment of the surface geometry, such as a graphical user interface (GUI) or other form of electronic interface. Including such mechanical, GUI-based, or other types of controls allows for relatively precise and straightforward tuning of the surface geometry. In some implementations, the surface geometry can be made to be planar, convex, concave, or another suitable surface geometry or combination of surface geometries.
• In some cases, the distance between sensors can vary. For example, an array of 4×4, 2×2, 6×6, 10×10, 18×9, 6×3, or 10×5 (horizontal×vertical)imaging elements200 may be arranged so that the spacing between sensors is approximately ¼ inch between sensor centers, approximately ½ inch between sensor centers, approximately one inch between sensor centers, approximately two inches between sensor centers, or approximately four inches between sensor centers.
• Theimaging elements200 may also be mounted on a framework that provides for variable distance between sensors. For example, an array of 4×4, 2×2, 6×6, 10×10, 18×9, 6×3, or 10×5 (horizontal×vertical)imaging elements200 may be arranged so that the spacing between sensors adjusts from approximately one inch between sensor centers to approximately four inches between sensor centers. In other cases, the spacing can be adjusted from approximately ¼ inch between sensor centers to approximately ½ inch, approximately 1 inch, or approximately 4 inches between sensor centers. In yet further embodiments, the spacing can be adjusted from approximately ½ inch between sensor centers to approximately 1 inch, or approximately 4 inches between sensor centers.
• WhileFIGS. 7B, 7C, 7E, and 7F illustrate arranging theimaging elements200a,200b,200cin a concave or convex fashion, other optical adjustments could be used to obtain the desired alignment. For example, U.S. Patent Publication 2009/0141352 titled “Liquid Optics Image Stabilization,” the entirety of which is incorporated by reference herein, describes the use of liquid lens cells to stabilize an image along one or more axis. The image stabilization described in that publication could be adapted to provide image alignment of the plurality of imaging elements as illustrated inFIG. 7A, thus avoiding the need for physically aligning the imaging devices in a concave or convex manner as illustrated inFIGS. 7B, 7C, 7E, and 7F.
• FIGS. 7B and 7E illustrate that varying the alignment of a plurality of imaging elements in a concave fashion can provide for multiple images to be substantially aligned, even if the field of view changes. The convex and concave configurations shown inFIGS. 7B and 7C can also be used to counteract the effect of parallax that may result from the spacing between the imaging elements. Additionally, in some cases, images can be captured in sequence from adjacent imaging elements to reduce parallax issues for video.
• Further,imaging elements200a,200b,200ccould use liquid lens cells to provide zoom and/or focus, as described in U.S. Pat. No. 7,855,838, titled “Liquid Optics Zoom Lens and Imaging Apparatus,” the entirety of which is incorporated by reference herein.
• FIGS. 7A to 7F illustrateimaging elements200a,200b,200cfor ease of illustration. However, it is to be understood that multiple imaging elements could be used and not limited in number to three. Theimaging elements200 could be arranged in a variety of patterns, including, for example, along a linear axis, along a square grid, along a diagonal grid, along a triangular grid, or in a hexagonal pattern. The arrangement could be planar, or, for example, concave or convex, or irregular.
• In some embodiments, a dense field imager can include a plurality offacets300, eachfacet300 having field of view. The dense field imager can be configured such that the field of view of facet substantially overlaps with a field of view of a neighboring or adjacent facet. This can be accomplished by mounting the facets on a concave surface or by orienting the various imaging elements on the facets to have their primary optical axes to be generally converging along at least one direction. The dense field imager can then use the image data in the overlapping regions to enhance a parameter of an output image generated by the image processing system, enhancement of the parameter based at least in part on information provided by pixels within the substantially overlapping fields of view of neighboring image blocks. The additional information provided by pixels in the overlapping region can be utilized in extracting additional information from a scene relative to a situation where a single image sensor is used to capture an image of a scene. Because the image sensors provide for some redundancy in the overlapping regions, the additional pixel data can be used to extract additional parameters or they can be tuned to extend the functionality of the imaging elements. For example, the enhanced image parameter can be a brightness of at least a portion of the image, a dynamic range of at least a portion of the image, or a color of at least a portion of the image, or any combination of these or other image properties. The overlapping fields of view can also be used to generate video of a scene with a higher effective frame rate by staggering the shutters of each of the imaging elements.
• FIG. 8 illustrates an exampledense field imager100 having an array ofimaging elements200 wherein eachimaging element200 includes a monochromatic sensor. The monochromatic sensors can be manufactured using techniques described herein. A monochromatic sensor can be configured to be sensitive to light falling within a region of the electromagnetic spectrum, which can span multiple “color” bands. For example, a monochromatic sensor can be sensitive to red light, to green light, to blue light, to white or broadband light (e.g., light that covers all or substantially all of the visible spectrum or light that covers from red to blue light), to infrared light, to ultraviolet light, and the like. When used in combination with other monochromatic sensors, thedense field imager100 can provide dense field data capable of reproducing a wide variety of colors and light intensities. Using an array of monochromatic sensors can reduce or eliminate crosstalk between color channels as compared to multi-chromatic image sensors (e.g., a Bayer sensor). By reducing or eliminating crosstalk between color channels, the sensors can be reduced in size. Reducing the size of the image sensor can result in improved depth of field and/or increased perceived sharpness of resolution. The increased sharpness can be based at least in part on a reduction in interpolation as compared to images produced using traditional multi-chromatic sensors. The manufacture of monochromatic sensors is described in greater detail herein.
• Themonochromatic imaging elements200 can be configured to have overlapping fields of view, as described herein, and overlap regions can be used to produce color information in an image. For example, in a region where a red, a blue, and a green imaging element acquire image data, an output image can be produced with an appropriate color at each output pixel or position. In some embodiments, a fourth imaging element can be included that has a white or broadband sensor (e.g., a panchromatic sensor). The white sensor can provide an ability to control neutral density and/or dynamic range of a resulting image.
• Using monochromatic sensors, color crosstalk can be reduced or eliminated. Color crosstalk can occur where a photon of one color produces an electrical output in a pixel of another color (e.g., photon conversion crosstalk), which can occur when a photon strikes a boundary between pixels or when a pixel becomes saturated and the signal from the saturated pixel affects neighboring pixels (e.g., bloom crosstalk). The bloom crosstalk can be particularly prevalent for white pixels which can saturate faster than pixels of other colors. In a monochromatic sensor, photon conversion crosstalk is reduced or eliminated, or the effect of crosstalk is reduced because pixels are all of the same color. Bloom crosstalk is reduced or eliminated as well because when one pixel saturates it is likely that the neighboring pixels of the same color will be saturated or nearly saturated as well, reducing the effect of bloom crosstalk.
• Images produced using imaging elements with monochromatic sensors can be improved by individually controlling sensors. Different color sensors can be treated differently based on a variety of conditions. Imaging elements can be controlled to have different shutter speeds, apertures, exposures, and the like based at least in part on the color of the sensor. For example, white sensors can be configured to have quicker exposure times to reduce saturation. Color sensors can have longer exposure times to increase photon collection in low-light situations. A particular color can have a longer or shorter exposure time for scenes where one or more colors may be more prevalent. This feature can be used to create a dense field imager that does not utilize a traditional Bayer configuration, where there are two green pixels for every red and blue pixel. The green imaging element can be configured to have a higher dynamic range, a longer exposure, a wider aperture, or any combination of these or the like to enhance green photon collection as compared to red and blue imaging elements. In some embodiments, thedense field imager100 includes electronics that control imaging elements individually. In some embodiments, the control electronics can be configured to treat imaging elements with similar monochromatic sensors in a similar or uniform fashion.
Example Planar Dense Field Imager
• FIGS. 9A to 9C show an example embodiment of a planardense field imager900 having integrated coolingelements910 and integrated acquisition andprocessing electronics915. In the illustrated example embodiment, thefacets930 are mounted on a common support and are coplanar, but, in some embodiments, thedense field imager900 can includefacets930 mounted on physically disparate supports and/or supports that are non-coplanar. In some embodiments, thefacets930 can be mounted on supports that are non-coplanar and that form planes that are not parallel to one another. As described herein, this can be used to create desirable overlap in image acquisition and/or increase a field of view of a resultant dense field representation. The illustrated example planardense field imager900 is configured to be a unitary structure, but, in some embodiments, thedense field imager900 can be configured to be modular where facets can be added and/or taken away to create a variety of configurations.
• The planardense field imager900 includesfacets930 where eachfacet930 includes 16imaging elements920, arranged in a 4×4 grid. Other configurations are possible as well, including varying the number and arrangement of theimaging elements920. For example, theimaging elements920 can be arranged in a rectangular or hexagonal grid, along concentric circles, in an irregular or random distribution, or any combination of these. The number ofimaging elements920 on afacet930 can be can be at least 2 and/or less than or equal to 400, at least 3 and/or less than or equal to 200, at least 4 and/or less than or equal to 100, or any number between 2 and 400 (e.g., 2, 3, 4, 5, 6, 7, 8, etc.). The number ofimaging elements920 perfacet930 can vary from facet to facet as well.
• The planardense field imager900 includes a grid of 12facets930, arranged in a 6×2 grid. As illustrated in this example, thefacets930 can be configured to be arranged in a regular rectangular grid, but other configurations are possible as well. For example, thefacets930 can be staggered, they can be configured as hexagonals and arranged on a hexagonal grid, thefacets930 can be distributed according to any regular pattern, thefacets930 can be arranged randomly or irregularly, or any combination of these. In some embodiments, thefacets930 are arranged to form a square or a rectangle whose width is shorter than the height.
• The example planardense field imager900 is approximately 36 cm wide by 26 cm high by 6 cm deep, including thecooling elements910 and theintegrated electronics915. Thedense field imager900 can have different sizes and can be larger or smaller than the illustrated example. For example, the width and/or height of thedense field imager900 can be at least about 6 cm and/or less than or equal to about 100 cm, at least about 9 cm and/or less than or equal to about 50 cm, or at least about 15 cm and/or less than or equal to about 40 cm. In some embodiments, thedense field imager900 is modular wherein each module can have a particular size. The combination of the dense field imager modules can have any size and/or configuration, based at least upon the desired application. For example, in some embodiments, a modulardense field imager900 can be configured to cover a horizontal field of view of about 360 degrees and/or a vertical field of view of about 180 degrees.
• Eachfacet930 has an imaging area (e.g., the portion of thefacet930 with the imaging elements920) that is approximately 6 cm by 6 cm and an electronics area (e.g., the portion of thefacet930 with the facet acquisition electronics) that is approximately 6 cm by 8 cm. The imaging area of thefacet930 can be at least about 1 cm²and/or less than or equal to about 600 cm², at least about 3 cm²and/or less than or equal to about 400 cm², at least about 10 cm²and/or less than or equal to about 100 cm². In some embodiments,facets930 can be implemented on a wafer and can have appropriate wafer-scale dimensions, as described herein. The electronics area of thefacet930 can have a variety of sizes and, in some embodiments, the facet electronics can be integrated into thedense field imager900 apart from thefacet930.
• In some embodiments, theimaging elements920 and/or thefacets930 can be configured to be movable relative to one another. Thedense field imager900 can include actuators configured to change a position and/or orientation ofindividual facets930 and/orindividual imaging elements920. Theimaging elements920 and/orfacets930 can be moved to increase or decrease an overlap in their fields of view. The movement of theimaging elements920 and/or thefacets930 can be controlled by an operator or it can be automatically controlled by thedense field imager900. Movement can be based at least in part on a target field of view, a targeted amount of overlapping image information, a viewing angle, or any combination of these.
Example Non-planar Dense Field Imagers
• FIGS. 10A to 10C illustrate an example embodiment of adense field imager1000 having a plurality offacets1030 that are mounted in a concave fashion. Thedense field imager1000 can include acommon support1005 that lies within an x-y plane (e.g., the plane of the paper inFIG. 10A). The plurality offacets1030 can be attached or secured to thecommon support1005 through mountingelements1040.
• The orientation of the plurality offacets1030 will now be described relative to a coordinate system having an x-axis and y-axis oriented as shown inFIG. 10A, where the z-axis would extend outward from the page. The top and side views respectively illustrated inFIGS. 10B and 10C show the relative configurations of the x-, y-, and z-axes such thatFIG. 10A lies in the x-y plane,FIG. 10B lies in the x-z plane, andFIG. 10C lies in the y-z plane. Within this coordinate system, a primaryoptical axis1002 of thedense field imager1000 can be defined as a line that is parallel to the z-axis and that intersects the x-y plane in the geometric center of the plurality offacets1030, as illustrated by the circle with a dot. Additionally, a facet plane can be defined as a plane that contains theimaging elements1020 of thatfacet1030. Each facet has a facetoptical axis1032 that can be defined as a line perpendicular to the facet plane that intersects the facet plane in the geometric center of the facet'simaging elements1020.
• The mountingelements1040 can be configured to orient one ormore facets1030 such that the facetoptical axes1032 are not parallel to the primaryoptical axis1002. In the example embodiment illustrated inFIGS. 10A to 10C, the mountingelements1040 secure thefacets1030 in such a way that the facetoptical axes1032 generally converge towards the primaryoptical axis1002 in both the x- and y-directions. Other configurations are possible as well. For example, the mountingelements1040 can be configured to orient one ormore facets1030 such that the facetoptical axes1032 remain parallel to the primaryoptical axis1002 in either the x- or y-direction while converging in the other direction (e.g., parallel in the x-direction and converging in the y-direction). The mountingelements1040 can be configured to orient one ormore facets1030 such that the facetoptical axes1032 diverge from the primaryoptical axis1002 in either the x- or y-directions while converging in the other direction (e.g., diverging in the x-direction and converging in the y-direction). The mountingelements1040 can be configured to orient one ormore facets1030 such that some of the facetoptical axes1032 remain parallel to the primary optical in a first direction while converging in a second, orthogonal direction, other facetoptical axes1032 remain parallel in the second direction and converge in the first direction, other facetoptical axes1032 diverge in a first or second direction and converge in the other direction, or any combination of these.
• In some embodiments, the mountingelements1040 generally orient the plurality offacets1030 along an interior surface of a sphere, an ellipsoid, a cylinder, a cylindrical ellipsoid, a paraboloid, a hyperboloid, a polyhedron, or some other regular or irregular surface. In some embodiments, the mountingelements1040 can be configured to be movable through mechanical adjustments and/or through actuators configured to change an orientation of one or more facet planes. The mountingelements1040 can be manually and/or automatically adjustable. Thefacets1030 can be mounted to thecommon support1005 to provide desired or targeted overlapping fields of view. The overlapping fields of view can be configured to provide image data from a majority ofimaging elements1020 and/orfacets1030 in a targeted region where on object or region of interest is located.
• FIGS. 10D to 10F illustrate fields of view of some of thefacets1030 on thedense field imager1000 illustrated inFIGS. 10A to 10C.FIG. 10D illustrates the overlapping fields of view of thefacets1030 as viewed in the y-z plane to show the vertical overlapped regions.FIG. 10E illustrates the overlapping fields of view of thefacets1030 as viewed in the x-z plane to show the horizontal overlapped regions.FIG. 10F illustrates the overlapping fields of view as viewed in a plane that is nearly parallel to the x-y plane to illustrate the horizontal and vertical overlapped regions. The exampledense field imager1000 provides a region of relativelyhigh coverage1060 where a majority offacets1030 have overlapping fields of view, as can most easily be seen inFIGS. 10D and 10E.
• FIGS. 11A to 11C illustrate an example embodiment of adense field imager1100 havingfacets1130 that are mounted in a convex fashion. Thedense field imager1100 can include acommon support1105 that lies within an x-y plane (e.g., the plane of the paper inFIG. 11A). The plurality offacets1130 can be attached or secured to thecommon support1105 through mountingelements1140.
• Following the coordinate system convention described with reference toFIGS. 10A to 10C, the mountingelements1140 can be configured to orient one ormore facets1130 such that the facetoptical axes1132 are not parallel to the primaryoptical axis1102. In the example embodiment illustrated inFIGS. 11A to 11C, the mountingelements1140 secure thefacets1130 in such a way that the facetoptical axes1132 generally diverge from the primaryoptical axis1102 in both the x- and y-directions. Other configurations are possible as well. For example, the mountingelements1140 can be configured to orient one ormore facets1130 such that the facetoptical axes1132 remain parallel to the primaryoptical axis1102 in either the x- or y-direction while diverging in the other direction (e.g., parallel in the x-direction and diverging in the y-direction). The mountingelements1140 can be configured to orient one ormore facets1130 such that the facetoptical axes1132 converge towards the primaryoptical axis1102 in either the x- or y-directions while diverging in the other direction (e.g., converging in the x-direction and diverging in the y-direction). The mountingelements1140 can be configured to orient one ormore facets1130 such that some of the facetoptical axes1132 remain parallel to the primary optical in a first direction while diverging in a second, orthogonal direction, some facetoptical axes1132 remain parallel in the second direction and diverge in the first direction, some facetoptical axes1132 converge in either the first or second direction and diverge in the other direction, or any combination of these.
• In some embodiments, the mountingelements1140 generally orient the plurality offacets1030 along an exterior surface of a sphere, an ellipsoid, a cylinder, a cylindrical ellipsoid, a paraboloid, a hyperboloid, a polyhedron, or some other regular or irregular surface. In some embodiments, the mountingelements1140 can be configured to be movable through mechanical adjustments and/or through actuators configured to change an orientation of one or more facet planes. The mountingelements1140 can be manually and/or automatically adjustable. Thefacets1130 can be mounted to thecommon support1105 to provide a desired or targeted combined field of view. The combined field of view can be configured to provide coverage for a relatively wide area to provide image information over the area using a single imaging device (e.g., the dense field imager1100) rather than a multitude of individual imaging devices.
• FIGS. 11D to 11F illustrate fields of view of some of thefacets1130 on thedense field imager1100 illustrated inFIGS. 11A to 11C.FIG. 11D illustrates the substantially adjacent fields of view of thefacets1130 as viewed in the y-z plane to show the vertical overlapped regions.FIG. 11E illustrates the substantially adjacent fields of view of thefacets1130 as viewed in the x-z plane to show the horizontal overlapped regions.FIG. 11F illustrates the fields of view of thefacets1130 as viewed in a plane that is nearly parallel to the x-y plane to illustrate a size of the combined field of view of thedense field imager1100. The exampledense field imager1100 can be configured to provide dense field data over a relatively wide field of view.
• Example Dense Field Imager with Display
• FIGS. 12A and 12B illustrate perspective views of an exampledense field imager1200 including a plurality ofimaging elements1220 arranged in an array. Theimaging elements1220 can include a sensor, electronics, and/or an optical system. Theimaging elements1220 are contained within abody1202. As such, thedense field imager1200 can be a substantially integrated, portable unit. Afirst face1204 of thebody1202 includes a transparent or substantiallytransparent surface1206, while asecond face1208 includes aviewfinder screen1210.
• Theimaging elements1220 are situated within thebody1202 such that their respective optical systems gather and focus light that is incident on and passing through thesurface1206. Thesurface1206 can be made of glass or plastic, for example. In one embodiment, the surface is partially opaque, and includes transparent windows that correspond to each of theindividual imaging elements1220. In some cases, thesurface1206 is treated with one or more anti-reflective coatings in order to reduce glare or lens flare. Thesurface1206 in some cases is a lens, and is shaped to refract light in a desired fashion. In some instances, one or more additional lenses or other optical elements are situated beneath thesurface1206, between the optical systems of the individual imaging devices and theface1204.
• Depending on the embodiment,individual imaging elements1220 in the array can be adjacent to one another, or can be spaced by an appropriate amount. Moreover, physical barriers can be included between theindividual imaging elements1220 to reduce cross-talk or other noise between adjacent imaging elements.
• Thedisplay1210 can include any type of monitoring device. For example, but without limitation, thedisplay1210 can include an LCD panel. In some embodiments, instead of or in addition to theintegral display1210, a display is connected to an infinitely adjustable mount configured to allow thedisplay1210 to be adjusted to any position relative to thebody1202 of thedense field imager1200 so that a user can view thedisplay1210 at any angle relative to thebody1202. In some embodiments, a separate display can be connected to thebody1202 through any type of appropriate cabling or other connection, or wirelessly.
• Thedisplay1210 may also include a “look-around” feature, in which thedisplay1210 shows images for playback that have a larger scene area than the those that are being recorded for playback and/or editing. For instance, thedense field imager1200 may be capable of capturing image data corresponding to a first image scene area but, due to the desired aspect ratio or other formatting parameters, the system only includes image data corresponding to a certain sub-set of the captured image scene area in the recorded video file. In such cases, the system may discard a certain amount of edge or boundary content for recording purposes. However, thedisplay1210 may depict some or all of the edge or boundary content in addition to the recorded content. Moreover, thedisplay1210 may include a rectangular box or other indication of what portions of the displayed video are actually being recorded. One advantage of this feature is that camera operators can anticipate when objects are about to enter the recorded scene.
• Thedense field imager1200 includes acontrol interface1212. Thecontrol interface1212 can include any of a variety of standard user interface features. For instance, thedisplay1210 may form part of thecontrol interface1212 and can be a touch screen, with integrated controls in the touch screen.Separate controls1214 such as one or more knobs, buttons, keypads and the like may also be used. In one embodiment, thedense field imager1200 includes a separate keyboard or other interface hingeably attached to an edge of thebody1202. Thecontrols1214 can provide a variety of functions including, for example, toggling thedense field imager1200 between motion and still modes, entering a record mode, operating one or more of the displays or other components of thedense field imager1200, and the like.
• As shown, one ormore handles1216 can be attached to thebody1202. While the illustrated embodiment includes twohandles1216 attached to the left and right sides of thebody1202, one ormore handles1216 can also be attached to the top and/or bottom sides of thebody1202. Thehandles1216 may be releasably attachable to the camera body in some cases. For instance, thehandles1216 can be attached for hand-held use, but removed for studio or other non-hand-held use.
• One or more of thehandles1216 can include an interface (not shown) including features for mechanically and/or electrically coupling the handle to a corresponding interface (not shown) on thebody1202. Thehandles1216 may be releasably attachable to thebody1202 via a variety of mechanisms including friction-fit, snap-fit, threaded components, and the like.
• In some embodiments, one or more of thehandles1216 further includes a variety of controls for operating thedense field imager1200. The controls may include exposure controls, focus controls, and the like, which may be user-definable and suitable for use in still and/or video applications. Thehandles1216 may be particularly suited for hand-held and light-weight tripod use. In certain embodiments, one or more of thehandles1216 includes a rechargeable battery, allowing for lightweight and low-profile remote use without a separate power source. For instance, the battery may be releasably insertable into a corresponding receptacle in thehandle1216. Alternatively, one or more batteries can be integrated into ahandle1216. When the battery reaches discharge, thehandle1216 can be removed from thedense field imager1200 and replaced with a second handle, containing a second, fully charged battery.
• While threehandles1216 are shown inFIGS. 12A and 12B, other numbers and types of handles can be used. In one configuration, a fourth handle is positioned on the top side of thebody1202, providing convenient handling regardless of the orientation of thedense field imager1200.
• Thedense field imager1200 can further include one ormore ports1218 providing output and/or input connectivity to and from external devices. A wide variety of types of ports can be used, including, without limitation, Ethernet, USB, USB2, USB3, IEEE 1394 (including but not limited toFireWire 400,FireWire 800, FireWire S3200, FireWire S800T, i.LINK, DV), SATA, SCSI, monitoring ports capable of outputting high resolution image data (e.g., 1080p, 2k, or 4k image data), such as HD-SDI, HDMI, etc.
• The dense field imager can further include one ormore mounting points1219 which serve as mounting points for a variety of components. For, instance, thedense field imager1200 can be compatible with various rails, rods, shoulder mounts, tripod mounts, helicopter mounts, matte boxes, follow focus controls, zoom controls, and other features and other accessories known in the art. And one or more of devices providing these functions can be mountable on one or more brackets or other mounting points on thedense field imager1200.
• In some cases, a remote unit (an example of which is shown inFIG. 13) includes a control interface and is in wireless communication with a wireless interface of thedense field imager1200, providing operators with the ability to control camera operation remotely. For example, the remote and the camera may include wireless transceivers. In yet other configurations, the wireless interface is configured to communicate data wirelessly. For instance, image data can be communicated to an external device for remote viewing purposes. This can be particularly useful in situations where theviewfinder1210 is not visible or accessible to the camera operator, such as where thedense field imager1200 is mounted to a vehicle.
• As shown, thedense field imager1200 has a depth d, width w, and height h, and the form factor is generally configured for straightforward handheld and other portable operation. The multi-imaging element configuration allows for a particularly low-profile design, while still providing the high performance video and/or still capture described herein. For instance, the depth d can be less than about 0.5 inches, less than about 1 inch, less than about 1.5 inches, less than about 2 inches, or less than about 3 inches, depending on the configuration.
• While theimaging elements1220 are arranged in a 3×6 (vertical×horizontal) array in the illustrated embodiment, the arrangement and number ofimaging elements1220 can vary depending on the embodiment. In another embodiment, theimaging elements1220 are arranged in a 9×18 (vertical×horizontal) array.
• Theimaging elements1220 of the illustrated embodiment can be grouped into facets where each facet hasimaging elements1220 arranged in a 2×3 (vertical×horizontal) array, a 3×3 array, a 2×1 array, or a 1×3 array, or any combination of these. In another embodiment, the number ofimaging elements1220 can differ and the facets can be configured accordingly.
• The resolutions of the individual sensors can vary. However, thedense field imager1200 can be configured to produce an output that is not dependent on the input resolution. For example, thedense field imager1200 can include a plurality of facets, each facet having a plurality ofimaging elements1220. Each facet can be configured to produce light field representations, and thedense field imager1200 can be configured to combine the light field representations from the plurality of facets to produce a dense field representation. From the dense field representation, viewpoints can be generated at a variety of output resolutions which are independent of the input resolution. The above was described in relationship to spatial resolution, but it also applies for temporal resolution. Accordingly, thedense field imager1200 can produce an output that can be used to generate viewpoints (e.g., images and/or video) that whose output resolution (e.g., spatial and/or temporal resolution) that is not dependent on the input resolution.
• The width w of thebody1202 may also be optimized. For instance, the width w of the box may be no more than about 20 inches, no more than about 18 inches, no more than about 12 inches, or no more than about 14 inches. For instance, in certain embodiments, the camera system provides dense field data and has a width w of no more than about 20 inches, no more than about 18 inches, no more than about 12 inches, or no more than about 14 inches, depending on the desired lens size and sensor spacing. In another embodiment, thedense field imager1200 provides dense field data and has a width w of no more than about 10 inches, no more than about 8 inches, no more than about 6 inches, or no more than about 4 inches, depending on the desired lens size and sensor spacing.
• Moreover, thedense field imager1200 also provides for a particularly large total imaging surface area in a low-profile package. For instance, the total span of the sensing surface, which can correspond to the sums of longest edges of the individual sensors, can be relatively large in comparison to the depth d of thebody1202 along the optical axis. As an example, where a 9×18 array (vertical by horizontal) of 5 mm×5 mm sensors are used, and the depth d is 25.4 mm (1 inch), the ratio of the total span of the sensing area, along the array horizontally, to the depth d, is about 3.54 (90 mm/25.4 mm). As another example, where a 20×40 (vertical by horizontal) array of 5 mm×5 mm sensors are used, and the depth d is 25.4 mm (1 inch), the ratio of the total span of the sensing area, along the array horizontally, to the depth d is about 7.87 (200 mm/25.4 mm). As yet another example, where a 3×6 array (vertical by horizontal) of 5 mm×5 mm sensors are used, and the depth d is 25.4 mm (1 inch), the ratio of the total span of the sensing area, along the array horizontally, to the depth d, is about 1.8 (30 mm/25.4 mm). These are just a few illustrative examples. Depending on the embodiment, the ratio of the sum of the sensing surface along a particular direction in the sensing plane (e.g., horizontally or vertically across the array of imaging elements1220), to the depth d of thebody1220, is at least about 0.25, 0.3, 0.5, 0.75, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, or 50.
• The total span of the lens surface can also be relatively large in comparison to the depth d of thedense field imager1200. For instance, the total span of the lens area can correspond to the sums of the widths (e.g., diameters for circular lenses) of the individual lenses at their widest points. As an example, where a 9×18 array of lenses having a diameter of 15 mm is used, and the depth d is 25.4 mm (1 inch), the ratio of the span of the lens surface along the horizontal direction across the array is 10.62 (270/25.4 mm). As another example, where a 20×30 array of lenses having a diameter of 15 mm is used, and the depth d is 25.4 mm (1 inch), the ratio of the span of the lens surface along the horizontal direction in the array is 17.7 (450/25.4 mm). These are just a couple of examples. Depending on the embodiment, the ratio of the sum of the total lens surface along a particular direction in the sensing plane (e.g., horizontally or vertically across the array of imaging devices), is at least about 1, 5, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 25, 30, 40, 50, 60, 70, 80, 90, or 100.
• In some embodiments, the height h can be at least about 6 in. and/or less than or equal to about 36 in., at least about 12 in. and/or less than or equal to about 24 in., or at least about 15 in. and/or less than or equal to about 20 in. In some embodiments, the width w can be at least about 4 in. and/or less than or equal to about 36 in., at least about 8 in. and/or less than or equal to about 24 in., or at least about 12 in. and/or less than or equal to about 20 in. In some embodiments, the depth d can be at least about 0.5 in. and/or less than or equal to about 8 in., at least about 1 in. and/or less than or equal to about 5 in., or at least about 2 in. and/or less than or equal to about 4 in.
• In some embodiments, the dense field camera can be configured to have a greater effective output image resolution with a smaller depth of the camera housing relative to a standard camera employing a single imaging element. In a standard camera to increase the effective output resolution, typically the resolution of the image sensor is increased, thereby increasing the physical size of the sensor. In addition to increasing the size of the sensor, the lens generally increases in length, increasing the depth of the imaging system. The dense field cameras disclosed herein can be configured to acquire image data using a plurality of imaging elements and join, combine, and/or stitch the image data together to generate an output image with a resolution greater than any individual image sensor. To increase the effective resolution of the output image, additional imaging elements can be added. Because the size of the sensors for each of the additional imaging elements does not increase, or because the sizes of the sensors can remain substantially the same when adding additional imaging elements, the size of the lenses for each imaging element can remain substantially the same. Advantageously, this allows the dense field imager to increase output resolution by adding additional imaging elements without increasing a depth d of the imager. Similarly, this allows the dense field imager to provide a similar effective output resolution relative to a typical imaging system with a single image sensor and lens pair with a smaller depth d, providing for a more compact package.
• According to some embodiments, thedense field imager1200 is modularly expandable to increase the number of facets andimaging elements1220. For instance, thedense field imager1200 can be extended via attachment of a dense field imaging module that includes one or more facets, each facet having a plurality of imaging elements. In some other embodiments, thedense field imager1200 may include a slot, bay or other receptacle for insertion of one or more additional dense field imaging modules. For instance, thebody1202 may include a slot on one of its sides for receiving additional dense field imaging modules. The dense field imaging modules may have the same number of facets, imaging elements, and/or a similar form factor as thedense field imager1200, or can have different numbers of facets, imaging elements, and/or form factors. In this manner, thedense field imager1200 can be scalable to achieve a particular field of view or other outcome. One or more mounting brackets, snap-fit or friction-fit mechanisms, or other appropriate components can be used to fasten the dense field imaging modules together.
• FIG. 12C shows a side perspective view of an embodiment of thedense field imager1200 having a plurality ofimaging elements1220 with large, removable lenses. Theimaging elements1220 can be configured to have removable lenses such that they can be exchanged and/or replaced. In addition, different types of lenses can be used ondifferent imaging elements1220. The lenses can also be changed or modified to include filters to provide desired optical properties. In embodiments such as those illustrated inFIGS. 12C to 12H, thetransparent surface1206 present in the dense field imager illustrated inFIGS. 12A and 12B can be excluded and/or removed. The relatively large lenses can provide an increased light sensitivity, increased zoom ratio, increased aperture, and the like when compared to smaller lenses. In addition, using a larger lens and/orimaging element1220 in general can increase a size of the image sensor associated with theimaging element1220, which can result in an increase in resolution for an individual imaging element (e.g., more pixels), and/or an increase in light sensitivity (e.g., larger pixels).
• FIG. 12D shows a side perspective view of an embodiment of thedense field imager1200 having a plurality ofimaging elements1220 with small, removable lenses. Similar to the embodiment illustrated inFIG. 12C, the small, removable lenses can allow for a variety of lens types to be included in thedense field imager1200. The relatively small lenses can provide for sharper images, larger depths of field, and the like. In addition, using a smaller lens and/orimaging element1220 in general can increase the number of imaging elements that can be used in a given space. Moreover, using smaller lenses and/orimaging elements1220 can correspond to using smaller imaging sensors which can produce images with a higher perceived sharpness.
• FIG. 12E shows a side perspective view of an embodiment of adense field imager1200 having a plurality ofimaging elements1220 with non-removable, low-profile lenses. The non-removable, low-profile lenses can advantageously reduce a size of thedense field imager1200, making it easier to handle and operate. In addition, by making the lenses non-removable, thedense field imager1200 can be more resistant to mechanical failures related to the addition and removal of lenses.
• FIGS. 12F-H show top perspective views of thedense field imager1200 with agrip attachment1216, thedense field imager1200 having a plurality ofimaging elements1220 with, respectively, large, removable lenses; small, removable lenses; and non-removable, low-profile lenses. Thegrip attachment1216 can be configured to be removable to allow thedense field imager1200 to be used in a variety of contexts. Thegrip attachment1216 can allow for a user to easily operate and handle thedense field imager1200, whereas removing thegrip attachment1216 may be desirable where the dense field imager will be mounted to a mounting device or where dense field imaging modules are to be attached. In some embodiments, thegrip attachment1216 includes user interface elements that allow the user to operate and configure properties of the dense field imager, such as a zoom, an orientation ofimaging elements1220 and/or facets, an acquisition mode (e.g., switching between image and video mode), an exposure time, and the like.
• FIGS. 12I and 12J show top perspective views of thedense field imager1200, respectively with and without agrip attachment1216, showing theviewfinder screen1210. As described above with reference toFIG. 12B, thedisplay1210 can be used to provide visual feedback to a user of the scene being captured. Because thedense field imager1200 can be used to acquire light field data and output dense field data, thedisplay1210 can be configured to receive image information from asingle imaging element1220. In some embodiments, theimaging element1220 providing information to thedisplay1220 can be automatically selected or it can be selected by a user. In some embodiments, theimaging element1220 providing image information to the display can be switched according to criteria and/or user selection. In some embodiments, a facet can provide image information to thedisplay1210 and the display can allow a user to interact with light field data provided by the facet. For example, the user can re-focus a scene, zoom in on a scene, change a depth of field of a scene, adjust a viewpoint, and the like based at least in part on the light field data from a facet. Similar to theimaging element1220, the facet can providing data to the display can be selected and/or switched automatically or manually. In some embodiments, the user can use thedisplay1210 to interact with dense field data provided by thedense field imager1200. In some embodiments, thedisplay1210 can be used to acquire dense field data, to manipulate dense field data, and/or to configure thedense field imager1200. Thedisplay1210 can be configured to display status information to the user, such as storage information, wireless connectivity information, radiance or luminance information, distances to objects, and the like.
• Thedense field imager1200 can include modular components and modular digital camera systems. The modular nature of thedense field imager1200 as well as the modular nature of any of its components can include elements, functionality, properties, and/or characteristics disclosed in U.S. Pat. No. 8,477,238, entitled “Modular Digital Camera,” issued Jul. 2, 2013, which is incorporated by reference herein in its entirety.
• FIG. 13 illustrates an embodiment of adense field imager1300 that is configured to wirelessly transmit acquired data to animage hub1360. Thedense field imager1300 can be any of the dense field imager embodiments described herein, such as those described with reference toFIGS. 1, 5A, 5B, 8, 9A-C,10A-F,11A-F, and/or12A-J. Theimage hub1360 can be any hardware system adapted to communicate wirelessly with thedense field imager1300 and to store the data received from thedense field imager1300.
• In some embodiments, theimage hub1360 is portable. For example, theimage hub1360 can be configured to be worn by a user (e.g., a camera operator), attached to the clothing of a user, carried inside a bag by a user, moved with thedense field imager1300, or the like. In some embodiments, theimage hub1360 is configured to have a substantially fixed location and can be located remote from thedense field imager1300. For example, theimage hub1360 can be an image processing system that is located at a central image processing location that is within a wireless communication distance from the dense field imager. The wireless communication distance can vary depending on the wireless protocol to be used. For example, the wireless communication distance can be less than or equal to about 500 feet, less than or equal to about 100 feet, less than or equal to about 20 feet, less than or equal to about 10 feet, less than or equal to about 5 feet, less than or equal to about 2 feet, less than or equal to about 1 foot, or less than or equal to about 6 inches.
• In some embodiments, theimage hub1360 is configured to communicate with thedense field imager1300 using a wireless protocol capable of transmitting data at a desired data transmission rate. For example, the wireless communication can be accomplished using 802.11 standards (e.g., WiFi and/or WiGig), Bluetooth®, CDMA, GSM, NFC, or the like. In some embodiments, the wireless data transmission rates can be greater than or equal to about 20 Gbps, at least about 500 Mbps and/or less than or equal to about 10 Gbps, at least about 1 Gbps and/or less than or equal to about 5 Gbps, or less than or equal to about 500 Mbps.
• In some embodiments, both thedense field imager1300 and theimage hub1360 include a plurality of transceivers and antennas configured to communicate data in a parallel fashion over a plurality of data communication channels. In this way, the data transmission rates can be increased to accommodate more data such as when the data acquisition rate of thedense field imager1300 increases. For example, the data acquisition rate of thedense field imager1300 can increase where an acquisition mode switches from a still image acquisition mode to a video acquisition mode or where the dense field imaging modules are attached to thedense field imager1300 thereby increasing the amount of data acquired.
• Theimage hub1360 can be configured to have a storage medium for recording the data received from thedense field imager1300. The storage medium can be any suitable computer storage such as hard disk drives, solid state drives, or the like. Theimage hub1360 can be configured to perform image processing as well as data storage. Theimage hub1360 can include any suitable processing systems such as computer processors, FPGAs, DSPs, ASICs, and the like configured to process data received from thedense field imager1300. For example, theimage hub1360 can be similar to the multi-image processing system110 described herein with reference toFIG. 1.
Multi-Imaging Device Systems
• FIG. 14A illustrates a top plan view andFIG. 14B illustrates a side elevational view of a multi-imaging device system1400. The multi-imaging device system1400 can include two or three ormore cameras1402 that are physically separate from one another, and, in some embodiments, one or more of thecameras1402 can be a stand-alone imaging device. For example, thecameras1402 can include dense field imagers, video cameras, cell phone cameras, DSLR cameras, webcams, or any imaging device configured to capture digital image data. Thecameras1402 can be different from one another and can have different imaging properties, such as, for example and without limitation, resolution, acquisition rate, aperture, focus, sensitivity, color gamut, and the like. Thecameras1402 can be positioned at spaced apart locations and oriented so that a primaryoptical axis1403 of each camera is generally directed toward an object of interest or a common viewing theater. Data from the multi-imaging device system1400 can be transmitted to a processing station where image data is combined. The combined image data can be light field data, dense field data, video data, image data, pixel data, mathematical representations of image data (e.g., representations using fractals or vectors), or any combination of these. The combined image data can be used to generate two dimensional or three dimensional images, in still images or motion. An example of such a processing station is described herein with reference toFIG. 1. In some embodiments, simultaneously captured video from one or more of thecameras1402 in the system1400 is combined to create a spatial and/or temporal resolution independent output. For example, each of thecameras1402 can be configured to acquire video data at an acquisition resolution at an acquisition frame rate where the acquisition resolution and/or frame rate can be different for eachcamera1402. The acquired video data can be combined to generate dense field data that can be used to generate high-resolution (e.g., 2k, 4k, or higher), high-frame rate (e.g., 30 fps, 60 fps, 120 fps, 240 fps, or higher) video output where the resolution and/or frame rate of the video output exceeds the acquisition resolution and/or frame rate of anyindividual camera1402. For some applications, non-simultaneous data is combined to create at least a portion of the output video stream.
• As discussed, theoptical axes1403 of thecameras1402 can be directed towards the common viewing theater. For instance, in certain embodiments, the field of view1404 of each of thecameras1402 at least partially overlaps with the field of view1404 of each of theother cameras1402. In some configurations, theoptical axes1403 of each of thecameras1402 intersect at the same point, or substantially the same point, in the viewing theater.
• In some embodiments, thecameras1402 transmit wireless data to one or more receivers, which route data to a processing station.Cameras1402 can be positioned to surround the viewing theater, and may be attached to support poles, buildings, trees, or other suitable existing or supplied support structure. In certain implementations, at least three, and in some applications at least 10, or at least 50 ormore cameras1402 are provided in the multi-imaging device system1400 for imaging a viewing theater.
• Due to the disparate arrangement of thecameras1402 about the viewing theater, thecameras1402 can image the viewing theater from generally any vantage point. In some cases, each of thecameras1402 is spaced from each of theother cameras1402 by at least 1, 10, 100 or 1000 feet. According to some arrangements, the viewing theater can be delineated into halves (e.g., with respect to a top plan perspective and/or side elevational perspective), and at least onecamera1402 in the array is positioned in a different half of the viewing theater than at least oneother camera1402 in the array. Or, the viewing theater may delineated into four equally sized segments (e.g., from a top plan perspective and/or side elevational perspective), and at least onecamera1402 in the array is positioned in each of the four segments. In yet further cases, at least onefirst camera1402ais at least partially facing at least oneother camera1402c, such that theother camera1402cat least partly resides in the field ofview1404aof thefirst camera1402a. Depending on the orientation of thecameras1402, thefirst camera1402acan also at least partly reside in the field ofview1404cof theother camera1402c.
• Thecameras1402 can be positioned about the perimeter of the viewing theater or, in some cases, one or more of thecameras1402 are also positioned within the viewing theater. Moreover, thecameras1402 can be arranged in some cases in an irregular manner about the viewing theater. In such cases, the system1400 can use a depth map to adjust for positional variations between thecameras1402 and the viewing theater or portions thereof. This capability gives camera operators flexibility in placing thecameras1402 with respect to the viewing theater. This can be particularly beneficial in environments where options for camera placement are constrained, such as where it is desirable to hide the cameras. In other cases, thecameras1402 or a subset of thereof are arranged in a regular manner with respect to one another, e.g., within the same plane, on a common hemispherical profile, symmetrically, etc.
• As shown inFIG. 14B, thecameras2 can also be positioned in different vertical positions with respect to the viewing theater. For instance, thecameras1402 may be divided into a first plurality, within a first vertical zone1405 (e.g., within about 5 or 10 vertical feet) of the local ground level. A second plurality ofcameras1402 may be positioned within a second zone1408 (e.g., from about 5 or 10 feet of the ground and about 20 or 30 feet of the ground) vertically above the firstvertical zone1405, and a third plurality of cameras may be positioned in athird zone1409, vertically above thesecond zone1408. The vertical separation of cameras may or may not be desired depending upon the nature of the activity under surveillance in the viewing theater.
• The orientation of one or more of thecameras1402 is adjustable according to further aspects. For instance, thecameras1402 can be outfitted or mounted on a motor-driven tripod or other apparatus that allows for user controlled or automated tilt, pan and/or zoom control. In one implementation, such as where thecameras1402 are in continuous or substantially continuous communication with the processing station, the processing station can maintain up-to-date orientation information for each of thecameras1402. In some embodiments, the processing station pings thecameras1402 periodically for status info, or thecameras1402 periodically broadcast current orientation. According to one method,cameras1402 report orientation information upon “wake-up” from a power-down or reduced power mode.
• Allcameras1402 in the multi-imaging device1400 may be stationary. Alternatively, a first plurality ofcameras1402 may be stationary and at least one or a second plurality ofcameras1402 may be moving or attached to a structure which is capable of moving (e.g., a vehicle, robot, personnel, drone or other aircraft, satellite). Allcameras1402 may be carried by a person or support which is moving or capable of moving.
• Data from thecameras1402 can be combined to generate dense field data, light field data, combined image data, or any combination of these or other image representations, as described herein. In some embodiments, combining the data from thecameras1402 can include determining correspondences between two or more images of approximately the same optical target. Determining the correspondences can include registering the images and can be intensity-based (e.g., comparing intensity patterns in a first image to intensity patterns in a second image) and/or feature-based (e.g., finding features in one image and seeing if the layout of a subset of features is similar to that in a second image). One or more reference points may be identified or introduced into the theater to facilitate correspondence determination. For example, one or more transmitters can be positioned within the theater, for transmitting a signal that can be recorded by the camera and recognized as a correspondence reference. The transmitted signal can be optical, RF and, in some environments, acoustical. Active continuous or intermittent transmitters may be desirable in some applications. Alternatively, passive references (e.g., RFID or other system which responds to interrogation from the camera) may be used. Optical signals may include a continuous or intermittent optical transmitter positioned within the theater (UV, visible or infra red) or an optical signal (e.g., laser) transmitted from one or more cameras into the theater.
• In general, where image data frommultiple cameras1402 is combined to create motion video or still images, any of the techniques described herein (or other techniques) for stitching together or otherwise combining the image data can be used. While in some cases, image data from all of thecameras1402 is combined together, data from a selected subset of thecameras1402 in the array can also be used. And, the current selected subset ofcameras1402 can adjust dynamically depending on the desired effect. For instance, image data from a first subset ofcameras1402 may be combined to create 2D or 3D image data from a first perspective, for a first period of time, and then, for a second period of time, the system combines image data from a second subset ofcameras1402, achieving a different viewing theater perspective, and so on. As one example, referring toFIG. 14A, the system1400 may combine image data from thecameras1402a,1402bto create video image data from a first perspective and then transition to combining image data from thecameras1402b,1402c. The next transition could be to any combination of cameras (e.g., back to thecameras1402a,1402b, to thecameras1402c,1402d, to thecameras1402a,1402b,1402c, to all the cameras, etc.). Transitioning to a subset ofcameras1402 having at least one camera in common with the current subset is preferable in some cases, as it can provide continuity of perspective. However, in some cases, the next subset does not include anycameras1402 in common with the current subset.
• The mechanism for adjusting the current subset of cameras can vary. For instance, a user interface (e.g., touch screen, joystick, voice command, etc.) can provide operators with real-time control of the desired viewing perspective. In other cases, the system automatically tracks certain image scene objects. As one example, where a person or other object of interest is moving through the image scene, the system1400 can dynamically adjust the subset of cameras (or otherwise process the combined image data) to provide an obstruction-free view of the person or object.
• Moreover, in certain cases, to conserve bandwidth or power, such as in live playback applications, image data for only the selected subset ofcameras1402 is transmitted for processing.Unselected cameras1402, in some cases, enter a power-saving mode in which they consume less power than the selected subset ofcameras1402.
• According to certain aspects, the multi-imaging device system1400 can also provide multi-vantage point audio capture capability for the viewing theater. For instance, eachcamera1402 may be provided with or be coupled to a microphone or other audio input device. In such cases, captured audio information associated with eachcamera1402 may be transmitted for processing, along with the image data, for example. Instead of or in addition to providing microphones that are integrated with thecameras1402, separate microphones can be positioned throughout the viewing theater, or otherwise be situated to record sound information from the viewing theater.
• Similar to the image data, audio information from thecameras1402 can be selectively combined. For instance, the system may automatically utilize image data captured from one or more cameras in a currently selected subset. In other cases, a user may be able to select the audio source(s). In some cases, a user may select audio data from asingle camera1402 or other audio source closest to a particular object or event of interest, while the selected image data is from a combination ofmultiple cameras1402. Depending on the circumstances, aparticular camera1402 may be used as an audio source even though it is not currently being utilized in the recorded video file (or for playback).
• The multi-camera images from the viewing theater can also be combined to adjust the dynamic range of the output video stream (e.g., by tone mapping or other appropriate mechanisms), or to otherwise control exposure. For instance, lighting conditions may be different for different cameras in array. In one case, highlight detail from one or more cameras1402 (e.g., with a higher exposure setting or positioned in relatively brighter light conditions) is combined with shadow details from one or more other cameras1402 (e.g., with a lower exposure setting or positioned in relatively low light conditions). In some cases, lighting conditions forunselected cameras1402 may be more favorable than lighting conditions forcameras1402 in the currently selected bank. In these and other circumstances, exposure information from one ormore cameras1402 that are not in the currently selected bank can be used to adjust exposure levels in the output stream. Additional compatible high dynamic range techniques are discussed below and are described in U.S. Patent Application Publication No. 2012/0044381 entitled “HIGH DYNAMIC RANGE VIDEO”, which is incorporated by reference herein in its entirety.
• One ormore cameras1402 in the multi-imaging device system1400 may be cell phone cameras, smart phone cameras, cameras on tablets, hand-held cameras, or the like. The system1400 can be configured to receive image data acquired using one or more of these disparate cameras and combine the information to produce video and/or still images from virtual viewpoints or having other characteristics, as described herein. Thus, the multi-imaging device system1400 can be a dynamic system, with a changing number ofcameras1402 that make up the system1400. In such embodiments, the image processing system can be part of a website or other publicly- or privately accessible resource that receives image data from users to generate output data. The input data can be used to generate video and/or still images that incorporate information from one or more of thecameras1402 to produce high-quality viewpoints of an object of interest. For example, multiple people may capture image data of a newsworthy event, yet the images may be partially occluded, acquired from a disadvantageous viewpoint, shaky, low-resolution, blurry, or the like. By combining the acquired image data from multiple viewpoints usingmultiple cameras1402, images and/or video may be generated that provide an advantageous, high-quality view of the event of interest. In some embodiments, the acquired image data can be combined and the combined data can be made available for others to use to generate output images and/or video. Software tools can be provided which allow users to generate images and video using the output data which can be light field data and/or dense field data, for example.
• One ormore cameras1402 in the multi-imaging device system1400 may be carried by one or more geosynchronous satellites, which have the advantage of remaining permanently in the same area of the sky, as viewed from a particular location on Earth. This allows permanent surveillance of a desired theater. Thus the multi-imaging device system1400 can include a geosynchronous component or a geosynchronous network as a portion of the system1400.
• At present, the GeoEye-1 satellite may be the highest resolution commercial imaging system and is able to collect images with a ground resolution of 0.41 meters (16 inches) in the panchromatic or black and white mode. Spatial resolution is defined as the pixel size of an image representing the size of the surface area (i.e. m2) being measured on the ground, determined by the sensors' instantaneous field of view (IFOV). While the satellite is able to collect imagery at 0.41 meters, GeoEye's operating license from the U.S. Government requires re-sampling the imagery to 0.5 meters for all customers not explicitly granted a waiver by the U.S. Government.
• The ground based component of the present multi-imaging device system1400 can add significant depth of data to current Geospatial Intelligence systems. Frequently referred to as GEOINT, Geospatial Intelligence is an intelligence discipline comprising the exploitation and analysis of geospatial data and information to describe, assess, and visually depict physical features (both natural and constructed) and geographically referenced activities on the Earth. Geospatial Intelligence data sources include imagery and mapping data, whether collected by commercial satellite, government satellite, aircraft (such as Unmanned Aerial Vehicles [UAV] or reconnaissance aircraft), or by other means, such as maps and commercial databases, census information, GPS waypoints, utility schematics, or any discrete data that have locations on earth. Integration of data from the optical network described herein can help GEOINT evolve from traditional compilations of geospatial information and imagery, towards an emphasis on knowledge.
• Depending upon the needs of a particular theater,cameras1402 can be hard-wired or in wireless communication with the data processing station. A variety of wireless technologies can be used, as described herein. In some embodiments, thecameras1402 implement a wireless link capable of transmitting relatively high resolution image data, such as a dual 2.970 Gigabit HD-SDI wireless link that operates in the 60 GHz frequency band. Data processing can occur locally, or towns or continents away.Cameras1402 can be powered by ground power, battery power, solar power or otherwise depending upon a particular assignment.
• As discussed, image data from thecameras1402 can be combined to provide improved video or still images of select image scene regions.FIG. 14C illustrates a top plan view of an array ofcameras1402 recording anobject1406 or other portion of interest in a viewing theater. As shown, one or more obstructingobjects1407 occlude the field of view1404 of thecameras1402. The processing station can stitch together or otherwise combine the image data from a plurality (e.g., 2, 3, 4, 5 or all) of thecameras1402 in the array to reduce or remove the occlusions in the combined video output. While the field of view1404 of eachcamera1402 shown inFIG. 14C is at least partially occluded, in some cases, some of thecameras1402 have un-obstructed fields of view1404.
Wide-Angle Persistent Surveillance
• The dense field imagers described herein can be used in a variety of contexts. For example, a dense field imager can be used to provide overhead surveillance. The dense field imaging systems described herein can be mounted on an aerial vehicle to perform wide-angle persistent surveillance. The dense field imager can include a plurality offacets300, each having a plurality ofimaging elements200 with image sensors, each image sensor having a resolution measured in pixels. The dense field imaging system can be configured to generate a plurality of light field representations from each of thefacets300, where eachfacet300 covers a field of view and the dense field imaging system covers an aggregate field of view. The plurality of light field representations can be combined or aggregated to generate an output image of at least a portion of the aggregate field of view wherein the output resolution of the output image is greater than any resolution of the image sensors of the plurality of imaging blocks. This can be accomplished by combining, fusing, and/or stitching the pixel data provided by the plurality ofimaging elements200. In some embodiments, a viewpoint (e.g., image) can be generated from each of the plurality of light field representations and the resulting viewpoints can be combined, fused, and/or stitched to generate a final output image. In some embodiments, the several viewpoints can be configured to provide a depth of field, focus, dynamic range, or the like that, when combined, provide a higher quality output image than would be provided by a combination of the raw pixel data from theimaging elements200.
• In some embodiments, the resolution of the output image is at least 3× greater than the resolution of any individual image sensor, at least 5× greater than the resolution of any individual image sensor, at least 10× greater than the resolution of any individual image sensor, at least 20× greater than the resolution of any individual image sensor, or at least 50× greater than the resolution of any individual image sensor. For example, where the dense field imager comprises 4 facets where each facet has 8 imaging elements with 5 MP sensors, the output image generated by the dense field imager can be at least 15 MP (e.g., 3× greater than the resolution of any individual sensor), 25 MP, 50 MP, 100 MP, or 250 MP. This can be accomplished by stitching and/or combining pixel data acquired by theimaging elements200 to effectively increase the output resolution of theindividual imaging elements200.
• In some implementations, the resolution of the output image is greater than the combined resolution of the imaging sensors. The combined resolution can be measured as the sum total of pixels of all the sensors in the dense field imager. For example, where the dense field imager comprises 4 facets where each facet has 8 imaging elements with 5 MP sensors, the combined resolution of the dense field imager can be considered to be 160 MP (4×8×5 MP). The resolution of the output image is at least 3× greater than the combined resolution the image sensors, at least 5× greater than the combined resolution the image sensors, at least 10× greater than the combined resolution the image sensors, at least 20× greater than the combined resolution the image sensors, or at least 50× greater than the combined resolution the image sensors. Using the example numbers from this paragraph, the output image generated by the dense field imager can be at least 480 MP (e.g., 3× greater than the combined resolution of the image sensors), 800 MP, 1.6 GP, 3.2 GP, or 8 GP. This can be accomplished by stitching and/or combining light field representations or viewpoints generated from light field representations generated using image data acquired from thefacets300 to effectively increase the combined output resolution of theimaging elements200.
• In some implementations, the dense field imaging system can be mounted on a gimbal on the aerial vehicle. In some implementations, the aerial vehicle can be an unmanned aerial vehicle, an airplane, a helicopter, a satellite, or the like. In some implementations, a ground sample distance of each pixel in the image sensors is at least about 0.5 m and/or less than about 4 m, at least about 0.75 m and/or less than about 2 m, at least about 0.9 m and/or less than about 1.25 m wherein the wide-angle persistent surveillance images the ground from the aerial vehicle. The ground sample distance is a measure of the distance covered by an individual pixel in an image sensor, and is related to the optical properties of the imaging element as well as the size of the pixel and the height (e.g., altitude) of the aerial vehicle.
• Mobile Device with a Dense Field Imager
• The dense field imagers described herein can be used in a variety of contexts. For example, a dense field imager can be incorporated into a mobile device such as a smartphone, cellular phone, PDA, tablet, laptop, or the like. Similar to mobile phone cameras, the dense field imager can be integrated into the mobile device such that the dense field imager can utilize the image processing capabilities and display capabilities of the mobile device to generate images and/or interact with the dense field imager. The dense field imaging system can include a plurality offacets300 each configured to acquire image data and generate one or more light field representations of a field of view. The image processing system can be included on the mobile device or it can be provided by an external system. Thefacets300 can each include a plurality ofimaging elements200 each comprising a lens and an image sensor. The imaging elements can be implemented using wafer-level optics for the lenses and a wafer-substrate for the image sensors.
Example Image Processing System
• FIG. 15 illustrates a block diagram of an exampleimage processing system1500 for receiving data from a plurality of sources of different types and generating light field data, dense field data, and/or viewpoints from received data. Theimage processing system1500 can receive image data fromimage data sources1501 which can include, for example, adense field imager1502, a plurality ofimaging devices1504, a plurality of lightfield data sources1506, or any combination of these. Theimage processing system1500 includes acontroller1508, memory1510, data storage1512, apre-processing module1514, a lightfield generator module1516, a densefield generator module1518, and aviewpoint generator module1520. The various components of theimage processing system1500 can be configured to communicate with one another and/or with external systems through communication bus1507.
• Theimage data sources1501 can provide image data to theimage processing system1500. The image data provided to theimage processing system1500 can include pixel data from a plurality of image sensors, light field representations, dense field representations, metadata, calibration information, and other information associated with the images that allow the image processing system to generate light field data, dense field data, viewpoints, and the like. Theimage data sources1501 can include adense field imager1502, examples of which are described herein with reference toFIGS. 1, 5A, 5B, and 8-13. Theimage data sources1501 can include a plurality ofimaging devices1504, as described herein with reference toFIGS. 14A-C. Theimage data sources1501 can include a plurality of lightfield data sources1506, which can include, for example and without limitation, imaging devices which provide light field data (e.g., plenoptic cameras, light-field cameras, etc.), stored light field data (e.g., computing devices with stored light field data, external storage with light field data, etc.), dense field imagers that output a plurality of light field representations, or any combination of these.
• Theimage processing system1500, in some embodiments, is configured to receive, store, and/or output raw image data. In other embodiments, the image data is not raw. In some cases, theimage processing system1500 stores data received directly from the plurality of sensors, without substantial modification, substantial processing, or without modification or processing, prior to recording and/or output. Examples of image storage and compression are provided in U.S. Patent Pub. No. 2013/0113951, entitled “Video Camera,” which is incorporated by reference herein in its entirety.
• Thevarious modules1514,1516,1518, and1520 of theimage processing system1500 may execute on thecontroller1508 residing in theimage processing system1500 and/or may include custom circuitry. One or more modules may process data that is stored in memory1510 and/or data storage1512, or one or more modules may process data as it comes from theimage data sources1501 which can then be stored in memory1510 and/or data storage1512.
• Theimage processing system1500 includes thecontroller1508. Thecontroller1508 can include one or more processors and can be used by any of the components of theimage processing system1500, such as thepre-processing module1514, the lightfield generator module1516, the densefield generator module1518, theviewpoint generator module1520, to process information. As used herein, the term “processor” refers broadly to any suitable device, logical block, module, circuit, or combination of elements for executing instructions. Thecontroller1508 can be any conventional general purpose single- or multi-chip microprocessor such as a Pentium® processor, a MIPS® processor, a Power PC® processor, AMD® processor, ARM® processor, or an ALPHA® processor. In addition, thecontroller1508 can be any conventional special purpose microprocessor such as a digital signal processor. The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor, such ascontroller1508, can be a conventional microprocessor, but thecontroller1508 can also be any conventional processor, controller, microcontroller, or state machine.Controller1508 can also be implemented as a combination of computing devices, e.g., a combination of a FPGA and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• Theimage processing system1500 includes memory1510. The memory1510 can be coupled to the other components of theimage processing system1500, such as thecontroller1508, data storage1512, thepre-processing module1514, the lightfield generator module1516, the densefield generator module1518, and theviewpoint generator1520. Memory1510 can refer to electronic circuitry and/or devices that allow information, typically processing instructions and/or data, to be stored for retrieval for use by theimage processing system1500 in a relatively quick manner. Memory1510 can refer to Random Access Memory (RAM) or various forms of Read Only Memory (ROM), which are directly connected to the one or more processors of theimage processing system1500. Other types of memory can include bubble memory and core memory.
• Theimage processing system1500 includes data storage1512. Data storage1512 can be coupled to the other components of theimage processing system1500, such as thecontroller1508, memory1510, thepre-processing module1514, the lightfield generator module1516, the densefield generator module1518, and theviewpoint generator1520. Data storage1512 can refer to electronic circuitry that allows information, typically computer data, to be stored and retrieved. Data storage1512 includes computer-readable memory devices and can be in the form of any type of computer-readable medium, such as, for example and without limitation, hard drives, solid-state drives, flash memory, optical discs, or any other type of memory device. In some embodiments, the size of the data storage1512 can be sufficiently large to store light field data, dense field data, image data, and the like from thevarious modules1514,1516,1518, and1520. Data storage1512 can refer to external devices or systems or internal devices or components which are directly connected to the one or more processors of theimage processing system1500.
• In some embodiments, the data storage1512 can be mounted on an exterior of a camera body (e.g., a dense field imager) or be inserted into a slot or other receptacle in the camera body. Further, in some embodiments, the data storage1512 can be connected to the other components of theimage processing system1500 through standard or custom communication ports, including, for example, but without limitation, Ethernet, USB, USB2, USB3, IEEE 1394 (including but not limited toFireWire 400,FireWire 800, FireWire S3200, FireWire S800T, i.LINK, DV), SATA and SCSI. Further, in some embodiments, the data storage1512 can comprise a plurality of hard drives, such as those operating under a RAID protocol. However, any type of storage device can be used.
• Data can be stored in data storage1512 using any variety of data storage methods including, without limitation, data stored in databases, as individual files, as a plurality of data files with configuration (e.g., XML files), in a table, index, map, or other data structure including metadata associated with the image data. For instance, data storage1512 can include raw image data from thedense field imager1502 where the raw image data includes pixel data for imaging elements on a plurality of facets. As another example, data storage1512 can include light field representations from thedense field imager1502 where each light field representation is provided by a facet in thedense field imager1502. As another example, data storage1512 can include a dense field representation resulting from combining light field data and/or raw image data. In yet another example, data storage1512 can include calibration information for thedense field imager1502 that provides depth and alignment information for pixels associated with imaging elements on facets. Data storage1512 can include information that associates particular sets of image data with particular imaging devices from the plurality ofimaging devices1504. Data storage1512 can include image device alignment information that can be used in generating light field data. In some cases, data storage1512 further includes exposure balancing information to balance exposure levels between theimaging devices1504. Data storage1512 can also include distortion correction information, calibration information, pixel correspondence information, and the like.
• In some embodiments, data storage1512 includes an image data set comprising a tensor or matrix derived from a first light field data set representing a first portion of a light field, a second light field data set representing a second portion of a light field, and information relating to a spatial relationship between the first and second portions of the light field. The first and second portions of the light field can be received directly from an imaging device (e.g., a dense field imager, a light-field camera, etc.), the light field data sets can be stored in data storage1512, or the light field data sets can be received from external data sources1521. The light field data sets can be functions of spatial coordinates, direction, wavelength, and/or time, and in some embodiments, the light field data sets can be represented using at least four dimensions. The light field data can be derived from pixel data acquired by imaging elements in a dense field imager. An array of imaging elements can make up a facet, and the pixel data acquired by the array of imaging elements of a facet can be combined into light field data. Accordingly, the first light field data can be derived from pixel data of a first facet and the second light field data can be derived from pixel data of a second facet. The first and second facets can be coplanar or non-coplanar and they can have fields of view that substantially overlap or that are substantially adjacent, with a portion of their fields of view overlapping.
• Theimage processing system1500 includes thepre-processing module1514 which is configured to receive data from theimage data sources1501 and prepare the data for conversion into light field data, dense field data, viewpoint images, or any combination of these. Thepre-processing module1514 can use one or more computer processors to prepare the data. Preparing the data can include, for example, calibrating the received data based on calibration data associated with the device that acquired the data. Calibration can include associating spatial and/or directional coordinates with pixels in the device acquiring the data. Using this correspondence map, thepre-processing module1514 can align corresponding portions of acquired images for later combination and/or conversion to light field representations or dense field representations, as described herein. Thepre-processing module1514 can adjust received data based on properties of the sensors acquiring the data, such as, for example, correcting for dynamic range differences, different sensitivities, different sizes, different resolutions, and the like. In some embodiments, thepre-processing module1514 receives raw image data from a plurality of imaging elements and/or imaging devices and creates separate images from the raw image data. In some embodiments, the separate images are aligned based at least in part on calibration data and/or registration of overlapping images. In some embodiments, the aligned images are compared to a predicted model and deviations from the model are assessed to identify outlier pixels and/or deviations from the model. Based at least in part on these differences, the alignment parameters can be adjusted to improve alignment of pixel data among imaging elements and/or imaging devices.
• In some embodiments, thepre-processing module1514 is configured to combine pixel blocks from image sensors in an array of image sensors to extract improved color information. For image sensor arranged in a Bayer pattern such that a block of pixels are used to generate a single output color (e.g., a 2×2 pixel pattern with a red pixel and a green pixel on top and a green pixel and a blue pixel on bottom), the blocks of pixels from multiple sensors can be combined to provide greater color data based at least in part on angular differences between a source and the pixel block capturing the incident light from the source. Because light color corresponds to a frequency of light and different frequencies of light behave differently through optical components when measured from different angles, an increase of color information can be extracted when comparing pixel blocks on different sensors because they see the object of interest from a different angle. This can improve processes such as edge detection and can reduce or eliminate halos from sharp transitions in a real world image that happens to split a pixel block. This can also lead to better output images, as they may be perceived as more realistic. Thus, by comparing color differences between pixel blocks that view an object from different angles, more accurate color information can be extracted.
• Theimage processing system1500 includes the lightfield generator module1516 which is configured to receive pixel data from a plurality of imaging elements and to derive a light field representation based at least in part on the received pixel data. Facets that include an array of imaging elements can be used to acquire data that is converted into a light field representation. The imaging elements can be coplanar and their optical axes can be primarily aligned along a direction. The lightfield generator module1516 can receive this data from the facets of a dense field imager (or after pre-processing with the pre-processing module1514), for example, and generate a light field representation by determining a radiance of the light in the scene as a function of position and direction. In some embodiments, the light field representation includes time information. In some embodiments, the spatial and directional components of the light field representation are expressed using four dimensions, five dimensions, or more than five dimensions. The lightfield generator module1516 can be configured to receive image data from a plurality ofimaging devices1504 and derive at least one light field representation using the received image data. The lightfield generator module1516 can also use calibration information, alignment information, and the like to generate a light field representation. The lightfield generator module1516 can be configured to receive image data from a variety of image sources and to generate one or more light field representations from the received image data.
• Theimage processing system1500 includes the densefield generator module1518 which is configured to receive a plurality of light field representations and to combine or aggregate them into a dense field data set or representation. The densefield generator module1518 can use one or more processors to derive an image data set comprising a plurality of light field representations and a tensor or matrix describing spatial relationships between the plurality of light field representations or the devices used to obtain the light field representations. In some embodiments, the derived image data set is represented using five dimensions for the light field (e.g., three spatial dimensions and two directional dimensions) with an additional one, two, three, four, or more than four dimensions for the matrix or tensor describing the spatial relationships (e.g., a 3D matrix can be used to represent a relative angular orientation of two facets). In some embodiments, thedense field generator1518 is configured to receive a plurality of light field representations, each representation expressed in five dimensions, and to derive the spatial relationship matrix or tensor on combined or aggregated light field representations. For example, two light field representations having five dimensions each can be combined into a representation having 10 dimensions. In some embodiments, the spatial relationship matrix or tensor can be determined using a representation with less than 5 dimensions, greater than 5 dimensions, greater than 10 dimensions, greater than 20 dimensions, greater than 40 dimensions, greater than 60 dimensions, greater than 80 dimensions, greater than 120 dimensions, greater than 200 dimensions, or greater than 500 dimensions. The densefield generator module1518 can determine the spatial relationship tensor on the 10-dimensional representation.
• In some embodiments, aggregating light field representations comprises representing the received light field representations and spatial and/or temporal information relating the light field representations. The spatial and/or temporal information can include data indicating the relative positions, orientations, and/or times of acquisition of the devices used to obtain the light field representations (e.g., facets on a dense field imager). Aggregating the light field representations can provide for an ability to generate viewpoints from the various light field representations, which viewpoints can then be joined (e.g., by stitching together output images). This can advantageously provide for a system which provides light-field like capabilities over an extended field of view without stitching or fusing the light field representations in vector space prior to generating viewpoints.
• Once the spatial relationship matrix or tensor is determined, the dimensionality of the combined light field representations can be reduced back down to five or fewer dimensions. The light field representation resulting from the process can be a representation that uses non-Euclidean geometry, or that is in a coordinate system that is not flat, but is curved or has non-planar or non-flat dimensions. The resulting spatial relationship matrix or tensor can include one dimension, two dimensions, three dimensions, or more than three dimensions. The resulting spatial relationship matrix or tensor can describe mappings or transformations of light field representations from the non-Euclidean geometry or the geometry of the combined light field representation into a Euclidean geometry or a traditional geometry, allowing theviewpoint generator module1520 to extract viewable images from the combined light field representation. Thus, the 10-dimensional combined light field representation can be reduced to fewer dimensions when represented as a dense field representation.
• This process can be implemented on more than two light field representations. This process can be used to combine light field data while maintaining the dimensionality of the light field representations, which can mean that combining a plurality of light field representations results in an intermediate light field representation having a number of dimensions that is equal to the sum of the dimensions of the input light field representations. For example, if four 5-D light field representations were combined, the intermediate light field representation would have 20 dimensions. Combining the input light field representations using this process can be accomplished wherein, during the combining process, a maximum total number of dimensions used is less than 5 dimensions, greater than or equal to 5 dimensions, greater than or equal to 8 dimensions, greater than or equal to 10 dimensions, greater than or equal to 20 dimensions, greater than or equal to 40 dimensions, greater than or equal to 60 dimensions, or greater than or equal to 80 dimensions. The process can be used to derive a data set that reduces the number of dimensions of the intermediate representation to one that has the same dimensionality as any of the input light field representations. In addition to the combined light field representation, the spatial relationship matrix or tensor can have an additional one, two, three, or more than three dimensions for the spatial relationship matrix or tensor. As a result, a plurality of multi-dimensional light field representations can be combined to form a dense field representation having the same dimensionality as the input light field representations with one or more additional dimensions to describe the spatial relationships of the input light field representations and/or the devices used to acquire the light field representations.
• Theimage processing system1500 includes theviewpoint generator module1520 which is configured to produce images and/or video using received image data, light field representations, and/or dense field representations. Theviewpoint generator module1520 can extract viewable images and/or video from a dense field representation using the spatial relationship matrix or tensor and the associated light field data. The spatial relationship matrix or tensor can be used to define extraction parameters so that a viewable image can be extracted from the combined light field representations. In a traditional light field representation, a 2D image can be extracted from the representation by effectively intersecting a plane at a desired location to determine radiance at each point on the plane, thus producing an image. Based at least in part on the non-Euclidean and/or non-linear nature of the combined light field representations, this process differs from the traditional process as the required geometry of intersection is not necessarily a plane. The spatial relationship matrix or tensor is used to define the correct operation to perform on the combined light field representation to extract the radiance at the desired points. The spatial relationship matrix or tensor can be used to extract 2D images, 3D images, stereoscopic images, images from multiple viewpoints, video, holograms, and the like from the combined light field representations. Described in another way, the dense field representation includes combined light field representations along with a spatial relationship matrix or tensor which can be used by theviewpoint generator module1520 to extract viewable images. Thus, the image data set generated by the densefield generator module1518 contains the radiance information for points within the scene as well as the parameters used to extract viewable images from the image data set. In some embodiments, theviewpoint generator module1520 can generate images using super-resolution techniques, thereby generating an output image with an increased resolution over any of the input images. In some embodiments, theviewpoint generator module1520 can generate viewpoints that effectively see around and/or through occlusions by combining data from multiple viewpoints. In some embodiments, theviewpoint generator module1520 can generate stereoscopic images and can vary an inter-ocular distance. Theviewpoint generator module1520 can be configured to change a point of view or a viewing angle of an output image, change a depth of field, change a focus depth, and to increase a dynamic range of an output image compared to individual images received from theimage data sources1501.
• Theimage processing system1500 as described can provide very powerful creative flexibility after image acquisition (e.g., in post-processing). Thus, apost-processing system1522 can be configured to receive image data, light field data, and/or dense field data from theimage processing system1500. In some embodiments, thepost-processing system1522 is separate from theimage processing system1500, and in some embodiments, thepost-processing system1522 is a part of theimage processing system1500. The creative flexibility provided by thepost-processing system1522 is based at least in part on the ability to manipulate light field data and/or dense field data to accomplish a number of post-processing tasks including, for example and without limitation, re-focusing, changing a depth of focus, relighting based on depth, changing a viewpoint, changing an inter-ocular distance for stereoscopic images, and the like. As one example, the user can re-focus in generally any desired manner using thepost-processing system1522. It can therefore be useful to provide some context regarding the intent of the cinematographer with regard to focus and other parameters when they shot the video. In this regard, metadata associated with a video file can be used to store certain cinematographic settings defined by the cinematographer or other camera operator during the shoot (e.g., focus region or quality, depth-of-field, 3D convergence, inter-ocular distance, white balance, etc.). The metadata can then be used by thepost-processing system1522 to access the user-defined settings. For instance, when the file is opened for playback, thepost-processing system1522 or other playback component may open the file according to the stored metadata, with the initial focus quality or depth-of-field defined by the cinematographer, for example. The metadata can be stored in data storage1512, for example. Thepost-processing system1522 can be configured to allow dynamic editing of a video, reducing a need for detailed plans for capturing the video. For example, the post-processing module can change a focus, zoom in or out, change a viewing angle, and the like to mimic similar effects traditionally performed by camera operators. Thus, thepost-processing system1522, using the dense field data, can change a focus of the creative process from acquiring the video in a particular way to editing the acquired video to achieve a desired effect. In some embodiments, thepost-processing system1522 can be provided as a tool to a community of users. Raw dense field data can then be provided to the community which could allow users to generate new and unique videos based off the originally acquired video. For example, a movie can be shot and a particular version of the movie can be generated for viewing by the general public. The raw data for the movie could be provided to the community of users to allow them to modify aspects of the movie to generate a personalized version of scenes from the movie. A different character can be followed through a scene, a focus of the scene can change, a viewing angle can change, or any combination of these to alter the users' experience. In some embodiments, thepost-processing system1522 can allow for dynamic focusing and tracking of objects within a scene. For example, at sports event in a stadium, the ball and the players can be tracked dynamically so that each can be the focus of the output video stream. This can allow for dynamically changing the focus depending on circumstances to, for example, show what a critical player was doing at a particular point during the event. In some embodiments, thepost-processing system1522 can enable a user to generate viewpoints from a large number of virtual cameras. This can reduce or eliminate the importance of placement of actual cameras and/or the importance of camera operators.
• Theimage processing system1500 may also be configured to compress image data for storage in data storage1512. For instance, image data corresponding to each imaging device in thearray1502 may be compressed separately, and the compressed image data may be associated with the corresponding imaging device in data storage1512. Image data compression and associated techniques are discussed in further detail below. In certain embodiments, thedense field imager1502 stores “compressed raw” image data.
• Theimage processing system1500 can include a laptop, workstation, or other appropriate computing device configured to execute thevarious modules1514,1516,1518, and1520, which can include software modules or programs. Theimage processing system1500 is in communication with theimage data sources1501 through any suitable connection (e.g., via Ethernet, USB, Wi-Fi, Bluetooth, WiGig, or some other appropriate connection).
• Thepost-processing system1522 can ingest image data and metadata from theimage processing system1500. Thepost-processing system1522 can then use the metadata to make creative adjustments to a variety of cinematographic parameters. Such parameters can include focus point or focus quality, depth of field, inter-ocular distance, 3D convergence, and the like. Adjustment of particular aspects will be discussed in greater detail below. Theimage processing system1500 and/or thepost-processing system1522 may additionally provide agraphical user interface1524 on adisplay1523 coupled to theimage processing system1500, for example.
• Because the image data includes image data from a plurality of imaging elements and/or facets in thedense field imager1502, or from each imaging device in thearray1504, or from a plurality oflight field sources1506 orexternal data sources1521, as well as corresponding metadata, video files can include a tremendous amount of image data. Thus, theimage processing system1500 and/or thepost-processing system1522 can additionally compress the received image data and/or metadata. In some cases, the image data and/or metadata is compressed according to a compression ratio of from between about 2:1 and about 30:1. In certain embodiments, the image data and/or metadata is compressed according to a compression ratio of from between about 2.5:1 and about 10:1. In one embodiment, the compression ratio is about 3:1. In other cases, the compression ratio can be at least about 2:1, at least about 4:1, at least about 8:1, at least about 16:1, at least about 20:1, at least about 25:1, or at least about 30:1.
• In other embodiments, theimage processing system1500 can be configured to perform post-processing tasks. In some embodiments, thedense field imager1502 can be configured to perform any or all of the functions described with respect to theimage processing system1500 and/or thepost-processing system1522.
• As described above, in certain embodiments, theimaging devices1504 are separate from one another. The array ofseparate imaging devices1504 can be oriented to capture a viewing theater, as described herein with reference toFIGS. 14A-C. Theimage processing system1500 can be configured to process data from theimaging devices1504. Each of theimaging devices1504 may be a single-lens/single-sensor camera, or can have multiple lenses and/or sensors. For instance, each of theimaging devices1504 could be one of the integrated multi-array imaging devices described herein. In any case, theimaging devices1504 can include memory and an image processing module similar to the corresponding components described with respect to thedense field imager1502 having the integrated array of imaging elements arranged in facets, examples of which are shown inFIGS. 1A-B,5A-B, and8-13.
Image Acquisition
• FIG. 16 illustrates a flow chart of anexample method1600 for acquiring image data with a dense field imager or an array of imaging devices having overlapping fields of view wherein a plurality of light field representations are generated using two or more coplanar imaging elements or imaging devices. Using theacquisition method1600, viewable images are extractable wherein the spatial and/or temporal resolution of the extracted images and/or video is substantially independent of the spatial and/or temporal resolution of the input image data. Thus, using a plurality of imaging devices or imaging elements that acquire image data at individual acquisition resolutions and an acquisition frame rates, video data can be extracted that has a resolution that is greater than any of the acquisition resolutions and a frame rate that is greater than any of the acquisition frame rates. Themethod1600 receives raw image data and generates dense field data through a process that includes pre-processing image data, combining image data, and generating spatial relationship information about the combined image data. For ease of description, themethod1600 will be described as being performed by an image acquisition system. However, any step or combination of steps in the method can be performed by theimage processing system1500 or a dense field imager. Furthermore, the image acquisition system, theimage processing system1500, and the dense field imager can perform any individual step or combination of steps in conjunction with the other steps being performed on other systems. Thus, the description that follows is not to be interpreted as limiting the performance of the acquisition method to a single system or to a system that is separate from the dense field imager or the image processing system.
• Inblock1605, the image acquisition system receives raw image data from a plurality of imaging elements and/or imaging devices. The raw image data can be received directly from the imaging elements, or it can be received from a camera or dense field imager. Receiving raw image data can include receiving an array of digitized values corresponding to quantities of light incident on a pixel of an image sensor. The raw image data can include digitized data corresponding to color bands. For example, raw image data can include an array of digital values arranged according to a known color pattern, e.g., a Bayer pattern. In some embodiments, monochromatic sensors can be used such that two, three, four, or more arrays of digitized values are received where each array corresponds to digitized values for a particular wavelength band. For example, a first array can correspond to a red sensor, a second array can correspond to a green sensor, a third array can correspond to a blue sensor, and a fourth array can correspond to a panchromatic or white sensor. The image acquisition system can combine color information for groups of pixels into a representative color at a location in the sensor. For example, to determine luminance, the image acquisition system can de-Bayer the raw image data. As another example, using monochromatic sensors can allow the image acquisition system to filter out crosstalk between color pixels, thus improving the luminance determination. The image acquisition system can apply any calibration information to the received raw data. For example, baseline values can be corrected for (e.g., “dark” current or noise), response characteristics can be corrected for (e.g., differing sensitivities), and the like.
• Inblock1610, the image acquisition system generates separate images from the received raw data. The image acquisition system can group calibrated raw image data into 2D bitmaps. When the raw data is provided by a dense field imager as described herein, the separate images from imaging elements on a particular facet will be substantially similar as they are pointed in generally the same direction and cover a similar portion of the scene being imaged. Separate images from other facets may be substantially similar (e.g., where the facets are coplanar), they may substantially overlap with images from adjacent facet (e.g., where the optical axes of the facets converge), or they may be substantially adjacent with images from adjacent facet (e.g., where the optical axes of the facets diverge).
• Inblock1615, the image acquisition system registers the separate images. Registering the separate images can be important for later mapping pixel data to real world coordinates. One of the goals of registering the separate images, especially for images from the same facet, is to determine relationships between pixels in the facet. When the image acquisition system registers the images, it can create a matrix of correspondence vectors. The correspondence vectors can be mappings from one pixel to corresponding pixels in other sensors on the facet. For example, for a first pixel on a first image sensor within a facet, a correspondence vector can be created that maps the first pixel to a corresponding pixel on the other sensors on the facet. The correspondence vectors can be created for pixels that acquire image data that is also acquired in other sensors. Thus, for “edge” pixels, or pixels that acquire image data that is not acquired by other sensors, there are no correspondence vectors. The correspondence vectors can be represented as a magnitude and an angle that maps a first pixel to a location of a corresponding pixel. For instance, for a facet with N imaging elements, each pixel in a sensor can have about N−1 correspondence vectors associated therewith, corresponding to vectors from the pixel's position to a position of a corresponding pixel in another imaging element in the facet. Registration can be performed using intensity-based and/or feature-based techniques.
• Inblock1620, the image acquisition system compares the registered images to predicted results and aligns the registered images. Comparing the registered images to predicted results can include modeling behavior of the imaging element, such as the optics, and filtering out expected signal from noise. This procedure can improve registration as it improves signals which in turn can improve registration based at least in part on being able to more closely match real values of pixels when performing intensity-based registration, for example. In some embodiments, the optics on the facets or other imaging devices makes them epipolar. To account for this, the image acquisition system can determine a curvature associated with the system and identify deviations from the expected for each pixel. Outlier pixels can be removed from the image data. The output of this step is a conformed set of correspondence vectors, where the conformation vectors represent corrections to the correspondence vectors based on models.
• The image acquisition system can align the registered images to assign weights to angular data associated with light field representations. Based at least in part on the registered images, pixel data can be associated with real world coordinates. By combining the separate images and comparing the real world coordinates, angular luminance information can be extracted for positions within the imaged scene. By comparing the luminance of different pixels, weights can be assigned to the conformed correspondence vectors according to warping, where the angle corresponds to the inverse of the warping. In some embodiments, it may be advantageous to de-Bayer on a pixel by pixel basis as four times the angular information is available by treating each filtered pixel as a separate pixel value rather than combining all filtered pixels within a block to determine a luminance. From the weighted conformed correspondence vectors, five dimensional light field data can be generated. In some embodiments, the five dimensions can be reduced to four dimensions through a choice of parameterization, as described herein.
• Inblock1625, the image acquisition system fuses the registered and aligned images. Fusing the registered and aligned images can include combining image and/or light field data on a facet-by-facet basis. The image acquisition system can take the images and/or light field data for a given facet along with the weighted conformed correspondence vectors to generate a five-dimensional tensor field. The five-dimensional tensor field includes information about all the light field representations generated for each facet. In some embodiments, data can be combined in an additive manner where light field data is aligned. In some embodiments, data is averaged over position and/or direction to fuse the light field data. In some embodiments, the image acquisition system can generate a single multi-dimensional data structure representing the image data for all the tiles in the dense field imager. The number of dimensions can correspond to the dimensionality of the input light field representations. For example, if five dimensions are used for the light field representations generated inblock1620, then the fused tensor will have five dimensions as well as it is a combination of the light field representations of the facets.
• Inblock1630, the image acquisition system stitches the fused images. Stitching can include projecting the fused tensor into a universal imaging space where the universal imaging space can be a space where all light field rays are parallel. The image acquisition system can then determine corresponding patches from one facet to adjacent facets. The image acquisition can then blend the patches to generate a single light field representation across the universal imaging space. The image acquisition system can extract depth information and flow field information from the fused tensor to enable stitching of the light field representations from adjacent facets.
• Inblock1635, the image acquisition system removes redundant data from the combined image data. After fusing and stitching the light field representations, the image acquisition system can identify and remove light rays that provide redundant information. As an example, after fusion and stitching, the combined light field representation can include millions of rays that provide substantially similar information. The image acquisition system can sample these rays to reduce the number of rays such that sufficient information is maintained to extract viewable images at targeted or desired resolutions and quality. The image acquisition system can be configured to determine which rays deliver substantially the same information and to collapse these rays into a single light ray. In some embodiments, this process can remove one or more dimensions from the combined light field representation.
• Inblock1640, the image acquisition system projects the combined image data into a multi-dimensional tensor representation. The image acquisition system can reduce the dimensionality of the combined light field representation by projecting it into a space having a target or desired number of dimensions. For example, the combined light field representation can be projected into a geometry or space where five dimensions are used to represent the light field data. As a result of this projection, an additional spatial relationship tensor is created which maps the resulting multi-dimensional light field representation from a first geometry which can be non-Euclidean, non-linear, curved, or otherwise irregular, into a Euclidean or linear geometry. The spatial relationship tensor can be one dimension, two dimensions, three dimensions, four dimensions, or more than four dimensions. The spatial relationship tensor can be used to extract viewable images from the multi-dimensional light field representation. A dense field data set or dense field representation can comprise the combination of the multi-dimensional light field representation with the spatial relationship tensor. Thus, dense field data can be a multi-dimensional tensor representation. This multi-dimensional tensor representation can then be stored on any suitable computer-readablememory device block1645, as described elsewhere herein.
Calibration
• FIG. 17 illustrates a flow chart of anexample calibration method1700 for determining corrective factors for imaging elements on a dense field imager or an array of imaging devices. The method generally uses a images of a well-known object to determine correspondences between pixels of different imaging elements or imaging devices. To extract position and direction information when generating light field data and/or dense field data, calibration information determine using themethod1700 can be used to correlate 2D pixel coordinates to 3D image scene coordinates. Thecalibration method1700 can be used to derive calibration information for pixels within an imaging element and/or within an imaging device, which calibration information can be used during image processing to generate quality light field representations and dense field representations. For ease of description, themethod1700 will be described as being performed by a calibration system. The calibration system can be a part of a dense field imager, an array of imaging devices, an image processing system, a module on a computing device, or it can be distributed across a plurality of these and other systems and/or devices. One or more steps in this method can be performed by any one or any combination of elements or modules in the devices and systems described herein.
• Inblock1705, the calibration receives images of a well-known object in space acquired with a dense field imager. It is to be understood, that although much of the description is presented as being performed with images acquired with the dense field imager, any array of imaging elements and/or imaging devices may be used. The well-known object in space can be a calibration target. Anexample calibration target1800 is illustrated inFIG. 18A. Theexample calibration target1800 is an internal corner of a cube with a plurality ofcircular control points1802 placed at known locations on thecalibration target1800. Thecalibration target1800 can be any suitable target that provides a plurality of control points at known locations. Some calibration targets can include an interior of a sphere, ellipsoid, or polyhedron, or control points placed at known locations without conforming to a regular shape. In some embodiments, the control points can be placed randomly if the locations are known or can be determined. Images of thecalibration target1800 can be acquired with the dense field imager, wherein the images include a plurality of images taken with the imaging elements of the dense field imager. In some embodiments, all of the imaging elements acquire images of thecalibration target1800. In some embodiments, less than all of the imaging elements acquire images of thecalibration target1800 and methods are used to interpolate calibration constants for imaging elements that did not acquire an image of thecalibration target1800.
• Inblock1710, the calibration system extractscircular control points1802 from the received images. The calibration system can use sub-pixel image processing to extract this information. In some embodiments, thecontrol points1802 are visually distinguishable from the calibration target by being a different color, a different brightness, having a different texture or pattern, or a different spectral signature when an image is acquired. The calibration system can be configured to identify the control points based at least in part on the different characteristics of the control points1802 on thecalibration target1800. In some embodiments, the calibration system begins with a predicted location for thecalibration targets1802 and iteratively corrects the predicted location until it finds an actual location of acontrol point1802. The difference between the predicted location and the actual location within the image can then be used, at least in part, to derive calibration information. In some embodiments, the calibration system is configured to extract position information for thecontrol points1802 based at least in part on the properties of the control points1802. For example, thecontrol points1802 can include a bar code, QR code, or other visual indication that can be extracted by an image processing system where the visual indication provides either absolute or relative position information. Extractingcontrol points1802 can include identifying pixels that include information about a control point and creating a table or other data structure that maps pixels to control points.
• Inblock1715, the calibration system associates a resulting set of 2D coordinates for each imaging element with a corresponding set of 3D coordinates derived from the physical dimensions of the calibration target. The calibration system can be configured to map a correspondence between a centroid of a control point to a centroid of the pixels that acquired image data of the control point. Based at least in part on the correspondences and the known locations of the pixels and control points, the calibration system generates a map between a pixel location and a corresponding location in 3D space in a scene being imaged.
• Inblock1720, the calibration system determines internal and external geometrical camera parameters from the set of 2D-to-3D correspondences. The internal geometrical camera parameters can include distortion coefficients for pixels within an imaging element and the external geometrical camera parameters can include relative orientation information for every imaging element in the dense field imager.
• Due in part to the imaging elements acquiring images of the calibration target from a variety of angles, there is a distortion in the images of thecalibration target1800 that can be corrected using the correspondence maps. Using internal camera parameters determined using themethod1700 or other similar method, images can be geometrically corrected and undistorted. For example,FIG. 18B illustrates anexample plot1830 showing results of corrected and non-corrected pixel data. The original positions of the control points without distortion removed show misalignment artifacts introduced by optics and other effects. Near the center of the image, where the control points are relatively straight-ahead of the imaging elements, there is little difference between the determined control point positions with and without distortion corrections made. However, the control points at the edges of the images, the control points with the most extreme angles within the images, have distorted positions. By applying the internal camera parameter calibration values to remove distortions, the actual positions of the control points can be recovered, as is shown by theplot1830.
• Imaging elements can have optical axes that are not parallel with one another and the correspondences can be used to determine relative orientations of imaging elements. With the generated external camera parameters, the position for every imaging element in the array relative to the calibration target can be calculated. For example,FIG. 18C illustrates aplot1860 of relative imaging element orientation determined from the external camera parameters. The relative orientation of two imaging elements is indicated by the axes labeled1862aand1862b. The relative orientation indicates the direction of the optical axes of the two imaging elements (the arrow pointing toward the calibration target) with additional axes defining the horizontal and vertical for the respective imaging elements.
• One measure for a quality of calibration is a vertical misalignment of corresponding control point locations in a set of two imaging elements after calibration corrections are applied. After applying thecalibration method1700 on a dense field imager, theplot1890 illustrated inFIG. 18D is achieved. Theplot1890 demonstrates that the vertical misalignment, on average, is approximately zero with a spread in the distribution of less than about 0.1 pixels.
• Inblock1725, the calibration system can store the resulting calibration parameters in data storage, such as the data storage1512 described with reference toFIG. 15. The calibration parameters can be retrieved and utilized by a dense field imager, an array of imaging devices, an image processing system, a computing device, or any combination of these to produce light field data, dense field data, image data, or the like.
Light Field Stitching
• FIG. 19 illustrates a flow chart of anexample method1900 for stitching light field representations. Stitching light field representations can include combining light field representations from one or more light field data sources. The light field data sources can include stored light field data, an array of plenoptic or light-field cameras, an array of imaging devices, a dense field imager, or any combination of these. Stitching light field representations can be used to expand a field of view covered by light field data. For example, a first light field representation can cover a first field of view and a second light field representation can cover a second field of view, the second field of view overlapping with the first field of view. Themethod1900 can be used to combine the first and second light field representations to produce a dense field representation that includes a combined light field representation and a spatial relationship tensor such that the combined field of view of the resulting dense field representation covers the first and second fields of view. For ease of description, themethod1900 will be described as being performed by the image processing system, but any suitable device or combination of devices or systems can be used to perform the method. For example, a dense field imager can be configured to stitch light field representations. Any one or any combination of the following steps of themethod1900 can be performed by one or more components of the image processing system, the dense field imager, a computing device, and the like.
• Inblock1905, the image processing system acquires pixel data from a plurality of imaging elements. In some embodiments, the pixel data can be acquired directly from the imaging elements on a facet of a dense field imager. In some embodiments, the pixel data is acquired and stored and then retrieved by the image processing system. The pixel data can include metadata or other configuration or calibration parameters that facilitate and/or improve processing of the pixel data. Alignment or registration information of the pixel data can be included that maps the 2D pixel data to 3D image scene data. In this way, pixel data can be mapped to spatial and directional coordinates which can facilitate further processing.
• Inblock1910, the image processing system generates a plurality of light field representations. Generating a plurality of light field representations can include grouping planar imaging elements to create a light field representation. In some embodiments, the planar imaging elements also have substantially parallel optical axes. The plurality of light field representations can be generated based at least in part on geometrical considerations. For example, a first light field representation can be generated using pixel data acquired from imaging elements aligned generally in a first plane and a second light field representation can be generated using pixel data acquired from imaging elements aligned generally in a second plane, the second plane non-coplanar with the first plane. In some embodiments, the plurality of light field representations are generated based at least in part on logical groupings. For example, a first set of imaging elements can be coupled to a first acquisition and/or processing electronics, such as in a facet of a dense field imager. A second set of imaging elements can be coupled to a second acquisition and/or processing electronics. The first set of imaging elements and the second set of imaging elements can be used to generate first and second light field representations, respectively. The first and second sets of imaging elements can be coplanar or the first set can be non-coplanar with the second set. In some embodiments, the fields of view of the generated light field representations overlap. Generating a light field representation can include using a plurality of images to define a data set that describes radiance as a function of position and direction. In some embodiments, the generated light field representations can be functions of time and/or wavelength. In some embodiments, the five dimensions representing space and direction can be reduced to four dimensions using a parameterized space, such as using a light slab, light spheres, a plane and angle, and the like.
• Inblock1915, the image processing system creates a spatial relationship matrix or tensor that defines spatial relationships among the light field representations and/or the respective devices or systems generating the light field representations. To aggregate or combine light field representations, the image processing system can join the light field representations into a representation of a higher dimensional order to accommodate the dimensionality of each of the component light field representations. For example, if the image processing system were to join two 5D light field representations, the image processing system could create a 10D representation accommodating the two 5D representations. The image processing system can then process the combined light field representation to identify spatial relationships between the light field representations. As a result of this determination, a spatial relationship matrix or tensor can be created that effectively maps light field representations in relation to one another or that expresses the relative positions and/or orientations of the light field representations. The spatial relationship matrix or tensor can be used, for example, to generate viewpoints from a resulting combined light field representation and/or to generate a viewpoint from a light field representation and combine that with a viewpoint generated from another light field representation as the relative orientations of both are provided by the spatial relationship matrix or tensor. For example, for a single light field representation a 2D bitmap can be generated by bisecting the light field representation with a flat plane. The spatial relationship matrix or tensor for this scenario could be used to identify the flat plane to use to extract the desired image. Similarly, for aggregated light field representations, the flat planes used to generate the 2D bitmaps can be related via the spatial relationship matrix or tensor, which can allow the flat planes to be configured to be parallel and/or co-planar. Similarly, for combined light field representations, the geometry may not be flat or Euclidean and the spatial relationship tensor can be used to identify the correct “shape” (analogous to a plane to generate a 2D bitmap from a single light field representation) to use to bisect the combined light field data to extract a desired viewpoint.
• In some embodiments, stitching light field representations comprises projecting light field representations into a universal imaging space where all rays are parallel. Light field data acquired with adjacent imaging devices or facets can then be joined by finding corresponding patches between the light representations and blending the light field representations at those corresponding patches. Finding the corresponding patches can, at least in part, provide information for determining the spatial relationship tensor as the correspondences are related to the spatial relationships between the light representations.
• Inblock1920, the image processing system uses the spatial relationship matrix or tensor to aggregate or combine the light field representations to create a dense field representation. Once the spatial relationship matrix or tensor is determined, the combined light field representation can be projected into a reduced-dimension representation, e.g., a 5D tensor. The combination of the 5D tensor representing the aggregated or combined light field representations and the spatial relationship matrix or tensor can comprise the dense field representation. With this information, a viewpoint generator can generate viewpoints and a post-processing system or module can achieve light field-type effects. The light representations have been effectively joined to create a dense field representation over the combined field of view.
• An example of where this method may be applied is shown inFIGS. 20A and 20B.FIG. 20A illustrates a scene to be imaged2000. A region ofinterest2002 is indicated with a rectangle and a field ofview2004 of a device providing light field data is indicated with a second, dashed rectangle. The field ofview2004 of the light field device is shown inFIG. 20B. To cover the region ofinterest2002, then, a plurality of light field representations can be aggregated or joined. Themethod1900 can be used to aggregate or join the light field representations by intersecting the light rays of the light field representations to be joined or by generating the respective light fields and storing the information along with the orientation and/or position information of the representations. Intersecting the light rays comprises matching vectors in 4 or more dimensions. By finding the intersection of the vectors, information can be determined that maps the light field representations into a single light field representation in a coherent geometry from which any cross section will result in a valid bitmap across the cumulative field of view. In some embodiments, the cross-section is not a plane and is determined using the spatial relationship tensor. In some embodiments, the cross-section for each aggregated light field representation is a plane and the spatial relationship matrix or tensor can be used to provide information about relative orientations and positions of the planes.
• Inblock1925, the image processing system stores the dense field representation. In some embodiments, the dense field representation can be stored in data storage1512 which is described with reference toFIG. 15. In some embodiments, the dense field representation can include the output aggregated or combined light field representation, the spatial relationship matrix or tensor, the acquired pixel data, the generated light field representations, metadata, or any combination of these.
• In some embodiments, combining the light field representations includes stitching together light field representations having substantially non-overlapping fields of view to create a dense field image set having a substantially wider field of view than the individual light field representations. The combined lateral field of view of the dense field image set can be greater than or equal to about 145 degrees in some embodiments.
• In some embodiments, themethod1900 of stitching light field representations is independent of where imaging elements and/or imaging devices are positioned relative to one another. This can be accomplished by intersecting the light rays of the input light field representations as that process will remove dependence, at least in part, on sensor position. In some embodiments, relationships between any number of light field representations that share a sufficient field of view should be combinable into a single tensor lattice (e.g., a combined light field tensor in combination with a spatial relationship tensor) from which images can be created.
Depth Mapping and Focus and Depth of Field Control
• A dense field imager and/or multiple imaging devices can be used to provide depth mapping. Close objects will shift more in position between imaging devices than objects that are far away. A depth map can be created using a determination of how much an object shifts between images. In one embodiment, some imaging devices are used to create a depth map, while other imaging devices are used to capture images with different focus settings. Depending on the embodiment, the depth mapping techniques described herein may be used with any of the systems described herein having an array of physically separate cameras (e.g., with respect toFIGS. 14A-C orFIG. 15) or, alternatively, to any of the camera systems having an integrated array of imaging elements (e.g., with respect toFIGS. 1A-B,5A-B, and8-13).
• For example, some of the imaging devices could use different focal distances, and the depth map would provide information about which imaging devices would best represent an object in focus. As another example, some of the imaging devices could use different apertures and the depth map could be used to select the desired depth of field for different portions of the resulting image. In yet another embodiment, the depth map could be used to artificially create or adjust focus, bokeh and/or depth of field during post-processing.
• Where the imaging devices have a relatively wide depth of field (e.g., at or near infinity), the native images captured from the imaging devices may be substantially in-focus, across the entire depth of the image scene. In such cases, the depth map can be used to create artificial bokeh, simulating out-of-focus quality in desired image regions. For instance, the depth map can be used to simulate a user-selected region of focus or depth-of-field by blurring image regions outside of the selected region of focus or depth of field.
• In one embodiment, the user selects a focus region while shooting. The user may select a region of focus using the viewfinder touch screen, cursor, voice command, or other appropriate input mechanism, and the camera system stores this desired focus region in metadata associated with the video file. The system may also add bokeh to the image that is displayed on the viewfinder in response to the user selection.
• As discussed, in post-processing, the focus or depth of field set by the cinematographer during shooting may be initially presented to the user. But, depending on the desired creative effect, and regardless of the cinematographer's settings, the user can adjust the focus or depth of field as desired, providing substantial creative flexibility.
• The depth map can also be used to control the 3-D effect of an image. The depth map may be shown in substantially real time. Industries such as, for example, robotics, biometrics, security, surveillance and autonomous vehicles can advantageously utilize depth information.
• The depth map can be created by the post-processing module. In another embodiment, the image processing module creates the depth map.
• Moreover, while the transition from in-focus to blurry regions in traditional cameras may have a certain characteristic response depending on the aperture of the lens, the synthetic blurring effect achievable using the multi-imaging device cameras described herein can be tailored to achieve generally any desired blurring effect. For instance, the transition from in-focus to blurry regions, can be linear, logarithmic, exponential, step-function, or a combination thereof.
• The multi-imaging device techniques can also be used to provide interactive focus and depth of field during playback. For instance, the user in some cases can select a first point of focus and/or depth of field to view an initial segment of a video clip, and then change the point of focus or depth of field as desired, during viewing. As one example, during instant replay of a catch in a football game, the video operator may be able to adjust the depth of field depending on the movement of the football or of the receiver. Where the imaging processing system on the camera generates the combined video image data on camera, the user may use the touch screen interface and/or controls on the camera to pause playback, modify the focus or depth of field, and then resume playback, as desired.
• In these and other ways, certain embodiments described herein can capture full light field effect image data and/or provide similar advantages to light-field and plenoptic cameras, such as the ability to adjust focus and/or depth of field in post-processing.
• Three Dimensional Video with an Imaging Device Array
• Images taken from two different perspectives can be combined in a variety of ways to create an illusion of depth. In one example, glasses containing liquid crystal may block or pass light in synchronization with an alternating display of images, so that the left eye sees a different image than the right eye. In another example, glasses containing linearly polarized lenses or circularly polarized lenses may block or pass light corresponding to the polarized projection of two different images. Other examples that use glasses include, for example, Infitec (interference filter technology),Inficolor 3D, complementary color anaglyphs,ColorCode 3D, Chromadepth. Other examples, which do not require glasses, are referred to as autostereoscopy.
• A stereoscopic image typically uses two or more pictures taken from different perspectives. For example,imaging devices200aand200binFIG. 7A, or any different subset of one ormore cameras1402 in any of theFIGS. 14A-C, provide pictures from one pair of different perspectives.Imaging devices200aand200cprovide pictures from another pair of different perspectives. The illusion of depth changes with the distance between the imaging devices, so different pairs of imaging devices provide different perceptions of depth. While the exemplary illustration inFIG. 7A provides three imaging devices, it is to be understood that multiple imaging devices could be used, providing the director with multiple perspective options.
• The perceived depth of objects in an image is affected by the distance and/or relative angle between perspectives. Multiple images with varying distances between the images, or inter-ocular distance, can be acquired to provide options for post processing. Moreover, multiple images with varying angles (e.g., 3D convergence values) between the images, can be acquired to provide options for post processing. As mentioned above, an initial inter-ocular distance and/or 3D convergence values can be set by the cinematographer during the shoot, and the post-processing software can load this pre-set value as a default when the video file is loaded for playback or editing. The user can then adopt the default inter-ocular distance or adjust the distance in post as desired.
• In some cases, a first set of at least one sensor is displaced from a second set of at least one sensor by a distance within the range of from about 1 inch to about 18 inches, often within the range of from about 1.5 inches to 6 inches within the same single hand held device, to capture stereoscopic or full light field effect image data. In other cases, separate cameras located at spaced apart locations in a viewing theater are used.
• For cases where there is a need to accurately represent motion, it is preferred that the multiple images are captured at the same time.
• High Resolution Video with an Imaging Device Array
• As discussed, an image processing system can combine or stitch together the image data from the individual imaging devices. And as mentioned, with respect toFIG. 7A andFIGS. 19 and 20A, for example, the stitching can result in an effective resolution of the combined image that is significantly higher than the individual sensors.
• Another technique that can be used to improve resolution involves generating multiple sequential image samples at slightly offset spatial positions and/or different incident angles. For example, a mechanically actuatable optical element can be positioned before an array of imaging devices, such as in the array of imaging elements illustrated in any ofFIGS. 8-13.
• The optical element can be configured to wobble or otherwise move to direct light rays onto the array at slightly different spatial positions and/or at incident angles. The optical element can include a transparent planar member, and can be formed of glass, plastic, or some other appropriate optically transmissive material. The optical element can include a refractive lens in some cases. In some embodiments, a mechanical actuator at each corner of the optical element can be included, although any appropriate mechanism can be used. Movement of the optical element can result in a corresponding angular variation in the light exiting the optical element. The effect of this variation is that, depending on the current position of the optical element, the light rays are directed towards slightly different portions of the array and at different angles of incidence.
• In some embodiments, the optical element is actuated to direct the light rays in a repeating pattern, which can vary, depending on the embodiment. For instance, a square pattern can be used, in which the optical element directs the rays one or more pixels (or a fraction of a pixel) right, then down, then left, and then up return to the original position. Or the light rays can be moved in a circular pattern. The imaging devices can be configured to sample the image at different points along the pattern trajectory.
• When the optical element is transitioned through the pattern rapidly, each individual pixel in the imaging devices captures a corresponding point in an image scene in relatively rapid succession, but at slightly offset spatial positions and differing incident angles. The resulting sequentially captured images can then be combined to create higher resolution image frames. For instance, the image processing system can combine the successively captured images to create high resolution video frames.
• Because the samples are captured sequentially, there can be some amount of motion in the image scene that occurs between successive samples. This can result in blur in the combined image. In some embodiments, the camera includes a control that allows the user to adjusting the sampling rate to achieve a desired level of blur. In other cases, such as where any amount of blur is undesirable or where lower resolutions are acceptable, movement of the optical element can be disabled. For instance, the camera can include a button, switch, touch screen control or other appropriate mechanism to disable movement of the optical element. In yet other configurations, the optical element is not included.
• The camera system can be configured to output video at a variety of resolution levels, and the output resolution levels can be independent of the input resolution levels. For instance, any of the imaging systems described herein can generate image data (e.g., compressed raw image data) having “6k” (e.g., 6144×3160), “5k” (e.g., 5120×2700), Quad HD (e.g., 3840×2160 pixels), “4.5k” resolution (e.g., 4,520×2540), “4k” (e.g., 4,096×2,540 pixels), “2k” (e.g., 2048×1152 pixels) or other resolutions. As used herein, in the terms expressed in the format of xk (such as 2k and 4k noted above), the “x” quantity refers to the approximate horizontal resolution. As such, “4k” resolution corresponds to about 4000 or more horizontal pixels and “2k” corresponds to about 2000 or more pixels. The dense field imager, for example, can acquire data that a viewpoint generator can use to output video image data having resolutions of at least 2k, at least 4k, at least 4.5 k, at least Quad HD, at least 5k, at least 6k, at least 10k, or at least 18k, at least 20k, or greater. In some embodiments, the camera can be configured to record image data (e.g., compressed raw image data) having a horizontal resolution of between at least any of the foregoing resolutions (e.g., between 2k and 20k, between 4k and 18k, between 5k and 10k, etc.).
• In certain embodiments, the imaging systems described herein can also achieve relatively high zoom ratios. For instance, in one embodiment, the dense field imager can produce output data that can achieve a zoom ration of 100:1. In various embodiments, at least about 10:1, at least about 25:1, at least about 50:1 and at least about 100:1 zoom ratios are possible. In addition, the imaging elements of the dense field imager can sample image data at relatively high frame rates. For instance, in one embodiment, the imaging elements can capture image data at a frame rate of at least 500 frames per second. In various embodiments, the imaging elements can capture image data at a frame rate of at least 150 frames per second, at least 200 frames per second, at least 300 frames per second, at least 400 frames per second, or at least 500 frames per second, and at a resolution of at least 2k. In other embodiments, the camera502 system can capture image data at a frame rate of at least 150 frames per second, at least 200 frames per second, at least 300 frames per second, at least 400 frames per second, or at least 500 frames per second, and at a resolution of at least 4k.
High Dynamic Range Video Using an Imaging Device Array
• The imaging devices described herein, for example, could be configured to capture images using different exposure times. The images could then be combined to create a high-dynamic range image. Motion may affect the quality of images captured using different exposure times.
• The imaging devices could also be configured to apply different gains to the signals coming from the sensors. This is comparable to adjusting the ISO setting on traditional cameras. Increasing the gain allows measurement in low-light situations, with the trade-off of increased noise in the measurement. The image could then be reconstructed using low-gain imaging devices for areas of the image that have sufficient light, and high-gain imaging devices for areas of the image that need a boost in the signal.
• A combination of different exposure times and different gain settings for the various imaging devices could also be used. Additional compatible high dynamic range techniques are discussed below and are described in U.S. Patent Application Publication No. 2012/0044381 entitled “HIGH DYNAMIC RANGE VIDEO”, which is incorporated by reference herein in its entirety.
High-Speed Videography
• FIG. 21 illustrates a timing diagram for capturing images using multiple imaging devices. The timing of the image captures may be arranged to increase the frame capture rate for video. For example, where a single imaging device might be capable of capturing 60 image frames per second, three imaging devices could be used to capture 180 image frames per second. In order to reduce the effects of “rolling shutter,” each frame could comprise rows obtained from multiple imaging devices. For example, the first third of the frame might come from a first imaging device, the second third of the frame might come from the second imaging device, and the last third of the frame might come from the third imaging device. As illustrated in the timing diagram shown inFIG. 21, the first third of the rows of a first sensor, the second third of the rows of a second sensor, and the last third of the rows of a third sensor are captured at approximately the same time. While three sensors were used for illustration purposes, it is to be understood that other numbers of sensors may be used.
Other Image Processing Operations
• The image processing systems described herein, such as theimaging processing system102 ofFIG. 1A, theimage processing system1500 ofFIG. 15, or the dense field imagers ofFIGS. 8-13 can perform a variety of other operations on the image data. For example, the image processing system may optionally compress the data from the imaging devices, perform pre-compression data-preparation (e.g., pre-emphasis and/or entropy reduction), format the compressed data, and the like. Examples of such techniques are described in greater detail in U.S. Pat. No. 7,830,967 entitled “VIDEO CAMERA” (the '967 patent), and U.S. Patent Pub. No. 2013/0113951, entitled “Video Camera,” which are both incorporated by reference herein in their entireties. In some instances, the image processing system processes the image data to tune the image streams to perceptual space. Additional compatible techniques include green data modification and pre-emphasis processes shown and described throughout the '967 patent (e.g., with respect to FIGS. 8-11 and columns 11-13). In general, certain embodiments described herein are compatible with and/or are components of embodiments described in the '967 patent. Thus, some or all of the features described herein can be used or otherwise combined with various features described in the '967 patent, including features described in the '967 patent which are not explicitly discussed herein.
Viewpoint Generation
• FIG. 22 illustrates a flow chart of anexample method2200 for generating a viewpoint using dense field data. Themethod2200 can be used to extract viewable images and/or video in a variety of formats and resolutions. The dense field data allows for the spatial and/or temporal output resolution to generally be independent of the input spatial and temporal resolutions. For ease of description, themethod2200 will be described as being performed by the viewpoint generator, but it is to be understood that any suitable image processing system, post-processing system, dense field imager, computing device, or the like can be used to perform any of the following steps and any combination of devices can be used to perform themethod2200.
• Inblock2205, the viewpoint generator receives a dense field representation. The dense field representation can be provided by, for example, an array of imaging devices, a dense field imager, an image processing system, data storage devices, a computing device, or any combination of these. The dense field representation can include light field data from a plurality of devices that cover a combined field of view. Light field data can be combined, as described herein with reference toFIG. 19, to generate dense field data. The dense field data can include light field information and spatial relationship information. The spatial relationship information can be used to selectively bisect a portion of the combined light field representation to extract a viewable image. The dense field data can provide post-processing capabilities similar to light field data, such as re-focusing capabilities, the ability to generate a variety of viewing angles, change inter-ocular distance for 3D images, and the like.
• Inblock2210, the viewpoint generator determines a portion of the dense field representation that contains the region of interest. Because the dense field data can cover a relatively large field of view (e.g., potentially 360 degrees in azimuth and 180 degrees in elevation), the viewpoint generator determines a region of interest within the scene data. The viewpoint generator then uses the spatial relationship matrix to determine a size and shape of an object to use to bisect the combined light field data to extract a viewable image of the region of interest. In some embodiments, determining the portion of the dense field data that contains the region of interest includes extracting one or more images from the dense field data and refining the bisection object to find the desired view of the region of interest.
• Inblock2215, the viewpoint generator calculates an optical flow and/or motion compensation related to the region of interest. The optical flow can include image artifacts related to positional and/or motion effects of the imaging elements used to acquire the image data and/or the objects being captured. The optical flow can be used to correct for aberrations that could potentially adversely affect an output image or video. Motion compensation can account for differing positions of objects when taken at different points in time. In some embodiments, the viewpoint generator can interpolate and produce images at viewpoints and/or times where no image data was actually acquired. For instance, image data from imaging elements positioned at a variety of viewpoints can be combined to create a virtual viewpoint. As another example, image data taken at different points in time can be used to generate a view of the region of interest at a time when no data was actually acquired.
• Inblock2220, the viewpoint generator digitally re-focuses on the region of interest. The viewpoint generator can manipulate the dense field data to produce an image that is digitally focused on an object of interest. The viewpoint generator can change a focus depth and/or a depth of focus. In this way, different objects can be in or out of focus and a greater or smaller percentage of the generated viewpoint can be focused.
• Inblock2225, the viewpoint generator renders an image or video of the region of interest. Using the optical flow, motion compensation, region of interest, and focus information derived in the preceding steps, the viewpoint generator can extract viewable images or video from the dense field generator. The output resolution can be independent of the resolution of the imaging devices used to acquire the image data.
• In this way, the viewpoint generator can be configured to synthesize images from any dense field data set (e.g., data acquired using multiple sensors, multiple cameras, array of cameras, etc.). The viewpoint generator can be configured to generate pixel data, to data for blocks of pixels, shapes data, and finally images. The viewpoint generator can provide a single integrated view from disparate sources. For example, data from disparate imaging devices can be combined into a dense field data set and the viewpoint generator can extract a single, integrated view of a scene from the disparate imaging devices. In some embodiments, the viewpoint generator can be configured to extract data at a variety of dimensions such as, for example, 1D pixels, 2D bitmaps, 3D temporal image sequences, 4D stereoscopic sequences, 5D light field representations (for holograms), etc. In some embodiments, the viewpoint generator can be configured to produce automated compositions based on input criteria. For example, the viewpoint generator can be configured to focus on an object of interest when a particular object is within a defined space. In some embodiments, the viewpoint generator can be coupled to a networked input source that receives information from multiple users over a social network to provide dynamic viewpoints according to user requests.
Manufacture of Imaging Elements
• In some embodiments, one or more liquid lens cells may be used with a small sensor in an imaging device. The imaging device may then be attached to an adjustable frame.
• Some optics may be implemented in the dielectric stack of the integrated circuit using refractive microlenses or diffractive gratings patterned in the metal layers.
• Making the sensor and optics small provides for mass-production of the imaging elements.
• In some embodiments, the image sensors manufactured as described herein can comprise monochromatic sensors. A monochromatic sensor can be configured to be sensitive to a targeted bandwidth of light, such as red light, blue light, green light, yellow light, and or white light (or a panchromatic sensor). The monochromatic sensors can be manufactured on a wafer-scale using wafer-scale optics as described herein. Color filters or dyes can be applied to any suitable stage and/or any suitable layer. For example, the lens elements can be coated with a filter or the lens elements can include a dye which substantially filters light outside of a targeted portion of the electromagnetic spectrum.
• In some embodiments, the monochromatic sensors do not include any filters and are configured to be used in conjunction with a lens plate that includes monochromatic filters. When the lens plate and the sensors are used in conjunction, monochromatic sensors are effectively created. In some embodiments, the lens plate is removable and allows for different color filter schemes to be used to achieve a desired result. In some embodiments, a wafer is configured to contain at least four image sensors and a lens plate is configured to transmit a particular wavelength band to an associated image sensor. For example, the lens plate can make a first sensor a red sensor, transmitting red light while substantially blocking all other wavelengths, the lens plate can make a second sensor a blue sensor, a third sensor a green sensor, and a fourth sensor a white sensor. Control of the individual sensors can be arranged through control electronics. In some embodiments, by making the image sensors monochromatic, cross-talk is reduced and image quality increases, as described herein.
• In some embodiments, any of the imaging systems described herein, including the dense field imagers, can include a camera array that is fabricated on a semiconductor chip. For example, U.S. Patent Publication 2011/0080487 entitled “Capturing and Processing of Images Using Monolithic Camera Array with Heterogeneous Imagers,” the entirety of which is incorporated by reference herein, describes the use of an array of imagers where each imager in the array comprises a plurality of light sensing elements and a lens stack fabricated on a semiconductor chip. The camera array may include two or more types of heterogeneous imagers, each imager including two or more sensor elements or pixels. Each one of the imagers may have different imaging characteristics. Alternatively, there may be two or more different types of imagers where the same type of imager shares the same imaging characteristics.
• In some embodiments, each imager has its own filter and/or optical element (e.g., lens). Specifically, each of the imagers or a group of imagers may be associated with spectral color filters to receive certain wavelengths of light. Example filters include a traditional filter used in the Bayer pattern (R, G, B or their complements C, M, Y), an IR-cut filter, a near-IR filter, a polarizing filter, and a custom filter to suit the needs of hyper-spectral imaging. Some imagers may have no filter to allow reception of both the entire visible spectra and near-IR, which increases the imager's signal-to-noise ratio. The number of distinct filters may be as large as the number of imagers in the camera array. Further, each of the imagers or a group of imagers may receive light through lenses having different optical characteristics (e.g., focal lengths) or apertures of different sizes.
• In some embodiments, the imagers in the camera array are spatially separated from each other by a separation distance. By increasing the spatial separation, the parallax between the images captured by the imagers may be increased. The increased parallax is advantageous where more accurate distance information is important. Separation between two imagers may also be increased to approximate the separation of a pair of human eyes. By approximating the separation of human eyes, a realistic stereoscopic 3D image may be provided to present the resulting image on an appropriate 3D display device.
• In some embodiments, multiple camera arrays are provided at different locations on a device to overcome space constraints. One camera array may be designed to fit within a restricted space while another camera array may be placed in another restricted space of the device. For example, if a total of 20 imagers are required but the available space allows only a camera array of 1×10 imagers to be provided on either side of a device, two camera arrays each including 10 imagers may be placed on available space at both sides of the device. Each camera array may be fabricated on a substrate and be secured to a motherboard or other parts of a device. In addition, such imagers do not have to be homogenous in size, and may have different x- and y-dimensions. The images collected from multiple camera arrays may be processed to generate images of desired resolution and performance.
• In some embodiments, the imaging system can include wafer level optics. The wafer level optics can include a plurality of lens elements, where each lens element covers one of the sensors in the array. For example, the imaging system can be an array of pixels overlaid with color filters and microlenses. The microlenses that sit on top of the color filters are used to focus light on the active area of each underlying pixel. The microlenses can be thought of as sampling the continuous light field in object space sampled by the main lens. Whereas the main lens samples the scene radiance light field, the micro-lenses sample the sensor irradiance light field.
• The main lens associated with each imager maps the points in the object space to points in the image space such at that the mapping is bijective (onto-to-one and onto). Each microlens samples a finite extent of the sensor irradiance light field. The sensor irradiance light field is continuous and is the result of a bijective mapping from the object space. Thus, the microlens sampling of a finite extent of the sensor irradiance light field is also a sampling of a corresponding finite extent of the scene radiance light field in object space.
• In some embodiments, the imaging systems and dense field imagers described herein can include monochromatic imaging elements. The monochromatic imaging elements can include color filters that are off the sensor, e.g., on the lens. The color filters can be modular, so that different filter plates can be switched and used with the same sensor array. The filter plates can be added at the point of manufacture, or by the consumer. Thus, the imaging system can include a plurality of imaging elements that are assembled in a uniform fashion, e.g., sensors that are sensitive to a broad spectrum of light, and the imaging system can be customized by implementing a combination of filter arrays. This type of configuration can reduce manufacturing costs and/or increase utility of the imaging system.
EXAMPLE EMBODIMENTS
• The following is a numbered list of example embodiments that are within the scope of this disclosure. The example embodiments that are listed should in no way be interpreted as limiting the scope of the embodiments. Various features of the example embodiments that are listed can be removed, added, or combined to form additional embodiments, which are part of this disclosure:
Example Dense Field Imaging Block and Array
• 1. An imaging block, comprising:
• • a support;
  • at least a first imaging element and a second imaging element carried by the support, each imaging element comprising a sensor and a lens;
  • a mechanical connector for mechanically connecting the imaging block into an array of imaging blocks; and
  • an electrical connector for electrically connecting the imaging block into an array of imaging blocks.
• 2. An imaging block as inembodiment 1, wherein at least the first imaging element comprises a monochromatic filter, wherein the sensor of the first imaging element detects substantially monochromatic light passing through the monochromatic filter.
• 3. An imaging block as inembodiment 1, wherein the support comprises a wafer substrate.
• 4. An imaging block as inembodiment 3, wherein the wafer substrate comprises a semiconductor material.
• 5. An imaging block as in embodiment 4, wherein the sensors are formed on the substrate.
• 6. An imaging block as inembodiment 3, wherein the lenses are wafer-level lenses.
• 7. An imaging array, comprising an array support, and at least two imaging blocks of any ofembodiments 1 through 6 carried by the array support, the at least two imaging blocks comprising a first imaging block and a second imaging block.
• 8. An imaging array as in embodiment 7, wherein at least one of the sensors in the first imaging block is non-coplanar with at least one of the sensors in the second imaging block.
• 9. An imaging array as in embodiment 7, wherein each of the sensors in the first imaging block are coplanar, each of the sensors in the second imaging block are coplanar, and each of the sensors in the first imaging block are non-coplanar with each of the sensors in the second imaging block.
• 10. An imaging array as in embodiment 7, wherein each imaging element has a primary optical axis, and the primary optical axis of at least one of the imaging elements in the first imaging block is substantially non-parallel with the primary optical axis of at least one of the imaging elements in the second imaging block.
• 11. An imaging array as in embodiment 7, wherein each imaging element has a primary optical axis, the primary optical axes of the imaging elements in the first imaging block are substantially parallel, the primary optical axes of the imaging elements in the second imaging block are substantially parallel, and the primary optical axes of the imaging elements in the first imaging block are substantially non-parallel with the primary optical axes of the imaging elements in the second imaging block.
• 12. An imaging array as in embodiment 7, wherein a primary optical axis of at least one imaging element of the first imaging block is angularly adjustable with respect to a primary optical axis of at least one imaging element of the second imaging block.
• 13. An imaging array as in embodiment 12, wherein the imaging array comprises a user-actuatable control for achieving the angular adjustment.
• 14. An imaging array as in embodiment 7, further comprising:
• • an image processing system configured to:
    • using image data captured by the first imaging block, generate a first image data set representative of a first portion of a light field; and
    • using image data captured by second imaging block, generate a second image data set representative of a second portion of the light field; and
    • derive a third image data set from the first and second image data sets.
• 15. An imaging array as in embodiment 14, wherein the image processing system is carried by the array support.
• 16. An imaging array as in embodiment 14, wherein the image processing system is physically separate from the array support and receives the first and second image data sets wirelessly.
• 17. An imaging array as in embodiment 14, wherein the image processing system derives the third image data set at least partly by creating a spatial relationship tensor that includes spatial relationship information between elements of the first and second image data sets and using the spatial relationship tensor to derive the third image data set.
• 18. An imaging array as in embodiment 17, wherein the image processing system derives the third image data set at least partly by using the spatial relationship tensor to combine together the first and second portions of the light field.
• 19. An imaging block as inembodiment 1, further comprising one or more processors carried by the support and configured to generate a light field representation based on pixel data acquired from the sensors.
• 20. An imaging block as inembodiment 1, wherein one or more of the lenses are removably replaceable with lenses having different optical characteristics.
• 21. An imaging block as inembodiment 1, wherein each imaging element has a primary optical axis, and most of the primary optical axes are substantially parallel.
• 22. An imaging block as inembodiment 1, wherein each imaging element has a primary optical axis, and at least two of the primary optical axes diverge in a direction leading away from the sensor.
• 23. An imaging block as inembodiment 1, wherein each imaging element has a primary optical axis, and at least two of the primary optical axes converge in a direction leading away from the sensor.
• 24. An imaging block as inembodiment 1, wherein each imaging element has a primary optical axis, and at least a first primary optical axis is angularly adjustable with respect to at least a second primary optical axis.
• 25. An imaging block as inembodiment 1, comprising at least 8 imaging elements.
• 26. An imaging block as in embodiment 25, wherein the imaging elements are arranged in two rows of 4.
• 27. An imaging block as inembodiment 1, comprising at least 16 imaging elements.
• 28. An imaging block as in embodiment 27, wherein the imaging elements are arranged in a 4×4 grid.
• 29. An imaging block as inembodiment 1, wherein at least one sensor is no larger than about 5 mm×5 mm.
• 30. An imaging block as inembodiment 1, further comprising an FPGA chip carried by the support.
• 31. An imaging array as inembodiment 1, wherein at least some of the sensors have one or more of different sizes, different resolutions, or different sensitivities.
• 32. An imaging array comprising an array support, and at least two imaging blocks of any of embodiments 19 through 31 carried by the array support.
Example Method of Compiling an Image Data Set
• 1. A method of compiling an image data set, comprising:
• • obtaining a first image data set representative of a first portion of a light field;
  • obtaining a second image data set representative of a second portion of the light field;
  • with one or more processors, deriving a third image data set that is based on at least the first image data set, the second image data set, and information relating to a spatial relationship between the first and second portions of the light fields; and
  • storing the third image data set in one or more memory devices.
• 2. A method as inembodiment 1, wherein the first image data set is derived from pixel data acquired by a first group of at least two imaging elements and the second image data set is derived from pixel data acquired by a second group of at least two imaging elements, each imaging element comprising a sensor and a lens.
• 3. A method as inembodiment 1, further comprising processing the first image data set and the second image data set to determine the spatial relationship between the first portion of the light field and the second portion of the light field.
• 4. A method as inembodiment 1, wherein said deriving comprises accessing a spatial relationship tensor representing the spatial relationship information to perform a geometric transform on the first image data set and second image data set.
• 5. A method as inembodiment 2, wherein the at least two imaging elements in the first group are coplanar, the at least two imaging elements in the second group are coplanar, and the at least two imaging elements in the first group are non-coplanar with respect to the at least two imaging elements in the second group.
• 6. A method as inembodiment 1, wherein the first and second image data sets additionally represent the first portion and the second portion of the light field as a function of time.
• 7. A method as inembodiment 1, wherein the third image data set comprises light field information represented as a function of time.
• 8. A method as inembodiment 1, wherein the first portion of the light field and the second portion of the light field comprise regions of the light field which at least partially overlap, and wherein the third data set comprises light field information derived from data in both of the first and second data sets that corresponds to the region of the light field lying within the overlap.
• 9. A method as inembodiment 1, wherein the first portion of the light field and the second portion of the light field comprise regions of the light field which only partially overlap, and wherein the third data set comprises light field information derived from:
• • data in the first image data set that corresponds to a first portion of a scene; and
  • data in the second image data set that corresponds to a second portion of the scene that does not overlap with the first portion.
• 10. A method as inembodiment 1, wherein the third image data set comprises at least 4D light field information.
• 11. A method as inembodiment 1, wherein said deriving the third image data set comprises deriving the third image data set while maintaining the dimensionality of functions that represent the first and second portions of the light field.
• 12. A method as inembodiment 1, wherein the first and second image data sets respectively represent the first and second portions of the light field as functions having at least four input parameters.
• 13. A method as inembodiment 1, wherein the first and second image data sets respectively represent the first and second portions of the light field as functions having at least five input parameters.
• 14. A method as inembodiment 1, wherein the first and second image data sets respectively represent the first and second portions of the light field as functions that represent luminance as a function of a position in space and a pointing direction.
• 15. A method ofembodiment 1, wherein viewable images are extractable from the third image data set.
• 16. A method as in embodiment 15, wherein the viewable images comprise 2D images.
• 17. A method as in embodiment 15, wherein the viewable images comprise 3D images.
• 18. A method as inembodiment 1, wherein viewable motion video is extractable from the third image data set.
• 19. A method as inembodiment 1, further comprising accessing a fourth image data set representative of a third portion of the light field, wherein said deriving comprises deriving the third image data set based on at least the first image data set, the second image data set, the fourth image data set, and spatial relationships between the first, second, and third portions of the light field.
Example Stored Image Data Set
• 1. A memory device containing a dense field image data set, comprising:
• • a storage medium; and
  • a dense field image data set stored in the storage medium and derived by relating a first image data set representative of a first portion of a light field to a second image data set representative of a second portion of the light field, using information relating to a spatial relationship between the first and second light fields.
• 2. A computer-readable memory device as inembodiment 1, wherein the image data set comprises light field information represented as a function of time.
• 3. A computer-readable memory device as inembodiment 1, wherein the image data set comprises at least 4D light field information.
• 4. A computer-readable memory device as inembodiment 1, wherein the first image data set was derived from pixel data acquired by a first imaging block comprising a support and at least two imaging elements carried by the support, and the second image data set was derived from pixel data acquired by a second imaging block comprising a support and at least two imaging elements carried by the support, each imaging element comprising a sensor and a lens.
• 5. A machine comprising the computer-readable memory device ofembodiment 1 and one or more processors, the one or more processors configured to derive the dense field image data set.
• 6. A machine comprising the computer-readable memory device ofembodiment 1 and one or more processors, the one or more processors configured to extract viewable images from the dense field image data set.
• 7. A computer-readable memory device as inembodiment 6, wherein the viewable images comprise 2D images.
• 8. A computer-readable memory device as inembodiment 6, wherein the viewable images comprise 3D images.
• 9. A computer-readable memory device as inembodiment 6, wherein the viewable images comprise motion video.
• Example Combining Light Fields to Create Dense Field Image Set with an Enhanced Field of View
• 1. A method of creating a dense field image set, the method comprising:
• • acquiring pixel data from a plurality of imaging elements, each imaging element comprising a sensor and a lens;
  • generating a plurality of light field representations, each of the light field representations generated using pixel data from at least two of the imaging elements;
  • creating a spatial relationship matrix representative of spatial relationships among the light field representations; and
  • utilizing the spatial relationship matrix, combining the light field representations to create a dense field image set.
• 2. A method as inembodiment 1, wherein said combining the light field representations comprises stitching together light field representations having at least partially non-overlapping fields of view to create a dense field image set having a significantly wider field of view than the individual light field representations.
• 3. A method as inembodiment 2, wherein the lateral field of view of the dense field image set is greater than or equal to about 145 degrees.
• 4. A method as inembodiment 1, wherein the imaging elements are arranged on a common support.
• 5. A method as inembodiment 1, wherein at least some of the imaging elements are not coplanar.
• 6. A method as inembodiment 1, wherein each light field representation is generated using pixel data from an imaging block comprising at least two imaging elements.
• 7. A method as inembodiment 6, wherein at least some of the imaging blocks are not coplanar.
• 8. A method as inembodiment 1, wherein at least one of the imaging elements comprises a monochromatic filter, and wherein light passing through the monochromatic filter is detected by the sensor of the at least one imaging element.
• 9. A method as inembodiment 1, wherein at least half of the imaging elements comprise monochromatic filters.
• 10. A method as inembodiment 1, wherein substantially all of the imaging elements comprise monochromatic filters.
• Example Dense Field Imager Creating Light Fields from Multiple Tiles Using Pixel Correspondence Information
• 1. An imaging system, comprising:
• • a plurality of imaging blocks each comprising at least two imaging elements, each of the imaging elements comprising an image sensor and a lens;
  • a dense field image processor module configured to:
    • for each imaging block of at least two of the plurality of imaging blocks:
      • generate pixel correspondence information for the imaging block, the pixel correspondence information representative of spatial relationships between pixels in each imaging element of the imaging block and corresponding pixels in other imaging elements of the imaging block;
      • utilizing the correspondence information, generate a light field representation using pixel data acquired by the respective imaging block.
• 2. An imaging system as inembodiment 1, wherein the dense field image processor module is further configured to:
• • create a spatial relationship matrix representative of spatial relationships among the light field representations; and
  • utilizing the spatial relationship matrix, combine the light field representations to create a dense field image set.
• 3. An imaging system as inembodiment 1, wherein the imaging blocks are arranged on a common support.
• 4. An imaging system as inembodiment 1, wherein the imaging blocks are formed in a wafer.
• 5. An imaging system as inembodiment 2, wherein the imaging elements of each imaging block are coplanar with respect to one another.
• 6. An imaging system as inembodiment 2, wherein at least some of the imaging blocks are not coplanar with respect to other ones of the imaging blocks.
• 7. An imaging system as inembodiment 1, wherein at least some of the imaging blocks are arranged on physically separate supports.
Example Dense Filed Imager Capable of Combining or Aggregating Light Fields
• 1. A dense field imaging system, comprising:
• • a plurality of imaging blocks, each of the imaging blocks comprising at least two sensor/lens pairs, wherein at least some of the imaging blocks are substantially non-coplanar with each other and have at least partially non-overlapping fields of view;
  • a dense field image processor module configured to:
    • for each imaging block, generate a light field representation using pixel data acquired by the sensor/lens pairs;
    • generate a spatial relationship matrix representative of spatial relationships between the light field representations; and
    • using the spatial relationship matrix, combine the light field representations to create a dense field imaging set having a substantially wider field of view than the individual light field representations.
• 2. A dense field imager as inembodiment 1, wherein each of the imaging blocks comprises a wafer substrate on which the sensors are formed.
• 3. A dense field imager as inembodiment 1, further comprising a control configured to adjust the angular relationship between sensor/lens pairs.
• 4. A dense field imager as inembodiment 3, wherein the control comprises a knob.
• Example Dense Field Imaging System with Monochromatic Sensors
• 1. A dense field imaging system, the dense field imaging system comprising:
• • a plurality of imaging blocks arranged on a support, each of the imaging blocks comprising at least two imaging elements, each of the imaging elements comprising a sensor and lens, wherein at least some of the imaging elements comprise monochromatic filters; and
  • a dense field image processor module configured to:
    • for each imaging block, generate a light field representation using pixel data acquired by the imaging elements of the imaging block; and
    • combine the light field representations to create a dense field image set.
• 2. A dense field imaging system as inembodiment 1, wherein most of the sensors are monochromatic.
• 3. A dense field imaging system as inembodiment 1, wherein each of the sensors is monochromatic.
• 4. A dense field imaging system as inembodiment 1, wherein each of the imaging blocks includes a support comprising a wafer substrate, and wherein the sensors are formed on the wafer substrate.
• 5. A dense field imaging system as in embodiment 4, wherein the lenses comprise wafer level lenses.
Example Dense Field Imager
• 1. A dense field imaging system, comprising:
• • a plurality of imaging blocks, each of the imaging blocks comprising:
    • a support;
    • at least a first imaging element and a second imaging element carried by the support, each imaging element comprising a sensor and a lens;
    • a mechanical connector for mechanically connecting the imaging block into an array of imaging blocks; and
    • an electrical connector for electrically connecting the imaging block into an array of imaging blocks;
  • an image processor system configured to:
    • for each imaging block, generate a light field representation using pixel data acquired at least by the first imaging element and the second imaging element; and
    • aggregate the light field representations to create a dense field image data set having, in aggregate, a wider field of view than the individual light field representations.
• 2. A dense field imager as inembodiment 1, wherein each of the imaging blocks comprises a wafer substrate on which the sensors of the first and second imaging elements are formed.
• 3. A dense field imager as inembodiment 2, wherein the lenses of the first and second imaging elements are wafer-level lenses.
• 4. A dense field imager as inembodiment 1, further comprising a control configured to adjust the angular relationship between the supports of the plurality of imaging blocks.
• 5. A dense field imager as inembodiment 1, further comprising a control configured to adjust the angular relationship between the first and second imaging elements on at least one of the plurality of imaging blocks.
• 6. A dense field imager as inembodiment 1, wherein at least one of the sensors in a first imaging block is non-coplanar with at least one of the sensors in a second imaging block.
• 7. A dense field imager as inembodiment 1, wherein each imaging element of an imaging block has a primary optical axis, and the primary optical axis of at least one of the imaging elements in a first imaging block is substantially non-parallel with the primary optical axis of at least one of the imaging elements in a second imaging block.
• 8. A dense field imager as inembodiment 1, wherein a primary optical axis of at least one imaging element of a first imaging block is angularly adjustable with respect to a primary optical axis of at least one imaging element of a second imaging block.
• 9. A dense field imager as inembodiment 8, further comprising a user-actuatable control for achieving the angular adjustment.
• 10. A dense field imager as inembodiment 1, further comprising an array support configured to support the array of imaging blocks, wherein the mechanical connector of each of the plurality of imaging blocks is configured to secure the imaging block to the array support.
• 11. A dense field imager as inembodiment 10, wherein the image processor system is carried by the array support.
• 12. A dense field imager as inembodiment 10, wherein the image processing system is physically separate from the array support and receives data acquired by the plurality of imaging blocks through a wireless transmission.
• 13. A dense field imager as inembodiment 1, wherein each of the plurality of imaging blocks further comprises one or more processors carried by the support and configured to generate a light field representation based on pixel data acquired from the sensors of the first and second imaging elements.
• 14. A dense field imager as inembodiment 1, wherein each of the plurality of imaging blocks comprises at least 8 imaging elements, each imaging element comprising a sensor and a lens.
• 15. A dense field imager as inembodiment 1, wherein a plurality of the imaging elements of the plurality of imaging blocks comprises monochromatic filters.
• 16. A dense field imager as inembodiment 1, wherein each of the imaging elements of the plurality of imaging blocks comprises monochromatic filters.
• 17. An imaging system mounted on an aerial vehicle and configured to perform wide-angle persistent surveillance, the imaging system comprising the dense field imager as in any ofembodiments 1 through 16, wherein an output resolution of an output image is greater than any resolution of the imaging elements of the plurality of imaging blocks.
• 18. The imaging system of embodiment 17, wherein the resolution of the output image is at least 3× greater than the resolution of any individual image sensor.
• 19. The imaging system of embodiment 17, wherein the resolution of the output image is at least 3× greater than the resolution of any individual image sensor.
• 20. The imaging system of embodiment 17, wherein the resolution of the output image is at least 5× greater than the resolution of any individual image sensor.
• 21. The imaging system of embodiment 17, wherein the resolution of the output image is at least 50× greater than the resolution of any individual image sensor.
• 22. The imaging system of embodiment 17, wherein the resolution of the output image is between 3× and 100× greater than the resolution of any individual image sensor.
• 23. The imaging system of embodiment 17, wherein the resolution of the output image is at least 3× greater than the combined resolution of the image sensors.
• 24. The imaging system of embodiment 17, wherein the resolution of the output image is at least 5× greater than the combined resolution of the image sensors.
• 25. The imaging system of embodiment 17, wherein the resolution of the output image is at least 50× greater than the combined resolution of the image sensors.
• 26. The imaging system of embodiment 17, wherein the resolution of the output image is between 3× and 100× greater than the combined resolution of the image sensors.
• 27. The imaging system of embodiment 17, wherein the imaging system is mounted on a gimbal on the aerial vehicle.
• 28. The imaging system of embodiment 17, wherein the aerial vehicle is an unmanned aerial vehicle, an airplane, a helicopter, or a satellite.
• 29. The imaging system of embodiment 17, wherein a ground sample distance of each pixel in at least one of the sensors is between 0.5 m and 4 m, wherein the imaging system is configured to acquire image data of the ground from the aerial vehicle.
• 30. An imaging system incorporated into a mobile device, the imaging system comprising the dense field imager as in any ofembodiments 1 through 16, wherein the mobile device is a smartphone, cellular phone, PDA, tablet, or laptop.
• 31. The imaging system ofembodiment 30, the lenses comprising wafer-level optics and the sensors comprising a wafer-substrate.
• 32. A dense field imager as in any ofembodiments 1 through 16, wherein each of the plurality of imaging blocks has a field of view and the fields of view of adjacent imaging blocks substantially overlap.
• 33. A dense field imager as in embodiment 32, wherein the image processing system is further configured to enhance a parameter of an output image generated by the image processing system, wherein enhancement of the parameter is based at least in part on data acquired by pixels of the sensors, the pixels acquiring data within the substantially overlapping fields of view.
• 34. A dense field imager as in embodiment 33, wherein the image parameter is a brightness of at least a portion of the image, a dynamic range of at least a portion of the image, or a color of at least a portion of the image.
Example Portable Dense Field Camera
• 1. A portable dense field camera comprising:
• • a housing having a first side with a plurality of imaging blocks attached thereto, and a second side, opposite the first side, having a display; and
  • an image processing system configured to generate a light field representation from the image data acquired by each of the plurality of imaging blocks,
  • wherein the housing has a length between 6 in. and 36 in.,
  • wherein the housing has a width between 4 in. and about 36 in.,
  • wherein the housing has a depth between 0.5 in. and 5 in.
• 2. The portable dense field camera ofembodiment 1, further comprising a wireless data transmission system configured to transmit image data to an off-camera processing system.
• 3. The portable dense field camera ofembodiment 1, wherein the dense field imager has a greater effective output image resolution with a smaller depth of the camera housing relative to a standard camera employing a single imaging element having an equivalent effective output image resolution.
• 4. A portable dense field camera comprising:
• • a housing having a surface with a plurality of imaging blocks attached thereto; and
  • an image processing system configured to generate a light field representation from the image data acquired by each of the plurality of imaging blocks,
  • wherein the surface has a modifiable geometrical configuration wherein a modification to the surface geometry modifies a field of view of the portable dense field camera.
• 5. The portable dense field camera of embodiment 4, wherein the surface geometry can be modified by a user of the dense field camera.
• 6. The portable dense field camera of embodiment 5, further comprising a single mechanical element configured to adjust the surface geometry.
• 7. The portable dense field camera ofembodiment 6, wherein the single mechanical element is a knob, slider, or rod.
• 8. The portable dense field camera of embodiment 4, wherein the surface geometry can be modified to be planar, convex, or concave.
Terminology/Additional Embodiments
• Embodiments have been described in connection with the accompanying drawings. However, it should be understood that the figures are not drawn to scale. Distances, angles, etc. are merely illustrative and do not necessarily bear an exact relationship to actual dimensions and layout of the devices illustrated. In addition, the foregoing embodiments have been described at a level of detail to allow one of ordinary skill in the art to make and use the devices, systems, etc. described herein. A wide variety of variation is possible. Components, elements, and/or steps can be altered, added, removed, or rearranged. While certain embodiments have been explicitly described, other embodiments will become apparent to those of ordinary skill in the art based on this disclosure.
• Conditional language used herein, such as, among others, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or states. Thus, such conditional language is not generally intended to imply that features, elements and/or states are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or states are included or are to be performed in any particular embodiment.
• Depending on the embodiment, certain acts, events, or functions of any of the methods described herein can be performed in a different sequence, can be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the method). Moreover, in certain embodiments, acts or events can be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors or processor cores, rather than sequentially. In some embodiments, the algorithms disclosed herein can be implemented as routines stored in a memory device. Additionally, a processor can be configured to execute the routines. In some embodiments, custom circuitry may be used.
• The various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. The described functionality can be implemented in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosure.
• The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein can be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor can be a microprocessor, but in the alternative, the processor can be any conventional processor, controller, microcontroller, or state machine. A processor can also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• The blocks of the methods and algorithms described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of computer-readable storage medium known in the art. An exemplary storage medium is coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.
• While the above detailed description has shown, described, and pointed out novel features as applied to various embodiments, it will be understood that various omissions, substitutions, and changes in the form and details of the devices or algorithms illustrated can be made without departing from the spirit of the disclosure. As will be recognized, certain embodiments of the inventions described herein can be embodied within a form that does not provide all of the features and benefits set forth herein, as some features can be used or practiced separately from others. The scope of certain inventions disclosed herein is indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims (1)

What is claimed is:

1. A dense field imaging system, comprising:

a plurality of imaging blocks, each of the imaging blocks comprising:

a support;

at least a first imaging element and a second imaging element carried by the support, each imaging element comprising a sensor and a lens;

a mechanical connector for mechanically connecting the imaging block into an array of imaging blocks; and

an electrical connector for electrically connecting the imaging block into an array of imaging blocks;

an image processor system configured to:

for each imaging block, generate a light field representation using pixel data acquired at least by the first imaging element and the second imaging element; and

aggregate the light field representations to create a dense field image data set having, in aggregate, a wider field of view than the individual light field representations.

Images