Detailed Description
In order to make the technical problems, technical solutions and advantageous effects to be solved by the embodiments of the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and the embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
It will be understood that when an element is referred to as being "secured to" or "disposed on" another element, it can be directly on the other element or be indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or be indirectly connected to the other element. The connection may be for fixation or for circuit connection.
It is to be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," and the like are used in an orientation or positional relationship indicated in the drawings for convenience in describing the embodiments of the present invention and to simplify the description, and are not intended to indicate or imply that the referenced device or element must have a particular orientation, be constructed in a particular orientation, and be in any way limiting of the present invention.
Furthermore, the terms "first", "second" and "first" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of the present invention, "a plurality" means two or more unless specifically limited otherwise.
FIG. 1 is a schematic view of a temporal depth of flight camera according to one embodiment of the invention. Time flight depth camera 10 includesemission module 11,gather module 12 andprocessing circuit 13, whereinemission module 11 providesemission beam 30 toobject 20 in the space with the illumination space in the target space, at leastpartial emission beam 30 forms reflectedlight beam 40 afterobject 20 reflects,reflected light beam 40 is at least partly gathered by gatheringmodule 12,processing circuit 13 is connected withemission module 11 andgathering module 12 respectively,synchronous emission module 11 andgather module 12's trigger signal is sent out and is gathered the required time ofmodule 12 receipt by gathering byemission module 11 with the calculation beam, namely, the flight time t betweenemission beam 30 and thereflected light beam 40, furthermore, total light flying distance D of corresponding point on the object can be calculated by the following formula:
D=c·t(1)
where c is the speed of light.
Theemitting module 11 includes alight source 111, alight beam modulator 112, a light source driver (not shown), and the like. Thelight source 111 may be a light source such as a Light Emitting Diode (LED), an Edge Emitting Laser (EEL), a Vertical Cavity Surface Emitting Laser (VCSEL), or a light source array composed of a plurality of light sources, and light beams emitted by the light sources may be visible light, infrared light, ultraviolet light, or the like. Thelight source 111 emits a light beam outwards under the control of a light source driver (which may be further controlled by the processing circuit 13), for example, in one embodiment, thelight source 111 emits a pulsed light beam under the control at a certain frequency, which may be set according to a measurement distance, for example, 1MHz to 100MHz, which may be set in a range of several meters to several hundred meters, and may be used in Direct time of flight (TOF) measurement; in one embodiment, thelight source 111 emits light beams whose amplitude is modulated under control to emit pulsed light beams, square wave light beams, sine wave light beams, and the like, which can be used in Indirect time of flight (inductively TOF) measurements. It will be appreciated that thelight source 111 may be controlled to emit the associated light beam by means of a part of theprocessing circuitry 13 or a sub-circuit present independently of theprocessing circuitry 13, such as a pulse signal generator.
Thebeam modulator 112 receives the light beam from thelight source 111 and emits a spatially modulated light beam, such as a floodlight beam with a uniform intensity distribution or a patterned light beam with a non-uniform intensity distribution, to the outside. It is to be understood that the distribution uniformity is a relative concept and is not absolutely uniform, and generally the light beam intensity at the edge of the field of view is allowed to be slightly lower, and the intensity for the imaging area in the middle can be changed within a certain threshold, for example, the intensity change can be allowed to be not more than 15% or 10%. In some embodiments, thebeam modulator 112 is also configured to expand the received beam to expand the field of view.
Thecollecting module 12 includes animage sensor 121, alens unit 122, and may further include a filter (not shown in the figure), thelens unit 122 receives and images at least part of the spatially modulated light beam reflected by the object on at least part of theimage sensor 121, and the filter needs to select a narrow band filter matched with the wavelength of the light source to suppress background light noise in the rest of the wavelength bands. Theimage sensor 121 may be a Charge Coupled Device (CCD), Complementary Metal Oxide Semiconductor (CMOS), Avalanche Diode (AD), Single Photon Avalanche Diode (SPAD), etc., with an array size representing the resolution of the depth camera, such as 320x240, etc. Generally, a readout circuit (not shown in the figure) composed of one or more of a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC), and the like is also included in connection with theimage sensor 121.
Generally, theimage sensor 121 comprises at least one pixel, each pixel then comprising a plurality of taps (for storing and reading or discharging charge signals generated by incident photons under control of the respective electrodes), for example 3 taps, for reading charge signal data.
In some embodiments, the time-of-flight depth camera 10 may further include a driving circuit, a power supply, a color camera, an infrared camera, an IMU, and so on, which are not shown in the drawings, and a combination of these devices may realize more abundant functions, such as 3D texture modeling, infrared face recognition, SLAM, and so on. Time-of-flight depth camera 10 may be embedded in a cell phone, tablet, computer, or other electronic product.
Theprocessing circuit 13 may be a stand-alone dedicated circuit, such as a dedicated SOC chip, an FPGA chip, an ASIC chip, etc. including a CPU, a memory, a bus, etc., or may include a general-purpose processing circuit, such as when the depth camera is integrated into an intelligent terminal, such as a mobile phone, a television, a computer, etc., and the processing circuit in the terminal may be at least a part of theprocessing circuit 13. In some embodiments, theprocessing circuit 13 is configured to provide a modulation signal (emission signal) required when thelight source 111 emits laser light, and the light source emits a pulse light beam to the object to be measured under the control of the modulation signal; theprocessing circuit 13 further provides a demodulation signal (acquisition signal) of a tap in each pixel of theimage sensor 121, and the tap acquires a charge signal generated by a light beam including a reflected pulse light beam reflected back by the object under the control of the demodulation signal, and generally, there are some light beams such as background light, interference light, and the like in addition to the reflected pulse light beam reflected back by the object; theprocessing circuit 13 may also provide auxiliary monitoring signals such as temperature sensing, over-current, over-voltage protection, dropout protection, etc.; theprocessing circuit 13 may also be configured to store and perform corresponding processing on raw data acquired by each tap in theimage sensor 121 to obtain specific position information of the object to be measured. The functions of the modulation and demodulation method, control, processing, etc. performed by theprocessing circuit 13 will be described in detail in the embodiments of fig. 2-5, and for convenience of explanation, the PM-ietf modulation and demodulation method is taken as an example for explanation.
Fig. 2 is a schematic diagram of a time-of-flight depth camera optical signal emission and collection method according to a first embodiment of the present invention. Fig. 2 shows an exemplary timing diagram of a laser emission signal (modulation signal), a reception signal, and a collection signal (demodulation signal) within a single frame period T, where the meaning of each signal is: sp represents a pulsed emission signal of the light source, each pulsed emission signal representing a pulsed light beam; sr represents the reflected light signal of the pulsed light reflected by the object, each reflected light signal represents the corresponding pulsed light beam reflected by the object to be measured, and the reflected light signal has a certain delay on a time line (horizontal axis in the figure) relative to the pulse emission signal, and the delayed time t is the flight time of the pulsed light beam to be calculated; s1 represents a pulse pickup signal of a first tap of the pixel, S2 represents a pulse pickup signal of a second tap of the pixel, and S3 represents a pulse pickup signal of a third tap of the pixel, each pulse pickup signal representing that the tap picks up a charge signal (electrons) generated by the pixel in a period corresponding to the signal; the period of the pulse emission signal, the reflected light signal and the pulse acquisition signal of each tap is Tp, the pulse width is Th, and Tp is N multiplied by Th, wherein N is the number of taps participating in pixel electron collection.
The entire frame period T is divided into two periods Ta and Tb, where Ta denotes a period during which charge collection and storage is performed by each tap of the pixel, and Tb denotes a period during which the charge signal is read out. In the charge collection and storage period Ta, the collected signal pulse of the nth tap has a phase delay time of (n-1) × Th with respect to the laser emission signal pulse, respectively, and when the reflected light signal is reflected back to the pixel by the object, each tap collects electrons generated at the pixel during its pulse period. In the present embodiment, the first tap is triggered in synchronization with the laser emission signal, and when the reflected light signal is reflected back to the pixel by the object, the first tap, the second tap, and the third tap respectively and sequentially perform charge collection and storage to obtain the charge amounts q1, q2, and q3, respectively, so as to complete a pulse period Tp, which is 3Th for the case of 3 taps. A plurality of pulses are transmitted within a single frame period T and divided into a plurality of (X) pulse groups, each having a period Ts of a plurality of (Y) pulses, the period Ts of the pulse group being Y · Tp. In the embodiment shown in fig. 2, 9 pulses are transmitted in a single frame period T, the pulses are divided into 3 pulse groups, and each pulse group has 3 pulse periods Tp, that is, Y is 3, it is understood that, here, only by way of illustration, in practice, hundreds or even tens of thousands of pulses may be transmitted in a single frame period, the specific number of pulses may be set arbitrarily according to actual needs, and in addition, the number of pulses in different frame periods may be different.
The total charge amount collected and read out by each tap in the Tb period is the sum of the charge amounts corresponding to the optical signals collected by each tap for a plurality of times (9 times in fig. 2) in the whole frame period T, and the total charge amount of each tap in a single frame period can be represented as follows:
Qi=∑qi,i=1,2,3 (2)
the total charge amount in a single frame period of the first tap, the second tap and the third tap can be obtained as Q1, Q2 and Q3 according to the formula (2).
If the reflected light signal is collected by the first tap and the second tap (the first tap and the second tap will also collect the ambient light signal), the third tap is used to collect the ambient light signal, so that based on the total charge collected by each tap, the processing circuit can calculate the distance of the object according to the following formula:
it is to be understood that the modulation and demodulation method and the distance calculation method described in equation (3) are only examples, and any other modulation and demodulation method and distance calculation method may be used in the present application.
When a plurality of time flight depth cameras work together, when an optical signal emitted from one of the time flight depth cameras is reflected by an object to be measured or directly enters an image sensor in the other depth camera, interference is generated on measurement, and measurement accuracy is influenced. In order to reduce interference, a plurality of different time intervals Tg are provided between the individual pulse groups in the present embodiment. Due to the addition of the time interval Tg, the external interference signal is difficult to continuously generate a continuously fixed influence on the number of electrons collected by each tap, when the group X is large enough, the influence of the external interference signal on each tap can be considered to be almost consistent, and similar to background signals (such as ambient light and the influence of dark current noise on the iToF device), the influence of optical signals in other depth cameras on the measurement accuracy can be effectively eliminated by some background signal elimination means (such as subtraction between tap signals).
In one embodiment, the number of pulses in each pulse group may be the same or may be set different. For different situations, the influence of external optical signals on the acquisition of each tap can be further reduced, so that the interference influence is better reduced.
In one embodiment, the time intervals Tg between groups of pulses are randomly set.
In one embodiment, the time intervals Tg between the groups of pulses are arranged pseudo-randomly, for example, an array of all time intervals Tg may be set in advance, and the values in the array are set to fixed values to form pseudo-random groups.
In one embodiment, the pulse periods in the respective pulse groups may be the same or set different. For different situations, the influence of external optical signals on the acquisition of each tap can be further reduced, so that the interference influence is better reduced.
The external light signal (including interference light, ambient light) and other noise sources may continuously excite the pixel to generate electrons in the intervening time interval Tg, and most of the generated electrons will enter the tap that is turned on first after the intervening time, for example, the first tap in the embodiment shown in fig. 2, which may cause the signal of the first tap to be higher, thereby causing measurement error. To reduce this effect, the present invention provides several solutions, see in detail fig. 3-5.
FIG. 3 is a schematic diagram of a method for transmitting and collecting optical signals of a time-of-flight depth camera according to a second embodiment of the present invention. In order to prevent the charge signal generated in each time interval from entering the first tap, in this embodiment, the demodulation signal is set so that each tap collects the generated charge signal in each time interval Tg, thereby avoiding a large error caused by entering a single tap. To reduce the error, the electrons generated during these time intervals should be split into the various taps as evenly as possible.
In one embodiment, each tap may be turned on in turn to acquire a respective time interval, as shown in FIG. 3. It can be understood that due to the difference of the time intervals, when the number of the pulse intervals is large enough, the sequential turning-on may obtain a better effect, but if the difference of the time intervals is large and/or the number of the pulse intervals is not large, the sequential turning-on effect may not ensure that the number of electrons in each tap is equally divided as much as possible, so that all the time intervals in the frame period may be processed in advance, and the taps to be turned on in each time interval are calculated and allocated in advance to ensure that the acquisition time of each tap is as same as possible.
It is to be understood that, for the first tap, since the time interval is connected to the time of the collected signal in the next pulse period, if the first tap starts the collected signal in the time interval, it can be considered to widen the collected signal time of the first tap in the next pulse period to collect the optical signal in the time interval and the optical signal in one pulse period Th. For the third tap, since the time interval is connected to the time of the acquisition signal in the previous pulse period, if the third tap starts the acquisition signal in the time interval, it can be considered that the acquisition signal time of the third tap in the previous pulse period is widened to acquire the optical signal in one pulse period Th and the time interval.
FIG. 4 is a schematic diagram of a method for transmitting and collecting optical signals of a time-of-flight depth camera according to a third embodiment of the present invention. As can be seen from fig. 4, this embodiment can be regarded as a further improvement of the embodiment shown in fig. 3, except that when each tap collects a signal within a time interval, the width of the collected signal is smaller than the time interval, so that the two collections can be distinguished from each other in terms of time, thereby avoiding the problem of difficulty in modulation of the processing circuit caused by the superposition of the two collected signals.
The sharing idea adopted in the embodiments shown in fig. 3 and 4 is difficult to fundamentally solve the error problem caused by electrons generated in a time interval. The present application will provide another solution.
FIG. 5 is a diagram illustrating a method for transmitting and collecting optical signals of a time-of-flight depth camera according to a fourth embodiment of the present invention. Compared with the previous embodiments, the difference is that the present embodiment further includes a fourth tap, and the fourth tap may be a tap for storing electrons or a tap for discharging (drain) electrons. The fourth tap is turned on for each time interval, for storing electrons generated in each time interval if it is a store electronic tap, and for discharging electrons generated in each time interval if it is a discharge electronic tap. Because the fourth tap collects or discharges electrons generated in the period of time in the time interval, the electrons are prevented from entering the tap in the next frame period, and the measurement accuracy is improved. It will be appreciated that the fourth tap may be turned on for the same time interval or less than the time interval.
2-4, the anti-interference modulation and demodulation method based on 3 taps is described, and fig. 5 is that the optimized anti-interference modulation and demodulation method based on 4 taps is realized by adding 1 tap on the basis of 3 taps. It will be appreciated that in some embodiments, the interference rejection modem approach shown in fig. 2-4 is equally applicable for pixels having 4 or more taps, only one or more of which need not be involved.
In some embodiments, for a modulation and demodulation method based on 4 or more taps (by default, four taps are all involved), the above-mentioned interference rejection method by grouping pulses and setting different time intervals between groups is also applicable in the embodiments. In addition, the optimized anti-interference modulation and demodulation method realized by adding one tap on the basis of the method is also applicable to the method similar to the method shown in fig. 5.
In the above embodiments, an indirect time-of-flight method (PM-ietf) based on a pulsed light signal is described as an example. It will be appreciated that the above method is equally applicable to an indirect time-of-flight method (CW-ietf) based on a continuous wave signal (such as a square wave or a sine wave), or a direct time-of-flight method (PM-dTOF or CW-dTOF) based on a pulsed signal or a continuous wave signal. For the continuous wave signal, a continuous signal of a single period Tp may be equivalent to a single pulse signal in each of the above embodiments, and therefore the pulse signal mentioned in the present application is understood to be a pulse signal in a broad sense, which may be a high-frequency pulse signal with an extremely short pulse width (Tp — 3Th in fig. 2, duty ratio 33%), a square wave continuous signal with a long pulse width (Tp — 2Th, duty ratio 50%), a sine wave signal with the same period as the pulse period Tp, or the like.
The invention has the advantages that the interference problem among multiple cameras is effectively reduced by grouping the emitted light signals and setting different time intervals among groups, thereby ensuring that a single camera can be suitable for more occasions. Compared with the traditional anti-interference methods of setting different frequencies, frequency conversion or frequency sweeping and the like, the method is easier to realize and more effective.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several equivalent substitutions or obvious modifications can be made without departing from the spirit of the invention, and all the properties or uses are considered to be within the scope of the invention.