技术领域technical field
本发明涉及光学测量技术领域，尤其涉及一种深度测量装置及距离测量方法。The invention relates to the technical field of optical measurement, in particular to a depth measurement device and a distance measurement method.
背景技术Background technique
ToF的全称是Time-of-Flight，即飞行时间，ToF测距技术是一种通过测量光脉冲在发射/接收装置和目标物体间的往返飞行时间来实现精确测距的技术。在ToF技术中直接对光飞行时间进行测量的技术被称为dToF(direct-TOF)；对发射光信号进行周期性调制，通过对反射光信号相对于发射光信号的相位延迟进行测量，再由相位延迟对飞行时间进行计算的测量技术被成为iToF(Indirect-TOF)技术。按照调制解调类型方式的不同可以分为连续波(Continuous Wave，CW)调制解调方法和脉冲调制(Pulse Modulated，PM)调制解调方法。The full name of ToF is Time-of-Flight, that is, time of flight. ToF ranging technology is a technology that achieves precise ranging by measuring the round-trip flight time of light pulses between the transmitting/receiving device and the target object. In ToF technology, the technology of directly measuring the time of flight of light is called dToF (direct-TOF); the emitted optical signal is periodically modulated, and the phase delay of the reflected optical signal relative to the emitted optical signal is measured, and then by The measurement technology for calculating the time of flight by phase delay is called iToF (Indirect-TOF) technology. According to different modulation and demodulation methods, it can be divided into a continuous wave (Continuous Wave, CW) modulation and demodulation method and a pulse modulation (Pulse Modulated, PM) modulation and demodulation method.
结构光测距技术则向空间物体发射结构光光束，其次采集被物体调制及反射后的结构光光束所形成的结构光图案，最后利用三角法进行深度计算以获取物体的深度数据。常用的结构光图案有不规则斑点图案、条纹图案、相移图案等。The structured light ranging technology emits a structured light beam to a space object, and then collects the structured light pattern formed by the structured light beam modulated and reflected by the object, and finally uses the triangulation method for depth calculation to obtain the depth data of the object. Commonly used structured light patterns include irregular spot patterns, stripe patterns, phase shift patterns, etc.
相比而言，TOF技术无需进行复杂的图像处理计算(如结构光图像匹配计算)，在中远距时能保持相对较高的测量精度；而结构光技术则在近距测量时具有非常高的精度，但随着距离增加精度会至少呈现线性增长。另外，结构光技术由于是通过采集反映反射光强度的图像来进行计算深度的，因此难免会受到环境光对光强度的影响，相对而言TOF则在环境光抗干扰方面要优于结构光技术。In contrast, TOF technology does not require complex image processing calculations (such as structured light image matching calculations), and can maintain relatively high measurement accuracy at medium and long distances; while structured light technology has very high accuracy at short-range measurements. Accuracy, but accuracy increases at least linearly with distance. In addition, since structured light technology calculates depth by collecting images reflecting the intensity of reflected light, it will inevitably be affected by ambient light on light intensity. Relatively speaking, TOF is superior to structured light technology in terms of ambient light anti-interference .
发明内容Contents of the invention
本发明为了解决现有的问题，提供一种深度测量装置及距离测量方法。In order to solve the existing problems, the present invention provides a depth measuring device and a distance measuring method.
为了解决上述问题中的至少一个，本发明采用的技术方案如下所述：In order to solve at least one of the above-mentioned problems, the technical solution adopted in the present invention is as follows:
深度测量装置，包括：光发射模组，包含光源及图案化光学元件，所述光源用于发射时序上振幅被调制的光束，所述图案化光学元件用于接收所述光束后向目标物体发射结构光光束；TOF图像传感器，包含至少一个像素，所述像素用于接收由所述目标物体反射的结构光光束并形成电信号；控制和处理电路，接收所述电信号并计算出所述反射的结构光光束的强度信息以形成结构光图案，以及，利用所述结构光图案计算所述目标物体的深度图像。The depth measurement device includes: a light emitting module, including a light source and a patterned optical element, the light source is used to emit a light beam whose amplitude is modulated in time sequence, and the patterned optical element is used to receive the light beam and emit it to the target object Structured light beam; TOF image sensor, including at least one pixel, the pixel is used to receive the structured light beam reflected by the target object and form an electrical signal; control and processing circuit, receive the electrical signal and calculate the reflection The intensity information of the structured light beam is used to form a structured light pattern, and the depth image of the target object is calculated by using the structured light pattern.
在本发明的一种实施例中，所述结构光光束包含不规则斑点图案光束、条纹图案光束、二维编码图案化光束；所述不规则斑点图案光束中每个斑点对应的光束的振幅在时序上以连续波、方波或脉冲的方式被调制。In an embodiment of the present invention, the structured light beam includes an irregular spot pattern beam, a stripe pattern beam, and a two-dimensionally coded patterned beam; the amplitude of the beam corresponding to each spot in the irregular spot pattern beam is Timing is modulated in the form of continuous wave, square wave or pulse.
在本发明的又一种实施例中，所述像素包括至少3个抽头，所述抽头用于在单个帧周期内分别采集由包含所述目标物体反射的结构光光束所产生的电信号。所述光发射模组发射时序上振幅被正弦波调制的光束，所述TOF图像传感器的各个像素至少包括4个抽头，分别用于在单个帧周期内分别采集4次光信号并转换成所述电信号C1、C2、C3和C4，4次采集的时间以及间隔相同；采用加权平均的方式计算所述结构光光束的强度B，具体如下：或，采用加入环境光消除因子的方式计算所述结构光光束的强度B，具体如下：所述光发射模组发射时序上振幅被脉冲调制的光束，所述TOF图像传感器的各个像素至少包括3个抽头，分别用于在单个帧周期内分别采集3次光信号并转换成电信号C1、C2和C3，第一抽头的触发时间与光脉冲的发射时间同步，且触发时长与脉冲的宽度相同，第二抽头、第三抽头在所述第一抽头结束时相继被触发，触发时长与所述第一抽头相同；采用加权平均的方式计算所述结构光光束的强度B，具体如下：或，采用加入环境光消除因子的方式计算所述结构光光束的强度B，具体如下：B＝C1+C2-2C3。In yet another embodiment of the present invention, the pixel includes at least 3 taps, and the taps are used to respectively collect electrical signals generated by structured light beams including the reflection of the target object within a single frame period. The light emitting module emits a light beam whose amplitude is modulated by a sine wave in time sequence, and each pixel of the TOF image sensor includes at least 4 taps, which are respectively used to collect 4 optical signals in a single frame period and convert them into the The electrical signals C1, C2, C3, and C4 have the same time and interval for four acquisitions; the intensity B of the structured light beam is calculated by weighted average, as follows: Or, calculate the intensity B of the structured light beam by adding an ambient light elimination factor, as follows: The light emitting module emits a light beam whose amplitude is pulse-modulated in time sequence, and each pixel of the TOF image sensor includes at least 3 taps, which are respectively used to collect 3 optical signals in a single frame period and convert them into electrical signals C1 . The first tap is the same; the intensity B of the structured light beam is calculated by weighted average, as follows: Alternatively, the intensity B of the structured light beam is calculated by adding an ambient light elimination factor, specifically as follows: B=C1 +C2 −2C3 .
本发明还提供一种距离测量方法，包括：利用光源发射时序上振幅被调制的光束，所述光束被图案化光学元件接收后形成结构光光束向目标物体发射；利用包含至少一个像素的TOF图像传感器的所述像素接收由所述目标物体反射的结构光光束并形成电信号；接收所述电信号并计算出所述反射的结构光光束的强度信息以形成结构光图案，以及，利用所述结构光图案计算所述目标物体的深度图像。The present invention also provides a distance measurement method, including: using a light source to emit a light beam whose amplitude is modulated in time sequence, and the light beam is received by a patterned optical element to form a structured light beam and emit it to a target object; using a TOF image containing at least one pixel The pixel of the sensor receives the structured light beam reflected by the target object and forms an electrical signal; receives the electrical signal and calculates the intensity information of the reflected structured light beam to form a structured light pattern, and uses the The structured light pattern calculates a depth image of the target object.
在本发明的一种实施例中，所述结构光光束包含不规则斑点图案光束、条纹图案光束、二维编码图案化光束；所述不规则斑点图案光束中每个斑点对应的光束的振幅在时序上以连续波、方波或脉冲的方式被调制。In an embodiment of the present invention, the structured light beam includes an irregular spot pattern beam, a stripe pattern beam, and a two-dimensionally coded patterned beam; the amplitude of the beam corresponding to each spot in the irregular spot pattern beam is Timing is modulated in the form of continuous wave, square wave or pulse.
在本发明的另一种实施例中，所述像素包括至少3个抽头，所述抽头用于在单个帧周期内分别采集由包含所述目标物体反射的结构光光束所产生的电信号。所述结构光光束是时序上振幅被正弦波调制的光束，各个所述像素至少包括4个抽头，分别用于在单个帧周期内分别采集4次光信号并转换成所述电信号C1、C2、C3和C4，4次采集的时间以及间隔相同；采用加权平均的方式计算所述结构光光束的强度B，具体如下：或，采用加入环境光消除因子的方式计算所述结构光光束的强度B，具体如下：所述结构光光束是时序上振幅被脉冲调制的光束，各个所述像素至少包括3个抽头，分别用于在单个帧周期内分别采集3次光信号并转换成电信号C1、C2和C3，第一抽头的触发时间与光脉冲的发射时间同步，且触发时长与脉冲的宽度相同，第二抽头、第三抽头在所述第一抽头结束时相继被触发，触发时长与所述第一抽头相同；采用加权平均的方式计算所述结构光光束的强度B，具体如下：或，采用加入环境光消除因子的方式计算所述结构光光束的强度B，具体如下：B＝C1+C2-2C3。In another embodiment of the present invention, the pixel includes at least 3 taps, and the taps are used to separately collect electrical signals generated by structured light beams including the reflection of the target object within a single frame period. The structured light beam is a beam whose amplitude is modulated by a sine wave in time series, and each pixel includes at least 4 taps, which are used to collect 4 optical signals in a single frame period and convert them into electrical signals C1 and C2. , C3 and C4, the time and interval of the four acquisitions are the same; the intensity B of the structured light beam is calculated in a weighted average manner, as follows: Or, calculate the intensity B of the structured light beam by adding an ambient light elimination factor, as follows: The structured light beam is a beam whose amplitude is modulated in time series, and each of the pixels includes at least 3 taps, which are respectively used to collect 3 optical signals in a single frame period and convert them into electrical signals C1, C2 and C3, The trigger time of the first tap is synchronized with the emission time of the optical pulse, and the trigger duration is the same as the width of the pulse. The second tap and the third tap are triggered successively when the first tap ends, and the trigger duration is the same as the first tap. Same; the intensity B of the structured light beam is calculated by weighted average, as follows: Alternatively, the intensity B of the structured light beam is calculated by adding an ambient light elimination factor, specifically as follows: B=C1 +C2 −2C3 .
本发明的有益效果为：提供一种深度测量装置及距离测量方法，将TOF图像传感器与结构光光源相结合，发射端采用了振幅调制，在接收端采用了多抽头像素采集，从而可以使得本发明的装置和方法拥有比传统方案更多的功能，实现传统方案难以实现的抗环境光干扰的深度测量方法。The beneficial effects of the present invention are: providing a depth measuring device and a distance measuring method, combining a TOF image sensor with a structured light source, adopting amplitude modulation at the transmitting end, and adopting multi-tap pixel acquisition at the receiving end, so that the present invention can The invented device and method have more functions than the traditional scheme, and realize the depth measurement method which is difficult to realize by the traditional scheme and resist the interference of ambient light.
附图说明Description of drawings
图1是本发明实施例中深度测量装置的结构示意图。Fig. 1 is a schematic structural diagram of a depth measuring device in an embodiment of the present invention.
图2是本发明实施例中一种深度测量装置原理示意图。Fig. 2 is a schematic diagram of the principle of a depth measuring device in an embodiment of the present invention.
图3是本发明实施例中又一种深度测量装置原理示意图。Fig. 3 is a schematic diagram of the principle of another depth measuring device in an embodiment of the present invention.
图4是本发明实施例中一种融合TOF与结构光方案的深度测量装置的结构示意图。Fig. 4 is a schematic structural diagram of a depth measurement device that combines TOF and structured light solutions in an embodiment of the present invention.
图5是本发明实施例中又一种融合TOF与结构光方案的深度测量装置的结构示意图。Fig. 5 is a schematic structural diagram of another depth measurement device that integrates TOF and structured light solutions in an embodiment of the present invention.
图6是本发明实施例中结构光与TOF时序控制示意图。Fig. 6 is a schematic diagram of structured light and TOF timing control in an embodiment of the present invention.
图7是本发明实施例中控制与处理电路架构示意图。Fig. 7 is a schematic diagram of the structure of the control and processing circuit in the embodiment of the present invention.
图8是本发明实施例中一种距离测量方法示意图。Fig. 8 is a schematic diagram of a distance measurement method in an embodiment of the present invention.
图9是本发明实施例中另一种距离测量方法示意图。Fig. 9 is a schematic diagram of another distance measuring method in an embodiment of the present invention.
图10是本发明实施例中一种对所述电信号进行处理方法示意图。Fig. 10 is a schematic diagram of a method for processing the electrical signal in an embodiment of the present invention.
图11是本发明实施例中另一种对所述电信号进行处理方法示意图。Fig. 11 is a schematic diagram of another method for processing the electrical signal in an embodiment of the present invention.
其中，10-深度测量装置，11-光发射模组，12-采集模组，13-控制与处理电路，14-泛光光源模组，111-光源，112-图案化光学元件，121-TOF图像传感器，122-透镜单元，20-目标物体，30-发射光束，40-反射光束，201-斑点结构光图案化光束，202-光斑，40-深度测量装置，41-光发射模组，42-采集模组，43-控制与处理电路，44-泛光光源模组，51-像素，52-光斑，70-芯片，71-相位计算模块，72-标定模块，73-第一后处理模块，74-1-分路器，74-2-复用器，75-振幅计算模块，76-前处理模块，77-匹配模块，78-第二后处理模块，79-融合模块。Among them, 10-depth measurement device, 11-light emission module, 12-acquisition module, 13-control and processing circuit, 14-flood light source module, 111-light source, 112-patterned optical element, 121-TOF Image sensor, 122-lens unit, 20-target object, 30-emitted beam, 40-reflected beam, 201-spot structured light patterned beam, 202-spot, 40-depth measuring device, 41-light emitting module, 42 -acquisition module, 43-control and processing circuit, 44-flood light source module, 51-pixel, 52-spot, 70-chip, 71-phase calculation module, 72-calibration module, 73-first post-processing module , 74-1-splitter, 74-2-multiplexer, 75-amplitude calculation module, 76-pre-processing module, 77-matching module, 78-second post-processing module, 79-fusion module.
具体实施方式Detailed ways
为了使本发明实施例所要解决的技术问题、技术方案及有益效果更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅仅用以解释本发明，并不用于限定本发明。In order to make the technical problems, technical solutions and beneficial effects to be solved by the embodiments of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.
需要说明的是，当元件被称为“固定于”或“设置于”另一个元件，它可以直接在另一个元件上或者间接在该另一个元件上。当一个元件被称为是“连接于”另一个元件，它可以是直接连接到另一个元件或间接连接至该另一个元件上。另外，连接即可以是用于固定作用也可以是用于电路连通作用。It should be noted that when an element is referred to as being “fixed” or “disposed on” another element, it may be directly on the other element or be indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, the connection can be used for both fixing and circuit communication.
需要理解的是，术语“长度”、“宽度”、“上”、“下”、“前”、“后”、“左”、“右”、“竖直”、“水平”、“顶”、“底”“内”、“外”等指示的方位或位置关系为基于附图所示的方位或位置关系，仅是为了便于描述本发明实施例和简化描述，而不是指示或暗示所指的装置或元件必须具有特定的方位、以特定的方位构造和操作，因此不能理解为对本发明的限制。It is to be understood that the terms "length", "width", "top", "bottom", "front", "rear", "left", "right", "vertical", "horizontal", "top" , "bottom", "inner", "outer" and other indicated orientations or positional relationships are based on the orientations or positional relationships shown in the drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, rather than indicating or implying Any device or element must have a specific orientation, be constructed and operate in a specific orientation and therefore should not be construed as limiting the invention.
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括一个或者更多该特征。在本发明实施例的描述中，“多个”的含义是两个或两个以上，除非另有明确具体的限定。In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly specifying the quantity of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of these features. In the description of the embodiments of the present invention, "plurality" means two or more, unless otherwise specifically defined.
图1是根据本发明一个实施例的深度测量装置。深度测量装置10包括光发射模组11、采集模组12以及控制与处理电路13，其中光发射模组11用于向目标物体20发射发射光束30，发射光束30是在时序上振幅被调制的结构光光束，该结构光光束至目标空间中以照明空间中的目标物体20，至少部分发射光束30经目标物体20反射后形成反射光束40，反射光束40的至少部分被采集模组12采集，控制与处理电路13分别与光发射模组11以及采集模组12连接，以控制光束的发射与采集，同时接收来自采集模组12所采集到的电信号，并对该电信号进行计算以获取目标物体的深度信息。Fig. 1 is a depth measurement device according to one embodiment of the present invention. The depth measurement device 10 includes a light emitting module 11, an acquisition module 12, and a control and processing circuit 13, wherein the light emitting module 11 is used to emit a light beam 30 to the target object 20, and the light beam 30 is modulated in amplitude in time sequence A structured light beam, the structured light beam enters the target space to illuminate the target object 20 in the space, at least part of the emitted beam 30 is reflected by the target object 20 to form a reflected beam 40, at least part of the reflected beam 40 is collected by the acquisition module 12, The control and processing circuit 13 is respectively connected with the light emission module 11 and the collection module 12 to control the emission and collection of the light beam, and simultaneously receives the electrical signal collected by the collection module 12, and calculates the electrical signal to obtain Depth information of the target object.
光发射模组11包括光源111、图案化光学元件112以及光源驱动器(图中未示出)等。光源111可以是发光二极管(LED)、边发射激光器(EEL)、垂直腔面发射激光器(VCSEL)等光源，也可以是多个光源组成的光源阵列，光源所发射的光束可以是可见光、红外光、紫外光等。光源111在光源驱动器(其可以进一步被控制和处理电路13控制)的控制下以一定的时序振幅被调制后向外发射光束，比如在一个实施例中，光源111在控制下以一定的频率发射脉冲调制光束、方波调制光束、正弦波调制光束等光束。可以理解的是，可以利用控制和处理电路13中的一部分或者独立于控制和处理电路13存在的子电路来控制光源111发射相关的光束，比如脉冲信号发生器。The light emitting module 11 includes a light source 111 , a patterned optical element 112 , and a light source driver (not shown in the figure). The light source 111 may be a light emitting diode (LED), an edge-emitting laser (EEL), a vertical cavity surface-emitting laser (VCSEL), or a light source array composed of multiple light sources. The light beam emitted by the light source may be visible light, infrared light, etc. , ultraviolet light, etc. The light source 111 is modulated with a certain timing amplitude under the control of the light source driver (which can be further controlled by the control and processing circuit 13) and then emits a light beam outwards. For example, in one embodiment, the light source 111 emits a light beam at a certain frequency under control. Pulse modulated beam, square wave modulated beam, sine wave modulated beam and other beams. It can be understood that a part of the control and processing circuit 13 or a sub-circuit existing independently of the control and processing circuit 13 can be used to control the light source 111 to emit relevant light beams, such as a pulse signal generator.
图案化光学元件112接收来自光源111的光束，并向外发射结构光光束，比如不规则排列的斑点图案化光束、条纹图案光束、二维编码图案化光束等。在一些实施例中，图案化光学元件112还用于将接收到的光束进行扩束，以扩大视场角。可以理解的是，通过图案化光学元件112调制后的光束其振幅依旧是以一定的时序被调制，即入射的正弦波调制光束，出射的依旧是正弦波调制光束。在本发明的一种实施例中，不规则斑点图案光束中每个斑点对应的光束的振幅在时序上以连续波、方波或脉冲的方式被调制。The patterned optical element 112 receives the light beam from the light source 111 and emits a structured light beam, such as irregularly arranged spot patterned light beams, stripe patterned light beams, two-dimensional coded patterned light beams, and the like. In some embodiments, the patterned optical element 112 is also used to expand the received light beam to expand the viewing angle. It can be understood that the amplitude of the light beam modulated by the patterned optical element 112 is still modulated in a certain time sequence, that is, the incident sine wave modulated light beam is still the outgoing sine wave modulated light beam. In an embodiment of the present invention, the amplitude of the light beam corresponding to each spot in the irregular spot pattern light beam is modulated in a continuous wave, square wave or pulse manner in time sequence.
采集模组12包括TOF图像传感器121、透镜单元122，还可以包含滤光片(图中未示出)，透镜单元122接收并将由目标物体反射回的至少部分结构光光束并成像在至少部分TOF图像传感器121上，滤光片需选择与光源波长相匹配的窄带滤光片，用于抑制其余波段的背景光噪声。TOF图像传感器121可以是电荷耦合元件(CCD)、互补金属氧化物半导体(CMOS)、雪崩二极管(AD)、单光子雪崩二极管(SPAD)等组成的图像传感器，阵列大小代表着该深度相机的分辨率，比如320x240等。一般地，与图像传感器121连接的还包括由信号放大器、时数转换器(TDC)、模数转换器(ADC)等器件中的一种或多种组成的读出电路(图中未示出)。The acquisition module 12 includes a TOF image sensor 121, a lens unit 122, and may also include a filter (not shown in the figure). The lens unit 122 receives and reflects at least part of the structured light beam reflected by the target object and images it on at least part of the TOF On the image sensor 121 , the filter needs to select a narrow-band filter that matches the wavelength of the light source to suppress background light noise in other bands. The TOF image sensor 121 can be an image sensor composed of a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), an avalanche diode (AD), a single photon avalanche diode (SPAD), etc., and the size of the array represents the resolution of the depth camera. rate, such as 320x240, etc. Generally, the image sensor 121 is connected to a readout circuit (not shown in the figure) composed of one or more of a signal amplifier, a time-to-digital converter (TDC), an analog-to-digital converter (ADC) and other devices. ).
一般地，TOF图像传感器121包括至少一个像素，与传统的仅用于拍照的图像传感器相比，这里每个像素则包含两个以上的抽头(tap，用于在相应电极的控制下存储并读取或者排出由入射光子产生的电荷信号)，比如包括3个抽头，在单个帧周期(或单次曝光时间内)内以一定的次序依次切换抽头以采集相应的光子，以用于接收光信号并转换成电信号。Generally, the TOF image sensor 121 includes at least one pixel. Compared with the traditional image sensor that is only used for taking pictures, each pixel here includes more than two taps (taps, used to store and read under the control of corresponding electrodes). Take or discharge the charge signal generated by the incident photon), for example, including 3 taps, switch the taps in a certain order in a single frame period (or a single exposure time) to collect the corresponding photons for receiving optical signals and convert it into an electrical signal.
控制与处理电路13可以是独立的专用电路，比如包含CPU、存储器、总线等组成的专用SOC芯片、FPGA芯片、ASIC芯片等等，也可以包含通用处理电路，比如当该深度相机被集成到如手机、电视、电脑等智能终端中去，终端中的处理电路可以作为该控制与处理电路13的至少一部分。The control and processing circuit 13 can be an independent dedicated circuit, such as a dedicated SOC chip, FPGA chip, ASIC chip, etc. that include a CPU, memory, bus, etc., or a general processing circuit, such as when the depth camera is integrated into a In smart terminals such as mobile phones, televisions, and computers, the processing circuit in the terminal can serve as at least a part of the control and processing circuit 13 .
控制与处理电路13用于提供光源111(或泛光光源模组14中的光源等)发射激光时所需的调制信号(发射信号)，光源在调制信号的控制下向目标物体发射光束。比如在一个实施例中，调制信号为连续波信号如正弦波信号，光源在该正弦波信号的调制下振幅在时序上以正弦波形式发射；在一个实施例中，调制信号为方波信号或脉冲信号，光源在该调制信号的调制下振幅被时序上调制以产生方波信号或者脉冲信号向外发射。The control and processing circuit 13 is used to provide the modulation signal (emission signal) required for the light source 111 (or the light source in the flood light source module 14, etc.) to emit laser light, and the light source emits a beam to the target object under the control of the modulation signal. For example, in one embodiment, the modulation signal is a continuous wave signal such as a sine wave signal, and the amplitude of the light source is emitted in the form of a sine wave in time sequence under the modulation of the sine wave signal; in one embodiment, the modulation signal is a square wave signal or A pulse signal, under the modulation of the modulation signal, the amplitude of the light source is time-sequentially modulated to generate a square wave signal or a pulse signal is emitted outward.
此外，控制与处理电路13还提供TOF图像传感器121各像素中各抽头的解调信号(采集信号)，抽头在解调信号的控制下采集由包含目标物体反射回的反射的光束所产生的电信号。该电信号与反射光束的强度相关，控制与处理电路13随后对该电信号进行处理并计算出反映反射光束强度的强度信息以形成结构光图案，最后基于该结构光图案进行像匹配计算、三角法计算等计算以获得目标物体的深度图像。In addition, the control and processing circuit 13 also provides the demodulation signal (acquisition signal) of each tap in each pixel of the TOF image sensor 121. Signal. The electrical signal is related to the intensity of the reflected beam, and the control and processing circuit 13 then processes the electrical signal and calculates the intensity information reflecting the intensity of the reflected beam to form a structured light pattern, and finally performs image matching calculation and triangulation based on the structured light pattern. The depth image of the target object is obtained by calculation such as method calculation.
在一些实施例中，深度测量装置10还包括泛光光源模组14，用于向外发射时序上振幅被调制的泛光光束。控制与处理电路13可以同步对泛光光源模组14与TOF图像传感器121进行调制与解调，以实现传统TOF测距方法以获取目标的深度图像，即通过对泛光光束由发射到被接收所需要的时间来进行深度值计算的方式。In some embodiments, the depth measurement device 10 further includes a floodlight light source module 14 for emitting a floodlight beam whose amplitude is modulated in timing. The control and processing circuit 13 can synchronously modulate and demodulate the floodlight light source module 14 and the TOF image sensor 121, so as to realize the traditional TOF ranging method to obtain the depth image of the target, that is, by adjusting the floodlight beam from being emitted to being received The way it takes time to perform depth value calculations.
在一些实施例中，深度测量装置10还可以包括驱动电路、电源、彩色相机、红外相机、IMU等器件，在图中并未示出，与这些器件的组合可以实现更加丰富的功能，比如3D纹理建模、红外人脸识别、SLAM等功能。深度测量装置10可以被嵌入到手机、平板电脑、计算机等电子产品中。In some embodiments, the depth measurement device 10 may also include devices such as a drive circuit, a power supply, a color camera, an infrared camera, and an IMU, which are not shown in the figure, and the combination of these devices can realize more abundant functions, such as 3D Texture modeling, infrared face recognition, SLAM and other functions. The depth measurement device 10 can be embedded in electronic products such as mobile phones, tablet computers, and computers.
图2是根据本发明第一实施例的深度测量装置原理示意图。在控制与处理电路13的控制下，光发射模组11向目标物体发射振幅被正弦调制的结构光光束30，在本实施例中结构光光束30为不规则斑点结构光，即由多个光斑202以不规则排列形成的斑点结构光图案化光束201，各个光斑202的振幅在时序上被正弦调制。采集模组12中的TOF图像传感器的各个像素至少包括4个抽头，分别用于在单个帧周期内分别采集4次光信号并转换成电信号C1、C2、C3和C4，4次采集的时间以及间隔相同。Fig. 2 is a schematic diagram of the principle of the depth measuring device according to the first embodiment of the present invention. Under the control of the control and processing circuit 13, the light emitting module 11 emits a structured light beam 30 whose amplitude is sinusoidally modulated to the target object. In this embodiment, the structured light beam 30 is irregular spot structured light, that is, a plurality of spots 202 irregularly arranged spot structured light patterned light beam 201, the amplitude of each light spot 202 is sinusoidally modulated in time sequence. Each pixel of the TOF image sensor in the collection module 12 includes at least 4 taps, which are respectively used to collect 4 optical signals in a single frame period and convert them into electrical signals C1, C2, C3 and C4. The time for the 4 collections and the same interval.
控制与处理电路13接收电信号C1、C2、C3和C4计算出结构光光束的强度信息。在一个实施例中的，强度信息根据下式计算：The control and processing circuit 13 receives the electrical signals C1, C2, C3 and C4 to calculate the intensity information of the structured light beam. In one embodiment, the intensity information is calculated according to the following formula:
可以理解的是，这一计算方式类似于传统未被振幅调制的结构光光束，即在单个帧周期内通过单抽头连续积分的方式获取光信号的强度信息，只不过这里进行了平均。在获取所有像素的强度信息之后，就可以形成结构光图案了，最后再利用结构光图案进行匹配计算以获取视差以及根据视差计算出深度图像。It can be understood that this calculation method is similar to the traditional unmodulated structured light beam, that is, the intensity information of the optical signal is obtained through single-tap continuous integration within a single frame period, except that it is averaged here. After obtaining the intensity information of all pixels, the structured light pattern can be formed, and finally the structured light pattern is used for matching calculation to obtain the parallax and calculate the depth image according to the parallax.
对于存在环境光信号时，这一光束强度计算方式与传统方式一样，均难以进行消除，从而导致最终的结构光图案信噪比较低。因此，在一个实施例，强度信息将根据下式计算：When there is an ambient light signal, this beam intensity calculation method is the same as the traditional method, which is difficult to eliminate, resulting in a low signal-to-noise ratio of the final structured light pattern. Therefore, in one embodiment, the intensity information will be calculated according to the following formula:
上述信息实质上计算的是振幅调制信号的振幅值，这一振幅值计算由于进行了抽头之间的相减以及平方根，在非常小时间范围内可以看成环境光不变，因此相减正好可以十分有效地去除由环境光引起的噪声，同时该振幅值也非常完好地反映出了目标物体表面纹理所对应的反射光强度信息。可以理解的是，也可以通过对振幅进行平方等处理以得到光束的强度信息。The above information essentially calculates the amplitude value of the amplitude modulation signal. Due to the subtraction and square root between the taps, the amplitude value calculation can be regarded as the ambient light in a very small time range. Therefore, the subtraction can be just It is very effective in removing noise caused by ambient light, and at the same time, the amplitude value also perfectly reflects the reflected light intensity information corresponding to the surface texture of the target object. It can be understood that the intensity information of the light beam can also be obtained by performing processing such as square on the amplitude.
上述基于4抽头的TOF图像传感器以及正弦波调制光发射信号的结构光图案获取方案同样也适用于其他抽头TOF图像传感器以及其他类型调制光发射信号的深度测量装置。The above structured light pattern acquisition scheme based on the 4-tap TOF image sensor and sinusoidal wave modulated light emission signal is also applicable to other tap TOF image sensors and other types of depth measurement devices with modulated light emission signal.
如图3所示，是根据本发明第二实施例的深度测量装置原理示意图。在控制与处理电路13的控制下，光发射模组11向目标物体发射振幅被方波或脉冲调制的发射光束30为结构光光束，在本实施例中结构光光束为不规则斑点结构光，即由多个光斑202以不规则排列形成的斑点结构光图案化光束201，各个光斑202的振幅在时序上被方波或脉冲调制。采集模组12中的TOF图像传感器的各个像素至少包括3个抽头，分别用于在单个帧周期内分别采集3次光信号并转换成电信号C1、C2和C3，第一抽头的触发时间与光脉冲的发射时间同步，且触发时长与脉冲的宽度相同，第二、三抽头在第一抽头结束时相继被触发，触发时长与第一抽头相同。As shown in FIG. 3 , it is a schematic diagram of the principle of the depth measuring device according to the second embodiment of the present invention. Under the control of the control and processing circuit 13, the light emitting module 11 emits to the target object the emission beam 30 whose amplitude is modulated by a square wave or a pulse, which is a structured light beam. In this embodiment, the structured light beam is irregular spot structured light. That is, the spot structured light patterned light beam 201 is formed by a plurality of light spots 202 arranged in an irregular manner, and the amplitude of each light spot 202 is modulated by a square wave or a pulse in time sequence. Each pixel of the TOF image sensor in the collection module 12 includes at least 3 taps, which are respectively used to collect 3 optical signals in a single frame period and convert them into electrical signals C1, C2 and C3. The trigger time of the first tap is the same as The emission time of the optical pulse is synchronized, and the trigger duration is the same as the pulse width. The second and third taps are triggered successively at the end of the first tap, and the trigger duration is the same as the first tap.
由此，反射回的结构光光束的将被第一、第二、第三抽头采集到，控制与处理电路13接收电信号C1、C2和C3计算出结构光光束的强度信息。在一个实施例中的，强度信息根据下式计算：Thus, the reflected structured light beam will be collected by the first, second and third taps, and the control and processing circuit 13 receives the electrical signals C1, C2 and C3 to calculate the intensity information of the structured light beam. In one embodiment, the intensity information is calculated according to the following formula:
同样地，这种利用加权平均的方式虽然可以计算出反映结构光光束强度的数值，但却与传统方法一样无法消除背景光所引起的噪声问题。因此在一个实施例中，强度信息将根据下式计算：Similarly, although this weighted average method can calculate the value reflecting the intensity of the structured light beam, it cannot eliminate the noise problem caused by the background light just like the traditional method. Thus in one embodiment the intensity information will be calculated according to:
B＝C1+C2-2C3 (4)B＝C1 +C2 -2C3 (4)
可以理解的是，也可以对上述取平均等计算以获取强度信息。与式(3)相比，由于第三抽头采集的是环境光信号，因此通过相减就可以有效消除环境光的影响从而提升信噪比。It can be understood that the above calculations such as averaging can also be performed to obtain intensity information. Compared with formula (3), since the third tap collects the ambient light signal, the influence of the ambient light can be effectively eliminated by subtraction to improve the signal-to-noise ratio.
以上图2，3所示实施例中，分别举例了发射端以正弦波或者脉冲调制的方式，接收端采用4抽头或者3抽头的采集方式。在进行结构光图案强度信息计算时，均可以采用加权平均或者加入环境光消除因子的方式进行计算。可以理解的是，类似的发射端调制方式、接收端采集方式以及强度信息计算方式的组合均包含在本发明的范围之内。In the above embodiments shown in FIG. 2 and FIG. 3 , the transmitting end adopts a sine wave or pulse modulation mode, and the receiving end adopts a 4-tap or 3-tap acquisition mode. When calculating the intensity information of the structured light pattern, the calculation may be performed by using a weighted average or adding an ambient light elimination factor. It can be understood that similar combinations of the modulation method at the transmitting end, the collection method at the receiving end, and the calculation method of the intensity information are all within the scope of the present invention.
与传统的结构光深度测量方法相比，本发明发射端采用了振幅调制，在接收端采用了多抽头像素采集，从而可以使得该方法拥有比传统方案更多的功能，比如可以实现传统方案难以实现的抗环境光干扰的深度测量方法。Compared with the traditional structured light depth measurement method, the present invention uses amplitude modulation at the transmitting end and multi-tap pixel acquisition at the receiving end, so that the method has more functions than the traditional solution, such as realizing A depth measurement method that is resistant to ambient light interference is realized.
基于上述实施例的深度测量装置，还可以实现结构光与TOF结合的融合深度测量方案。图4是根据本发明一个实施例的融合TOF与结构光方案的深度测量装置的结构示意图。深度测量装置40包括光发射模组41、采集模组42、泛光光源模组44以及控制与处理电路43。与前述各实施例相似，其中由光发射模组41、采集模组42以及控制与处理电路43可以组成结构光深度测量子单元，其获取深度图像的原理与图1～3所示实施例相同。此外，深度测量装置40中由泛光光源模组44、采集模组42以及控制与处理电路43组成了TOF深度测量子单元，类似地，泛光光源模组44中的光源同样被控制与处理电路43以一定的时序进行振幅调制，比如在一个实施例中，为了方便设计控制与处理电路43，泛光光源模组44中的光源与光发射模组41中的光源以相同的方式被调制，比如振幅调制强度、调制波型等参数均相同。Based on the depth measurement device of the above embodiments, a fusion depth measurement solution combining structured light and TOF can also be realized. Fig. 4 is a schematic structural diagram of a depth measurement device combining TOF and structured light solutions according to an embodiment of the present invention. The depth measurement device 40 includes a light emission module 41 , a collection module 42 , a flood light source module 44 and a control and processing circuit 43 . Similar to the foregoing embodiments, the structured light depth measurement subunit can be composed of the light emitting module 41, the acquisition module 42, and the control and processing circuit 43, and its principle of acquiring depth images is the same as that of the embodiments shown in Figures 1 to 3 . In addition, in the depth measurement device 40, the TOF depth measurement subunit is composed of the flood light source module 44, the acquisition module 42, and the control and processing circuit 43. Similarly, the light source in the flood light source module 44 is also controlled and processed. The circuit 43 performs amplitude modulation with a certain timing. For example, in one embodiment, in order to facilitate the design of the control and processing circuit 43, the light source in the flood light source module 44 is modulated in the same way as the light source in the light emitting module 41. , such as amplitude modulation intensity, modulation waveform and other parameters are the same.
控制与处理电路43也可以包含多个子电路分别对应于结构光深度测量子单元以及TOF深度测量子单元。TOF深度测量子单元与结构光深度测量子单元不同的是控制与处理电路43中所执行的深度计算原理，对于结构光深度测量子单元而言，如前述各实施例所述，控制与处理电路43对由采集模组42所形成的电信号进行处理并计算出反映反射光束强度的强度信息以形成结构光图案，再基于结构光图案利用结构光三角法进行深度值计算；而对于TOF深度测量子单元而言，控制与处理电路43则对采集模组42所形成的电信号进行处理以计算出相位差，并基于该相位差计算反射光束由发射端发射到接收端接收所用的飞行时间，进一步基于该飞行时间计算出目标的深度值。控制与处理电路43可以是高度集成的电路元器件，比如FPGA、SOC、ASIC等芯片，其内部包含可分别进行结构光计算以及TOF计算的电路模块，本发明同样提供了一个具体的实施例，将在图7所示的实施例中进行详述。The control and processing circuit 43 may also include a plurality of sub-circuits respectively corresponding to the structured light depth measurement subunit and the TOF depth measurement subunit. The difference between the TOF depth measurement subunit and the structured light depth measurement subunit is the depth calculation principle implemented in the control and processing circuit 43. For the structured light depth measurement subunit, as described in the foregoing embodiments, the control and processing circuit 43 Process the electrical signal formed by the acquisition module 42 and calculate the intensity information reflecting the intensity of the reflected beam to form a structured light pattern, and then use the structured light triangulation method to calculate the depth value based on the structured light pattern; and for TOF depth measurement As far as the subunit is concerned, the control and processing circuit 43 processes the electrical signal formed by the acquisition module 42 to calculate the phase difference, and based on the phase difference, calculates the flight time of the reflected light beam from the transmitting end to the receiving end for reception, The depth value of the target is further calculated based on the time-of-flight. The control and processing circuit 43 can be highly integrated circuit components, such as FPGA, SOC, ASIC and other chips, which contain circuit modules that can perform structured light calculation and TOF calculation respectively. The present invention also provides a specific embodiment. The details will be described in the embodiment shown in FIG. 7 .
回到图4，在本实施例中，结构光深度测量子单元以及TOF深度测量子单元分别用于测量范围在ΔZ1以及ΔZ2内的目标，并且ΔZ1≠ΔZ2。在一个实施例中，ΔZ1为近距，ΔZ2为远距，即由结构光深度测量子单元对近距的目标物体进行测量，由TOF深度测量子单元对远距的目标物体进行测量。由于结构光算法精度在近距离时较高且随距离增加会急速降低甚至无法测量，而TOF算法则在近距离测量时深度分辨率较低、而对远距离拥有较高的精度。可以同时启动两个子单元进行测量，比如前后帧切换的形式，在分别获取结构光深度图像以及TOF深度图像之后，可以将两幅深度图像进行融合以获取大范围、高精度的深度图像，融合方式可以采用基于置信度的加权平均等方式。比如在一个实施例中，根据相对精度设计置信度，对于结构光深度图像而言，深度值越大，置信度越低，而对于TOF深度图像而言，深度值越大，置信度越高。本实施例的方案所带来的有益效果是，可以使得深度测量装置40实现更大的测量范围，同时在大范围内均保持较高的测量精度。Referring back to FIG. 4 , in this embodiment, the structured light depth measurement subunit and the TOF depth measurement subunit are respectively used to measure targets within the ranges of ΔZ1 and ΔZ2 , and ΔZ1 ≠ΔZ2 . In one embodiment, ΔZ1 is a short distance, and ΔZ2 is a long distance, that is, the structured light depth measurement subunit measures a short-distance target object, and the TOF depth measurement subunit measures a long-distance target object. Since the accuracy of the structured light algorithm is high at short distances and will decrease rapidly or even be impossible to measure as the distance increases, while the TOF algorithm has low depth resolution at short distances and high accuracy at long distances. Two sub-units can be started at the same time for measurement, such as the form of front and rear frame switching. After obtaining the structured light depth image and TOF depth image respectively, the two depth images can be fused to obtain a large-scale, high-precision depth image. The fusion method Ways such as weighted average based on confidence can be used. For example, in one embodiment, the confidence level is designed according to the relative accuracy. For the structured light depth image, the larger the depth value, the lower the confidence level, while for the TOF depth image, the larger the depth value, the higher the confidence level. The beneficial effect brought by the solution of this embodiment is that the depth measurement device 40 can achieve a larger measurement range while maintaining high measurement accuracy in a large range.
图5是根据本发明又一实施例的融合TOF与结构光方案的深度测量装置的结构示意图。该深度测量装置同样包含结构光深度测量子单元与TOF深度测量子单元，可以理解的是，TOF深度测量子单元即可以是由泛光光源模组与采集模组组成，也可以与结构光深度测量子单元相同由光发射模组以及采集模组组成，区别在于控制与处理电路对电信号的处理方式不同，对于TOF深度测量子单元而言，其计算的是相位信息。在本实施例中，默认TOF深度测量子单元由泛光光源模组与采集模组组成，其可以获取采集模组各个像素的TOF深度值。Fig. 5 is a schematic structural diagram of a depth measurement device combining TOF and structured light solutions according to another embodiment of the present invention. The depth measurement device also includes a structured light depth measurement subunit and a TOF depth measurement subunit. It can be understood that the TOF depth measurement subunit can be composed of a flood light source module and an acquisition module, or can be combined with a structured light depth measurement The measurement subunit is also composed of a light emission module and an acquisition module. The difference is that the control and processing circuits process electrical signals differently. For the TOF depth measurement subunit, it calculates phase information. In this embodiment, the default TOF depth measurement subunit is composed of a flood light source module and an acquisition module, which can obtain the TOF depth value of each pixel of the acquisition module.
由于结构光深度计算需要进行匹配计算，匹配计算是整个深度计算环节中最消耗资源也是对精度影响最大的环节。匹配计算一般包括初始深度值估计、迭代优化、亚像素高精度深度值计算等几个步骤，其中初始深度值估计直接影响到后面步骤的效率与计算精度。本实施例中，将由TOF深度测量子单元所获取的TOF深度值作为结构光深度计算的初始深度值，如图5所示，采集模组中的TOF图像传感器包含多个像素51，光发射模组所发射的斑点光束经目标物体反射后成像在TOF图像传感器的一些像素上形成光斑52，可以理解的是，这里假定光斑52大致占2x2＝4个像素，实际上光斑52可以是其他大小，在此不做限定。首先通过TOF深度测量子单元获取各个像素51的TOF深度值(对于由光发射模组及采集模组组成的TOF深度测量子单元而言，只能获取光斑52对应像素的TOF深度值D1～D4)，其次通过结构光深度测量子单元采集包含各个斑点图案的结构光图案，最后进行结构光深度值计算时将TOF深度值作为匹配计算的初始深度值d。Since structured light depth calculation requires matching calculation, matching calculation is the link that consumes the most resources and has the greatest impact on accuracy in the entire depth calculation process. Matching calculation generally includes several steps such as initial depth value estimation, iterative optimization, and sub-pixel high-precision depth value calculation, among which the initial depth value estimation directly affects the efficiency and calculation accuracy of subsequent steps. In this embodiment, the TOF depth value acquired by the TOF depth measurement subunit is used as the initial depth value for structured light depth calculation. As shown in FIG. The speckle beam emitted by the group is reflected by the target object and forms a light spot 52 on some pixels of the TOF image sensor. It can be understood that the light spot 52 is assumed to roughly occupy 2×2=4 pixels. In fact, the light spot 52 can be other sizes. It is not limited here. First, obtain the TOF depth value of each pixel 51 through the TOF depth measurement subunit (for the TOF depth measurement subunit composed of a light emitting module and an acquisition module, only the TOF depth values D1-D4 of the pixels corresponding to the light spot 52 can be obtained ), and then the structured light depth measurement sub-unit collects the structured light pattern including each speckle pattern, and finally uses the TOF depth value as the initial depth value d for matching calculation when calculating the structured light depth value.
由于TOF深度值本身就具备较高的精度，以此作为结构光深度匹配计算的初始值，再进行迭代计算实现亚像素的深度值估计，可以实现比单独结构光或者TOF计算更高精度的深度计算。Since the TOF depth value itself has high precision, it is used as the initial value for the structured light depth matching calculation, and then iterative calculation is performed to realize the sub-pixel depth value estimation, which can achieve a higher precision depth than the single structured light or TOF calculation. calculate.
图4、5所示实施例中均涉及到结构光深度测量子单元与TOF深度测量子单元的融合，在实际操作过程中，两个子单元将以一定的时序进行工作，比如图6是根据本发明一个实施例的结构光与TOF时序控制示意图，在本实施例中，结构光与TOF以一定的频率切换工作，即在周期T1、T3、T5等中深度测量装置以结构光模式进行工作，而在周期T2、T4、T6等中以TOF模式进行工作。可以理解的是，根据实际需要也可以通过其他时序进行控制，比如对于图5所示实施而言，可以TOF一帧、结构光两帧的方式进行控制。The embodiments shown in Figures 4 and 5 all involve the integration of the structured light depth measurement subunit and the TOF depth measurement subunit. In actual operation, the two subunits will work in a certain sequence. Schematic diagram of timing control of structured light and TOF in an embodiment of the invention. In this embodiment, structured light and TOF are switched at a certain frequency, that is, the depth measurement device works in structured light mode in periods T1, T3, T5, etc. In the periods T2, T4, T6, etc., it works in the TOF mode. It can be understood that other time sequences can also be used for control according to actual needs. For example, for the implementation shown in FIG. 5 , one frame of TOF and two frames of structured light can be used for control.
图7是根据本发明一个实施例的控制与处理电路架构示意图。为了尽可能提升计算效率以及降低功耗，将一些计算、控制功能设计成专用芯片，比如FPGA、ASIC或者SOC芯片的形式是非常有效的方式。本实施例提供一种可实现上述各实施例中控制或计算功能的专用控制与处理电路芯片架构。芯片70用于接收来自TOF图像传感器的电信号，通过内部各个模块的计算后输出深度图像，在计算过程中也会与其他器件进行数据通信，比如存储器等。FIG. 7 is a schematic diagram of a control and processing circuit architecture according to an embodiment of the present invention. In order to improve computing efficiency and reduce power consumption as much as possible, it is very effective to design some computing and control functions into dedicated chips, such as FPGA, ASIC or SOC chips. This embodiment provides a dedicated control and processing circuit chip architecture capable of realizing the control or computing functions in the above-mentioned embodiments. The chip 70 is used to receive electrical signals from the TOF image sensor, output depth images after calculation by various internal modules, and communicate with other devices during the calculation process, such as memory.
芯片70可以实现多重模式深度计算，TOF深度计算模式、结构光深度计算模式以及融合深度计算模式。The chip 70 can realize multi-mode depth calculation, TOF depth calculation mode, structured light depth calculation mode and fusion depth calculation mode.
(1)TOF深度计算模式。(1) TOF depth calculation mode.
当在TOF深度计算模式下进行TOF深度计算时，芯片70接收来自TOF图像传感器的电信号，比如图2或图3所示实施例中的像素各个抽头所采集到的电信号值。电信号将首先进入分路器74-1，分路器74-1在主控制器(图中未示出)的控制下根据当前的模式选择性让电信号进入下一计算模块，在TOF深度计算模式下电信号将进入相位计算模块71。相位计算模块71执行对电信号的相位计算以获取相位值，该相位值与深度值之间存在线性关系，因此在一些实施例中，该相位计算模块也可以直接计算出深度值。随后相位值被送入标定模块进行标定，由于TOF测量常常受到噪声干扰，使得测量值与实际值之间存在一定的误差，因此在实际使用前将采用一个标定步骤，比如在一定的测量区间内每隔一段距离设置标定板，并且标定板的实际深度值已知，随后逐次对不同距离上的标定板进行实际测量得到各个距离对应的测量值，测量值与实际值之间的关系就可以作为预标定参数被存储到存储器中，标定模块在标定时将从存储器中调用预标定参数对当前测量值进行标定。这里的预标定参数即可以是实际值与测量值的对照表(index)，此时标定模块72标定过程实际上是查表过程；也可以是通过一定的数学手段对误差进行建模，并通过预先的多次测量以计算得到模型中的未知参数，标定模块72的标定过程实际上就是基于模型、测量值计算出实际值的过程。经标定后的相位值/深度值将被送入第一后处理模块73，第一后处理模块73对相位值/深度值进行例如图像增强、插值计算等对其进行优化，比如孔洞填充、边缘优化等。被第一后处理模块73处理后的相位值/深度值经由复用器74-2输出。When performing TOF depth calculation in the TOF depth calculation mode, the chip 70 receives electrical signals from the TOF image sensor, such as the electrical signal values collected by each tap of the pixel in the embodiment shown in FIG. 2 or FIG. 3 . The electrical signal will first enter the splitter 74-1. Under the control of the main controller (not shown in the figure), the splitter 74-1 selectively allows the electrical signal to enter the next calculation module according to the current mode. In the calculation mode, the electrical signal will enter the phase calculation module 71 . The phase calculation module 71 performs phase calculation on the electrical signal to obtain a phase value, and there is a linear relationship between the phase value and the depth value, so in some embodiments, the phase calculation module can also directly calculate the depth value. Then the phase value is sent to the calibration module for calibration. Since TOF measurement is often disturbed by noise, there is a certain error between the measured value and the actual value. Therefore, a calibration step will be used before actual use, such as within a certain measurement interval Calibration boards are set at intervals, and the actual depth values of the calibration boards are known, and then the actual measurement of the calibration boards at different distances is carried out successively to obtain the measured values corresponding to each distance. The relationship between the measured value and the actual value can be used as The pre-calibration parameters are stored in the memory, and the calibration module will call the pre-calibration parameters from the memory to calibrate the current measured value during calibration. The pre-calibration parameters here can be the comparison table (index) of the actual value and the measured value. At this time, the calibration process of the calibration module 72 is actually a table look-up process; Multiple measurements in advance are used to calculate unknown parameters in the model, and the calibration process of the calibration module 72 is actually a process of calculating actual values based on the model and measured values. The calibrated phase value/depth value will be sent to the first post-processing module 73, and the first post-processing module 73 will optimize the phase value/depth value, such as image enhancement, interpolation calculation, etc., such as hole filling, edge optimization etc. The phase value/depth value processed by the first post-processing module 73 is output via the multiplexer 74-2.
(2)结构光深度计算模式。(2) Structured light depth calculation mode.
当在结构光深度计算模式下进行结构光深度计算时，芯片70接收来自TOF图像传感器的电信号，比如图2或图3所示实施例中的像素各个抽头所采集到的电信号值。电信号将首先进入分路器74-1，分路器74-1在主控制器(图中未示出)的控制下根据当前的模式选择性让电信号进入下一计算模块，在结构光深度计算模式下电信号将进入振幅计算模块75。振幅计算模块75执行对电信号的相位计算以获取反映光束强度的强度信息，比如振幅值。随后振幅值被送入前处理模块76进行去噪、对比度增强等处理，也可以对振幅值执行图像畸变校正等前处理任务。被前处理后的振幅值随后进入匹配模块77进行匹配计算，在进行匹配计算时匹配模块77将从存储器中调用被预先存储的参考图像，在一个实施例中，匹配模块77采用零均值归一化最小平方距离函数对振幅值对应的图像与参考图像进行像素偏离值的匹配估计。根据结构光三角法，像素偏离值与目标的深度值之间存在一定的关系，因此在一些实施例中，匹配模块77也可以直接进行深度值的计算，当然深度值的计算也可以放在后续其他模块执行。匹配模块77输出的偏离值/深度值随后被送入第二后处理模块78进行例如图像增强、插值计算等对其进行优化，比如孔洞填充、边缘优化等。被第二后处理模块78处理后的偏离值/深度值经由复用器74-2输出。When the structured light depth calculation is performed in the structured light depth calculation mode, the chip 70 receives the electrical signal from the TOF image sensor, such as the electrical signal value collected by each tap of the pixel in the embodiment shown in FIG. 2 or FIG. 3 . The electrical signal will first enter the splitter 74-1. Under the control of the main controller (not shown in the figure), the splitter 74-1 selectively allows the electrical signal to enter the next computing module according to the current mode. In the depth calculation mode, the electrical signal will enter the amplitude calculation module 75 . The amplitude calculation module 75 performs phase calculation on the electrical signal to obtain intensity information reflecting the beam intensity, such as an amplitude value. Then the amplitude value is sent to the pre-processing module 76 for processing such as denoising and contrast enhancement, and pre-processing tasks such as image distortion correction can also be performed on the amplitude value. The pre-processed amplitude value then enters the matching module 77 for matching calculation. When performing the matching calculation, the matching module 77 will call the pre-stored reference image from the memory. In one embodiment, the matching module 77 uses zero-mean normalization The image corresponding to the amplitude value and the reference image are matched and estimated by the least square distance function. According to the structured light triangulation method, there is a certain relationship between the pixel deviation value and the depth value of the target, so in some embodiments, the matching module 77 can also directly calculate the depth value, and of course the calculation of the depth value can also be placed in the subsequent other modules execute. The offset value/depth value output by the matching module 77 is then sent to the second post-processing module 78 for image enhancement, interpolation calculation, etc. to optimize it, such as hole filling, edge optimization, etc. The offset value/depth value processed by the second post-processing module 78 is output via the multiplexer 74-2.
(3)融合深度计算模式。(3) Fusion depth calculation mode.
当在融合深度计算模式下进行深度计算时，上述两种模式下各个模块均进行工作，并由第一后处理模块73输出TOF深度图像，以及由第二后处理模块78输出结构光深度图像，TOF深度图像与结构光深度图像随后被送入融合模块79进行融合以输出最终的融合深度图像，融合深度图像经由复用器74-2进行输出。具体融合的方式可参考图4所示实施例。When the depth calculation is performed in the fusion depth calculation mode, each module in the above two modes works, and the TOF depth image is output by the first post-processing module 73, and the structured light depth image is output by the second post-processing module 78, The TOF depth image and the structured light depth image are then sent to the fusion module 79 for fusion to output a final fusion depth image, and the fusion depth image is output through the multiplexer 74-2. For a specific fusion manner, reference may be made to the embodiment shown in FIG. 4 .
在一些实施例中，由第一后处理模块73输出的TOF深度图像后，直接进入匹配模块77，以作为匹配计算的初始值，具体的融合方案参考图5所示实施例。In some embodiments, the TOF depth image output by the first post-processing module 73 directly enters the matching module 77 as an initial value for matching calculation. For the specific fusion scheme, refer to the embodiment shown in FIG. 5 .
在一些实施例中，相位计算模块71与振幅计算模块75可以是同一个计算模块，该模块将同步基于电信号计算出相位以及振幅，相位与振幅将被直接送入下一模块计算，或者被送入分路器74-1，分路器74-1根据当前的模式将相位和/或振幅送入相应的模块进行计算。In some embodiments, the phase calculation module 71 and the amplitude calculation module 75 can be the same calculation module, and this module will synchronously calculate the phase and amplitude based on the electrical signal, and the phase and amplitude will be directly sent to the next module for calculation, or be Send it to the splitter 74-1, and the splitter 74-1 sends the phase and/or amplitude to the corresponding module for calculation according to the current mode.
在一些实施例中，上述三种模式可以同步进行，复用器74-2可以实时同步或连续输出不同模式所输出的深度图像。In some embodiments, the above three modes can be performed synchronously, and the multiplexer 74-2 can output depth images output by different modes synchronously or continuously in real time.
在一些实施例中，芯片70还可以包括如总线、主控制器、输入/输出接口等模块，比如若芯片70为集成SOC芯片时，将包括这些模块，分路器74-1通过输入接口接收数据，复用器74-2的数据通过输出接口输出，输入/输出接口可以是USB、MIPI、DVP等形式的接口；总线负责SOC内部各个模块以及外部模块之间的通信、传输；主控制器则负责内部各个模块之间的资源调配与功能触发等工作。In some embodiments, the chip 70 may also include modules such as a bus, a main controller, and an input/output interface. For example, if the chip 70 is an integrated SOC chip, these modules will be included, and the splitter 74-1 receives Data, the data of the multiplexer 74-2 is output through the output interface, and the input/output interface can be an interface in the form of USB, MIPI, DVP, etc.; the bus is responsible for the communication and transmission between the various modules inside the SOC and the external modules; the main controller It is responsible for resource allocation and function triggering among various internal modules.
基于上述各实施例中的深度测量装置，本申请还提供下述深度测量方法。Based on the depth measurement devices in the above embodiments, the present application also provides the following depth measurement methods.
如图8所示，本申请中一种距离测量方法如下：As shown in Figure 8, a distance measurement method in this application is as follows:
利用光源发射时序上振幅被调制的光束，所述光束被图案化光学元件接收后形成结构光光束向目标物体发射；A light source is used to emit a light beam whose amplitude is modulated in time sequence, and the light beam is received by a patterned optical element to form a structured light beam and emit it to a target object;
利用包含至少一个像素的TOF图像传感器的所述像素接收由所述目标物体反射的结构光光束并形成电信号；Using the pixel of the TOF image sensor comprising at least one pixel to receive the structured light beam reflected by the target object and form an electrical signal;
接收所述电信号并计算出所述反射的结构光光束的强度信息以形成结构光图案，以及，利用所述结构光图案计算所述目标物体的深度图像。receiving the electrical signal and calculating intensity information of the reflected structured light beam to form a structured light pattern; and calculating a depth image of the target object by using the structured light pattern.
如图9所示，本申请中一种距离测量方法如下：As shown in Figure 9, a distance measurement method in this application is as follows:
利用光发射模组的光源发射时序上振幅被调制的光束，所述光束被图案化光学元件接收后形成结构光光束向目标物体发射；Using the light source of the light emitting module to emit a light beam whose amplitude is modulated in time sequence, the light beam is received by the patterned optical element to form a structured light beam and emit it to the target object;
利用泛光光源模组向所述目标物体发射时序上振幅被调制的泛光光束；Utilizing the floodlight light source module to emit a floodlight beam whose amplitude is modulated in time sequence to the target object;
利用包含至少一个像素的TOF图像传感器的所述像素接收由所述目标物体反射的结构光光束并形成第一电信号；以及，接收所述目标物体反射的泛光光束并形成第二电信号；Using the pixel of the TOF image sensor comprising at least one pixel to receive the structured light beam reflected by the target object and form a first electrical signal; and receive the flood light beam reflected by the target object and form a second electrical signal;
对所述光发射模组和所述泛光光源模组的时序进行控制，按所述时序分别接收所述目标物体反射的所述结构光光束形成的所述第一电信号，以及，所述目标物体反射的所述泛光光束形成的所述第二电信号；分别对所述第一电信号和第二电信号进行处理获得所述目标物体的深度图像。Controlling the timing of the light emitting module and the flood light source module, respectively receiving the first electrical signal formed by the structured light beam reflected by the target object according to the timing, and the The second electrical signal formed by the flood light beam reflected by the target object; respectively processing the first electrical signal and the second electrical signal to obtain a depth image of the target object.
如图10所示，一种实施例中对所述电信号进行处理包括：As shown in Figure 10, in an embodiment, processing the electrical signal includes:
根据所述结构光光束形成的所述第一电信号计算出所述结构光光束的强度信息以形成结构光图案，利用所述结构光图案计算所述目标物体的第一深度图像；calculating intensity information of the structured light beam according to the first electrical signal formed by the structured light beam to form a structured light pattern, and using the structured light pattern to calculate a first depth image of the target object;
根据所述泛光光束形成的所述第二电信号计算出相位差，并基于所述相位差计算所述泛光光束由发射到被接收所需要的飞行时间，基于所述飞行时间计算所述目标物体的深度值，利用所述深度值获取所述目标物体的第二深度图像；Calculate the phase difference according to the second electrical signal formed by the flood beam, and calculate the time-of-flight required for the flood beam from being emitted to being received based on the phase difference, and calculate the time-of-flight based on the time-of-flight a depth value of the target object, using the depth value to acquire a second depth image of the target object;
将所述第一深度图像和所述第二深度图像融合获得所述目标物体的深度图像。The first depth image and the second depth image are fused to obtain a depth image of the target object.
如图11所示，另一种实施例中对所述电信号进行处理包括：As shown in Figure 11, in another embodiment, processing the electrical signal includes:
根据所述泛光光束形成的所述第二电信号计算出相位差，并基于所述相位差计算所述泛光光束由发射到被接收所需要的飞行时间，基于该所述飞行时间计算各个所述像素的TOF深度值；Calculate the phase difference according to the second electrical signal formed by the flood light beam, and calculate the flight time required for the flood light beam from being emitted to being received based on the phase difference, and calculate each the TOF depth value of the pixel;
根据所述结构光光束形成的所述第一电信号计算出所述结构光光束的强度信息以形成结构光图案；calculating intensity information of the structured light beam according to the first electrical signal formed by the structured light beam to form a structured light pattern;
将所述TOF深度值作为所述结构光图案匹配计算的初始深度值，利用所述结构光图案计算所述目标物体的深度图像。The TOF depth value is used as an initial depth value for the structured light pattern matching calculation, and the structured light pattern is used to calculate the depth image of the target object.
本发明实现上述实施例方法中的全部或部分流程，也可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一计算机可读存储介质中，该计算机程序在被处理器执行时，可实现上述各个方法实施例的步骤。其中，所述计算机程序包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读介质可以包括：能够携带所述计算机程序代码的任何实体或装置、记录介质、U盘、移动硬盘、磁碟、光盘、计算机存储器、只读存储器(ROM，Read-OnlyMemory)、随机存取存储器(RAM，Random Access Memory)、电载波信号、电信信号以及软件分发介质等。需要说明的是，所述计算机可读介质包含的内容可以根据司法管辖区内立法和专利实践的要求进行适当的增减，例如在某些司法管辖区，根据立法和专利实践，计算机可读介质不包括电载波信号和电信信号。The present invention realizes all or part of the processes in the methods of the above embodiments, and can also be completed by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program is processed by the processor. During execution, the steps in the above-mentioned various method embodiments can be realized. Wherein, the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM, Read-OnlyMemory), Random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, computer-readable media Excludes electrical carrier signals and telecommunication signals.
本发明达到的有益效果为，提供了一种基于振幅时序调制的结构光发射端与TOF图像传感器组成的深度测量装置，并提出了基于该装置的多种深度图像获取的方案，以实现目前传统方案难以实现的高精度、大范围深度测量。同时还提供了一种深度计算芯片架构以从芯片级实现该方案，最终实现低功耗、高效率计算的目的。The beneficial effect achieved by the present invention is that it provides a depth measuring device composed of a structured light transmitter based on amplitude timing modulation and a TOF image sensor, and proposes a variety of depth image acquisition schemes based on the device, in order to realize the current traditional It is difficult to achieve high-precision and wide-range depth measurement. At the same time, it also provides a deep computing chip architecture to realize the solution from the chip level, and finally achieve the purpose of low power consumption and high efficiency computing.
以上内容是结合具体的优选实施方式对本发明所作的进一步详细说明，不能认定本发明的具体实施只局限于这些说明。对于本发明所属技术领域的技术人员来说，在不脱离本发明构思的前提下，还可以做出若干等同替代或明显变型，而且性能或用途相同，都应当视为属于本发明的保护范围。The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments, and it cannot be assumed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art to which the present invention belongs, several equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and those with the same performance or use should be deemed to belong to the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910632082.4ACN110333501A (en) | 2019-07-12 | 2019-07-12 | Depth measuring device and distance measuring method |
| PCT/CN2020/089758WO2021008209A1 (en) | 2019-07-12 | 2020-05-12 | Depth measurement apparatus and distance measurement method |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910632082.4ACN110333501A (en) | 2019-07-12 | 2019-07-12 | Depth measuring device and distance measuring method |
| Publication Number | Publication Date |
|---|---|
| CN110333501Atrue CN110333501A (en) | 2019-10-15 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910632082.4APendingCN110333501A (en) | 2019-07-12 | 2019-07-12 | Depth measuring device and distance measuring method |
| Country | Link |
|---|---|
| CN (1) | CN110333501A (en) |
| WO (1) | WO2021008209A1 (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110954029A (en)* | 2019-11-04 | 2020-04-03 | 深圳奥比中光科技有限公司 | Three-dimensional measurement system under screen |
| CN110954917A (en)* | 2019-12-18 | 2020-04-03 | 深圳奥比中光科技有限公司 | Depth measuring device and depth measuring method |
| CN111025317A (en)* | 2019-12-28 | 2020-04-17 | 深圳奥比中光科技有限公司 | Adjustable depth measuring device and measuring method |
| CN111025319A (en)* | 2019-12-28 | 2020-04-17 | 深圳奥比中光科技有限公司 | Depth measuring device and measuring method |
| CN111045029A (en)* | 2019-12-18 | 2020-04-21 | 深圳奥比中光科技有限公司 | Fused depth measuring device and measuring method |
| CN111123289A (en)* | 2019-12-18 | 2020-05-08 | 深圳奥比中光科技有限公司 | Depth measuring device and measuring method |
| CN111189617A (en)* | 2020-02-21 | 2020-05-22 | 深圳奥比中光科技有限公司 | MEMS micro-galvanometer monitoring device and method based on optical super-surface |
| CN111239729A (en)* | 2020-01-17 | 2020-06-05 | 西安交通大学 | ToF depth sensor with fusion speckle and flood projection and its ranging method |
| CN111323205A (en)* | 2020-02-21 | 2020-06-23 | 深圳奥比中光科技有限公司 | Micro-mirror array-based MEMS micro-galvanometer monitoring device and method |
| CN111366941A (en)* | 2020-04-20 | 2020-07-03 | 深圳奥比中光科技有限公司 | A TOF depth measurement device and method |
| CN111626282A (en)* | 2020-05-12 | 2020-09-04 | 深圳阜时科技有限公司 | Image sensing device and electronic apparatus |
| WO2021008209A1 (en)* | 2019-07-12 | 2021-01-21 | 深圳奥比中光科技有限公司 | Depth measurement apparatus and distance measurement method |
| CN112462388A (en)* | 2020-11-12 | 2021-03-09 | 深圳博升光电科技有限公司 | Three-dimensional imaging device and method and electronic equipment |
| CN112615979A (en)* | 2020-12-07 | 2021-04-06 | 江西欧迈斯微电子有限公司 | Image acquisition method, image acquisition apparatus, electronic apparatus, and storage medium |
| CN112712476A (en)* | 2020-12-17 | 2021-04-27 | 豪威科技(武汉)有限公司 | Denoising method and denoising device for TOF (time of flight) ranging and TOF camera |
| TWI728667B (en)* | 2019-11-06 | 2021-05-21 | 奇景光電股份有限公司 | Structured light imaging device |
| CN112929519A (en)* | 2019-12-06 | 2021-06-08 | Oppo广东移动通信有限公司 | Depth camera, imaging device, and electronic apparatus |
| CN112969019A (en)* | 2021-02-26 | 2021-06-15 | 深圳荆虹科技有限公司 | TOF module and electronic device |
| CN113031001A (en)* | 2021-02-24 | 2021-06-25 | Oppo广东移动通信有限公司 | Depth information processing method, depth information processing apparatus, medium, and electronic device |
| CN113075672A (en)* | 2021-03-03 | 2021-07-06 | 深圳市灵明光子科技有限公司 | Ranging method and system, and computer readable storage medium |
| CN113596330A (en)* | 2021-07-28 | 2021-11-02 | 北京水木东方医用机器人技术创新中心有限公司 | Method, device and equipment for improving refresh rate of infrared optical positioner |
| WO2021238213A1 (en)* | 2020-05-24 | 2021-12-02 | 奥比中光科技集团股份有限公司 | Tof-based depth measurement apparatus and method, and electronic device |
| CN114236504A (en)* | 2021-12-17 | 2022-03-25 | 奥诚信息科技(上海)有限公司 | dToF-based detection system and light source adjusting method thereof |
| WO2022110947A1 (en)* | 2020-11-25 | 2022-06-02 | Oppo广东移动通信有限公司 | Control method for electronic device, electronic device, and computer-readable storage medium |
| CN115097474A (en)* | 2022-06-23 | 2022-09-23 | 深圳市汇顶科技股份有限公司 | Time difference of flight ranging module, method and electronic equipment |
| WO2023011091A1 (en)* | 2021-08-03 | 2023-02-09 | Congying Sui | An imaging method and system based on wise-pixels with valved modulation |
| CN116165679A (en)* | 2023-02-16 | 2023-05-26 | 奥比中光科技集团股份有限公司 | Multi-spectral depth camera, spectrum and depth measurement method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115079187A (en)* | 2021-03-10 | 2022-09-20 | 奥比中光科技集团股份有限公司 | Flight time measuring method and device and time flight depth camera |
| CN116981957A (en)* | 2021-03-15 | 2023-10-31 | 索尼半导体解决方案公司 | Lighting circuit, lighting method and time-of-flight module |
| CN113326802A (en)* | 2021-06-23 | 2021-08-31 | 深圳博升光电科技有限公司 | 3D recognition device, 3D recognition system and recognition method thereof |
| CN113960562A (en)* | 2021-08-17 | 2022-01-21 | 科沃斯机器人股份有限公司 | Structured light module and self-moving equipment |
| CN113960569A (en)* | 2021-10-19 | 2022-01-21 | 深圳奥锐达科技有限公司 | Distance measuring system and distance measuring method |
| CN114236505B (en)* | 2021-12-20 | 2025-06-03 | 奥诚信息科技(上海)有限公司 | A measurement system and method based on DToF |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104903677A (en)* | 2012-12-17 | 2015-09-09 | Lsi公司 | Methods and apparatus for merging depth images generated using distinct depth imaging techniques |
| US20180059224A1 (en)* | 2016-08-26 | 2018-03-01 | Samsung Electronics Co., Ltd. | Time-of-flight (tof) image sensor using amplitude modulation for range measurement |
| CN107765260A (en)* | 2016-08-22 | 2018-03-06 | 三星电子株式会社 | For obtaining the method, equipment and computer readable recording medium storing program for performing of range information |
| CN107852472A (en)* | 2016-06-15 | 2018-03-27 | 索尼公司 | Imaging device, imaging method and program |
| WO2018140107A1 (en)* | 2017-01-25 | 2018-08-02 | Google Llc | System for 3d image filtering |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR101788032B1 (en)* | 2011-03-24 | 2017-10-19 | 삼성전자주식회사 | Depth sensor, depth information error compensation method thereof, and signal processing system having the depth sensor |
| US9958758B2 (en)* | 2015-01-21 | 2018-05-01 | Microsoft Technology Licensing, Llc | Multiple exposure structured light pattern |
| CN107067428B (en)* | 2017-03-10 | 2020-06-30 | 深圳奥比中光科技有限公司 | Augmented reality projection device and method |
| CN108490632B (en)* | 2018-03-12 | 2020-01-10 | Oppo广东移动通信有限公司 | Laser projection module, depth camera and electronic device |
| CN108881717B (en)* | 2018-06-15 | 2020-11-03 | 深圳奥比中光科技有限公司 | Depth imaging method and system |
| CN109283508A (en)* | 2018-11-21 | 2019-01-29 | 深圳奥比中光科技有限公司 | Flight time calculation method |
| CN109343070A (en)* | 2018-11-21 | 2019-02-15 | 深圳奥比中光科技有限公司 | Time flight depth camera |
| CN110333501A (en)* | 2019-07-12 | 2019-10-15 | 深圳奥比中光科技有限公司 | Depth measuring device and distance measuring method |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104903677A (en)* | 2012-12-17 | 2015-09-09 | Lsi公司 | Methods and apparatus for merging depth images generated using distinct depth imaging techniques |
| CN107852472A (en)* | 2016-06-15 | 2018-03-27 | 索尼公司 | Imaging device, imaging method and program |
| CN107765260A (en)* | 2016-08-22 | 2018-03-06 | 三星电子株式会社 | For obtaining the method, equipment and computer readable recording medium storing program for performing of range information |
| US20180059224A1 (en)* | 2016-08-26 | 2018-03-01 | Samsung Electronics Co., Ltd. | Time-of-flight (tof) image sensor using amplitude modulation for range measurement |
| WO2018140107A1 (en)* | 2017-01-25 | 2018-08-02 | Google Llc | System for 3d image filtering |
| Title |
|---|
| 刘力维: "基于TOF深度摄像机的深度超分辨率恢复和深度融合研究", 《中国博士学位论文全文数据库信息科技辑》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2021008209A1 (en)* | 2019-07-12 | 2021-01-21 | 深圳奥比中光科技有限公司 | Depth measurement apparatus and distance measurement method |
| CN110954029A (en)* | 2019-11-04 | 2020-04-03 | 深圳奥比中光科技有限公司 | Three-dimensional measurement system under screen |
| TWI728667B (en)* | 2019-11-06 | 2021-05-21 | 奇景光電股份有限公司 | Structured light imaging device |
| CN112929519B (en)* | 2019-12-06 | 2022-10-25 | Oppo广东移动通信有限公司 | Depth Cameras, Imaging Devices and Electronics |
| CN112929519A (en)* | 2019-12-06 | 2021-06-08 | Oppo广东移动通信有限公司 | Depth camera, imaging device, and electronic apparatus |
| CN111045029A (en)* | 2019-12-18 | 2020-04-21 | 深圳奥比中光科技有限公司 | Fused depth measuring device and measuring method |
| WO2021120402A1 (en)* | 2019-12-18 | 2021-06-24 | 深圳奥比中光科技有限公司 | Fused depth measurement apparatus and measurement method |
| WO2021120403A1 (en)* | 2019-12-18 | 2021-06-24 | 深圳奥比中光科技有限公司 | Depth measurement device and method |
| CN111123289A (en)* | 2019-12-18 | 2020-05-08 | 深圳奥比中光科技有限公司 | Depth measuring device and measuring method |
| CN111045029B (en)* | 2019-12-18 | 2022-06-28 | 奥比中光科技集团股份有限公司 | Fused depth measuring device and measuring method |
| CN110954917A (en)* | 2019-12-18 | 2020-04-03 | 深圳奥比中光科技有限公司 | Depth measuring device and depth measuring method |
| WO2021128587A1 (en)* | 2019-12-28 | 2021-07-01 | 深圳奥比中光科技有限公司 | Adjustable depth measuring device and measuring method |
| CN111025319A (en)* | 2019-12-28 | 2020-04-17 | 深圳奥比中光科技有限公司 | Depth measuring device and measuring method |
| CN111025317A (en)* | 2019-12-28 | 2020-04-17 | 深圳奥比中光科技有限公司 | Adjustable depth measuring device and measuring method |
| CN111239729A (en)* | 2020-01-17 | 2020-06-05 | 西安交通大学 | ToF depth sensor with fusion speckle and flood projection and its ranging method |
| CN111239729B (en)* | 2020-01-17 | 2022-04-05 | 西安交通大学 | ToF depth sensor with fusion speckle and flood projection and its ranging method |
| CN111189617A (en)* | 2020-02-21 | 2020-05-22 | 深圳奥比中光科技有限公司 | MEMS micro-galvanometer monitoring device and method based on optical super-surface |
| CN111323205A (en)* | 2020-02-21 | 2020-06-23 | 深圳奥比中光科技有限公司 | Micro-mirror array-based MEMS micro-galvanometer monitoring device and method |
| CN111189617B (en)* | 2020-02-21 | 2021-08-03 | 奥比中光科技集团股份有限公司 | MEMS micro-galvanometer monitoring device and method based on optical super-surface |
| CN111323205B (en)* | 2020-02-21 | 2021-08-03 | 奥比中光科技集团股份有限公司 | Micro-mirror array-based MEMS micro-galvanometer monitoring device and method |
| CN111366941A (en)* | 2020-04-20 | 2020-07-03 | 深圳奥比中光科技有限公司 | A TOF depth measurement device and method |
| WO2021212917A1 (en)* | 2020-04-20 | 2021-10-28 | 奥比中光科技集团股份有限公司 | Device and method for measuring tof depth |
| CN111626282A (en)* | 2020-05-12 | 2020-09-04 | 深圳阜时科技有限公司 | Image sensing device and electronic apparatus |
| WO2021238213A1 (en)* | 2020-05-24 | 2021-12-02 | 奥比中光科技集团股份有限公司 | Tof-based depth measurement apparatus and method, and electronic device |
| CN112462388A (en)* | 2020-11-12 | 2021-03-09 | 深圳博升光电科技有限公司 | Three-dimensional imaging device and method and electronic equipment |
| WO2022110947A1 (en)* | 2020-11-25 | 2022-06-02 | Oppo广东移动通信有限公司 | Control method for electronic device, electronic device, and computer-readable storage medium |
| CN112615979A (en)* | 2020-12-07 | 2021-04-06 | 江西欧迈斯微电子有限公司 | Image acquisition method, image acquisition apparatus, electronic apparatus, and storage medium |
| CN112712476B (en)* | 2020-12-17 | 2023-06-02 | 豪威科技(武汉)有限公司 | Denoising method and device for TOF ranging and TOF camera |
| CN112712476A (en)* | 2020-12-17 | 2021-04-27 | 豪威科技(武汉)有限公司 | Denoising method and denoising device for TOF (time of flight) ranging and TOF camera |
| CN113031001B (en)* | 2021-02-24 | 2024-02-13 | Oppo广东移动通信有限公司 | Depth information processing method, depth information processing device, medium and electronic apparatus |
| CN113031001A (en)* | 2021-02-24 | 2021-06-25 | Oppo广东移动通信有限公司 | Depth information processing method, depth information processing apparatus, medium, and electronic device |
| CN112969019A (en)* | 2021-02-26 | 2021-06-15 | 深圳荆虹科技有限公司 | TOF module and electronic device |
| CN112969019B (en)* | 2021-02-26 | 2023-08-08 | 深圳荆虹科技有限公司 | TOF module and electronic device |
| CN113075672A (en)* | 2021-03-03 | 2021-07-06 | 深圳市灵明光子科技有限公司 | Ranging method and system, and computer readable storage medium |
| CN113596330B (en)* | 2021-07-28 | 2023-04-07 | 北京水木东方医用机器人技术创新中心有限公司 | Method, device and equipment for improving refresh rate of infrared optical positioner |
| CN113596330A (en)* | 2021-07-28 | 2021-11-02 | 北京水木东方医用机器人技术创新中心有限公司 | Method, device and equipment for improving refresh rate of infrared optical positioner |
| WO2023011091A1 (en)* | 2021-08-03 | 2023-02-09 | Congying Sui | An imaging method and system based on wise-pixels with valved modulation |
| CN114236504A (en)* | 2021-12-17 | 2022-03-25 | 奥诚信息科技(上海)有限公司 | dToF-based detection system and light source adjusting method thereof |
| CN115097474A (en)* | 2022-06-23 | 2022-09-23 | 深圳市汇顶科技股份有限公司 | Time difference of flight ranging module, method and electronic equipment |
| CN116165679A (en)* | 2023-02-16 | 2023-05-26 | 奥比中光科技集团股份有限公司 | Multi-spectral depth camera, spectrum and depth measurement method |
| Publication number | Publication date |
|---|---|
| WO2021008209A1 (en) | 2021-01-21 |
| Publication | Publication Date | Title |
|---|---|---|
| CN110333501A (en) | Depth measuring device and distance measuring method | |
| CN110471080A (en) | Depth measurement device based on TOF imaging sensor | |
| CN110456379A (en) | Fusion depth measurement device and distance measurement method | |
| CN111123289B (en) | Depth measuring device and measuring method | |
| CN110488240A (en) | Depth calculation chip architecture | |
| CN111045029B (en) | Fused depth measuring device and measuring method | |
| US20210181317A1 (en) | Time-of-flight-based distance measurement system and method | |
| CN110596721B (en) | Flight time distance measuring system and method of double-shared TDC circuit | |
| CN110221272B (en) | Time flight depth camera and anti-interference distance measurement method | |
| CN110221274B (en) | Time flight depth camera and multi-frequency modulation and demodulation distance measuring method | |
| CN110596725B (en) | Time-of-flight measurement method and system based on interpolation | |
| CN206650757U (en) | a device | |
| CN110596723B (en) | Dynamic histogram drawing flight time distance measuring method and measuring system | |
| WO2021128587A1 (en) | Adjustable depth measuring device and measuring method | |
| CN110320528B (en) | Time depth camera and noise reduction distance measurement method for multi-frequency modulation and demodulation | |
| CN110596724B (en) | Method and system for measuring flight time distance during dynamic histogram drawing | |
| CN111427048B (en) | ToF depth measuring device, control method and electronic equipment | |
| CN109343070A (en) | Time flight depth camera | |
| CN110361751B (en) | Time flight depth camera and distance measuring method for reducing noise of single-frequency modulation and demodulation | |
| CN110221273A (en) | Time flight depth camera and the distance measurement method of single-frequency modulation /demodulation | |
| US20220043129A1 (en) | Time flight depth camera and multi-frequency modulation and demodulation distance measuring method | |
| US11709271B2 (en) | Time of flight sensing system and image sensor used therein | |
| CN111538024A (en) | Filtering ToF depth measurement method and device | |
| WO2022241942A1 (en) | Depth camera and depth calculation method | |
| CN109283508A (en) | Flight time calculation method |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | Address after:11-13 / F, joint headquarters building, high tech Zone, 63 Xuefu Road, Yuehai street, Nanshan District, Shenzhen, Guangdong 518000 Applicant after:Obi Zhongguang Technology Group Co.,Ltd. Address before:12 / F, joint headquarters building, high tech Zone, 63 Xuefu Road, Nanshan District, Shenzhen, Guangdong 518000 Applicant before:SHENZHEN ORBBEC Co.,Ltd. | |
| CB02 | Change of applicant information | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20191015 | |
| RJ01 | Rejection of invention patent application after publication |