CN111612826A - A high-precision three-dimensional motion trajectory acquisition, positioning and motion process reproduction method based on binocular video sensor - Google Patents

Abstract

Translated fromChinese

本发明公开了一种基于双目视频传感器的高精度三维运动轨迹采集与运动过程再现方法，本发明主要应用于对田径、游泳及冰上比赛等竞速项目的运动员运动情况数据采集与综合分析，帮助运动员教练员有效分析运动特点，改善运动策略提升运动成绩。本发明主要包括两个方面：首先，它采用了高精度视频目标跟踪技术与伺服控制跟踪系统，用于确定目标角度方位。其次，它通过相机标定与双目立体视觉技术使跟踪到的像素坐标数据，转台转角数据转换为三维空间坐标，进而得到运动目标在三维空间中的运动轨迹。然后可以通过高精度伺服控制系统对某一空间平面如运动场地平面进行光学投影，从而再现原目标的运动过程，使每一时刻投影的位置、速度和加速度等与之前所采集的目标运动过程一致。

The invention discloses a high-precision three-dimensional motion trajectory acquisition and motion process reproduction method based on a binocular video sensor. The present invention is mainly applied to data acquisition and comprehensive analysis of athletes' motion conditions in athletic events such as track and field, swimming and ice competitions. , to help athletes and coaches effectively analyze sports characteristics, improve sports strategies and improve sports performance. The invention mainly includes two aspects: firstly, it adopts high-precision video target tracking technology and servo control tracking system to determine the target angle and orientation. Secondly, it converts the tracked pixel coordinate data and turntable angle data into three-dimensional space coordinates through camera calibration and binocular stereo vision technology, and then obtains the motion trajectory of the moving target in three-dimensional space. Then a high-precision servo control system can be used to optically project a space plane such as a sports field plane, so as to reproduce the motion process of the original target, so that the projected position, speed and acceleration at each moment are consistent with the previously collected target motion process .

Description

Translated fromChinese

一种基于双目视频传感器的高精度三维运动轨迹采集定位与运动过程再现方法A high-precision three-dimensional motion trajectory acquisition and positioning based on binocular video sensorMovement process reproduction method

技术领域technical field

本发明属于计算机视觉与计算机控制领域，具体涉及机器视觉跟踪技术、双目视觉定位技术、高精度伺服控制等技术领域，是一种基于双目视频传感器的高精度定位方法。The invention belongs to the field of computer vision and computer control, and specifically relates to the technical fields of machine vision tracking technology, binocular vision positioning technology, high-precision servo control and the like, and is a high-precision positioning method based on binocular video sensors.

背景技术Background technique

近年来，随着计算机硬件水平的提高，计算机视觉技术飞速发展。其中计算机视觉中的目标跟踪技术也随之受益。不过遗憾的是现有的目标跟踪技术和云台跟踪系统的结合多用于安防监控领域而鲜有其他。近些年，随着冬奥会的即将到来，体育事业的发展也逐渐被重视。传统的冰上运动员，滑雪运动员一般通过记录整段滑行时间、滑行距离以及回放录像等来数据分析滑行质量。但由于无法分析运动员每一时刻的运动状态以及整个滑行轨迹，使得运动员不能达到理想的训练效果。基于诸如类似的辅助训练系统缺陷，急需引入双目视觉技术对运动员进行精确定位。但由于早期的计算机算力限制等原因导致无法使用实时的高精度跟踪算法、并且加上用于安防领域的云台伺服角输出精度不够。因此无法将这类设备应用于高精度定位事业。In recent years, with the improvement of computer hardware level, computer vision technology has developed rapidly. Among them, the target tracking technology in computer vision also benefits. Unfortunately, the combination of existing target tracking technology and PTZ tracking system is mostly used in the field of security monitoring and few others. In recent years, with the approaching of the Winter Olympics, the development of sports has gradually been paid attention to. Traditional ice athletes and skiers generally analyze the gliding quality by recording the entire gliding time, gliding distance, and replaying the video. However, due to the inability to analyze the movement state of the athlete at every moment and the entire gliding trajectory, the athlete cannot achieve the ideal training effect. Based on similar defects in auxiliary training systems, it is urgent to introduce binocular vision technology to accurately locate athletes. However, due to the limitation of early computer computing power and other reasons, the real-time high-precision tracking algorithm cannot be used, and the output accuracy of the PTZ servo angle used in the security field is not enough. Therefore, such equipment cannot be applied to the high-precision positioning business.

发明内容SUMMARY OF THE INVENTION

有鉴于此，本发明提出了一种基于双目视频传感器的高精度定位方法，将高精度目标跟踪技术作为视觉传感器，应用于基于高精度转台的反馈控制系统中的测量元件，在提高伺服跟踪技术的基础上，利用双目视觉技术对目标进行定位。因此该发明可广泛应用于类似于冰上、雪上的高速运动并记录高精度数据的场合。In view of this, the present invention proposes a high-precision positioning method based on a binocular video sensor, which uses the high-precision target tracking technology as a visual sensor and is applied to the measuring element in the feedback control system based on the high-precision turntable. On the basis of technology, use binocular vision technology to locate the target. Therefore, the invention can be widely used in high-speed motion similar to ice and snow and high-precision data recording.

本发明是一种基于双目视频传感器的高精度定位方法，包括如下步骤：The present invention is a high-precision positioning method based on a binocular video sensor, comprising the following steps:

第一步，设计三台高精度二维转台，并通过控制器，驱动器，运算器组成高精度伺服控制系统。The first step is to design three high-precision two-dimensional turntables, and form a high-precision servo control system through the controller, driver and arithmetic unit.

第二步，设计两个电控伺服调焦镜头，分别将两台相机与镜头固定至第一步所述转台上方，两转台可放置最远距离为80米。The second step is to design two electronically controlled servo focusing lenses, respectively, to fix the two cameras and lenses above the turntable described in the first step, and the two turntables can be placed at a maximum distance of 80 meters.

第三步，将激光手电筒置于第三台转台上，此转台置于步骤二所述两转台中间。The third step is to place the laser flashlight on the third turntable, which is placed in the middle of the two turntables described in the second step.

第四步，利用视频目标跟踪算法对每台相机传入的每帧图片中目标进行跟踪，并利用自适应PID控制算法，将跟踪得到的目标中心点与图像中心点做比较，使转台能够稳定跟踪目标转动。利用步骤二的变焦镜头，基于图像跟踪框与整幅图像作比较，使镜头稳定变焦。最终让运动目标稳定的呈现在两幅画面中心。The fourth step is to use the video target tracking algorithm to track the target in each frame of pictures passed in by each camera, and use the adaptive PID control algorithm to compare the center point of the tracked target with the center point of the image, so that the turntable can be stabilized. Track the target rotation. Using the zoom lens instep 2, compare the entire image based on the image tracking frame to stabilize the zoom of the lens. Finally, the moving target is stably presented in the center of the two pictures.

第五步，通过相机标定，获取相机的内外参数并得到世界坐标系与图像坐标系关系的对应矩阵，通过双目立体视觉技术将每一帧运动目标的像素坐标转换成世界坐标。In the fifth step, through camera calibration, the internal and external parameters of the camera are obtained and the corresponding matrix of the relationship between the world coordinate system and the image coordinate system is obtained, and the pixel coordinates of each frame of moving objects are converted into world coordinates through binocular stereo vision technology.

第六步，通过世界坐标系与步骤三转台的转角关系，利用插值计算可让激光器复现运动目标的运动状态和轨迹。In the sixth step, through the angle relationship between the world coordinate system and the turntable in step 3, the laser can reproduce the motion state and trajectory of the moving target by using interpolation calculation.

3、如权利要求1所述一种基于双目视频传感器的高精度定位方法，其特征在于，利用先进的KCF相关滤波跟踪技术，使图像传感器具有回传速度快，运算量小，跟踪精度高等特点。3. A high-precision positioning method based on a binocular video sensor as claimed in claim 1, characterized in that, using advanced KCF correlation filtering and tracking technology, the image sensor has the advantages of fast return speed, small calculation amount and high tracking accuracy. Features.

4、如权利要求1所述一种基于双目视频传感器的高精度定位方法，其特征在于，应用自适应PID控制算法对于非线性时变系统具有良好的控制效果，针对于不同运动目标可使云台自动跟踪控制更稳定，捕捉目标位置更精确。4. A high-precision positioning method based on a binocular video sensor according to claim 1, characterized in that the application of the adaptive PID control algorithm has a good control effect on nonlinear time-varying systems, and can be used for different moving targets. The automatic tracking control of the gimbal is more stable, and the target position is captured more accurately.

5、如权利要求1所述一种基于双目视频传感器的高精度定位方法，其特征在于，利用双目视觉定位技术和云台控制系统能够记录运动目标每个时刻运动速度、加速度大小，并能通过激光手电筒复现运动目标轨迹。5. A high-precision positioning method based on a binocular video sensor as claimed in claim 1, wherein the binocular vision positioning technology and the PTZ control system can be used to record the movement speed and acceleration of the moving target at each moment, and The trajectory of the moving target can be reproduced through the laser flashlight.

附图说明Description of drawings

图1是本发明系统整体结构框图；Fig. 1 is the overall structure block diagram of the system of the present invention;

图2是高精度云台设计图与实物图；Figure 2 is the high-precision gimbal design drawing and physical drawing;

图3是系统闭环控制原理图；Figure 3 is a schematic diagram of the system closed-loop control;

图4是双目视觉技术原理图。Figure 4 is a schematic diagram of binocular vision technology.

具体实施方式Detailed ways

下面结合附图和具体实施例对本发明进行详细说明。应指出的是，所描述的实例仅旨在便于对本发明的理解，而对其不起任何限定作用。The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the described examples are only intended to facilitate the understanding of the present invention without any limitation thereto.

该发明的总体硬件设计结构框图如图1所示，系统由三台高精度伺服转台、两台高帧率工业相机、一台激光器、一台运动控制器、一台目标跟踪器、与一台 PC上位机组成，其中目标跟踪器负责对相机传输的图片做跟踪处理计算，上位机则只负责显示视频与向控制器发送指令。总体过程如下，架在云台上的相机向目标跟踪器同步传输图像数据，跟踪器对每一帧图像进行目标跟踪处理，将跟踪结果数据传送给PC机，PC机会根据得到的结果自动向控制器发送控制指令，进而使云台跟踪转动。待目标运动过程结束后，PC机会记录目标轨迹并向继续控制器发送指令，使激光器云台复现整个运动轨迹。The overall hardware design structure block diagram of the invention is shown in Figure 1. The system consists of three high-precision servo turntables, two high-frame rate industrial cameras, one laser, one motion controller, one target tracker, and one It consists of a PC host computer, in which the target tracker is responsible for tracking and processing the pictures transmitted by the camera, and the host computer is only responsible for displaying the video and sending instructions to the controller. The overall process is as follows: the camera mounted on the PTZ transmits image data synchronously to the target tracker, the tracker performs target tracking processing on each frame of image, and transmits the tracking result data to the PC. The controller sends control commands to make the gimbal track and rotate. After the target movement process is over, the PC will record the target trajectory and send instructions to the continuous controller, so that the laser head can reproduce the entire movement trajectory.

第一，设计三台高精度二维转台，转台设计图与实物图如图2所示。本系统搭建的伺服旋转平台为高精度直驱伺服平台，它是一种采用直线电机直接驱动的运动控制平台,它改变过去依赖于机械转换装置才能将旋转电机的旋转运动转化为直线运动的约束,克服了传统机械转换机构的传动链长、体积大、效率低、精度差等缺陷。系统中所用的电机配有编码器，因此可以利用采集编码器的信号作为系统反馈，构建一个闭环控制系统，来实现伺服控制。本系统的总体方案如下：即使用两个直驱伺服电机，直接带动负载实现角度跟踪和定位，通过直驱电机的内置编码器实时检测伺服电机转动的角度，以实现位置、速度及加速度的精确闭环控制。First, three high-precision two-dimensional turntables are designed, and the design drawing and physical drawing of the turntable are shown in Figure 2. The servo rotating platform built by this system is a high-precision direct-drive servo platform, which is a motion control platform directly driven by a linear motor. , Overcome the defects of the traditional mechanical conversion mechanism such as long transmission chain, large volume, low efficiency and poor precision. The motor used in the system is equipped with an encoder, so the signal collected from the encoder can be used as system feedback to construct a closed-loop control system to realize servo control. The overall scheme of this system is as follows: even if two direct-drive servo motors are used to directly drive the load to achieve angle tracking and positioning, and the built-in encoder of the direct-drive motor is used to detect the rotation angle of the servo motor in real time to achieve accurate position, speed and acceleration. Closed-loop control.

第二，设计两个电控伺服调焦镜头结构，分别将两台相机与镜头固定至第一步所述转台上方，两转台可放置最远距离为80米。然后将激光手电筒安装至第三台转台上，并将此转台置于上述两转台中间。Second, two electronically controlled servo focusing lens structures are designed, and the two cameras and lenses are respectively fixed to the top of the turntable described in the first step, and the two turntables can be placed at a maximum distance of 80 meters. Then install the laser flashlight on the third turntable, and place this turntable in the middle of the above two turntables.

第三，利用跟踪运算器将两台相机所传入的视频流分别进行基于KCF算法的目标跟踪处理，跟踪器对每台相机处理每一帧图像，都会向控制器传入一个像素点坐标。如图3所示的闭环控制回路的原理图，每一帧图像产生的像素点坐标作为反馈变量，都会使系统产生新的误差。再通过设计好的自适应PID回路，使得每一帧图像都能对应伺服电机的每一次的速度输出，将离散的速度输出过程连续化就能够得到稳定的输出结果，实现转台对目标的稳定跟踪。Third, the tracking operator is used to perform target tracking processing based on the KCF algorithm on the video streams input from the two cameras. The tracker processes each frame of image for each camera and transmits a pixel coordinate to the controller. As shown in the schematic diagram of the closed-loop control loop shown in Figure 3, the pixel coordinates generated by each frame of image are used as feedback variables, which will cause the system to generate new errors. Through the designed adaptive PID loop, each frame of image can correspond to each speed output of the servo motor, and the discrete speed output process can be continuous to obtain stable output results and realize the stable tracking of the target by the turntable. .

第四，利用张氏标定法得到相机参数，标定和建立图像坐标系与世界坐标系的关系矩阵，利用如图4的双目立体视觉技术将每一帧运动目标的像素坐标转换成世界坐标。双目机器视觉是利用仿真生物视觉系统的原理，使用双摄像机从不同的角度获取同一三维场景的两幅数字图像，通过立体匹配计算两幅图像像素间的位置偏差(即视差)来获取该三维场景的几何信息与深度信息，并重建该场景的三维形状与位置。它的原理如图4所示，O_l，O_r分别为左右相机的光心位置，该位置可通过转台转动角度根据几何关系精确求得，T为左右相机光心之间的距离即基线距离，f为相机焦距，左右相同，P为待测目标点，Z为待测目标点到基线的距离，即待测目标点的深度值。(x^l,y^l),(x^r,y^r)分别为P点在左右相机的像点坐标，由以下三角形相似关系可得出待测目标点的精确坐标。Fourth, use Zhang's calibration method to obtain camera parameters, calibrate and establish the relationship matrix between the image coordinate system and the world coordinate system, and use the binocular stereo vision technology as shown in Figure 4 to convert the pixel coordinates of each frame of moving objects into world coordinates. Binocular machine vision is to use the principle of simulated biological vision system, use dual cameras to obtain two digital images of the same three-dimensional scene from different angles, and calculate the positional deviation (ie parallax) between the two image pixels through stereo matching to obtain the three-dimensional image. Geometry and depth information of the scene, and reconstruct the 3D shape and position of the scene. Its principle is shown in Figure 4, where O_l and_Or are the positions of the optical centers of the left and right cameras, respectively, which can be accurately obtained according to the geometric relationship through the rotation angle of the turntable, and T is the distance between the optical centers of the left and right cameras, that is, the baseline distance , f is the focal length of the camera, the left and right are the same, P is the target point to be measured, Z is the distance from the target point to be measured to the baseline, that is, the depth value of the target point to be measured. (x^l , y^l ), (x^r , y^r ) are the image coordinates of point P on the left and right cameras respectively. The exact coordinates of the target point to be measured can be obtained from the following triangular similarity relationship.

最后可以利用世界坐标系与步骤三转台的转角关系，经过投影计算可让转台投影系统复现运动目标的运动状态和轨迹。Finally, the relationship between the world coordinate system and the turntable in step 3 can be used to make the projection system of the turntable reproduce the motion state and trajectory of the moving target through projection calculation.

综上所述，以上仅为本发明的较佳实施例而已，并非用于限定本发明的保护范围。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。To sum up, the above are only preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims (4)

Translated fromChinese

1.一种基于双目视频传感器的高精度三维运动轨迹采集与运动过程再现方法，其特征在于：1. a high-precision three-dimensional motion track collection and motion process reproduction method based on binocular video sensor, is characterized in that:1，包含三台高精度二维转台，并通过控制器，驱动器，运算器组成高精度伺服控制系统。1. It contains three high-precision two-dimensional turntables, and forms a high-precision servo control system through controllers, drivers and operators.2，包含两个电控伺服调焦镜头，分别将两台相机与镜头固定至第一步所述转台上。2. It includes two electronically controlled servo focusing lenses, respectively fixing the two cameras and lenses to the turntable described in the first step.3，将激光器或其他投影装置置于第三台转台上构成投影系统。3. Place a laser or other projection device on the third turntable to form a projection system.4，利用视频目标跟踪算法对相机传入的图片目标位置进行跟踪，利用自适应控制算法，将跟踪得到的目标中心点与图像中心点偏差量做为反馈量，控制转台转动从而稳定跟踪目标运动。利用步骤二的变焦镜头，基于图像跟踪框与整幅图像作比较，使镜头稳定变焦，保持目标大小稳定，图像清晰。最终让运动目标稳定的呈现在两幅画面中心位置附近。4. Use the video target tracking algorithm to track the target position of the picture input by the camera, and use the adaptive control algorithm to take the deviation between the tracked target center point and the image center point as the feedback amount, and control the rotation of the turntable to track the target movement stably. . Using the zoom lens in step 2, compare the entire image based on the image tracking frame, so that the lens is zoomed stably, the size of the target is kept stable, and the image is clear. Finally, the moving target is stably presented near the center of the two pictures.5，通过相机标定，获取相机的内外参并得到世界坐标系与图像坐标系间的变换矩阵，通过数学变换将同一时刻通过两台相机采集图形所获得的运动目标的像素坐标位置及两套转台系统的转角，相机光学变焦参数等变换成三维世界坐标。5. Through the camera calibration, the internal and external parameters of the camera are obtained and the transformation matrix between the world coordinate system and the image coordinate system is obtained. Through mathematical transformation, the pixel coordinate position of the moving target and the two sets of turntables obtained by collecting graphics from two cameras at the same time are converted through mathematical transformation. The rotation angle of the system, the optical zoom parameters of the camera, etc. are transformed into three-dimensional world coordinates.6，通过世界坐标系与步骤三转台的转角关系，利用插值计算让投影系统复现运动目标的运动状态和轨迹。6. Through the angle relationship between the world coordinate system and the turntable in step 3, use interpolation calculation to make the projection system reproduce the motion state and trajectory of the moving target.2.如权利要求1所述一种基于双目视频传感器的高精度定位方法，其特征在于，利用滤波跟踪技术，稳定提取运动目标在图像中所处位置。2 . The high-precision positioning method based on a binocular video sensor according to claim 1 , wherein the filter tracking technology is used to stably extract the position of the moving target in the image. 3 .3.如权利要求1所述一种基于双目视频传感器的高精度定位方法，其特征在于，应用自动控制算法实现转台控制，令目标始终位于转台所承载相机视野内，直至轨迹测量结束。3. A kind of high-precision positioning method based on binocular video sensor as claimed in claim 1 is characterized in that, applying automatic control algorithm to realize turntable control, so that the target is always located in the field of view of the camera carried by the turntable, until the end of the trajectory measurement.4.如权利要求1所述一种基于双目视频传感器的高精度定位方法，其特征在于，利用双目视觉定位技术和云台控制系统记录运动目标每个时刻的坐标位置，并能通过投影系统复现运动目标的完整运动轨迹。4. a kind of high-precision positioning method based on binocular video sensor as claimed in claim 1 is characterized in that, utilize binocular vision positioning technology and PTZ control system to record the coordinate position of each moment of moving target, and can pass projection The system reproduces the complete motion trajectory of the moving target.

Images