技术领域technical field
本发明涉及视线跟踪瞄准操控领域，尤其涉及一种视线跟踪瞄准操控系统及方法。The present invention relates to the field of line-of-sight tracking and aiming control, in particular to a line-of-sight tracking and aiming control system and method.
背景技术Background technique
视觉是人们获取信息的主要方式，由于视线具有直接性、自然性和双向性等特点，使得视线跟踪技术在人机交互、人机协作、商场购物注意力分析、驾驶员行为分析、广告网页等用户心理及行为分析等多个领域有着广泛的应用。但目前现有的视线跟踪技术多应用于跟踪视线在显示屏幕或场景摄像机拍摄图像视频等二维平面上的位置，直接跟踪使用者在真实空间中三维注视点位置坐标的系统及方法较少。特别是在让使用者直接用眼睛操控机器设备对真实空间中的目标进行瞄准锁定跟踪的领域，暂时还没有相关的系统及方法。Vision is the main way for people to obtain information. Due to the characteristics of directness, naturalness and bidirectionality of sight, sight tracking technology is widely used in human-computer interaction, human-computer collaboration, shopping mall attention analysis, driver behavior analysis, advertising web pages, etc. It has a wide range of applications in many fields such as user psychology and behavior analysis. However, the existing gaze tracking technology is mostly used to track the position of the gaze on a two-dimensional plane such as a display screen or a scene camera to capture images and videos, and there are few systems and methods that directly track the coordinates of the user's three-dimensional gaze point in real space. Especially in the field of allowing the user to directly use the eyes to control the machine equipment to target and lock the target in the real space, there is no related system and method for the time being.
发明内容Contents of the invention
基于现有技术所存在的问题，本发明的目的是提供一种视线跟踪瞄准操控系统及方法，能让使用者用眼睛操控机器设备快速瞄准锁定跟踪所注视的目标。Based on the problems existing in the prior art, the object of the present invention is to provide a sight-tracking and aiming control system and method, which allow the user to quickly aim, lock, and track the gazed target by manipulating the machine with the eyes.
本发明的目的是通过以下技术方案实现的：The purpose of the present invention is achieved through the following technical solutions:
本发明实施方式提供一种视线跟踪瞄准操控系统，包括：An embodiment of the present invention provides a line-of-sight tracking and aiming control system, including:
视线跟踪设备，能跟踪计算在该视线跟踪设备坐标系下使用者视线方向及注视点位置坐标；Eye-tracking device capable of tracking and calculating the user's line-of-sight direction and gaze point coordinates in the coordinate system of the eye-tracking device;
相对位置姿态获取单元，分别与视线跟踪设备和瞄准操控设备通信连接，能获取所述视线跟踪设备坐标系和所述瞄准操控设备坐标系之间的相对位置姿态；The relative position and posture acquisition unit is respectively connected to the line of sight tracking device and the aiming control device in communication, and can obtain the relative position and posture between the coordinate system of the line of sight tracking device and the coordinate system of the aiming and control device;
瞄准操控设备，分别与所述视线跟踪设备和相对位置姿态获取单元通信连接，能根据所述视线跟踪设备提供的视线方向及注视点位置坐标和所述相对位置姿态获取单元提供的相对位置姿态计算得出使用者注视点目标在该瞄准操控设备坐标系下的方向和位置坐标，并通过计算出的方向和位置坐标操控设备瞄准锁定跟踪使用者注视点目标。The aiming control device is connected to the line of sight tracking device and the relative position and posture acquisition unit respectively, and can be calculated according to the line of sight direction and gaze point position coordinates provided by the line of sight tracking device and the relative position and posture provided by the relative position and posture acquisition unit Obtain the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming control device, and use the calculated direction and position coordinates to manipulate the device to aim, lock and track the user's gaze point target.
本发明实施方式还提供一种视线跟踪瞄准操控方法，采用本发明所述的视线跟踪瞄准操控系统，包括以下步骤：The embodiment of the present invention also provides a line-of-sight tracking and aiming control method, using the line-of-sight tracking and aiming control system of the present invention, including the following steps:
步骤(a)，通过所述系统的视线跟踪设备采集使用者的眼动数据，根据所述使用者的眼动数据计算使用者视线方向和注视点位置坐标；Step (a), collecting the user's eye movement data through the sight tracking device of the system, and calculating the user's line of sight direction and fixation point position coordinates according to the user's eye movement data;
步骤(b)，通过所述系统的相对位置姿态获取单元获取所述系统的视线跟踪设备坐标系与所述瞄准操控设备坐标系的相对位置姿态；Step (b), acquiring the relative position and posture of the line-of-sight tracking device coordinate system of the system and the aiming and manipulation device coordinate system through the relative position and posture acquisition unit of the system;
步骤(c)，由所述系统的瞄准操控设备根据所述视线方向及注视点位置坐标和相对位置姿态计算出使用者注视点目标在该瞄准操控设备坐标系下的方向和位置坐标，操控设备瞄准锁定跟踪使用者注视点目标。Step (c), the aiming control device of the system calculates the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming control device according to the line of sight direction, the gaze point position coordinates and the relative position and posture, and controls the device Aim lock tracks the target at the user's gaze point.
由上述本发明提供的技术方案可以看出，本发明实施例提供的视线跟踪瞄准操控系统及方法，其有益效果为：It can be seen from the above-mentioned technical solutions provided by the present invention that the eye-tracking aiming control system and method provided by the embodiments of the present invention have the following beneficial effects:
通过设置相互通信连接的视线跟踪设备、相对位置姿态获取单元和瞄准操控设备相互配合，可以跟踪计算出使用者视线方向及注视点位置坐标，获取视线跟踪设备坐标系和瞄准操控设备坐标系之间的相对位置姿态，两者结合进而计算出使用者注视点目标在瞄准操控设备坐标系下的方向和位置坐标，让使用者用眼睛操控机器设备快速瞄准锁定跟踪所注视的目标。本发明的系统及方法在人与机器人协作，飞机、车辆、舰船、航天飞行器等的辅助操控，VR/AR全浸入式人机交互等多个领域有广泛应用。By setting the gaze tracking device connected with each other, the relative position and posture acquisition unit and the aiming control device to cooperate with each other, the user's line of sight direction and gaze point position coordinates can be tracked and calculated, and the distance between the coordinate system of the gaze tracking device and the coordinate system of the aiming control device can be obtained. The relative position and posture of the user is combined to calculate the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming and controlling device, allowing the user to use the eyes to control the machine device to quickly aim, lock and track the gazed target. The system and method of the present invention are widely used in many fields such as human-robot collaboration, auxiliary control of aircraft, vehicles, ships, aerospace vehicles, etc., and VR/AR full-immersion human-computer interaction.
附图说明Description of drawings
为了更清楚地说明本发明实施例的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域的普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他附图。In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For Those of ordinary skill in the art can also obtain other drawings based on these drawings on the premise of not paying creative work.
图1为本发明实施例提供的视线跟踪瞄准操控系统结构图；FIG. 1 is a structural diagram of a line-of-sight tracking and aiming control system provided by an embodiment of the present invention;
图2为本发明实施例提供的视线跟踪瞄准操控方法流程图；Fig. 2 is a flow chart of the sight-tracking and aiming control method provided by the embodiment of the present invention;
图3为本发明实施例提供的视线跟踪瞄准操控系统应用场景示意图。FIG. 3 is a schematic diagram of an application scenario of the gaze tracking and aiming control system provided by the embodiment of the present invention.
具体实施方式detailed description
下面结合本发明的具体内容，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明的保护范围。The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the specific content of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.
如图1所示，本发明实施例提供一种视线跟踪瞄准操控系统，可以让使用者用眼睛操控机器设备快速瞄准锁定跟踪所注视的目标，包括：As shown in Figure 1, the embodiment of the present invention provides a line of sight tracking and aiming control system, which allows the user to use the eyes to control the machine to quickly aim, lock and track the target, including:
视线跟踪设备，能跟踪计算在该视线跟踪设备坐标系下使用者视线方向及注视点位置坐标；Eye-tracking device capable of tracking and calculating the user's line-of-sight direction and gaze point coordinates in the coordinate system of the eye-tracking device;
相对位置姿态获取单元，分别与视线跟踪设备和瞄准操控设备通信连接，能获取所述视线跟踪设备坐标系和所述瞄准操控设备坐标系之间的相对位置姿态；The relative position and posture acquisition unit is respectively connected to the line of sight tracking device and the aiming control device in communication, and can obtain the relative position and posture between the coordinate system of the line of sight tracking device and the coordinate system of the aiming and control device;
瞄准操控设备，分别与所述视线跟踪设备和相对位置姿态获取单元通信连接，能根据所述视线跟踪设备提供的视线方向及注视点位置坐标和所述相对位置姿态获取单元提供的相对位置姿态计算得出使用者注视点目标在该瞄准操控设备坐标系下的方向和位置坐标，并通过计算出的方向和位置坐标操控设备瞄准锁定跟踪使用者注视点目标,使用者注视点目标是指使用者注视点处的目标。The aiming control device is connected to the line of sight tracking device and the relative position and posture acquisition unit respectively, and can be calculated according to the line of sight direction and gaze point position coordinates provided by the line of sight tracking device and the relative position and posture provided by the relative position and posture acquisition unit Obtain the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming control device, and control the device to aim and lock to track the user's gaze point target through the calculated direction and position coordinates. The user's gaze point target refers to the user's gaze point target. The target at the gaze point.
上述系统还包括：瞄准校正单元，与所述瞄准操控设备通信连接，对使用者注视点目标进行标定校准。设置该瞄准校正单元能提高系统的视线跟踪瞄准锁定跟踪精度。The above-mentioned system further includes: an aiming correction unit, connected in communication with the aiming control device, to calibrate and calibrate the target of the user's fixation point. Setting the aiming correction unit can improve the line-of-sight tracking aiming locking tracking accuracy of the system.
上述系统中，视线跟踪设备包括：In the above system, the gaze tracking device includes:
眼动数据采集装置和视线方向及注视点的位置坐标估算单元；其中，An eye movement data collection device and a line of sight direction and a location coordinate estimation unit of a gaze point; wherein,
所述眼动数据采集装置，能采集使用者单眼或者双眼的眼动数据；The eye movement data collection device can collect eye movement data of one or both eyes of the user;
所述的视线方向及注视点位置坐标估算单元，与所述眼动数据采集装置通信连接，能从所述眼动数据采集装置采集的眼动数据中提取与视线方向及注视点相关的眼动参数信息，并通过眼球模型和标定训练建立眼动参数信息和视线方向及注视点之间的映射关系，再结合眼动参数和映射关系计算得出使用者单眼或双眼视线方向和注视点目标在该视线跟踪设备坐标系下的位置坐标。The gaze direction and fixation point position coordinate estimation unit is connected to the eye movement data acquisition device in communication, and can extract eye movements related to the gaze direction and fixation point from the eye movement data collected by the eye movement data acquisition device parameter information, and establish the mapping relationship between eye movement parameter information, line of sight direction and fixation point through eyeball model and calibration training, and then combine the eye movement parameters and mapping relationship to calculate the user's monocular or binocular line of sight direction and fixation point target. The gaze tracks the position coordinates in the device coordinate system.
上述系统中，若所述视线跟踪设备与所述瞄准操控设备的位置能相对移动，所述相对位置姿态获取单元采用：标定定位单元、惯性导航定位单元、卫星定位单元中的至少一种；In the above system, if the positions of the sight tracking device and the aiming control device can move relatively, the relative position and attitude acquisition unit adopts at least one of: a calibration positioning unit, an inertial navigation positioning unit, and a satellite positioning unit;
若所述视线跟踪设备与所述瞄准操控设备的位置不能相对移动，所述相对位置姿态获取单元采用：测量确定单元或输入接收单元。If the positions of the gaze tracking device and the aiming control device cannot move relative to each other, the relative position and posture acquisition unit adopts: a measurement determination unit or an input receiving unit.
具体的，当系统包括的视线跟踪设备与瞄准操控设备会有相对移动时，所述的相对位置姿态获取单元包括但不限于标定定位单元、惯性导航定位单元、卫星定位单元等相对位置姿态获取单元，且可以融合多个测算单元的数据，在室内外等范围内实现更高的测算精度和鲁棒性。当视线跟踪设备与瞄准操控设备安装固定在同一平台上无相对运动时，所述的视线跟踪设备与瞄准操控设备的相对位置姿态可以是在设计和生产安装时直接测量确定获得。Specifically, when the line-of-sight tracking device and the aiming control device included in the system move relatively, the relative position and attitude acquisition unit includes but is not limited to a calibration positioning unit, an inertial navigation positioning unit, a satellite positioning unit, and other relative position and attitude acquisition units. , and the data of multiple measurement units can be fused to achieve higher measurement accuracy and robustness in indoor and outdoor areas. When the line-of-sight tracking device and the aiming control device are installed and fixed on the same platform without relative movement, the relative position and posture of the line-of-sight tracking device and the aiming control device can be directly measured and determined during design, production and installation.
上述系统中，标定定位单元包括：In the above system, the calibration positioning unit includes:
标记点装置和信号发射感知装置，分别安装在所述视线跟踪设备和所述瞄准操控设备上，其中一个设备上安装标记点装置，另一个设备安装信号发射感知装置。The marking point device and the signal emission sensing device are respectively installed on the line of sight tracking device and the aiming control device, wherein a marking point device is installed on one device, and a signal emission sensing device is installed on the other device.
上述系统中，标记点装置为：按照一定结构排列安装在一个设备上的多个标定点装置，并已知各标定点装置在所安装的设备坐标系下的位置坐标；In the above system, the marking point device is: a plurality of calibration point devices installed on a piece of equipment arranged according to a certain structure, and the position coordinates of each calibration point device under the coordinate system of the installed equipment are known;
所述信号发射感知装置，能发射或接收信号，确定各标定点装置在该信号发射感知装置坐标系下的位置方向或坐标，该信号发射装置坐标系在所安装的设备平台坐标系下的位置姿态和旋转平移矩阵已知。优选的，上述标定定位单元采用摄像头标定、激光扫描标定、光源定位、声源定位、电磁信号源定位中的至少一种。The signal transmitting sensing device can transmit or receive signals, determine the position direction or coordinates of each calibration point device under the coordinate system of the signal transmitting sensing device, and the position of the signal transmitting device coordinate system under the installed equipment platform coordinate system The pose and rotation-translation matrices are known. Preferably, the above-mentioned calibration and positioning unit adopts at least one of camera calibration, laser scanning calibration, light source localization, sound source localization, and electromagnetic signal source localization.
具体的，所述的标定定位单元包括安装在某一设备上的标记点装置和安装在另一设备上的信号发射感知装置。所述的标记点装置是按照一定结构排列安装在设备上的多个标记点装置，已知各点在安装设备坐标系下的位置坐标。所述的安装在另一设备上的信号发射感知装置可以发射或接受一定信号，确定多个标记点装置在信号发射感知装置坐标系下的位置方向或坐标。该信号发射装置坐标系在安装设备平台坐标系下的位置姿态和旋转平移矩阵已知。所述的标定定位单元可以通过信号的发射感知，确定标记点装置的多个标定点在信号发射感知装置坐标系下的位置方向或坐标，结合多个标定点在其安装设备坐标系下的位置坐标信息可以测算出两设备坐标系之间的旋转平移矩阵。Specifically, the calibration and positioning unit includes a marking point device installed on a certain device and a signal emission sensing device installed on another device. The marking point device is a plurality of marking point devices arranged and installed on the equipment according to a certain structure, and the position coordinates of each point in the coordinate system of the installation equipment are known. The signal emission sensing device installed on another device can transmit or receive a certain signal to determine the position, direction or coordinates of multiple marking point devices in the coordinate system of the signal emission sensing device. The position, attitude and rotation-translation matrix of the coordinate system of the signal transmitting device under the coordinate system of the installation equipment platform are known. The calibration and positioning unit can determine the position direction or coordinates of multiple calibration points of the marking point device under the coordinate system of the signal transmission sensing device through the transmission perception of the signal, and combine the positions of multiple calibration points under the coordinate system of the installation equipment The coordinate information can measure and calculate the rotation and translation matrix between the two device coordinate systems.
所述的标定定位单元确定多个标记点装置在信号发射感知装置坐标系下的位置方向或坐标的方法可以是基于摄像头标定、激光扫描、光源定位、声源定位、电磁信号源定位等方法中的至少一种方法实现，但不限于以上方式。The method for the calibration and positioning unit to determine the position, direction or coordinates of multiple marking point devices in the coordinate system of the signal emission sensing device may be based on methods such as camera calibration, laser scanning, light source positioning, sound source positioning, and electromagnetic signal source positioning. At least one method of implementation, but not limited to the above methods.
上述系统中，惯性导航定位单元包括：在视线跟踪设备和瞄准操控设备上分别安装的惯性导航定位设备，通过各惯性导航定位设备分别测算出视线跟踪设备坐标系和瞄准操控设备坐标系相对于大地坐标系的变化，准确计算出视线跟踪设备坐标系和瞄准操控设备坐标系的相对位置姿态；具体的，惯性导航定位单元包括加速度计、陀螺仪、磁力计等部分，在视线跟踪设备和瞄准操控设备上均安装惯性导航定位设备。所述的惯性导航单元可以融合三轴加速度、角速度和磁场方向等信息测算出视线跟踪设备坐标系和瞄准操控设备坐标系相对于大地坐标系的变化，进而在短时间内可以准确的计算出视线跟踪设备坐标系和瞄准操控设备坐标系的相对位置姿态。In the above system, the inertial navigation and positioning unit includes: inertial navigation and positioning equipment respectively installed on the line of sight tracking equipment and the aiming control equipment, through which the coordinate system of the line of sight tracking equipment and the coordinate system of the aiming and control equipment are respectively measured and calculated relative to the earth. The change of the coordinate system can accurately calculate the relative position and attitude of the coordinate system of the eye-tracking equipment and the coordinate system of the aiming and controlling equipment; specifically, the inertial navigation and positioning unit includes accelerometers, gyroscopes, magnetometers and other parts, which are used in the line-of-sight tracking equipment and aiming and controlling equipment. Inertial navigation and positioning equipment are installed on the equipment. The inertial navigation unit can fuse information such as triaxial acceleration, angular velocity, and magnetic field direction to measure and calculate changes in the line-of-sight tracking device coordinate system and the aiming control device coordinate system relative to the earth coordinate system, and then accurately calculate the line-of-sight in a short time Track the relative position and attitude of the device coordinate system and aim to manipulate the device coordinate system.
所述卫星定位单元包括：在视线跟踪设备和瞄准操控设备上分别安装的卫星定位设备，通过卫星定位设备进行卫星定位测算出视线跟踪设备坐标系和瞄准操控设备坐标系在卫星系统坐标系下的位置姿态，计算出视线跟踪设备坐标系和瞄准操控设备坐标系相对位置姿态。具体的，卫星定位单元基于GPS、北斗等全球卫星导航定位系统实现，在在视线跟踪设备和瞄准操控设备上均安装卫星定位设备。该单元可以定位测算出视线跟踪设备坐标系和瞄准操控设备坐标系在卫星系统坐标系下的位置姿态，进而可以计算出视线跟踪设备坐标系和瞄准操控设备坐标系相对位置姿态。The satellite positioning unit includes: satellite positioning equipment respectively installed on the line of sight tracking equipment and the aiming control equipment, the coordinate system of the line of sight tracking equipment and the coordinate system of the aiming control equipment are under the satellite system coordinate system through satellite positioning measurement and calculation by the satellite positioning equipment Position and posture, calculate the relative position and posture of the eye-tracking device coordinate system and the aiming and manipulation device coordinate system. Specifically, the satellite positioning unit is implemented based on global satellite navigation and positioning systems such as GPS and Beidou, and satellite positioning equipment is installed on both the line-of-sight tracking equipment and the aiming control equipment. This unit can locate and calculate the position and attitude of the coordinate system of the eye-tracking equipment and the coordinate system of the aiming and controlling equipment in the satellite system coordinate system, and then can calculate the relative position and attitude of the coordinate system of the eye-tracking equipment and the coordinate system of the aiming and controlling equipment.
上述系统中，瞄准操控设备包括：机器平台和瞄准锁定跟踪装置；其中，In the above system, the aiming control equipment includes: a machine platform and an aiming locking and tracking device; wherein,
所述机器平台，与所述视线跟踪设备通信连接，获取使用者的视线方向及注视点位置坐标，结合相对位置姿态信息计算出使用者注视点目标在该瞄准操控设备坐标系下的方向和位置坐标；所述的机器平台还具有静止、地面移动、地下移动、空中飞行、水面移动、水下移动等功能中的至少一种；The machine platform communicates with the gaze tracking device, obtains the user's gaze direction and gaze point position coordinates, and calculates the direction and position of the user's gaze point target in the coordinate system of the aiming control device in combination with the relative position and posture information coordinates; the machine platform also has at least one of the functions of stationary, ground movement, underground movement, air flight, water surface movement, and underwater movement;
所述瞄准锁定跟踪装置包括：云台和瞄准装置；其中，所述云台装置安装在所述机器平台上，所述瞄准装置安装在所述云台装置上；该瞄准锁定跟踪装置与所述机器平台通信连接，能根据测算出的使用者注视点目标在该瞄准操控设备坐标系下的方向和位置坐标，通过所述云台装置控制瞄准装置在所述机器平台上前后左右运动，上下左右旋转，瞄准锁定跟踪使用者注视点目标。所述的瞄准装置可以是摄像头、激光笔、机器设备等装置中的任意一种或多种。The aiming locking and tracking device comprises: a pan-tilt and an aiming device; wherein, the said pan-tilt is installed on the machine platform, and the aiming is installed on the said pan-tilt; The communication connection of the machine platform can control the aiming device to move forward, backward, left, and right on the machine platform through the pan-tilt device according to the calculated direction and position coordinates of the user's gaze point target in the coordinate system of the aiming control device, up, down, left, and right Rotate, Aim Lock tracks the user's gaze point target. The aiming device may be any one or more of devices such as a camera, a laser pointer, and a machine device.
如图2所示，本发明实施例还提供一种视线跟踪瞄准操控方法，采用上述的视线跟踪瞄准操控系统，包括以下步骤：As shown in FIG. 2 , an embodiment of the present invention also provides a line-of-sight tracking and targeting control method, using the above-mentioned line-of-sight tracking and targeting control system, including the following steps:
步骤(a)，通过所述系统的视线跟踪设备采集使用者的眼动数据，根据所述使用者的眼动数据计算出使用者视线方向和注视点位置坐标；Step (a), collecting the user's eye movement data through the sight tracking device of the system, and calculating the user's line of sight direction and fixation point position coordinates according to the user's eye movement data;
步骤(b)，通过所述系统的相对位置姿态获取单元获取所述系统的视线跟踪设备坐标系与所述瞄准操控设备坐标系的相对位置姿态；Step (b), acquiring the relative position and posture of the line-of-sight tracking device coordinate system of the system and the aiming and manipulation device coordinate system through the relative position and posture acquisition unit of the system;
步骤(c)，由所述系统的瞄准操控设备根据所述视线方向及注视点位置坐标和相对位置姿态计算出使用者注视点目标在该瞄准操控设备坐标系下的方向和位置坐标，操控设备瞄准锁定跟踪使用者注视点目标。Step (c), the aiming control device of the system calculates the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming control device according to the line of sight direction, the gaze point position coordinates and the relative position and posture, and controls the device Aim lock tracks the target at the user's gaze point.
进一步的，上述方法还包括步骤(d)，通过所述系统的瞄准校正单元对使用者注视点目标进行标定校准，提高对注视点目标的瞄准锁定跟踪精度。Further, the above method further includes step (d), using the aiming correction unit of the system to calibrate and calibrate the user's fixation point target, so as to improve the aiming locking and tracking accuracy of the fixation point target.
下面对本发明实施例具体作进一步地详细描述。The embodiments of the present invention will be further described in detail below.
本实施例提供一种视线跟踪瞄准操控系统，可以跟踪计算出使用者视线方向及注视点位置坐标，测算出的视线跟踪设备坐标系和瞄准操控设备坐标系之间的相对位置姿态，两者结合进而计算出使用者注视点目标在瞄准操控设备坐标系下的方向和位置坐标，让使用者用眼睛操控机器设备快速瞄准锁定跟踪所注视的目标。本发明在人与机器人协作，飞机、车辆、舰船、航天飞行器等的辅助操控，VR/AR全浸入式人机交互等多个领域有广泛应用。This embodiment provides a line of sight tracking and aiming control system, which can track and calculate the direction of the user's line of sight and the position coordinates of the point of gaze, and measure the relative position and posture between the coordinate system of the line of sight tracking equipment and the coordinate system of the aiming and control equipment. Then calculate the direction and position coordinates of the target of the user's gaze point in the coordinate system of the aiming and controlling device, so that the user can use the eyes to control the machine and equipment to quickly aim, lock and track the target that he is looking at. The invention is widely used in many fields such as human-robot collaboration, auxiliary control of aircraft, vehicles, ships, aerospace vehicles, etc., and VR/AR full-immersion human-computer interaction.
如图1所示，所述的视线跟踪瞄准操控系统，包括：As shown in Figure 1, the line of sight tracking and aiming control system includes:
视线跟踪设备，可以跟踪计算在该设备坐标系下使用者视线方向及注视点位置坐标；Gaze tracking device, which can track and calculate the user's gaze direction and gaze point position coordinates in the device coordinate system;
相对位置姿态获取单元，可以测算视线跟踪设备坐标系和瞄准操控设备坐标系之间的相对位置姿态；The relative position and attitude acquisition unit can measure the relative position and attitude between the coordinate system of the sight tracking device and the coordinate system of the aiming and controlling device;
瞄准操控设备，可以根据所述的视线方向及注视点位置坐标和相对位置姿态等计算出使用者注视点目标在瞄准操控设备坐标系下的方向和位置坐标，从而操控设备瞄准锁定跟踪使用者注视点目标。The aiming control device can calculate the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming control device according to the line of sight direction, the position coordinates of the gaze point, and the relative position and posture, so as to control the device to aim, lock and track the user's gaze. point target.
此外，该视线跟踪瞄准操控系统还可以包括瞄准校正单元，通过使用者注视已知目标进行标定校准，提高系统的视线跟踪瞄准锁定跟踪精度。In addition, the line-of-sight tracking and aiming control system may also include an aiming correction unit, which can be calibrated and calibrated by the user looking at a known target, so as to improve the line-of-sight tracking, aiming and locking tracking accuracy of the system.
所述的视线跟踪设备包括眼动数据采集装置和视线方向及注视点的位置坐标估算单元。The gaze tracking device includes an eye movement data collection device and a gaze direction and gaze point position coordinate estimation unit.
所述的眼动数据采集装置，可以采集使用者单眼或者双眼的眼动数据，包括但不限于眼部区域的可见光、红外光等视频数据、眼球运动引起的眼部电压变化数据和其他向眼部区域主动发射接收电磁场或者红外光间接获得的眼动数据等；The eye movement data collection device can collect eye movement data of one eye or both eyes of the user, including but not limited to video data such as visible light and infrared light in the eye area, eye voltage change data caused by eye movement, and other eye movement data. Eye movement data obtained indirectly by actively transmitting and receiving electromagnetic fields or infrared light in the internal area;
所述的视线方向及注视点位置坐标估算单元，可以从眼动数据中提取与视线方向及注视点相关的眼动参数信息，通过眼球模型和标定训练的方法建立眼动参数信息和视线方向及注视点之间的映射关系，再结合眼动参数和映射关系估算使用者单眼或双眼视线方向和注视点目标在视线跟踪设备坐标系下的位置坐标。The gaze direction and fixation point position coordinate estimation unit can extract eye movement parameter information related to the gaze direction and fixation point from the eye movement data, and establish eye movement parameter information, gaze direction and The mapping relationship between the fixation points, combined with the eye movement parameters and the mapping relationship, is used to estimate the user's monocular or binocular gaze direction and the position coordinates of the fixation point target in the coordinate system of the gaze tracking device.
所述的眼动参数信息可以是基于眼部视频数据提取的使用者双眼或单眼的瞳孔、光斑、眼角、角膜球心的二维或三维坐标和瞳孔椭圆平面法向量等眼球参数，也可以是基于眼部电压变化数据提取的电压值和波形等。The eye movement parameter information can be eyeball parameters such as pupils, faculae, eye corners, corneal sphere center two-dimensional or three-dimensional coordinates and pupil ellipse plane normal vectors extracted based on eye video data, and can also be The voltage value and waveform extracted based on the eye voltage change data, etc.
所述的建立映射的方法，可以是单独基于眼部视线模型或基于标定训练的方法，也可以是眼部视线模型和标定训练相结合的方法。The method for establishing the mapping may be based solely on the eye sight model or calibration training, or may be a combination of the eye sight model and calibration training.
此外，所述的视线跟踪设备，可以是随使用者身体或头部运动的穿戴式设备，包括但不限于眼镜、头盔等设备，也可以是不随使用者运动的桌面式设备，可以安装或固定在其他物体设备上，安装或固定的设备包括但不限于桌面、显示器、车辆、飞机、机器人平台等设备。In addition, the gaze tracking device can be a wearable device that moves with the user's body or head, including but not limited to glasses, helmets, etc., or it can be a desktop device that does not move with the user, and can be installed or fixed On other physical devices, installed or fixed devices include but are not limited to desktops, monitors, vehicles, aircraft, robot platforms and other devices.
所述的相对位置姿态获取单元，当系统包括的视线跟踪设备与瞄准操控设备会有相对移动时，所述的相对位置姿态获取单元包括但不限于标定定位单元、惯性导航定位单元、卫星定位单元等相对位置姿态获取单元，且可以融合多个测算单元的数据，在室内外等范围内实现更高的测算精度和鲁棒性。当视线跟踪设备与瞄准操控设备安装固定在同一平台上无相对运动时，所述的视线跟踪设备与瞄准操控设备的相对位置姿态可以是在设计和生产安装时直接测量确定。The relative position and posture acquisition unit, when the line of sight tracking device and the aiming control device included in the system will move relative to each other, the relative position and posture acquisition unit includes but is not limited to a calibration positioning unit, an inertial navigation positioning unit, a satellite positioning unit Equivalent relative position and attitude acquisition unit, and can fuse the data of multiple measurement and calculation units to achieve higher measurement accuracy and robustness in indoor and outdoor areas. When the line-of-sight tracking device and the aiming control device are installed and fixed on the same platform without relative movement, the relative position and posture of the line-of-sight tracking device and the aiming control device can be directly measured and determined during design, production and installation.
所述的标定定位单元包括安装在某一设备上的标记点装置和安装在另一设备上的信号发射感知装置。所述的标记点装置是按照一定结构排列安装在设备上的多个标记点装置，已知各点在安装设备坐标系下的位置坐标。所述的安装在另一设备上的信号发射感知装置可以发射或接受一定信号，确定多个标记点装置在信号发射感知装置坐标系下的位置方向或坐标。该信号发射装置坐标系在安装设备平台坐标系下的位置姿态和旋转平移矩阵已知。所述的标定定位单元可以通过信号的发射感知，确定标记点装置的多个标定点在信号发射感知装置坐标系下的位置方向或坐标，结合多个标定点在其安装设备坐标系下的位置坐标信息可以测算出两设备坐标系之间的旋转平移矩阵。The calibration and positioning unit includes a marking point device installed on a certain device and a signal emission sensing device installed on another device. The marking point device is a plurality of marking point devices arranged and installed on the equipment according to a certain structure, and the position coordinates of each point in the coordinate system of the installation equipment are known. The signal emission sensing device installed on another device can transmit or receive a certain signal to determine the position, direction or coordinates of multiple marking point devices in the coordinate system of the signal emission sensing device. The position, attitude and rotation-translation matrix of the coordinate system of the signal transmitting device under the coordinate system of the installation equipment platform are known. The calibration and positioning unit can determine the position direction or coordinates of multiple calibration points of the marking point device under the coordinate system of the signal transmission sensing device through the transmission perception of the signal, and combine the positions of multiple calibration points under the coordinate system of the installation equipment The coordinate information can measure and calculate the rotation and translation matrix between the two device coordinate systems.
所述的标定定位单元确定多个标记点装置在信号发射感知装置坐标系下的位置方向或坐标的方法可以是基于摄像头标定、激光扫描、光源定位、声源定位、电磁信号源定位等方法中的至少一种方法实现，但不限于以上方法。The method for the calibration and positioning unit to determine the position, direction or coordinates of multiple marking point devices in the coordinate system of the signal emission sensing device may be based on methods such as camera calibration, laser scanning, light source positioning, sound source positioning, and electromagnetic signal source positioning. At least one method of implementation, but not limited to the above methods.
所述的基于摄像头标定的方法中的标记点装置中的标定点可以是LED灯、灰度颜色差别大的圆点或棋盘格交点，信号发射感知装置可以是单个或多个摄像头。LED灯和摄像头的感应或发射的光波长可以是可见光、红外光、紫外光波段等，但必须相互对应。通过图像处理的方法可以获得标定点在摄像头拍摄图像中的位置，结合摄像头的成像模型可以获取标定点其在摄像头坐标系下的位置方向或坐标，结合多个标定点在其安装设备坐标系下的位置坐标信息可以测算出两设备坐标系之间的旋转平移矩阵。The calibration points in the marking point device in the camera-based calibration method can be LED lights, dots with large grayscale and color differences, or checkerboard intersections, and the signal emission sensing device can be a single or multiple cameras. The wavelengths of light sensed or emitted by LED lights and cameras can be visible light, infrared light, ultraviolet light, etc., but they must correspond to each other. The position of the calibration point in the image captured by the camera can be obtained through image processing, and the position, direction or coordinates of the calibration point in the camera coordinate system can be obtained by combining the imaging model of the camera, and multiple calibration points can be obtained in the coordinate system of the installation device The position coordinate information can measure the rotation and translation matrix between the two device coordinate systems.
所述的基于激光扫描的方法中的标记点装置是光电接收装置，信号发射接收装置是单个或多个激光发射器。光电接收装置和激光发射器的光波长可以是可见光、红外光、紫外光波段等，但须相互对应。所述的激光发射器按照一定规律周期性的发射覆盖全空间的光同步信号并沿激光发射器坐标系的X轴和Y轴匀速交替旋转发射线激光束，扫描覆盖整个空间。当发射覆盖全空间的光同步信号和激光束扫描到标定点的光电接收装置时，光电接收装置感应到较强的激光信号。根据接收到的激光束扫描到标定点的时间在整个同步周期中位置，可以确定光电接收装置在激光发射器坐标系中的位置方向，在多个激光发射器系统中，可以进一步确定光电接收装置的位置坐标，结合多个光电接收装置在其安装设备坐标系下的位置坐标信息可以测算出两设备坐标系之间的旋转平移矩阵。The marking point device in the method based on laser scanning is a photoelectric receiving device, and the signal transmitting and receiving device is a single or multiple laser transmitters. The light wavelengths of the photoelectric receiving device and the laser transmitter can be visible light, infrared light, ultraviolet light bands, etc., but must correspond to each other. The laser emitter periodically emits optical synchronization signals covering the entire space according to a certain rule, and alternately rotates the emission line laser beam along the X-axis and Y-axis of the laser emitter coordinate system at a uniform speed, scanning and covering the entire space. When the optical synchronization signal covering the whole space is emitted and the laser beam scans to the photoelectric receiving device at the marked point, the photoelectric receiving device senses a strong laser signal. According to the time when the received laser beam scans to the calibration point and the position in the entire synchronization period, the position and direction of the photoelectric receiving device in the laser transmitter coordinate system can be determined. In multiple laser transmitter systems, the photoelectric receiving device can be further determined. Combining the position coordinates of multiple photoelectric receiving devices in their installation equipment coordinate system, the rotation and translation matrix between the two equipment coordinate systems can be calculated.
所述的基于光源定位、声源定位、电磁信号源定位的方法中的标记点装置中的标定点是光源、声源或电磁信号发射源等，信号发射感知装置是按照一定结构排列安装在设备上的多个光信号、声信号或电磁信号接收器，已知各接收器在安装设备坐标系下的位置坐标。信号接收装置根据多个信号接收器接收到的信号的强度和方向等信息，结合多个信号接收器在其安装设备坐标系下的位置坐标信息，可以确定信号发射源在信号接收装置坐标系中的位置方向和坐标，再结合多个信号发射源在其安装设备坐标系下的位置坐标信息可以测算出两设备坐标系之间的旋转平移矩阵。In the method based on light source localization, sound source localization, and electromagnetic signal source localization, the calibration point in the marking point device is a light source, a sound source, or an electromagnetic signal emission source, etc., and the signal emission sensing device is arranged and installed on the equipment according to a certain structure. Multiple optical signal, acoustic signal or electromagnetic signal receivers on the system, the position coordinates of each receiver in the installation equipment coordinate system are known. The signal receiving device can determine that the signal transmitting source is in the coordinate system of the signal receiving device according to the information such as the strength and direction of the signal received by multiple signal receivers, combined with the position coordinate information of multiple signal receivers in the coordinate system of the installation equipment The direction and coordinates of the position, combined with the position coordinate information of multiple signal emitting sources in the coordinate system of the installation equipment, can measure the rotation and translation matrix between the two equipment coordinate systems.
所述的惯性导航定位单元包括加速度计、陀螺仪、磁力计等部分，在视线跟踪设备和瞄准操控设备上均安装惯性导航定位设备。所述的惯性导航单元可以融合三轴加速度、角速度和磁场方向等信息测算出视线跟踪设备坐标系和瞄准操控设备坐标系相对于大地坐标系的变化，进而在短时间内可以准确的计算出视线跟踪设备坐标系和瞄准操控设备坐标系的相对位置姿态。The inertial navigation and positioning unit includes accelerometers, gyroscopes, magnetometers and other parts, and inertial navigation and positioning devices are installed on the line of sight tracking equipment and the aiming control equipment. The inertial navigation unit can fuse information such as triaxial acceleration, angular velocity, and magnetic field direction to measure and calculate changes in the line-of-sight tracking device coordinate system and the aiming control device coordinate system relative to the earth coordinate system, and then accurately calculate the line-of-sight in a short time Track the relative position and attitude of the device coordinate system and aim to manipulate the device coordinate system.
所述的卫星定位单元基于GPS、北斗等全球卫星导航定位系统实现，在在视线跟踪设备和瞄准操控设备上均安装卫星定位设备。该单元可以定位测算出视线跟踪设备坐标系和瞄准操控设备坐标系在卫星系统坐标系下的位置姿态，进而可以计算出视线跟踪设备坐标系和瞄准操控设备坐标系相对位置姿态。The satellite positioning unit is realized based on global satellite navigation and positioning systems such as GPS and Beidou, and satellite positioning equipment is installed on the line of sight tracking equipment and the aiming control equipment. This unit can locate and calculate the position and attitude of the coordinate system of the eye-tracking equipment and the coordinate system of the aiming and controlling equipment in the satellite system coordinate system, and then can calculate the relative position and attitude of the coordinate system of the eye-tracking equipment and the coordinate system of the aiming and controlling equipment.
所述的视线跟踪设备与瞄准操控设备相对位置姿态获取单元，还可以融合标定定位单元、惯性导航定位单元、卫星定位单元等部分的信息，在室内外等范围内实现更高的测算精度和鲁棒性。标定定位单元，可以在精确的计算出视线跟踪设备坐标系和机器人平台设备坐标系的相对位置姿态和旋转平移矩阵，但距离较近，基于光学的系统要在视距范围内使用。卫星定位系统的定位范围大，但测算精度较低，特别是在建筑物内等顶层有遮挡的地方难以使用。惯性导航定位单元短时间内定位精度较高，但误差会随时间累积，精度持续降低。融合标定定位单元、惯性导航定位单元、卫星定位单元等部分的数据信息，不仅可以在室内外各种环境下的视距范围外也保持较精确的测算精度，还能提高视距范围内的测算精度。The relative position and attitude acquisition unit of the line of sight tracking device and the aiming control device can also fuse the information of the calibration positioning unit, inertial navigation positioning unit, satellite positioning unit, etc., to achieve higher measurement accuracy and robustness in indoor and outdoor areas. Stickiness. The calibration and positioning unit can accurately calculate the relative position, attitude and rotation-translation matrix of the line-of-sight tracking device coordinate system and the robot platform device coordinate system, but the distance is relatively short, and the optical-based system should be used within the line-of-sight range. The positioning range of the satellite positioning system is large, but the calculation accuracy is low, especially it is difficult to use in places where the top floor of the building is sheltered. The inertial navigation positioning unit has high positioning accuracy in a short period of time, but the error will accumulate over time, and the accuracy will continue to decrease. Integrating the data information of the calibration positioning unit, inertial navigation positioning unit, satellite positioning unit, etc., not only can maintain a relatively accurate measurement accuracy outside the line-of-sight range in various indoor and outdoor environments, but also improve the measurement within the line-of-sight range precision.
所述的瞄准操控设备，包括机器平台和瞄准锁定跟踪设备。The aiming control device includes a machine platform and an aiming locking and tracking device.
所述的机器平台，可以与视线跟踪设备通信，获得使用者的视线方向及注视点位置坐标，结合相对位置姿态信息计算出使用者注视点目标在瞄准操控设备坐标系下的方向和位置坐标。所述的机器平台还具有静止、地面移动、地下移动、空中飞行、水面移动、水下移动等功能中的至少一种。The machine platform can communicate with the gaze tracking device to obtain the user's gaze direction and gaze point position coordinates, and combine the relative position and posture information to calculate the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming control device. The machine platform also has at least one of the functions of being stationary, moving on the ground, moving underground, flying in the air, moving on the water surface, and moving underwater.
所述的瞄准锁定跟踪设备，包括云台和瞄准装置。所述的云台装置安装在机器平台上，所述的瞄准装置加装在云台装置上。该瞄准锁定跟踪设备可以根据测算出的使用者注视点目标在瞄准操控设备坐标系下的方向和位置坐标，通过云台控制瞄准装置在机器平台上前后左右运动，上下左右旋转，瞄准锁定跟踪使用者注视点目标。所述的瞄准装置可以是摄像头、激光笔、机器设备等装置中的任意一种或多种。The aiming and locking tracking device includes a pan-tilt and an aiming device. The pan-tilt device is installed on the machine platform, and the aiming device is additionally installed on the pan-tilt device. The aiming lock tracking device can control the aiming device to move forward, backward, left, and right on the machine platform through the pan/tilt according to the calculated direction and position coordinates of the user's gaze point target in the coordinate system of the aiming control device, and to use for aiming lock and tracking. The reader gazes at the point target. The aiming device may be any one or more of devices such as a camera, a laser pointer, and a machine device.
此外，所述的具备移动功能的瞄准操控设备还可以根据测算出的与视线跟踪设备的相对位置姿态跟随视线跟踪设备及使用者移动和调整朝向以更好的定位和瞄准。In addition, the aiming control device with the moving function can also follow the gaze tracking device and the user to move and adjust the orientation according to the calculated relative position and posture with the gaze tracking device, so as to better locate and aim.
如图2所示，采用上述系统进行视线跟踪瞄准操控方法，包括以下步骤：As shown in Figure 2, using the above-mentioned system to perform line-of-sight tracking and aiming manipulation method includes the following steps:
步骤(a)，采集使用者的眼动数据，根据眼动数据计算出使用者视线方向和注视点位置坐标；Step (a), collecting the user's eye movement data, and calculating the user's line of sight direction and fixation point position coordinates according to the eye movement data;
步骤(b)，获取视线跟踪设备坐标系与瞄准操控设备坐标系的相对位置姿态；Step (b), obtaining the relative position and posture of the coordinate system of the sight tracking device and the coordinate system of the aiming and manipulating device;
步骤(c)，根据所述的视线方向及注视点位置坐标和相对位置姿态计算出使用者注视点目标在瞄准操控设备坐标系下的方向和位置坐标，操控设备瞄准锁定跟踪使用者注视点目标。Step (c), calculate the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming and controlling device according to the line of sight direction, the gaze point position coordinates and the relative position and posture, and the manipulation device aims at, locks and tracks the user's gaze point target .
进一步的，上述方法还包括步骤(d)，通过使用者注视已知目标进行标定校准，提高对注视点目标的瞄准锁定跟踪精度。Further, the above method further includes step (d), by performing calibration by the user staring at a known target, so as to improve the aiming-locking tracking accuracy of the gaze-point target.
图3所示为本发明实施例提供的视线跟踪瞄准操控系统的实际应用构成，其采用的是标定定位单元作为相对位置姿态获取单元2，标定定位单元由多个标记点装置21和多个信号发射接收装置22构成的，多个标记点装置21设在视线跟踪设备1，多个信号发射接收装置22设在瞄准操控设备3上，可以跟踪计算出使用者视线方向及注视点位置坐标，测算出的视线跟踪设备坐标系和瞄准操控设备坐标系之间的相对位置姿态，两者结合进而计算出使用者注视点目标在瞄准操控设备坐标系下的方向和位置坐标，让使用者用眼睛操控机器设备快速瞄准锁定跟踪所注视的目标5。该系统还可以通过使用者注视已知目标进行标定校准，提高系统的视线跟踪瞄准锁定跟踪精度。本发明的系统及方法在人与机器人协作，飞机、车辆、舰船、航天飞行器等的辅助操控，VR/AR全浸入式人机交互等多个领域有广泛应用。Fig. 3 shows the actual application composition of the line-of-sight tracking aiming control system provided by the embodiment of the present invention, which adopts a calibration positioning unit as the relative position and posture acquisition unit 2, and the calibration positioning unit is composed of a plurality of marking point devices 21 and a plurality of signals Composed of transmitting and receiving devices 22, multiple marking point devices 21 are set on the line of sight tracking device 1, and multiple signal transmitting and receiving devices 22 are set on the aiming control device 3, which can track and calculate the user's line of sight direction and gaze point position coordinates, and measure The relative position and posture between the coordinate system of the sight tracking device and the coordinate system of the aiming and controlling device are obtained, and the two are combined to calculate the direction and position coordinates of the user's gaze point target in the coordinate system of the aiming and controlling device, so that the user can control it with the eyes The equipment quickly aims at and locks on the fixed target 5 . The system can also be calibrated and calibrated by the user staring at a known target to improve the line-of-sight tracking, aiming, locking and tracking accuracy of the system. The system and method of the present invention are widely used in many fields such as human-robot collaboration, auxiliary control of aircraft, vehicles, ships, aerospace vehicles, etc., and VR/AR full-immersion human-computer interaction.
以上所述，仅为本发明较佳的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明披露的技术范围内，可轻易想到的变化或替换，都应涵盖在本发明的保护范围之内。因此，本发明的保护范围应该以权利要求书的保护范围为准。The above is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person familiar with the technical field can easily conceive of changes or changes within the technical scope disclosed in the present invention. Replacement should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the protection scope of the claims.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710365090.8ACN107014378A (en) | 2017-05-22 | 2017-05-22 | A kind of eye tracking aims at control system and method |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710365090.8ACN107014378A (en) | 2017-05-22 | 2017-05-22 | A kind of eye tracking aims at control system and method |
| Publication Number | Publication Date |
|---|---|
| CN107014378Atrue CN107014378A (en) | 2017-08-04 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710365090.8APendingCN107014378A (en) | 2017-05-22 | 2017-05-22 | A kind of eye tracking aims at control system and method |
| Country | Link |
|---|---|
| CN (1) | CN107014378A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108334191A (en)* | 2017-12-29 | 2018-07-27 | 北京七鑫易维信息技术有限公司 | Based on the method and apparatus of the determination blinkpunkt of eye movement analysis equipment |
| CN108732746A (en)* | 2018-05-29 | 2018-11-02 | 北京理工大学 | A kind of wear-type visual sight control system |
| CN108874147A (en)* | 2018-07-12 | 2018-11-23 | 深圳沃利创意工程有限公司 | A kind of VR-BOX experience interactive device |
| CN108981690A (en)* | 2018-06-07 | 2018-12-11 | 北京轻威科技有限责任公司 | Optical-inertia fusion positioning method, device and system |
| CN109445599A (en)* | 2018-11-13 | 2019-03-08 | 宁波视睿迪光电有限公司 | Interaction pen detection method and 3D interactive system |
| CN109828663A (en)* | 2019-01-14 | 2019-05-31 | 北京七鑫易维信息技术有限公司 | Determination method and device, the operating method of run-home object of aiming area |
| CN109857102A (en)* | 2019-01-21 | 2019-06-07 | 大连理工大学 | Wheeled robot formation and tracking control method based on relative position |
| CN109949655A (en)* | 2017-12-21 | 2019-06-28 | 泰勒斯公司 | Dual coordination method and system for a head mounted display system for conforming the display of aircraft pilot information to the external real world |
| CN110033652A (en)* | 2019-03-11 | 2019-07-19 | 中国民航大学 | A kind of radar dynamic object scaling method and system based on eye movement tracking |
| CN110134222A (en)* | 2018-02-02 | 2019-08-16 | 上海集鹰科技有限公司 | A kind of VR shows positioning sighting system and its positioning method of sight |
| CN110375582A (en)* | 2019-08-13 | 2019-10-25 | 北京缔科新技术研究院(有限合伙) | A kind of laser of view-based access control model information takes aim at mirror and focusing and zooming method |
| CN110554777A (en)* | 2019-09-24 | 2019-12-10 | 大连理工大学 | Eyeball gaze angle measuring method based on electromagnetic wave tracking |
| CN111190488A (en)* | 2019-12-30 | 2020-05-22 | 华为技术有限公司 | Device control method, communication device, and storage medium |
| WO2020220284A1 (en)* | 2019-04-30 | 2020-11-05 | 深圳市大疆创新科技有限公司 | Aiming control method, mobile robot and computer-readable storage medium |
| CN112332917A (en)* | 2020-10-23 | 2021-02-05 | 军事科学院系统工程研究院网络信息研究所 | Laser communication autonomous tracking and aiming method controlled by inertial navigation |
| CN112489804A (en)* | 2019-09-12 | 2021-03-12 | 北京七鑫易维信息技术有限公司 | Aiming prediction method and device |
| CN112509007A (en)* | 2020-12-14 | 2021-03-16 | 科大讯飞股份有限公司 | Real fixation point positioning method and head-wearing sight tracking system |
| CN112633128A (en)* | 2020-12-18 | 2021-04-09 | 上海影创信息科技有限公司 | Method and system for pushing information of interested object in afterglow area |
| CN113424012A (en)* | 2019-02-11 | 2021-09-21 | D·富尼 | On-board device with network-connected sighting telescope to allow multiple other devices to track target simultaneously |
| CN113473015A (en)* | 2021-06-08 | 2021-10-01 | 荣耀终端有限公司 | Holder control method and electronic equipment |
| CN113536909A (en)* | 2021-06-08 | 2021-10-22 | 吉林大学 | A method, system and device for calculating preview distance based on eye movement data |
| CN113932831A (en)* | 2021-09-24 | 2022-01-14 | 成都飞机工业（集团）有限责任公司 | Aircraft magnetic heading calibration method for laser aiming without reference object |
| CN114035186A (en)* | 2021-10-18 | 2022-02-11 | 北京航天华腾科技有限公司 | Target position tracking and indicating system and method |
| CN114500839A (en)* | 2022-01-25 | 2022-05-13 | 青岛根尖智能科技有限公司 | Vision holder control method and system based on attention tracking mechanism |
| CN115639755A (en)* | 2022-10-31 | 2023-01-24 | 歌尔科技有限公司 | Device control method, electronic device, and computer-readable storage medium |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080218436A1 (en)* | 2007-03-08 | 2008-09-11 | Lockheed Martin Corporation | Zero-lag image response to pilot head mounted display control |
| CN106168853A (en)* | 2016-06-23 | 2016-11-30 | 中国科学技术大学 | A kind of free space wear-type gaze tracking system |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080218436A1 (en)* | 2007-03-08 | 2008-09-11 | Lockheed Martin Corporation | Zero-lag image response to pilot head mounted display control |
| CN106168853A (en)* | 2016-06-23 | 2016-11-30 | 中国科学技术大学 | A kind of free space wear-type gaze tracking system |
| Title |
|---|
| 孟举等: "基于视频图像处理的头部位置跟踪方法探索", 《现代制造工程》* |
| 寇超等: "自行高炮模拟仿真实验平台设计与实现", 《火力与指挥控制》* |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109949655A (en)* | 2017-12-21 | 2019-06-28 | 泰勒斯公司 | Dual coordination method and system for a head mounted display system for conforming the display of aircraft pilot information to the external real world |
| CN109949655B (en)* | 2017-12-21 | 2022-05-13 | 泰勒斯公司 | Dual coordination method and system for head mounted display system |
| WO2019128677A1 (en)* | 2017-12-29 | 2019-07-04 | 北京七鑫易维信息技术有限公司 | Method and apparatus for determining gazing point based on eye movement analysis device |
| CN108334191A (en)* | 2017-12-29 | 2018-07-27 | 北京七鑫易维信息技术有限公司 | Based on the method and apparatus of the determination blinkpunkt of eye movement analysis equipment |
| CN110134222A (en)* | 2018-02-02 | 2019-08-16 | 上海集鹰科技有限公司 | A kind of VR shows positioning sighting system and its positioning method of sight |
| CN108732746A (en)* | 2018-05-29 | 2018-11-02 | 北京理工大学 | A kind of wear-type visual sight control system |
| CN108981690A (en)* | 2018-06-07 | 2018-12-11 | 北京轻威科技有限责任公司 | Optical-inertia fusion positioning method, device and system |
| CN108874147A (en)* | 2018-07-12 | 2018-11-23 | 深圳沃利创意工程有限公司 | A kind of VR-BOX experience interactive device |
| CN109445599A (en)* | 2018-11-13 | 2019-03-08 | 宁波视睿迪光电有限公司 | Interaction pen detection method and 3D interactive system |
| CN109828663A (en)* | 2019-01-14 | 2019-05-31 | 北京七鑫易维信息技术有限公司 | Determination method and device, the operating method of run-home object of aiming area |
| CN109857102A (en)* | 2019-01-21 | 2019-06-07 | 大连理工大学 | Wheeled robot formation and tracking control method based on relative position |
| CN109857102B (en)* | 2019-01-21 | 2021-06-15 | 大连理工大学 | A wheeled robot formation and tracking control method based on relative position |
| CN113424012A (en)* | 2019-02-11 | 2021-09-21 | D·富尼 | On-board device with network-connected sighting telescope to allow multiple other devices to track target simultaneously |
| CN110033652A (en)* | 2019-03-11 | 2019-07-19 | 中国民航大学 | A kind of radar dynamic object scaling method and system based on eye movement tracking |
| CN110033652B (en)* | 2019-03-11 | 2021-06-04 | 中国民航大学 | Radar dynamic target calibration method and system based on eye tracking |
| WO2020220284A1 (en)* | 2019-04-30 | 2020-11-05 | 深圳市大疆创新科技有限公司 | Aiming control method, mobile robot and computer-readable storage medium |
| CN110375582A (en)* | 2019-08-13 | 2019-10-25 | 北京缔科新技术研究院(有限合伙) | A kind of laser of view-based access control model information takes aim at mirror and focusing and zooming method |
| CN112489804A (en)* | 2019-09-12 | 2021-03-12 | 北京七鑫易维信息技术有限公司 | Aiming prediction method and device |
| CN110554777A (en)* | 2019-09-24 | 2019-12-10 | 大连理工大学 | Eyeball gaze angle measuring method based on electromagnetic wave tracking |
| CN111190488A (en)* | 2019-12-30 | 2020-05-22 | 华为技术有限公司 | Device control method, communication device, and storage medium |
| CN112332917A (en)* | 2020-10-23 | 2021-02-05 | 军事科学院系统工程研究院网络信息研究所 | Laser communication autonomous tracking and aiming method controlled by inertial navigation |
| CN112509007A (en)* | 2020-12-14 | 2021-03-16 | 科大讯飞股份有限公司 | Real fixation point positioning method and head-wearing sight tracking system |
| CN112509007B (en)* | 2020-12-14 | 2024-06-04 | 科大讯飞股份有限公司 | Real gaze point positioning method and head-mounted gaze tracking system |
| CN112633128A (en)* | 2020-12-18 | 2021-04-09 | 上海影创信息科技有限公司 | Method and system for pushing information of interested object in afterglow area |
| CN113473015B (en)* | 2021-06-08 | 2022-03-08 | 荣耀终端有限公司 | Holder control method and electronic equipment |
| CN113536909A (en)* | 2021-06-08 | 2021-10-22 | 吉林大学 | A method, system and device for calculating preview distance based on eye movement data |
| CN113473015A (en)* | 2021-06-08 | 2021-10-01 | 荣耀终端有限公司 | Holder control method and electronic equipment |
| CN113932831A (en)* | 2021-09-24 | 2022-01-14 | 成都飞机工业（集团）有限责任公司 | Aircraft magnetic heading calibration method for laser aiming without reference object |
| CN113932831B (en)* | 2021-09-24 | 2023-12-08 | 成都飞机工业（集团）有限责任公司 | Aircraft magnetic heading calibration method for laser aiming without reference object |
| CN114035186A (en)* | 2021-10-18 | 2022-02-11 | 北京航天华腾科技有限公司 | Target position tracking and indicating system and method |
| CN114035186B (en)* | 2021-10-18 | 2022-06-28 | 北京航天华腾科技有限公司 | Target position tracking and indicating system and method |
| CN114500839A (en)* | 2022-01-25 | 2022-05-13 | 青岛根尖智能科技有限公司 | Vision holder control method and system based on attention tracking mechanism |
| CN114500839B (en)* | 2022-01-25 | 2024-06-07 | 青岛根尖智能科技有限公司 | Visual cradle head control method and system based on attention tracking mechanism |
| CN115639755A (en)* | 2022-10-31 | 2023-01-24 | 歌尔科技有限公司 | Device control method, electronic device, and computer-readable storage medium |
| Publication | Publication Date | Title |
|---|---|---|
| CN107014378A (en) | A kind of eye tracking aims at control system and method | |
| US11042028B1 (en) | Relative pose data augmentation of tracked devices in virtual environments | |
| CN108022302B (en) | Stereo display device of Inside-Out space orientation's AR | |
| US8885177B2 (en) | Medical wide field of view optical tracking system | |
| CN110458961B (en) | Augmented reality based system | |
| CN114730210B (en) | Co-location pose estimation in a shared artificial reality environment | |
| Foxlin et al. | Flighttracker: A novel optical/inertial tracker for cockpit enhanced vision | |
| US11914762B2 (en) | Controller position tracking using inertial measurement units and machine learning | |
| US12112440B2 (en) | Mixed-reality visor for in-situ vehicular operations training | |
| US9323055B2 (en) | System and method to display maintenance and operational instructions of an apparatus using augmented reality | |
| EP3076892B1 (en) | A medical optical tracking system | |
| US20060210111A1 (en) | Systems and methods for eye-operated three-dimensional object location | |
| US10802606B2 (en) | Method and device for aligning coordinate of controller or headset with coordinate of binocular system | |
| US11017745B2 (en) | Method, computer readable storage medium having instructions, apparatus and system for calibrating augmented reality goggles in a transportation vehicle, transportation vehicle suitable for the method, and augmented reality goggles suitable for the method | |
| US20120120070A1 (en) | System and method to display maintenance and operational instructions of an apparatus using augmented reality | |
| AU2018222619A1 (en) | Device and method for real-time eye-gaze analysis | |
| US11269400B2 (en) | Positioning system | |
| KR101662032B1 (en) | UAV Aerial Display System for Synchronized with Operators Gaze Direction | |
| US11914763B1 (en) | System and method for conformal head worn display (HWD) headtracker alignment | |
| CN207164367U (en) | AR glasses and its tracing system | |
| JP4924342B2 (en) | Motion tracker device | |
| JP2016151566A (en) | Motion tracker device | |
| US20250076973A1 (en) | Wearable system with controller localization using headset cameras and controller fiducials | |
| CN208314856U (en) | A kind of system for the detection of monocular airborne target | |
| EP3819656A1 (en) | Spatial localization using augmented reality |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | Application publication date:20170804 | |
| RJ01 | Rejection of invention patent application after publication |