技术领域technical field
本发明属于人机交互技术领域，具体是涉及一种眼动控制的多模块智能设备。The invention belongs to the technical field of human-computer interaction, and in particular relates to a multi-module intelligent device controlled by eye movement.
背景技术Background technique
随着智能穿戴设备的更新迭代，我们看到越来越多的工业作业场景中出现了智能穿戴设备的身影。但是目前大部分这类智能穿戴设备的操作方向和人的身体绑定，并不能通过一种直观的人机交互方式控制其灵活运动。With the update and iteration of smart wearable devices, we see smart wearable devices appearing in more and more industrial operation scenarios. However, at present, the operating direction of most of these smart wearable devices is bound to the human body, and its flexible movement cannot be controlled through an intuitive human-computer interaction method.
眼球运动跟踪技术是通过一系列算法获取人眼视线信息。目前比较广为使用的眼球追踪的方法是瞳孔-角膜反射技术(PCCR)，该技术的基本原理是通过使用红外线照射眼睛，使用红外摄像机采集从角膜和视网膜上反射的红外光线，由于眼球的生理结构和物理性质，在光源和头部相对位置不变的前提下，角膜反射形成的光斑不会移动，视网膜上反射的光线方向标示了瞳孔的朝向(光源光线从瞳孔射入，视网膜反射光线从瞳孔射出)。因此根据角膜与瞳孔反射光线之间的角度可以计算出眼动的方向。PCCR眼动技术现已被广泛应用于眼动识别设备中，且相应的开源识别算法可以直接使用。Eye movement tracking technology is to obtain human eye gaze information through a series of algorithms. At present, the widely used method of eye tracking is Pupil-Corneal Reflection (PCCR). The structure and physical properties, under the premise that the relative position of the light source and the head remain unchanged, the light spot formed by the corneal reflection will not move, and the direction of the light reflected on the retina indicates the direction of the pupil (the light from the light source enters from the pupil, and the light reflected from the retina passes from pupil ejection). Therefore, the direction of eye movement can be calculated according to the angle between the cornea and the reflected light from the pupil. PCCR eye movement technology has been widely used in eye movement recognition equipment, and the corresponding open source recognition algorithm can be used directly.
而将眼动信息作为操控输入则可以实现由眼动控制的人机交互方式，该交互方式相较于其他交互方式拥有门槛低，应用场景灵活性高，且符合人的直觉的独特优势，拥有广阔的应用前景。因此不少穿戴设备中也出现了眼动控制相关技术，但目前使用眼动控制的智能穿戴设备并没有很好地实现从眼球运动凝视位置到所控制的设备朝向的准确映射方式，即持续的眼球追踪；且功能设计单一，应用场景受局限。如专利申请CN107388201A公开了一种医用头戴式眼动控制手术照明灯，其确定用户视线位置的方式是通过“重心法算法提取”，将用户正视前方时的“瞳孔重心”与该时刻用户的“瞳孔中心”比较移动的像素数，该视线方向识别算法并不精准，不能得到用户真正的视线方向；没有考虑到用户与被照射物体之间的距离以及其他因素，从用户视线到控制LED灯的映射方式是线性的且考虑不周全的；且其使用场景单一受限，仅能用于照明。专利申请CN114500839A提出了一种基于注意力跟踪机制的视觉云台控制方法及系统，其在实现调整云台时，考虑了目标注视点坐标的因素，实现对云台角度的调节控制，考虑了目标注视点因素，提高了控制精度；但其针对球赛的观看者的看球注意力，调整智能化自动导播云台的控制，从而使云台拍摄的画面兼顾观众注意力与画面中的被跟踪主体；使用用途单一，仅能用于画面追踪拍摄，所提出的设备系统可移动性差，使用方式与使用地点受局限；云台控制方式并不是实时跟踪实现位置的，而是将实现作为一种跟踪的辅助输入，用于画面跟踪。Using eye movement information as the control input can realize the human-computer interaction method controlled by eye movement. Compared with other interactive methods, this interaction method has the unique advantages of low threshold, high flexibility of application scenarios, and in line with human intuition. Broad application prospects. Therefore, eye movement control related technologies have also appeared in many wearable devices, but the smart wearable devices that use eye movement control do not realize the accurate mapping from the gaze position of the eyeball movement to the controlled device orientation, that is, continuous Eye tracking; and the function design is single, and the application scenarios are limited. For example, patent application CN107388201A discloses a medical head-mounted eye-movement control surgical lighting. The way to determine the position of the user's line of sight is through the "centre of gravity method algorithm extraction", and the "pupil center of gravity" when the user is looking straight ahead is compared with the user's position at that moment. The "pupil center" compares the number of moving pixels. The line-of-sight recognition algorithm is not accurate and cannot obtain the user's real line-of-sight direction; it does not take into account the distance between the user and the illuminated object and other factors, from the user's line of sight to the control of the LED light The mapping method of is linear and not well considered; and its use scene is single and limited, and can only be used for lighting. Patent application CN114500839A proposes a visual pan-tilt control method and system based on an attention tracking mechanism. When adjusting the pan-tilt, it takes into account the coordinates of the target gaze point, realizes the adjustment and control of the pan-tilt angle, and considers the target The focus point factor improves the control accuracy; however, it adjusts the control of the intelligent automatic directing pan/tilt according to the attention of the viewers watching the game, so that the pan/tilt takes into account both the audience's attention and the tracked subject in the picture. ; The purpose of use is single, and it can only be used for screen tracking and shooting. The proposed equipment system has poor mobility, and the use method and location are limited; the PTZ control method does not track the position in real time, but uses the realization as a tracking Auxiliary input for picture tracking.
发明内容Contents of the invention
为解决上述技术问题，本发明提供了一种眼动控制的多模块智能设备，其可实现从眼球运动凝视位置到所控制的设备朝向的准确映射，并可更换多种模块，使得功能多样化，应用场景广泛。In order to solve the above technical problems, the present invention provides a multi-module smart device controlled by eye movement, which can realize accurate mapping from the gaze position of the eyeball movement to the orientation of the controlled device, and can replace various modules to make the functions diversified , a wide range of application scenarios.
本发明所述的一种眼动控制的多模块智能设备，包括单片机、单片机连接的云台电机模块、模式切换模块、眼动图像采集模块、视觉追踪算法模块、陀螺仪模块、视线到云台映射算法模块；所述单片机用于接收其他模块传输的数据，并发出控制指令给对应模块进行相应的操作；A multi-module intelligent device for eye movement control according to the present invention includes a single-chip microcomputer, a pan-tilt motor module connected to the single-chip microcomputer, a mode switching module, an eye movement image acquisition module, a visual tracking algorithm module, a gyroscope module, and a line of sight to the pan-tilt A mapping algorithm module; the single-chip microcomputer is used to receive data transmitted by other modules, and send control instructions to corresponding modules to perform corresponding operations;
其中，眼动图像采集模块分别对左眼和右眼进行眼动图像采集；Wherein, the eye movement image acquisition module performs eye movement image acquisition on the left eye and the right eye respectively;
视觉追踪算法模块用于处理眼动图像采集模块获取的画面数据，计算得到眼球的视线方向；The visual tracking algorithm module is used to process the picture data acquired by the eye movement image acquisition module, and calculate the line of sight direction of the eyeball;
模式切换模块与单片机相连接，通过输出信号给单片机，单片机根据收到的信号切换云台的跟踪状态，即视线追踪模式或云台方向锁定模式；The mode switching module is connected with the single-chip microcomputer, and the single-chip microcomputer switches the tracking state of the gimbal according to the received signal, that is, the sight tracking mode or the gimbal direction locking mode;
陀螺仪模块输出三维矢量信息，实现云台方向锁定；The gyroscope module outputs three-dimensional vector information to realize the direction locking of the gimbal;
视线到云台映射算法模块根据眼球的视线方向及陀螺仪模块输出的信息，得到云台应该指向的方向；The line-of-sight to gimbal mapping algorithm module obtains the direction the gimbal should point to according to the line-of-sight direction of the eyeball and the information output by the gyroscope module;
单片机根据得到的云台应指向的方向信息控制云台电机模块转动。The single-chip microcomputer controls the rotation of the motor module of the pan-tilt according to the direction information that the pan-tilt should point to.
进一步的，所述眼动图像采集模块为红外摄像机，其分别对左眼和右眼进行眼动图像采集。Further, the eye movement image collection module is an infrared camera, which collects eye movement images for the left eye and the right eye respectively.
进一步的，视觉追踪算法模块采用开源的眼动识别代码实现眼球3D信息的获取；基于红外照明的眼球画面，通过计算比较画面像素深度来获得二维的瞳孔信息，再将该数据输入到三维眼球模型算法中，从而获得视线的三维矢量方向、眼球的中心位置以及瞳孔直径。Furthermore, the visual tracking algorithm module uses open-source eye movement recognition codes to obtain 3D eyeball information; based on the infrared lighting eyeball picture, the two-dimensional pupil information is obtained by calculating and comparing the pixel depth of the picture, and then the data is input into the three-dimensional eyeball In the model algorithm, the three-dimensional vector direction of the line of sight, the center position of the eyeball and the diameter of the pupil are obtained.
进一步的，模式切换模块由一个自锁开关组成，该模块输出自锁开关的信号给单片机，单片机根据信号来切换视线追踪模式。Further, the mode switching module is composed of a self-locking switch, which outputs the signal of the self-locking switch to the single-chip microcomputer, and the single-chip microcomputer switches the eye-tracking mode according to the signal.
进一步，通过视线到云台映射算法模块根据眼动识别算法计算得到的眼球的视线方向及陀螺仪模块输出的信息，以控制云台指向准确的实现方向，具体为：Further, through the line of sight to the gimbal mapping algorithm module according to the eye movement recognition algorithm to calculate the direction of the eyeball and the information output by the gyroscope module, to control the accurate realization direction of the gimbal pointing, specifically:
设备初始化，用户佩戴该设备，自然平视前方，此时进行初始化，设备将认为当前的方向为初始化方向，以此作为参考；经过初始化后单片机不断监测模式切换模块输出的信息：若监测到模式切换模块的信号为低，云台进入视线追踪模式，通过眼动识别算法获得视线方向，利用视线到云台映射算法模块控制云台指向视线方向；若单片机监测到模式切换模块的信号为高，单片机接收到自锁开关信息后，切换云台跟踪模式为云台方向锁定模式。The device is initialized. The user wears the device and looks forward naturally. At this time, the initialization is performed, and the device will consider the current direction as the initialization direction as a reference; after initialization, the single-chip microcomputer continuously monitors the information output by the mode switching module: When the signal of the module is low, the gimbal enters the line-of-sight tracking mode, obtains the line-of-sight direction through the eye movement recognition algorithm, uses the line-of-sight to the gimbal mapping algorithm module to control the gimbal to point to the line-of-sight direction; if the MCU detects that the signal of the mode switching module is high, the MCU After receiving the self-locking switch information, switch the gimbal tracking mode to gimbal direction lock mode.
进一步的，通过眼动识别算法获得从左右眼发射出的视线的方向矢量，根据两眼距离计算视线交点位置，该位置相对于云台的方向即为云台应指向的方向。Further, the direction vector of the line of sight emitted from the left and right eyes is obtained through the eye movement recognition algorithm, and the position of the line of sight intersection is calculated according to the distance between the two eyes. The direction of this position relative to the gimbal is the direction the gimbal should point to.
进一步的，当云台跟踪方式为云台方向锁定模式时，可以控制云台不随着视线运动；具体为：不断根据陀螺仪获得信号进行补偿，即切换模式切换模块信号，记录此时云台的方位，且同时开始通过不断获取陀螺仪获得当前头部运动方向，将切换模式切换模块信号时的云台方向减去陀螺仪获得的头部运动方向，实现云台方向锁定。Further, when the gimbal tracking mode is the gimbal direction lock mode, the gimbal can be controlled not to move with the line of sight; the specific method is: continuously compensate according to the signal obtained from the gyroscope, that is, switch the mode to switch the module signal, and record the gimbal at this time Orientation, and at the same time start to obtain the current head movement direction by continuously acquiring the gyroscope, subtract the head movement direction obtained by the gyroscope from the gimbal direction when switching the mode switch module signal, and realize the gimbal direction locking.
进一步的，云台电机模块为三轴云台，有三个互相垂直的电机马达构成，通过单片机进行转动方向的控制；Further, the gimbal motor module is a three-axis gimbal, which is composed of three mutually perpendicular motors, and the direction of rotation is controlled by a single-chip microcomputer;
通过将当前云台的方向与云台应指向的方向作差取绝对值，得到电机运动的幅度大小；电机运动速度与正向相关，且云台的旋转速度呈缓动(即先快后慢)的速度变化，实现更快地响应用户视线的运动。By taking the absolute value of the difference between the current direction of the gimbal and the direction the gimbal should point to, the magnitude of the motor motion can be obtained; the motor motion speed is related to the positive direction, and the rotation speed of the gimbal is slow (that is, it is fast first and then slow ) to achieve faster response to the movement of the user's gaze.
进一步的，所述云台表面还设有通讯模块，其他功能模块通过通讯模块的接口与单片机通信；所述其他功能模块为测距模块或测温模块或照明模块或测速模块。Further, the surface of the cloud platform is also equipped with a communication module, and other functional modules communicate with the single-chip microcomputer through the interface of the communication module; the other functional modules are distance measuring modules, temperature measuring modules, lighting modules or speed measuring modules.
本发明所述的有益效果为：本发明通过视线追踪模块实现双眼视线追踪，通过单片机计算提供快速精准的云台视线追踪效果，真正做到“随眼动”；本发明利用模块化设计，基于云台上的通讯模块，允许用户根据所需要的适用场景自由更换组件，通过红外测距模块测距、红外测温模块测温，使用高功率LED灯照明模块照明，以及使用激光测速模块测速，拓宽了此设备的使用应用场景；设计了云台方向锁定模式，利用陀螺仪实时补偿头部运动，满足对维持某一方向数据测量的需求，进一步拓展了该设备的使用场景。The beneficial effects of the present invention are as follows: the present invention realizes binocular gaze tracking through the gaze tracking module, and provides a fast and accurate pan-tilt gaze tracking effect through single-chip computer calculations, truly achieving "following the eyes"; the present invention utilizes a modular design, based on The communication module on the gimbal allows users to freely replace components according to the required applicable scenarios, measure the distance through the infrared ranging module, measure the temperature with the infrared temperature measuring module, use the high-power LED lighting module for lighting, and use the laser speed measuring module to measure the speed. The use and application scenarios of this device are broadened; the gimbal direction locking mode is designed, and the gyroscope is used to compensate the head movement in real time to meet the demand for maintaining data measurement in a certain direction, and further expand the use scenarios of the device.
附图说明Description of drawings
图1是本发明所述设备的结构示意图；Fig. 1 is the structural representation of equipment described in the present invention;
图2是本发明所述设备的功能示意图；Fig. 2 is a functional schematic diagram of the device of the present invention;
图3是眼动图像采集模块使用示意图；Fig. 3 is a schematic diagram of the use of the eye movement image acquisition module;
图4是视觉追踪算法模块使用示意图；Figure 4 is a schematic diagram of the use of the visual tracking algorithm module;
图5是视线到云台映射算法模块原理图；Fig. 5 is a schematic diagram of the line of sight to the cloud platform mapping algorithm module;
图6是视线到云台映射算法模块进行映射的方法流程图Fig. 6 is a flow chart of the method for mapping the line of sight to the pan-tilt mapping algorithm module
图7是本发明所述设备的工作原理图。Fig. 7 is a working principle diagram of the device of the present invention.
图8是本发明所述设备的云台工作模式切换原理图。Fig. 8 is a schematic diagram of switching working modes of the pan/tilt of the device according to the present invention.
具体实施方式Detailed ways
为了使本发明的内容更容易被清楚地理解，下面根据具体实施例并结合附图，对本发明作进一步详细的说明。In order to make the content of the present invention more clearly understood, the present invention will be further described in detail below based on specific embodiments and in conjunction with the accompanying drawings.
本发明所述的一种多模块智能眼动交互设备结构如图1所示，功能设计如图2所示，包括stm32单片机、云台电机模块、模式切换模块、眼动图像采集模块、视觉追踪算法模块、陀螺仪模块、视线到云台映射算法模块、云台与功能模块的通讯模块以及各类功能模块(如测距模块、测温模块、照明模块、测速模块)；单片机系统中的数据缓存模块可以将一些常用的数据或指令存储在缓存中，这样当系统需要使用这些数据或指令时，就可以直接从缓存中获取，而不需要每次都从主存中读取，大大提高系统的响应速度，减少延迟时间。以stm32单片机作为主控制器控制系统其他芯片，进行图像以及其他数据的采集、传输、处理、计算、缓存以及输出等操作。The structure of a multi-module intelligent eye movement interaction device according to the present invention is shown in Figure 1, and the functional design is shown in Figure 2, including stm32 single-chip microcomputer, pan-tilt motor module, mode switching module, eye movement image acquisition module, visual tracking Algorithm module, gyroscope module, line-of-sight to pan-tilt mapping algorithm module, communication module between pan-tilt and functional modules, and various functional modules (such as distance measurement module, temperature measurement module, lighting module, speed measurement module); data in the single-chip system The cache module can store some commonly used data or instructions in the cache, so that when the system needs to use these data or instructions, they can be directly obtained from the cache instead of reading from the main memory every time, which greatly improves the performance of the system. Response speed, reduce delay time. The stm32 single-chip microcomputer is used as the main controller to control other chips in the system to perform operations such as image and other data collection, transmission, processing, calculation, caching, and output.
眼动图像采集模块为红外摄像机，用于采集使用者眼部特写画面数据，为视觉追踪算法模块提供数据。该穿戴设备可设计为类似头盔，在使用者左右眼附近有两台红外摄像机分别对左眼与右眼进行眼动图像采集。如图3所示，红外摄像机的位置与角度可以调整，以便适配更多使用者。一般人的瞳距范围在58-64mm内。The eye movement image acquisition module is an infrared camera, which is used to collect the close-up image data of the user's eyes and provide data for the visual tracking algorithm module. The wearable device can be designed like a helmet, and there are two infrared cameras near the left and right eyes of the user to collect eye movement images for the left and right eyes respectively. As shown in Figure 3, the position and angle of the infrared camera can be adjusted to suit more users. The range of interpupillary distance for ordinary people is within 58-64mm.
视觉追踪算法模块：用于处理眼动图像采集模块获取的画面数据，通算法计算得到眼球的视线方向。Vision tracking algorithm module: used to process the picture data acquired by the eye movement image acquisition module, and calculate the gaze direction of the eyeball through the algorithm.
本实施例采用开源的眼动识别代码实现眼球3D信息的获取。源代码使用C++语言编写，易于编译，之后装载入stm32单片机中进行运行。In this embodiment, an open-source eye movement recognition code is used to obtain eyeball 3D information. The source code is written in C++ language, easy to compile, and then loaded into the stm32 microcontroller for operation.
该代码的编写是基于红外照明的眼球画面，通过计算比较画面像素深度来获得二维的瞳孔信息，再将该数据输入到三维眼球模型算法中，从而获得视线的三维矢量方向、眼球的中心位置以及瞳孔直径等，如图4所示。The coding of this code is based on the eyeball picture of infrared illumination. The two-dimensional pupil information is obtained by calculating and comparing the pixel depth of the picture, and then the data is input into the three-dimensional eyeball model algorithm to obtain the three-dimensional vector direction of the line of sight and the center position of the eyeball. And pupil diameter, etc., as shown in Figure 4.
如图8所示，模式切换模块由一个自锁开关组成，该模块输出自锁开关的信号给单片机，单片机根据信号来切换视线追踪模式。当模式切换模块的信号为低时，云台进入视线追踪模式；当模式切换模块的信号为高时，单片机接收到自锁开关信息后，切换云台跟踪模式为云台方向锁定模式。As shown in Figure 8, the mode switching module is composed of a self-locking switch, which outputs the signal of the self-locking switch to the single-chip microcomputer, and the single-chip microcomputer switches the eye-tracking mode according to the signal. When the signal of the mode switching module is low, the pan/tilt enters the line-of-sight tracking mode; when the signal of the mode switching module is high, after the single-chip microcomputer receives the self-locking switch information, the pan/tilt tracking mode is switched to the pan/tilt direction locking mode.
陀螺仪模块：用于获得该设备目前的三维旋转信息，用于实现云台方向锁定功能；该模块输出三维矢量信息。Gyroscope module: used to obtain the current three-dimensional rotation information of the device, and used to realize the direction locking function of the gimbal; this module outputs three-dimensional vector information.
视线到云台映射算法模块：用于实现视觉追踪算法模块获得的三维信息对云台电机的正确操控，良好的映射关系决定了眼动控制云台的精确度与体验性。Line-of-sight to gimbal mapping algorithm module: used to realize the correct control of the gimbal motor by the three-dimensional information obtained by the visual tracking algorithm module. A good mapping relationship determines the accuracy and experience of eye movement control gimbal.
设备初始化，用户佩戴该设备，自然平视前方，此时进行初始化，设备将认为当前的方向为初始化方向，以此作为参考；经过初始化后单片机不断监测模式切换模块输出的信息，判断云台进入哪种模式。When the device is initialized, the user wears the device and looks straight ahead naturally. At this time, the device will consider the current direction as the initialization direction and use it as a reference; mode.
当云台进入视线追踪模式时，视线到云台映射算法模块根据眼动识别算法计算得到的眼球的视线方向，同时根据陀螺仪模块输出的信息，以控制云台指向准确的实现方向，具体为：由设备物理形态决定的各部件之间的几何关系，我们可以构建一个本地坐标系，以两眼之间的中点作为坐标系原点O，在O点垂直上方的G点代表云台上所承载设备的中心，记为云台视角原点。左右眼分别记为点El与Er。如图5所示。设备本身的几何结构(如瞳距ElEr、云台到两眼中点的距离GO等)默认为已知的常数。而每个使用者的面部都是不同的，因此在使用前应当先输入给设备使用者的瞳距ElEr。When the gimbal enters the line-of-sight tracking mode, the line-of-sight-to-gimbal mapping algorithm module controls the gimbal to point to an accurate realization direction based on the eye gaze direction calculated by the eye movement recognition algorithm and the information output by the gyroscope module, specifically: : The geometric relationship between the components determined by the physical shape of the device, we can build a local coordinate system, take the midpoint between the two eyes as the origin O of the coordinate system, and the G point vertically above the O point represents the position on the gimbal. The center of the bearing device is recorded as the origin of the PTZ viewing angle. The left and right eyes are respectively marked as points El and Er. As shown in Figure 5. The geometric structure of the device itself (such as the interpupillary distance ElEr, the distance GO from the gimbal to the midpoint of the two eyes, etc.) is a known constant by default. And each user's face is different, so the interpupillary distance ElEr that should be input to the device user before use.
通过眼动识别算法可以获得从左右眼发射出的视线的方向矢量分别为与，经过计算可以得到视线交点，即为云台应指向的方向。具体如图6所示，通过设备内部的陀螺仪获得了设备的俯仰信息，得到面ElErG的法向量，即设备的正视方向R0，在初始化的过程中可以得到水平状态的方向向量，以此作为参考。此时，在三角形P0GO中，由三角函数得到云台应当指向的方向，即记作Vgoal。。将此时云台的朝向记作向量V0，单片机控制云台电机模块转动，将运动到Vgoal的方向。Through the eye movement recognition algorithm, the direction vectors of the line of sight emitted from the left and right eyes can be obtained respectively, and the intersection point of the line of sight can be obtained after calculation, which is the direction that the gimbal should point to. Specifically, as shown in Figure 6, the pitch information of the device is obtained through the gyroscope inside the device, and the normal vector of the surface El Er G is obtained, that is, the front view direction R0 of the device. During the initialization process, the direction of the horizontal state can be obtained Vector to use as a reference. At this time, in the triangle P0GO, the direction the pan/tilt should point to is obtained by the trigonometric function, namely Denote it as Vgoal . . The direction of the gimbal at this time is recorded as a vector V0 , and the single-chip microcomputer controls the rotation of the gimbal motor module to move to the direction of Vgoal .
当云台跟踪方式为云台方向锁定模式时，可以控制云台不随着视线运动；具体为：不断根据陀螺仪获得信号进行补偿，即切换模式切换模块信号，记录此时云台的方位，且同时开始通过不断获取陀螺仪获得当前头部运动方向，将切换模式切换模块信号时的云台方向减去陀螺仪获得的头部运动方向，实现云台方向锁定，从而满足更多使用需求。When the gimbal tracking mode is the gimbal direction lock mode, it can control the gimbal not to move with the line of sight; specifically: continuously compensate according to the signal obtained from the gyroscope, that is, switch the mode to switch the module signal, record the orientation of the gimbal at this time, and At the same time, start to obtain the current head movement direction by continuously obtaining the gyroscope, subtract the head movement direction obtained by the gyroscope from the gimbal direction when switching the mode switch module signal, and realize the gimbal direction locking, so as to meet more usage needs.
云台电机模块：用于实现功能模块在三个维度上的运动；该模块为三轴云台，由三个相互垂直的电机马达构成，通过单片机进行转动方向的控制；通过将当前云台的方向与云台应指向的方向作差取绝对值，得到电机运动的幅度大小；电机运动速度与正向相关，且云台的旋转速度呈缓动(即先快后慢)的速度变化，实现更快地响应用户视线的运动。Pan-tilt motor module: used to realize the movement of functional modules in three dimensions; this module is a three-axis pan-tilt, which is composed of three mutually perpendicular motors, and the direction of rotation is controlled by a single-chip microcomputer; Take the absolute value of the difference between the direction and the direction the pan/tilt should point to, and get the magnitude of the motor movement; the motor movement speed is related to the positive direction, and the rotation speed of the pan/tilt changes slowly (that is, first fast and then slow), realizing Faster response to user gaze movements.
各类功能模块：用于提供各种不同功能，包括：测距模块、测温模块、照明模块、测速模块等，通过接口与单片机通讯。Various functional modules: used to provide various functions, including: distance measurement module, temperature measurement module, lighting module, speed measurement module, etc., communicate with the microcontroller through the interface.
各个功能模块的几何中心应当设计为一致，从而保证各个模块能精准指向视线交点。The geometric center of each functional module should be designed to be consistent, so as to ensure that each module can accurately point to the line of sight intersection.
测距模块：使用红外测距；测温模块：使用红外线测温；照明模块：使用高功率LED灯；测速模块：使用激光测速模块。Distance measurement module: use infrared distance measurement; temperature measurement module: use infrared temperature measurement; lighting module: use high-power LED lights; speed measurement module: use laser speed measurement module.
本发明所述的一种眼动控制的多模块智能设备的系统流程如图7所示；以一次正常使用为例，首先用户的瞳距先输入到设备中，之后佩戴好设备，启动程序。设备开始进入校准模式。此时会同时启动陀螺仪与位于用户双眼旁的两个红外摄像机开启。The system flow of an eye-movement-controlled multi-module intelligent device according to the present invention is shown in Figure 7; taking a normal use as an example, the user's interpupillary distance is first input into the device, and then the device is worn and the program is started. The device starts to enter calibration mode. At this time, the gyroscope and the two infrared cameras located next to the user's eyes will be activated at the same time.
陀螺仪会持续获得用户此时的头部运动角度信息；同时，两个红外摄像机不断获取眼球画面，眼动识别算法在单片机中不断处理这些画面，获得两个眼球的视线方向。我们已经知道用户的瞳距以及设备的物理结构关系，就可以计算出使用者当前时刻的视线交点，这也就是云台需要运动的目标。The gyroscope will continue to obtain the user's head movement angle information at this time; at the same time, the two infrared cameras will continuously obtain eyeball images, and the eye movement recognition algorithm will continuously process these images in the single-chip microcomputer to obtain the line of sight directions of the two eyeballs. We already know the user's interpupillary distance and the physical structure of the device, so we can calculate the intersection point of the user's line of sight at the current moment, which is the target that the gimbal needs to move.
当设备初次使用时，进入校准流程：使用者正视前方，同时保持头的水平。设备陀螺仪与眼球模型初始化完毕，校准结束，开始正常使用。When the device is first used, it enters the calibration process: the user looks straight ahead while keeping their head level. The device gyroscope and eyeball model are initialized, the calibration is completed, and normal use begins.
在正常使用过程中，用户可以使用视线控制云台对准自己注视的地方。用户根据自己的使用需求更换云台上的模块。云台上所安装的模块可以输出数据给单片机，包括测距，测速，测温模块从而记录数据，存储在数据缓存模块中。During normal use, the user can use the line of sight to control the gimbal to align with where they are looking. Users can replace the modules on the gimbal according to their own needs. The modules installed on the cloud platform can output data to the single-chip microcomputer, including distance measurement, speed measurement, and temperature measurement modules to record data and store them in the data cache module.
使用者还可以通过开启设备上的自锁开关以开启云台方向锁定的功能，可以控制云台不随着实现运动，从而满足诸如持续照明前方道路，或者是持续测量某一方向上的距离等更多使用需求。The user can also turn on the self-locking switch on the device to enable the function of the gimbal direction lock, which can control the gimbal not to move along with it, so as to meet the requirements such as continuous lighting of the road ahead, or continuous measurement of the distance in a certain direction, etc. Usage requirements.
以上所述仅为本发明的优选方案，并非作为对本发明的进一步限定，凡是利用本发明说明书及附图内容所作的各种等效变化均在本发明的保护范围之内。The above descriptions are only preferred solutions of the present invention, and are not intended to further limit the present invention. All equivalent changes made by using the description and drawings of the present invention are within the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310478072.6ACN116560503A (en) | 2023-04-28 | 2023-04-28 | A multi-module intelligent device controlled by eye movement |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310478072.6ACN116560503A (en) | 2023-04-28 | 2023-04-28 | A multi-module intelligent device controlled by eye movement |
| Publication Number | Publication Date |
|---|---|
| CN116560503Atrue CN116560503A (en) | 2023-08-08 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310478072.6APendingCN116560503A (en) | 2023-04-28 | 2023-04-28 | A multi-module intelligent device controlled by eye movement |
| Country | Link |
|---|---|
| CN (1) | CN116560503A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025081535A1 (en)* | 2023-10-20 | 2025-04-24 | 中国科学院深圳先进技术研究院 | Wearable head-mounted eye movement tracking device, method and system, and processing device |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2025081535A1 (en)* | 2023-10-20 | 2025-04-24 | 中国科学院深圳先进技术研究院 | Wearable head-mounted eye movement tracking device, method and system, and processing device |
| Publication | Publication Date | Title |
|---|---|---|
| CN104113680B (en) | Gaze tracking system and method | |
| US10521026B2 (en) | Passive optical and inertial tracking in slim form-factor | |
| US11127380B2 (en) | Content stabilization for head-mounted displays | |
| CN104834381B (en) | Wearable device and sight line focus localization method for sight line focus location | |
| US9122053B2 (en) | Realistic occlusion for a head mounted augmented reality display | |
| US20140375680A1 (en) | Tracking head movement when wearing mobile device | |
| US7657079B2 (en) | Single constraint at a time (SCAAT) tracking of a virtual reality (VR) display | |
| CN110345815A (en) | A kind of creeper truck firearms method of sight based on Eye-controlling focus | |
| KR20180064413A (en) | Eye tracking using structured light | |
| CN108170279A (en) | The eye of aobvious equipment is moved moves exchange method with head | |
| Bohme et al. | Remote eye tracking: State of the art and directions for future development | |
| WO2013155217A1 (en) | Realistic occlusion for a head mounted augmented reality display | |
| US11557020B2 (en) | Eye tracking method and apparatus | |
| CN113160260B (en) | Head-eye double-channel intelligent man-machine interaction system and operation method | |
| US11983310B2 (en) | Gaze tracking apparatus and systems | |
| CN113963416A (en) | Eye movement interaction method and system based on laser visual feedback | |
| US11743447B2 (en) | Gaze tracking apparatus and systems | |
| CN116560503A (en) | A multi-module intelligent device controlled by eye movement | |
| CN119576126B (en) | Mixed reality eye-tracking interaction system and method based on dense map semantic segmentation | |
| GB2596541A (en) | Video processing | |
| EP4425304A2 (en) | Radar-assisted three-dimensional (3d) detection for near-eye display devices | |
| CN109727317B (en) | Augmented reality system and control method | |
| KR100664832B1 (en) | 2D monitor pointing device using eye movement tracking | |
| KR100374346B1 (en) | Head pose tracking method using image processing | |
| CN223244896U (en) | Visual smart mirror |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |