CN114947848A - A real-time monitoring system for pilots' attention and steady state based on EEG and eye movement data - Google Patents
A real-time monitoring system for pilots' attention and steady state based on EEG and eye movement dataDownload PDFInfo
- Publication number
- CN114947848A CN114947848ACN202210464894.4ACN202210464894ACN114947848ACN 114947848 ACN114947848 ACN 114947848ACN 202210464894 ACN202210464894 ACN 202210464894ACN 114947848 ACN114947848 ACN 114947848A
- Authority
- CN
- China
- Prior art keywords
- data
- eye movement
- electroencephalogram
- eeg
- attention
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/0016—Operational features thereof
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/0016—Operational features thereof
- A61B3/0025—Operational features thereof characterised by electronic signal processing, e.g. eye models
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/11—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for measuring interpupillary distance or diameter of pupils
- A61B3/112—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for measuring interpupillary distance or diameter of pupils for measuring diameter of pupils
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B3/00—Apparatus for testing the eyes; Instruments for examining the eyes
- A61B3/10—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions
- A61B3/113—Objective types, i.e. instruments for examining the eyes independent of the patients' perceptions or reactions for determining or recording eye movement
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/163—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state by tracking eye movement, gaze, or pupil change
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
- A61B5/168—Evaluating attention deficit, hyperactivity
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
- A61B5/374—Detecting the frequency distribution of signals, e.g. detecting delta, theta, alpha, beta or gamma waves
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7225—Details of analogue processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/725—Details of waveform analysis using specific filters therefor, e.g. Kalman or adaptive filters
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7253—Details of waveform analysis characterised by using transforms
- A61B5/726—Details of waveform analysis characterised by using transforms using Wavelet transforms
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Veterinary Medicine (AREA)
- Biomedical Technology (AREA)
- Surgery (AREA)
- Molecular Biology (AREA)
- Medical Informatics (AREA)
- Public Health (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Heart & Thoracic Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- Pathology (AREA)
- Psychiatry (AREA)
- Signal Processing (AREA)
- Psychology (AREA)
- Developmental Disabilities (AREA)
- Ophthalmology & Optometry (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physiology (AREA)
- Educational Technology (AREA)
- Social Psychology (AREA)
- Child & Adolescent Psychology (AREA)
- Hospice & Palliative Care (AREA)
- Human Computer Interaction (AREA)
- Power Engineering (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
Translated fromChinese
Description
Translated fromChinese技术领域technical field
本发明涉及人机环系统技术领域，涉及一种飞行员在监控任务过程中，注意的稳定性状态(包括注意正常和注意过度集中状态)的实时监测系统。The invention relates to the technical field of man-machine loop systems, and relates to a real-time monitoring system for a pilot's attention stability state (including normal attention and excessive attention concentration state) during the monitoring task process.
背景技术Background technique
飞行员注意稳定状态实时监测是人机环中人体状态识别技术的一种。人机环人体状态识别的目的是通过对人的行为、认知和情感状态进行实时的识别，以实现对人的能力的评价、系统安全的监控和人机交互效率的提升等效果。注意力、情绪、疲劳、工作负荷和操作意图是最受关注的、最有应用价值的几种人体状态。The real-time monitoring of the pilot's attention to the steady state is a kind of human state recognition technology in the man-machine loop. The purpose of human-machine-loop human state recognition is to realize real-time recognition of human behavior, cognition and emotional state, so as to realize the evaluation of human ability, the monitoring of system security and the improvement of human-computer interaction efficiency. Attention, emotion, fatigue, workload, and operational intention are some of the most concerned and valuable human states.
在飞行作战任务中，注意稳定性有时是影响战斗胜败的关键因素。比如，飞行员常在锁定敌机时进入注意力过度集中的状态，产生注意隧道效应，而没能注意到本机被尾随攻击，导致被击落。因此，如果座舱可以对飞行员注意稳定状态进行实时的监测，在飞行员处于注意过度集中状态时进行预警和干预，可以提高飞行员的战斗力。In flying combat missions, attention to stability is sometimes the key factor affecting the victory or defeat of the battle. For example, pilots often get into a state of overattention when locking on the enemy aircraft, resulting in attention tunneling effect, but fail to notice that the aircraft is being followed and attacked, resulting in being shot down. Therefore, if the cockpit can monitor the steady state of the pilot's attention in real time, and provide early warning and intervention when the pilot is in a state of excessive concentration, the pilot's combat effectiveness can be improved.
现有的注意稳定状态监测主要有三方法。第一，通过操作行为特征的变化来实现检测注意稳定状态。比如，在医学上，可以通过让人在一段时间内进行警觉性任务并测量其反应时的变化，来测量一个人维持注意稳定性的能力。第二，通过眼动行为模式的变化来实现检测注意稳定状态。比如，通过眼睛是否注视主任务区域，可以判断人的注意力是否被分散。第三，通过脑活动特征的变化来实现检测注意稳定状态。比如，通过脑电仪采集人的脑电数据，并分析其变化模式，可以判断人处于注意集中状态或分心状态。There are three main methods for existing attention steady state monitoring. First, the detection of attentional steady-state is achieved by manipulating changes in behavioral characteristics. In medicine, for example, a person's ability to maintain attentional stability can be measured by performing an alertness task over a period of time and measuring changes in their responses. Second, the detection of attentional steady-state is achieved through changes in eye-movement behavior patterns. For example, whether people's attention is distracted can be judged by whether the eyes are fixed on the main task area. Third, the detection of attentional steady state is achieved through changes in the characteristics of brain activity. For example, by collecting a person's EEG data through an EEG and analyzing its change pattern, it can be judged whether a person is in a state of concentration or distraction.
以上三种方法各有其局限性。通过操作行为特征的变化来实现注意稳定状态识别的方法一般只能用于识别人维持注意力的能力，当用于实时监测时，会存在时间延迟的缺点(当注意分散并导致行为绩效下降以后，才能监测出注意的分散)。通过眼睛是否注视主任务区域来实现注意稳定状态识别的方法只能由于在空间上能区分不同任务区域的任务场景。通过脑活动特征的变化来实现检测注意稳定状态的方法有的需要在飞机座舱上难以采用的多通道的脑电采集设备来采集多通道的脑电信号作为输入，有的则针对特定使用场景建模(如学生听课)、不能迁移到飞行场景。综上所述，现有的注意稳定状态监测方法都没能解决在飞机座舱中对飞行员注意稳定状态有效地实时监测的问题。Each of the above three methods has its limitations. The method of realizing attentional steady state recognition by manipulating changes in behavioral characteristics can generally only be used to recognize the ability of people to maintain attention. When used for real-time monitoring, there will be the disadvantage of time delay (when attention is distracted and leads to a decline in behavioral performance later , in order to monitor the distraction of attention). The method of realizing attention steady state recognition by whether the eyes are fixed on the main task area can only be due to the task scene that can distinguish different task areas in space. Some methods of detecting the steady state of attention through changes in brain activity characteristics require multi-channel EEG acquisition equipment that is difficult to use in the aircraft cockpit to collect multi-channel EEG signals as input, while others are designed for specific usage scenarios. Models (such as students listening to lectures) cannot be migrated to flight scenarios. To sum up, none of the existing attentional steady state monitoring methods can solve the problem of effective real-time monitoring of the pilot's attentional steady state in the aircraft cockpit.
发明内容SUMMARY OF THE INVENTION
本发明的目的是提供一种基于脑电与眼动数据的飞行员注意稳定状态实时监测系统，利用便携式的脑电和眼动设备所采集的数据，实现在飞机座舱中实现可行的、有效的飞行员注意稳定状态实时监测。The purpose of the present invention is to provide a real-time monitoring system for pilots' attention and steady state based on EEG and eye-tracking data, using the data collected by portable EEG and eye-tracking equipment to realize a feasible and effective pilot in the cockpit of the aircraft Pay attention to real-time monitoring of steady state.
一种基于脑电与眼动数据的飞行员注意稳定状态实时监测系统，包括脑电和眼动数据采集模块、神经生理数据处理模块、神经生理特征校准模块和注意稳定性计算模块，其中：A real-time monitoring system for pilots' attention stability state based on EEG and eye movement data, comprising an EEG and eye movement data acquisition module, a neurophysiological data processing module, a neurophysiological feature calibration module and an attention stability calculation module, wherein:
脑电和眼动数据采集模块与用户佩戴的便携式脑电仪和眼动仪连接，并对脑电仪和眼动仪采集的脑电数据、眼动数据进行收集；The EEG and eye movement data acquisition module is connected to the portable EEG and eye tracker worn by the user, and collects the EEG data and eye movement data collected by the EEG and the eye tracker;
神经生理数据处理模块用于对所采集的脑电数据进行滤波、去伪迹、校准处理、特征提取，得到脑电数据特征；对眼动数据进行瞳孔大小异常变化次数特征提取并进行校准处理，得到眼动数据特征；其中，利用脑电基线特征参数对去伪迹后的脑电数据进行校准处理，利用眼动基线特征参数对眼动数据提取后的特征进行校准处理；The neurophysiological data processing module is used to filter, remove artifacts, calibrate and extract features from the collected EEG data to obtain EEG data features; perform feature extraction and calibration processing on the eye movement data for abnormal changes in pupil size. Obtaining eye movement data features; wherein, the EEG data after artifact removal is calibrated by using the EEG baseline characteristic parameters, and the extracted features of the eye movement data are calibrated by using the eye movement baseline characteristic parameters;
神经生理特征校准模块利用所述脑电和眼动数据采集模块获取用户在特定状态下的脑电基线数据和眼动基线数据，并输出至所述神经生理数据处理模块对脑电基线数据进行滤波、去伪迹处理，对眼动基线数据进行瞳孔大小异常变化次数特征提取，得到脑电基线特征参数和眼动基线特征参数；The neurophysiological feature calibration module uses the EEG and eye movement data acquisition module to obtain the EEG baseline data and eye movement baseline data of the user in a specific state, and outputs to the neurophysiological data processing module to filter the EEG baseline data , Artifact-removing processing, extracting the number of abnormal changes in pupil size for eye movement baseline data, and obtaining EEG baseline characteristic parameters and eye movement baseline characteristic parameters;
注意稳定性计算模块用于将神经生理数据处理模块输出的脑电数据特征和眼动数据特征作为输入，通过训练好的注意稳定性状态模型，输出注意稳定性状态标签；所述注意稳定性状态模型采用非线性支持向量机模型。The attentional stability calculation module is used to take the EEG data features and eye movement data features output by the neurophysiological data processing module as input, and output the attentional stability state label through the trained attentional stability state model; the attentional stability state The model adopts the nonlinear support vector machine model.
进一步地，所述脑电数据的滤波过程为：Further, the filtering process of the EEG data is:
对于每段实时获取的脑电数据，滤波方法为4阶Butterworth滤波器，获得0.5Hz到100Hz的信号，以去除慢信号和高频噪声，由此得到滤波后的脑电原始信号。For each piece of EEG data acquired in real time, the filtering method is a 4th-order Butterworth filter to obtain signals from 0.5Hz to 100Hz to remove slow signals and high-frequency noise, thereby obtaining the filtered EEG original signal.
进一步地，所述脑电数据的去伪迹处理过程为：Further, the de-artifact processing process of the EEG data is:
首先，采用小波包分析的方法分辨出经滤波处理后的脑电原始信号中伪迹所在的时间区间：First, the wavelet packet analysis method is used to distinguish the time interval of the artifact in the filtered EEG original signal:
采用Daubechies 4作为母小波的小波包分析，把脑电原始信号分解到7层，对第1-4层的小波包系数设为0，对第5-7层的小波包系数进行一个软阈值过滤处理，具体做法为把第5-7层中每一层的小波包系数的集合提取出来，通过软阈值公式计算出阈值；然后把大小超过软阈值的小波包系数设为0，而其它不变；其中，软阈值公式为：Using Daubechies 4 as the wavelet packet analysis of the mother wavelet, the original EEG signal is decomposed into 7 layers, the wavelet packet coefficients of the 1st to 4th layers are set to 0, and the wavelet packet coefficients of the 5th to 7th layers are filtered by a soft threshold. The specific method is to extract the set of wavelet packet coefficients of each layer in the 5th to 7th layers, and calculate the threshold value through the soft threshold formula; ; Among them, the soft threshold formula is:
以上公式中，Thres为某一层小波包系数的软阈值，median()为中位数，abs()为绝对值，coef为由某一层小波包系数的集合组成的数组；In the above formula, Thres is the soft threshold of the wavelet packet coefficients of a certain layer, median() is the median, abs() is the absolute value, and coef is an array composed of a set of wavelet packet coefficients of a certain layer;
小波包分析重构后的信号即只包含伪迹的信号；在只包含伪迹的信号中，找到大于3倍方差的点，这些点即为需要纠正伪迹；根据眨眼伪迹信号特性，需要纠正的伪迹所在的时间区间还需要往前、往后延伸预设的时间段，以覆盖眨眼带来的电信号；由此获得的时间区间即为伪迹区间；The signal reconstructed by wavelet packet analysis is the signal containing only artifacts; in the signal containing only artifacts, find the points greater than 3 times the variance, these points are the artifacts that need to be corrected; The time interval in which the corrected artifact is located also needs to be extended forward and backward by a preset time period to cover the electrical signal brought by the blinking; the time interval thus obtained is the artifact interval;
其次，对于伪迹区间进行信号替换处理：Second, perform signal replacement processing for the artifact interval:
将所述滤波后的脑电原始信号经过Savitzky-Golay滤波处理；将伪迹区间中的数据替换成脑电原始信号经过Savitzky-Golay滤波处理的数据，即完成去伪迹处理。The filtered original EEG signal is processed by Savitzky-Golay filtering; the data in the artifact interval is replaced with the data processed by the original EEG signal processed by Savitzky-Golay filtering, that is, the artifact removal process is completed.
进一步地，所述脑电数据的校准处理过程为：Further, the calibration process of the EEG data is:
从神经生理特征校准模块获得用户在特定状态下的脑电基线特征参数，包括脑电数据的平均值MEEG与方差SDEEG，并将滤波后的信号作如下变换：Obtain the EEG baseline characteristic parameters of the user in a specific state from the neurophysiological feature calibration module, including the mean valueMEEG and variance SDEEG of the EEG data, and transform the filtered signal as follows:
其中，EEGInd为校准处理后的脑电数据，EEGFiltered为完成了去伪迹处理的脑电数据。Among them, EEGInd is the EEG data after calibration processing, and EEGFiltered is the EEG data after the de-artifact processing has been completed.
进一步地，所述脑电数据特征包括：Further, the EEG data features include:
整个信号的平均值；整个信号的方差；整个信号的偏度；整个信号的峰度；整个信号的Hjorth移动性；整个信号的Hjorth复杂度；整个信号的熵值；整个信号的Higuchi分形维数；基于小波包分析的delta波(0.5-4Hz)的强度；基于小波包分析的delta波(0.5-4Hz)的偏度；基于小波包分析的delta波(0.5-4Hz)的峰度；基于小波包分析的theta波(4-8Hz)的强度；基于小波包分析的theta波(4-8Hz)的偏度；基于小波包分析的theta波(4-8Hz)的峰度；基于小波包分析的alpha波(8-12Hz)的强度；基于小波包分析的alpha波(8-12Hz)的偏度；基于小波包分析的alpha波(8-12Hz)的峰度；基于小波包分析的beta波(12-30Hz)的强度；基于小波包分析的beta波(12-30Hz)的偏度；基于小波包分析的beta波(12-30Hz)的峰度；基于小波包分析的beta波(30-100Hz)的强度；基于小波包分析的beta波(30-100Hz)的偏度；基于小波包分析的beta波(30-100Hz)的峰度。The mean value of the entire signal; the variance of the entire signal; the skewness of the entire signal; the kurtosis of the entire signal; the Hjorth mobility of the entire signal; the Hjorth complexity of the entire signal; the entropy of the entire signal; the Higuchi fractal dimension of the entire signal ; Intensity of delta wave (0.5-4Hz) based on wavelet packet analysis; Skewness of delta wave (0.5-4Hz) based on wavelet packet analysis; Kurtosis of delta wave (0.5-4Hz) based on wavelet packet analysis; Wavelet based Intensity of theta wave (4-8Hz) based on packet analysis; skewness of theta wave (4-8Hz) based on wavelet packet analysis; kurtosis of theta wave (4-8Hz) based on wavelet packet analysis; The intensity of the alpha wave (8-12Hz); the skewness of the alpha wave (8-12Hz) based on the wavelet packet analysis; the kurtosis of the alpha wave (8-12Hz) based on the wavelet packet analysis; the beta wave based on the wavelet packet analysis ( 12-30Hz) intensity; wavelet packet analysis-based beta wave (12-30Hz) skewness; wavelet packet analysis-based beta wave (12-30Hz) kurtosis; wavelet packet analysis-based beta wave (30-100Hz) ) intensity; the skewness of beta waves (30-100 Hz) based on wavelet packet analysis; the kurtosis of beta waves (30-100 Hz) based on wavelet packet analysis.
进一步地，所述眼动数据的瞳孔大小异常变化次数确定方法为：Further, the method for determining the number of abnormal changes in pupil size of the eye movement data is:
对于每段实时获取的眼动数据中的瞳孔大小眼动数据，首先，对该数据进行小波包分析，采用Daubechies 8作为母小波，分解到第2层，并对第2层的小波包系数进行软阈值过滤处理；过滤后的非零小波包系数的个数即为瞳孔大小异常变化次数。For the pupil size eye movement data in the eye movement data obtained in real time, firstly, the wavelet packet analysis is performed on the data, and Daubechies 8 is used as the mother wavelet, decomposed into the second layer, and the wavelet packet coefficients of the second layer are analyzed. Soft threshold filtering; the number of non-zero wavelet packet coefficients after filtering is the number of abnormal changes in pupil size.
进一步地，所述眼动数据的校准处理与特征提取过程为：Further, the calibration processing and feature extraction process of the eye movement data are as follows:
将瞳孔大小异常变化次数减去从神经生理特征校准模块获得用户在特定状态下的眼动基线特征参数，即瞳孔大小异常变化次数的平均值，得到校准后的瞳孔大小异常变化次数特征，作为从眼动数据中提取的唯一眼动数据特征。The number of abnormal changes in pupil size is subtracted from the baseline characteristic parameters of the user's eye movement in a specific state obtained from the neurophysiological feature calibration module, that is, the average value of the number of abnormal changes in pupil size, to obtain the number of abnormal changes in pupil size after calibration. Unique eye movement data feature extracted from eye movement data.
进一步地，针对每个用户，需要在用户处于即静息休息状态时，采集T秒的脑电数据和眼动数据作为脑电基线数据和眼动基线数据；Further, for each user, it is necessary to collect T seconds of EEG data and eye movement data as EEG baseline data and eye movement baseline data when the user is in a resting state;
对于脑电基线数据，以t秒为一个窗口，通过神经生理数据处理模块中的滤波和去伪迹操作后，计算每个窗口的数据的平均值mEEG与方差sdEEG，并把所有窗口的平均值MEEG与方差SDEEG储存作为脑电基线特征参数；For the EEG baseline data, taking t seconds as a window, after filtering and removing artifacts in the neurophysiological data processing module, calculate the mean value mEEG and variance sdEEG of the data in each window, and put the MeanMEEG and variance SDEEG were stored as EEG baseline characteristic parameters;
对于眼动基线数据，以t秒为一个窗口，通过神经生理数据处理模块中的瞳孔大小异常变化次数确定方法计算每个窗口的瞳孔大小异常变化次数值，并计算所有窗口的平均值，即瞳孔大小异常变化次数的平均值作为眼动基线特征参数。For the eye movement baseline data, taking t seconds as a window, the number of abnormal changes in pupil size for each window is calculated by the method for determining the number of abnormal changes in pupil size in the neurophysiological data processing module, and the average value of all windows is calculated, that is, the pupil size The average of the number of abnormal changes in size was used as the baseline characteristic parameter of eye movement.
一种基于脑电与眼动数据的飞行员注意稳定状态实时监测方法，包括：A real-time monitoring method for pilots' attention steady state based on EEG and eye movement data, including:
步骤1，打开便携式脑电仪和眼动仪，确认脑电仪和眼动仪工作正常，驾驶员佩戴上脑电仪和眼动仪；Step 1, turn on the portable EEG and eye tracker, confirm that the EEG and eye tracker are working properly, and the driver wears the EEG and eye tracker;
步骤2，如果该驾驶员首次接受监测，则需要进行校准数据的采集，否则进入步骤3；校准数据的采集具体操作为：Step 2, if the driver is monitored for the first time, it is necessary to collect calibration data, otherwise, go to Step 3; the specific operations for the collection of calibration data are:
输入驾驶员编号、姓名、性别等基本信息；让驾驶员维持稳定坐姿，身体放松，双眼注视座舱内显示屏，待驾驶员处于静息休息状态后，进入校准数据采集模式，利用神经生理特征校准模块、脑电和眼动数据采集模块开始采集T秒的脑电基线数据和眼动基线数据，然后结合神经生理数据处理模块，计算脑电基线特征参数和眼动基线特征参数作为校准数据；校准数据采集完成后，系统进入待机状态，进入步骤4；Enter the basic information such as driver number, name, gender, etc.; let the driver maintain a stable sitting posture, relax, and look at the display screen in the cockpit. The module, EEG and eye movement data acquisition module starts to collect EEG baseline data and eye movement baseline data for T seconds, and then combines the neurophysiological data processing module to calculate the EEG baseline characteristic parameters and eye movement baseline characteristic parameters as calibration data; After the data collection is completed, the system enters the standby state and goes to step 4;
步骤3，如果该驾驶员已经进行过校准数据采集，则在系统中输入驾驶员编号，以调出驾驶员的校准数据；如果需要进行重新校准，则按照步骤2的操作进行重新采集，否则进入步骤4；Step 3, if the driver has already collected the calibration data, enter the driver number in the system to call up the driver's calibration data; step 4;
步骤4，在系统中开始注意稳定性监测：Step 4, start paying attention to stability monitoring in the system:
开始检测后，脑电和眼动数据采集模块将持续地接收来自脑电仪和眼动仪的脑电数据、眼动数据，并发送给神经生理数据处理模块；神经生理数据处理模块将以t秒为移动窗口，并结合神经生理特征校准模块中该驾驶员的校准数据，对脑电数据进行滤波、去伪迹、校准处理、特征提取，对眼动数据进行瞳孔大小异常变化次数特征提取并进行校准处理，最终从t秒的数据中提取脑电数据特征和眼动数据特征，并输出至注意稳定性计算模块；注意稳定性计算模块根据脑电数据特征和眼动数据特征，最终输出“注意正常”或“注意过度集中”两种注意状态标签。After the detection starts, the EEG and eye movement data acquisition module will continuously receive the EEG data and eye movement data from the EEG and eye tracker, and send them to the neurophysiological data processing module; the neurophysiological data processing module will use t Second is the moving window, and combined with the calibration data of the driver in the neurophysiological feature calibration module, the EEG data is filtered, de-artifacted, calibrated, and feature extracted, and the eye movement data is subjected to abnormal changes in pupil size. Carry out calibration processing, and finally extract the EEG data features and eye movement data features from the data of t seconds, and output them to the attention stability calculation module; the attention stability calculation module finally outputs " There are two types of attention state labels: normal attention” or “attention excessively concentrated”.
与现有技术相比，本发明具有以下技术特点：Compared with the prior art, the present invention has the following technical characteristics:
1.本发明的注意稳定性计算模块中的注意稳定性状态实时算法模型针对便携式脑电仪和眼动仪进行了优化，只需要单通道脑电数据和瞳孔大小数据便可以有效地输出注意稳定状态，解决了现有注意稳定状态监测方法在飞机座舱中应用的可行性问题。1. The attentional stability state real-time algorithm model in the attentional stability calculation module of the present invention is optimized for portable EEG and eye trackers, and only needs single-channel EEG data and pupil size data to effectively output attentional stability It solves the feasibility problem of the existing attention-steady state monitoring method applied in the aircraft cockpit.
2.本发明通过对脑电和眼动的实时采集、处理和分析，并通过集成脑电个性化校准模块对特定个体进行基于渐变连续操作任务的基线采集和数据校准，实现了注意稳定性状态的个性化的实时监测。2. The present invention realizes the stable state of attention by collecting, processing and analyzing the EEG and eye movement in real time, and by integrating the EEG individualized calibration module to perform baseline collection and data calibration based on the gradient continuous operation task for a specific individual. personalized real-time monitoring.
3.本发明通过经过心理学研究验证的渐变连续操作任务进行算法建模和数据校准，一定程度上脱离具体任务场景的注意稳定性状态监测，解决了以往希望不能跨场景进行注意稳定性状态监测的问题。3. The present invention performs algorithm modeling and data calibration through the gradual and continuous operation task verified by psychological research, and to a certain extent separates the attentional stability state monitoring of specific task scenarios, and solves the problem of not being able to perform attentional stability state monitoring across scenarios in the past. The problem.
附图说明Description of drawings
图1为本发明的系统结构示意图；Fig. 1 is the system structure schematic diagram of the present invention;
图2为本发明的使用流程示意图；Fig. 2 is the use flow schematic diagram of the present invention;
图3为便携式脑电采集仪示意图；3 is a schematic diagram of a portable EEG acquisition instrument;
图4为便携式眼动仪示意图；4 is a schematic diagram of a portable eye tracker;
图5为渐变连续操作任务实验范式实验材料示意图；Figure 5 is a schematic diagram of experimental materials for the experimental paradigm of the gradual continuous operation task;
图6为渐变连续操作任务实验范式实验注意状态分析示意图。Figure 6 is a schematic diagram of the experimental attention state analysis of the gradient continuous operation task experimental paradigm.
具体实施方式Detailed ways
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下多获得的所有其它实施例，都属于本发明保护的范围。The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.
本发明采用经过渐变连续操作任务实验范式进行人因学实验，采用便携式脑电和眼动设备采集数据，提取与注意稳定状态相关的神经生理特征，构建基于支持向量机的机器学习模型，并将所构建的模型用于飞行员注意稳定状态的实时监测。The invention adopts the experimental paradigm of continuous operation task through gradual change to conduct human factors experiments, adopts portable EEG and eye-tracking equipment to collect data, extracts neurophysiological features related to the steady state of attention, constructs a machine learning model based on support vector machine, and integrates The constructed model is used for real-time monitoring of pilot attention steady state.
参阅图1所示，本发明为一种基于脑电与眼动数据的飞行员注意稳定状态实时监测系统，包含脑电和眼动数据采集模块、神经生理数据处理模块、神经生理特征校准模块和注意稳定性计算模块，其中：Referring to Figure 1, the present invention is a real-time monitoring system for pilots' attention and steady state based on EEG and eye movement data, including an EEG and eye movement data acquisition module, a neurophysiological data processing module, a neurophysiological feature calibration module and an attention module. Stability calculation module, where:
1.脑电和眼动数据采集模块1. EEG and eye movement data acquisition module
脑电和眼动数据采集模块与用户佩戴的便携式脑电仪和眼动仪连接，并对脑电仪和眼动仪采集的脑电数据、眼动数据进行收集。The EEG and eye movement data acquisition module is connected to the portable EEG and eye tracker worn by the user, and collects the EEG data and eye movement data collected by the EEG and the eye tracker.
其中，便携式脑电仪可以采用北京汇心联科技的轻便式脑电采集仪，该脑电采集仪主要采集前额叶左右双通道脑电；便携式眼动仪可以采用Tobii公司的眼镜式眼动仪，该眼动仪可以采集瞳孔大小数据。注意，本发明不限于采用上述品牌型号的仪器。所述便携式脑电仪需要能收集1个通道、位于前额叶左侧单通道脑电信号，采样率为250Hz或以上。眼动仪需要能够采集单眼瞳孔大小，以毫米为单位，采样率须达到30Hz或以上。上述两种仪器都需要支持数据的实时传输功能，即通过蓝牙或wifi的形式把所采集的数据实时传输到PC机。Among them, the portable EEG instrument can use the portable EEG acquisition instrument of Beijing Huixinlian Technology, which mainly collects dual-channel EEG of the left and right prefrontal lobes; the portable eye tracker can use the glasses-type eye tracker of Tobii Company , the eye tracker can collect pupil size data. Note that the present invention is not limited to the use of the above-mentioned brands and models of instruments. The portable EEG instrument needs to be able to collect 1-channel, single-channel EEG signals located on the left side of the prefrontal lobe, with a sampling rate of 250 Hz or above. The eye tracker needs to be able to capture the pupil size of a single eye, measured in millimeters, at a sampling rate of 30Hz or more. Both of the above two instruments need to support the real-time transmission function of data, that is, real-time transmission of the collected data to the PC in the form of bluetooth or wifi.
2.神经生理数据处理模块2. Neurophysiological data processing module
神经生理数据处理模块用于对所采集的脑电数据进行滤波、去伪迹、校准处理、特征提取，得到脑电数据特征；对眼动数据进行瞳孔大小异常变化次数特征提取并进行校准处理，得到眼动数据特征；其中，利用脑电基线特征参数对去伪迹后的脑电数据进行校准处理，利用眼动基线特征参数对眼动数据提取后的特征进行校准处理。The neurophysiological data processing module is used to filter, remove artifacts, calibrate and extract features from the collected EEG data to obtain EEG data features; perform feature extraction and calibration processing on the eye movement data for abnormal changes in pupil size. Eye movement data features are obtained; wherein, the EEG data after artifact removal is calibrated by using the EEG baseline characteristic parameters, and the extracted features of the eye movement data are calibrated by using the eye movement baseline characteristic parameters.
神经生理数据处理模块以2秒为单位的滑动窗口对脑电和眼动数据进行处理。The neurophysiological data processing module processes the EEG and eye movement data with a sliding window of 2 seconds.
2.1脑电数据的处理过程2.1 The process of EEG data processing
2.1.1脑电数据的滤波2.1.1 Filtering of EEG data
对于每段实时获取的长度为2秒的脑电数据，首先进行滤波处理；滤波方法为4阶Butterworth滤波器，获得0.5Hz到100Hz的信号，以去除慢信号和高频噪声，由此得到滤波后的脑电原始信号。For each piece of EEG data acquired in real time with a length of 2 seconds, filtering is performed first; the filtering method is a 4th-order Butterworth filter to obtain signals from 0.5Hz to 100Hz to remove slow signals and high-frequency noise, thereby obtaining a filter post EEG raw signals.
2.1.2脑电数据的去伪迹处理2.1.2 De-artifact processing of EEG data
滤波处理后，进行去伪迹处理：脑电信号的伪迹主要为眨眼带来的伪迹，去伪迹处理过程为：After filtering, the artifact removal process is performed: the artifact of the EEG signal is mainly caused by blinking, and the artifact removal process is as follows:
首先，采用小波包分析的方法分辨出所述脑电原始信号中伪迹所在的时间区间：First, the wavelet packet analysis method is used to distinguish the time interval where the artifact in the original EEG signal is located:
采用Daubechies 4作为母小波的小波包分析，把脑电原始信号分解到7层，对第1-4层的小波包系数设为0，对第5-7层的小波包系数进行一个软阈值过滤处理，具体做法为把第5-7层中每一层的小波包系数的集合提取出来，通过软阈值公式计算出阈值；然后把大小超过软阈值的小波包系数设为0，而其它不变；其中，软阈值公式为：Using Daubechies 4 as the wavelet packet analysis of the mother wavelet, the original EEG signal is decomposed into 7 layers, the wavelet packet coefficients of the 1st to 4th layers are set to 0, and the wavelet packet coefficients of the 5th to 7th layers are filtered by a soft threshold. The specific method is to extract the set of wavelet packet coefficients of each layer in the 5th to 7th layers, and calculate the threshold value through the soft threshold formula; ; Among them, the soft threshold formula is:
以上公式中，Thres为某一层小波包系数的软阈值，median()为中位数，abs()为绝对值，coef为由某一层小波包系数的集合组成的数组。In the above formula, Thres is the soft threshold of the wavelet packet coefficients of a certain layer, median() is the median, abs() is the absolute value, and coef is an array composed of a set of wavelet packet coefficients of a certain layer.
小波包分析重构后的信号即只包含伪迹的信号；在只包含伪迹的信号中，找到大于3倍方差的点，这些点即为需要纠正伪迹；根据眨眼伪迹信号特性，需要纠正的伪迹所在的时间区间还需要往前延伸0.16秒、往后延伸0.84秒，以覆盖眨眼带来的电信号；由此获得的1秒区间即为伪迹区间。The signal reconstructed by wavelet packet analysis is the signal containing only artifacts; in the signal containing only artifacts, find the points greater than 3 times the variance, these points are the artifacts that need to be corrected; The time interval in which the corrected artifact is located also needs to be extended forward by 0.16 seconds and backward by 0.84 seconds to cover the electrical signal brought by the blinking; the 1-second interval thus obtained is the artifact interval.
其次，对于伪迹区间进行信号替换处理：Second, perform signal replacement processing for the artifact interval:
将所述滤波后的脑电原始信号经过Savitzky-Golay滤波处理；其中Savitzky-Golay滤波器阶数为3阶，窗口宽度为41；将伪迹区间中的数据替换成脑电原始信号经过Savitzky-Golay滤波处理的数据，即完成去伪迹处理。The filtered EEG original signal is processed by Savitzky-Golay filtering; wherein the order of the Savitzky-Golay filter is 3, and the window width is 41; the data in the artifact interval is replaced with the original EEG signal and processed by Savitzky-Golay The data processed by Golay filtering, that is, the de-artifact processing is completed.
2.1.3脑电数据的校准处理2.1.3 Calibration processing of EEG data
去伪迹处理后、提取特征之前，要对脑电数据进行校准处理，主要处理为从神经生理特征校准模块获得用户在特定状态，即静息休息状态下的脑电基线特征参数，包括脑电数据的平均值MEEG与方差SDEEG，并将滤波后的信号作如下变换：After de-artifact processing and before feature extraction, the EEG data should be calibrated. The main process is to obtain the EEG baseline characteristic parameters of the user in a specific state, that is, the resting state, from the neurophysiological feature calibration module, including EEG. The mean MEEG and variance SDEEG of the data, and the filtered signal is transformed as follows:
其中，EEGInd为校准处理后的脑电数据，EEGFiltered为完成了去伪迹处理的脑电数据。Among them, EEGInd is the EEG data after calibration processing, and EEGFiltered is the EEG data after the de-artifact processing has been completed.
2.1.4脑电数据的特征提取2.1.4 Feature extraction of EEG data
校准处理后，从脑电信号中计算出以下脑电数据特征：After the calibration process, the following EEG data features were calculated from the EEG signals:
(a)整个信号的平均值；(a) the average value of the entire signal;
(b)整个信号的方差；(b) the variance of the entire signal;
(c)整个信号的偏度；(c) the skewness of the entire signal;
(d)整个信号的峰度；(d) the kurtosis of the entire signal;
(e)整个信号的Hjorth移动性，其公式为
(f)整个信号的Hjorth复杂度，其公式为
(g)整个信号的熵值；(g) the entropy value of the entire signal;
(h)整个信号的Higuchi分形维数；(h) Higuchi fractal dimension of the entire signal;
(i)基于小波包分析的delta波(0.5-4Hz)的强度，即对校准处理后的脑电信号的脑电信号进行小波包分解，找出对应0.5-4Hz频段的小波包系数的数组，并求出该数组的平均值；(i) The intensity of the delta wave (0.5-4Hz) based on the wavelet packet analysis, that is, the EEG signal of the calibrated EEG signal is decomposed by the wavelet packet, and the array of wavelet packet coefficients corresponding to the 0.5-4Hz frequency band is found, and find the average of the array;
(j)基于小波包分析的delta波(0.5-4Hz)的偏度，即对校准处理后的脑电信号的脑电信号进行小波包分解，找出对应0.5-4Hz频段的小波包系数的数组，并求出该数组的偏度；(j) Skewness of delta wave (0.5-4Hz) based on wavelet packet analysis, that is, the EEG signal of the calibrated EEG signal is decomposed by wavelet packet, and the array of wavelet packet coefficients corresponding to the frequency band of 0.5-4Hz is found. , and find the skewness of the array;
(k)基于小波包分析的delta波(0.5-4Hz)的峰度，即对校准处理后的脑电信号的脑电信号进行小波包分解，找出对应0.5-4Hz频段的小波包系数的数组，并求出该数组的峰度；(k) The kurtosis of the delta wave (0.5-4Hz) based on the wavelet packet analysis, that is, the EEG signal of the calibrated EEG signal is decomposed by the wavelet packet, and the array of wavelet packet coefficients corresponding to the 0.5-4Hz frequency band is found. , and find the kurtosis of the array;
(l)基于小波包分析的theta波(4-8Hz)的强度，参考(i)特征的计算；(l) Intensity of theta wave (4-8 Hz) based on wavelet packet analysis, refer to the calculation of (i) feature;
(m)基于小波包分析的theta波(4-8Hz)的偏度，参考(j)特征的计算；(m) Skewness of theta waves (4-8 Hz) based on wavelet packet analysis, refer to the calculation of (j) features;
(n)基于小波包分析的theta波(4-8Hz)的峰度，参考(k)特征的计算；(n) kurtosis of theta wave (4-8Hz) based on wavelet packet analysis, refer to the calculation of (k) feature;
(o)基于小波包分析的alpha波(8-12Hz)的强度，参考(i)特征的计算；(o) Intensity of alpha wave (8-12 Hz) based on wavelet packet analysis, refer to the calculation of (i) feature;
(p)基于小波包分析的alpha波(8-12Hz)的偏度，参考(j)特征的计算；(p) Skewness of alpha wave (8-12 Hz) based on wavelet packet analysis, refer to the calculation of (j) feature;
(q)基于小波包分析的alpha波(8-12Hz)的峰度，参考(k)特征的计算；(q) The kurtosis of the alpha wave (8-12 Hz) based on the wavelet packet analysis, refer to the calculation of the (k) feature;
(r)基于小波包分析的beta波(12-30Hz)的强度，参考(i)特征的计算；(r) Intensity of beta waves (12-30 Hz) based on wavelet packet analysis, with reference to the calculation of (i) features;
(s)基于小波包分析的beta波(12-30Hz)的偏度，参考(j)特征的计算；(s) Skewness of beta waves (12-30 Hz) based on wavelet packet analysis, refer to the calculation of (j) features;
(t)基于小波包分析的beta波(12-30Hz)的峰度，参考(k)特征的计算；(t) kurtosis of beta waves (12-30 Hz) based on wavelet packet analysis, with reference to the calculation of (k) features;
(u)基于小波包分析的beta波(30-100Hz)的强度，参考(i)特征的计算；(u) Intensity of beta waves (30-100 Hz) based on wavelet packet analysis, with reference to the calculation of (i) features;
(v)基于小波包分析的beta波(30-100Hz)的偏度，参考(j)特征的计算；(v) Skewness of beta waves (30-100 Hz) based on wavelet packet analysis, with reference to the calculation of (j) features;
(w)基于小波包分析的beta波(30-100Hz)的峰度，参考(k)特征的计算。(w) Kurtosis of beta waves (30-100 Hz) based on wavelet packet analysis, with reference to the calculation of (k) features.
2.1.5眼动数据的瞳孔大小异常变化次数确定2.1.5 Determination of the number of abnormal changes in pupil size of eye movement data
对于眼动数据，需要利用眼动数据中的瞳孔大小数据，从中提取瞳孔大小异常变化次数特征。For the eye movement data, it is necessary to use the pupil size data in the eye movement data to extract the feature of the number of abnormal changes in pupil size.
对于每段实时获取的长度为2秒的瞳孔大小眼动数据，首先，对该数据进行小波包分析，采用Daubechies 8作为母小波，分解到第2层，并对第2层的小波包系数进行一个软阈值过滤处理；处理方式同脑电数据的软阈值过滤处理，在此不赘述；过滤后的非零小波包系数的个数即为瞳孔大小异常变化次数。For each segment of pupil size eye movement data with a length of 2 seconds obtained in real time, first, the data is subjected to wavelet packet analysis, using Daubechies 8 as the mother wavelet, decomposed into the second layer, and the wavelet packet coefficients of the second layer are analyzed. A soft threshold filtering process; the processing method is the same as the soft threshold filtering process of EEG data, and will not be repeated here; the number of non-zero wavelet packet coefficients after filtering is the number of abnormal changes in pupil size.
2.1.6眼动数据的校准处理与特征提取2.1.6 Calibration processing and feature extraction of eye movement data
将瞳孔大小异常变化次数减去从神经生理特征校准模块获得用户在特定状态，即静息休息状态下的眼动基线特征参数，即瞳孔大小异常变化次数的平均值，得到校准后的瞳孔大小异常变化次数特征，作为从眼动数据中提取的唯一眼动数据特征。Subtract the number of abnormal changes in pupil size from the baseline characteristic parameters of the user's eye movement in a specific state, namely the resting state, obtained from the neurophysiological feature calibration module, that is, the average number of abnormal changes in pupil size, to obtain the abnormal pupil size after calibration The number of changes feature, as the only eye movement data feature extracted from the eye movement data.
最后，从长度为2秒的脑电和眼动信号中，共提取24个特征(脑电数据特征23个，眼动数据特征1个)；这24个特征以数组的形式输入给注意稳定性计算模块。Finally, from the EEG and eye movement signals with a length of 2 seconds, a total of 24 features (23 features for EEG data and 1 feature for eye movement data) are extracted; these 24 features are input to attention stability in the form of an array calculation module.
3.神经生理特征校准模块3. Neurophysiological feature calibration module
神经生理特征校准模块利用所述脑电和眼动数据采集模块获取用户在特定状态下的脑电基线数据和眼动基线数据，并输出至所述神经生理数据处理模块对脑电基线数据进行滤波、去伪迹处理，对眼动基线数据进行瞳孔大小异常变化次数特征提取，得到脑电基线特征参数和眼动基线特征参数。该模块还用于记录用户的基本信息，包括用户编号、姓名、性别等。The neurophysiological feature calibration module uses the EEG and eye movement data acquisition module to obtain the EEG baseline data and eye movement baseline data of the user in a specific state, and outputs to the neurophysiological data processing module to filter the EEG baseline data , Artifact removal processing, extracting the number of abnormal changes in pupil size from the eye movement baseline data to obtain the EEG baseline characteristic parameters and the eye movement baseline characteristic parameters. This module is also used to record the user's basic information, including user ID, name, gender, etc.
针对每个用户，需要在用户处于特定状态，即静息休息状态时，采集30秒的脑电数据和眼动数据作为脑电基线数据和眼动基线数据，以提高注意稳定状态识别准确率。For each user, it is necessary to collect 30 seconds of EEG data and eye movement data as EEG baseline data and eye movement baseline data when the user is in a specific state, that is, in a resting state, so as to improve the recognition accuracy of the steady state of attention.
对于脑电基线数据，30秒的基线数据需要以2秒为一个单位(固定窗口)，通过上述“神经生理数据处理模块”中的滤波和去伪迹操作后，计算每个窗口的数据的平均值mEEG与方差sdEEG，并把所有窗口的平均值MEEG与方差SDEEG储存作为脑电基线特征参数，并实时输入到“神经生理数据处理模块”以进行脑电数据校准。For EEG baseline data, the 30-second baseline data needs to take 2 seconds as a unit (fixed window). After filtering and de-artifacting operations in the above "neurophysiological data processing module", calculate the average of the data in each window The values mEEG and variance sdEEG are stored, and the mean values of all windows MEEG and variance SDEEG are stored as EEG baseline characteristic parameters, and are input to the "neurophysiological data processing module" in real time for EEG data calibration.
对于眼动基线数据，30秒的基线数据需要以2秒为一个单位(固定窗口)，通过上述“神经生理数据处理模块”中的瞳孔大小异常变化次数确定方法计算每个窗口的瞳孔大小异常变化次数值，并计算所有窗口的平均值，即瞳孔大小异常变化次数的平均值作为眼动基线特征参数，并实时输入到“神经生理数据处理模块”以进行眼动数据校准。For the eye movement baseline data, the 30-second baseline data needs to be taken as a unit of 2 seconds (fixed window), and the abnormal pupil size changes in each window are calculated by the method for determining the number of abnormal pupil size changes in the above "neurophysiological data processing module". and calculate the average value of all windows, that is, the average number of abnormal changes in pupil size, as the baseline characteristic parameter of eye movement, and input it to the "neurophysiological data processing module" in real time for eye movement data calibration.
4.注意稳定性计算模块4. Pay attention to the stability calculation module
注意稳定性计算模块用于将神经生理数据处理模块输出的脑电数据特征和眼动数据特征作为输入，通过训练好的注意稳定性状态模型，输出注意稳定性状态标签。The attentional stability calculation module is used to take the EEG data features and eye movement data features output by the neurophysiological data processing module as input, and output the attentional stability state label through the trained attentional stability state model.
本方案中，所述注意稳定性状态模型采用非线性支持向量机模型。In this solution, the attentional stability state model adopts a nonlinear support vector machine model.
注意稳定性状态模型训练完成后，接收“神经生理数据处理模块”输出的脑电数据特征和眼动数据特征，具体样式为{f1,f2,…,f24}，其中，f1-f24为以2秒为窗口、从脑电数据和眼动数据中所提取的24个特征；该模型的输出为“注意正常”或“注意过度集中”两种标签。Note that after the training of the stability state model is completed, the EEG data features and eye movement data features output by the "neurophysiological data processing module" are received, and the specific pattern is {f1 ,f2 ,...,f24 }, where f1 - f24 is the 24 features extracted from EEG data and eye movement data with a 2-second window; the output of the model is two labels of "attention normal" or "over-attention".
实施例：Example:
注意稳定性状态模型的训练主要通过渐变连续操作任务实验收集数据并建模；本实施例中，建模数据包括参与实验的30名被试。Note that the training of the stable state model mainly collects data and models through the gradient continuous operation task experiment; in this embodiment, the modeling data includes 30 subjects participating in the experiment.
渐变连续操作任务的实验材料包含两类圆形灰度图，分别是城市风光和山地景色(各10张)，图1为“城市”和“山”的图例。实验任务中，“城市”和“山”分别以90％和10％的概率随机呈现，相邻试次不会重复呈现同一张图片。前一张图片到下一张图片的切换是通过透明度的渐变来实现的，相邻图片的过渡期为0.8秒。例如，第一个0.8秒内，第一张图片的透明度从0％(完全清晰)渐变为100％(完全透明)，与此同时，第二张图片的透明度从100％渐变为0％；下一个周期内，第二张图片的透明度从0％渐变为100％，同时，第三张图片的透明度从100％变为0％，依次类推。除每组正式实验中的第一张图片外，其余每张图片的透明度是从100％渐变到0％再渐变到100％，跨越两个周期，共计1.6秒。任务要求被试验者看到“城市”(90％)按空格键，看到“山”(10％)不做按键反应。The experimental materials for the gradient continuous operation task consist of two types of circular grayscale images, namely city scenery and mountain scenery (10 images each). Figure 1 shows the legend of "city" and "mountain". In the experimental task, "city" and "mountain" were randomly presented with a probability of 90% and 10%, respectively, and the same picture would not be presented repeatedly in adjacent trials. The transition from the previous image to the next image is achieved through a gradient of transparency, and the transition period between adjacent images is 0.8 seconds. For example, in the first 0.8 seconds, the transparency of the first image fades from 0% (completely clear) to 100% (completely transparent), and at the same time, the transparency of the second image fades from 100% to 0%; In one cycle, the transparency of the second image fades from 0% to 100%, while the transparency of the third image changes from 100% to 0%, and so on. Except for the first image in each set of formal experiments, the transparency of each image is ramped from 100% to 0% and then to 100%, spanning two cycles for a total of 1.6 seconds. The task required subjects to press the space bar when they saw "city" (90%), and did not respond to the key when they saw "mountain" (10%).
被试到达实验室后，需要先阅读并签署知情同意书。实验正式开始之前，被试需要坐在屏幕前，调整座椅高度与腮托(用于固定下巴，保持眼睛与屏幕中央达到要求距离)，之后进行眼睛的校准。完成校准后，呈现实验指导语，要求被试大概了解实验流程并熟悉全部20张实验使用的图片(被标记为“城市”和“山”的图片)，然后被试验者需要进行2分钟的练习以掌握任务要求，之后实验正式开始。实验共包含3组正式实验，每组时长为8分钟，两组实验的中间有2分钟休息时间。After the subjects arrived at the laboratory, they were required to read and sign the informed consent form. Before the experiment officially started, the subjects needed to sit in front of the screen, adjust the seat height and chin rest (used to fix the chin and keep the eyes and the center of the screen at the required distance), and then calibrate the eyes. After completing the calibration, the experimental instructions are presented, and the subjects are required to have a general understanding of the experimental process and be familiar with all 20 pictures used in the experiment (the pictures marked as "city" and "mountain"), and then the subjects need to practice for 2 minutes To master the task requirements, the experiment officially begins. The experiment consisted of 3 groups of formal experiments, each group was 8 minutes long, and there was a 2 minute rest period between the two groups of experiments.
被试的注意状态由被试在任务中的反应时经计算得出。其中，数据集中的反应时数据平均每0.8秒有一个值，对应每张图片的呈现时长。对反应时数据的处理如下：首先，从反应时数据中计算时间进程变异性值。时间进程变异性值是度量反应时可变性的指标，对于每一个被试来说，将其所有的反应时转化成对应的z分数(即标准分数)，每个试次与其总体平均反应时的绝对偏差则为时间进程变异性值。划分不同注意状态时，先对每组实验内时间进程变异性值时间序列进行高斯核平滑处理。以平滑处理后时间进程变异性值(每组内)的75分位数值为划分标准，低于75分位数值的状态划分为“注意正常”，高于75分位数值的状态划分为“注意过度集中”。由此，数据集中每0.8秒为一段的时间窗口就对应了一个注意状态的标签。The subject's attentional state was calculated from the subject's reaction time in the task. Among them, the reaction time data in the dataset has an average value every 0.8 seconds, corresponding to the presentation time of each picture. The reaction time data were processed as follows: First, time course variability values were calculated from the reaction time data. The time course variability value is an index to measure the variability of response times. For each subject, all of its response times are converted into corresponding z-scores (ie, standard scores), and the difference between each trial and its overall average response time. The absolute deviation is the time course variability value. When dividing different attention states, Gaussian kernel smoothing is first performed on the time series of time course variability values within each group of experiments. The 75th quantile value of the time course variability value (within each group) after smoothing is used as the classification standard, the state below the 75th quantile value is divided into "attention to normal", and the state higher than the 75th quantile value is divided into "attention" Too concentrated". Thus, every 0.8 second time window in the dataset corresponds to a label of an attention state.
构建训练数据集时，由于数据标签以0.8秒为单位标记，而脑电和眼动特征提取则以2秒为单位进行，因此需要做以下处理：When building the training dataset, since the data labels are marked in units of 0.8 seconds, and the EEG and eye movement feature extraction is performed in units of 2 seconds, the following processing is required:
首先，脑电和眼动特征提取以2秒为移动窗口、0.5秒为步长进行提取，在每个窗口中提出得出前述24个特征；此24个特征的数据以及该移动窗口所对应的开始和结束时间记为一行数据：{f1,f2,…,f24,ts,te}，其中，f1-f24为所提取的24个特征，ts为时间窗口开始时间，te为时间窗口的结束时间。然后，检查时间窗口内所对应的所有注意状态标签，如果窗口内标签均为“注意正常”，则该行数据成为一个样本，所对应的标签为“注意正常”；如果窗口内标签均为“注意过度集中”，则该行数据成为一个样本，所对应的标签为“注意过度集中”；如果窗口内标签既有“注意正常”、也有“注意过度集中”，则舍弃窗口内数据。最后，用于训练的样本格式为{f1,f2,…,f24,label}，其中，f1-f24为所提取的24个特征，label为标签。First, the EEG and eye movement feature extraction is performed with a moving window of 2 seconds and a step of 0.5 seconds, and the aforementioned 24 features are proposed in each window; the data of these 24 features and the corresponding data of the moving window The start and end times are recorded as a row of data: {f1 ,f2 ,…,f24 ,ts,te}, where f1 -f24 are the 24 extracted features, ts is the start time of the time window, and te is The end time of the time window. Then, check all the corresponding attention status labels in the time window. If the labels in the window are all "Attention Normal", the row of data becomes a sample, and the corresponding label is "Attention Normal"; if the labels in the window are "Attention Normal" Pay attention to over-concentration", the row of data becomes a sample, and the corresponding label is "attention over-concentration"; if the label in the window has both "attention normal" and "attention over-concentration", the data in the window is discarded. Finally, the sample format used for training is {f1 ,f2 ,...,f24 ,label}, where f1 -f24 are the extracted 24 features, and label is the label.
按照本方案建立系统以后，在实际应用时，对于一个待进行监测的驾驶员，需要经过以下步骤：After the system is established according to this scheme, in practical application, a driver to be monitored needs to go through the following steps:
步骤1，打开便携式脑电仪和眼动仪，确认脑电仪和眼动仪工作正常，驾驶员佩戴上脑电仪和眼动仪。Step 1. Turn on the portable EEG and eye tracker, confirm that the EEG and eye tracker are working properly, and the driver wears the EEG and eye tracker.
步骤2，如果该驾驶员首次接受监测，则需要进行校准数据的采集，否则进入步骤3；校准数据的采集具体操作为：Step 2, if the driver is monitored for the first time, it is necessary to collect calibration data, otherwise, go to Step 3; the specific operations for the collection of calibration data are:
输入驾驶员编号、姓名、性别等基本信息；让驾驶员维持稳定坐姿，身体放松，双眼注视座舱内显示屏，待驾驶员处于静息休息状态后，进入校准数据采集模式，利用神经生理特征校准模块、脑电和眼动数据采集模块开始采集30秒的脑电基线数据和眼动基线数据，然后结合神经生理数据处理模块，计算脑电基线特征参数和眼动基线特征参数作为校准数据；校准数据采集完成后，系统进入待机状态，进入步骤4。Enter the basic information such as driver number, name, gender, etc.; let the driver maintain a stable sitting posture, relax, and look at the display screen in the cockpit. The module, EEG and eye movement data acquisition module starts to collect the EEG baseline data and eye movement baseline data for 30 seconds, and then combines the neurophysiological data processing module to calculate the EEG baseline characteristic parameters and eye movement baseline characteristic parameters as calibration data; calibration; After the data collection is completed, the system enters the standby state and goes to step 4.
步骤3，如果该驾驶员已经进行过校准数据采集，则在系统中输入驾驶员编号，以调出驾驶员校准数据；如果需要进行重新校准，则按照步骤2的操作进行重新采集，否则进入步骤4。Step 3, if the driver has already collected calibration data, enter the driver number in the system to call up the driver calibration data; if recalibration is required, follow the operation of step 2 to recollect, otherwise go to step 2 4.
步骤4，在系统中开始注意稳定性监测：Step 4, start paying attention to stability monitoring in the system:
开始检测后，脑电和眼动数据采集模块将持续地接收来自脑电仪和眼动仪的脑电数据、眼动数据，并发送给神经生理数据处理模块；神经生理数据处理模块将以2秒为移动窗口，并结合神经生理特征校准模块中该驾驶员的校准数据，对脑电数据进行滤波、去伪迹、校准处理、特征提取，对眼动数据进行瞳孔大小异常变化次数特征提取并进行校准处理，最终从2秒的数据中提取24个脑电数据特征和眼动数据特征，并输出至注意稳定性计算模块；注意稳定性计算模块根据脑电数据特征和眼动数据特征，最终输出“注意正常”或“注意过度集中”两种注意状态标签。After the detection starts, the EEG and eye movement data acquisition module will continuously receive the EEG data and eye movement data from the EEG and eye tracker, and send them to the neurophysiological data processing module; the neurophysiological data processing module will use 2 Second is the moving window, and combined with the calibration data of the driver in the neurophysiological feature calibration module, the EEG data is filtered, de-artifacted, calibrated, and feature extracted, and the eye movement data is subjected to abnormal changes in pupil size. Perform calibration processing, and finally extract 24 EEG data features and eye movement data features from the 2-second data, and output them to the attentional stability calculation module; Outputs two attention state labels, "attention is normal" or "attention is excessively concentrated".
值得注意的是，上述系统实施例中，所包括的各个单元只是按照功能逻辑进行划分的，但并不局限于上述的划分，只要能够实现相应的功能即可；另外，各功能单元的具体名称也只是为了便于相互区分,并不用于限制本发明的保护范围。It is worth noting that, in the above system embodiment, the units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units It is only for the convenience of distinguishing from each other, and is not used to limit the protection scope of the present invention.
另外，本领域普通技术人员可以理解实现上述各实施例方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，相应的程序可以存储于一计算机可读取存储介质中，所述的存储介质，如ROM/RAM、磁盘或光盘等。In addition, those of ordinary skill in the art can understand that all or part of the steps in implementing the methods of the above embodiments can be completed by instructing relevant hardware through a program, and the corresponding program can be stored in a computer-readable storage medium. storage medium, such as ROM/RAM, magnetic disk or optical disk, etc.
以上实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的精神和范围，均应包含在本申请的保护范围之内。The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: The recorded technical solutions are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application, and should be included in the application. within the scope of protection.
Claims (9)
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210464894.4ACN114947848A (en) | 2022-04-29 | 2022-04-29 | A real-time monitoring system for pilots' attention and steady state based on EEG and eye movement data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210464894.4ACN114947848A (en) | 2022-04-29 | 2022-04-29 | A real-time monitoring system for pilots' attention and steady state based on EEG and eye movement data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114947848Atrue CN114947848A (en) | 2022-08-30 |
Family
ID=82979212
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210464894.4APendingCN114947848A (en) | 2022-04-29 | 2022-04-29 | A real-time monitoring system for pilots' attention and steady state based on EEG and eye movement data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114947848A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115844404A (en)* | 2023-03-01 | 2023-03-28 | 中国民航大学 | Controller attention characteristic evaluation method and device based on eye movement data |
| CN118708067A (en)* | 2024-08-27 | 2024-09-27 | 天目山实验室 | Real-time monitoring system and method for attention status of low-altitude air traffic controllers based on EEG characteristics |
| CN118806299A (en)* | 2024-09-18 | 2024-10-22 | 博睿康医疗科技(上海)有限公司 | Signal artifact detection method and detection system, signal artifact removal method |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070273611A1 (en)* | 2004-04-01 | 2007-11-29 | Torch William C | Biosensors, communicators, and controllers monitoring eye movement and methods for using them |
| CN103956028A (en)* | 2014-04-23 | 2014-07-30 | 山东大学 | Automobile multielement driving safety protection method |
| CN106236027A (en)* | 2016-08-23 | 2016-12-21 | 兰州大学 | Depressed crowd's decision method that a kind of brain electricity combines with temperature |
| CN109431497A (en)* | 2018-10-23 | 2019-03-08 | 南京医科大学 | A kind of brain-electrical signal processing method and epilepsy detection system |
| CN110801237A (en)* | 2019-11-10 | 2020-02-18 | 中科搏锐（北京）科技有限公司 | Cognitive ability assessment system and method based on eye movement and electroencephalogram characteristics |
| CN111956219A (en)* | 2020-08-27 | 2020-11-20 | 济南大学 | Electroencephalogram signal-based emotion feature identification method and system and emotion feature identification and adjustment system |
| CN113143273A (en)* | 2021-03-23 | 2021-07-23 | 陕西师范大学 | Intelligent detection system and method for attention state of learner in online video learning |
- 2022
- 2022-04-29CNCN202210464894.4Apatent/CN114947848A/enactivePending
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20070273611A1 (en)* | 2004-04-01 | 2007-11-29 | Torch William C | Biosensors, communicators, and controllers monitoring eye movement and methods for using them |
| CN103956028A (en)* | 2014-04-23 | 2014-07-30 | 山东大学 | Automobile multielement driving safety protection method |
| CN106236027A (en)* | 2016-08-23 | 2016-12-21 | 兰州大学 | Depressed crowd's decision method that a kind of brain electricity combines with temperature |
| CN109431497A (en)* | 2018-10-23 | 2019-03-08 | 南京医科大学 | A kind of brain-electrical signal processing method and epilepsy detection system |
| CN110801237A (en)* | 2019-11-10 | 2020-02-18 | 中科搏锐（北京）科技有限公司 | Cognitive ability assessment system and method based on eye movement and electroencephalogram characteristics |
| CN111956219A (en)* | 2020-08-27 | 2020-11-20 | 济南大学 | Electroencephalogram signal-based emotion feature identification method and system and emotion feature identification and adjustment system |
| CN113143273A (en)* | 2021-03-23 | 2021-07-23 | 陕西师范大学 | Intelligent detection system and method for attention state of learner in online video learning |
Non-Patent Citations (2)
| Title |
|---|
| 张娟;朱文强;李晓伟;刘浩学;: "连续长坡路段组合线形与驾驶员瞳孔大小关系的试验分析", 安全与环境学报, no. 01, 25 February 2018 (2018-02-25)* |
| 王庆敏;姚永杰;李科华;时粉周;刘秋红;: "飞行员眼动追踪与航空工效学的研究现状", 海军医学杂志, no. 01, 28 January 2016 (2016-01-28)* |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115844404A (en)* | 2023-03-01 | 2023-03-28 | 中国民航大学 | Controller attention characteristic evaluation method and device based on eye movement data |
| CN115844404B (en)* | 2023-03-01 | 2023-05-12 | 中国民航大学 | Eye movement data-based controller attention characteristic evaluation method and device |
| CN118708067A (en)* | 2024-08-27 | 2024-09-27 | 天目山实验室 | Real-time monitoring system and method for attention status of low-altitude air traffic controllers based on EEG characteristics |
| CN118806299A (en)* | 2024-09-18 | 2024-10-22 | 博睿康医疗科技(上海)有限公司 | Signal artifact detection method and detection system, signal artifact removal method |
| CN118806299B (en)* | 2024-09-18 | 2024-12-06 | 博睿康医疗科技(上海)有限公司 | Signal artifact detection method, signal artifact detection system and signal artifact removal method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114947848A (en) | A real-time monitoring system for pilots' attention and steady state based on EEG and eye movement data | |
| Minhad et al. | Happy-anger emotions classifications from electrocardiogram signal for automobile driving safety and awareness | |
| Kleifges et al. | BLINKER: Automated extraction of ocular indices from EEG enabling large-scale analysis | |
| JP7373555B2 (en) | Quantification of motor function using EEG signals | |
| Varone et al. | Finger pinching and imagination classification: A fusion of CNN architectures for IoMT-enabled BCI applications | |
| Schwarz et al. | Analyzing and decoding natural reach-and-grasp actions using gel, water and dry EEG systems | |
| Tcheslavski et al. | Alcoholism-related alterations in spectrum, coherence, and phase synchrony of topical electroencephalogram | |
| CN117918863A (en) | A method and system for real-time artifact processing and feature extraction of electroencephalogram signals | |
| Dovgialo et al. | Assessment of statistically significant command-following in pediatric patients with disorders of consciousness, based on visual, auditory and tactile event-related potentials | |
| WO2018081980A1 (en) | Neurovascular coupling analytical method based on electroencephalogram and functional near infrared spectroscopy technology | |
| CN112957049A (en) | Attention state monitoring device and method based on brain-computer interface equipment technology | |
| Rajan et al. | Feasibility analysis of portable electroencephalography based abnormal fatigue detection and tele-surveillance system | |
| Campanella et al. | PPG and EDA dataset collected with Empatica E4 for stress assessment | |
| Mai et al. | 1D-CNN-based BCI system for detecting Emotional states using a Wireless and Wearable 8-channel Custom-designed EEG Headset | |
| Zhao et al. | Wearable EEG-based real-time system for depression monitoring | |
| CN110569968B (en) | Method and system for evaluating entrepreneurship failure resilience based on electrophysiological signals | |
| Masaki et al. | A quantitative evaluation on the software use experience with electroencephalogram | |
| Saidatul et al. | Automated system for stress evaluation based on EEG signal: A prospective review | |
| Pandiyan et al. | Mental stress level classification using eigenvector features and principal component analysis | |
| CN112587796A (en) | Method and device for quantifying wake-up promotion effect of deep brain electric stimulation | |
| Romaniuk et al. | Eye movement assessment methodology based on wearable eeg headband data analysis | |
| Khatter et al. | Study of various automatic eeg artifact removal techniques | |
| Abreu et al. | PreEpiSeizures: description and outcomes of physiological data acquisition using wearable devices during video-EEG monitoring in people with epilepsy | |
| Sutharsan et al. | Electroencephalogram signal processing with independent component analysis and cognitive stress classification using convolutional neural networks | |
| CN106473704B (en) | Method and system for removing ocular artifacts in sleep state analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |