技术领域Technical field
本发明属于运动想象脑电信号解码技术领域，尤其涉及一种运动想象脑电信号解码方法、系统、介质及设备。The invention belongs to the technical field of motor imagination EEG signal decoding, and in particular relates to a motor imagination EEG signal decoding method, system, medium and equipment.
背景技术Background technique
本部分的陈述仅仅是提供了与本发明相关的背景技术信息，不必然构成在先技术。The statements in this section merely provide background technical information related to the present invention and do not necessarily constitute prior art.
脑机接口(Brain-computer Interface,BCI)是一种建立大脑与外部设备的直接连接的技术，被广泛的应用于人机交互、运动康复和疾病治疗等领域。常用的脑机接口范式有稳态视觉诱发(Steady-State Visual Evoked Potentials,SSVEP)、P300和运动想象(Motor Imagery,MI)，其中MI是最有应用前景的一种范式。MI BCI通常使用脑电图(EEG)信号来检测运动想象，可以让使用者通过想象运动来控制设备，例如移动电动轮椅、光标和上肢机器人等。然而，由于大脑活动的不稳定和低信噪比(SNR)，脑电图信号会产生不同的结果。并且脑电图信号的高维性、主题依赖性和通道相关性使得脑信号的分析和分类成为一项具有挑战性的任务。Brain-computer Interface (BCI) is a technology that establishes a direct connection between the brain and external devices. It is widely used in fields such as human-computer interaction, sports rehabilitation, and disease treatment. Commonly used brain-computer interface paradigms include Steady-State Visual Evoked Potentials (SSVEP), P300 and Motor Imagery (MI), among which MI is the most promising paradigm. MI BCI usually uses electroencephalogram (EEG) signals to detect motor imagination, which allows users to control devices such as mobile electric wheelchairs, cursors, and upper-limb robots by imagining movements. However, EEG signals can produce different results due to instability in brain activity and low signal-to-noise ratio (SNR). And the high dimensionality, subject dependence, and channel correlation of EEG signals make the analysis and classification of brain signals a challenging task.
目前用于MI EEG信号解码的方法主要有传统机器学习(ML)和深度学习(DL)两种方法。其中，传统的机器学习算法例如支持向量机(SVM)、线性判别分析(LDA)和K近邻(KNN)被用来MI的分类。然而，特征的提取需要丰富的先验知识，机器学习算法过度依赖特征选择，因此为不同任务提取合适的特征和选择合适的算法仍然是一个挑战。另外，卷积神经网络(CNN)是MI分类中最广泛使用的结构，但是卷积神经网络无法提取时间序列数据的长时间的依赖特征，而且使用单一卷积模式和卷积核大小，无法有效地提取多尺度高级时空特征，限制了MI脑电信号的分类性能。Currently, there are two main methods used for MI EEG signal decoding: traditional machine learning (ML) and deep learning (DL). Among them, traditional machine learning algorithms such as support vector machine (SVM), linear discriminant analysis (LDA) and K nearest neighbor (KNN) are used for MI classification. However, feature extraction requires rich prior knowledge, and machine learning algorithms overly rely on feature selection, so extracting appropriate features and selecting appropriate algorithms for different tasks remains a challenge. In addition, convolutional neural network (CNN) is the most widely used structure in MI classification, but convolutional neural network cannot extract long-term dependence features of time series data, and it cannot be effective using a single convolution mode and convolution kernel size. Extracting multi-scale advanced spatiotemporal features limits the classification performance of MI EEG signals.
综上所述，传统的用于MI EEG信号解码的方法针对非平稳性和个体差异性的脑电信号存在解码精度低的问题。In summary, traditional methods for MI EEG signal decoding have problems with low decoding accuracy for non-stationary and individual differences in EEG signals.
发明内容Contents of the invention
为了解决上述背景技术中存在的技术问题，本发明提供一种运动想象脑电信号解码方法及系统，其基于多头注意力机制和时间卷积网络对运动想象脑电信号进行解码，能够有效的提升运动想象脑电信号的被试内和跨被试的解码性能。In order to solve the technical problems existing in the above background technology, the present invention provides a motor imagination EEG signal decoding method and system, which decodes the motor imagination EEG signal based on a multi-head attention mechanism and a temporal convolution network, and can effectively improve Within-subject and cross-subject decoding performance of motor imagery EEG signals.
为了实现上述目的，本发明采用如下技术方案：In order to achieve the above objects, the present invention adopts the following technical solutions:
本发明的第一个方面提供一种运动想象脑电信号解码方法。A first aspect of the present invention provides a motor imagery EEG signal decoding method.
一种运动想象脑电信号解码方法，其包括：A method for decoding electroencephalogram signals from motor imagery, which includes:
获取运动想象脑电信号并进行归一化处理；Obtain motor imagery EEG signals and normalize them;
利用训练完成的EISATC-Fusion模型解码归一化的运动想象脑电信号，预测出运动想象脑电信号的类别；其中，所述EISATC-Fusion模型由EEGInc模块、多头自注意力模块、时间卷积网络模块、融合模块及输出模块构成；Use the trained EISATC-Fusion model to decode the normalized motor imagery EEG signal and predict the category of the motor imagery EEG signal; wherein, the EISATC-Fusion model consists of an EEGInc module, a multi-head self-attention module, and a temporal convolution Composed of network module, fusion module and output module;
利用EISATC-Fusion模型解码归一化的运动想象脑电信号的过程为：The process of using the EISATC-Fusion model to decode the normalized motor imagery EEG signal is:
利用EEGInc模块提取归一化的运动想象脑电信号的频率特征、特定频率的空间特征及多尺度时间信息，得到初始时空特征；Use the EEGInc module to extract the frequency characteristics, spatial characteristics of specific frequencies and multi-scale time information of the normalized motor imagery EEG signal to obtain the initial spatio-temporal characteristics;
将初始时空特征经多头自注意力模块处理，得到全局具有长时依赖的特征；The initial spatio-temporal features are processed by the multi-head self-attention module to obtain global features with long-term dependence;
将初始时空特征与全局具有长时依赖的特征在深度的维度上进行特征融合，再经时间卷积网络模块提取高维度时间特征；The initial spatio-temporal features and global long-term dependency features are fused in the depth dimension, and then high-dimensional temporal features are extracted through the temporal convolution network module;
将EEGInc模块的输出与时间卷积网络模块的输出分别作为输入，经全连接层对应得到运动想象脑电信号属于每个类别的两个预测结果，再利用可学习张量对两个预测结果进行决策级融合；The output of the EEGInc module and the output of the temporal convolutional network module are used as input respectively. Through the fully connected layer, two prediction results of the motor imagery EEG signal belonging to each category are obtained, and then the learnable tensor is used to perform the two prediction results. Decision-level integration;
在输出模块中，将决策融合的结果输入到Softmax函数中，得到最终的运动想象脑电信号的类别。In the output module, the decision fusion results are input into the Softmax function to obtain the final motor imagery EEG signal category.
作为一种实施方式，所述EISATC-Fusion模型训练过程分为两个阶段：As an implementation manner, the EISATC-Fusion model training process is divided into two stages:
在第一阶段，训练集被用来训练模型，在验证集上使用早停策略获取最优训练模型；其中，在早停策略中，将验证集的准确率和损失作为模型训练早停的指标；In the first stage, the training set is used to train the model, and the early stopping strategy is used on the verification set to obtain the optimal training model; in the early stopping strategy, the accuracy and loss of the verification set are used as indicators for early stopping of model training. ;
在第二阶段，训练集和验证集都被用来训练第一阶段获取的模型，当模型在验证集上的损失小于或等于模型在第一阶段训练集上的最小损失，则结束第二阶段的训练。In the second stage, both the training set and the verification set are used to train the model obtained in the first stage. When the loss of the model on the verification set is less than or equal to the minimum loss of the model on the training set of the first stage, the second stage ends. training.
作为一种实施方式，使用Z-score标准化方法对运动想象脑电信号进行归一化处理。As an implementation method, the Z-score normalization method is used to normalize the motor imagination EEG signals.
作为一种实施方式，所述EEGInc模块由时间卷积、空间卷积和Inception块构成；所述时间卷积和空间卷积用于分别提取归一化处理后的运动想象脑电信号的频率特征和特定频率的空间特征；所述Inception块用于进一步提取多尺度时间信息。As an implementation manner, the EEGInc module is composed of temporal convolution, spatial convolution and Inception blocks; the temporal convolution and spatial convolution are used to respectively extract the frequency characteristics of the normalized motor imagery EEG signal. and spatial features of specific frequencies; the Inception block is used to further extract multi-scale temporal information.
作为一种实施方式，所述多头自注意力(MSA)模块通过查询、键和值这三个组件来模拟；输入至多头自注意力模块的特征通过线性变换得到查询向量、键向量和值向量；将查询向量、键向量和值向量分割成若干个子向量，取查询向量、键向量和值向量的子向量各一个构成一个注意力头。As an implementation method, the multi-head self-attention (MSA) module is simulated by three components: query, key and value; the features input to the multi-head self-attention module are linearly transformed to obtain the query vector, key vector and value vector. ; Divide the query vector, key vector and value vector into several sub-vectors, and take one sub-vector of the query vector, key vector and value vector to form an attention head.
作为一种实施方式，所述时间卷积网络(TCN)模块由两个残差块堆叠而成，每个残差块由两个膨胀因果卷积组成，每个膨胀因果卷积后面依次跟着批归一化、指数线性单元和Dropout，残差块的后面都跟着一个指数线性单元；残差连接在输入和输出维度不一致时会使用一个点卷积将输入维度转换为与输出维度一致；As an implementation manner, the temporal convolutional network (TCN) module is stacked by two residual blocks. Each residual block is composed of two dilated causal convolutions. Each dilated causal convolution is followed by a batch Normalization, exponential linear unit and Dropout, the residual block is followed by an exponential linear unit; the residual connection uses a point convolution to convert the input dimension to be consistent with the output dimension when the input and output dimensions are inconsistent;
作为一种实施方式，所述融合模块由特征融合和决策级融合构成，将EEGInc模块输出的特征和MSA模块输出的特征在深度维度上进行特征融合，将EEGInc模块的输出与TCN模块输出的高维度时间特征分别作为输入，经全连接层对应得到运动想象脑电信号属于每个类别的两个预测结果，再利用可学习张量对两个预测结果进行决策级融合，可学习张量通过Sigmoid限制在[0,1]范围内。As an implementation manner, the fusion module is composed of feature fusion and decision-level fusion. The features output by the EEGInc module and the features output by the MSA module are feature fused in the depth dimension, and the output of the EEGInc module is combined with the high-level feature output by the TCN module. The dimensional and time features are used as input respectively, and the two prediction results of the motor imagery EEG signal belonging to each category are obtained through the fully connected layer. The learnable tensor is then used to perform decision-level fusion of the two prediction results. The learnable tensor is passed through Sigmoid Limited to the range [0,1].
作为一种实施方式，所述输出模块由一层Softmax构成。As an implementation manner, the output module is composed of a layer of Softmax.
本发明的第二个方面提供一种运动想象脑电信号解码系统。A second aspect of the present invention provides a motor imagery EEG signal decoding system.
一种运动想象脑电信号解码系统，其包括：A motor imagery EEG signal decoding system, which includes:
归一化处理模块，其用于对获取的运动想象脑电信号进行归一化处理；A normalization processing module, which is used to normalize the acquired motor imagination EEG signals;
类别预测模块，其用于利用训练完成的EISATC-Fusion模型解码归一化的运动想象脑电信号，预测出运动想象脑电信号的类别；其中，所述EISATC-Fusion模型由EEGInc模块、多头自注意力模块、时间卷积网络模块、融合模块及输出模块构成；A category prediction module, which is used to use the trained EISATC-Fusion model to decode the normalized motor imagery EEG signal and predict the category of the motor imagery EEG signal; wherein, the EISATC-Fusion model is composed of the EEGInc module and the multi-head automatic EEG signal. Composed of attention module, temporal convolutional network module, fusion module and output module;
模型训练模块，其用于利用两阶段训练策略训练EISATC-Fusion模型；Model training module, which is used to train the EISATC-Fusion model using a two-stage training strategy;
利用EISATC-Fusion模型解码归一化的运动想象脑电信号的过程为：The process of using the EISATC-Fusion model to decode the normalized motor imagery EEG signal is:
利用EEGInc模块提取归一化的运动想象脑电信号的频率特征、特定频率的空间特征及多尺度时间信息，得到初始时空特征；Use the EEGInc module to extract the frequency characteristics, spatial characteristics of specific frequencies and multi-scale time information of the normalized motor imagery EEG signal to obtain the initial spatio-temporal characteristics;
将初始时空特征经多头自注意力模块处理，得到全局具有长时依赖的特征；The initial spatio-temporal features are processed by the multi-head self-attention module to obtain global features with long-term dependence;
在融合模块中对初始时空特征与全局具有长时依赖的特征在深度的维度上进行特征融合，再经时间卷积网络模块提取高维度时间特征；将EEGInc模块输出的初始时空特征与时间卷积网络模块输出的高维度时间特征分别作为输入，经全连接层对应得到运动想象脑电信号属于每个类别的两个预测结果，再利用可学习张量对两个预测结果进行决策级融合。In the fusion module, the initial spatio-temporal features and global long-term dependency features are fused in the depth dimension, and then the high-dimensional time features are extracted through the temporal convolution network module; the initial spatio-temporal features output by the EEGInc module are convolved with time The high-dimensional temporal features output by the network module are used as inputs respectively. After corresponding through the fully connected layer, two prediction results of the motor imagery EEG signal belonging to each category are obtained, and then the learnable tensor is used to perform decision-level fusion of the two prediction results.
在输出模块中，将决策融合的结果输入到Softmax函数中，得到最终的运动想象脑电信号的类别。In the output module, the decision fusion results are input into the Softmax function to obtain the final motor imagery EEG signal category.
利用两阶段训练策略训练EISATC-Fusion模型的过程为：The process of training the EISATC-Fusion model using the two-stage training strategy is:
在第一阶段，训练集被用来训练模型，在验证集上使用早停策略获取最优训练模型；其中，在早停策略中，将验证集的准确率和损失作为模型训练早停的指标；In the first stage, the training set is used to train the model, and the early stopping strategy is used on the verification set to obtain the optimal training model; in the early stopping strategy, the accuracy and loss of the verification set are used as indicators for early stopping of model training. ;
在第二阶段，训练集和验证集都被用来训练第一阶段获取的模型，当模型在验证集上的损失小于或等于模型在第一阶段训练集上的最小损失，则结束第二阶段的训练。In the second stage, both the training set and the verification set are used to train the model obtained in the first stage. When the loss of the model on the verification set is less than or equal to the minimum loss of the model on the training set of the first stage, the second stage ends. training.
本发明的第三个方面提供一种计算机可读存储介质。A third aspect of the invention provides a computer-readable storage medium.
一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如上述所述的运动想象脑电信号解码方法中的步骤。A computer-readable storage medium has a computer program stored thereon. When the program is executed by a processor, the steps in the above-mentioned method for decoding motor imagery EEG signals are implemented.
本发明的第四个方面提供一种电子设备。A fourth aspect of the present invention provides an electronic device.
一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现如上述所述的运动想象脑电信号解码方法中的步骤。An electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-mentioned method for decoding motor imagery brain signals. step.
与现有技术相比，本发明的有益效果是：Compared with the prior art, the beneficial effects of the present invention are:
(1)本发明利用EISATC-Fusion模型同时提取脑电信号的时域特征、特定频率的空域特征、多尺度时间信息、具有全局性的长时依赖特征和高维度的时间特征，并且通过融合的方式充分利用了各层特征和不同模型的信息，提高了模型的分类准确率和鲁棒性，以及运动想象脑电信号的被试内和跨被试的解码性能。(1) The present invention uses the EISATC-Fusion model to simultaneously extract the time domain features of the EEG signal, the spatial domain features of specific frequencies, multi-scale time information, global long-term dependence features and high-dimensional time features, and through the fusion The method makes full use of the features of each layer and the information of different models to improve the classification accuracy and robustness of the model, as well as the intra-subject and cross-subject decoding performance of motor imagery EEG signals.
(2)本发明采用两阶段训练策略训练EISATC-Fusion模型，在第一阶段中，将验证集的准确率和损失作为模型训练早停的指标，进而在验证集上获取鲁棒性更好的模型，在第二阶段同时使用训练集和验证集训练模型，进一步提高了模型的解码性能，并且该策略可以应用于任何深度学习模型中，提高了深度学习方法解码运动想象信号的性能。(2) The present invention uses a two-stage training strategy to train the EISATC-Fusion model. In the first stage, the accuracy and loss of the verification set are used as indicators for early stopping of model training, so as to obtain better robustness on the verification set. Model, in the second stage, both the training set and the validation set are used to train the model, which further improves the decoding performance of the model, and this strategy can be applied to any deep learning model, improving the performance of the deep learning method in decoding motor imagery signals.
(3)本发明利用EISATC-Fusion模型解码归一化的运动想象脑电信号，预测运动想象脑电信号的类别，是端到端的网络设计方法，不需要复杂的人工预处理，能够进行脑机接口的在线应用。(3) The present invention uses the EISATC-Fusion model to decode the normalized motor imagery EEG signal and predict the category of the motor imagery EEG signal. It is an end-to-end network design method that does not require complex manual preprocessing and is capable of brain-computer processing. Online application of the interface.
本发明附加方面的优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本发明的实践了解到。Additional advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
附图说明Description of drawings
构成本发明的一部分的说明书附图用来提供对本发明的进一步理解，本发明的示意性实施例及其说明用于解释本发明，并不构成对本发明的不当限定。The description and drawings that constitute a part of the present invention are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention.
图1为本发明实施例的EISATC-Fusion模型的整体结构图；Figure 1 is an overall structural diagram of the EISATC-Fusion model according to the embodiment of the present invention;
图2为本发明实施例的EISATC-Fusion模型中EEGInc模块的详细结构图；Figure 2 is a detailed structural diagram of the EEGInc module in the EISATC-Fusion model according to the embodiment of the present invention;
图3为本发明实施例的EEGInc模块中Inception块的详细结构图；Figure 3 is a detailed structural diagram of the Inception block in the EEGInc module according to the embodiment of the present invention;
图4为本发明实施例的EISATC-Fusion模型中MSA模块的详细结构和缩放点卷积的计算过程图；Figure 4 is a detailed structure of the MSA module and a calculation process diagram of scaled point convolution in the EISATC-Fusion model according to the embodiment of the present invention;
图5为本发明实施例的EISATC-Fusion模型中TCN模块的详细结构和TCN模块计算过程示意图；Figure 5 is a schematic diagram of the detailed structure of the TCN module and the calculation process of the TCN module in the EISATC-Fusion model according to the embodiment of the present invention;
图6为本发明实施例的EISATC-Fusion模型在数据集BCI-2a和BCI-2b上所有被试的平均混淆矩阵图。Figure 6 is a diagram of the average confusion matrix of all subjects on the data sets BCI-2a and BCI-2b of the EISATC-Fusion model according to the embodiment of the present invention.
具体实施方式Detailed ways
下面结合附图与实施例对本发明作进一步说明。The present invention will be further described below in conjunction with the accompanying drawings and examples.
应该指出，以下详细说明都是示例性的，旨在对本发明提供进一步的说明。除非另有指明，本文使用的所有技术和科学术语具有与本发明所属技术领域的普通技术人员通常理解的相同含义。It should be noted that the following detailed description is exemplary and is intended to provide further explanation of the present invention. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
需要注意的是，这里所使用的术语仅是为了描述具体实施方式，而非意图限制根据本发明的示例性实施方式。如在这里所使用的，除非上下文另外明确指出，否则单数形式也意图包括复数形式，此外，还应当理解的是，当在本说明书中使用术语“包含”和/或“包括”时，其指明存在特征、步骤、操作、器件、组件和/或它们的组合。It should be noted that the terms used herein are for the purpose of describing specific embodiments only, and are not intended to limit the exemplary embodiments according to the present invention. As used herein, the singular forms are also intended to include the plural forms unless the context clearly indicates otherwise. Furthermore, it will be understood that when the terms "comprises" and/or "includes" are used in this specification, they indicate There are features, steps, operations, means, components and/or combinations thereof.
实施例一Embodiment 1
本实施例提供了一种运动想象脑电信号解码方法，其具体包括如下步骤：This embodiment provides a motor imagery EEG signal decoding method, which specifically includes the following steps:
步骤1：获取运动想象脑电信号并进行归一化处理。Step 1: Obtain motor imagery EEG signals and perform normalization processing.
在本实施例中，使用Z-score标准化方法对运动想象脑电信号进行归一化处理，这样能够减少信号的波动，其计算公式如下：In this embodiment, the Z-score normalization method is used to normalize the motor imagery EEG signal, which can reduce signal fluctuations. The calculation formula is as follows:
其中，x0为标准化输出，xi为待分类的运动想象脑电信号，μ和σ2分别为训练数据的均值和方差，并且直接用于训练和测试数据的标准化。Among them, x0 is the standardized output,xi is the motor imagery EEG signal to be classified, μ and σ2 are the mean and variance of the training data respectively, and are directly used for the standardization of training and test data.
步骤2：利用训练完成的EISATC-Fusion模型解码归一化的运动想象脑电信号，预测出运动想象脑电信号的类别；其中，所述EISATC-Fusion模型由EEGInc模块、多头自注意力模块、时间卷积网络模块、融合模块及输出模块构成，如图1所示。Step 2: Use the trained EISATC-Fusion model to decode the normalized motor imagery EEG signal and predict the category of the motor imagery EEG signal; wherein, the EISATC-Fusion model consists of the EEGInc module, the multi-head self-attention module, It consists of a temporal convolutional network module, a fusion module and an output module, as shown in Figure 1.
例如：运动想象脑电信号的类别包括但不限左手、右手、双脚和舌头运动的想象。For example: the categories of motor imagination EEG signals include but are not limited to imagination of movements of the left hand, right hand, feet and tongue.
此处需要说明的是，本领域技术人员可根据实际情况来具体设置运动想象脑电信号的类别，此处不再详述。It should be noted here that those skilled in the art can specifically set the categories of motor imagery EEG signals according to actual conditions, which will not be described in detail here.
在步骤2中，利用EISATC-Fusion模型解码归一化的运动想象脑电信号的过程为：In step 2, the process of using the EISATC-Fusion model to decode the normalized motor imagery EEG signal is:
步骤2.1：利用EEGInc模块提取归一化的运动想象脑电信号的频率特征、特定频率的空间特征及多尺度时间信息，得到初始时空特征。Step 2.1: Use the EEGInc module to extract the frequency characteristics, spatial characteristics of specific frequencies and multi-scale time information of the normalized motor imagery EEG signal to obtain the initial spatiotemporal characteristics.
具体地，所述EEGInc模块由时间卷积、空间卷积和Inception块构成；所述时间卷积和空间卷积用于分别提取归一化处理后的运动想象脑电信号的频率特征和特定频率的空间特征；所述Inception块用于进一步提取多尺度时间信息。Specifically, the EEGInc module is composed of temporal convolution, spatial convolution and Inception blocks; the temporal convolution and spatial convolution are used to respectively extract the frequency characteristics and specific frequencies of the normalized motor imagery EEG signal. spatial features; the Inception block is used to further extract multi-scale temporal information.
EEGInc模块的主要结构由三个卷积层构成，具体结构如图2所示。第一层为时间卷积层，使用F1＝32个尺寸为(1,32)的卷积核，学习不同频率段的滤波器，提取运动想象脑电信号的频率信息，并且这些滤波器也扩充了数据。第二层为通道卷积，使用包含F2个尺寸为(C,1)的深度(DW)卷积学习特定频率的空间滤波器，提取出运动想象信号的空间特征。其中C为EEG的通道数，F2确定了EEGInc模块输出特征的尺寸，并且F2＝D×F1，其中D代表了连接到前一层每个时间滤波器的空间滤波的数量，D依据经验设置为2。紧接着通道卷积的是一层平均池化层，池化核的尺寸为(1,8)步长为(1,8)，将信号的采样率降到～32Hz。第三层为由DW卷积构成的Inception块，它包含了三条由DW卷积构成的支路，以及一条由池化和DW卷积构成的残差支路，其结构如图3所示。第一到第三条支路的卷积核尺寸分别为(1,Ki)、(1,2×Ki)和(1,4×Ki)，其中Ki＝4，分别实现了8Hz、4Hz和2Hz的高通滤波，同时每条支路的滤波器数为F2/4，groups＝F2/4。第四条支路采用了ResNet的结构，通过一个(3,3)的最大池化进行信息融合。为了确保输出尺寸不变，在最大池化的后面紧跟着一层卷积核为(1,1)的点卷积，并设置其滤波器个数为F2/4，然后四条分支的输出在深度上进行连接。紧接着Inception块的是第二层平均池化层，池化核的尺寸为(1,8)步长为(1,8)，将信号的采样率降到～4Hz。每一层卷积后面都跟着批归一化(Batch Normalization,BN)来提高模型的泛化能力，第二和第三层卷积后面还跟着指数线性单元(Exponential Linear Unit,ELU)激活函数赋予模型非线性表达能力。每个池化层后面都跟着一个Dropout层，并且丢弃率为0.5。The main structure of the EEGInc module consists of three convolutional layers. The specific structure is shown in Figure 2. The first layer is the temporal convolution layer, which uses F1 = 32 convolution kernels of size (1, 32) to learn filters in different frequency bands and extract the frequency information of motor imagery EEG signals, and these filters are also expanded data. The second layer is channel convolution, which uses F2 depth (DW) convolutions of size (C,1) to learn spatial filters of specific frequencies to extract the spatial features of the motor imagination signal. Where C is the number of EEG channels, F2 determines the size of the output feature of the EEGInc module, and F2 = D × F1, where D represents the number of spatial filters connected to each temporal filter of the previous layer, and D is empirically set to 2. Following the channel convolution is an average pooling layer with a pooling kernel size of (1,8) and a stride of (1,8), which reduces the sampling rate of the signal to ~32Hz. The third layer is the Inception block composed of DW convolution, which contains three branches composed of DW convolution, and a residual branch composed of pooling and DW convolution. Its structure is shown in Figure 3. The convolution kernel sizes of the first to third branches are (1,Ki ), (1,2×Ki ) and (1,4×Ki ) respectively, where Ki =4, achieving 8Hz respectively. , 4Hz and 2Hz high-pass filtering, and the number of filters in each branch is F2/4, groups=F2/4. The fourth branch adopts the structure of ResNet and performs information fusion through a (3,3) maximum pooling. In order to ensure that the output size remains unchanged, the maximum pooling is followed by a layer of point convolution with a convolution kernel of (1,1), and the number of filters is set to F2/4, and then the output of the four branches is Connect in depth. Following the Inception block is a second average pooling layer with a pooling kernel size of (1,8) and a stride of (1,8), which reduces the sampling rate of the signal to ~4Hz. Each layer of convolution is followed by Batch Normalization (BN) to improve the generalization ability of the model. The second and third layers of convolution are also followed by an Exponential Linear Unit (ELU) activation function. Model nonlinear expression ability. Each pooling layer is followed by a dropout layer with a dropout rate of 0.5.
EEGInc模块的输出为的时间序列，其中Te＝T/(8×8)为输出序列的时间长度，T为归一化的运动想象脑电信号的长度，d＝F2为序列的数据维度。The output of the EEGInc module is time series, whereTe =T/(8×8) is the time length of the output sequence, T is the length of the normalized motor imagery EEG signal, and d=F2 is the data dimension of the sequence.
步骤2.2：将初始时空特征经多头自注意力模块(MSA)处理，得到全局具有长时依赖的特征。Step 2.2: Process the initial spatio-temporal features through the multi-head self-attention module (MSA) to obtain global features with long-term dependence.
其中，所述多头自注意力模块通过查询(Q)、键(K)和值(V)这三个组件来模拟；输入至多头自注意力模块的特征通过线性变换得到查询向量、键向量和值向量；线性变换为：Among them, the multi-head self-attention module is simulated by three components: query (Q), key (K) and value (V); the features input to the multi-head self-attention module are linearly transformed to obtain the query vector, key vector and Value vector; linear transformation is:
Q＝ELU(WqLN(Ye))， (2)Q＝ELU(Wq LN(Ye )), (2)
K＝ELU(WkLN(Ye))， (3)K=ELU(Wk LN(Ye )), (3)
V＝ELU(WvLN(Ye))， (4)V=ELU(Wv LN(Ye )), (4)
其中LN为LayerNorm层，用来对输入MSA模型的数据进行归一化处理，Wq、Wk和Wv分别为向量Q、K和V的线性变换矩阵。MAS模块的详细结构如图4所示。Among them, LN is the LayerNorm layer, which is used to normalize the data input to the MSA model. Wq , Wk and Wv are the linear transformation matrices of the vectors Q, K and V respectively. The detailed structure of the MAS module is shown in Figure 4.
将查询向量、键向量和值向量分割成若干个子向量，取查询向量、键向量和值向量的子向量各一个构成一个注意力头。Divide the query vector, key vector and value vector into several sub-vectors, and take one sub-vector each of the query vector, key vector and value vector to form an attention head.
例如：将Q、K和V向量分割成h个子向量，取Q、K和V子向量各一个构成一个注意力头，通过分割我们就可以得到h个头，依据经验，设置头的个数为8。然后计算每个头的注意力权重，这些权重表示注意力头所关注的输入序列的哪个部分。我们采用缩放点积注意力计算这些权重，首先通过Q和K计算注意力值(ATsores)：For example: Divide the Q, K and V vectors into h sub-vectors, take one Q, K and V sub-vector each to form an attention head. By dividing, we can get h heads. Based on experience, set the number of heads to 8 . Attention weights are then calculated for each head, which represent which part of the input sequence the attention head focuses on. We calculate these weights using scaled dot product attention, first calculating the attention values (ATsores) through Q and K:
其中k表示键向量的维度。为了防止过拟合以及让注意力更加的突出，对计算的注意力值增加了一个Dropout的操作，并设置丢弃率为0.3。然后将注意力值与V相乘，得到缩放点积注意力的输出：where k represents the dimension of the key vector. In order to prevent overfitting and make attention more prominent, a Dropout operation is added to the calculated attention value, and the dropout rate is set to 0.3. The attention value is then multiplied by V to get the output of the scaled dot product attention:
Attention(Q,K,V)＝ATscores·V。 (6)Attention(Q,K,V)=ATscores·V. (6)
将每个注意力头的输出力拼接起来即可得到多头自注意力的输出：By splicing the output of each attention head together, we can get the output of multi-head self-attention:
其中h＝8代表了头的个数，分别代表了第i个头的查询、键值和值，Tm为MSA输出序列的长度，并且Tm＝Te。Among them, h=8 represents the number of heads, represent the query, key value and value of the i-th header respectively, Tm is the length of the MSA output sequence, and Tm =Te .
步骤2.3：将初始时空特征与全局具有长时依赖的特征在深度的维度上进行特征融合，再经时间卷积网络(TCN)模块提取高维度时间特征。Step 2.3: Fusion of initial spatio-temporal features and global features with long-term dependence in the depth dimension, and then extract high-dimensional temporal features through the temporal convolution network (TCN) module.
将EEGInc模块输出的特征和MSA模块输出的特征在深度维度上进行融合，得到表征能力更全面的融合特征，将融合特征输入到TCN模块，提取更高维度的时间依赖信息。The features output by the EEGInc module and the features output by the MSA module are fused in the depth dimension to obtain fused features with more comprehensive representation capabilities. The fused features are input to the TCN module to extract higher-dimensional time-dependent information.
具体地，时间卷积网络(TCN)模块由两个残差块堆叠而成，每个残差块由两个膨胀因果卷积组成，每个膨胀因果卷积后面依次跟着BN、ELU和Dropout(p＝0.3),残差块的后面都跟着一个ELU，具体结构如图5A所示。TCN模块的残差连接在输入和输出维度不一致时会使用一个点卷积将输入维度转换为与输出维度一致，但是本发明使用了identitymapping，因为本发明的模型的TCN模块的输入和输出的维度都是2×F2。Specifically, the Temporal Convolutional Network (TCN) module is stacked by two residual blocks. Each residual block consists of two dilated causal convolutions. Each dilated causal convolution is followed by BN, ELU and Dropout ( p=0.3), the residual block is followed by an ELU, and the specific structure is shown in Figure 5A. The residual connection of the TCN module will use a point convolution to convert the input dimension to be consistent with the output dimension when the input and output dimensions are inconsistent. However, the present invention uses identitymapping because the dimensions of the input and output of the TCN module of the model of the present invention are Both are 2×F2 .
TCN模块中残差块的膨胀因果卷积的膨胀系数随着堆叠残差块数量L呈指数增长，例如，第i个残差块的膨胀系数为2i-1因此TCN模块的感受野(Receptive Filed Size,RFS)为：The expansion coefficient of the causal convolution of the residual block in the TCN module grows exponentially with the number of stacked residual blocks L. For example, the expansion coefficient of the i-th residual block is 2i-1 , so the receptive field of the TCN module (Receptive Filed Size (RFS) is:
RFS＝1+2(Kt-1)(2L-1)， (8)RFS＝1+2(Kt -1)(2L -1), (8)
其中Kt为TCN模块中卷积层的核的尺寸。在EISATC-Fusion模型中TCN模块的输入的序列长度为15，堆叠的残差块数量L＝2，只有当感受野的长度大于输入序列的长度时才不会有信息损失，因此在本实施例中设置Kt＝4，此时RFS＝19＞15。TCN模块的计算过程示意图如图5中的B所示。where Kt is the size of the kernel of the convolutional layer in the TCN module. In the EISATC-Fusion model, the input sequence length of the TCN module is 15, and the number of stacked residual blocks L=2. There will be no information loss only when the length of the receptive field is greater than the length of the input sequence. Therefore, in this embodiment Set Kt = 4 in , at this time RFS = 19>15. The schematic diagram of the calculation process of the TCN module is shown in B in Figure 5.
步骤2.4：将EEGInc模块输出的初始时空特征与时间卷积网络模块输出的高维度时间特征分别作为输入，经全连接层对应得到运动想象脑电信号属于每个类别的两个预测结果，再利用可学习张量对两个预测结果进行决策级融合。Step 2.4: Use the initial spatio-temporal features output by the EEGInc module and the high-dimensional temporal features output by the temporal convolution network module as input respectively, and obtain two prediction results of the motor imagery EEG signal belonging to each category through the fully connected layer, and then use Learnable tensors perform decision-level fusion of two predictions.
将EEGInc模块和TCN模块的输出输入全连接层得到每个类别的预测结果和其中NC为类别个数，然后创建一个可学习张量β作为两个预测结果的融合系数，并通过Sigmoid将融合系数限制在[0,1]，则融合后的预测结果为Input the output of the EEGInc module and TCN module into the fully connected layer to obtain the prediction results of each category. and Where NC is the number of categories, then create a learnable tensor β as the fusion coefficient of the two prediction results, and limit the fusion coefficient to [0,1] through Sigmoid, then the fusion prediction result is
P＝f(β)·Pe+(1-f(β))·Pt， (9)P＝f(β)·Pe +(1-f(β))·Pt , (9)
其中f()为Sigmoid函数。where f() is the Sigmoid function.
步骤2.5：在输出模块中，将决策融合的结果输入到Softmax函数中，得到最终的运动想象脑电信号的类别。Step 2.5: In the output module, input the decision fusion result into the Softmax function to obtain the final motor imagery EEG signal category.
在具体实施过程中，在训练EISATC-Fusion模型之前，将原始脑电信号使用Z-score标准化方法进行归一化后，再将归一化后的数据进行划分，将数据划分为训练数据和测试数据，将训练数据中的80％作为训练集剩余的20％作为验证集，测试数据作为测试集。In the specific implementation process, before training the EISATC-Fusion model, the original EEG signals are normalized using the Z-score standardization method, and then the normalized data are divided into training data and testing data. Data, 80% of the training data is used as the training set, the remaining 20% is used as the validation set, and the test data is used as the test set.
使用5折交叉验证，记录测试集的准确率，并使用5折交叉验证的平均准确率作为每个被试解码性能的衡量标准。使用Adam优化器训练模型，学习率、β1、β2和权重衰减分别为0.001、0.9、0.999和0.001。使用交叉熵作为整个框架的损失函数。Using 5-fold cross-validation, record the accuracy of the test set, and use the average accuracy of the 5-fold cross-validation as a measure of each subject's decoding performance. The model is trained using the Adam optimizer, with the learning rate, β1 , β2 and weight decay being 0.001, 0.9, 0.999 and 0.001 respectively. Use cross-entropy as the loss function for the entire framework.
所述EISATC-Fusion模型训练过程分为两个阶段：The EISATC-Fusion model training process is divided into two stages:
在第一阶段，训练集被用来训练模型，在验证集上使用早停策略获取最优训练模型；其中，在早停策略中，将验证集的准确率和损失作为模型训练早停的指标；In the first stage, the training set is used to train the model, and the early stopping strategy is used on the verification set to obtain the optimal training model; in the early stopping strategy, the accuracy and loss of the verification set are used as indicators for early stopping of model training. ;
在第二阶段，训练集和验证集都被用来训练第一阶段获取的模型，当模型在验证集上的损失小于或等于模型在第一阶段训练集上的最小损失，则结束第二阶段的训练。In the second stage, both the training set and the verification set are used to train the model obtained in the first stage. When the loss of the model on the verification set is less than or equal to the minimum loss of the model on the training set of the first stage, the second stage ends. training.
在早停策略中，准确率作为主要的早停指标，当验证集上的准确率在预定的早停轮数内不再改进，则停止训练，同时考虑模型在验证集上的loss，如果当前训练轮次在验证集上的损失为最优损失，且对应的准确率高于或等于之前的最优准确率，则重置早停轮数。在这个过程中验证准确率最高的模型被保存。In the early stopping strategy, accuracy is the main early stopping indicator. When the accuracy on the verification set no longer improves within the predetermined number of early stopping rounds, training will be stopped. At the same time, the loss of the model on the verification set will be considered. If the current If the loss of the training round on the validation set is the optimal loss, and the corresponding accuracy is higher than or equal to the previous optimal accuracy, the number of early stopping rounds will be reset. In this process, the model with the highest verification accuracy is saved.
例如设置第一阶段训练轮数为3000，预设的早停轮数为300，第二阶段的训练轮数为800，batch size为64。For example, set the number of training rounds in the first stage to 3000, the default number of early stopping rounds to 300, the number of training rounds in the second stage to 800, and the batch size to 64.
下面使用的数据集和实验结果描述如下：The datasets used below and the experimental results are described below:
数据集：采用两个常用的MI数据集，即BCI Competition IV Dataset 2a和BCICompetition IV Dataset 2b进行模型性能评估。每个数据集采用不同的实验范式，并且有不同数量的样本数据，能够验证本发明方法的泛化能力。Datasets: Two commonly used MI data sets, namely BCI Competition IV Dataset 2a and BCICompetition IV Dataset 2b, are used for model performance evaluation. Each data set adopts different experimental paradigms and has different amounts of sample data, which can verify the generalization ability of the method of the present invention.
BCI-2a数据集由9个受试者的脑电图数据组成，有四种不同的运动想象任务，包括左手、右手、双脚和舌头运动的想象。每个受试者在不同日期记录两次，每次由288次试验组成。使用22个Ag/AgCl电极以250Hz的采样率记录EEG。视觉提示出现后0-4秒的时间段被用作一条样本数据。The BCI-2a data set consists of EEG data from nine subjects with four different motor imagination tasks, including left hand, right hand, foot and tongue movement imagination. Each subject was recorded twice on different days, each consisting of 288 trials. EEG was recorded using 22 Ag/AgCl electrodes at a sampling rate of 250Hz. The period of 0-4 seconds after the visual cue appeared was used as a piece of sample data.
BCI-2b数据集由9个受试者的脑电图数据组成，有二种不同的运动想象任务，包括左手、右手。使用3个三个双极脑电通道(C3、Cz和C4)以250Hz的采样率记录EEG。视觉提示出现后0-4秒的时间段被用作一条样本数据。The BCI-2b data set consists of EEG data of 9 subjects, with two different motor imagination tasks, including left and right hands. EEG was recorded using 3 bipolar EEG channels (C3, Cz and C4) at a sampling rate of 250Hz. The period of 0-4 seconds after the visual cue appeared was used as a piece of sample data.
实验结果与分析：Experimental results and analysis:
为了验证本发明的有效性和通用性，在公开的数据集上进行了跨范式实验、消融实验、训练方法对比实验和跨被试实验。In order to verify the effectiveness and versatility of the present invention, cross-paradigm experiments, ablation experiments, training method comparison experiments and cross-subject experiments were conducted on public data sets.
(1)跨范式实验(1) Cross-paradigm experiment
将本发明的方法在两个数据集上进行了被试内的性能评估，并与一些先进的算法做了对比实验。每个被试的分类准确率，所有被试分类准确率的平均值和标准差，所有被试的平均kappa值，所提方法与先进算法的威尔逊符号秩检验的p值都被呈现在了表1和2中。最高的准确率和kappa值与最小的标准差都被突出显示。结果表明本发明方法的性能优于其他的方法。The method of the present invention was evaluated within subjects on two data sets, and comparative experiments were conducted with some advanced algorithms. The classification accuracy of each subject, the mean and standard deviation of the classification accuracy of all subjects, the average kappa value of all subjects, and the p value of the Wilson signed rank test of the proposed method and the advanced algorithm are presented in the table 1 and 2. The highest accuracy and kappa values and the smallest standard deviation are highlighted. The results show that the performance of the method of the present invention is better than other methods.
表1提出的模型与现有方法在BCI-2a上的被试内分类结果Table 1 Intra-subject classification results of the proposed model and existing methods on BCI-2a
表2提出的模型与现有方法在BCI-2b上的被试内分类结果Table 2 Intra-subject classification results of the proposed model and existing methods on BCI-2b
本发明的方法性能相对于其他的先进算法都有显著的提升(p<0.05)，具有最高的平均准确率和kappa值，以及最小的标准差，并且大多数被试的准确率都是最高的。本发明的方法在两个数据集上分类的混淆矩阵如图6所示。这些结果表明本发明的方法能够在实现高分类性能的同时具有更好的鲁棒性。The performance of the method of the present invention is significantly improved compared to other advanced algorithms (p<0.05), with the highest average accuracy, kappa value, and the smallest standard deviation, and the accuracy of most subjects is the highest . The confusion matrix classified by the method of the present invention on the two data sets is shown in Figure 6. These results show that the method of the present invention can achieve high classification performance while having better robustness.
(2)消融实验(2)Ablation experiment
EISATC-Fusion模型主要由四个模块组成，为了探究每个模块对模型分类性能的贡献，本发明在BCI-2a上进行了消融实验。每个被试的准确率以及所有被试的平均指标都呈现在了表3中，最优的数据都被突出显示。The EISATC-Fusion model mainly consists of four modules. In order to explore the contribution of each module to the model classification performance, the present invention conducted an ablation experiment on BCI-2a. The accuracy of each subject and the average index of all subjects are presented in Table 3, and the best data are highlighted.
表3在BCI-2a上的消融实验结果(EI：EEGInc模块，FF：特征融合，DF：决策融合，FM：融合模块)Table 3 Ablation experimental results on BCI-2a (EI: EEGInc module, FF: feature fusion, DF: decision fusion, FM: fusion module)
当只使用EEGInc模块时，它的平均准确率比EEGNet提高了2.43％。在EEG-Inc模块的基础上加上TCN模块让模型平均准确率增加了6.03％。MSA模块与其他模块组合的时候，准确率都比只使用EEGInc模块时高。在模型中加入融合模块模型的平均准确率提高了4.51％，并且标准差也极大的减小。以上结果表明，EISATCN-Fusion模型中的每个模块都对提高模型的解码性能有贡献。When only using the EEGInc module, its average accuracy improves by 2.43% over EEGNet. Adding the TCN module to the EEG-Inc module increased the average accuracy of the model by 6.03%. When the MSA module is combined with other modules, the accuracy is higher than when only the EEGInc module is used. The average accuracy of the model increased by 4.51% after adding the fusion module to the model, and the standard deviation was also greatly reduced. The above results show that each module in the EISATCN-Fusion model contributes to improving the decoding performance of the model.
(3)训练方法对比实验(3) Comparison experiment of training methods
针对本实施例的两阶段训练策略在BCI-2a上使用提出的模型和其他先进的算法进行了实验，所有被试的平均准确率和标准差以及平均kappa值(括号中的内容)都呈现在了表4中。For the two-stage training strategy of this embodiment, experiments were conducted on BCI-2a using the proposed model and other advanced algorithms. The average accuracy and standard deviation of all subjects and the average kappa value (content in brackets) are presented in in Table 4.
表4提出的模型和其他先进的算法在BCI-2a上的使用不同策略进行训练的实验结果Table 4 Experimental results of training the proposed model and other advanced algorithms on BCI-2a using different strategies
结果表明，无论是使用一阶段训练策略还是使用二阶段训练策略，将损失作为模型训练的早停指标都能够提高模型的脑电信号解码性能，证明本发明改进的模型训练策略是有效的。The results show that whether using a one-stage training strategy or a two-stage training strategy, using loss as an early stopping indicator for model training can improve the EEG signal decoding performance of the model, proving that the improved model training strategy of the present invention is effective.
(4)跨被试实验(4) Cross-subject experiment
将本发明的方法在BCI-2a上进行了跨被试的解码性能评估，并与一些先进的算法做了对比实验。每个被试的分类准确率，所有被试的平均指标都被呈现在了表5中。最优的数据都被突出显示。The method of the present invention was evaluated on the decoding performance across subjects on BCI-2a, and a comparative experiment was conducted with some advanced algorithms. The classification accuracy of each subject and the average index of all subjects are presented in Table 5. The best data are highlighted.
表5提出的模型与现有方法在BCI-2a上的跨被试分类结果Table 5 Cross-subject classification results of the proposed model and existing methods on BCI-2a
本实施例的该方法性能相对于其他的先进算法都有显著的提升(p<0.05)，具有最高的平均准确率和kappa值，以及最小的标准差，并且大多数被试的准确率都是最高的。这表明本发明的方法相比于其他的方法具有更好的鲁棒性和泛化性，能够在脑机接口实际应用中在面对一个全新的被试时具有更高的分类性能。The performance of this method in this embodiment is significantly improved compared to other advanced algorithms (p<0.05), with the highest average accuracy and kappa value, and the smallest standard deviation, and the accuracy of most subjects is Highest. This shows that the method of the present invention has better robustness and generalization than other methods, and can have higher classification performance when facing a brand-new subject in the practical application of brain-computer interface.
本实施例针对运动想象脑电信号的非平稳性和个体差异性导致的解码精度低的问题，提出EISATC-Fusion融合模型，模型主要包括四个部分，EEGInc模块提取信的时域、空域和多尺度信息，使用多头自注意力(MSA)模块提取全局具有长时依赖的特征，融合EEGInc和MSA模块的输出，并输入到时间卷积网络(TCN)模块进一步提取脑电信号的高维度时间特征，融合EEGInc和TCN模块的输出实现多个模型的决策级融合。为进一步提高模型的分类性能，使用两阶段训练训练策略对提出的模型进行训练，将验证集的损失和准确率作为模型训练的早停指标。本实施例为运动想象脑电信号解码提供精度高且鲁棒性好的方法。This embodiment proposes an EISATC-Fusion fusion model to address the problem of low decoding accuracy caused by the non-stationarity and individual differences of motor imagery EEG signals. The model mainly includes four parts. The EEGInc module extracts the time domain, spatial domain and multi-dimensional information of the signal. For scale information, the multi-head self-attention (MSA) module is used to extract global features with long-term dependencies, the outputs of the EEGInc and MSA modules are fused, and input to the temporal convolution network (TCN) module to further extract high-dimensional temporal features of the EEG signal. , merging the output of EEGInc and TCN modules to achieve decision-level fusion of multiple models. In order to further improve the classification performance of the model, the proposed model is trained using a two-stage training strategy, and the loss and accuracy of the validation set are used as early stopping indicators for model training. This embodiment provides a method with high accuracy and good robustness for motor imagery EEG signal decoding.
实施例二Embodiment 2
本实施例提供了一种运动想象脑电信号解码系统，其具体包括如下模块：This embodiment provides a motor imagery EEG signal decoding system, which specifically includes the following modules:
归一化处理模块，其用于对获取的运动想象脑电信号进行归一化处理；A normalization processing module, which is used to normalize the acquired motor imagination EEG signals;
类别预测模块，其用于利用训练完成的EISATC-Fusion模型解码归一化的运动想象脑电信号，预测出运动想象脑电信号的类别；其中，所述EISATC-Fusion模型由EEGInc模块、多头自注意力模块、时间卷积网络模块、融合模块及输出模块构成；A category prediction module, which is used to use the trained EISATC-Fusion model to decode the normalized motor imagery EEG signal and predict the category of the motor imagery EEG signal; wherein, the EISATC-Fusion model is composed of the EEGInc module and the multi-head automatic EEG signal. Composed of attention module, temporal convolutional network module, fusion module and output module;
模型训练模块，其用于利用两阶段训练策略训练EISATC-Fusion模型；Model training module, which is used to train the EISATC-Fusion model using a two-stage training strategy;
利用EISATC-Fusion模型解码归一化的运动想象脑电信号的过程为：The process of using the EISATC-Fusion model to decode the normalized motor imagery EEG signal is:
利用EEGInc模块提取归一化的运动想象脑电信号的频率特征、特定频率的空间特征及多尺度时间信息，得到初始时空特征；Use the EEGInc module to extract the frequency characteristics, spatial characteristics of specific frequencies and multi-scale time information of the normalized motor imagery EEG signal to obtain the initial spatio-temporal characteristics;
将初始时空特征经多头自注意力模块处理，得到全局具有长时依赖的特征；The initial spatio-temporal features are processed by the multi-head self-attention module to obtain global features with long-term dependence;
将初始时空特征与全局具有长时依赖的特征在深度的维度上进行特征融合，再经时间卷积网络模块提取高维度时间特征；The initial spatio-temporal features and global long-term dependency features are fused in the depth dimension, and then high-dimensional temporal features are extracted through the temporal convolution network module;
将EEGInc模块的输出与时间卷积网络模块的输出分别作为输入，经全连接层对应得到运动想象脑电信号属于每个类别的两个预测结果，再利用可学习张量对两个预测结果进行决策级融合。The output of the EEGInc module and the output of the temporal convolutional network module are used as input respectively. Through the fully connected layer, two prediction results of the motor imagery EEG signal belonging to each category are obtained, and then the learnable tensor is used to perform the two prediction results. Decision-level integration.
在输出模块中，将决策融合的结果输入到Softmax函数中，得到最终的运动想象脑电信号的类别。In the output module, the decision fusion results are input into the Softmax function to obtain the final motor imagery EEG signal category.
利用两阶段训练策略训练EISATC-Fusion模型的过程为：The process of training the EISATC-Fusion model using the two-stage training strategy is:
在第一阶段，训练集被用来训练模型，在验证集上使用早停策略获取最优训练模型；其中，在早停策略中，将验证集的准确率和损失作为模型训练早停的指标；In the first stage, the training set is used to train the model, and the early stopping strategy is used on the verification set to obtain the optimal training model; in the early stopping strategy, the accuracy and loss of the verification set are used as indicators for early stopping of model training. ;
在第二阶段，训练集和验证集都被用来训练第一阶段获取的模型，当模型在验证集上的损失小于或等于模型在第一阶段训练集上的最小损失，则结束第二阶段的训练。In the second stage, both the training set and the verification set are used to train the model obtained in the first stage. When the loss of the model on the verification set is less than or equal to the minimum loss of the model on the training set of the first stage, the second stage ends. training.
此处需要说明的是，本实施例中的各个模块与实施例一中的各个步骤一一对应，其具体实施过程相同，此处不再累述。It should be noted here that each module in this embodiment corresponds to each step in Embodiment 1, and the specific implementation process is the same, which will not be described again here.
实施例三Embodiment 3
本实施例提供了一种计算机可读存储介质，其上存储有计算机程序，该程序被处理器执行时实现如上述所述的运动想象脑电信号解码方法中的步骤。This embodiment provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the motor imagery EEG signal decoding method described above are implemented.
实施例四Embodiment 4
本实施例提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时实现如上述所述的运动想象脑电信号解码方法中的步骤。This embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the motor imaging EEG as described above is implemented. Steps in the signal decoding method.
本发明是参照本发明实施例的方法、设备(系统)和计算机程序产品的流程图和/或方框图来描述的。应理解可由计算机程序指令实现流程图和/或方框图中的每一流程和/或方框以及流程图和/或方框图中的流程和/或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和/或方框图一个方框或多个方框中指定的功能的装置。The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.
以上所述仅为本发明的优选实施例而已，并不用于限制本发明，对于本领域的技术人员来说，本发明可以有各种更改和变化。凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311330780.1ACN117609852A (en) | 2023-10-13 | 2023-10-13 | Method, system, medium and equipment for decoding motor imagery electroencephalogram signals |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311330780.1ACN117609852A (en) | 2023-10-13 | 2023-10-13 | Method, system, medium and equipment for decoding motor imagery electroencephalogram signals |
| Publication Number | Publication Date |
|---|---|
| CN117609852Atrue CN117609852A (en) | 2024-02-27 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311330780.1APendingCN117609852A (en) | 2023-10-13 | 2023-10-13 | Method, system, medium and equipment for decoding motor imagery electroencephalogram signals |
| Country | Link |
|---|---|
| CN (1) | CN117609852A (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117972395A (en)* | 2024-03-22 | 2024-05-03 | 清华大学 | Multi-channel data processing method, device, electronic device and storage medium |
| CN118260672A (en)* | 2024-04-17 | 2024-06-28 | 杭州电子科技大学 | Motor imagery decoding method based on multi-tested data combined countermeasure training |
| CN119047515A (en)* | 2024-10-30 | 2024-11-29 | 安徽大学 | Method for generating imagined voice reconstruction model and imagined voice reconstruction method |
| CN119319561A (en)* | 2024-09-29 | 2025-01-17 | 黑龙江大学 | Auxiliary device for person without arm disabilities |
| CN119475035A (en)* | 2024-11-07 | 2025-02-18 | 北京国润健康医学投资有限公司 | EEG signal cross-device decoding and classification model training method, decoding and classification method and device |
| CN119884889A (en)* | 2025-03-24 | 2025-04-25 | 北京师范大学 | Lightweight motor imagery classification method and device guided by frequency priori features |
| CN120493081A (en)* | 2025-07-18 | 2025-08-15 | 吉林大学 | Bridge scour damage identification method and system based on deep learning |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117972395A (en)* | 2024-03-22 | 2024-05-03 | 清华大学 | Multi-channel data processing method, device, electronic device and storage medium |
| CN118260672A (en)* | 2024-04-17 | 2024-06-28 | 杭州电子科技大学 | Motor imagery decoding method based on multi-tested data combined countermeasure training |
| CN119319561A (en)* | 2024-09-29 | 2025-01-17 | 黑龙江大学 | Auxiliary device for person without arm disabilities |
| CN119047515A (en)* | 2024-10-30 | 2024-11-29 | 安徽大学 | Method for generating imagined voice reconstruction model and imagined voice reconstruction method |
| CN119047515B (en)* | 2024-10-30 | 2025-01-03 | 安徽大学 | Method for generating imagined speech reconstruction model, imagined speech reconstruction method |
| CN119475035A (en)* | 2024-11-07 | 2025-02-18 | 北京国润健康医学投资有限公司 | EEG signal cross-device decoding and classification model training method, decoding and classification method and device |
| CN119884889A (en)* | 2025-03-24 | 2025-04-25 | 北京师范大学 | Lightweight motor imagery classification method and device guided by frequency priori features |
| CN120493081A (en)* | 2025-07-18 | 2025-08-15 | 吉林大学 | Bridge scour damage identification method and system based on deep learning |
| CN120493081B (en)* | 2025-07-18 | 2025-09-19 | 吉林大学 | Bridge scour damage identification method and system based on deep learning |
| Publication | Publication Date | Title |
|---|---|---|
| CN117609852A (en) | Method, system, medium and equipment for decoding motor imagery electroencephalogram signals | |
| CN113627518B (en) | Method for realizing neural network brain electricity emotion recognition model by utilizing transfer learning | |
| Zhao et al. | Deep representation-based domain adaptation for nonstationary EEG classification | |
| Islam et al. | EEG channel correlation based model for emotion recognition | |
| Chakladar et al. | EEG based emotion classification using “correlation based subset selection” | |
| Zhang et al. | Multivariate time series missing data imputation using recurrent denoising autoencoder | |
| CN113180692B (en) | Electroencephalogram signal classification and identification method based on feature fusion and attention mechanism | |
| CN116304815A (en) | Motor imagery electroencephalogram signal classification method based on self-attention mechanism and parallel convolution | |
| CN110555468A (en) | Electroencephalogram signal identification method and system combining recursion graph and CNN | |
| CN116431004B (en) | Control method and system for interactive behavior of rehabilitation robot | |
| CN118349906A (en) | Motor imagery EEG signal classification method based on multi-scale convolution and self-attention | |
| CN112200016A (en) | Electroencephalogram signal emotion recognition based on ensemble learning method AdaBoost | |
| CN113017645B (en) | A P300 Signal Detection Method Based on Atrous Convolutional Neural Network | |
| Xu et al. | A framework for motor imagery with LSTM neural network | |
| Jiang et al. | Emotion recognition via multiscale feature fusion network and attention mechanism | |
| CN107766898A (en) | The three classification mood probabilistic determination methods based on SVM | |
| CN117332300A (en) | Motor imagery electroencephalogram classification method based on self-attention improved domain adaptation network | |
| CN117874570A (en) | EEG signal multi-classification method, device and medium based on hybrid attention mechanism | |
| Lin et al. | Motor imagery EEG task recognition using a nonlinear Granger causality feature extraction and an improved Salp swarm feature selection | |
| US12383206B2 (en) | System and method for testing for SARS-COV-2/COVID-19 based on wearable medical sensors and neural networks | |
| CN114027786A (en) | Sleep-disordered breathing detection method and system based on self-supervised memory network | |
| Li et al. | Self-supervised contrastive learning for EEG-based cross-subject motor imagery recognition | |
| Mohdiwale et al. | Investigating Feature Ranking Methods for Sub‐Band and Relative Power Features in Motor Imagery Task Classification | |
| Chaudhary et al. | A two-stage transformer based network for motor imagery classification | |
| CN120203520A (en) | Obstructive sleep apnea detection method, system, storage medium and electronic device based on multi-scale convolutional neural network |
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |